Generate CSV

The CSV generator creates a CSV dump from the RDF data.

Usage

uv run python -m oc_meta.run.meta.generate_csv -c <CONFIG> -o <OUTPUT_DIR> [OPTIONS]

Required arguments

Argument	Description
-c, --config	Path to Meta configuration file
-o, --output_dir	Directory where CSV files will be stored

Optional arguments

Argument	Default	Description
--redis-host	localhost	Redis server hostname
--redis-port	6379	Redis server port
--redis-db	2	Redis database number for caching
--workers	4	Number of parallel workers
--clean	-	Clear checkpoint and Redis cache before starting

Example

uv run python -m oc_meta.run.meta.generate_csv \
    -c meta_config.yaml \
    -o /data/csv_dump \
    --workers 8

Input

The script reads RDF data from the directory specified by output_rdf_dir in the configuration file. It expects:

• JSON-LD files compressed in ZIP archives
• Standard OpenCitations Meta directory structure (br/, ra/, id/, ar/, re/)
• Files organized by supplier prefix and numeric ranges

The script only processes bibliographic resources (BR), but resolves related entities:

• Identifiers (ID): DOI, PMID, PMCID, ISBN, ISSN, etc.
• Responsible agents (RA): Authors, editors, publishers with their identifiers
• Agent roles (AR): Links between BR and RA with ordering via hasNext
• Resource embodiments (RE): Page numbers

Output

CSV files

Output files are named output_1.csv, output_2.csv, etc., with a maximum of 3000 rows per file.

The CSV format matches the standard Meta input format:

Column	Description
id	Space-separated identifiers (OMID + external IDs)
title	Publication title
author	Semicolon-separated authors in format Family, Given [identifiers]
issue	Issue number
volume	Volume number
venue	Venue title with identifiers
page	Page range (e.g., 123-456)
pub_date	Publication date
type	Publication type (e.g., journal article, book chapter)
publisher	Publisher with identifiers
editor	Semicolon-separated editors

Checkpoint file

The script creates processed_br_files.txt in the output directory to track which RDF files have been processed. This enables resumability: if the script is interrupted, it will skip already processed files on restart.

Redis cache

Processed OMIDs are stored in Redis to avoid duplicates. The cache persists across runs unless --clean is specified.

Processing details

Skipped entities

The following entity types are skipped as standalone records (they appear only as venue containers):

• Journal volumes (fabio:JournalVolume)
• Journal issues (fabio:JournalIssue)

Author ordering

Authors, editors, and publishers are ordered by following the oc:hasNext chain from agent roles. The script detects the first agent role (one not referenced by any other hasNext) and follows the chain to maintain correct ordering.

Venue hierarchy

For journal articles, the script traverses the frbr:partOf hierarchy:

Article → Issue → Volume → Journal

Extracting issue number, volume number, and journal title with identifiers.

Cycle detection

The script includes safeguards against:

• Cycles in hasNext chains (max iterations limit)
• Cycles in venue hierarchy (visited set + max depth of 5)

Warnings are printed when cycles are detected.

Resumability

The script supports resuming interrupted processing:

1. File-level checkpoint: Tracks processed RDF files in processed_br_files.txt
2. Entity-level cache: Stores processed OMIDs in Redis

To start fresh, use the --clean flag:

uv run python -m oc_meta.run.meta.generate_csv \
    -c meta_config.yaml \
    -o /data/csv_dump \
    --clean

This removes the checkpoint file and clears the Redis cache.

Performance

• Uses multiprocessing with configurable worker count
• LRU cache (2000 entries) for loaded JSON files
• Redis pipeline batching for efficient cache operations
• Progress bar shows processing status and time estimates