Preprocessing

Preprocessing#

The preprocessing script filters and optimizes CSV files before they enter the main Meta pipeline. This step removes duplicates and filters out IDs that already exist in the database.

What preprocessing does#

  1. Filters existing IDs that are already in Redis or a SPARQL triplestore

  2. Removes duplicates across all input files

  3. Splits output into smaller, manageable chunks

Storage backends#

The script supports two backends for checking whether identifiers already exist.

Redis#

The default mode checks IDs against a Redis database that maps external identifiers to OMIDs. This database is generated by the meta2redis.py script from the OpenCitations Index repository.

The meta2redis.py script populates three Redis databases, but this preprocessing script only uses database 10 (db_br), which contains bibliographic resource identifiers (DOI, ISBN, PMID, etc.).

Each identifier is stored as a Redis set (using SADD), with the identifier as key and the corresponding OMIDs as set members:

doi:10.1234/example -> {omid:br/0601234}
issn:1234-5678 -> {omid:br/0605678}

The preprocessing script uses EXISTS to check if an identifier is present in the database. Redis checks are parallelized per-file across multiple workers.

SPARQL#

When a --sparql-endpoint is provided, the script queries a SPARQL triplestore (QLever) instead of Redis. This mode:

  1. Reads all CSV files in parallel

  2. Collects all unique identifiers across files

  3. Batches identifiers into SPARQL VALUES queries (batch size: 30)

  4. Executes queries in parallel using ProcessPoolExecutor (groups of 100 queries per worker)

  5. Filters out rows where all identifiers already exist

This approach minimizes network roundtrips by querying all unique identifiers at once, rather than one at a time.

Basic usage#

With Redis:

uv run python -m oc_meta.run.meta.preprocess_input <INPUT_DIR> <OUTPUT_DIR> --redis-port <PORT>

With SPARQL:

uv run python -m oc_meta.run.meta.preprocess_input <INPUT_DIR> <OUTPUT_DIR> \
  --sparql-endpoint "http://localhost:8805?access-token=TOKEN"

Example with all options#

Redis mode:

uv run python -m oc_meta.run.meta.preprocess_input input/ output/ \
  --redis-port 6379 \
  --rows-per-file 5000 \
  --redis-host 192.168.1.100 \
  --redis-db 10 \
  --workers 8

SPARQL mode:

uv run python -m oc_meta.run.meta.preprocess_input input/ output/ \
  --sparql-endpoint "http://localhost:8805?access-token=TOKEN" \
  --rows-per-file 5000 \
  --workers 24

Options#

Option

Required

Default

Description

input_dir

Yes

-

Directory containing input CSV files

output

Yes

-

Output path: directory for split files, or path ending in .csv for single file

--redis-port

Yes (Redis mode)

-

Redis port

--sparql-endpoint

Yes (SPARQL mode)

-

SPARQL endpoint URL (alternative to Redis)

--rows-per-file

No

3000

Split output into files of N rows each

--single-file

No

-

Write all output rows to a single CSV file

--redis-host

No

localhost

Redis hostname

--redis-db

No

10

Redis database number

--workers

No

4

Number of parallel workers

Either --redis-port or --sparql-endpoint must be provided. When --sparql-endpoint is set, Redis options are ignored.

--rows-per-file and --single-file are mutually exclusive. With --single-file, the output path determines the file name: if it ends in .csv, that exact path is used; otherwise, a merged.csv file is created in the given directory.

Progress display#

During execution, the script shows a progress bar with:

  • Current phase being processed

  • Progress percentage and count (e.g., 50% 5/10)

  • Elapsed time

  • Estimated time remaining

Output report#

When finished, the script prints a summary table:

               Processing Report
┏━━━━━━━━━━━━━━��━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Metric                       ┃ Value    ┃
┡━━━━━━━━━��━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total input files processed  │ 10       │
│ Total input rows             │ 150000   │
│ Rows discarded (duplicates)  │ 12500    │
│ Rows discarded (existing IDs)│ 8200     │
│ Rows written to output       │ 129300   │
│                              │          │
│ Duplicate rows %             │ 8.3%     │
│ Existing IDs %               │ 5.5%     │
│ Processed rows %             │ 86.2%    │
└──────��───────────────────────┴──────────┘