RDF to N-Quads

RDF to N-Quads#

Recursively searches for ZIP files containing JSON-LD data and converts the content to N-Quads format.

Usage#

uv run python -m oc_meta.run.migration.rdf_to_nquads <input_dir> <output_dir> [options]

Parameters#

Parameter

Required

Default

Description

input_dir

Yes

-

Directory containing ZIP files (searched recursively)

output_dir

Yes

-

Output directory for converted .nq files

-m, --mode

No

all

Mode: all for all ZIP files, data for entity data only, prov for provenance only

-w, --workers

No

CPU count

Number of worker processes

-c, --compress

No

disabled

Compress output files using 7z format

Modes#

All mode (default)#

Processes all ZIP files, both entity data and provenance.

uv run python -m oc_meta.run.migration.rdf_to_nquads /srv/oc_meta/rdf /data/nquads

Data mode#

Searches for numeric ZIP files (e.g., 1000.zip, 2000.zip) excluding se.zip and files inside prov/ directories.

uv run python -m oc_meta.run.migration.rdf_to_nquads /srv/oc_meta/rdf /data/meta_nquads --mode data

Provenance mode#

Searches for se.zip files in prov/ directories. These contain provenance snapshots.

uv run python -m oc_meta.run.migration.rdf_to_nquads /srv/oc_meta/rdf /data/provenance_nquads --mode prov

Compression#

By default, output files are written as plain text .nq files. Use the --compress flag to compress each output file individually using 7z format:

uv run python -m oc_meta.run.migration.rdf_to_nquads /srv/oc_meta/rdf /data/nquads --compress

Each N-Quads file is compressed into its own .nq.7z archive. 7z offers better compression ratios than ZIP, reducing storage requirements for large datasets.

Process#

  1. Recursively finds ZIP files based on the selected mode

  2. For each archive, extracts the JSON-LD content

  3. Converts JSON-LD to N-Quads using rdflib

  4. Writes the output to a flat directory with filenames derived from the path

Output naming#

Output filenames are derived from the relative path of the source file, with path separators replaced by dashes:

  • Input: ra/0610/10000/1000/prov/se.zip → Output: ra-0610-10000-1000-prov-se.nq

  • Input: br/060/10000/1000.zip → Output: br-060-10000-1000.nq

With --compress enabled:

  • Input: br/060/10000/1000.zip → Output: br-060-10000-1000.nq.7z

Examples#

Convert all RDF data:

uv run python -m oc_meta.run.migration.rdf_to_nquads /srv/oc_meta/rdf /data/all_nquads \
    --workers 8

Convert entity data only:

uv run python -m oc_meta.run.migration.rdf_to_nquads /srv/oc_meta/rdf /data/meta_nquads \
    --mode data --workers 8

Convert provenance only:

uv run python -m oc_meta.run.migration.rdf_to_nquads /srv/oc_meta/rdf /data/provenance_nquads \
    --mode prov --workers 8

Convert with 7z compression:

uv run python -m oc_meta.run.migration.rdf_to_nquads /srv/oc_meta/rdf /data/compressed_nquads \
    --compress --workers 8

Report#

At completion, the script reports the number of successfully processed files and failures:

Final report
  Success: 12345
  Failed:  0