triplets.parser¶

CIM/RDF XML parser package.

Provides pluggable engines for parsing CIM RDF/XML to DataFrames or Arrow tables: - python_lxml_pandas (pure Python + lxml → pd.DataFrame, always available, default) - python_lxml_arrow (pure Python + lxml → Arrow RecordBatch, needs pyarrow) - cython_pugixml_arrow (Cython + pugixml C++ → Arrow RecordBatch, needs build + pyarrow)

Fallback: cython_pugixml_arrow → python_lxml_arrow → python_lxml_pandas

Usage:: from triplets.parser import parse, read_rdf df = parse([“file.xml”, “data.zip”], engine=”python_lxml_pandas”) df = parse(path, engine=”auto”) # best available table = parse(path, return_type=”arrow”)

triplets.parser.register_engine(name: str, module_or_factory: Any) → None[source]¶: Register a custom engine for future extensibility.

triplets.parser.get_engine(name: str = 'auto')[source]¶: Resolve parser engine name (with aliases) and return (name, module).

triplets.parser.parse(list_of_paths_to_zip_globalzip_xml: str | List | Any, debug: bool = False, max_workers: int | None = None, engine: str = 'auto', return_type: str = 'pandas', categorical_columns: Sequence[str] | None = ('INSTANCE_ID', 'KEY'), shorten_resources: bool = True, string_type: str = 'auto') → Any[source]¶

Main entry: parse CIM RDF/XML (or zips) using chosen engine.

Parameters:

debug (bool, default False) – Enable verbose debug output (file discovery, row counts, timing in some engines, etc.). When False but the logger is at DEBUG level (logging.basicConfig(level=logging.DEBUG) or getLogger(“triplets.parser”).setLevel(logging.DEBUG)), debug output is auto-enabled.
engine (str, default "auto") – Parser engine. “auto” picks best available. Options: “python_lxml_pandas”, “python_lxml_arrow”, “cython_pugixml_arrow”.
return_type (str, default "pandas") – Output format: “pandas”, “arrow”, or “polars”.
categorical_columns (tuple or None, default ("INSTANCE_ID", "KEY")) – Columns to dictionary-encode for memory savings. Pass None to disable.
shorten_resources (bool, default True) – Shorten http(s) resource values to their #fragment (CIM instance data convention). Pass False for lossless URIs (e.g. RDFS schema parsing); only the python engines support this.
string_type (str, default "auto") – Arrow layout of the ID and VALUE string columns (arrow/polars output, and pandas via ArrowDtype): “utf8” (32-bit offsets), “large_utf8” (64-bit), or “string_view” (polars’ native layout, adopted zero-copy; needs pyarrow >= 16). “auto” picks the layout the return_type adopts zero-copy: string_view for polars, utf8 otherwise. Dictionary-encoded columns are unaffected (consumers use the indices). Ignored by the pandas engine (python_lxml_pandas).

triplets.parser.parse_batches(list_of_paths_to_zip_globalzip_xml: str | List | Any, debug: bool = False, engine: str = 'auto', shorten_resources: bool = True, max_workers: int | None = None) → Any[source]¶

Parse CIM RDF/XML lazily into a pyarrow.RecordBatchReader.

One RecordBatch per XML file, produced as the reader is consumed — the dataset is never materialized in Python (out-of-core ingest, e.g. the duckdb read_rdf). Fixed all-utf8 schema [ID, KEY, VALUE, INSTANCE_ID]: no dictionary encoding and no string_type layout, because per-file dictionaries would differ batch to batch and database consumers re-encode internally anyway.

max_workers parses up to that many files ahead on a thread pool — a bounded, in-order prefetch: multi-file ingest parallelizes while memory stays bounded by max_workers+1 batches (None = fully sequential, one batch alive at a time). Batch order always follows file order.

Requires an arrow parser engine (“auto” resolves one whenever pyarrow is installed); raises ValueError otherwise — no silent pandas fallback.

triplets.parser.read_rdf(*args: Any, **kwargs: Any) → Any[source]¶: Alias for parse (for pandas.read_rdf registration).

triplets.parser.python_lxml_pandas¶

python_lxml_pandas engine: pure Python + lxml → list-of-tuples → pandas DataFrame.

The default parser engine. Requires only lxml + pandas (core deps, no pyarrow). Restored from the original rdf_parser.py load_RDF_to_list logic with fixes: - rdf:nodeID support (parity with arrow engines) - Empty string instead of None for missing values (parity with arrow engines) - Namespace map None key → “” (lxml uses None for default namespace)

triplets.parser.python_lxml_pandas.load_rdf_to_dataframe(path_or_fileobject: str | IO, debug: bool = False, shorten_resources: bool = True) → DataFrame[source]¶

Parse single RDF/XML file to pandas DataFrame using lxml + list-of-tuples.

This is the old proven path: lxml parse → iterate → build Python list → pd.DataFrame. No pyarrow dependency.

triplets.parser.python_lxml_arrow¶

python_lxml_arrow engine: pure Python + lxml + pyarrow streaming to RecordBatch.

Uses lxml for XML parsing but streams to Arrow StringBuilder builders instead of Python lists, producing pa.RecordBatch. Better for polars interop and dictionary-encoding (categorical columns). Requires pyarrow.

triplets.parser.python_lxml_arrow.load_rdf_to_dataframe(path_or_fileobject: str | IO, debug: bool = False, shorten_resources: bool = True) → pyarrow.RecordBatch[source]¶

Parse single RDF/XML (path or fileobj) to pyarrow RecordBatch using lxml + lists.

Streaming in the sense of column-wise collection then direct Arrow (no 4-tuple list).

triplets.parser.utils¶

Shared utilities for CIM/RDF XML parsers (python_lxml and cython_pugixml).

Extracted/adapted from rdf_parser.py and rdf_parser_lxml_arrow.py cues.

triplets.parser.utils.clean_ID(ID: Any) → str[source]¶: Removes ID prefixes used in CIM - urn:uuid:, #_, _ .

triplets.parser.utils.iter_all_xml(list_of_paths_to_zip_globalzip_xml: str | List | Any, debug: bool = False)[source]¶

Yield XML file objects and/or str paths, one at a time (lazy).

Supports str paths, file-like, .xml/.rdf, .zip (recursive). Same items and order as find_all_xml(): direct .xml/.rdf items first in input order, then zip members in zip order. Zip members are read into memory only when yielded, so a consumer that processes-and-drops each file keeps at most one member in RAM (out-of-core ingest). Zip handles this function opens are closed after their archive is exhausted.

triplets.parser.utils.find_all_xml(list_of_paths_to_zip_globalzip_xml: str | List | Any, debug: bool = False) → List[source]¶

Returns list of XML file objects and/or paths in ZIP file.

Supports str paths, file-like, .xml/.rdf, .zip (recursive). Eager form of iter_all_xml() — every zip member is read into memory up front.

triplets.parser.utils.get_namespace_map_from_root(root: Any) → dict[source]¶: Best effort ns map + xml_base from element (lxml or pygixml style).

triplets.parser.nquads¶

N-Quads reader — the inverse of the N-Quads export.

read_nquads turns N-Quads / N-Triples text (path, bytes, or file-like) back into a triplet DataFrame [ID, KEY, VALUE, INSTANCE_ID], applying the inverse of the export conventions (triplets.export.nquads_utils): urn:uuid: stripped, CIM namespace shortened, rdf:type → ‘Type’, datatype / language annotations dropped (values keep their lexical form), graph → INSTANCE_ID (absent → None).

Everything is vectorized pandas string ops. terms_to_triplets is the shared term-level conversion, also used by the SPARQL engines to decode CONSTRUCT/DESCRIBE results: the qlever engine feeds it Arrow-decoded term columns, the oxigraph engine feeds read_nquads its serialized result bytes.

triplets.parser.nquads.read_nquads(source, return_type='pandas')[source]¶

Parse N-Quads (or N-Triples) into a triplet DataFrame.

Parameters:

source (str/Path, bytes, or file-like) – Path to a .nq/.nt file, the serialized content as bytes/str, or an open file object (text or binary).
return_type (str, default "pandas") – “pandas”, “polars”, or “arrow”.

Returns:

Triplet DataFrame [ID, KEY, VALUE, INSTANCE_ID] — the round-trip inverse
of export_to_nquads (datatype annotations drop to lexical form, which is
the triplets convention (everything is a string). Lines without a graph)
term (N-Triples) get INSTANCE_ID null.

triplets.parser.nquads.terms_to_triplets(frame)[source]¶

N-Triples-form term columns → triplet values, converted in place.

frame carries columns [ID, KEY, VALUE] and optionally INSTANCE_ID (the graph term; a missing column → None, a constructed graph has no source instance). Term shapes: <iri>, _:bnode, "literal" (optionally with a ^^<datatype> / @lang suffix — dropped, the value keeps its lexical form; string escapes decoded), or bare turtle-shorthand numbers/booleans. IRIs lose urn:uuid: and the CIM namespace, rdf:type → ‘Type’.