triplets.validation package

Submodules

triplets.validation.context module

Optional context enrichment for violations frames.

The slower opt-in pass (validate(..., context=True) or a standalone enrich call) that adds human/schema context columns to the canonical violations frame: which instance/file the object came from, what the object is, what the shape says about itself, and what the export schema defines for the class and attribute. Every source is optional — absent sources leave their columns null, so the output schema is stable.

All lookups are vectorized maps built once from the sources — cost is O(data), not O(violations × data).

triplets.validation.context.enrich(violations, data=None, shapes=None, rdf_map=None)

Return a copy of the violations frame with ENRICHMENT_COLUMNS added.

Parameters:

• violations (DataFrame in VIOLATION_COLUMNS (any engine's output).)

• data (triplet DataFrame, optional) – Source data — fills INSTANCE_ID / INSTANCE_LABEL (the parsed file name) / OBJECT_TYPE / OBJECT_NAME. An object present in several instances is attributed to its first occurrence.

• shapes (CompiledShapes or shape source, optional) – Fills SHAPE_NAME / SHAPE_DESCRIPTION (sh:name / sh:description of the source shape; a property shape without its own inherits the parent NodeShape’s). Pass the same shapes object the validation ran with — anonymous property shapes get fresh blank-node ids per parse, so a re-parsed graph cannot be matched to the violations.

• rdf_map (dict or str, optional) – Export schema — fills SCHEMA_DESCRIPTION / SCHEMA_MULTIPLICITY for the violation’s KEY and CLASS_DESCRIPTION for the object’s type.

triplets.validation.locations module

Exact source locations for violations — one grep-style pass over the CIM/XML.

RDF objects carry no text coordinates through the triplets frame, so locations are recovered from the original files at export time: each source file is read once and every rdf:ID / rdf:about definition is indexed with its line number — one sequential pass per file, cost independent of how many IDs are wanted. For each violation, the violated property’s own position is then searched inside that object’s text window, so the annotation lands on the offending property element, not just the object.

locate_violations is the public pass: it stamps LOCATION_COLUMNS (SOURCE_URI, SOURCE_LINE, SOURCE_SNIPPET — the located line’s text) onto a violations frame — both the SARIF and the sh:ValidationReport exports call it when given sources=, and it is exposed as violations.shacl.locate(sources=...).

The parse/validate hot paths are untouched: nothing here runs unless sources are handed to an export. When the same object is defined in several files (rdf:about continuation across profiles), the first definition wins — a violated KEY living in a later profile file falls back to the definition position. Positions are whole lines (1-based), deliberately: RDF text serializations put one statement per line, so the line IS the reference — columns add per-format anchor rules (byte vs UTF-16 units, element shapes) and broke SARIF viewers, for no extra information. Line-only is also what keeps the pass extensible beyond XML: IDs and KEYs appear as literal substrings in every serialization (N-Quads, Turtle, JSON-LD).

triplets.validation.locations.locate(wanted, sources)

Find the wanted objects in the source XML.

Parameters:

• wanted (dict {ID: set of KEYs}) – Objects to locate; the KEY sets name the violated properties whose own lines are worth pinpointing (may be empty).

• sources (str/Path to .xml/.rdf/.zip, file-like, or a list of those —) – the same shapes parse()/read_rdf accept.

Returns:

dict {ID – “keyLines”: {KEY: (line, snippet)}}} IDs not present in the sources are absent from the result.

Return type:

{“uri”: str, “startLine”: int, “snippet”: str,

triplets.validation.locations.locate_violations(violations, sources)

Stamp LOCATION_COLUMNS onto a violations frame — one locate() pass.

Per row (ID, KEY): the violated property element’s own position when it was found inside the object’s text window, else the object definition’s; rows whose object is not in the sources get nulls.

triplets.validation.sarif module

SARIF 2.1.0 export of validation results.

Turns the canonical violations frame into a SARIF log any SARIF consumer can ingest (GitHub, SonarQube, VS Code viewers, …). Grouping is the default: a model with 100k identical findings is one issue — one result per rule with occurrenceCount and sample instances (first 3 + last 3) — so reports stay reviewable; group=False emits one result per violation.

RDF objects have no text coordinates, so results point at the model through logicalLocations (object id / type / name) plus the source file as artifactLocation when the enrichment pass can trace it. Passing data/shapes/rdf_map runs that pass automatically (see context.enrich).

triplets.validation.sarif.export_to_sarif(violations, data=None, shapes=None, rdf_map=None, group=True, sources=None, path=None, export_to_memory=False)

Violations frame → SARIF 2.1.0 log.

Parameters:

• violations (DataFrame in VIOLATION_COLUMNS (any engine's output).) – A frame already carrying the enrichment columns is used as-is.

• data (optional) – Run the context enrichment pass first (see context.enrich) — fills object names, source files and shape/schema descriptions into the log. Pass the same shapes object the validation ran with.

• shapes (optional) – Run the context enrichment pass first (see context.enrich) — fills object names, source files and shape/schema descriptions into the log. Pass the same shapes object the validation ran with.

• rdf_map (optional) – Run the context enrichment pass first (see context.enrich) — fills object names, source files and shape/schema descriptions into the log. Pass the same shapes object the validation ran with.

• group (bool, default True) – One result per rule (occurrenceCount + sample instances) instead of one result per violation row.

• sources (optional) – The original CIM/XML files (paths/zips/file-likes, same shapes read_rdf accepts). When given, the reported instances are located in the text (one grep-style pass per file, at export time only) and results carry a whole-line physicalLocation.region (startLine == endLine) on the violated property element’s line (or the object definition’s) — what GitHub code scanning needs to annotate lines.

• path (str or Path, optional) – Output file (default “report.sarif”). Ignored with export_to_memory.

• export_to_memory (bool, default False) – Return a BytesIO (with .name) instead of writing to disk.

triplets.validation.sarif.build_sarif(violations, group=True, sources=None)

Violations frame → SARIF document dict (I/O only when sources is given — the locate_violations pass reads just the source files; a frame already carrying LOCATION_COLUMNS is used as-is).

triplets.validation.shacl_duckdb module

SHACL DuckDB engine — compiled-IR executor for larger-than-memory data.

Every constraint compiles to one SQL query against the connection’s configured triplets table ([ID, KEY, VALUE, INSTANCE_ID]), so validation streams through DuckDB’s vectorized executor and spills to disk instead of requiring the dataset in RAM. Input is a DuckDB connection holding the table (con.read_rdf(...)); any other flavor (pandas/polars/arrow) is registered into an in-memory connection, which makes the engine uniformly testable.

Semantics are identical to the pandas/polars engines (same IR, same canonical violations schema, same lexical-form datatype deviation). The nested and query components (sh:or, sh:and, sh:not, sh:node, sh:sparql) delegate to the pandas implementations — they materialize the (scoped) table, which is fine: those are a handful of rows even in the real profiles, and sh:sparql needs an rdflib graph anyway.

Explicitly selected (engine="duckdb"), not in the auto order: polars owns the in-memory fast path; this engine is the deliberate choice when the data does not fit.

triplets.validation.shacl_duckdb.validate(data, compiled, rdf_map=None, scope=None, components=None, max_workers=None, table=None, schema=None, table_name=None, **kwargs)

Validate triplet data against the compiled constraint table (DuckDB SQL).

Parameters mirror shacl_pandas.validate, plus:

table / schema / table_name

Triplets relation when data is a DuckDB connection (defaults from the connection, else triplets).

triplets.validation.shacl_ir module

Compile SHACL shapes once into an engine-agnostic IR (constraint table).

rdflib is the definitive shapes parser: shapes are parsed exactly once into CompiledShapes — the shapes graph (consumed by the pyshacl reference engine) plus a flat constraint table (consumed by the pandas/polars/duckdb executors, which never touch rdflib). Compilation is cached by content hash, and each engine caches its own compiled artifact (LazyFrame plan builders, SQL) in CompiledShapes.plans — re-validating new data against the same shapes never recompiles anything.

IR: one row per shape x path x constraint component.

shape_id, target_class, path, inverse, component, params, severity, message, name, description

params holds the component’s parameter: a scalar (sh:minCount), a list (sh:in), or nested row-dict lists (sh:or / sh:and / sh:not). Unknown components are kept as rows and logged — engines skip what they don’t implement; pyshacl still covers the full spec.

class triplets.validation.shacl_ir.CompiledShapes(graph: object, ir: DataFrame, hash: str, sources: tuple = (), language: str = 'shacl', stats: dict = <factory>, plans: dict = <factory>)

Bases: object

Shapes compiled once, shared by all validation engines.

graph: object

ir: DataFrame

hash: str

sources: tuple = ()

language: str = 'shacl'

stats: dict

plans: dict

triplets.validation.shacl_ir.compile_shapes(shapes) → CompiledShapes

Parse SHACL shapes (str/path | list of paths | rdflib.Graph) once.

Returns the cached CompiledShapes when the same shape content was compiled before (content hash — path identity does not matter).

triplets.validation.shacl_ir.split_rules(ir, implemented, fallback_components, engine)

IR rows → (vectorized, fallback, skipped components) against an engine’s registries.

Engines cache the result in CompiledShapes.plans[engine] so the split runs once per compiled shapes, not once per validate call; validate() reads the skipped components into the report metadata.

triplets.validation.shacl_ir.parse_ir(graph) → DataFrame

Walk NodeShapes → property shapes → one IR row per constraint component.

A NodeShape may declare several sh:targetClass (the ENTSO-E profiles do); every constraint row is emitted once per target. sh:targetSubjectsOf targets compile the same way with target_kind=”subjectsOf” — the engines resolve the focus as the subjects carrying that KEY instead of a class’s instances.

triplets.validation.shacl_pandas module

SHACL pandas engine — compiled-IR executor (debugging reference for the vectorized engine family).

Operates directly on the triplet DataFrame’s raw string VALUEs — no rdflib, no rdf_map. That enables the one deliberate deviation from pyshacl: the datatype check judges the actual lexical form. rdflib reads "1"^^xsd:float as simply valid; here it is reported, on two levels:

• value outside the declared type’s lexical space (“abc” for xsd:float) → VIOLATION_TYPE sh:datatype, the shape’s declared severity

• value valid but written in a non-canonical / narrower form (“1” for xsd:float — integer form; “0” for xsd:boolean) → VIOLATION_TYPE triplets:lexicalForm, severity Warning

Structure: one pure function per constraint component (context, rule) → violations DataFrame, registered in CONSTRAINT_VALIDATORS. Every validator sees the rule’s path normalized to (FOCUS, PATH_VALUE) pairs — inverse paths (sh:inversePath) swap the columns, so direction is handled once in _Context.path_rows.

sh:sparql constraints are delegated to triplets.sparql (auto engine — qlever when built, else oxigraph, else rdflib): the engine loads the data once (content-hash cached), each constraint runs as one SELECT with the focus nodes bound via VALUES, and max_workers runs the constraint queries in parallel processes on the rdflib path only (fork — copy-on-write shares the dataset; threads don’t help GIL-bound rdflib; the embedded engines are ms-scale sequentially).

Known limits: - sh:nodeKind — triplets store every value as a string, so term kind is decided

like the N-Quads exporter decides it: by the schema when rdf_map names the path (a datatype key is Literal even when its values look like UUIDs), by value form otherwise (known ID / UUID / URI scheme / enum PhaseCode.ABC)

class triplets.validation.shacl_pandas.SchemaKind

Bases: object

Schema-driven term-kind decision, shared by the vectorized engines’ contexts.

Mirrors the N-Quads exporter’s classification (build_key_metadata): a key with a schema datatype (incl. xsd:string) holds literals; an enumeration key holds IRIs; anyURI/unknown keys return None (decide by value form). Without rdf_map everything is None.

rdf_map = None

key_kind(key)

“literal” / “iri” / None — what the export schema says values at key are.

triplets.validation.shacl_pandas.validate(data, compiled, rdf_map=None, scope=None, components=None, max_workers=None, **kwargs)

Validate triplet data against the compiled constraint table.

Parameters:

• data (triplet DataFrame (pandas/polars), arrow, or DuckDB connection)

• compiled (CompiledShapes) – From triplets.validation.compile — this engine executes the IR.

• rdf_map (dict or str, optional) – Used only for the sh:sparql constraints (typed literals in the queried graph); every other component reads the raw lexical forms directly.

• scope (iterable of INSTANCE_ID, optional) – Validate only rows of these instances.

• components (iterable of component names, optional) – Restrict to a subset (e.g. ("sh:datatype",) for the lexical supplement run next to pyshacl). None = everything implemented.

• max_workers (int, optional) – Run the sh:sparql constraint queries in parallel processes (fork). None = sequential.

triplets.validation.shacl_polars module

SHACL polars engine — compiled-IR executor (performance).

Lazy by construction: every constraint becomes one LazyFrame plan against a shared base, and everything executes in a single polars.collect_all — parallel across plans, with common subplans (the per-path filters, the Type scan) eliminated once. Shared indices (per-class focus IDs, the set of all IDs) are materialized eagerly once and reused across plans — speed over memory, per the engine guidance in docs/validation.md. No Python UDFs, no per-constraint collects, no streaming.

Semantics are identical to the pandas engine (same IR, same canonical violations schema, same lexical-form datatype deviation — see shacl_pandas). The rare nested/query components (sh:or, sh:and, sh:not, sh:node, sh:sparql — a handful of rows even in the real profiles) are delegated to the pandas implementations, so coverage is complete while the hot path stays lazy.

Per-rule plan builders are cached in CompiledShapes.plans["polars"] — the IR is split and normalized once per compiled shapes, then only re-bound to new data.

Note: sh:pattern runs on Rust regex (no lookarounds). The ENTSO-E profiles use plain character-class patterns; exotic patterns belong to pyshacl.

triplets.validation.shacl_polars.validate(data, compiled, rdf_map=None, scope=None, components=None, max_workers=None, **kwargs)

Validate triplet data against the compiled constraint table (lazy polars).

Parameters mirror shacl_pandas.validate. Vectorized components run as one polars.collect_all over per-constraint LazyFrame plans; the nested and query components (FALLBACK_COMPONENTS) are delegated to the pandas engine so results are identical across both.

triplets.validation.shacl_pyshacl module

SHACL reference engine — pyshacl (spec-complete, rdflib-based).

Correctness-first reference; the data is loaded into an in-memory rdflib graph via the N-Quads export. Consumes CompiledShapes.graph (shapes are parsed once by the IR compiler); the constraint table is for the vectorized engines.

triplets.validation.shacl_pyshacl.validate(data, compiled, rdf_map=None, scope=None, inference='none', advanced=True, abort_on_first=False, store='memory', **kwargs)

Validate triplet data against compiled shapes; return a violations DataFrame.

Parameters:

• data (triplet DataFrame (pandas/polars), arrow, or DuckDB connection)

• compiled (CompiledShapes) – From triplets.validation.compile — this engine uses the shapes graph.

• rdf_map (dict or str, optional) – Export schema — xsd-typed literals in the data graph (optional).

• scope (iterable of INSTANCE_ID, optional) – Validate only these instances’ named graphs — data outside the scope is not loaded, so references into unscoped instances count as absent. Include dependency instances in the scope (or validate the full union, scope=None) for cross-instance checks.

• inference (passed to pyshacl.validate.)

• advanced (passed to pyshacl.validate.)

• abort_on_first (passed to pyshacl.validate.)

• **kwargs (other engines' options (components, max_workers, table_name) —) – accepted and ignored, per the shared engine contract.

• store (str, default "memory") – rdflib store backend for the data graph (see load_dataset): “oxigraph” loads through the oxigraph engine’s cached store (Rust N-Quads parse instead of rdflib’s Python parser); “auto” picks it when pyoxigraph + oxrdflib are installed. Measured (2026-07-12): results are identical, but memory stays the default — pyshacl force-clones the data graph into rdflib Memory regardless (advanced=True), and the clone through the oxrdflib wrapper costs more than the Rust parse saves (95k warm: 1.8 s vs 1.2 s; 1.14M load+clone: 40 s vs 33 s). Opt in when the store is already loaded for SPARQL anyway — then the load leg is free.

triplets.validation.shacl_report module

Map a pyshacl/SHACL ValidationReport graph to the canonical violations DataFrame, and the inverse: export a violations DataFrame as a standard sh:ValidationReport.

Canonical violations schema (identical across all current and future SHACL engines, so the later vectorized engines can produce it natively):

[ID, KEY, VALUE, VIOLATION_TYPE, MESSAGE, SEVERITY, SOURCE_SHAPE]

triplets.validation.shacl_report.report_to_violations(report_graph)

ValidationReport rdflib graph → violations DataFrame (single columnar pass).

triplets.validation.shacl_report.violations_to_report_graph(violations, report_source=None, report_references=None)

Violations DataFrame → sh:ValidationReport rdflib graph (inverse of report_to_violations; KEYs expand to the CIM namespace unless already URIs).

A result carries several plain-text ``sh:resultMessage``s when the frame has the context/location columns: the engine message, the shape and schema descriptions (context.enrich) and the source position (locations.locate_violations) — SHACL has no location vocabulary, so a message is the interoperable carrier.

Report-level metadata (always): prov:generatedAtTime, dcterms:creator (tool + version). Optional: dcterms:source / dcterms:references from report_source / report_references (str or sequence — validated file name(s) / shape file name(s)).

Defaults come from violations.attrs["validation"] — the metadata validate() stamps on the frame (timestamp of the validation run, tool version, data/shape file names). Explicit arguments override it.

triplets.validation.shacl_report.message_prefix(violation_type, source=None, language='shacl')

[<language>_message] = the text is the shape’s/schema’s own message; [engine_message] = the engine worded it. validate() stamps MESSAGE_SOURCE on every row; frames without it fall back to the violation-type namespace (triplets:* → engine). The constraint language (“shacl”; “rdfs” for schema-compiled runs) names the whole tag family in both report formats.

triplets.validation.shacl_report.export_to_shacl_report(violations, sources=None, path=None, export_to_memory=False, format=None, report_source=None, report_references=None)

Violations frame → standard sh:ValidationReport (any rdflib format).

Parameters:

• violations (DataFrame in VIOLATION_COLUMNS (any engine's output).) – Enrichment/location columns, when present, become additional ``sh:resultMessage``s (Description/Schema/Source).

• sources (optional) – The original CIM/XML files — runs the locate_violations pass so each result carries a “Source: file line N” message (a frame already carrying LOCATION_COLUMNS is used as-is).

• path (str or Path, optional) – Output file. Default report.<ext> from the resolved format. Suffix selects format when format is None (.xml/.rdf → RDF/XML, .ttl → turtle, …).

• export_to_memory (bool, default False) – Return a BytesIO (with .name) instead of writing to disk.

• format (str or None, default None) – rdflib serialize format. When None, derived from path suffix (unknown/missing → turtle). Explicit value always wins.

• report_source (str or sequence, optional) – dcterms:source on the ValidationReport (validated file name(s)). Metadata only — sources is the one that runs the locate pass. Default: the data file names validate() stamped in violations.attrs["validation"].

• report_references (str or sequence, optional) – dcterms:references on the ValidationReport (shape file name(s)). Plain labels — not the shapes object to_sarif(shapes=) takes. Default: the shape file names from violations.attrs["validation"].

triplets.validation.shacl_report.violations_to_csv(violations, path='violations.csv', export_to_memory=False)

Violations frame → CSV, plus a <name>_meta.<ext> sidecar carrying the validation metadata (violations.attrs["validation"]) as KEY,VALUE rows (no sidecar when the frame carries none). export_to_memory=True returns the BytesIO objects (with .name) instead of writing to disk — same convention as the other exports.

triplets.validation.shacl_report.violations_to_excel(violations, path='violations.xlsx', export_to_memory=False)

Violations frame → Excel; the validation metadata (violations.attrs["validation"]) goes to a second “metadata” sheet. export_to_memory=True returns a BytesIO (with .name).

Module contents

SHACL validation over triplet data.

Engines (registry dispatch, mirroring triplets.parser / triplets.sparql): - pyshacl — reference, spec-complete, rdflib-based; always available with the

validation extra

• polars — compiled-IR executor (performance; one lazy plan per constraint, single collect_all; auto-preferred when polars is installed)

• pandas — compiled-IR executor (debugging; same complete registry and semantics, eager. sh:sparql delegates to triplets.sparql with optional max_workers; sh:node runs the compile-time-expanded referenced shape against the value nodes; sh:nodeKind is decided by the rdf_map schema)

• duckdb — compiled-IR executor for larger-than-memory data (one SQL query per constraint against the triplets table, streams/spills via DuckDB; explicit engine=”duckdb”, not in auto — polars owns the in-memory fast path)

Compile once: compile(shapes) parses the shapes with rdflib exactly once into CompiledShapes (shapes graph + flat constraint table, cached by content hash). Engines receive the compiled object:

validate(data, compiled, rdf_map=None, scope=None, **kwargs) → violations DataFrame

pyshacl consumes the graph; the vectorized engines consume the IR (and cache their own plan per engine in CompiledShapes.plans) — they never touch rdflib. sh:sparql IR rows are delegated to triplets.sparql by the vectorized engines (pyshacl evaluates them itself via advanced=True).

One deliberate deviation from pyshacl: the datatype check inspects the raw lexical form of VALUE (see shacl_pandas) — with lexical=True (default) those findings are appended to any engine’s report.

triplets.validation.register_engine(name: str, module: Any) → None

Register a custom validation engine for future extensibility.

triplets.validation.get_engine(name: str = 'auto')

Resolve validation engine name (with aliases) and return (name, module).

triplets.validation.validate(data, shapes, rdf_map=None, scope=None, engine='auto', lexical=True, context=False, **kwargs)

Validate triplet data against SHACL shapes; return a violations DataFrame.

Parameters:

• data (triplet DataFrame (pandas/polars), arrow, or DuckDB connection)

• shapes (str | path | list of paths | rdflib.Graph | CompiledShapes) – SHACL shapes (format auto-detected by extension). Pass the result of compile(shapes) to reuse the parsed shapes across validations.

• rdf_map (dict or str, optional) – Export schema — xsd-typed literals in the data graph (optional).

• scope (iterable of INSTANCE_ID, optional) – Validate only these instances’ named graphs — data outside the scope is not loaded, so references into unscoped instances count as absent. Include dependency instances in the scope (or validate the full union, scope=None) for cross-instance checks.

• engine (str, default "auto") – “polars” (performance), “pandas” (debugging), “duckdb” (larger-than-memory) or “pyshacl” (reference). “auto” picks polars → pandas → pyshacl; duckdb is always an explicit choice.

• lexical (bool, default True) – Append the lexical-form datatype findings (the deliberate deviation from pyshacl — see shacl_pandas) to the engine’s report.

• context (bool, default False) – Run the slower enrichment pass (triplets.validation.context.enrich): adds instance/file, object type/name, shape name/description and schema definition columns to the report.

• in (The returned frame carries the validation-run metadata)

• duration (violations.attrs["validation"] (start/end timestamps and)

:param : :param engine: :param tool version: :param data/shape file names: :param shape and constraint counts: :param : :param and any shapes/components the run skipped) — every report exporter reads: :param it: :type it: ValidationReport and the csv/excel exports tell the :param so SARIF: :type so SARIF: ValidationReport and the csv/excel exports tell the :param the sh: :type the sh: ValidationReport and the csv/excel exports tell the :param same story.:

triplets.validation.validate_schema(data, rdf_map, engine='auto', closed=False, profiles=None, **kwargs)

Validate triplet data against the export schema — per instance, per declared profile; profiles are never merged.

Every INSTANCE_ID is validated separately (the scope filter), against each profile its own header declares: the header’s profile-identity fields (conformsTo, Model.profile, keyword, Model.messageType) are matched against the schema profiles’ declared identity (versionIRI / conformsTo / keyword / section key; legacy 2.4 profile URLs by substring). One instance may declare several profiles — each runs on its own, so per-profile constraints (e.g. mRID 1..1 in every CGMES 3.0 profile) are checked against that instance’s rows alone.

profilessequence of profile identifiers, optional

Explicit override applied to EVERY instance (section key, keyword or profile URI) — for header-less or legacy data. Unknown identifiers raise ValueError. Default None = resolve from each instance’s header; instances resolving to nothing are skipped and reported in the run metadata coverage (skipped_shapes).