SHACL Validation Architecture¶
Status & known limitations¶
The validation API is frozen as of 0.2.0rc1. Known engine gaps (each
reported in the run’s coverage metadata, skipped_shapes /
skipped_components):
Vectorized engines walk
sh:targetClassandsh:targetSubjectsOf. Shapes reached solely throughsh:targetNode/sh:targetObjectsOf/sh:target(or usingsh:xone) are invisible to polars/pandas/duckdb —compile()logs a warning naming them; useengine="pyshacl"for full spec coverage.Property paths: direct,
sh:inversePath, and the two-step sequencesh:path ( assoc rdf:type )(the ENTSO-E “valueType” pattern — the constraint applies to the referenced object’s type; a dangling reference yields no value node per SHACL path semantics). Longer sequences and the*OrMorePathforms are skipped with a compile warning;engine="pyshacl"covers them.Lexical datatype checks: with
lexical=True(the default) datatype checks judge the raw lexical form of values — see the dedicated section below.engine="reference"always gives the pure pyshacl view.sh:nodeKindBlankNode(+combo) cases are intentionally not implemented in the vectorized engines (triplets data has no blank nodes).The duckdb engine streams/spills by design, but a true larger-than-RAM validation has not been exercised yet.
Engines¶
Registry dispatch (mirroring the parser), so compiled engines plug in without touching the public API:
Engine |
File |
Requires |
Role |
|---|---|---|---|
|
|
pyshacl + rdflib ( |
reference — spec-complete, rdflib-based. |
|
|
core (+ |
compiled-IR executor for debugging; complete registry — |
|
|
polars |
compiled-IR executor for performance: one LazyFrame plan per constraint, single |
|
|
duckdb |
compiled-IR executor for larger-than-memory data: one SQL query per constraint against the connection’s triplets table (streams/spills via DuckDB’s executor). Defaults come from the connection ( |
Auto order: polars → pandas → pyshacl (first importable).
engine="reference" always gives the pure pyshacl view. Custom
engines via triplets.validation.register_engine(name, module).
Compile Once (shacl_ir.py)¶
Shapes are parsed by rdflib exactly once into CompiledShapes, cached by
content hash of the shape sources:
shapes.ttl ──rdflib──► CompiledShapes
├── graph (rdflib shapes graph — what pyshacl consumes)
├── ir (flat constraint table — what the vectorized engines consume)
├── hash (content hash, the cache key)
└── plans {engine name → compiled artifact, filled lazily}
The IR is a pandas DataFrame, one row per shape × target × constraint component.
compiled.ir is inspectable (see examples/shacl_validation.py); the engine
registries that consume it are internal.
field |
type |
meaning |
|---|---|---|
|
str |
full shape IRI (blank-node id for anonymous property shapes) — becomes |
|
str |
local name: the class for |
|
|
which target declaration produced the row |
|
str | None |
the property as one triplet KEY (local name); None for node-level constraints ( |
|
bool |
|
|
bool |
the |
|
str |
the dispatch key ( |
|
object |
component parameter, shape varies (next table) |
|
str |
local name of |
|
str | None |
|
|
str | None |
|
params per component:
components |
|
|---|---|
|
int |
|
float (integer bounds are coerced) |
|
str |
|
list[str] (IRIs shortened to local names) |
|
list[str] — the fully resolved allowed KEY list: |
|
list[list[dict]] — one inner list of nested IR row dicts per alternative |
|
list[dict] — nested IR row dicts |
|
|
|
|
Compile-time behaviors worth knowing: a NodeShape with several sh:targetClass
(the ENTSO-E profiles do this) emits every row once per target class; shape
reference cycles through sh:node / sh:or / sh:and / sh:not are detected,
warned about, and dropped; supported sh:path forms are a direct IRI,
sh:inversePath, the ( assoc rdf:type ) sequence, and sh:alternativePath
only when one member carries a nested inverse (any other path form warns and
skips the property shape — pyshacl covers it). Unknown components are kept and
logged — engines skip what they don’t implement.
CompiledShapes.plans is where each engine caches its own compiled artifact
(the polars/duckdb split_rules partition) — re-validating new data against
the same shapes never recompiles anything.
Component coverage per engine¶
The shared contract lives in shacl_ir: KNOWN_COMPONENTS (all 24 keys) and
FALLBACK_COMPONENTS (sh:or/and/not/node/sparql — the nested/query components
every vectorized engine delegates to the pandas implementations via
split_rules). A test (test_shacl_ir.py::test_component_registries_agree)
pins the registries together:
engine |
registry |
coverage |
|---|---|---|
pandas |
|
all 24 (the fallback target) |
polars |
|
19 vectorized + 5 delegated |
duckdb |
|
19 vectorized + 5 delegated |
pyshacl |
consumes |
full spec; report vocabulary mapped back via |
The compile cache participates in the shared engine-state lifecycle:
triplets.clear_caches() drops cached CompiledShapes (with their plans)
together with the SPARQL engines’ loaded state, and
with triplets.cache_scope(): bounds the state created inside the block —
see sparql.md.
compiled = triplets.validation.compile(["equipment.ttl", "topology.ttl"])
triplets.validation.validate(data_a, compiled)
triplets.validation.validate(data_b, compiled) # shapes parsed once, plans reused
Engine Contract¶
Every engine module implements:
validate(data, compiled: CompiledShapes, rdf_map=None, scope=None, **kwargs) → violations DataFrame
pyshacl consumes
compiled.graph(data goes through_rdflib_loader).pandas/polars/duckdb consume
compiled.ir— they never touch rdflib and read the raw stringVALUEs directly (rdf_mapmatters only for their sh:sparql delegation, where it types the queried graph).sh:sparql IR rows: pyshacl evaluates them natively (
advanced=True). The vectorized engines delegate them totriplets.sparql(auto order: qlever when built, else oxigraph when installed, else rdflib): the data is loaded into one dataset, each constraint runs as a single SELECT with the focus nodes bound viaVALUES ?this {...}and$PATHsubstituted from the IR, andmax_workers=Nruns the constraint queries in parallel processes on the rdflib path only (fork gives copy-on-write sharing of the dataset; threads don’t help rdflib — it is GIL-bound pure Python; qlever and oxigraph are ms-scale sequentially). For sh:sparql-heavy profiles build the qlever extension orpip install triplets[oxigraph]; the rdflib fallback runs in minutes. No query fixing: constraint queries run exactly as authored. A constraint query a strict engine rejects is reported as atriplets:invalidSparqlWarning row naming the shape and is still evaluated on the lenient rdflib engine.
The Lexical-Form Datatype Deviation¶
The one deliberate divergence from pyshacl. rdflib compares a literal’s
declared datatype, so "1"^^xsd:float is simply valid — the vectorized
engines see the raw lexical form and can judge it. Two levels:
Finding |
Example (declared |
|
|
|---|---|---|---|
outside the lexical space |
|
|
shape’s declared severity |
valid but narrower/non-canonical form |
|
|
|
validate(..., lexical=True) (the default) appends these findings to any
engine’s report (duplicates dropped on
[ID, KEY, VALUE, VIOLATION_TYPE, SOURCE_SHAPE, SEVERITY]).
Parity tests treat triplets:lexicalForm rows as documented extras: engines
must never lose a violation pyshacl reports.
rdf_map still matters for the pyshacl path: without it every VALUE is an
untyped string, so sh:datatype xsd:float trips on everything; with it the
N-Quads export attaches real xsd types and only genuinely broken values fail.
Result Structure¶
validate always returns the canonical violations DataFrame — empty =
conforms — identical across all engines:
Column |
Meaning |
|---|---|
|
focus node (instance UUID, |
|
property path (CIM short name, e.g. |
|
offending value |
|
constraint component ( |
|
message from the shape |
|
|
|
shape URI that produced the result |
Violations DataFrames export like any other DataFrame (export_to_csv, Excel, …),
as a standard sh:ValidationReport (violations.shacl.to_shacl_report(...) — the
exact inverse of the pyshacl report mapping in shacl_report.py) or as SARIF 2.1.0
(violations.shacl.to_sarif(...), see below). Source positions come from one shared
pass — violations.shacl.locate(sources=...) stamps LOCATION_COLUMNS
(SOURCE_URI, SOURCE_LINE) onto the frame; both exports run it
automatically when given sources=, or reuse the columns when already present.
The SHACL report serializes via rdflib: format=None (default) derives the format
from the path suffix (.ttl → turtle, .xml/.rdf → RDF/XML, …); an explicit
format= always wins.
Validation-run metadata is stamped once, by validate(), onto the returned
frame as violations.attrs["validation"]. Every report exporter reads it, so
all formats tell the same story:
key |
meaning |
|---|---|
|
validation start/end (UTC, Zulu form) and wall-clock duration of the engine run (shapes compilation excluded — cache-independent) |
|
the engine that produced the report |
|
tool + version |
|
data file names (from the data’s Distribution label meta rows) |
|
shape file names (recorded at compile) |
|
shape count / compiled IR constraint rows |
|
feature-level gaps THIS run did not evaluate: unreachable targets ( |
|
constraint components the engine neither vectorizes nor delegates |
The coverage keys turn the compile/engine warnings into data: a report that
says “0 violations” also says whether every shape actually ran. Empty coverage
is stated, not implied — SARIF keeps the [], the csv/excel metadata rows keep
a blank-valued row.
exporter |
carries the metadata as |
|---|---|
|
|
|
|
|
a |
|
a second |
Every report export takes export_to_memory=True and returns BytesIO
object(s) with .name instead of touching the filesystem — the same
convention as export_to_cimxml/export_to_csv (to_csv returns a list:
data file + sidecar).
Every message states its origin with a prefix, and the constraint text is
never rewritten — raw sh:message / engine wording stays verbatim behind its
tag:
Exactly one of [shacl_message]/[engine_message] appears per result — the constraint
text has one author.
tag |
carries |
|---|---|
|
the shape’s own |
|
engine-worded constraint text (default messages, |
|
what the constraint requires, worded from the IR parameter ( |
|
the offending value itself (the bad literal, or the reference) — self-contained errors, no need to open the instance data |
|
what was actually found — reference targets (id, Type, name / dangling), duplicate values on maxCount — the |
|
the validated object ( |
|
the shape’s |
|
the rdf_map property (attribute/association) definition + multiplicity (context.enrich with |
|
the rdf_map description of the object’s class (context.enrich with |
|
source file + line, and the located line’s text (locate pass) |
|
SARIF text only: the shape’s declared path, grouped totals, sample objects |
Report size is controlled by existing dials, no dedicated flags: SARIF groups
by default (group=True — repeated entries like [schema_class] appear once
per rule, not per violation; group=False opts into the verbose per-violation
form). The SHACL report cannot group (one sh:result per focus node is spec
semantics), but every message entry is column-driven — drop a column
(violations.drop(columns=["CLASS_DESCRIPTION"])) and its entry disappears;
enrichment itself is opt-in.
The SHACL report additionally embeds the violated shapes’ defining triples
(CBD, stamped by validate() into the run metadata), so sh:sourceShape is
never an empty blank node — the sh:in list and every other constraint
parameter are machine-recoverable from the report alone. SARIF carries the
located line as the native region.snippet.text. All of this happens in the
post-validate/context/locate passes — the engine hot path is untouched.
validate() stamps the message/engine distinction as a MESSAGE_SOURCE
column (authored messages are known from the compiled IR); bare frames fall
back to the violation-type namespace. The SHACL report’s dcterms:creator
names the engine ("triplets 0.2.0 (engine: polars)"); SARIF carries it in
the run properties. The SHACL
report carries them as separate sh:resultMessages (results stay one per
violation — merging them would break sh:ValidationReport semantics); SARIF
carries them as newline-separated blocks in one message.text, adds
[count] / [examples] blocks for grouped results, and puts the occurrence
count in the rule title (Line completeness (8×) — the ruleId stays stable,
so GitHub alert matching is unaffected).
enrich and locate preserve the attrs. On to_shacl_report, explicit
report_source= / report_references= override the stamped values (plain
labels — distinct from the sources= locate pass and the shapes object
to_sarif(shapes=) takes). Frames without the attrs (bare frames,
report_to_violations output) carry no run metadata in SARIF/csv/excel; the
SHACL report node still gets prov:generatedAtTime (export time) and
dcterms:creator — a standard report always says when and by what it was
written.
Schema validation (validate_schema / compile_schema)¶
Validate data directly against the export schema (rdf_map) — no SHACL shapes needed, no rdflib on this path:
violations = triplets.validation.validate_schema(data, schemas.ENTSOE_CGMES_3_0_0_552_ED1)
violations = data.shacl.validate_schema(rdf_map, engine="duckdb", closed=True)
violations = data.shacl.validate_schema(rdf_map, profiles=("EQ", "SSH")) # explicit override
compiled = triplets.validation.compile_schema(rdf_map) # the whole profile set, cached
compiled.get("http://iec.ch/TC57/ns/CIM/CoreEquipment-EU/3.0") # lookup by profile URI
Semantics: per instance, per declared profile — profiles are never merged.
The schema JSON is a SET of profiles; compile_schema compiles every section
separately into a CompiledSchema addressable by each profile’s own declared
identity (versionIRI / conformsTo URI / keyword / section key). Every
INSTANCE_ID is validated on its own (the scope filter) against each profile
its header declares — resolved from conformsTo (NC dcat headers),
Model.profile/Model.messageType (CGMES), keyword, with the legacy 2.4
profile-URL substring fallback. One instance may declare several profiles;
each runs separately. This is why merging is wrong: every CGMES 3.0 profile
serializes IdentifiedObject.mRID 1..1 — a union frame carries duplicate
mRID rows per re-declared object (false maxOccurs), while SSH’s own mRID
requirement must be met by SSH’s own rows, not EQ’s.
profiles= overrides resolution for header-less or legacy data (identifiers
may be section keys, keywords or URIs; unknown ones raise). Instances that
resolve to no profile are skipped and reported in the run-metadata coverage
(skipped_shapes); the applied profiles land in attrs["validation"]["profiles"],
each violation carries a PROFILE column, reports carry [rdfs_profile] EQ
entries, and SARIF alert titles read RDFS <profile> <Class> <attr> (N×).
Per profile the checks are: cardinality from the resolved xsd:minOccours/
xsd:maxOccours (incl. mRID — the schemas are profile-accurate: CGMES 2.4
declares it 0..1/not serialized, CGMES 3.0/NCP 1..1), datatype lexical checks
from xsd:type, enumeration membership from values, association targets
from range expanded to concrete subclasses via inheritance (the expansion
index spans all sections — inheritance is model knowledge). A referenced
object conforms when ANY of its types is in the range set; dangling
references are silent (cross-instance references resolve outside the scope).
closed=True adds an unknown-property check per class and profile
(schema:domainIncludes). The same vectorized engines run everything (plans
cached per compiled profile); the pyshacl engine refuses schema-compiled IR.
Results do not masquerade as SHACL: constraint language "rdfs",
violation types xsd:minOccurs/xsd:maxOccurs/xsd:type/rdfs:range/
schema:domainIncludes — real vocabulary IRIs in sh:sourceConstraintComponent,
message tags [rdfs_expected], [rdfs_path], [rdfs_profile], …
Call Sequence¶
data.shacl.validate(shapes)
|
'-> validation.validate(data, shapes, engine="auto", lexical=True)
|
|-> compile(shapes) # cached by content hash
| '-> _load_shapes -> parse_ir # rdflib parses ONCE
|
|-> get_engine("auto") # polars (first importable)
| '-> shacl_polars.validate(data, compiled, rdf_map)
| # reads raw VALUEs; lexical findings emitted inline
| # (pyshacl path instead: load_dataset -> scoped_graph ->
| # pyshacl.validate(compiled.graph) -> report_to_violations)
|
'-> + shacl_pandas datatype/lexical supplement # ONLY for engine="pyshacl"
File Layout¶
triplets/
|-- _rdflib_loader.py # shared with sparql: load_dataset(), scoped_graph()
'-- validation/
|-- __init__.py # validate() + compile() dispatcher, engine registry
|-- shacl_ir.py # shapes -> CompiledShapes (IR compiler, content-hash cache)
|-- shacl_pyshacl.py # reference engine: data + compiled.graph -> report
|-- shacl_pandas.py # compiled-IR executor (full registry; eager, debugging)
|-- shacl_polars.py # compiled-IR executor (lazy plans + collect_all, performance)
|-- shacl_duckdb.py # compiled-IR executor (SQL per constraint, larger-than-memory)
|-- shacl_report.py # ValidationReport <-> violations; multi-format export
|-- context.py # optional enrichment pass (instance/object/shape/schema context)
|-- locations.py # violations -> source line/column (the sources= grep pass)
'-- sarif.py # violations -> SARIF 2.1.0 (grouped by default)
Usage¶
import pandas
import triplets
from triplets.export_schema import schemas
data = pandas.read_RDF(["grid.zip"])
# validate against a shapes file (format auto-detected by extension) — empty = conforms
violations = data.shacl.validate("shapes.ttl", rdf_map=schemas.ENTSOE_CGMES_3_0_0_552_ED1)
# compile once, validate many
compiled = triplets.validation.compile(["equipment.ttl", "topology.ttl"])
violations = data.shacl.validate(compiled)
# scope filters to the named graphs (validated instances) — out-of-scope data
# is not loaded; include dependency instances for cross-instance references
one_instance = str(data["INSTANCE_ID"].astype(str).iloc[0])
violations = data.shacl.validate("shapes.ttl", scope=[one_instance])
# engines: auto is polars -> pandas -> pyshacl; works on any input flavor
data.shacl.validate("shapes.ttl", engine="pyshacl") # or "polars" / "pandas" / "duckdb"
triplets.validation.validate(con, "shapes.ttl") # DuckDB connection
violations = data.shacl.validate("shapes.ttl", lexical=False) # pure pyshacl report
# pyshacl on the oxigraph engine's cached store (identical results; opt in
# when the store is already loaded for SPARQL — Memory is faster otherwise)
data.shacl.validate("shapes.ttl", engine="pyshacl", store="oxigraph")
# slower optional context pass — adds instance/file, object type/name,
# shape sh:name/sh:description and schema definition columns
violations = data.shacl.validate(compiled, rdf_map=..., context=True)
violations = violations.shacl.enrich(data=data, shapes=compiled, rdf_map=...) # same, standalone
# SARIF 2.1.0 for GitHub / SonarQube / any SARIF viewer
violations.shacl.to_sarif(path="report.sarif")
# standard sh:ValidationReport for SHACL tooling (format from path suffix,
# or format=); sources= adds an "[context_location] file line N" message per
# result (SHACL has no location vocabulary — plain-text messages travel
# everywhere); the dcterms metadata (timestamp, creator, data/shape file
# names) comes from violations.attrs — stamped by validate(), overridable
# with report_source=/report_references=
violations.shacl.to_shacl_report(path="report.ttl", sources=["grid.zip"])
violations.shacl.to_shacl_report(path="report.xml") # RDF/XML via .xml
# tabular exports with the same metadata: CSV writes a report_meta.csv
# sidecar, Excel a second "metadata" sheet
violations.shacl.to_csv(path="report.csv")
violations.shacl.to_excel(path="report.xlsx")
# the location pass standalone — SOURCE_URI/LINE/COLUMN/COLUMN_END columns,
# reused by both report exports
violations = violations.shacl.locate(sources=["grid.zip"])
pyshacl pass-through options (inference, advanced, abort_on_first,
store) are forwarded as keyword arguments. Runnable end-to-end demos live in
examples/shacl_validation.py (engine behavior walkthrough) and
examples/shacl_reports.py (uv-runnable: performance engines + both report
exports with source locations — uv run examples/shacl_reports.py).
Context enrichment (validation/context.py)¶
validate(..., context=True) — or standalone
enrich(violations, data=, shapes=, rdf_map=) — appends ENRICHMENT_COLUMNS
to the report. Every source is optional; absent sources leave their columns
null, so the output schema is stable:
Columns |
Source |
Content |
|---|---|---|
|
data |
instance and its parsed file name (the |
|
data |
the focus object’s |
|
shapes |
|
|
rdf_map |
the violated attribute’s schema definition |
|
rdf_map |
the object’s class description |
Pass the same shapes object the validation ran with — anonymous property shapes get fresh blank-node ids per parse, so a re-parsed graph cannot be matched. An object present in several instances is attributed to its first occurrence.
SARIF 2.1.0 export (validation/sarif.py)¶
triplets.validation.export_to_sarif(violations, ...) /
violations.shacl.to_sarif(...) — a spec-valid SARIF 2.1.0 log for GitHub,
SonarQube or any SARIF viewer. Passing data=/shapes=/rdf_map= runs the
enrichment pass first (an already-enriched frame is used as-is).
Grouped by default (
group=True): a model can carry 100k identical findings that are really one issue — each rule becomes one result withoccurrenceCountand the first-3 + last-3 sample instances (all listed when ≤ 6).group=Falseemits one result per violation row.Severity maps
Violation/Warning/Info→error/warning/note; rules carry the shape’ssh:name/sh:description;MESSAGEfalls back to a generated text (message is mandatory in SARIF).RDF objects carry no text coordinates through the triplets frame: results always point at the model via
logicalLocations(Type/ID+ object name). Passingsources=(the original CIM/XML files — paths, zips or file-likes) runs the sharedlocate_violationspass (validation/locations.py) and adds a fully bounded whole-linephysicalLocation.region(startLine==endLine) on the violated property element’s line (or the object definition) — what GitHub code scanning needs to annotate lines. One grep-style pass per file; the parse/validate hot paths are untouched. A frame already carryingLOCATION_COLUMNS(fromviolations.shacl.locate(sources=...)) is used as-is. Without either,artifactLocation.urifalls back to the enrichment-traced file label, region-less. Columns are 1-based UTF-16 code units (SARIF’s unit) — a start-only region cannot be displayed by GitHub, so regions always carry their end.Everything domain-specific (triplet coordinates, schema descriptions, sample IDs) rides in the
propertiesbags.
Naming Convention¶
Engine files follow {purpose}_{engine}.py, mirroring the parser and export
modules. Shared loading lives in _rdflib_loader.py; the IR compiler in
shacl_ir.py; the report normalizer in shacl_report.py.