OKF Semantic Layer

Build data pipelines by describing them in natural language — with the safeguards of a machine-readable specification that both humans and AI agents can read, validate and diff.

Someone says "pull the supplier master in and add it to the order fact". An AI agent does not write a pipeline in response. It interviews them — what is one row of this, does anyone need to see it as it was last year, which number do you already trust — profiles the actual source, and only then writes a concept: a Markdown file whose frontmatter is a strictly validated contract, with a closed transformation vocabulary, declared types, declared business keys and declared maturity. The pipeline is generated from that contract.

The result is a workflow where the input is a conversation and the output is deterministic: the same specification produces the same code, and a change in the specification produces a predictable change in the code. A pull request shows the intent and the implementation side by side, in one diff.

The interview is not improvisation either. The questions, their order, and what to do when nobody knows the answer are a versioned convention in the bundle; so is the playbook for the twenty situations where a source misbehaves. When a table has no unique key, the agent asks — it does not invent one.

Built on Open Knowledge Format 0.1, targeting Databricks (Unity Catalog, Lakeflow Declarative Pipelines, asset bundles).

The One Rule

knowledge/ is the truth. pipelines/ is an artifact.

Generated code is never edited by hand. A manual edit disappears on the next generation and makes the diff unreadable. If something in pipelines/ is wrong, the specification is missing something — that is where you fix it.

Why Not Just Let the Agent Write the Pipeline?

Because the failure mode of AI-written data pipelines is not code that crashes. It is code that runs cleanly and returns a wrong number.

Five mechanisms guard against that, and they are the whole substance of this repo:

Mechanism	What it prevents
Closed transformation vocabulary	Free text like "convert the date" generating different code every run. There are ~18 allowed rules; the validator rejects everything else.
A specified interview	An agent guessing at what it did not ask. The question catalogue is a convention, every answer is recorded with where it came from — measured, stated, defaulted or guessed — and a blocking unknown holds its concept out of production.
spec_status	Code generated from a guess about a source reaching production. assumed concepts are physically routed to a dev-only directory that the test and prod targets do not read.
Two CI gates	A specification that no longer matches its generated code merging anyway. validate.py checks the bundle; check_release_gate.py checks that every generated file's header still matches its concept.
Generated reconciliation tests	Trusting code nobody read. Correctness is established by comparing aggregates against a source the business already believes — not by review.

Two health metrics are printed on every validator run. expr() — the escape hatch, which requires review_required: true — counts what the vocabulary could not express. Open questions count what nobody has answered yet. The fewer of each, the more the specification is actually carrying the work.

Quickstart

git clone <your-fork> && cd okf-semantic-layer
python3 -m venv .venv && source .venv/bin/activate
pip install pyyaml jsonschema
python3 knowledge/schema/validate.py       # expect: PASS
python3 scripts/check_release_gate.py      # expect: PASS

Both gates run with no Databricks access at all. To deploy the one worked example — a calendar dimension that needs no source system — see docs/02-first-run.md.

Layout

Path	Content	Hand-editable
knowledge/	OKF bundle — the specification of every data product	yes, this is where you work
knowledge/conventions/	The binding rules: the interview, the modeling playbook, layers, naming, vocabularies, ingress contracts	yes — start here when adapting
knowledge/intake/	What each onboarding asked, measured, and still does not know	yes, written during discovery
knowledge/schema/	JSON Schema and validator	yes, with care
pipelines/released/	Generated pipelines from validated concepts	no
pipelines/preview/	Generated pipelines from draft/assumed, dev only	no
pipelines/lib/	Shared transform helpers	no
tests/	Generated DQ and reconciliation tests	no
resources/	Schemas and job definitions	jobs: no
databricks.yml	Bundle and environments	yes, rarely
.claude/skills/	The five agent skills that make the above happen	yes
docs/	Human documentation	yes

Documentation

Start at docs/index.md. If you are in a hurry:

• Start Here — for the person who is not a data engineer and just wants their data in. One page, no vocabulary, with an annotated interview
• The Approach — the idea in five minutes
• First Run — clone to a deployed pipeline
• Making It Yours — adapting it to your own platform
• Cheat Sheet — every rule and command on one page

What Ships With This Repo

Five agent skills (okf-discover, okf, okf-authoring, okf-generate, okf-deploy), eight conventions, a JSON Schema, two CI gates, a Databricks asset bundle, and one worked example: the calendar dimension d_date. It needs no source system, which makes it the smallest slice that exercises the whole chain — concept, pipeline, test, job, deployment — so you can prove the machinery works before connecting anything real.

Everything else is an empty index file waiting for your first concept. That is deliberate: there is no sample domain to delete.

Agent Support

The skills under .claude/skills/ are written for Claude Code and load automatically there. They are plain Markdown with YAML frontmatter, so any agent harness that can read a directory of instructions can use them — and CLAUDE.md is a readable contract on its own if you would rather drive the process by hand.

Worth adding once you connect a real source: Databricks ships an MCP server and a set of agent skills that give an agent read access to Unity Catalog and a SQL warehouse. With those configured, the agent profiles the source table before writing the concept — is the key actually unique, is that column epoch milliseconds or a string, how many distinct status codes exist — instead of guessing and labelling the guess assumed. Exploration stays read-only, and neither production data nor catalog names ever enter knowledge/. See docs/02-first-run.md, Part 4.

License

MIT — see LICENSE.