Demos repo tools

The HTTP API, MCP server, and eval harness are not in pip install featuremesh. They live in the demos repo (source: featuremesh-demos). Clone that repo, then adapt compose, env, ports, and backends to the FeatureMesh install you already run.

BatchClient() in Python talks to a local engine from the package and does not start these servers. Package methods: Python library tools .

What you clone and what you change

Typical local work from featuremesh-demos:

1. Clone the repo and install its Jupyter / stack extras.
2. Point .env at your registry, identity token, and backends (FEATUREMESH_IDENTITY_TOKEN, warehouse credentials, optional OPENROUTER_API_KEY for eval).
3. Start only the pieces you need (just api-server, just mcp-server, Jupyter).
4. Confirm the stack with GET http://localhost:8101/capabilities (host and port may differ after you adapt the compose file).

A common adapted stack exposes:

There is no FeatureMesh-hosted MCP and no FeatureMesh-operated run of your warehouse SQL.

HTTP API server (:8101)

JSON interface. Paths match the Python BatchClient methods. Send Content-Type: application/json.

Common fields

Use "structured": true whenever your script needs rows, SQL, or typed errors. The Python package already returns those fields without this flag.

Leave debug at "off" unless you are investigating a compiler issue and need on-disk pipeline logs.

Routes

GET /capabilities — enabled endpoints, backends, and optional serving executors.

GET /health — liveness.

POST /query — run FeatureQL. Body: query (FeatureQL text only, not SLT harness blocks), backend, structured, optional DuckDB timeit. Structured responses include sql, dataframe, errors, warnings, and timings. Errors look like:

{
  "code": "UE/…",
  "message": "…",
  "location": [{"line": 1, "column": 10}],
  "hint": "… optional"
}

POST /translate — same body as /query, without execution.

POST /validate — formatting, feature graph, optional SQL dry-run. detail is "compact" (default) or "full". Structured fields include formatted_featureql, output_schema, features, warnings, sql_valid, generated_sql. Method meaning: Python library tools .

POST /diagnose — incremental run until the first failure. Fields: failed_step, steps, explain_json.

POST /describe — body prefixes (at least one non-empty namespace). Returns sources (NAME + FORMULA), one Grains table (entity / input / keyset), a POSITION-ranked derived sample (max 20) with derived_total, plus next_queries (SHOW … IS_DOWNSTREAM_OF, full names) when structured is true. Markdown uses short names — qualify with FROM <ns> or <ns>.FEATURE_NAME.

POST /help — terms, detail (minimal / normal / full), examples.

POST /sltest and POST /sltest_stream — run documentation tests. Bodies are mutually exclusive: where + limit (FeatureQL predicate; dependency-aware) or source (must return NAME and CONTENT). Guide: Conformance tests .

POST /sltest_multi — multi-backend execution when the server is configured for it.

POST /direct_sql_query — raw SQL through the server’s optional SQL executor — not FeatureQL.

A successful HTTP status can still include errors: […]. Invalid request bodies return VALIDATION-ERROR with the same shape.

BASE=http://localhost:8101

curl -s "$BASE/capabilities"

curl -s -X POST "$BASE/query" \
  -H "Content-Type: application/json" \
  -d '{"query":"RETURN F := 1;","backend":"duckdb","structured":true}'

MCP server (:8100)

Connect your AI client to the MCP port on the stack you adapted. Tool names match the HTTP routes and the Python methods:

How to connect a client, what to ask, and what to review: FeatureQL with AI . The assistant’s own loop is help('assistant_workflow').

Eval harness

The runner lives in the demos stack — you do not install a second Python package. Two campaign kinds share it:

A model can pass one and fail the other. Faithful translation of an oracle is not the same skill as reading a question.

From featuremesh-demos, with Jupyter running, and the HTTP API (:8101) and MCP (:8100) up (just api-server, just mcp-server if needed). Put OPENROUTER_API_KEY in the demos .env next to FEATUREMESH_IDENTITY_TOKEN.

Question to FeatureQL

A suite is a folder under apps/jupyter/eval/benchmarks/<name>/. The folder name is the suite id. Harbour in FeatureMesh is docs/website/authored/benchmarks/harbour_eval/ — copy that folder here.

questions.md and hints.md may contain only those # T<number>. Title headings — no preamble, no extra sections. Expected rows and SQL do not belong in Markdown. Each task names three snippets; they must agree on rows. Scratch and SQL depend on the dataset snippet; prebuilt depends on the last prebuilt setup snippet.

Do not put file paths in benchmark.yaml. Snippet names are unique, so the exporter finds them.

{
  "dataset": "harbour_eval_data_setup",
  "health": "SELECT COUNT(*) AS count FROM harbour_eval.port_calls",
  "profiles": {
    "scratch": ["harbour_eval_scratch"],
    "prebuilt": ["harbour_eval_prebuilt_reset", "harbour_eval_prebuilt_model", "harbour_eval_prebuilt_mapping"]
  },
  "tasks": [
    {"id": "T1", "difficulty": 1, "solutions": {"scratch": "harbour_t1_vessel_draught", "prebuilt": "harbour_t1_vessel_draught_prebuilt", "sql": "harbour_t1_vessel_draught_sql"}}
  ],
  "episodes": [
    {"id": "first-block", "tasks": ["T1", "T2", "T3"]}
  ]
}

Campaign files live in apps/jupyter/eval/campaigns/. The filename is the campaign id and the log folder.

"models": [
  "openai/gpt-4.1-mini",
  {"id": "openai/gpt-6-luna", "reasoning": {"effort": "high"}}
]

tasks may be task ids or episode ids. "thread" is per_task, sequential, or together.

just bench-eval-export harbour_eval
just bench-eval-run harbour-harness-v1.json

The first command splits the suite into questions (subject) and hidden answers (oracle) under eval/packages/. It compares the authored expected tables of the scratch, prebuilt, and SQL keys and refuses the package if they disagree. The second command runs every combination in the campaign file.

Each attempt writes under eval/logs/<campaign-file-stem>/. A campaign with both scratch and prebuilt writes one folder per profile. Scoring is the result table against the hidden expected rows. FROM SQL() in a FeatureQL submission fails when the FeatureQL oracle does not use it. If a run dies before cleanup, just bench-eval-reconcile deletes leftover projects.

Another suite: copy it to apps/jupyter/eval/benchmarks/my_suite/, add eval/campaigns/my_suite.json with "suite": "my_suite", then export and run. Suites and scoring notes: benchmark catalog .

SQL to FeatureQL

This campaign starts from a known SQL oracle and walks through Spine Normal Form (SNF) so each hop can be graded on rows. It does not ask a model to invent FeatureQL from a question.

SQL and FeatureQL organize work differently. SQL joins at whatever grain is convenient. FeatureQL names an entity, binds its keys, and computes features at that grain. A one-shot “rewrite this SELECT as FeatureQL” mixes three problems: did the model keep the same rows, did it pick a grain, and did it use the language?

SNF is ordinary DuckDB SQL with a fixed shape. It keeps the original answer and makes the grain explicit. The eval then asks two smaller questions:

1. Does this SNF still return the same bag of rows as the oracle SQL?
2. Does the FeatureQL return that same bag?

If hop 1 fails, the FeatureQL was never the issue. If hop 1 passes and hop 2 fails, the miss is the language mapping. The compiler still only does FeatureQL → SQL. This path is a client translator and an eval, not a new compiler mode.

Every SNF query is built around one spine: the key universe of the output entity, unfiltered.

• One CTE per grain. A CTE is one row per key of one entity.
• Many-to-one lookups are allowed. One-to-many joins are not — collapse the finer grain with GROUP BY the spine key.
• Predicates become boolean columns (flags). Aggregates use FILTER (WHERE flag).
• The base CTE assembles the spine with LEFT JOINs. The final SELECT keeps the original column order, plus ORDER BY / LIMIT.
• Correlated subqueries, EXISTS, and IN (SELECT …) are rewritten as collapsed CTEs. EXCEPT / INTERSECT stay fragment limits.

A comment above each CTE names its grain (-- grain: vessel). That comment is for the next hop, not for DuckDB.

-- original
SELECT id, name
FROM harbour_eval.vessels
WHERE draught_m > 11.0;

-- SNF
WITH spine AS (
    SELECT id FROM harbour_eval.vessels          -- grain: vessel (full universe)
),
base AS (                                        -- grain: vessel
    SELECT s.id, v.name,
           v.draught_m > 11.0 AS is_deep
    FROM spine s
    JOIN harbour_eval.vessels v ON v.id = s.id
)
SELECT id, name
FROM base
WHERE is_deep
ORDER BY id;

The WHERE on is_deep is allowed because it filters spine-grain columns in the presentation SELECT. The spine CTE itself stays complete.

Each SNF CTE becomes features at that grain. The spine key becomes ENTITY() plus BIND_COLUMNS. Pointwise flags stay predicates. Collapsed children become RELATED() / TRANSFORM() aggregates. The presentation SELECT is the same columns, in the same order.

DEFINE
    VESSELS := ENTITY(),
    VESSEL_ID := INPUT(BIGINT#VESSELS),
    TABLES.VESSELS := EXTERNAL_COLUMNS(
        id BIGINT#VESSELS BIND TO VESSEL_ID,
        name VARCHAR,
        draught_m DOUBLE
        FROM TABLE(harbour_eval.vessels)
    ),
    VESSEL_NAME := TABLES.VESSELS[name],
    IS_DEEP := TABLES.VESSELS[draught_m] >.1 11.0e0
RETURN
    VESSEL_ID,
    VESSEL_NAME
FOR
    VESSEL_ID := BIND_COLUMNS(id FROM TABLE(harbour_eval.vessels))
WHERE
    IS_DEEP
ORDER BY
    VESSEL_ID;

The universe is EXTERNAL_COLUMNS from the qualified table, the filter is a feature, and FOR binds the same key the spine selected.

Language patterns stay on their own help pages. The eval injects a shape card — help('sql2fql_lookup'), help('sql2fql_child_aggregation'), and the rest under help('sql2fql') — that shows only the SNF template and the FeatureQL template for that plan tag.

Input is the suite’s SQL oracle plus context/data_schema.yaml (tables, types, primary keys, foreign keys). data_notes.md keeps enums, nulls, units, and path caveats. The question text is not in the prompt.

A later optional step can persist features into a catalog. The first eval grades each query on its own.

Campaigns in the demos repo are *-sql2fql-v0.json. Suites live under authored/benchmarks/ (harbour, semantic, Contoso, analytics, Spider, TPC-H).

just sql2fql-plan harbour_eval
just sql2fql-run harbour-sql2fql-v0.json

sql2fql-plan is the histogram with no LLM. sql2fql-run walks the two hops. Threads and summary.md land under eval/logs/.

Related

• Python library tools
• FeatureQL with AI
• Conformance tests
• Shape cards: help('sql2fql')
• Language-first mapping when you are not in the eval: help('mental_model'), help('sql_to_featureql')