Conformance tests

FeatureMesh ships 4,000+ executable tests built from the same examples as the docs and playgrounds (SQL Logic Tests, or SLT). Each test is a FeatureQL snippet plus the rows it should return. Running that suite on your DuckDB, Trino, BigQuery, or serving setup is how you verify “same answer across backends” for yourself.

You do not need to write new test files to start. Point sltest at the corpus that already ships, on the client you already use.

Why run them

How it works

1. Documentation snippets become rows in SHOW DOCS (NAME + CONTENT).
2. sltest fetches a set of those rows, runs each query, and compares results to the expected table under ----.
3. Snippets that need tables or persisted features declare # depends: on setup snippets. The runner follows those links — it does not rely on “whatever appears earlier in the file.”
4. Python (client.sltest), HTTP (POST /sltest), and MCP (featuremesh_sltest) are the same runner.

query runs FeatureQL and returns a DataFrame. sltest runs the test harness (setup, expected rows, multi-snippet files). Use each for its job.

What a test looks like

# my_setup
statement ok
CREATE TABLE …;

# my_example
# depends: my_setup
# schema: BIGINT|BIGINT
query A:I,B:I
SELECT A := 1, B := A + 1;
----
1	2

To author new snippets for your own docs or CI, see help('authoring_slt').

Focused payloads (# match:) and productivity-tool calls (client …)

Exact TSV under ---- is the default. When the interesting part is a JSON cell, markdown text, or a structured tool result — not every row — use a payload matcher instead of # schema::

Allowlisted ops: help, describe, validate, diagnose, translate. Body is one compact JSON object (terms / prefixes / query / detail / …). Do not use client query — use ordinary query for execute-and-compare.

# match: jmespath
client validate
{"query": "SELECT F := 1;", "detail": "compact"}
----
contains(text, 'Formatted FeatureQL')	true
output_schema[0].NAME	"F"
output_schema[0].TYPE	"BIGINT"

# match: jmespath
client help
{"terms": ["getting_started"], "detail": "minimal"}
----
matched_tags	["getting_started"]
length(doc_pages)	`1`

Expected values may be JSON ("F", true) or JMESPath literals (`1`). Live examples live under tests/15-querytools/querytools_client.slt.

Run a first slice

From Python

from featuremesh import BatchClient

client = BatchClient()  # or managed mode with your sql_executor

rows = client.sltest(
    where="NAME LIKE '%array%#%'",
    halt_on_fail=False,
)

for row in rows:
    if row["status"] == "FAIL":
        print(row["name"])

Use backend="trino", "bigquery", or "serving" when your client is wired to those engines.

From the demos HTTP API

curl -s -X POST "http://localhost:8101/sltest" \
  -H "Content-Type: application/json" \
  -d '{"backend":"duckdb","where":"NAME LIKE '\''%array%#%'\'","halt_on_fail":false}'

For live progress, POST /sltest_stream streams events as tests finish. Check GET /capabilities for which backends your stack exposes.

Choosing which tests to run

Every test name looks like path/to/file.slt#10.snippet_id.

Filter on the file path before #, so setup snippets in that file are included:

NAME LIKE '%array%#%'
NAME LIKE '%5-related%#%'

Prefer filtering on the file path before #. A title-only pattern after # is harder to read and, with custom source, can still drop setup from the fetch universe.

Two ways to select tests:

Simple filter (where / limit)

client.sltest(
    where="NAME LIKE '%extend%#%'",  # FeatureQL, not DuckDB SQL
    limit=50,
    halt_on_fail=False,
)

where is still a FeatureQL predicate (same NAME LIKE … fragments as before). Built-in where / limit does not drop # depends: setup. Internally the client:

1. Embeds that FeatureQL predicate in SHOW DOCS and marks matches with RUN.
2. Wraps the result in a DuckDB hybrid outer query that keeps only RUN = TRUE plus the transitive # depends: closure.
3. Returns that smaller set to the runner (one round-trip; the client does not materialize the whole corpus).

Only the fetch client must be DuckDB (for step 2). The execution backend can be anything:

• backend="duckdb" (fetch defaults to the same client), or
• fetch_client= / fetch_backend= pointing at DuckDB while backend executes on Trino / BigQuery / serving.

# Execute on BigQuery; fetch docs + resolve depends on DuckDB
# where= stays FeatureQL either way
client_bq.sltest(
    where="NAME LIKE '%extend%#%'",
    fetch_client=client_duckdb,
    halt_on_fail=False,
)

If the fetch client is not DuckDB, the runner raises ValueError up front. For a non-DuckDB fetch path, use custom source and keep the universe wide enough for depends yourself.

Custom fetch with source

client.sltest(
    source="""
    SHOW DOCS (INCLUDE (CONTENT))
    WHERE CATEGORY = 'CODE_SAMPLE'
      AND NAME LIKE '%extend%#%'
      AND NAME NOT LIKE '%strict.slt%'
    ORDER BY NAME
    """,
    halt_on_fail=False,
)

where and source are alternatives — do not send both. If you need a row limit with source, put LIMIT inside the FeatureQL string. source is not tied to the DuckDB depends-CTE; you control the universe and optional RUN column.

Dependencies between tests (# depends:)

Order in the file does not matter. Only # depends: does.

Typical shape for a tutorial or domain file:

# demo_setup
# demo_tables      ← depends on demo_setup
# demo_features    ← depends on demo_tables
# demo_q1          ← depends on demo_features
# demo_q2          ← depends on demo_features   (not on demo_q1)
# demo_teardown    ← depends on demo_q1 and demo_q2

Practical rules:

• Point every example at a shared setup, not at a sibling example.
• Point teardown at the examples that used the data, so cleanup does not run too early.
• With custom source, keep the fetch universe wide enough that setup rows are present (built-in where / limit already expands depends on DuckDB).

On a long-lived demos process, setup that already ran stays loaded until restart. You can load data once, then explore with ordinary query calls.

Narrow a run with RUN

When source returns a boolean RUN column:

• Rows with RUN = TRUE are the tests you care about.
• Their # depends: setup still runs, even if those setup rows have RUN = FALSE.
• If RUN is absent, every fetched row runs.

Example — explore RELATED / EXTEND coverage (capped)

Fetch a wide set of samples so dependencies resolve; execute only the first fifty whose content mentions extend/related:

SHOW DOCS (
    INCLUDE (
        GLOBAL_CRITERIA := CATEGORY='CODE_SAMPLE' AND NAME LIKE ANY ('%3-feature%#%'),
        RUN_CRITERIA := CONTENT LIKE ANY ('%extend%', '%related%'),
        CONTENT,
        RUN := ROW_NUMBER() WITHIN (WHERE GLOBAL_CRITERIA AND RUN_CRITERIA) OVER (ORDER BY NAME) < 50
    )
)
WHERE GLOBAL_CRITERIA
ORDER BY NAME

Example — re-run a short list of failures

Keep the same wide fetch; set RUN from the failing names. Setup still comes along via # depends::

SHOW DOCS (
    INCLUDE (
        GLOBAL_CRITERIA := CATEGORY='CODE_SAMPLE' AND NAME LIKE ANY ('%3-feature%#%'),
        CONTENT,
        RUN := NAME LIKE ANY (
            '%related_simple_equiv_extend%',
            '%related_aggregation_multithreshold_as_input%'
        )
    )
)
WHERE GLOBAL_CRITERIA
ORDER BY NAME

When the short list is green, widen the filter (file → area → full suite) before you treat the backend as verified.

Check one query without a docs file

You can build NAME / CONTENT in the source query itself — useful in notebooks or with an assistant that should assert an expected table:

client.sltest(source="""
SELECT
    NAME := 'check#001.ok',
    CONTENT := '# schema: BIGINT|BIGINT
query A:I,B:I
SELECT A := 1, B := A + 1;
----
1	2
';
""")

Keep in mind:

1. NAME must look like something#001.name
2. Put real line breaks inside CONTENT (FeatureQL does not treat as a newline)
3. Double single quotes inside the string ('')

Reading results

sltest returns a list of per-test records:

Save the JSON once and filter locally (for example with jq) instead of re-running a long suite just to inspect the same result.

Useful options

halt_on_fail — Default true stops the rest of a batch after a failure (later dependents often show as BLOCKED). Set false when you want a full picture of what still fails.

Transient retries — Registry timeouts and similar network blips (common under max_workers > 1) are retried automatically: a few quick retries on the failed snippet, then one full run-batch restart if the failure is still transient after a few seconds. Real assertion mismatches are not retried.

fetch_client / fetch_backend — Client used only for the initial docs fetch. Must be DuckDB when using built-in where / limit (outer depends CTE — where itself remains FeatureQL). Optional otherwise (e.g. fetch large source payloads on DuckDB while executing on BigQuery).

Backends and labels — "backend": "duckdb" (or trino, …) selects the execution engine. Some tests are written only for certain backends; those appear as SKIP rather than FAIL.

Row order — When order is not guaranteed, tests use rowsort and list expected rows in sorted order. Stable ORDER BY queries omit it.

Serving fixtures — A few tests talk to Redis or Postgres through named executors configured on your stack (serving_executors in /capabilities). They still use # depends: like everything else.

A practical cross-backend checklist

1. Get the area you care about green on DuckDB.
2. Re-run the same filter with backend="trino" / "bigquery" / "serving" on a stack that has those engines.
3. Treat SKIP as “not applicable here”; treat FAIL as something to investigate.
4. Use halt_on_fail=False on first passes so one failure does not hide the rest.

That is the concrete meaning of “conformance tests you can run yourself.”

Next

• Working with an assistant
• Productivity tools
• HTTP Batch API — request fields for /sltest
• Authoring your own snippets: help('authoring_slt')