Logging served features
A model is best trained on the values it will see in production. Logging what the serving engine computed, at the time it computed it, gives training samples free of training/serving skew, and a trace of every answer for debugging.
A logger is a feature, like a prepared statement. It names the prepared statement whose calls it logs, the sink it writes to, and optionally which columns to drop and which rows to keep. It never changes the statement: adding, changing or removing a logger does not recompile it, and a logger never slows or fails a call.
What to log: CAPTURE
A prepared statement returns one feature. To log more, list features in CAPTURE: they are computed in the same call, from the same reads, and never returned to the caller.
CREATE OR REPLACE FEATURES IN FM.TUTORIALS.LOGGING AS
PERSIST
USER_ID := INPUT(BIGINT),
REQ_TS := INPUT(TIMESTAMP),
BASKET_VALUE := USER_ID * 12,
SCORE := ROW(BASKET_VALUE / 100.0 AS value, 'v1' AS model),
SCORE_PS := PREPARED_STATEMENT(SCORE USING (USER_ID, REQ_TS := NOW()) CAPTURE (BASKET_VALUE));| feature_name VARCHAR | status VARCHAR | message VARCHAR |
|---|---|---|
| FM.TUTORIALS.LOGGING.USER_ID | CREATED | Feature created as not exists |
| FM.TUTORIALS.LOGGING.REQ_TS | CREATED | Feature created as not exists |
| FM.TUTORIALS.LOGGING.BASKET_VALUE | CREATED | Feature created as not exists |
| FM.TUTORIALS.LOGGING.SCORE | CREATED | Feature created as not exists |
| FM.TUTORIALS.LOGGING.SCORE_PS | CREATED | Feature created as not exists |
Where to write: SINK
A sink is a feature too, so several loggers can share one:
CREATE OR REPLACE FEATURE FM.TUTORIALS.LOGGING.ML_LOGS AS SINK('file:///tmp/featuremesh_logs' WITH (FORMAT = 'jsonl'));| feature_name VARCHAR | status VARCHAR | message VARCHAR |
|---|---|---|
| FM.TUTORIALS.LOGGING.ML_LOGS | CREATED | Feature created as not exists |
| Scheme | Formats | Written as |
|---|---|---|
s3://, gs://, az:// | 'parquet' (default), 'jsonl' | Files rolled every minute or 64 MB, credentials from the server's environment |
file:// | 'parquet' (default), 'jsonl' | Parquet files as above; JSON lines appended, a new file every hour |
stdout:// | 'jsonl' | One JSON object per line, with "fm_logger" set to the logger's name |
Files are partitioned the way DuckDB, Spark, Trino, BigQuery and Snowflake read them: <sink>/<logger>/statement=<hash>/date=YYYY-MM-DD/hour=HH/<file>. The statement hash changes when the statement or the logged columns change, so one partition has one schema.
Each sink sets how its files are cut, so a 10,000 QPS endpoint and a 50 QPS one can log to sinks of their own:
| Property | Values | Default |
|---|---|---|
COMPRESSION | 'zstd', 'snappy', 'gzip', 'none' (Parquet only) | 'zstd' |
ROLL_SECONDS | 1 to 3600: a file is closed and uploaded once this old | 60 (the server's serving.logging.roll_interval); file:// JSON lines: every hour |
ROLL_MB | 1 to 1024: a file is closed once this large | 64 (the server's serving.logging.roll_size_mb) |
-- A busy endpoint: fewer, bigger files
CREATE OR REPLACE FEATURE FM.SHOP.SCORE_LOGS AS
SINK('s3://ml-logs/score' WITH (FORMAT = 'parquet', ROLL_SECONDS = 300, ROLL_MB = 256)); An open file is held in memory until it is uploaded, and a crash loses it: a longer ROLL_SECONDS means fewer files, later rows and a larger loss.
stdout:// and file:// with 'jsonl' are meant for a collector (Vector, Fluent Bit, an OpenTelemetry collector) that ships the lines where you want them.
The logger
CREATE OR REPLACE FEATURE FM.TUTORIALS.LOGGING.SCORE_LOGGER AS
LOGGER(
FM.TUTORIALS.LOGGING.SCORE_PS TO FM.TUTORIALS.LOGGING.ML_LOGS
EXCLUDE (FM.TUTORIALS.LOGGING.SCORE[model])
WHERE HASH01('log/' || FM.TUTORIALS.LOGGING.USER_ID::VARCHAR) <.6 0.5
);| feature_name VARCHAR | status VARCHAR | message VARCHAR |
|---|---|---|
| FM.TUTORIALS.LOGGING.SCORE_LOGGER | CREATED | Feature created as not exists |
EXCLUDE (...)drops inputs, the output, captures, or fields of a ROW (SCORE[model],SCORE[embedding][dim]).WHEREkeeps a row when it is TRUE. Hashing the key, rather than drawing at random, keeps whole histories: a user is either always logged or never.- A call that fails is always logged, whatever
WHEREsays, with its inputs and its error.
WHERE reads the call's row only: the statement's inputs, its output and its captures. To filter on anything else, capture it first:
CREATE OR REPLACE FEATURE FM.TUTORIALS.LOGGING.BAD_LOGGER AS
LOGGER(FM.TUTORIALS.LOGGING.SCORE_PS TO FM.TUTORIALS.LOGGING.ML_LOGS WHERE INPUT(BIGINT) > 1);A logged row
The caller receives the score, not the captured basket value:
FM.TUTORIALS.LOGGING.SCORE_PS{"input_1": [[1], [2], [3]]} | USER_ID BIGINT | REQ_TS VARCHAR | SCORE_PS ROW |
|---|---|---|
| 1 | * | {value: 0.12, model: v1} |
| 2 | * | {value: 0.24, model: v1} |
| 3 | * | {value: 0.36, model: v1} |
Each row the logger keeps holds:
| Column | Type | Content |
|---|---|---|
call_id | VARCHAR | UUIDv7 of the call, shared by its rows; sorts by time |
call_index | BIGINT | Position of the row in the call, from 1. (call_id, call_index) is the key |
call_ts | TIMESTAMP | Time the call started (UTC), the value of an input bound to NOW() |
call_details | ROW | status (OK, ERROR), error_code, error_message, latency_ms, row_count, server_version, boot_id |
| inputs, output, captures | their types | Named like query outputs (fm.tutorials.logging.basket_value), minus EXCLUDE |
Loggers are served definitions: SHOW SERVING STATUS lists them, and a logger whose sink cannot be opened is ERROR while calls are still served.
SHOW SERVING STATUS LIKE 'FM.TUTORIALS.LOGGING.SCORE_LOGGER';| FEATURE VARCHAR | KIND VARCHAR | STATUS VARCHAR | MESSAGE VARCHAR | SINCE VARCHAR |
|---|---|---|---|---|
| LOADED |
SHOW SERVING LOGS [LIKE '<logger>'] [LIMIT n]; lists the last rows each logger kept (100 per logger), with WHERE and EXCLUDE applied: what the sink receives, checked without opening it. Each row starts with LOGGER, its SINK and SINK_URL (without credentials), then the call columns as in the sink. RECORD holds the inputs, output and captures as JSON. It is a quick check: read the sink for the typed rows.
SHOW SERVING LOGS LIKE 'FM.TUTORIALS.LOGGING.SCORE_LOGGER' LIMIT 3;| LOGGER VARCHAR | SINK VARCHAR | SINK_URL VARCHAR | CALL_ID VARCHAR | CALL_INDEX BIGINT | CALL_TS TIMESTAMP | CALL_DETAILS ROW(status VARCHAR, error_code VARCHAR, error_message VARCHAR, latency_ms DOUBLE, row_count BIGINT, server_version VARCHAR, boot_id VARCHAR) | RECORD VARCHAR |
|---|---|---|---|---|---|---|---|
| FM.TUTORIALS.LOGGING.SCORE_LOGGER | FM.TUTORIALS.LOGGING.ML_LOGS | file:///tmp/featuremesh_logs | OK | "fm.tutorials.logging.basket_value": |
From logs to a training set
Logs are features at call time; labels come later from your own tables. Join them on the key and keep the label observed after the call:
SELECT l."fm.tutorials.logging.user_id" AS user_id,
l."fm.tutorials.logging.basket_value" AS basket_value,
l."fm.tutorials.logging.score_ps".value AS score,
o.converted
FROM read_parquet('s3://acme-ml/serving-logs/fm.tutorials.logging.score_logger/*/*/*/*.parquet', hive_partitioning = true) l
JOIN outcomes o ON o.user_id = l."fm.tutorials.logging.user_id" AND o.observed_at > l.call_ts
WHERE l.call_details.status = 'OK'; A logged call can be replayed in a query: bind the inputs it logged in FOR, REQ_TS included, and compare.
With the Python library
The serving engine embedded in the Python library (ServingDeployment.EMBEDDED) runs the same loggers, WHERE and EXCLUDE included, but hands the rows over synchronously, right after each call:
from featuremesh import LogBatch, ServingClient
def ship(batch: LogBatch) -> None: # batch.logger, batch.sink, batch.dsn, batch.records
queue.put(batch.records) # enqueue, don't ship, in the handler
client = ServingClient(log_handler=ship) With a handler, the sink is only a routing key: the same definitions write to S3 when served by featuremeshd. Without one, the client writes stdout:// and file:// ('jsonl') sinks itself, and a logger to any other sink is ERROR in SHOW SERVING STATUS. A handler that raises never fails the call: the error is counted in client.log_stats.