Logging served features

A model is best trained on the values it will see in production. Logging what the serving engine computed, at the time it computed it, gives training samples free of training/serving skew, and a trace of every answer for debugging.

A logger is a feature, like a prepared statement. It names the prepared statement whose calls it logs, the sink it writes to, and optionally which columns to drop and which rows to keep. It never changes the statement: adding, changing or removing a logger does not recompile it, and a logger never slows or fails a call.

What to log: CAPTURE

A prepared statement returns one feature. To log more, list features in CAPTURE: they are computed in the same call, from the same reads, and never returned to the caller.

FeatureQL
CREATE OR REPLACE FEATURES IN FM.TUTORIALS.LOGGING AS
PERSIST
    USER_ID := INPUT(BIGINT),
    REQ_TS := INPUT(TIMESTAMP),
    BASKET_VALUE := USER_ID * 12,
    SCORE := ROW(BASKET_VALUE / 100.0 AS value, 'v1' AS model),
    SCORE_PS := PREPARED_STATEMENT(SCORE USING (USER_ID, REQ_TS := NOW()) CAPTURE (BASKET_VALUE));
Result
feature_name VARCHARstatus VARCHARmessage VARCHAR
FM.TUTORIALS.LOGGING.USER_IDCREATEDFeature created as not exists
FM.TUTORIALS.LOGGING.REQ_TSCREATEDFeature created as not exists
FM.TUTORIALS.LOGGING.BASKET_VALUECREATEDFeature created as not exists
FM.TUTORIALS.LOGGING.SCORECREATEDFeature created as not exists
FM.TUTORIALS.LOGGING.SCORE_PSCREATEDFeature created as not exists

Where to write: SINK

A sink is a feature too, so several loggers can share one:

FeatureQL
CREATE OR REPLACE FEATURE FM.TUTORIALS.LOGGING.ML_LOGS AS SINK('file:///tmp/featuremesh_logs' WITH (FORMAT = 'jsonl'));
Result
feature_name VARCHARstatus VARCHARmessage VARCHAR
FM.TUTORIALS.LOGGING.ML_LOGSCREATEDFeature created as not exists

SchemeFormatsWritten as
s3://, gs://, az://'parquet' (default), 'jsonl'Files rolled every minute or 64 MB, credentials from the server's environment
file://'parquet' (default), 'jsonl'Parquet files as above; JSON lines appended, a new file every hour
stdout://'jsonl'One JSON object per line, with "fm_logger" set to the logger's name

Files are partitioned the way DuckDB, Spark, Trino, BigQuery and Snowflake read them: <sink>/<logger>/statement=<hash>/date=YYYY-MM-DD/hour=HH/<file>. The statement hash changes when the statement or the logged columns change, so one partition has one schema.

Each sink sets how its files are cut, so a 10,000 QPS endpoint and a 50 QPS one can log to sinks of their own:

PropertyValuesDefault
COMPRESSION'zstd', 'snappy', 'gzip', 'none' (Parquet only)'zstd'
ROLL_SECONDS1 to 3600: a file is closed and uploaded once this old60 (the server's serving.logging.roll_interval); file:// JSON lines: every hour
ROLL_MB1 to 1024: a file is closed once this large64 (the server's serving.logging.roll_size_mb)
-- A busy endpoint: fewer, bigger files
CREATE OR REPLACE FEATURE FM.SHOP.SCORE_LOGS AS
SINK('s3://ml-logs/score' WITH (FORMAT = 'parquet', ROLL_SECONDS = 300, ROLL_MB = 256));
sql

An open file is held in memory until it is uploaded, and a crash loses it: a longer ROLL_SECONDS means fewer files, later rows and a larger loss.

stdout:// and file:// with 'jsonl' are meant for a collector (Vector, Fluent Bit, an OpenTelemetry collector) that ships the lines where you want them.

The logger

FeatureQL
CREATE OR REPLACE FEATURE FM.TUTORIALS.LOGGING.SCORE_LOGGER AS
LOGGER(
    FM.TUTORIALS.LOGGING.SCORE_PS TO FM.TUTORIALS.LOGGING.ML_LOGS
    EXCLUDE (FM.TUTORIALS.LOGGING.SCORE[model])
    WHERE HASH01('log/' || FM.TUTORIALS.LOGGING.USER_ID::VARCHAR) <.6 0.5
);
Result
feature_name VARCHARstatus VARCHARmessage VARCHAR
FM.TUTORIALS.LOGGING.SCORE_LOGGERCREATEDFeature created as not exists

  • EXCLUDE (...) drops inputs, the output, captures, or fields of a ROW (SCORE[model], SCORE[embedding][dim]).
  • WHERE keeps a row when it is TRUE. Hashing the key, rather than drawing at random, keeps whole histories: a user is either always logged or never.
  • A call that fails is always logged, whatever WHERE says, with its inputs and its error.

WHERE reads the call's row only: the statement's inputs, its output and its captures. To filter on anything else, capture it first:

FeatureQL
CREATE OR REPLACE FEATURE FM.TUTORIALS.LOGGING.BAD_LOGGER AS
LOGGER(FM.TUTORIALS.LOGGING.SCORE_PS TO FM.TUTORIALS.LOGGING.ML_LOGS WHERE INPUT(BIGINT) > 1);
Result
ERROR UE/LOGGER-WHERE-OUTSIDE-CALL FM.TUTORIALS.LOGGING.BAD_LOGGER: WHERE reads (INPUT(BIGINT)), which the call's row does not hold.

A logged row

The caller receives the score, not the captured basket value:

Prepared
FM.TUTORIALS.LOGGING.SCORE_PS
Call 1
{"input_1": [[1], [2], [3]]}
USER_ID BIGINTREQ_TS VARCHARSCORE_PS ROW
1*{value: 0.12, model: v1}
2*{value: 0.24, model: v1}
3*{value: 0.36, model: v1}

Each row the logger keeps holds:

ColumnTypeContent
call_idVARCHARUUIDv7 of the call, shared by its rows; sorts by time
call_indexBIGINTPosition of the row in the call, from 1. (call_id, call_index) is the key
call_tsTIMESTAMPTime the call started (UTC), the value of an input bound to NOW()
call_detailsROWstatus (OK, ERROR), error_code, error_message, latency_ms, row_count, server_version, boot_id
inputs, output, capturestheir typesNamed like query outputs (fm.tutorials.logging.basket_value), minus EXCLUDE

Loggers are served definitions: SHOW SERVING STATUS lists them, and a logger whose sink cannot be opened is ERROR while calls are still served.

FeatureQL
SHOW SERVING STATUS LIKE 'FM.TUTORIALS.LOGGING.SCORE_LOGGER';
Result
FEATURE VARCHARKIND VARCHARSTATUS VARCHARMESSAGE VARCHARSINCE VARCHAR
LOADED

SHOW SERVING LOGS [LIKE '<logger>'] [LIMIT n]; lists the last rows each logger kept (100 per logger), with WHERE and EXCLUDE applied: what the sink receives, checked without opening it. Each row starts with LOGGER, its SINK and SINK_URL (without credentials), then the call columns as in the sink. RECORD holds the inputs, output and captures as JSON. It is a quick check: read the sink for the typed rows.

FeatureQL
SHOW SERVING LOGS LIKE 'FM.TUTORIALS.LOGGING.SCORE_LOGGER' LIMIT 3;
Result
LOGGER VARCHARSINK VARCHARSINK_URL VARCHARCALL_ID VARCHARCALL_INDEX BIGINTCALL_TS TIMESTAMPCALL_DETAILS ROW(status VARCHAR, error_code VARCHAR, error_message VARCHAR, latency_ms DOUBLE, row_count BIGINT, server_version VARCHAR, boot_id VARCHAR)RECORD VARCHAR
FM.TUTORIALS.LOGGING.SCORE_LOGGERFM.TUTORIALS.LOGGING.ML_LOGSfile:///tmp/featuremesh_logsOK"fm.tutorials.logging.basket_value":

From logs to a training set

Logs are features at call time; labels come later from your own tables. Join them on the key and keep the label observed after the call:

SELECT l."fm.tutorials.logging.user_id" AS user_id,
       l."fm.tutorials.logging.basket_value" AS basket_value,
       l."fm.tutorials.logging.score_ps".value AS score,
       o.converted
FROM read_parquet('s3://acme-ml/serving-logs/fm.tutorials.logging.score_logger/*/*/*/*.parquet', hive_partitioning = true) l
JOIN outcomes o ON o.user_id = l."fm.tutorials.logging.user_id" AND o.observed_at > l.call_ts
WHERE l.call_details.status = 'OK';
sql

A logged call can be replayed in a query: bind the inputs it logged in FOR, REQ_TS included, and compare.

With the Python library

The serving engine embedded in the Python library (ServingDeployment.EMBEDDED) runs the same loggers, WHERE and EXCLUDE included, but hands the rows over synchronously, right after each call:

from featuremesh import LogBatch, ServingClient

def ship(batch: LogBatch) -> None:   # batch.logger, batch.sink, batch.dsn, batch.records
    queue.put(batch.records)          # enqueue, don't ship, in the handler

client = ServingClient(log_handler=ship)
python

With a handler, the sink is only a routing key: the same definitions write to S3 when served by featuremeshd. Without one, the client writes stdout:// and file:// ('jsonl') sinks itself, and a logger to any other sink is ERROR in SHOW SERVING STATUS. A handler that raises never fails the call: the error is counted in client.log_stats.