HTTP data sources

HTTP integration enables FeatureQL to fetch data from REST APIs, microservices, and web endpoints. This is particularly useful for real-time feature enrichment, external service integration, and dynamic data retrieval.

Creating an HTTP data source

Define reusable HTTP endpoints with default configurations:

Connection configuration:

• URL: Base endpoint URL for the HTTP service
• Query parameters: Default query parameters using ROW() syntax
• Timeout: Request timeout settings (e.g., '500ms')
• Headers: Authentication, content-type, and other HTTP headers (if needed)

The SOURCE_HTTP() function creates a reusable HTTP endpoint configuration that can be shared across multiple feature definitions.

Making HTTP requests with EXTERNAL_HTTP

Use EXTERNAL_HTTP() to fetch data from HTTP endpoints with dynamic parameters:

Key components explained:

1. Source reference: FROM FM.DEMO1.HTTP_SOURCE1 uses the predefined HTTP source
2. Dynamic parameters: ROW(FIRSTNAME AS name) provides request-specific query parameters
3. Parameter override: Request parameters override defaults from the source definition
4. Batch processing: Multiple input values generate multiple HTTP requests automatically

Request flow:

• For FIRSTNAME := 'Alice' → GET http://host.docker.internal:8010/hello?name=Alice
• For FIRSTNAME := 'Bob' → GET http://host.docker.internal:8010/hello?name=Bob
• For FIRSTNAME := 'Charlie' → GET http://host.docker.internal:8010/hello?name=Charlie

Note: The output of the HTTP call is always a VARCHAR. You can parse it as a JSON object using the JSON_PARSE_AS() function.

Fallback: If the HTTP request fails, it returns NULL. Use COALESCE to set a fallback strategy.

Performance considerations

1. Batch optimization: FeatureQL can parallelize multiple HTTP requests
2. Timeout management: Set appropriate timeouts for your latency requirements
3. Retry policy: There is no retry policy.
4. Rate limiting: Be aware of API rate limits.
5. Overload: Be careful of not overloading your databases. FeatureMesh can generate violent burst of queries.
6. Network latency: HTTP requests add network latency to feature computation
7. Error handling: External HTTP services can be unreliable; implement fallbacks

Use cases

• API integration: Connect to third-party services and APIs
• Microservice communication: Fetch data from internal microservices
• ML model serving: Call external ML models for predictions