Feature stores and feature pipelines: offline/online consistency

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Concept: Feature store · also covers Feature pipelines · Step 7 of 35 in MLOps / ML platform engineer · version 1

Practice this

On this page

A feature store is the system that defines features once and serves the same values to training and to online inference. It pairs an offline store holding timestamped history with an online store holding the latest value per entity, plus a registry of definitions. Its main purpose is correctness: training rows must see only what was known at the time, and serving must compute features the same way training did.

Read more

How it is done in practice

A team keeps feature definitions in a Git repository next to the pipelines that compute them; CI runs a plan against the registry and merges apply the change. Batch pipelines write timestamped feature tables to the offline store, an orchestrator task runs incremental materialization after each successful load, and a streaming job pushes the few features that need minute-level freshness. Data scientists build training sets by passing an entity dataframe with label timestamps to historical retrieval, and the serving code fetches the same feature service by name. The platform team owns the online store capacity, freshness alerts per feature view, access control on feature groups, and a periodic offline versus online consistency check.

Technologies

FeastAmazon SageMaker Feature StoreTectonRedisAmazon DynamoDBBigQuerySnowflakeApache SparkApache Kafka

Diagram

flowchart LR
  P[Batch and streaming pipelines] --> OFF[Offline store with history]
  DEF[Feature definitions in Git] --> REG[Registry]
  REG --> OFF
  REG --> ON[Online store latest values]
  OFF -->|materialize| ON
  P -->|push fresh values| ON
  ED[Entity dataframe with timestamps] --> PIT[Point-in-time join]
  OFF --> PIT
  PIT --> TRAIN[Training job]
  ON --> SVC[Model service]
  SVC --> LOG[Served feature log]
  LOG --> OFF
flowchart LR
  P[Batch and streaming pipelines] --> OFF[Offline store with history]
  DEF[Feature definitions in Git] --> REG[Registry]
  REG --> OFF
  REG --> ON[Online store latest values]
  OFF -->|materialize| ON
  P -->|push fresh values| ON
  ED[Entity dataframe with timestamps] --> PIT[Point-in-time join]
  OFF --> PIT
  PIT --> TRAIN[Training job]
  ON --> SVC[Model service]
  SVC --> LOG[Served feature log]
  LOG --> OFF

Code

python
from datetime import timedelta
from feast import Entity, FeatureView, Field, FileSource
from feast.types import Float32, Int64

driver = Entity(name='driver', join_keys=['driver_id'])

source = FileSource(
    path='data/driver_stats.parquet',
    timestamp_field='event_timestamp',
)

driver_stats = FeatureView(
    name='driver_hourly_stats',
    entities=[driver],
    ttl=timedelta(days=1),
    schema=[Field(name='conv_rate', dtype=Float32),
            Field(name='avg_daily_trips', dtype=Int64)],
    source=source,
)
python
import pandas as pd
from feast import FeatureStore

store = FeatureStore(repo_path='.')
features = ['driver_hourly_stats:conv_rate',
            'driver_hourly_stats:avg_daily_trips']

# training: as-of join at each label timestamp
entity_df = pd.DataFrame({
    'driver_id': [1001, 1002],
    'event_timestamp': pd.to_datetime(['2026-01-05 10:00', '2026-01-06 09:30']),
})
train_df = store.get_historical_features(
    entity_df=entity_df, features=features).to_df()

# serving: latest values from the online store
online = store.get_online_features(
    features=features, entity_rows=[{'driver_id': 1001}]).to_dict()

Subtopics

Architecture: registry, offline store, online store

Three components and two read paths

The registry holds feature definitions as versioned metadata: which entities exist, which features belong to which view, where the source data lives. The offline store is a warehouse or lakehouse table set with full history, read in bulk to build training sets. The online store is a key-value database holding only the latest values per entity key, read at request time with low latency. Feast is an open-source implementation that plugs into existing stores instead of replacing them; it does not run the transformations that produce batch features, which stay in upstream pipelines. Amazon SageMaker Feature Store offers the same split as a managed service through feature groups with an online store, an offline store, or both.

When a feature store is worth it

The cost is another system to operate, so the case rests on reuse and correctness. It pays off when several models share features, when models serve online and need the same values that were used offline, and when training needs history joined at the correct time. A single batch-scored model with features computed in one SQL query usually does not need one. Platform teams often start with only the offline path and conventions for timestamped feature tables, then add online serving when the first real-time model arrives.

Defining entities, feature views and feature services

The definition objects

In Feast, an entity is the business object features describe, identified by a join key such as a user id. A data source points to a table or file with an event timestamp column. A feature view groups a set of features over one source for one or more entities, with a schema and a time-to-live that bounds how far back a value may be used. A feature service names the exact set of features one model version consumes, which gives a stable contract between the model and the store. Definitions are Python files in a repository; running the apply command registers them and prepares the infrastructure.

On-demand and streaming features

Some features cannot be precomputed because they depend on the request, for example the ratio between a transaction amount in the request and a stored average. On-demand feature views express such transformations so they run in both historical retrieval and online retrieval, keeping one definition. Features with tight freshness needs arrive from a stream processor that pushes values into the online store and logs them to the offline store. In SageMaker Feature Store the equivalent unit is a feature group, where every record carries a record identifier and an event time.

Point-in-time correct training sets

The join a feature store performs

Historical retrieval takes an entity dataframe, one row per training example with entity keys and an event timestamp (usually the moment the label's prediction would have been made). For each row and each feature view, the store selects the most recent feature row whose timestamp is at or before the example timestamp and no older than the view's time-to-live; otherwise the feature is null. This as-of join is what prevents label leakage from future values. It is expensive on large tables, so offline stores implement it as a windowed SQL query, and teams keep feature tables partitioned by time to bound the scan.

Materialization and online retrieval

Loading the online store

Materialization copies the latest feature values per entity from the offline store into the online store for a time range. An incremental materialize run processes only the interval since the previous run, and is scheduled from an orchestrator at the cadence the freshness requirement allows. Because the online store keeps one row per entity key per view, a rerun overwrites the same keys and is safe to retry. Failures here are silent for the model, which keeps serving stale values, so the platform tracks the last successful materialization time per view and alerts when it exceeds the expected interval.

Serving path

At inference the model service requests features for a list of entity rows, either through the SDK in process or through a feature server over the network. Latency is dominated by the online store read, so features for one request are batched into a single call, feature services limit the fetch to what the model needs, and missing entities return nulls that the model must handle with defaults. Capacity planning treats the online store as a tier-one dependency of the model endpoint: it shares the endpoint's availability target, timeouts and fallbacks.

Detecting and preventing training-serving skew

Sources of skew

Skew is any difference between the feature values a model trained on and the values it receives in production. Common causes are two implementations of the same logic (SQL offline, application code online), different handling of nulls and defaults, timezone or window boundary differences, stale online values after a failed materialization, and backfills that use data which was not yet available at the original time. A feature store removes the first cause by having one definition and one retrieval interface for both paths, but freshness and backfill discipline remain the team's responsibility.

Controls

The strongest control is to log the features actually served with each prediction and train on those logs, or at least compare them with offline-computed values for the same entity and timestamp and track the mismatch rate per feature. Other controls are schema and range validation on feature tables before materialization, pinning a model version to a feature service version, and treating a feature definition change like an API change with review and a migration plan. Distribution monitoring on served features catches drift that tests cannot.

Sources

Related lessons

Practice this
Storage for ML: lakehouse, warehouse and operational data storesOrchestrating ML pipelines with Airflow