Stateful Stream Processing with Apache Flink
A two-day, hands-on course on Apache Flink for engineers who run systems that never stop — especially the infrastructure around AI models and agents. You learn the execution model (partitioned state, event time and watermarks, distributed snapshots, exactly-once) by breaking it on purpose, then operate a real-time metering pipeline for an LLM gateway that survives TaskManager deaths, late data, stalled partitions, slow dependencies and hot tenants without ever billing a minute twice. After the twelve modules you can reason architecturally about stateful stream processing and build a metering or monitoring pipeline whose numbers you would put on an invoice.
About This Course
Flink's central idea is small: a computation over an unbounded stream is a set of parallel operators, each with local state partitioned by key, whose progress through time is measured by watermarks and whose state is made recoverable by consistent distributed snapshots. Everything else — windows, timers, backpressure, savepoints, exactly-once sinks — follows from those four mechanisms.
This course does not follow the word-count-to-Kafka-connector path. The one program, GatewayMeter, is the analytics job behind an LLM gateway: it turns request events into per-tenant cost, per-model latency and error alerts, continuously, with the guarantees a bill requires. Every module adds one capability to that job and one failure you inject to prove it. Day 1 is the machine — tasks and slots, keys, state, time, watermarks, windows. Day 2 is correctness and operations — checkpoints, exactly-once, backpressure, rescaling and upgrades, production, and a capstone in which every failure happens at once.
The labs run on a self-hosted Flink cluster and Kafka in docker compose, with the Python DataStream API. The Web UI is not an appendix: reading a checkpoint, a watermark and a backpressure signal off it is a skill this course drills from module 5 onward.
Course design original to SciMigo, developed from a two-day learning path for stateful stream processing. Reading pages adapt the Apache Flink documentation, the Flink Kafka connector documentation and the Flink Agents documentation (Apache License 2.0, © The Apache Software Foundation); see THIRD_PARTY_NOTICES.md. Apache Flink, Flink and the Flink logo are trademarks of The Apache Software Foundation; this course is independent and not endorsed by the ASF or the Flink project.
Prerequisites
- Comfortable Python; some familiarity with async/await
- Have run a service that consumed a queue or a log (Kafka, SQS, Kinesis) and had to think about duplicates
- Docker installed locally (labs run Flink and Kafka in containers); no prior Flink
What You Will Learn
- Explain what a Flink job is at runtime — JobManager, TaskManagers, slots, tasks, chains — and what survives a TaskManager death with and without checkpoints
- Choose keys deliberately and explain why keyed state can only live behind a keyBy
- Use keyed state, state TTL and timers to keep per-entity state bounded
- State the difference between event time and processing time and choose the right one for a billing computation
- Explain what a watermark claims, who generates it, how it propagates through parallel operators, and diagnose a stalled watermark
- Build tumbling, sliding and session windows with triggers, allowed lateness and side outputs; predict when a late event changes an emitted result
- Explain checkpoint barriers, alignment and unaligned checkpoints, and trace a recovery from checkpoint k
- Qualify every use of 'exactly-once' with its scope and build an end-to-end exactly-once path with a transactional or idempotent sink
- Read backpressure off the metrics and relieve it with async I/O, chaining and buffer configuration
- Rescale and upgrade a stateful job through savepoints without losing state — and know the two ways to lose it
- Detect and mitigate a hot key; read the six metrics that predict trouble
- Design and defend a production metering pipeline on Flink: failure contract per operator, key topology, time, checkpoints, sinks, observability
Terminology Mapping
How classic concepts map to the terminology used in this course.
| Classic | This Course (Python) |
|---|---|
| Job / JobGraph / ExecutionGraph | StreamExecutionEnvironment.get_execution_environment(); env.execute('GatewayMeter') submits the graph |
| Operator / Task / Subtask | each .map / .key_by / .process / .window call is an operator; parallel instances are subtasks; chained operators form one task |
| Keyed State | ValueState / ListState / MapState obtained from runtime_context.get_state(...) inside a KeyedProcessFunction after key_by |
| Event Time Watermark | WatermarkStrategy.for_bounded_out_of_orderness(Duration.of_seconds(20)).with_timestamp_assigner(...).with_idleness(...) |
| Window / Trigger / Allowed Lateness | .window(TumblingEventTimeWindows.of(Time.minutes(1))).allowed_lateness(...).side_output_late_data(tag).aggregate(...) |
| Timer | ctx.timer_service().register_event_time_timer(ts) in process_element; on_timer(ts, ctx) fires it |
| Checkpoint | env.enable_checkpointing(10_000, CheckpointingMode.EXACTLY_ONCE); env.get_checkpoint_config() for timeout, min pause, unaligned |
| Savepoint | flink stop --savepointPath ...; flink run -s <path> (CLI); job.uid('tenant-cost') on every stateful operator |
| Exactly-once sink | KafkaSink.builder().set_delivery_guarantee(DeliveryGuarantee.EXACTLY_ONCE).set_transactional_id_prefix(...) |
| Async I/O | AsyncDataStream.unordered_wait(stream, AsyncFunction, timeout, capacity, output_type) |
Your Learning Path
Each module builds on the last. Take your time—the AI tutor is with you at every step.
The Machine — JobManager, TaskManagers, slots, operators, parallelism — and what a job is when nothing is checkpointed
What runs where. A Flink job is a dataflow graph deployed as parallel tasks across TaskManagers, coordinated by a JobManager. GatewayMeter reads the gateway's event stream and counts events per model. You kill a TaskManager and learn the first hard fact: without checkpoints there is nothing to recover from.
Streams, Keys, and Operators — Partitioning, keyBy, key groups, operator chains — the same request must always meet the same subtask
How records find their operator. keyBy is a routing decision that also decides where state can live. GatewayMeter assembles each request's lifecycle from its events; drop the keyBy and one request's start and completion land on different subtasks and no record is ever emitted.
Stateful Operators — Keyed state, operator state, TTL — the running total that outlives the record that changed it
State is a local key/value store partitioned with the stream. GatewayMeter keeps an open-request record per request and a running dollar total per tenant. Requests that never complete leave state behind forever; you bound it with TTL and a cleanup timer.
Event Time and Processing Time — Which clock a minute belongs to — and why the gateway's timestamp, not the wall clock, decides the bill
Two notions of time, one of them wrong for billing. GatewayMeter computes cost per tenant per minute; a 30-second retry delay makes processing-time windows bill the wrong minute. Event time bills the right one, at the price of deciding when a minute is over.
Watermarks — Bounded out-of-orderness, per-partition generation, the minimum rule, idleness — who decides that a minute is over
A watermark is a claim that no earlier event will arrive. It is generated per source partition from the data seen, and an operator's event time is the minimum of its inputs. One slow partition holds every window in the job hostage; you diagnose it from the watermark metrics and fix it with idleness and alignment.
Windows and Timers — Tumbling, sliding, session; triggers, allowed lateness, side outputs; event-time timers — a late event can change an answer already sent
Windows are state plus a trigger plus a timer. GatewayMeter gains sliding p99 latency per model, session-like abandoned-request detection, and error-rate alerts. A late RequestCompleted re-fires a closed cost window and a second result for the same minute reaches the sink — the first appearance of the duplicate the rest of the course is about.
Checkpoints — Barriers, alignment, snapshots, recovery — the distributed snapshot problem, and Flink's answer to it
The module the course is built around. A checkpoint is a consistent cut across every operator's state and every source's position, drawn by barriers that flow with the records. You kill a TaskManager between checkpoints and watch state restore to checkpoint k and Kafka rewind to S_k: per-tenant totals neither lost nor double-counted inside Flink.
Exactly Once — Replayable source, checkpointed state, transactional or idempotent sink — what the phrase means and where it stops
Exactly-once state semantics are Flink's; end-to-end exactly-once needs the source and the sink to cooperate. The sink writes a cost record, the TaskManager dies before the checkpoint completes, the record is written again after recovery, and the billing consumer double-bills. Two fixes: a transactional Kafka sink with read_committed consumers, or an idempotent upsert keyed by tenant and window.
Backpressure — Network buffers, credit-based flow control, chaining, async I/O — what a slow downstream call does to everything upstream
Flink propagates slowness backward by design. GatewayMeter enriches records with a negotiated price from the billing service; when that service takes two seconds per call the whole pipeline slows, Kafka lag climbs and checkpoints time out. You read backpressure off the metrics, then relieve it with async I/O, capacity and unaligned checkpoints.
Scale and Upgrade — Savepoints, rescaling, max parallelism, operator UIDs, state schema evolution — changing the job without losing the state
A savepoint is a checkpoint you own. GatewayMeter goes from 4 slots to 32 through a savepoint; you meet the max-parallelism ceiling and the missing-uid trap, then evolve the state schema of a running job and upgrade it in place.
Operating GatewayMeter in Production — Skew and hot keys, the metrics that matter, HA, disaggregated state — one tenant becomes 40% of traffic
Production is the job you cannot restart on a whim. One tenant becomes a hot key and pins one subtask while 31 idle; you split the aggregation in two stages, read the metrics that would have told you first, and see what high availability and disaggregated state change about the operational picture.
Capstone: GatewayMeter Under Fire — The whole job, a failure contract per operator, and every failure at once
Derive the production GatewayMeter from its requirements rather than assemble it from the modules: write the failure contract for every operator first, then draw the job. The injection suite runs a TaskManager death, a stalled partition, a slow billing service and a hot tenant simultaneously, and the bill must still be right.