Which Runtime Does an Agent Belong In? Temporal, Flink, Flink Agents

Twelve modules of this course and eleven of its sibling both use the word durable, and both mean it. Temporal keeps AgentRun alive through a kill -9; Flink keeps GatewayMeter's per-tenant totals correct through a docker kill. Both survive failure, both manage state, both replay. Someone building the platform around an LLM gateway will be offered both, and lately a third thing, Flink Agents, that puts an agent inside a Flink job. This page is the Flink-side counterpart of the Temporal course's "Where Temporal ends" section: what each runtime's primary abstraction is, what that abstraction makes cheap and what it makes awkward, and a procedure for deciding.

Two primary abstractions

Temporal's primary abstraction is a durable execution: one program's decisions, persisted as an Event History and replayed to reconstruct the program's state on whatever Worker picks it up next. AgentRun is one execution with one Workflow ID. Its state is self.step, self.goal, self.context_summary, held in memory on a Worker and rebuilt from History when that Worker dies. call_llm and execute_tool are Activities: the effects the program has on the world, recorded as scheduled, started and completed. The runtime's question is where is this run?

Flink's primary abstraction is a continuously evolving computation over streams and partitioned state. Upstream's lowest layer is "stateful and timely stream processing"; the job is a graph of operators, each holding keyed state, each advancing through event time by watermarks. The records are persisted upstream, in gateway-events; the state is snapshotted at checkpoint k; recovery restores the snapshot and rewinds the source to the offsets S_k recorded with it. Nothing about a single request's "program" is stored; what is stored is the current truth for every key. The runtime's question is what is true now across all keys?

Both replay. They replay different things. Temporal replays decisions, so Workflow code must be deterministic and every effect lives behind an Activity. Flink replays records, so the operator must be a pure function of (state, record) and every effect lives behind a sink, or is accepted as at-least-once.

The properties, side by side

Read the 30-day row twice, because it is the one that decides most cases. Both waits are cheap in compute. The difference is what the wait is attached to. Temporal attaches it to a run: the run exists to wait, and when it wakes the next line of run() executes. Flink attaches it to a key inside a job that is doing other things: the timer fires in on_timer, with whatever state the key has by then, and only if the job's clock got there, which for event time means the watermark, and module 5 showed what an idle partition does to that. The question "what should happen for this tenant in 30 days" is a natural Flink question when the answer is "emit a record". It is a Temporal question when the answer is "resume a conversation".

Which runtime for which sentence

The last three rows are the gateway this course has been metering. The pattern in them is the pattern of the whole page: the stream decides that something should happen and for which key; a run decides what happens next when that involves waiting.

Flink Agents, as documented

Everything in this section is from the Flink Agents documentation at the commit the course pins (as of 2026-09; see upstream page). Where a sentence is upstream's, it is quoted.

What it is. Upstream's description: "Apache Flink Agents is a streaming Agent OS for enterprise, production-grade scenarios, built as a sub-project of the Apache Flink community. It brings AI agents into the Flink streaming pipeline - an agent becomes a first-class operator in your real-time datastream, making AI decisions in the flow of live events rather than in response to human prompts." The last clause is the design commitment: an agent run is triggered by a record, not by a person.

Agents as operators. The unit of work is the agent run, and upstream defines it in terms you already own: "An agent run refers to a complete execution of an agent to process an input event. Each record from upstream will trigger a new agent run." Inputs "are partitioned by their keys. This corresponds to how data are partitioned by keys in Flink's Keyed DataStream." So an agent in Flink Agents is a keyed operator whose per-record work happens to involve a model. Everything from modules 2, 3 and 7 applies: it has a key, it has state behind that key, and it is checkpointed with the rest of the job.

Two agent shapes. A ReAct agent is constructed, not written: "the user only needs to specify the goal with prompt and provide available tools, and the LLM will decide how to achieve the goal and take actions autonomously." The Python shape upstream shows is ReActAgent(chat_model=ResourceDescriptor(...), prompt=my_prompt, output_schema=MyBaseModelDataType). A workflow agent is "an agent whose reasoning and behavior are organized as a directed workflow of modular steps, called actions, connected by events"; it is a subclass of Agent whose methods carry @action(EventType.InputEvent), send a ChatRequestEvent through ctx.send_event(...), and handle the ChatResponseEvent in a second action that emits an OutputEvent. Upstream notes that "this event-driven workflow forms a directed graph that may contain cycles." The built-in chat action closes the tool loop on its own: "If the model asks to call tools, chat_model_action sends a ToolRequestEvent instead of a final ChatResponseEvent. After the tools finish, it receives the matching ToolResponseEvent, appends the tool results to the chat history, and calls the model again."

Memory. Upstream classifies memory by visibility, retention and derivation, and every kind is single-key. Sensory memory lives for one run, in Flink state, and "is checkpointed by Flink for fault tolerance"; it is cleared when the run completes. Short-term memory lives across runs of the same key, also in Flink state, with exact retrieval and optional expiration. Long-term memory is "a persistent storage mechanism in Flink Agents for storing information across multiple agent runs with semantic search capabilities"; it "currently supports the Mem0 backend", needs a chat model, an embedding model and a vector store declared as resources, and is isolated at job, partition (key) and memory-set level. Note what the key means here: an alert-investigating agent keyed by model remembers what it learned about gpt-5.5 last time, and nothing about any other model.

Tools and MCP. A tool is a local function marked @tool on the agent class or registered on the environment with agents_env.add_resource("name", ResourceType.TOOL, Tool.from_callable(fn)), then named in the chat model's tools=[...] list; upstream uses the function's docstring to build the tool schema. An MCP server is declared with @mcp_server returning ResourceDescriptor(clazz=ResourceName.MCP_SERVER, endpoint=...), after which "all tools and prompts from the MCP server are automatically registered" and referenced by name like local ones. Parameters the model must not choose, upstream's example being a tenant id, can be marked injected and read from config or memory.

Integration with a DataStream or Table job. This is the part that makes it a Flink job rather than a framework beside one. Upstream's Python shape, kept exact:

env = StreamExecutionEnvironment.get_execution_environment()
agents_env = AgentsExecutionEnvironment.get_execution_environment(env)

output_stream = (
    agents_env.from_datastream(
        input=input_stream, key_selector=lambda x: x.id
    )
    .apply(your_agent)
    .to_datastream()
)

"The input DataStream must be KeyedStream, or user should provide KeySelector." The Table side is from_table(input=..., key_selector=...) and to_table(schema=..., output_type=...), with the note that Python currently requires both the schema and the type information. For GatewayMeter the input would be the alerts stream keyed by model, and the output another DataStream that any module-8 sink can take.

Deployment and monitoring. "Submitting Flink Agent jobs to the Flink Cluster is the same as submitting PyFlink jobs": flink run --jobmanager <addr> --python job.py, against a cluster upstream requires to be Flink 1.20.3 or higher, with Python 3.10 to 3.12. Metrics are Flink metrics in the Flink metric groups: numOfEventProcessed, numOfActionsExecuted, per-action counts, and token usage as action.<action_name>.model.<model_name>.promptTokens and completionTokens, visible in the same Web UI the course reads from module 5 on. Which means the meter of the agent's own spend is a metric on the job that meters everyone else's.

The caveats

Status. Upstream's own words: "Apache Flink Agents 0.x releases are preview versions. They may contain known or unknown issues, including potential security risks. The APIs and configuration are experimental and may change in backward-incompatible ways before 1.0." The same documentation lists "Suitable Use Case: Production" on its deployment page; hold both sentences at once. The version the installation page pins in its Maven example is flink-agents 0.3.0 against Flink 2.2.1, its source-build example names 0.4-SNAPSHOT, and Python 3.12 "requires Flink 2.1 or above and Flink Agents 0.3 or above" (all as of 2026-09; see upstream page). Nothing in modules 1 to 12 depends on any of it.

The LLM call is an external effect inside an operator. Reason from module 8. Flink's checkpoint covers state and offsets; it does not cover what an operator did to the outside world between checkpoints. Upstream says exactly this for its own base guarantee: "After recovery from a checkpoint, Flink Agents reprocess events that arrived after that checkpoint. As a result, any actions triggered by those events may be executed again." That is exactly-once output consistency, and upstream is careful that it "does not mean each event is processed only once." A model call re-issued on recovery costs tokens twice; a tool that sends an email sends it twice.

Upstream offers two mechanisms above that floor, and both are opt-in. Exactly-once action consistency needs "an external action state store" (Kafka or Fluss, as of 2026-09): "After recovering from a checkpoint, Flink Agents consult the external store and will not re-execute actions that were already completed." Durable execution is finer: wrap the call in ctx.durable_execute(fn, *args) and "the framework persists the result and replays it on recovery when the same call is encountered, so the function will not be called again." The built-in tool action runs each tool through durable execution. The Temporal shape is recognisable, and so is its edge: "If a failure happens after a function starts but before it completes and its result is persisted, the call will be re-executed"; the answer offered is a reconciler that asks the provider what happened, which is the module-4 idempotency key by another name. Two further limits are upstream's: action-level exactly-once "is guaranteed only if, after recovering from the same checkpoint, inputs for each key arrive in the same order as before recovery", otherwise it "falls back to exactly-once output consistency"; and durable replay is best-effort, so a call whose arguments differ on replay is re-executed. The consequence for the question in every module's title: if a TaskManager dies during an agent run, the state and the offsets recover to checkpoint k, the run for every record after S_k starts again, and whether the model is called again depends on which of these you configured and whether the call had finished.

No waiting. Python async actions "only support await ctx.durable_execute_async(...)"; asyncio.sleep and the rest "are NOT supported because there is no asyncio event loop." An agent run is a bounded piece of work on one record. It cannot pause for a person. That is not a gap in the framework; it is the abstraction.

The decision procedure

Five questions, in order. The first answer that is clear usually settles it.

1. Is the unit of work a single long-lived execution, or a key in a stream? A thing with a name and a lifetime (agent-42) is a Workflow. A thing that is true per key and changes with every record is an operator.
2. Must something wait for a human, or a day? If yes, Temporal, or a Temporal handoff from wherever the trigger came from. A Flink timer emits a record; it does not resume a conversation.
3. Is the answer "now, across all keys", or "this run's next step"? p99 per model is the first. "What should this agent do next" is the second.
4. Which side of the effect boundary is expensive to duplicate? If the costly effect is a sink write, Flink's transactional or idempotent sinks already solve it. If the costly effect is a model call or a tool with consequences, Temporal's Activity-with-key is the default that does not need configuring; in Flink Agents the same protection is a store and a durable_execute you must set up.
5. Who operates it? A Temporal run is debugged from one History; a Flink job is debugged from checkpoints, watermarks and backpressure. Pick the runtime whose instrument your on-call engineer already reads.

GatewayMeter and AgentRun, together

The two courses describe one platform. The gateway's metering is Flink: GatewayMeter reads gateway-events, and tenant-cost, model-latency and alerts are what is true now, per key, with the guarantees an invoice needs. The agent runs the gateway meters are Temporal: each AgentRun is one execution whose call_llm and execute_tool Activities are the requests that become RequestStarted and AgentStepCompleted events on the way in. The AgentStepCompleted(step, tokens_in, tokens_out) event type exists in the spine for exactly this reason: the stream is how the run's cost reaches the bill.

The seam between them is the alerts topic. An error-rate alert per model is a keyed record; "investigate this model" is a bounded, per-record piece of agent work; that is the row where Flink Agents would sit, reading alerts keyed by model, remembering per model what it found last time, and emitting a finding downstream. The moment the investigation needs to page someone and wait for the reply, it leaves the job: an Activity-shaped tool that Signal-With-Starts an AgentRun, which then owns the wait. Nothing in this appendix changes the capstone. Module 12's GatewayMeter has no agent in it, and the Temporal capstone's tenth question, where Temporal ends and Kafka begins, has the same answer from this side: the stream is the source of record for what happened; the run is the source of record for what was decided.

Where this is used

Nowhere in the labs, by design. Lab 12 (labs/12-capstone-gatewaymeter/) asks for a failure contract per operator, and exercise 12.4's defense document may name the alerts seam as the place an agent would attach and say why it is not attached today. The Temporal course's capstone (labs/11-capstone-durable-agent-runtime/, exercise 11.4) asks its tenth question from the other side; a reader who has done both should give one consistent answer to both.