Durable Execution with Temporal
A two-day, hands-on course on Temporal for engineers who build long-running systems — especially AI agents. You learn the execution model (history, replay, determinism, activity failure semantics) by breaking it on purpose, then build a durable agent-job runtime that survives Worker crashes, pauses for days, accepts new instructions, delegates to sub-agents, and outlives deployments. After the eleven modules you can reason architecturally about Temporal and build a serious durable agent runtime — a narrower promise than 'Temporal expert in two days', and the one this course keeps.
About This Course
Temporal's central idea is small: your application logic is normal code, but Temporal persists the decisions that code makes so execution can be reconstructed after failure. The Service stores an append-only Event History; Workers execute your code; Workflow code must be deterministic because the SDK may replay history to rebuild state; everything non-deterministic — HTTP, databases, LLM calls, files — lives in Activities.
This course does not follow the usual hello-world-to-order-processing path. The capstone is a durable agent runtime, because that is what Temporal is unusually good at: long-running execution, pause and resume, external tools, retries, checkpoints, human intervention and code upgrades. Every module adds one capability to that runtime and one failure you inject to prove it: the same program, AgentRun, grows from a one-step Workflow in module 1 to the production design you defend in module 11. Day 1 is the machine — history, replay, determinism, activities, messages. Day 2 is architecture — continue-as-new, children, sagas, testing, versioning, and the final design defense.
The labs run on a self-hosted Temporal dev server with the Python SDK. The Web UI is not an appendix: reading Event History before reading code is a skill this course drills from module 2 onward.
Course design original to SciMigo, developed from a two-day learning path for durable agent runtimes. Reading pages adapt Temporal's MIT-licensed documentation and Python samples (Temporal Technologies Inc.); see THIRD_PARTY_NOTICES.md. Temporal is a trademark of Temporal Technologies; this course is independent and not endorsed by Temporal.
Prerequisites
- Comfortable Python, including async/await basics
- Have shipped a service that talked to a database or an API and had to handle retries
- Docker installed locally (labs run a Temporal dev server in a container); no prior Temporal
What You Will Learn
- Explain what survives a Worker crash and how replay reconstructs Workflow state from Event History
- State the determinism rule and recognize the non-deterministic code patterns that break replay
- Choose Activity timeouts and retry policies deliberately; design idempotent Activities that tolerate at-least-once execution
- Make a running Workflow interactive with Signals, Queries and Updates, and say which is which
- Bound history with Continue-As-New and choose between Activities and Child Workflows for delegated work
- Orchestrate compensation (sagas) and reason about cancellation of in-flight Activities
- Write Workflow, Activity and replay tests; run days-long Workflows in a time-skipping test environment
- Deploy changed Workflow code safely with patching and Worker Versioning
- Read a broken Workflow's history in the Web UI and answer: what happened, what did Temporal think happened, what will happen next
- Design and defend a production agent runtime on Temporal: task-queue topology, idempotency, versioning, history bounds, observability
Terminology Mapping
How classic concepts map to the terminology used in this course.
| Classic | This Course (Python) |
|---|---|
| Workflow Definition / Type / Execution | @workflow.defn class with one @workflow.run method; started by Client.start_workflow |
| Activity | @activity.defn function, invoked via workflow.execute_activity with timeouts and a RetryPolicy |
| Event History | temporal workflow show --workflow-id ... ; WorkflowHistory in temporalio.client |
| Signal / Query / Update | @workflow.signal / @workflow.query / @workflow.update handlers; Client handle .signal() / .query() / .execute_update() |
| Durable Timer | await asyncio.sleep(...) inside Workflow code (the sandbox makes it durable) |
| Continue-As-New | workflow.continue_as_new(args) |
| Child Workflow | workflow.execute_child_workflow(...) |
| Replay test | temporalio.worker.Replayer(...).replay_workflow(history) |
| Patching | workflow.patched('id') / workflow.deprecate_patch('id') |
Your Learning Path
Each module builds on the last. Take your time—the AI tutor is with you at every step.
The Machine — Service, Worker, Client, Namespace, Task Queue — and where the program counter lives
How a Temporal application is split between the Service that records and the Workers that execute. You start a dev server, run one Workflow with a durable sleep, kill the Worker mid-sleep, and watch execution continue — then answer the first real question: where was the program counter stored?
Event History and Replay — The append-only log that lets any Worker reconstruct a Workflow's state
The most important hour of the course. Event History is the durable source of truth; replay is how a fresh Worker rebuilds Workflow state from it. You read a real history in the Web UI and in JSON, map every Command to its Event, and see why a DAG engine and Temporal are not the same kind of thing.
Determinism — What Workflow code may and may not do, taught by breaking it
Replay only works if the code makes the same decisions from the same history. You write a Workflow that flips a coin, watch the SDK throw a non-determinism error on replay, and derive the rule: Workflow = decisions, Activity = effects. Covers the sandbox, deterministic time and randomness, and side effects.
Activities and Failure Semantics — Retries, the four timeouts, heartbeats, and why an Activity can run twice
Activities are the boundary between Workflow code and the outside world. A Worker can die after the side effect and before the completion, so Temporal may run the Activity again: assume every Activity can execute more than once — Temporal guarantees durable orchestration, not exactly-once execution of your effect. You trigger each of the four timeouts on purpose, add a heartbeat to a long Activity, and design an idempotent tool call for an LLM agent.
Signals, Queries, and Updates — Talking to a running Workflow: asynchronous writes, reads, and tracked synchronous writes
Signals, Queries and Updates are the three ways to talk to a running Workflow Execution: an asynchronous write, a read, and a tracked synchronous write. You add pause/resume/cancel Signals, a status Query, and a change-goal Update to the agent, see how handlers interleave with the main loop, then leave it paused, kill every Worker, and resume it a day later.
Continue-As-New and Child Workflows — Keeping history bounded and choosing the right unit of delegation
Event History is bounded, and a long-running agent loop will reach the bound. Continue-As-New checkpoints the state you pass into a fresh run under the same Workflow ID; the trigger is the Service's suggestion, not a number. You also choose, for each delegated task, between an Activity and a Child Workflow.
Cancellation and Compensation — Retry repeats an operation; compensation undoes one that succeeded
Temporal will retry for you but it cannot know your rollback semantics. You build reserve-GPU → allocate-sandbox → start-model → register-endpoint, make the last step fail, and orchestrate the compensating steps in reverse — the Saga pattern as ordinary durable code. Then you cancel a running Workflow and see what cancellation actually does to in-flight Activities.
Testing and Replay Tests — Unit tests, time-skipping, replaying production histories against new code
Three kinds of tests: Workflow tests in the time-skipping environment (a three-day Workflow finishes in milliseconds), Activity tests, and replay tests that take histories from production and replay them under new code to catch non-determinism before deploy. Then failure injection: Worker crash, Activity timeout, duplicate execution, cancellation.
Deploying Changed Workflow Code — Patching and Worker Versioning: changing code that has been running for months
A Workflow started two months ago may replay against the code you deploy today. You learn both mechanisms — patching inside the code, and Worker Versioning (GA since March 2026) which pins executions to a deployment version and supports ramped rollout and rollback — and how they interact with Continue-As-New.
Operating Temporal in Production — Task Queue lanes, Worker pools, visibility, priority — and where Temporal ends
AgentRun leaves the laptop. Task Queues become compute lanes with their own Worker pools; backlog becomes the scaling signal; Search Attributes and the UI answer what any run is doing; Priority and Fairness arbitrate which queued work reaches a shared GPU lane, while the compute platform still owns GPU placement and lifecycle. And the boundary question: Temporal decides where execution logically is — Kubernetes, Kafka and Postgres still own what they own.
Capstone: A Durable Agent Runtime — Design and defend AgentRun — every box, every failure semantic
The final exam, and the thing you came to build. Derive the AgentRun architecture from its requirements alone — 72 hours, 1,000+ model calls, vanishing GPUs, twice-daily deploys, pause and re-instruction, sub-agents, bounded history, exact diagnostics — defend every box's failure semantics, then compare it with a real one: the course-job workflow inside the engine that rendered these slides.