COURSE LAB

Lab 1: The Machine

Objective

Why: A restart demo only proves that a process can restart. This experiment shows where a Temporal program lives while no Worker process exists.

By the end: You will have one Workflow run, two Worker process identities, one model call, and a history showing which work belonged to the Temporal Service.

Before you run

  • Predict: If the Worker dies during a timer, what can advance without it?
  • Watch: Keep the Temporal Web UI open on agent-42 and watch its Event History.
  • Do not start twice: After the crash, restart only the Worker. The existing Workflow will continue.
  • Two routes, one experiment: The checkpoints name the browser workspace's buttons. If you chose the terminal route in setup, run the matching commands in Terminal reference at the end instead; the evidence you collect is the same.

AgentRun is the program this course grows. In this first stage, Workflow code plans one step, the call_llm Activity returns an answer, and a durable timer waits before the Workflow finishes.

Swipe to explore diagram →
Worker outage timeline Four moments in two lanes. Worker A runs and then dies. While no Worker exists, the Temporal Service records TimerFired and queues a task. Worker B later replays history and completes the run. 1. Before crash2. Crash 3. Outage4. Recovery WorkerService A runsthe code A is killedduring timer No Workeris running B replaysand finishes Result savedin history TimerStartedalready logged TimerFiredTask queued Run resultrecorded Only the Service lane advances during the outage.
The Service records TimerFired and queues a Workflow Task while no Worker runs. Worker B replays when it returns.
Run the experiment in the local workspace

After the one-time setup below, use the address printed by lab_server.py, choose Lab 01, and follow its controls. They prepare the environment, start the Worker and Workflow, show process output, and crash the Worker.

First time here? Set up the local workspace

You need Docker and Python 3.11 or newer (check with python3 -V; macOS ships 3.9, so brew install python@3.12 and type python3.12 where these steps say python3). Everything runs on your machine against a local Temporal dev server; no Temporal Cloud account is used.

1. Download the starter (all eleven labs, free during the public beta) and start Temporal:

curl -fL https://scimigo.com/downloads/temporal-durable-execution/free/latest/temporal-course-starter.zip -o temporal-course-starter.zip
curl -fL https://scimigo.com/downloads/temporal-durable-execution/free/latest/temporal-course-starter.zip.sha256 -o temporal-course-starter.zip.sha256
shasum -a 256 -c temporal-course-starter.zip.sha256
unzip temporal-course-starter.zip
cd temporal-course-labs
docker compose up -d

On Linux, use sha256sum -c temporal-course-starter.zip.sha256 for the checksum step.

Keep the Temporal Web UI open for the experiment.

Then choose a route. Both run the same experiment and reach the same checkpoints. The browser workspace gives you a button for each step and needs Agent Runtime installed once (steps 2 and 3). The terminal route skips both: create the Python environment below, then follow Terminal reference in place of the buttons.

Terminal route only:

python3 -m venv .venv
./.venv/bin/pip install -r requirements.txt

2. Browser workspace: install Agent Runtime. On macOS, run this command by itself and wait for Installed Agent Runtime. If the download times out, retry it before continuing.

curl -fsSL https://raw.githubusercontent.com/SciMigo/agent-runtime/main/scripts/install-macos.sh | bash

Only after installation succeeds, start the runtime and leave it running:

agent-runtime serve --port 9477
Linux or manual Agent Runtime installation
git clone https://github.com/SciMigo/agent-runtime.git
cd agent-runtime
python3 -m venv .venv
./.venv/bin/pip install -e .
./.venv/bin/agent-runtime serve --port 9477

3. Browser workspace: in another terminal, start the lab pages:

cd temporal-course-labs
python3 lab_server.py

Open the http://127.0.0.1:... address the server prints. It uses port 3000, or the next free port if another app has 3000. The status strip should show both Agent Runtime and Temporal as ready. If Chrome changes that address to /en, it has likely cached a redirect from another local app; use an Incognito window.

Goal: crash the process running AgentRun, watch Temporal advance the run without application compute, and use the history to explain how a new Worker continues it.

Run the experiment

Use the local browser workspace linked from this page. Its action cards choose the directory, Python environment, and lab-01 Task Queue for you.

Checkpoint 1 — Start, then crash

  1. Click Prepare this lab once.
  2. Click Run Worker, then Start Workflow.
  3. Open the Worker's Output. As soon as call_llm key=... appears, click Crash Worker. The Activity has completed and the Workflow's 30-second timer is now pending, so there is no need to race the clock while copying evidence.
  4. Leave the Worker dead. Do not click Start Workflow again.

In the Temporal Web UI, open agent-42 and confirm that its history contains ActivityTaskCompleted and TimerStarted. Record the Worker identity on the latest Workflow Task and the one call_llm line from Worker Output. The Workflow should remain Running: it has not failed, rolled back, or completed.

Checkpoint 2 — Observe progress with no Worker

Wait until at least 45 seconds after TimerStarted, then refresh the Event History. The 30-second timer fires first; the dead Worker's sticky queue then needs roughly 10 seconds to time out.

Look for this sequence (event numbers can differ if you reran the experiment):

TimerFired             the Service's clock advanced
WorkflowTaskScheduled  first offered on the dead Worker's sticky queue
WorkflowTaskTimedOut   no Worker took that sticky task
WorkflowTaskScheduled  reoffered on the shared queue, waiting for compute

This is the observation the lab is built around: new Events appeared while none of your application code was running.

Checkpoint 3 — Resume the existing run

  1. Click Run Worker again. This starts a new Worker process.
  2. Watch the existing agent-42 run complete. Do not start another Workflow.
  3. Compare the final Workflow Task identity with the identity you recorded before the crash.
  4. Open Worker Output and count call_llm key=... again. On your first attempt, the total is still one. The browser keeps output from earlier attempts, so after a rerun count only the lines belonging to the current experiment.

The new Worker finished code that uses the Activity result, but it did not call the model again. The recorded ActivityTaskCompleted Event supplied that result during replay.

Explain the evidence

First, complete this short ownership table from what you observed:

Thing Worker or Temporal Service?
Run plan()
Run the call_llm Activity
Store the Event History
Count down the durable timer
Record TimerFired
Rebuild Workflow state after the crash

Then connect each observation to a claim:

Evidence What it proves
TimerFired appeared with no Worker
A Workflow Task waited until a new Worker arrived
The final Workflow Task has a different Worker identity
call_llm ran once in this experiment

Finally, answer one question in your own words:

Where was the program counter while the Worker was dead?

Check your answer Temporal persisted Events. A Worker re-executed deterministic Workflow code against those Events to reconstruct the state. Temporal did not serialize and restore the Python coroutine stack.

Optional extension — Two Workers (terminals only)

The browser workspace intentionally gives one process to each action card, so clicking Run Worker twice reports that the Worker is already running. Use two terminal windows for this extension.

Start two copies of ../.venv/bin/python worker.py before starting a fresh Workflow ID. Kill the Worker that handled the first Workflow Tasks while the timer is pending. Compare the delay after TimerFired with the no-Worker outage above.

The surviving Worker can take the Task immediately. Sticky execution is a cache optimization, not the source of durability; a Worker without the cache can reconstruct state by replay.

Terminal reference

Use this route instead of the browser action cards, not in addition to them. From the repository root, start the dev server and install the Python environment once:

docker compose up -d
python3 -m venv .venv        # Python 3.11 or newer: check with python3 -V
./.venv/bin/pip install -r requirements.txt

In the first lab terminal:

cd 01-the-machine
export TASK_QUEUE=lab-01
../.venv/bin/python worker.py

In a second terminal, enter the same directory and use the same Task Queue:

export TASK_QUEUE=lab-01
../.venv/bin/python starter.py

During the timer, run the kill -9 <pid> command printed by worker.py. After the history reaches the shared-queue WorkflowTaskScheduled Event, run ../.venv/bin/python worker.py again.

For the two-Worker extension, open a third terminal in 01-the-machine, export the same Task Queue, and start a second ../.venv/bin/python worker.py before starting a fresh Workflow ID.

You are done when

You can point to one Event or log line for each claim:

  • the Service owned the timer;
  • a new Worker continued the existing run;
  • replay reused the completed Activity result; and
  • no Python stack was restored.

Files

Every file in this lab, as committed. Copy or clone; nothing here is hidden.

01-the-machine/starter.py
import asyncio
import os
import sys

sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
from common import TASK_QUEUE, connect, show_history  # noqa: E402
from workflows import AgentRun  # noqa: E402
from temporalio.exceptions import WorkflowAlreadyStartedError


async def main() -> None:
    client = await connect()
    workflow_id = os.environ.get("WORKFLOW_ID", "agent-42")
    try:
        handle = await client.start_workflow(
            AgentRun.run, "summarize the Temporal docs", id=workflow_id, task_queue=TASK_QUEUE
        )
    except WorkflowAlreadyStartedError:
        print(
            f"{workflow_id} is already running. After crashing the Worker, run the Worker again "
            f"to resume this Workflow; do not start {workflow_id} a second time.",
            file=sys.stderr,
        )
        raise SystemExit(2) from None
    print(f"started {handle.id} on {TASK_QUEUE!r} run={handle.result_run_id}; open http://localhost:8233")
    print("result:", await handle.result())
    await show_history(client, handle.id)


if __name__ == "__main__":
    asyncio.run(main())
01-the-machine/worker.py
import asyncio
import logging
import os
import sys

sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
from common import TASK_QUEUE, connect  # noqa: E402
from temporalio.worker import Worker  # noqa: E402
from workflows import AgentRun, call_llm  # noqa: E402

# Show the `call_llm key=...` line, so you can count how many times the Activity really ran (1.2).
logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s")


async def main() -> None:
    client = await connect()
    worker = Worker(
        client,
        task_queue=TASK_QUEUE,
        workflows=[AgentRun],
        activities=[call_llm],
    )
    print(f"worker pid={os.getpid()} polling {TASK_QUEUE!r}; kill -9 {os.getpid()} to crash it", flush=True)
    await worker.run()


if __name__ == "__main__":
    asyncio.run(main())
01-the-machine/workflows.py
"""Lab 1: AgentRun, stage 1 — plan one step, call the model once, sleep, finish.

This is the program the whole course grows. Nothing here is Temporal-magic: an
Activity performs external work, and its outcome becomes part of the Event
History; a Workflow is deterministic code whose Commands, and the results that
came back for them, are recorded, so a fresh Worker can rebuild its state by
replaying them. Temporal never stores `self.step`; it stores enough history to
recompute it.

The Task Queue name is deployment configuration, not Workflow logic, so it lives
in common/ and only worker.py and starter.py read it.
"""
from __future__ import annotations

import asyncio
import hashlib
from dataclasses import dataclass
from datetime import timedelta

from temporalio import activity, workflow


@dataclass
class Step:
    kind: str          # "llm" | "finish"
    prompt: str = ""


@activity.defn
async def call_llm(prompt: str, key: str) -> str:
    """The model call as an Activity. `key` is the idempotency key a retry
    would reuse (module 4 puts a cache behind it). Stage 1 fakes the model."""
    activity.logger.info("call_llm key=%s", key)
    return f"[model answer to: {prompt[:40]}]"


@workflow.defn
class AgentRun:
    def __init__(self) -> None:
        self.step = 0
        self.status = "running"
        self.goal = ""
        self.context_summary = ""

    def plan(self) -> Step:
        """Workflow code: a pure function of state. It never calls a model."""
        if self.step == 0:
            return Step("llm", prompt=f"Make a one-line plan for: {self.goal}")
        return Step("finish")

    @workflow.run
    async def run(self, goal: str, sleep_seconds: int = 30) -> str:
        self.goal = goal
        while True:
            step = self.plan()
            if step.kind == "finish":
                self.status = "done"
                return self.context_summary
            key = f"{workflow.info().workflow_id}:{self.step}:{hashlib.sha256(step.prompt.encode()).hexdigest()[:12]}"
            answer = await workflow.execute_activity(
                call_llm, args=[step.prompt, key], start_to_close_timeout=timedelta(seconds=10)
            )
            self.context_summary = answer
            self.step += 1
            # A durable timer: recorded as TimerStarted; fires as TimerFired; no process waits.
            await asyncio.sleep(sleep_seconds)

Lab source: labs/01-the-machine in the course repository. Verified against temporal server start-dev with the Python SDK 1.32 on 2026-09-26.