A4 — From a Workflow to a product: the evaluation console

Eleven modules taught you to make one execution durable. This page is about what sits on top of it: the API a browser calls, the list a user scrolls, the button that cancels. The scenario is deliberately not AgentRun — no web agent, no tool loop — but every decision in it is one the capstone already made, plus a product-side piece the course did not teach. Those product-side pieces (a read model, cursor pagination, an outbox, Server-Sent Events) are general engineering, not Temporal, and are stated here as such.

Six decisions follow. Each names the Temporal mechanism the course taught and the product piece the course stopped short of.

1. The run is a Workflow Execution

EvalRun is a Workflow and its Workflow ID is the run id (run-8c1f), so the API, the reviewer and the operator all address one execution, as they addressed agent-42. Each case is an Activity, run_case(run_id, case_id, model, config), on a gpu-eval Task Queue — module 10's lane: the set of Activities registered on gpu-eval is exactly the set that needs a GPU. It heartbeats, because a case runs for minutes and a preempted node should fail the attempt within a heartbeat timeout rather than at Start-to-Close. Its idempotency key is f"{run_id}:{case_id}", enforced by what the Activity calls — the inference endpoint's request id, the results table's primary key — not by the Activity itself. A key built from the Workflow ID and the case id is stable across attempts and across a continue-as-new; one built from the Run ID is not (module 4).

Cases are not Child Workflows. A case has one recorded result and no lifecycle anyone inspects on its own, and upstream's advice stands: when in doubt, use an Activity. A case becomes a Child Workflow when it acquires steps of its own — generate, judge with a second model, human-grade the disagreements — because then the retry unit is the step, not the case, and someone wants to open it in the UI. Hundreds of cases as Activities also fit the parent's history; a parent should not spawn more than 1,000 Child Workflow Executions.

The loop is AgentRun's with plan() replaced by a case list: fan out with bounded concurrency, record each result, and check is_continue_as_new_suggested() between batches with the remaining case ids as the snapshot.

2. The read model

The Runs list is the first thing the prompt asks for and the first trap. Do not build it on client.list_workflows(). Visibility is a search index Temporal updates asynchronously; upstream says it is built for finding and filtering across many executions, not for reading the current state of one, and that to react to a Workflow's progress you should follow its Event History or model the dependent work as Child Workflows rather than poll Visibility. List and Count calls also share one Visibility rate limit. A product list needs strongly consistent reads, joins to users and suites, and a page size independent of how many executions the Namespace retains. Upstream's own recommendation for business logic that needs execution state is to store it in an external datastore through Activities and fetch it directly from the store.

So the product database owns the read model and the Workflow writes to it through a projection Activity. This is the general pattern of a projection, or outbox: the Workflow is the write side, Postgres the read side, one Activity carries state across.

@activity.defn
async def record_run_event(ev: RunEvent) -> None:
    """Upsert one projection row. Idempotent by (run_id, seq): a retried attempt lands once."""
    async with pool.acquire() as db, db.transaction():
        await db.execute(
            "INSERT INTO run_events (org_id, run_id, seq, kind, payload, at) "
            "VALUES ($1, $2, $3, $4, $5, now()) ON CONFLICT (run_id, seq) DO NOTHING",
            ev.org_id, ev.run_id, ev.seq, ev.kind, ev.payload,
        )
        await db.execute(
            "UPDATE runs SET status = $2, done = $3, last_seq = $4, updated_at = now() "
            "WHERE run_id = $1 AND last_seq < $4",
            ev.run_id, ev.status, ev.done, ev.seq,
        )

seq is a counter in Workflow state, incremented once per emitted event, so it is monotonic per run and identical on every replay; the ON CONFLICT and the last_seq < guard are what make at-least-once execution harmless. The Workflow calls the Activity after each state change: running, case done, awaiting review, and a terminal one of published, cancelled or failed.

The list endpoint is a Postgres query with keyset (cursor) pagination on (created_at, run_id). The cursor is the last row's pair, so a page stays stable while new runs arrive, which offset pagination does not.

SELECT run_id, model, suite, status, done, total, created_at
FROM runs
WHERE org_id = $1
  AND ($2::text IS NULL OR status = $2) AND ($3::text IS NULL OR model = $3)
  AND (created_at, run_id) < ($4, $5)              -- the cursor
ORDER BY created_at DESC, run_id DESC LIMIT 50;

Search Attributes still exist: Org, RunStatus, Model, Suite, set at start and upserted where the status changes, as AgentStatus was. They are the operator's view — temporal workflow list --query "RunStatus = 'running' AND Suite = 'gsm8k'" during an incident — and defense in depth: a projection row that disagrees with the execution's Search Attributes is a bug report that writes itself. They are not the product's list.

3. Live progress and the log tail

Progress is the projection: done / total on the runs row, written by the same Activity per completed case. The log tail is a second append-only table, run_logs (run_id, seq, line), written by run_case itself as lines arrive — an Activity may write to a database; that is what Activities are for — with seq from a per-run sequence the database owns, since these writes happen outside the Workflow's counter.

The UI reads both through one Server-Sent Events endpoint that resumes from a cursor. SSE is a browser standard, not a Temporal feature; the property that matters is that the client re-sends the last id: it saw on reconnect, so a stream keyed by seq never duplicates and never skips.

@app.get("/runs/{run_id}/events")
async def run_events(run_id: str, after: int = 0, org: Org = Depends(current_org)):
    await require_run_in_org(run_id, org)                     # tenancy, decision 6
    async def gen():
        cursor = after
        while True:
            rows = await db.fetch(
                "SELECT seq, kind, payload FROM run_events WHERE run_id = $1 AND seq > $2 "
                "ORDER BY seq LIMIT 200", run_id, cursor)
            for r in rows:
                cursor = r["seq"]
                yield f"id: {cursor}\nevent: {r['kind']}\ndata: {r['payload']}\n\n"
            if rows and rows[-1]["kind"] in TERMINAL:              # published, cancelled, failed
                return
            await asyncio.sleep(1.0)                          # or LISTEN/NOTIFY
    return StreamingResponse(gen(), media_type="text/event-stream")

Why not a Query? The status Query is the strongly consistent answer for one run, and the course used it that way. But a Query is computed by a Worker: lab 11 recorded that with no Worker alive a Query hits its deadline while Describe answers. A browser tab polling a Query every second is a Workflow Task per poll per tab, served by the pool that should be evaluating, and it stops working at the moment the operator most wants the page — when the Workers are down. The projection answers from Postgres regardless. Polling GET /runs/{id}?after=<seq> with the same cursor is the fallback for clients that cannot hold a connection; rate-limit it per org.

Temporal now has a native answer to this decision, and you should know it exists. Workflow Streams is a durable, offset-addressed event channel hosted inside the Workflow: publishers append events, subscribers long-poll by Workflow ID and resume from their own offset, publishing is exactly-once per (publisher_id, sequence), the log is carried across Continue-As-New, and a stateless SSE proxy is upstream's own example. It is Public Preview in the Python SDK as of 2026-09 (temporalio.contrib.workflow_streams; see the upstream page), targets modest fan-out — tens of subscribers per Workflow — and subscribing is an Update per poll. It can replace run_logs and the polling loop. It does not replace the read model, because the Runs list still needs a store you can filter and join.

4. Cancel and retry failed cases

handle.cancel() is a request, not a stop: the Service records WorkflowExecutionCancelRequested, a Workflow Task is scheduled, and the Workflow code handles it. The API returns 202 when the RPC returns. When handle.cancel() returns, the API writes a cancel_requested event to the projection itself and the UI moves to cancelling; the run moves to cancelled when the Workflow's cancel handler — the except branch module 7 wrote — emits its terminal event and the execution closes as Cancelled. "Reflected quickly" is a promise about the projection, not about the GPU.

The GPU is the slow part. A case in flight on a gpu-eval Worker stops only because run_case heartbeats and checks: Activities must heartbeat to receive cancellations from a Temporal Service, and the request rides back on the heartbeat response. An Activity that does not heartbeat runs to completion after the cancel. So the except CancelledError in run_case is where the inference process is killed, the heartbeat interval bounds how long best-effort takes, and — module 7's warning — cancelling the Workflow does not terminate an external process on its own.

retry_failed starts a new Workflow Execution: not a Signal to the parent, which is closed, and not a restart under the parent's Workflow ID, which would erase the lineage.

n = await db.fetchval("SELECT count(*) + 1 FROM runs WHERE parent_run_id = $1", parent)
await client.start_workflow(
    EvalRun.run,
    EvalRunInput(run_id=f"{parent}-retry-{n}", org=p.org, model=p.model, suite=p.suite,
                 case_ids=failed_case_ids, parent_run_id=parent),
    id=f"{parent}-retry-{n}", task_queue=CONSOLE_QUEUE,
    id_reuse_policy=WorkflowIDReusePolicy.REJECT_DUPLICATE,
    memo={"parent_run_id": parent, "retried_cases": len(failed_case_ids)},
    search_attributes=TypedSearchAttributes([
        SearchAttributePair(ORG, p.org), SearchAttributePair(PARENT_RUN_ID, parent),
        SearchAttributePair(RUN_STATUS, "created"),
    ]),
)

The Workflow ID is the idempotency key on the start. Temporal guarantees at most one open Workflow Execution per Workflow ID, and REJECT_DUPLICATE extends that to closed ones within the retention period, so a double-clicked Retry raises WorkflowAlreadyStartedError on the second click instead of starting a second run. The parent id appears three times on purpose. As a Workflow argument it is the input the run depends on. As a Memo — non-indexed metadata returned when you describe or list executions — it is what an operator opening the child in the UI sees; upstream cautions that Memos are eventually consistent and should not hold data critical to the execution, which is why the argument exists. As the ParentRunId Search Attribute it makes ParentRunId = 'run-8c1f' a List Filter. The product's lineage is the parent_run_id column.

5. The approval gate

Publishing to the leaderboard is a state the run cannot leave on its own. After the last case the Workflow emits awaiting_review to the projection and blocks on workflow.wait_condition(lambda: self.approved_by is not None). A paused Workflow consumes no Worker compute while waiting, and a reviewer may take a week.

Approval is an Update, not a Signal, because the reviewer must see the write confirmed and because a bad request must leave no trace:

@workflow.update
async def approve(self, reviewer: str, command_id: str) -> str:
    self.processed_command_ids.add(command_id)
    self.approved_by = reviewer
    return f"approved by {reviewer}"

@approve.validator
def validate_approve(self, reviewer: str, command_id: str) -> None:
    if reviewer not in self.reviewers:                   # the org's reviewer set, a run input
        raise ValueError("not a reviewer for this run")
    if command_id in self.processed_command_ids:
        raise ValueError(f"command {command_id} already applied")
    if self.status != "awaiting_review":
        raise ValueError(f"run is {self.status}")

A validator is a read operation that may not block; it may check arguments and current state, so the duplicate check belongs there. If it rejects, the client is informed and the Workflow has no indication the Update was ever requested — no Event, no handler to drain. Lab 11's change_goal made the other choice, accepting the duplicate and returning ignored; both are correct, and rejecting is the one that keeps a retried click out of History. The API maps WorkflowUpdateFailedError to 403 or 409, command_id is minted when the button is pressed and reused on every retry (module 5), and the reviewer's identity comes from the API session, authorised there, and checked again here. Publishing is then an Activity publish_leaderboard(run_id), idempotent by run id, and published is the terminal event emitted after it. The UI shows awaiting_review from the projection like any other status.

6. Tenancy

Authorization lives in the API. org comes from the session, every query above carries WHERE org_id = $1, and require_run_in_org runs before any Workflow handle is obtained. Temporal does not know what an org is.

Two Temporal-side guards back that up. The Org Search Attribute — an opaque id, never a name or an email, because Search Attribute values are stored unencrypted in the Visibility store — lets an operator scope an incident to one tenant and lets a nightly reconciliation compare Postgres against Org = 'org_7f3a' AND ExecutionStatus = 'Running'. Per-org Task Queue prefixes (gpu-eval.org_7f3a) are how a dedicated GPU pool is sold to one tenant; the default is one shared lane with Fairness keyed by org, as module 10 did.

Namespaces are not the tenancy boundary, and upstream says so. A Namespace is a unit of isolation providing Workflow ID uniqueness, resource isolation and configuration boundaries — retention and archival are per Namespace — and a single Namespace is still multi-tenant: multiple applications or teams can share one but must coordinate on Workflow ID and Task Queue naming. Upstream's Namespace best practices start from one Namespace per use case and environment (payments-prd, orders-dev) and split later — per service or per domain, as in payments-checkout-prd — when rate-limit pressure, security requirements, ownership or troubleshooting justify the extra operational cost.

For customers, upstream's multi-tenant patterns page ranks four designs, most recommended first: a Task Queue per tenant; one Task Queue with Fairness keyed by tenant; shared Workflow Task Queues with an Activity Task Queue per tenant; and, last, a Namespace per tenant. That last design buys complete isolation — separate rate limits, credentials and dashboards — and upstream calls it practical only for a small number of high-value tenants that need a credential, rate-limit or compliance boundary: most teams manage it for fewer than 50. Each such tenant needs a new Namespace, its own Worker pool (at least two Workers for high availability) and its own credentials (as of 2026-09; see upstream pages) — and, from A3 and the definition above, its own Search Attribute registration and retention setting.

This console follows that ranking. The eval console and the agent runtime are separate Namespaces, and so are staging and production. Customers share one Namespace and are separated by authorization and by Org. The GPU lane uses the second design, which upstream suggests for many tenants on different service tiers, and switches to the first when a tenant buys a dedicated pool. A Namespace per customer is kept for the contract that demands its own credentials or data boundary.

The failure contract for this design

The capstone's ten questions, minus the two that belong to the loop (continue-as-new and versioning), asked of the two boxes this page added.

One ordering rule the table implies. The API inserts the runs row with status = 'creating' and only then calls start_workflow, with the Workflow ID chosen before either. If the start fails, a sweeper retries it with the same id, and a start that already happened is refused, so the sweeper is safe. If the insert fails, nothing started. Never start first and insert second: a crash between the two is a run the user cannot see. In general-engineering terms that is outbox-style: a durable intent row plus idempotent reconciliation. A canonical transactional outbox goes further — it writes the outbound message in the same database transaction and has a relay publish it — and the distinction is worth keeping straight if an interviewer presses on it.

Transfer table

Try it

Exercise 11.5 in labs/11-capstone-durable-agent-runtime/ is a working slice of this page against the dev server: in console/: an EvalRun Workflow on a console Task Queue whose cases are run_case Activities on gpu-eval — a fake GPU that sleeps in heartbeating ticks — a FastAPI app with a SQLite projection standing in for Postgres, the record_run_event upsert keyed by (run_id, seq), GET /runs with cursor pagination, GET /runs/{id}/events?after=<seq> as SSE, cancel through handle.cancel(), and retry-failed as a new Workflow with parent_run_id in a Memo and a Search Attribute. What it demonstrates is the failure contract above: kill the Worker between a projection write and its completion and count rows (one); disconnect the stream mid-run and reconnect with the last seq (no duplicate, no gap); submit Retry twice (one child); cancel during a case and time the gap between cancelling and cancelled against the heartbeat interval. The lab README records what each of those produced on the course dev server, and a six-test suite pins the same contract without Docker.

Where this is used

Lab 11.4's defense document is one Workflow with a failure column. A product is that Workflow plus a read model, a stream, a gate and a tenant boundary — the four things an interviewer means by "build the UI + API". Write the six decisions for your own capstone into the defense in the same order and check each against the transfer table: a decision with no product-side half is a Workflow, not yet a product.