Memory
prod orders-api-00089-kix

Judging freeze is on. Everything below is evaluated live; nothing is written to GitLab.

Memory Console · Orders API · GitLab Duo Agent Platform

Is it safe to let the agents ship?

Not on their own yet. 3 agent changes wait for a human, because no agent has reached the 0.80 hands-off line in the areas it touches; the best score is 0.55. 1 incident is still in the loop. 1 of 1 promise kept in production.

Incident #2 on the outer ring, 5 of 6 stages done. Inside: immune memory, promises, trust 0.55. Drag to turn it.
The trigger DB_POOL_SIZE=5 One default, with a 1.0 s wait for a free connection.
The cost · incident #2 40.1%of 631 orders failed 503 database_busy under a 40-user spike; p99 15,617 ms.
The memory 0 of 40fail, and it stays fixed Pool 20 from the agent's MR. Every later change to the pool is replayed against incident #2 first.
Review !5 supervised A one-minute 3D replay of incident #2. Also opens from ?mode=judge.
Scene 1 of 8
The mechanism

The immune loop

Outer ring: an incident's way from alert to antibody. Inner rings: what the system remembers: past incidents, promises and each agent's trust. The gate on the right decides whether a change ships alone or waits for you.

1 antibody · 1 promise · 4 agents

1/8
Before · pool of 5
40.1% failed

631 order requests on staging, 40 users, database slowed to 400 ms. p99 latency 15,617 ms.

Mention to fix MR
2:34min

The Incident Responder checked the immune memory, named the cause in 74 s, and opened MR !1 with a regression test.

After · pool of 20
0.0% failed

40 of 40 concurrent requests on a 400 ms database in the regression test. Production now runs a pool of 20.

The record

Incident #2, step by step

Times are from the issue, the merge requests and the ledger. Choose a step to see it in the loop.

Open incident #2
    Explainability

    Every decision, with its reason

    Live from GitLab and production, checked 01:19 UTC. Open a row to inspect the payload behind it.

    !5 INC-2 action 2: alert on sustained 503 database_busy
    gate label autonomy::supervisedno agent at 0.80 for its areachange
    waits for you
    Updated 2026-10-27 10:18 UTC. A person approves it; then the pipeline canaries it and the agent's trust moves with the result.
    !6 INC-2 action 3: add Retry-After to 503 database_busy
    gate label autonomy::supervisedno agent at 0.80 for its areachange
    waits for you
    Updated 2026-10-27 10:09 UTC. A person approves it; then the pipeline canaries it and the agent's trust moves with the result.
    !4 INC-2 action 1: runbook for 503 database_busy and DB pool sizing
    gate label autonomy::supervisedno agent at 0.80 for its areachange
    waits for you
    Updated 2026-10-27 10:04 UTC. A person approves it; then the pipeline canaries it and the agent's trust moves with the result.
    #2 Orders API returns 503 database_busy under order spikes
    next stage: Closedfix !1 mergedseverity 2
    5/6 stages
    An incident closes only when its fix is merged and it has become an antibody. Opened 2026-10-27 07:57 UTC. 
    [x] Opened
[x] Diagnosed
[x] Fix MR
[x] Fix merged
[x] Antibody
[ ] Closed   ← next
    kept
    The exact requests production must keep answering. A broken one reopens issue #3 and wakes Contract Breach. 
    above_threshold_requires_verification
POST /orders {"item": "contract-check-high", "quantity": 1, "total": 650}
expect {"equals": "pending_verification", "json_path": "$.status", "max_ms": 2000, "status": 202}
    just_above_threshold_requires_verification
POST /orders {"item": "contract-check-boundary-above", "quantity": 1, "total": 500.01}
expect {"equals": "pending_verification", "json_path": "$.status", "max_ms": 2000, "status": 202}
    at_threshold_is_confirmed
POST /orders {"item": "contract-check-boundary", "quantity": 1, "total": 500}
expect {"equals": "confirmed", "json_path": "$.status", "max_ms": 2000, "status": 201}
    below_threshold_is_confirmed
POST /orders {"item": "contract-check-low", "quantity": 1, "total": 20}
expect {"equals": "confirmed", "json_path": "$.status", "max_ms": 2000, "status": 201}
    INC-2 Orders API 503 database_busy under order spikes (DB pool 5 / 1.0s timeout too small)
    replayed when app/orders_api/db.py changesreplayed when app/orders_api/settings.py changesreplayed when app/orders_api/main.py changesreplayed when infra/cloudrun.tf changesreplayed when infra/variables.tf changes
    antibody
    Must hold on every replay: 
    error_rate < 0.01
p95_ms < 2500
p99_ms < 3500
fault {"slow_query_ms": 400}
    Read the postmortem
    Earned trust

    Earned trust, per agent and code area

    A change that survives the canary adds 0.05; a rollback takes 0.20. The gate merges on its own only at hands-off, 0.80.

    Ledger
    Agentauthcap superviseddataapiinfracap superviseddocsmemory
    Incident responder0 shipped 0.50
    supervised     		0.50
    supervised     		0.50
    supervised     		0.50
    supervised     		0.50
    supervised     		0.50
    supervised
    Close the loop0 shipped 0.50
    supervised     		0.50
    supervised     		0.50
    supervised     		0.50
    supervised     		0.50
    supervised     		0.50
    supervised
    Contract keeper1 shipped 0.50
    supervised     		0.50
    supervised     		0.50
    supervised     		0.50
    supervised     		0.50
    supervised     		0.55
    supervised
    Contract breach0 shipped 0.50
    supervised     		0.50
    supervised     		0.50
    supervised     		0.50
    supervised     		0.50
    supervised     		0.50
    supervised
    earned scorehands-off line, 0.80Below 0.50: assistedCapped areas never rise above their cap, whatever the score
    Act

    One step is yours.

    The gate held 3 changes because trust isn't earned yet. Approving starts the canary; the result moves the agent's score.

    Review !5 Pipelines Agent sessions .memory folder
    • Nothing merges on its own unless tests, immune replay, every promise and the security audit are green.
    • auth, infra and anything unmapped are capped at supervised, whatever an agent's score.
    • A rollback costs 0.20 trust and drops the agent back toward supervision.
    • Judging freeze is on: the system evaluates and reports, and writes nothing to GitLab.
    Check promise #3 against production
    curl -s -X POST https://orders-api-rjpdzrsxra-uc.a.run.app/orders \
  -H 'content-type: application/json' \
  -d '{"item":"judge-check","quantity":1,"total":650}'
# expect 202 and "status": "pending_verification"
    Ask the memory why a file matters (MCP)
    curl -s -X POST /mcp -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"why","arguments":{"path":"app/orders_api/db.py"}}}'
    Read the whole memory as JSON
    curl -s /api/memory