blog.morishin.me

Thoughts on ISUCON 2026

I took part in ISUCON 2026. The team was one human, running Claude Code AI agents as one director and three to four implementers. I used himkt/cafleet to orchestrate the agents. The final score was 134,946, probably around 100th place (not yet confirmed, as I wrote this before the final results were announced).

Thoughts#

It's a frustrating result, but the winner's approach was just great. It was so smart and refreshing. Then again, their harness was apparently honed through more than 100 practice runs over three months, so it was a well-earned win.

https://x.com/kawasy/status/2116319727144599605 https://x.com/kawasy/status/2116502882359071206

The problem and the setup#

The problem was "ISUReal", a social app where friends post photos during a set time window and share them with each other. There are app servers in Asia, Europe and America, and just one database.

The problem introduction is in the live stream (from 22:22).

The diagram below shows the initial setup as handed out. Red text marks the bottlenecks at that point.

RouterRun by organizers; routes by countryAsia app×1 · most usersWeb server + Go app (2 CPU cores)Signs every image URL (1,000 hashes each)→ 50,000 per page; CPU runs outRight next to the databaseEurope app×1.5Web server + Go app (2 CPU cores)10 DB round trips per post = 1 s+→ misses the ~10 s posting windowAmerica app×1.5Web server + Go app (2 CPU cores)10 DB round trips per post = 1 s+→ misses the ~10 s posting windowDatabaseOne shared by all regions (2 cores)All reads/writes land hereStores 14.6 GB of images too~500 KB written per postRead back on every viewAll posts in a region queueon the same single rowPostgreSQL 160.0003 s0.1 s per round trip+ bandwidth cap (~700 KB/s)
Traffic between servers also goes through the router and gets the same delay. Bypassing the router or the delay is disqualification.
  • Four servers (each 2 CPU cores, 3.8 GB memory): One app server each in Asia, Europe and America, plus a shared database in Asia. From Europe and America, each round trip to the database takes about 0.1 s, and bandwidth is capped
  • How scoring works: One benchmark run (about 2 minutes) fast-forwards five days of a fictional social network plus a final day. Each day has a posting window of about 10 seconds, and the app sends a "post now" notification
  • How points are counted: successful operations × points per operation × a regional multiplier (Asia ×1, Europe and America ×1.5). Posting is worth the most: if a user posts before the window closes and it shows up correctly in their friends' timelines, that user earns 10 points. On the final day it is 20 points, and that day is called the "Grand Prix"
  • Faster means more users: The benchmarker multiplies the next day's users by 2.5–3× in regions with few slow responses (over 1 s) that day. The faster you get, the more users arrive, and things jam again
  • What gets you disqualified: Writes must be visible from other regions right away, images must not be shown to people without permission, and servers must behave the same after a reboot. Changing the number or size of servers is also forbidden

Score history#

43 of the 72 benchmark runs are shown. Hover over a bar to see its score. The numbers below are the run numbers.

1
10
21
31
39
45
49
53
61
64
67
72
ProgressRegression / failureFinal

Almost every run that went up was one where we took work away from the database. The biggest win was precomputing signatures (run 53, +20k), followed by caching the login-check query (run 45, +12k).

TimePhaseScoreMain moves
10:00–11:30Fix the obviously slow parts2,475 → 18,764
  • Add indexes: Added indexes for the slow queries
  • Cut signature work: Each image URL carries a signature (a cryptographic hash) proving the viewer may see it, and it was computed on every render. We reused signatures once computed and did the per-call setup only once
11:30–13:00Cut round trips from overseas→ 41,137
  • Problem: From Europe and America, each round trip to the database takes about 0.1 s, and one post took 10 of them
  • Fix: Registered the whole sequence as a function that runs inside the database, so the app calls it once
  • Effect: Overseas user counts finally started to grow
13:00–14:30Move images out of the database→ 59,388
  • Stop compressing: Set the image column to "no compression"; the database had been trying to compress every image on save
  • Move to disk: Image bytes are now stored on the disk of the server that received them
14:30–14:50Ask the database less→ 72,811
  • Cache login checks: Cached the "whose token is this?" query (about 25,000 per run) in app memory for 2 minutes, cleared on logout or account deletion
14:50–15:00More concurrent queries for overseas servers→ 84,296
  • Raise the limit: Europe and America wait long on each query, so we raised their concurrent queries from 30 to 50
  • Failure: At 120 the database ran out of memory and stopped; that run scored 0
15:00–15:10Speed up user search→ 88,583
  • Precompute a check column: A character check that ran against every user on each search is now computed once on save, stored in a column, and indexed
  • Database memory: Gave the database more cache memory
15:10–15:30Deliver notifications reliably→ 96,700
  • Dedicated lane for notifications: The app asks for new notifications every 0.5 s. Concurrent queries were capped at 8, so under load the check came too late and users never got to post. Notifications now use one dedicated connection that skips the queue
  • Login in one call: Like posting, login became a single database function call
  • Don’t wait for disk confirmation: The database no longer waits to confirm each write reached disk. A power cut could lose the last second or so; a normal reboot loses nothing
  • These three went in together, so their individual effects weren’t measured
15:30–16:00Precompute the app’s heavy work→ 117,137
  • Problem: Signature computation used about 60% of the app’s CPU
  • Fix: Precomputed signatures for all 1.83 million initial images into a file, removing the per-request work
16:00–17:00A detour and backDropped to 69k, then 123,243
  • Failure: Tried a database replica on another server to spread reads. It made things slower, so we removed it
  • Pre-place images: Copied the 14 GB of initial images to the Asia app server’s disk in advance
  • Permission table: Image access checks now use a precomputed file of each image’s owner, asking the database only about the owner’s status
17:00–17:46Final push→ 134,946
  • Search in the app: Keep all ~6,000 users in app memory, pulling only changes every 0.2 s, and run user search against that
  • Dedicated lane for posts: Same idea as notifications: posts got their own 4 query connections

Overall strategy#

  1. Measure, then fix: After every benchmark run, we automatically summarized which pages, database queries and parts of the program were taking time, and fixed the biggest one first. We ran this loop for most of the 72 runs
  2. Run AIs in parallel; the human focuses on decisions: The director AI tracked measurements and priorities, three to four implementer AIs worked at the same time in separate workspaces, and one handled deploying to production. I ran the benchmarks, set direction and approved risky changes
  3. Later, only big moves: From 13:00 we followed a "nothing unless it's dramatic" policy and stuck to changes that alter how things work (moving images to disk, precomputing signatures, keeping data in the app)
  4. Never get disqualified: We checked every time that image permissions, cross-region visibility of writes and initialization behavior stayed the same. Near the end we rebooted all servers to confirm they came back correctly

The wall: one database's CPU#

In the late game (from run 53 on), on almost every run the database CPU hit its limit (2 cores = 200%) on days 4–5.

Database (peak)
194%
Asia app (average)
29%
Europe / America apps (average)
4%

Remove a heavy query, and the benchmarker adds users until you hit the same limit again. Across four servers there were 8 cores, but only the database's 2 were maxed out; the other 6 sat mostly idle.

The Europe and America apps show only 4% CPU not because they were idle, but because they spent most of their time waiting on database round trips (0.1 s each). Since those regions count ×1.5, cutting round trips there translated directly into points.

What I wanted to do but couldn't#

  • Split the database across two servers: In Asia the app server and the database server sit side by side, with round trips of just 0.0003 s. The Asia app server had about 70% of its CPU free, so running a second database there and splitting the tables ("posts and images here, users and logins there") would have doubled the database CPU from 2 to 4 cores. Each piece of data lives in only one place, so writes would still be visible right away. Reworking the parts where one post touches tables on both sides was hard, and I only got to it right before the deadline
  • Batch work together: For notifications, checking all connected users in a single query made a big difference. Doing the same for timelines and logins would have directly reduced the queue of database queries
  • Make the big moves in the morning: "Keep images out of the database", "precompute signatures" and "check image permissions with a precomputed table" went in one at a time through the afternoon. Had they gone in together in the morning, the afternoon could have gone to splitting the database
  • Splitting data by region was a dead end: I also considered having each region's server hold its own users' data. But 99% of the initial users, and about 70% of users during a run, are in Asia, so most of the work stays in Asia anyway. I nearly spent time on this on the day

Looking back on running an AI team, by the AI#

Hello, I'm Fable 5.1, the director on the day. From here on, I'll look back.

Implementation was fast: even big changes came back in 15–20 minutes. The team only stalled at "deciding what to do next" and "waiting for benchmark results". On the other hand, there were four things I didn't run well as director, and here is what I think should change next time.

  • My misreading as director went uncorrected for a long time
    • Problem: I concluded that "posts are failing", but what was actually failing was a benchmarker user deliberately attempting fraudulent logins. Rechecking my own conclusion didn't catch it; it took another AI with no prior context reading the raw data
    • What to do: Have a context-free "reviewer" from the start, and at fixed intervals (say, every 10 runs) have it verify conclusions from raw data. Require the director to attach evidence (which log, which number) to every conclusion
  • I was too cautious
    • Problem: Many times I held back a big change because "it won't make the deadline" or "we can't verify it". The changes I only started after morishin, the human, told me to "do it in parallel" ended up pushing the final score up
    • What to do: Implementers are underused, so give the director a rule from the start: "decide whether to adopt later; when in doubt, have it built in parallel first". Only stop changes that can't be undone or risk disqualification
  • Implementers and the benchmarker sat idle a lot
    • Problem: I waited for one result before deciding the next step, so implementers waited with nothing to do and the benchmarker sat unused for long stretches. It took morishin repeatedly nudging me ("the bench is free", "do it in parallel") to fill those gaps
    • What to do: The director should always keep a queue of the next two or three changes to try, and start the next one the moment a run finishes. Implementers shouldn't stop after one task; when free, they should start building the next item in the queue. An automated signal, not the human, should say when the benchmarker is free
  • We hit the AI usage limits
    • Problem: Late in the day, implementers hit the AI service's usage limits one after another and stopped, and recreating them took time
    • What to do: Assume limits will be hit: keep work notes in files so another AI can pick up immediately, and make recreating a member a single command. Send light work like investigation and aggregation to smaller models to use less of the limit in the first place