Add up the wait.
A user's wait is a stack of smaller waits. Pick a scenario, hover a stage, and toggle fixes. Some fixes save a pile of time. Some save nothing because that layer was not your problem.
Request waterfall · one visible action
rough model · plausible numbersDisabled fixes target stages that are already near zero. That is how you know the shiny optimization has no job here.
Averages hide the story.
Most requests can look fine while a small slow path ruins sessions. Add a lock wait, cold start, or cache miss to a tiny slice of traffic and watch the median stay calm while the tail catches fire.
"A p50 tells you what a typical user saw on a typical day. A p95 or p99 tells you where trust starts to erode."
from the article
Latency distribution · 4,000 sampled requests
simulated · lognormal + slow pathof page loads contain at least one request slower than your p99, when a page fires 30 backend requests.
P(hit tail) = 1 - 0.99k · assumes independent requests
A "1% of requests" problem becomes a "lots of sessions" problem once one page fans out to dozens of backend calls. Your p99 is not rare. It is just diluted.
A pool is an admission controller, not a throughput knob.
Here is the incident version: Poisson arrivals, a connection pool, and an 8 core database. Past about 8 concurrent queries, work starts sharing CPU and paying a contention tax. You show up with the pool open at 100 and p99 in seconds. Shrink the pool and the tail collapses. That is the HikariCP pool-sizing lesson in miniature: sometimes the fix is letting fewer requests in.
Pool simulator · 30 s per run
discrete-event · seededPool wait vs. execution
p99 vs. pool size: the U curve · swept with your current rps and query work
At pool = 100, pool wait looks great because the queue moved inside the database. Execution gets fat instead. Pull the pool back toward the cores line and the wait becomes visible and cheap. Go too far, down around 4, and you starve the database. That is the left side of the U.
OFFSET is an explicit latency bill.
Even with a matching ordered index, rows skipped by OFFSET must still be visited before
PostgreSQL discards them. Keyset pagination resumes from the prior page's cursor and reads the next 50 rows.
On a stable ordered dataset, both produce the same next page for very different bills. Drag the depth and race
them.
Pagination race · 1,000,000-row ordered tenant slice
cost model · ordered rows visitedOFFSET winner
KEYSET winner
This models sequential navigation with the prior page's cursor already available, not a random jump to
page 400. It assumes a stable snapshot so both queries identify the same rows; concurrent inserts can make
OFFSET pages drift. Keyset needs an index matching
(tenant_id, created_at DESC, id DESC).
Utilization is not the goal.
An inference server batches requests to keep the GPU busy, but the batcher buys utilization with queue time. This model puts a dynamic batcher (max batch 8) in front of one GPU. Raise arrival rate until utilization looks perfect. Then look at goodput: the rate of requests that meet the TTFT SLO. It can move the other way.
Dynamic batcher · 60 s simulated · TTFT vs. utilization
event-driven · seededBatch size distribution
SLO attainment vs. max_queue_delay · swept with your current rps and SLO
Two regimes are worth finding. Below ~14 rps, a batch of one keeps up, so forced delay only adds a TTFT
floor. Past ~14 rps, batches of one fall behind. At D=0 the GPU is pegged at 100%, the dashboard looks
happy, and the queue grows anyway. There, delay buys real capacity: SLO attainment rises, so goodput rate
jumps while utilization falls. Keep pushing D and the delay floor eats your SLO. The knob is
max_queue_delay_microseconds; the sweep chart plots attainment and utilization.
Same latency. Different wait.
Both responses take exactly the same total time to finish. One hides the answer until the end. The other shows progress after the first token. Streaming does not make the model faster; it moves the wait from silence to reading.
Perceived-wait race · identical end-to-end time
TTFC = time to first content. The streamed text makes the point while it runs.
The language fight is #7.
The article's default order of attack for ordinary web work. The ranking matters because every layer above your current fix keeps charging rent while you argue about the lower one. Hover a row to see which demo maps to it.
For specialized systems, follow the wait: model and data path first for AI, event topology first for realtime, storage layout first for analytics. Frameworks and languages still matter. They rarely matter first.