What performance really means in software engineering

A system is never slow in the abstract. Some piece of work is waiting on the browser's main thread, a TLS handshake, a cold serverless runtime, a connection pool, a row lock, or a GPU scheduler. Performance engineering begins by locating that wait. It then makes the queue visible, puts a bound on it, and decides whether the work belongs on the user's path at all.

The clock does not outrank correctness, cost, reliability, operability, security, accessibility, consistency, or the hardware users actually hold. Serving stale private data instantly is an incident. Serializing JSON in microseconds does little for a product that ships a 3 MB client bundle. Record token throughput is beside the point when the person at the other end is still staring at an empty box. Language rankings, framework leaderboards, and a green Lighthouse screenshot pasted into a launch document can all describe part of the system; none describes the user's wait by itself.

A user's wait now crosses the browser, network, CDN, edge or serverless runtime, application service, database pools, locks, queues, third-party dependencies, realtime fanout, analytics paths, and increasingly an inference engine. Treating "the system" as one process hides most of the places where latency accumulates.

This guide uses one deliberately narrow model: performance is queue ownership. That includes the queues in software and infrastructure as well as the human-operated incident process. Once the wait has an owner and a budget, it can be shortened, bounded, or moved somewhere the product can afford it.

Start with the dominant wait

No stack is "most optimized" for every kind of software. A sound default removes the dominant wait from the user's path; the location of that wait does more to determine the architecture than a language or framework ranking does.

Without more information about a serious web application, I would start here:

HTML-first frontend + CDN/edge cache
regional Go, .NET, Java, Rust, Node, Rails, Django, or Laravel service chosen by team fit
managed PostgreSQL in the same region as the write path
Redis/Valkey only for derived/cacheable state
object storage for files
queue/workflow engine for slow or unreliable work
OpenTelemetry + RUM + explicit SLOs

This starting point is fast enough for a wide range of products, inexpensive to operate, observable, and reversible. It also avoids turning an application into a distributed-systems incident before the workload requires distribution.

Its cloud mapping is mostly mechanical. On GCP, one version uses Astro, SvelteKit, or Next in static or hybrid mode behind Cloudflare, Fastly, or Cloud CDN; Cloud Run, with GKE reserved for workloads that need it; Cloud SQL PostgreSQL; Memorystore; Cloud Tasks, Pub/Sub, and Workflows; and Cloud Storage. On AWS, the corresponding pieces are CloudFront, ECS Fargate or Lambda, RDS PostgreSQL Multi-AZ, ElastiCache, SQS, EventBridge, Step Functions, and S3. The vendor names can change without disturbing the shape: deliver HTML first, cache where correctness allows, keep compute near the data it reads and writes, put slow work behind queues, and measure what users receive rather than what a staging machine reports.

Give every layer a veto

A performance review should resolve these competing views in a matrix, not a leaderboard:

1. Backend throughput charts are irrelevant when the app ships too much JavaScript, blocks on hydration, lazy-loads the LCP image, or lets a tag manager wreck INP.
2. Most public pages should never touch a centralized origin at request time.
3. Edge compute does not help when every request still crosses an ocean to reach one write database.
4. A live product needs room locality, sequencing, backpressure, and reconnection semantics; adding WebSockets to CRUD supplies none of them.
5. GPU queues, vector filters, OLAP scans, and prompt bloat can dominate an otherwise quick web stack.
6. Incident risk, staffing, rollback speed, and the monthly bill remain part of the performance design.

Optimize the path in this order

For end-user software, optimize in this order unless your product has a strong, specific reason to invert it:

1. Perceived responsiveness: LCP, INP, CLS, time to first useful content, and (for AI) time to first token.
2. Tail latency: p75 for web experience, p95/p99 for APIs, queue age for async work, goodput under SLO for inference.
3. Data locality: put compute near the data it must read or write, which often matters more than being near the user.
4. Browser work: JavaScript parse/compile/execute, hydration, long tasks, layout shifts, images, fonts, third-party scripts.
5. Network delivery: DNS, TLS, HTTP/2 or HTTP/3, CDN hit rate, compression, cache keys, media transformation.
6. Backend execution: serialization, allocation, event-loop stalls, pool waits, locks, retries, GC pauses, cold starts.
7. Overload behavior: admission control, backpressure, queue bounds, retry policy, load shedding, circuit breakers.
8. Operability: observability, rollback speed, restore tests, failure isolation, cost, and incident simplicity.

Write the user's wait as a sum. Each term is a queue you can own:

User-visible wait =
  browser scheduling
+ network and protocol setup
+ CDN cache miss / revalidation
+ runtime startup or queueing
+ application execution
+ database connection wait
+ database execution and lock wait
+ downstream dependency wait
+ realtime fanout or inference queueing
+ response transfer
+ browser rendering or token streaming

Averages conceal the users who wait longest. The p50 describes a typical observation. The p95 and p99 expose the tail where a system becomes unreliable enough for users to stop trusting it, and regaining that trust costs more than the few milliseconds that were saved elsewhere.

Use public numbers as a baseline

Core Web Vitals remain the most useful public baseline for web experience: Largest Contentful Paint within 2.5 seconds, Interaction to Next Paint at 200 milliseconds or less, and Cumulative Layout Shift at 0.1 or less, all measured at the 75th percentile and segmented by mobile and desktop.¹ INP became an official Core Web Vital on March 12, 2024, replacing First Input Delay, because FID only ever captured the first interaction and missed most of the real responsiveness story.²

The public web is improving, but slowly. The 2025 Web Almanac, drawing primarily from July 2025 HTTP Archive and CrUX measurements, reports good Core Web Vitals for only 48% of mobile websites and 56% of desktop websites.³ Page weight is one reason: the median home page is 2,559 KB on mobile and 2,862 KB on desktop. Images are the largest byte category, followed by JavaScript and fonts.⁴

Some case studies connect speed with business outcomes, but their scope matters. A Google-commissioned study by fifty-five and Deloitte found that a 0.1 second improvement in mobile site speed correlated with 8.4% higher retail conversions and 10.1% higher travel conversions.⁵ Rakuten 24 reported a 53.37% lift in revenue per visitor and a 33.13% lift in conversion rate from its Core Web Vitals A/B test.⁶ These results are directional evidence, not laws of nature or a promise that a rewrite will pay for itself.

Write the budget before choosing the stack

A budget forces the team to define "fast" and name the tradeoffs before an infrastructure choice hardens around vague expectations.

web:
  lcp_p75_mobile: '<= 2.5s'
  inp_p75_mobile: '<= 200ms'
  cls_p75: '<= 0.1'
  initial_js_content_pages_compressed: '<= 150KB target; exceptions require review'
  lcp_image_lazy_loaded: false
  rum_required: true

api:
  common_interaction_p95: '150-300ms regional target'
  common_interaction_p99: '500-800ms regional target'
  dependency_timeout_budget: 'explicit per downstream'
  queue_age_slo: 'route-specific'
  retries: 'bounded, jittered, only where safe'

database:
  pool_wait_p95: 'tracked separately from query execution'
  slow_query_threshold: 'route-specific'
  lock_wait_alerting: true
  connection_pool_policy: 'small, measured, admission-controlled'

realtime:
  sender_to_receiver_p95: 'product-specific'
  per_client_queue_bound: 'hard limit'
  reconnect_recovery_tested: true
  ephemeral_messages_may_drop: true
  durable_messages_sequenced: true

ai:
  ttft_p50: 'route-specific'
  ttft_p95: 'route-specific'
  itl_or_tpot: 'route-specific'
  slo_attainment_target: 'percentage of requests meeting TTFT + TPOT + E2E targets'
  goodput_target: 'requests per second meeting TTFT + TPOT + E2E targets'
  cost_per_successful_request: 'tracked'

The values vary by product, but the structure should remain recognizable. A proposed change that cannot be tied to one of these budget lines is still speculative.

Browser performance: bytes before frameworks

Browser performance is largely a problem of bytes, scheduling, and priority. Before showing anything useful, the median page asks the browser to download, parse, execute, lay out, paint, hydrate, and coordinate more work than the user's device can comfortably absorb. A different server router does not remove any of it.

Ship less JavaScript before the page is useful

JavaScript costs more than its transfer size. The browser has to decompress, parse, compile, and execute it while coordinating rendering and hydration. Code that does not contribute to the first useful interaction should stay off that path.

• Split by route and interaction, not just by package boundary.
• Don't hydrate static content that will never become interactive.
• Prefer islands, partial hydration, or server components where they cut client work without hiding waterfalls.
• Push low-value analytics and third-party scripts behind consent, interaction, idle time, or a server-side alternative.
• Re-measure long tasks and INP after every dependency bump, not once a quarter.

Real-user measurement belongs in production from the beginning:

// rum/web-vitals.ts
import { onCLS, onINP, onLCP, type Metric } from 'web-vitals';

type RumPayload = {
  name: Metric['name'];
  value: number;
  rating: Metric['rating'];
  id: string;
  route: string;
  deviceMemory: number | undefined;
  connection: string | undefined;
  ts: number;
};

function send(metric: Metric) {
  const nav: Navigator & {
    connection?: { effectiveType?: string };
    deviceMemory?: number;
  } = navigator;

  const payload: RumPayload = {
    name: metric.name,
    value: metric.value,
    rating: metric.rating,
    id: metric.id,
    route: document.body.dataset.route ?? 'unknown',
    deviceMemory: nav.deviceMemory,
    connection: nav.connection?.effectiveType,
    ts: Date.now(),
  };

  const body = new Blob([JSON.stringify(payload)], { type: 'application/json' });
  navigator.sendBeacon('/rum/web-vitals', body);
}

onLCP(send);
onINP(send);
onCLS(send);

A sitewide average is only an entry point. Break the data down by route, template, country, device class, network type, logged-in state, and release version. A regression confined to one of those dimensions can disappear inside an acceptable headline number.

Put the LCP element on the critical path deliberately

The LCP element is usually a product image, hero media, headline block, or card. Its discovery and rendering deserve the same care as an API dependency because the user's first impression is waiting on it.

A high-priority image path might look like this:

<img
  src="/images/hero-1280.avif"
  srcset="/images/hero-640.avif 640w, /images/hero-1280.avif 1280w, /images/hero-1920.avif 1920w"
  sizes="(max-width: 1280px) 100vw, 1280px"
  width="1280"
  height="720"
  fetchpriority="high"
  alt="Product screenshot showing the main workflow"
/>

Most LCP failures can be avoided with a short set of habits. Do not lazy-load the image likely to appear above the fold or hide it behind client-only rendering. Give it explicit dimensions to prevent layout shifts, serve responsive AVIF or WebP variants instead of one oversized original, and preload only when the browser cannot discover it early. A redundant preload competes with the resources it was meant to prioritize.

Fonts need the same discipline: use WOFF2, subset aggressively, limit weights, avoid loading an entire family for one heading, and choose font-display behavior deliberately. Preloading too many fonts reverses the intended priority.

Cache immutable assets like immutable assets

Hashed JavaScript, CSS, fonts, and media receive a new URL whenever their contents change. They can therefore be cached for a long time and marked immutable. MDN defines immutable for responses that will not be updated while fresh, which matches cache-busted URLs.⁷

# /assets/app.4f3a9c.js
Cache-Control: public, max-age=31536000, immutable

HTML can carry deploy rollout, personalization, experiments, authentication state, and asset-hash references. Cache it in a shared store only when the correctness model for those inputs is explicit.

# Anonymous content page that can tolerate brief staleness
Cache-Control: public, max-age=0, s-maxage=60, stale-while-revalidate=300

# Account-specific page
Cache-Control: private, no-store

An immutable bundle and a logged-in dashboard do not share a correctness model. Caching them as though they do can put private data into a shared cache.

CDN and protocol performance: shorten the route safely

A CDN shortens distance, reuses connections, terminates TLS near users, caches safe responses, and absorbs bursts. In exchange, it introduces cache fragmentation, accidental personalization leaks, Vary explosions, stale pricing, and protocol choices that can perform well in a lab but poorly on users' last-mile networks.

The 2025 Web Almanac CDN chapter shows ecosystem patterns, but its DNS, TLS, and TTFB measurements come from simulated Chrome connections on controlled infrastructure rather than real-user measurement.⁸ Protocol policy should come from your own RUM.

For a working CDN:

• Keep static assets on long-lived, immutable URLs.
• Cache anonymous HTML and API responses only when correctness is safe.
• Use s-maxage for shared-cache lifetime and stale-while-revalidate where stale data is acceptable.
• Keep Vary small; Vary: User-Agent alone can shred your hit ratio.
• Use cache tags or surrogate keys for product and category invalidation.
• Measure CDN TTFB, origin TTFB, cache-hit ratio, and origin-shield hit ratio as separate numbers.
• Test HTTP/2 and HTTP/3 against your audience, not against a global average.

An anonymous product-list response using Fastly might carry these headers. The capture uses Fastly-Debug: 1, which prevents Fastly from stripping Surrogate-Key before delivery:⁹

Cache-Control: public, max-age=0, s-maxage=120, stale-while-revalidate=600
Surrogate-Key: category:running-shoes brand:acme catalog:v42
Vary: Accept-Encoding
Server-Timing: cdn-cache;desc="HIT", origin;dur=0

Compare it with this:

Cache-Control: public, s-maxage=3600
Vary: User-Agent, Cookie, Accept-Language, X-Experiment, X-Geo, Authorization

The second response is unlikely to hit cache. Worse, any personalization input missing from Vary can leak state. A well-run CDN should produce high hit ratios, predictable invalidation, low regional variance, no accidental private data, and enough telemetry to explain a miss.

What framework benchmarks can measure

Synthetic server benchmarks can measure routing overhead, JSON serialization, a controlled database access pattern, and whether a technology imposes an obvious ceiling on a specific class of workload. They cannot decide whether a product belongs in Go, Rust, .NET, Node, Python, or Rails. The benchmark harness does not contain the team, production data, or likely failure modes.

TechEmpower, once the most visible public web-framework benchmark, ended on March 24, 2026. Its repository was archived read-only after more than a decade.¹⁰ Round 23 is now historical ceiling data for a narrow workload.

Use such benchmarks for order-of-magnitude checks, comparisons between implementation styles in one ecosystem, and early detection of hard ceilings in specialized workloads. They do not predict p99 under real traffic, choose a product architecture, or justify rewriting an application whose framework was never the bottleneck. Five serial external calls, OFFSET 50000, unbounded database connections, hydration of static content, and no path for slow work will overwhelm any benchmark winner.

Edge and serverless compute: cold starts are design variables

Serverless platforms make different tradeoffs around startup, isolation, concurrency, and placement, so "convenient but slow" is no longer a useful description of the category. The mechanics matter more than the label.

Cloudflare Workers run on V8 isolates, lightweight contexts that start very quickly inside an existing runtime and allow one runtime instance to host hundreds or thousands of isolates.¹¹ The model suits routing, auth gates, personalization, A/B decisions, cache orchestration, and small API facades. It fits less well when work is CPU-heavy, dependencies are large, mutable in-memory state is assumed to be durable, or the request must cross regions to reach a write database.

Vercel Fluid compute allows multiple invocations to share a function instance, favors existing idle resources before allocating new ones, and adds bytecode caching and pre-warming.¹² It reduces rather than eliminates cold-start effects: first requests, newly scaled paths, and cold regions can still incur startup cost. Enable it with "fluid": true in vercel.json when the workload benefits from concurrency, streaming, and warmer instances.

AWS Lambda SnapStart initializes a function version at publish time, takes a Firecracker microVM snapshot of that state, and resumes new execution environments from the encrypted cache rather than initializing them from scratch.¹³ It supports Java 11+, Python 3.12+, and .NET 8+. SnapStart mitigates startup latency but supplies no correctness layer: functions must still handle uniqueness, credentials, sockets, and network connections correctly after a restore, as AWS explicitly warns.¹³

Choose among these platforms by matching startup, concurrency, limits, and topology to the workload.

Databases: where backend latency accumulates

After frontend and network costs are controlled, backend latency often leads to the database. The usual causes are missing indexes, inefficient pagination, excess round trips, lock contention, long transactions, N+1 queries, and connection pools sized as though a database were a thread pool.

Connection pools enforce concurrency limits

PostgreSQL's max_connections caps concurrent connections, and the server sizes certain resources, including shared memory, directly from that value.¹⁴ A connection pool is therefore an admission-control mechanism, not a throughput dial. Below saturation, more connections can reduce application-side waiting. At saturation, they move the queue inside the database, where it is harder to observe. Beyond that point, wider pools add context switching, memory pressure, lock contention, and p99 latency.

HikariCP's pool-sizing guidance reports a case in which shrinking the pool alone reduced application response times from roughly 100 ms to roughly 2 ms.¹⁵ Small pools are not universally faster; the result applies once the database is saturated and each additional caller slows the rest.

With PgBouncer transaction pooling, many stateless application instances can share fewer server connections because a client holds one only for the duration of a transaction.¹⁶

; pool-sizing excerpt from pgbouncer.ini
[databases]
app = host=postgres.internal port=5432 dbname=app

[pgbouncer]
listen_port = 6432
pool_mode = transaction
max_client_conn = 1000
default_pool_size = 25
reserve_pool_size = 5
server_idle_timeout = 60

Transaction pooling breaks some session-based PostgreSQL features, so the application must account for the different semantics. Review session state, session-level advisory locks, temporary-table assumptions, and some prepared-statement behaviors before enabling it.¹⁶

Composite indexes follow the query shape

A multicolumn B-tree index is useful when the query constraints align with its leading columns. PostgreSQL specifies that equality constraints on those columns, plus an inequality on the first column without one, limit the portion of the index that must be scanned.¹⁷

Begin with the query:

SELECT id, status, total_cents, created_at
FROM orders
WHERE tenant_id = $1
  AND status = $2
  AND created_at < $3
ORDER BY created_at DESC
LIMIT 50;

Then match the index to its shape:

CREATE INDEX CONCURRENTLY idx_orders_tenant_status_created_at
ON orders (tenant_id, status, created_at DESC);

Build indexes from observed predicates, sort order, cardinality, and access frequency, then confirm the plan with EXPLAIN (ANALYZE, BUFFERS).

OFFSET charges for every skipped row

PostgreSQL still computes rows skipped by OFFSET, so large offsets become slow.¹⁸ This is a poor fit for a hot path:

SELECT *
FROM events
WHERE tenant_id = $1
ORDER BY created_at DESC, id DESC
LIMIT 50 OFFSET 50000;

Keyset pagination avoids computing and discarding the preceding rows:

SELECT *
FROM events
WHERE tenant_id = $1
  AND (created_at, id) < ($2, $3)
ORDER BY created_at DESC, id DESC
LIMIT 50;

CREATE INDEX CONCURRENTLY idx_events_tenant_created_id
ON events (tenant_id, created_at DESC, id DESC);

Keyset pagination is faster because the database no longer has to recount and discard fifty thousand rows on every click. It also matches the interaction: the user moves from a known item to the next page. The cursor can be the (created_at, id) of the last row shown; the first page omits the cursor predicate, and created_at must be non-null for the comparison to be total.

Keep slow work out of the request path

Emails, exports, webhooks, billing retries, image processing, AI enrichment, and long third-party calls should almost always run outside the request path.

// request handler: enqueue and return
await queue.send('order.created', {
  orderId: order.id,
  idempotencyKey: `order.created:${order.id}`,
});

// worker: derive one stable key per idempotent side effect
const { idempotencyKey, orderId } = job.data;
await withIdempotency(`${idempotencyKey}:receipt`, () => sendReceipt(orderId));
await withIdempotency(`${idempotencyKey}:index`, () => updateSearchIndex(orderId));
await withIdempotency(`${idempotencyKey}:warehouse`, () => notifyWarehouse(orderId));

The receipt still has to be sent; moving it does not erase the work. The queue bounds the user-facing path, allows safe retries, and turns queue age into an explicit product-health metric.

Realtime architecture starts above the transport

A realtime design has to specify who owns room state, how ordering works, which events require durability, which may be dropped, what happens after reconnect, how missed events are replayed, how fanout is bounded, and what one slow client costs everyone else. Selecting WebSockets answers only the transport question.

WebSocket remains the compatibility baseline because it is reliable, ordered, bidirectional, and deployable nearly everywhere. Many products need nothing more.

MDN marks WebTransport as Baseline as of March 2026.¹⁹ Over HTTP/3, it provides reliable streams alongside unreliable, UDP-like datagrams. That combination fits games, live cursors, telemetry streams, and collaborative interactions in which some events may be dropped. Browser support does not guarantee passage through every enterprise proxy, old device, or corporate firewall, so a public product should feature-detect WebTransport, handle connection failure, and fall back to WebSocket:

type RealtimeTransport = {
  send(data: ArrayBuffer | string): void;
  close(): void;
};

export async function connectRealtime(url: string): Promise<RealtimeTransport> {
  if ('WebTransport' in globalThis && url.startsWith('https://')) {
    try {
      const transport = new WebTransport(url);
      await transport.ready;

      // Use a reliable, ordered byte stream to match the WebSocket fallback's
      // delivery guarantees. Message framing, heartbeats, and reconnect are omitted.
      const stream = await transport.createUnidirectionalStream();
      const writer = stream.getWriter();

      return {
        send(data) {
          const bytes = typeof data === 'string' ? new TextEncoder().encode(data) : data;
          void writer.ready.then(() => writer.write(bytes)).catch(() => transport.close());
        },
        close() {
          transport.close();
        },
      };
    } catch {
      // Fall through to WebSocket. Browser support is not the same as network support.
    }
  }

  const wsUrl = new URL(url);

  if (wsUrl.protocol === 'https:') {
    wsUrl.protocol = 'wss:';
  } else if (wsUrl.protocol === 'http:') {
    wsUrl.protocol = 'ws:';
  } else if (wsUrl.protocol !== 'wss:' && wsUrl.protocol !== 'ws:') {
    throw new Error(`Unsupported realtime protocol: ${wsUrl.protocol}`);
  }

  const socket = new WebSocket(wsUrl);

  await new Promise<void>((resolve, reject) => {
    socket.onopen = () => resolve();
    socket.onerror = () => reject(new Error('WebSocket failed'));
  });

  return {
    send(data) {
      socket.send(data);
    },
    close() {
      socket.close();
    },
  };
}

The example omits authentication, reconnect, heartbeats, backoff, replay, and capability negotiation. Production code needs all of them, with the fallback designed at the same time as the preferred transport.

Durable Objects and room authority

Cloudflare Durable Objects can coordinate many WebSocket clients in one instance. The Hibernation API keeps clients connected while an object is evicted from memory. An incoming message wakes it, but the lost in-memory state must be rebuilt from serialized attachments or storage.²⁰ This model fits chat rooms, multiplayer lobbies, presence channels, and collaborative sessions with a natural object boundary.

Here is the smallest version of a room:

import { DurableObject } from 'cloudflare:workers';

export class Room extends DurableObject {
  fetch(_request: Request): Response {
    const pair = new WebSocketPair();
    const [client, server] = Object.values(pair);
    this.ctx.acceptWebSocket(server);
    return new Response(null, { status: 101, webSocket: client });
  }

  webSocketMessage(ws: WebSocket, message: string | ArrayBuffer): void {
    for (const peer of this.ctx.getWebSockets()) {
      if (peer !== ws) {
        peer.send(message);
      }
    }
  }
}

A production room also needs sequence numbers, per-client queue bounds, authentication, replay, rate limits, and durable append logs for messages that cannot be lost.

Broadcast, presence, and database changes are different tools

Supabase Realtime's benchmark documentation separates Broadcast, Presence, and Postgres Changes, and reports k6 results against explicit cluster topologies. It also notes that Postgres Changes must check access for every subscribed user and recommends Broadcast patterns for higher-scale fanout.²¹

These tools remain distinct beyond Supabase. Broadcast handles application-level fanout. Presence holds ephemeral state, such as who is online. Database change streams notify consumers about durable state; they are not a general-purpose, high-volume pub/sub bus. Collaborative editing needs CRDTs or operational transform. Yjs-style systems pair naturally with append-only update logs, snapshots, and compaction, and their performance budget includes merge cost, replay cost, storage growth, and cold-client catch-up in addition to bytes on the wire.

AI performance: measure goodput under the SLO

One aggregate number cannot describe interactive LLM serving. The relevant metrics pull in different directions and need to remain separate:

Anyscale distinguishes raw throughput, the work performed each second, from goodput, the SLO-compliant work completed each second.²² If a system increases tokens per second by making users queue longer, it is busier without improving interactive performance.

Stream by default for interactive UX

A minimal SSE proxy can expose generation as it happens:

import * as v from 'valibot';

const modelRequestSchema = v.strictObject({
  messages: v.array(
    v.strictObject({
      role: v.picklist(['system', 'user', 'assistant']),
      content: v.pipe(v.string(), v.minLength(1)),
    }),
  ),
});

export async function POST(request: Request): Promise<Response> {
  const body: unknown = await request.json().catch(() => undefined);
  const input = v.safeParse(modelRequestSchema, body);

  if (!input.success) {
    return Response.json({ error: 'invalid_request' }, { status: 400 });
  }

  const modelUrl = process.env.MODEL_URL;
  const modelApiKey = process.env.MODEL_API_KEY;

  if (modelUrl === undefined || modelApiKey === undefined) {
    return Response.json({ error: 'model_not_configured' }, { status: 500 });
  }

  const upstream = await fetch(modelUrl, {
    method: 'POST',
    headers: {
      authorization: `Bearer ${modelApiKey}`,
      'content-type': 'application/json',
    },
    body: JSON.stringify({ messages: input.output.messages, stream: true }),
    signal: request.signal,
  });

  if (!upstream.ok || upstream.body === null) {
    return Response.json({ error: 'model_unavailable' }, { status: 502 });
  }

  return new Response(upstream.body, {
    headers: {
      'content-type': 'text/event-stream; charset=utf-8',
      'cache-control': 'no-store',
      'x-accel-buffering': 'no',
    },
  });
}

Streaming leaves model speed unchanged but exposes progress, reducing the perceived wait. A long answer that begins quickly is easier to tolerate than a shorter one hidden behind a silent spinner. Forwarding the incoming abort signal also makes client cancellation explicit upstream instead of relying on runtime-specific stream teardown.

KV cache memory is a first-order constraint

During autoregressive decoding, transformer serving retains attention keys and values in memory. That memory is often a tighter constraint than the computation itself. vLLM's PagedAttention work targeted this constraint and reported up to 24x higher throughput than HuggingFace Transformers in its benchmark setting. Its v0.6.0 update reported 2.7x throughput and 5x faster TPOT on Llama 3 8B versus v0.5.3, plus 1.8x throughput and 2x lower TPOT on 70B.²³²⁴ Those multipliers belong to their benchmark configurations. The transferable finding is that memory layout, cache fragmentation, batching, scheduling, and the distributions of prompt and output lengths all shape serving performance.

A self-hosted server exposes those choices as configuration:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 8192 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.90

Test the flags against your prompt lengths, output lengths, hardware, and SLOs. Their useful values vary enough that guessing wastes GPU time.

Batching is a latency-throughput trade

NVIDIA Triton uses dynamic batching to combine inference requests and increase throughput, primarily for stateless models. Stateful workloads use its sequence batcher instead.²⁵

# config.pbtxt
max_batch_size: 8

dynamic_batching {
  max_queue_delay_microseconds: 5000  # 5 ms; tune against your TTFT SLO
}

Batching trades queue time for accelerator utilization. Too much delay degrades p95 TTFT even while utilization improves. For an interactive workload, goodput under SLO determines whether the trade is worthwhile.

An inference dashboard should report queue time, prefill time, TTFT, ITL/TPOT, token counts, GPU and KV-cache utilization, batch size distribution, cancellation rate, goodput, and cost per successful request separately. The product determines which of them leads:

Benchmark inference engines with your model, hardware, prompt distribution, batching policy, and SLO. A winner under someone else's inputs may lose under yours.

Vector search: begin inside PostgreSQL

For many products, pgvector inside PostgreSQL is the right first choice. Embeddings, metadata, permissions, tenancy, and transactional state remain together, keeping operations simple while comfortably serving small-to-medium RAG systems.

A dedicated vector system becomes justified when the working set outgrows comfortable PostgreSQL operation, recall and latency tuning require controls pgvector does not expose, metadata filtering becomes the bottleneck, index maintenance interferes with OLTP work, or retrieval must scale independently of transactions. Qdrant, Milvus, Pinecone, Weaviate, or another dedicated serving path can then take over retrieval.

Until one of those conditions creates a measured bottleneck or real operational risk, a separate vector database adds an operational boundary without removing one.

Analytics: separate scans from transactions

Transactional databases serve writes, point reads, constraints, and consistency. Analytics brings scans, aggregations, joins, and time windows. Running both on the OLTP primary lets analytical demand degrade the write path.

A typical analytics-heavy stack:

PostgreSQL for OLTP
CDC or event stream into ClickHouse, BigQuery, Snowflake, or Redshift
object storage with Parquet
DuckDB for embedded/local/ad hoc analysis
cached dashboard queries with honest freshness labels

Once analytics matters to users, separate it from transactions. Pre-aggregate common dashboard paths, use materialized views where the freshness tolerance permits them, run exports as background jobs, and show freshness in the UI. "Updated 3 minutes ago" is usually preferable to an 8-second wait for data that did not need to be live.

CPU-bound and batch work: build around the job

CPU-bound jobs often do not need the web stack. A practical default is a Rust, C, C++, Zig, or Go binary running in a container on Cloud Run Jobs, AWS Batch, ECS RunTask, or bare metal, with object storage for input and output and a queue or manifest table assigning the work:

wordfreq \
  --input 's3://raw-corpus/2026-06/*.txt.zst' \
  --output s3://derived/wordfreq/2026-06-23/result.parquet \
  --shards 32

Track bytes/sec, records/sec, CPU seconds, peak memory, and cost per job. A good native binary on one machine is often enough. Distribution becomes necessary only when that machine misses a real SLO; a job that consumes a file, CPU, and memory to produce an artifact has no inherent need for Kafka, Kubernetes, or a service mesh.

Application-specific defaults

Each row in the opening matrix implies a different operating model.

Public content, docs, marketing, editorial

Astro, Eleventy, Hugo, or static Next/SvelteKit
static HTML on a CDN
image pipeline (Cloudflare Images, imgix, Cloudinary, or equivalent)
headless CMS with webhook rebuild or on-demand revalidation
RUM for Core Web Vitals

Because public pages are mostly read-only, their HTML can be generated ahead of time and deliver meaningful content before JavaScript arrives. Avoid full client-side rendering when SEO or performance matters, dynamic SSR for content that can be cached or regenerated, hydration of static content, and lazy-loading the LCP image.

Ecommerce and product catalogs

Astro or Next.js/SvelteKit hybrid
static/ISR catalog, product, category, and editorial pages
edge CDN with cache tags or surrogate keys
dynamic regional cart, checkout, payment, and inventory service
image CDN with responsive AVIF/WebP transforms
RUM tied to conversion metrics

Operating rules:

• Don't SSR the whole product page because one price fragment changes; split volatile fragments from cacheable HTML.
• Keep payment, fraud, and fulfillment calls out of browse paths.
• Prioritize product hero media and critical CSS.
• Hold tag managers and third-party scripts to an explicit INP budget.

CRUD SaaS, internal tools, and B2B dashboards

HTML-first frontend or a disciplined SPA
regional service in Go, .NET, Java, Rust, Node, Python, Ruby, or PHP by team fit
managed PostgreSQL in the same region
PgBouncer, RDS Proxy, or a managed equivalent for bursty clients
Redis/Valkey for derived reads, sessions, and rate limits
queue or workflow engine for slow work
OpenTelemetry traces, metrics, and logs

In most SaaS products, users spend their time waiting on queries, connections, row locks, frontend waterfalls, and external APIs rather than the application language. One well-run region usually outperforms a premature global distribution.

Operating rules:

• Keep the service and the write database in the same region.
• Shape composite indexes from tenant, equality filters, and sort needs.
• Use cursor pagination.
• Make queue consumers idempotent.
• Cap pools and measure pool wait.
• Cache derived reads, not business truth.
• Set downstream timeouts; use jittered, bounded retries only where safe.

Edge compute fits routing, auth gates, locale selection, cacheable reads, and bot protection. An edge function that immediately calls a distant primary database restores the latency it was meant to remove.

Realtime collaboration, chat, multiplayer, presence

Collaboration: Yjs + Tiptap/CodeMirror/Lexical; Liveblocks, Hocuspocus,
  Y-Sweet, PartyKit, or your own room actors; append-only persistence
  with snapshots and compaction; presence outside durable document state

Chat/notifications: Phoenix Channels or Go/Node/Elixir room services;
  PostgreSQL for durable messages; NATS or managed fanout for regional
  distribution; Redis only where at-most-once ephemeral behavior is fine

Multiplayer/live cursors: an authoritative room process; bounded
  per-client queues; deltas instead of full state; database outside
  the tick loop

In each case, put active rooms near active users, sequence durable events per room, coalesce cursor and presence updates, and drop ephemeral events for slow clients rather than buffering them without a bound.

AI chat, RAG, agents

streaming UI over SSE
thin gateway in Go, Rust, Node, or FastAPI
hosted model API for fast iteration, or vLLM/SGLang/TensorRT-LLM for control
for self-hosting, GPUs colocated with the app and data path
PostgreSQL for app state; pgvector for small/medium RAG
queue or workflow engine for long jobs
OLAP store for product analytics

For interactive paths, stream by default, keep prompts short, place stable prompt prefixes first when prefix caching is available, separate online inference from batch enrichment, and keep vector search near inference. Self-hosting fits steady, high volume; privacy or residency requirements; and teams able to tune batching, KV cache, and scheduling. Hosted models fit early iteration, bursty traffic, or periods when model quality changes faster than infrastructure tuning can repay its cost.

Analytics-heavy SaaS and CPU batch

For analytics-heavy SaaS, use the OLTP/OLAP split and display honest freshness labels. For CPU batch, use the native job binary with object storage. Heavy BI scans should leave the OLTP primary once users depend on its write path, while a single job should remain outside a microservice mesh.

What I would avoid by default

My default is no on the following patterns because teams commonly adopt them before the workload earns their cost:

• Full client-side rendering for public pages with SEO or performance requirements.
• Dynamic SSR for content that could be cached, statically generated, or regenerated.
• Edge compute that immediately calls a faraway primary database.
• Direct database connections from thousands of serverless instances with no pooling or admission control.
• PostgreSQL in the realtime loop for cursors, presence, or game ticks.
• Redis Pub/Sub as a durable delivery mechanism.
• Analytics dashboards on the OLTP primary once they matter to users.
• Kubernetes for one or two services, unless the team already has the platform maturity to run it.
• Multi-region active-active writes, unless the product needs that availability model enough to pay the consistency and operations tax.
• Benchmark-winner frameworks whose winning implementation style isn't how your product will be written.
• Inference benchmarks that don't match your model, prompt lengths, hardware, batching policy, and SLO.

Instrument before you change anything

Few performance mistakes cost more than changing the easiest layer instead of the limiting one. Before touching the stack, make sure the limiting queues are visible:

• Browser: LCP, INP, and CLS by route, device class, and network type; JavaScript bytes split first-party/third-party; long tasks and hydration time; the LCP element's request waterfall.
• CDN and network: cache-hit and origin-shield hit ratios; CDN TTFB versus origin TTFB; HTTP/2 versus HTTP/3 on real traffic; cache-key cardinality and Vary behavior.
• Runtime: p50/p95/p99 by route; queue time versus execution time; cold-start count and latency; retry rate and retry amplification; concurrency per instance.
• Database: pool wait time; slow queries and their plans; lock waits and deadlocks; rows scanned versus rows returned; transaction duration; replication lag.
• Realtime: active sockets; fanout size distribution; sender-to-receiver latency; reconnect rate; missed-event replay latency; backpressure and dropped-message policy.
• AI inference: TTFT, ITL/TPOT, and E2E latency; queue time; goodput under SLO; GPU and KV-cache utilization; batch size distribution; cancellation rate; cost per successful request.

Route-level instrumentation exposes much of it. The appendix expands the list and includes a small OpenTelemetry wrapper.

Where to look first

For an ordinary web application, I investigate in this order:

1. Frontend payload, rendering, hydration, images, fonts, third-party scripts
2. CDN/cache behavior and network distance
3. Data access shape: indexes, pools, locks, queues, N+1s
4. Architecture of the critical path
5. Tail latency and overload behavior
6. Backend framework/runtime
7. Backend language

Specialized systems reorder the list around their dominant wait: the model and data path precede the web framework for AI; event topology precedes the database for realtime; storage layout precedes the API language for analytics; and algorithm and runtime lead for CPU batch. Framework and language still matter, but starting with them usually leaves a larger constraint unexamined.

Work from evidence

Set the web, API, database, realtime, AI, and operations budgets first, then enforce them in CI, dashboards, and release review. Without a budget, optimization becomes a sequence of unrelated local improvements.

Fix the first bottleneck in the user path. For consumer web, the common sequence is to reduce critical bytes, fix the LCP resource, remove render-blocking and hydration bottlenecks, put correct content behind the CDN, reduce origin TTFB, repair queries and pool behavior, and finally move compute closer to the user or data when distance is material. Internal tools and AI products will order the work differently, but neither benefits from optimizing a bottleneck outside the critical path.

Prefer changes that are easy to reverse. Removing unused JavaScript, compressing images, fixing cache headers, adding an index for an observed query, replacing hot-path OFFSET with keyset pagination, capping an overloaded connection pool, and putting a queue in front of a fragile dependency are all low-regret. Framework rewrites, database swaps, CDN migrations, new realtime transports, new vector databases, new inference engines, Kubernetes, and multi-region active-active have a larger blast radius and require stronger evidence.

Match benchmark evidence to the question it can answer. Core Web Vitals describe public web experience; the Web Almanac supplies ecosystem baselines, not measurements of your site; framework benchmarks reveal ceilings, not product architecture. Database and inference benchmarks become relevant only with your schema, data distribution, query mix, model, prompt lengths, hardware, and SLOs. A valid benchmark can still be irrelevant to the workload, and treating it as decisive directs effort toward the wrong layer.

Source quality and caveats

The sources are mostly primary documentation and ecosystem reports: Google's web performance docs; HTTP Archive's Web Almanac; documentation from AWS, Vercel, Cloudflare, PostgreSQL, PgBouncer, and NVIDIA; vLLM's posts; and Anyscale's benchmarking guidance. Their limits affect how the claims should be read:

• The 2025 Web Almanac is an annual ecosystem snapshot, not a substitute for your own RUM, and its CDN timing data is simulated browser measurement.⁸
• Rakuten 24 and Milliseconds Make Millions are vendor-published case studies: directional evidence, not guaranteed ROI.
• Cloud and edge performance always depends on region, runtime, traffic shape, warmness, limits, and configuration.
• SnapStart needs a restore-safety review for uniqueness, credentials, and network state.¹³
• WebTransport's browser support improved materially in March 2026, but deployment still needs a fallback strategy.¹⁹
• vLLM and Triton figures are benchmark- and configuration-dependent; their lasting value is in pointing at the real levers: memory management, batching, scheduling, and SLO-aware serving.
• The connection-pool guidance applies after saturation. Below saturation, more connections can reduce queueing; after it, they tend to inflate tail latency.

Keep the critical path short

Performance engineering controls waiting without abandoning correctness, reliability, or cost. The responsible layer depends on where the wait occurs: critical bytes and main-thread contention in the browser; cache correctness and network distance at the CDN; startup, concurrency, and placement in the runtime; connections, indexes, and query plans in the database; fanout, ordering, and reconnection in realtime systems; goodput under SLO in AI serving.

For most web products, the HTML-first frontend, regional service, and regional PostgreSQL default remains a sensible starting point. Specialize only when the dominant wait requires it: static-first for public pages, regional-database-first for SaaS, room-first for realtime, inference-first for AI, OLAP-first for analytics, and native-job-first for CPU batch.

A benchmark score almost never identifies the fastest stack. Measurements of the user's path do: they show which queue to expose, where to bound it, and whether to shorten the work or move it off the path. The resulting architecture should keep that path short, observable, and resilient.

Appendix: the operational checklist

The checklist expands the instrumentation in the body. If one of these signals is absent from telemetry, add it before changing the stack.

Browser and frontend

• LCP by route, device class, geography, and network type.
• INP by route and interaction type.
• CLS by template and component.
• JavaScript bytes by route, split into first-party and third-party.
• Long tasks and hydration time.
• LCP element type and its request waterfall.
• Font loading and layout-shift behavior.

CDN and network

• CDN cache-hit ratio and origin-shield hit ratio.
• CDN TTFB versus origin TTFB.
• HTTP/2 versus HTTP/3 on real traffic.
• TLS and connection-reuse behavior.
• Cache-key cardinality and Vary behavior.
• Regional latency and error rates.

Runtime and application

• p50/p95/p99 latency by route.
• Queue time versus execution time.
• Cold-start count and cold-start latency.
• Runtime memory and CPU saturation.
• Error rate by dependency.
• Retry rate and retry amplification.
• Concurrency per instance.

A minimal wrapper:

import { SpanStatusCode, trace } from '@opentelemetry/api';

const tracer = trace.getTracer('app-api');

export async function instrumentedRoute<T>(name: string, fn: () => Promise<T>): Promise<T> {
  return tracer.startActiveSpan(name, async (span) => {
    try {
      return await fn();
    } catch (error) {
      span.recordException(error instanceof Error ? error : String(error));
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw error;
    } finally {
      span.end();
    }
  });
}

Add attributes for tenant, route template, cache result, database pool wait, downstream service, cold start, and queue time. Do not put secrets or raw user input in spans.

Database

• Active connections and pool wait time.
• Slow queries and their query plans.
• Lock waits and deadlocks.
• Buffer cache hit ratio.
• Rows scanned versus rows returned.
• Index usage and unused indexes.
• Transaction duration.
• Replication lag.

Realtime

• Active sockets.
• Messages per second.
• Fanout size distribution.
• Sender-to-receiver latency.
• Reconnect rate.
• Missed-event replay latency.
• Per-room memory and CPU.
• Backpressure and dropped-message policy.

AI inference

• TTFT, ITL/TPOT, and E2E latency.
• Queue time.
• Tokens/sec and requests/sec.
• Goodput under SLO.
• GPU utilization and KV-cache memory pressure.
• Batch size distribution.
• Prompt-length and output-length distribution.
• Prefix/prompt/tool cache hit rates.
• Cancellation rate.
• Cost per successful request.

References