Mean and standard deviation are two of the most widely taught statistical tools, which is precisely why they are so often misapplied in load testing. Their familiarity creates a false sense of confidence: the numbers are easy to compute, easy to read on a dashboard — and dangerously easy to misinterpret. This page explains what these metrics actually measure, where they break down in the context of performance testing, and why you should think carefully before relying on them as your primary analysis lens.Documentation Index
Fetch the complete documentation index at: https://mintlify.com/gatling/gatling.io-doc/llms.txt
Use this file to discover all available pages before exploring further.
Definitions
Mean (Arithmetic Average)
The mean describes the central value of a data set. It is defined as the sum of all values divided by the number of values:mean = (x₁ + x₂ + … + xₙ) / nThe arithmetic mean is simple to calculate and communicate, which explains its ubiquity. For a perfectly symmetric, bell-shaped distribution, it accurately represents the “typical” value in the data set.
Variance
Variance describes how much individual values deviate from the mean. It is calculated by squaring the difference between each value and the mean, summing those squares, and dividing by the count:variance = Σ(xᵢ − mean)² / nVariance is expressed in squared units of the original metric — which makes it difficult to interpret directly when the original values are in milliseconds.
Standard Deviation
The standard deviation is the square root of the variance, bringing the unit back to the same scale as the original measurements:stdDev = √varianceStandard deviation is often described as “a measure of variability” — how spread out the values are around the mean. Two distributions can share the same mean but have very different standard deviations, reflecting different degrees of spread.
Why These Metrics Mislead in Load Testing
The Gaussian Assumption
Mean and standard deviation are most informative when the underlying distribution is Gaussian (normal): symmetric, unimodal, and bell-shaped. In that specific case, the mean tells you where the center is, and the standard deviation tells you how tightly values cluster around it. Load test response times are almost never Gaussian. The most common patterns are:Multi-modal distributions
Response times cluster around two or more peaks — for example, cache hits (fast) and cache misses (slow) in the same dataset. The mean falls in a trough between the modes, representing a time that almost no actual request takes.
Long-tailed / skewed distributions
A small number of very slow outlier requests drag the mean upward, making typical performance appear worse than it is, or hiding the severity of tail latency.
The Same Mean, Completely Different Shapes
It is mathematically straightforward to construct datasets that share an identical mean and standard deviation but have radically different shapes — from tight clusters to uniform spreads to bimodal distributions. This is not a theoretical curiosity. Under load, your response time distribution can shift shape entirely (e.g., from unimodal to bimodal as a caching tier saturates) while the mean and standard deviation remain superficially stable. This is sometimes illustrated with datasets that produce identical summary statistics but look completely different when plotted — serving as a stark reminder that summary statistics can never substitute for distribution awareness.Sensitivity to Outliers
The arithmetic mean is highly sensitive to extreme values. A handful of timeout-level responses (say, 30,000 ms) can shift the mean by hundreds of milliseconds even if 99% of requests completed in under 200 ms. Standard deviation amplifies this further, since deviations are squared before averaging. This sensitivity is harmful in both directions:- False pessimism: a few extreme outliers inflate the mean far above the typical user experience.
- False optimism: a bimodal distribution where half of requests are fast and half are slow can produce a mean that looks acceptable while hiding the fact that a large cohort of users has a poor experience.
What to Use Instead
The core problem is that mean and standard deviation describe a distribution by its center and spread, but do not tell you its shape. For load testing, shape matters enormously. Percentiles are a far more robust alternative:- The 50th percentile (median) tells you the response time that half of users experienced or better — completely immune to outliers.
- The 95th percentile (p95) tells you the worst experience of the 95% fastest requests — directly actionable for SLO definitions.
- The 99th percentile (p99) captures near-worst-case behavior, revealing tail latency that the mean conceals.
- If p95 is close to p50, the distribution is tight (most users have similar experiences).
- If p99 is dramatically higher than p95, you have a long tail worth investigating.
- If p50 and p99 are both high, the entire distribution has shifted — the system is under genuine stress.
Summary
| Metric | When it works | When it misleads |
|---|---|---|
| Mean | Symmetric, unimodal distributions | Skewed, multi-modal, or outlier-heavy data |
| Standard deviation | Gaussian distributions | Any non-Gaussian shape |
| Percentiles (p50, p95, p99) | Any distribution shape | Rarely — they are distribution-shape agnostic |