Scaling Predictive Analytics to 1M+ Events
World
Engineering
Growth
Scaling Predictive Analytics to 1M+ Events
Building high-throughput infrastructure to handle massive data streams in real-time.
Raw throughput is largely a budget question — add partitions, add consumers, move on. What breaks first is not volume but the tail: the p99 query that quietly climbs from 200ms to four seconds once the system is under sustained pressure.
Coordinated omission in the metrics Our early latency dashboards measured only requests the system managed to accept, which hid the queue entirely. Fixing the measurement changed the priorities. Hot partitions A handful of very large accounts generated a disproportionate share of events. Sub-partitioning by entity within those accounts flattened the skew. Garbage collection pauses Long pauses on the hot path produced most of our worst outliers. Moving the rolling window to off-heap storage removed the largest source of jitter.
We alert on p99 latency and queue depth rather than average throughput. Throughput looks healthy right up until the moment the system stops keeping up, which makes it a poor early warning signal.
"Averages tell you the system is fine. The tail tells you whether your customers agree."
No spam. Unsubscribe anytime.