the.com/data streaming
processing the river instead of waiting for the lake to fill.
means a way of handling data as a continuous flow of events, analyzed piece by piece the instant it arrives instead of stored first and queried later.
from emerged from 1990s telecom and sensor networks, but exploded with kafka at linkedin in 2011, when batch processing couldn't keep up with clicks, trades, and logs happening every millisecond.
kafka originnamed after franz kafka, for its complex bureaucratic-sounding design
latency scalereal streaming systems react in milliseconds, not hours
vs batchbatch waits for data to pile up, streaming never waits
for instance
apache kafka — linkedin built it in 2011, now moves trillions of messages daily
nyse trade feeds — stream millions of stock ticks per second to traders
netflix event pipeline — tracks every play, pause, and scroll in real time
uber's real-time eta — streams gps pings to recalculate arrival every few seconds