Have you ever wondered how streaming giants like YouTube, Netflix or Amazon Prime suggest content from the same creators you’re currently watching, recommend similar videos, or even pitch specific products in real-time?
We know this as targeted marketing, driven by your watch history, genres, and preferred content length. To a business, this data is pure gold. But to an engineer, the real challenge is: How do we process this data "on the fly"?
Suppose you are the Chief System Architect of YouTube. You are tasked with building a system that collects and analyzes this massive influx of "gold." How would you process a vast, never-ending stream of data without the system buckling?
In this scenario, you turn to Stream Processing.
What is Stream Processing?
Martin Kleppmann defines it as:
“Stream processing is a computing paradigm focused on continuously processing data as it is generated, rather than storing it first and processing it in batches. It allows systems to react to events in near real-time, enabling low-latency analytics, monitoring, and decision making. Stream processing systems ingest data streams, apply transformations or computations, and emit results while the input is still being produced.”
Essentially, instead of storing data and running a massive batch job at 2:00 AM, you process it the moment it arrives. But how do we implement this?
This is where Kafka Streams enters the picture.
By textbook definition:
“Kafka Streams is a lightweight, Java-based library for building real-time, scalable stream processing applications that read from and write to Apache Kafka topics. It provides high-level abstractions for continuous processing such as filtering, mapping, grouping, windowing, and aggregations, while handling fault tolerance and state management internally.”
Now that we know what to do and which tool to use, let’s build our stream pipeline.
NOTE: This is a simplified mental model to explain the role of stream processing and Kafka Streams, not an exact representation of YouTube’s internal architecture. A giant like YouTube uses multiple stream processors, batch + streaming, ML pipelines, feature stores, etc to provide a seamless user experience.
Designing the Stream Pipeline
In Kafka Streams, we map our logic into a Topology. A topology is a directed acyclic graph (DAG) of processing nodes that represent the transformation steps applied to the data stream.
We start with Watch History, User Activities as our source of truth. In technical terms, this is our Source Processor (reading from a Kafka Topic).
Using the Kafka Streams DSL (Domain Specific Language), we can define three distinct operations:
1: Data Masking and Sanitization
Before deriving any higher-level signals, it is often necessary to sanitize incoming events.
This node:
- consumes raw user interaction events
- removes or masks unnecessary or sensitive fields
- standardizes the event structure
This step ensures that downstream processors operate only on relevant and safe data, reducing coupling and improving maintainability.
The output of this node is a sanitized event stream, which becomes the input for subsequent processing steps.
2: Similar Content Recommendation
To power this, we need the User ID, Channel Name, and Genre. For example, if you watch a WWE video, the genre is Professional Wrestling. The goal is to immediately suggest related promotions like AEW or TNA.
In this node, we take the raw KStream, apply a map or transform operation to extract the relevant metadata, and pass it to a Sink Processor. This sink then emits the event into a new Kafka topic: similar-content.
3: Preferred Video Length
Here, we focus on user behaviour. Does the user prefer 30-second Shorts or 20-minute video essays?
We transform the incoming KStream into a specialized object containing the User ID and duration metrics. This transformed data is then streamed into a dedicated topic: preferred-content-length.
4: Product Discovery
If a user searches for specific items within the platform, we can extract these signals immediately. By filtering search events within the topology, we can transform them into product-intent objects and emit them into a product-recommendations topic.
Now that the data is emitted as well-defined events, downstream applications can analyze it independently and serve users far more effectively — and you get to keep your high-paying job, all thanks to stream processing and Kafka Streams 😉
Kafka Streams as a Transformer, Not the Brain'
Many descriptions label Kafka Streams as the "brain" or "heart" of an application (which, in some cases, may be true). However, in this architecture, Kafka Streams acts as a high-performance Transformer and Supplier.
It cleans, shapes, and routes data so that downstream microservices can act on it. This is the hallmark of a well-designed Event-Driven Architecture.
Congratulations! You’ve just scratched the surface of real-time data orchestration.
But a question remains: Why not just use a traditional database? Beyond the sheer volume of "heavy writes," what are the structural drawbacks of using a database for this?
Stay tuned for Part 2.
Top comments (0)