1. Postman Community Kolkata - Aug '23

Title: Set up your first Mock Server in Postman

Slide

2. GDSC FIEM - Oct '23

Title: Simplifying Web Development with Fastn: Unleashing the Power of OSS

Slide

3. Just Send IT Bangalore - Oct '23

Title: Introduction to Web3 and Walkthrough of SendIT wallet

Slide

4. GDSC MAKAUT - Nov '23

Title: Open Source Unleashed: Introduction to Git & GitHub

Slide

5. ReactPlay Bangalore - Jan '24

Title: ReactJS in the Era of Micro Frontends

Slide

If you open the official documentation of Kafka, you will find 'Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.'

Let's focus on the words event streaming and distributed.

Event streaming means creating real-time streams and processing real-time streams. Sounds a bit odd right? Let me explain with the below example.

In the picture, you can see that multiple Paytm users are actively doing transactions and sending requests. All the requests are coming to Kafka Server first. This is called creating or generating a real-time stream of data. Now suppose, Paytm limits the transactions to 10 per user and for that, they are using a client application. So, this checking and counting of no. of transactions in real-time are called processing a real-time stream of data.

Now let's understand what Distributed means. So, in the microservices world, distributed means distributing different nodes to multiple computers or regions to balance the load and avoid downtime. As Kafka is a distributed event streaming platform we can distribute the load to different regions as shown in the above picture. In case any server goes down, another server will come into the picture and avoid application downtime.

Where does Kafka come from?

Kafka was originally developed at LinkedIn and was subsequently open-sourced in early 2011. Jay Kreps, Neha Narkhede and Jun Rao helped co-create Kafka. Jay Kreps chose to name the software after the author Franz Kafka because it is "a system optimized for writing", and he liked Kafka's work. Currently, Kafka is developed and maintained under Apache Software Foundation.

Why do we need Kafka?

Again let's understand this with an example. Suppose in a product, there are 4 different applications viz. Frontend, Hadoop, Database Slave & Chat server. And each of these communicates with 5 other services.

This looks complex but what is the problem exactly? If we try to understand, there could be 3 major problems:

1. Data format: Here the front end wants to send a payload to the database server and also to security systems. In each case, the data format will be different. Now Hadoop also sends data to different services. So you can understand maintaining the data format in each communication is very difficult and complexity would arise in the schema.

2. Connection type: While the applications communicate with the different services, they might create different connections like HTTP connection, Socket connection, TCP connection etc. So the connection types will be complex to maintain with multiple services.

3. Number of Connections: In this example, each application is connected to 5 different services. So, just to maintain 9 different applications or services we need to maintain 20 different connections. If the product grows in scale or is an enterprise-level application then this number would be even more.

How can we overcome the challenges?

All the 3 major challenges are handled by Kafka itself. Now if the frontend app wants to send some data to the database server, it will pass through the Kafka and from there it will get the specific schema. Kafka handles the connection too.

How does Kafka work in a nutshell?

Kafka is a distributed system consisting of servers and clients that communicate via a high-performance TCP network protocol. It can be deployed on bare-metal hardware, virtual machines, and containers in on-premise as well as cloud environments.

1. Servers: Kafka is run as a cluster of one or more servers that can span multiple data centers or cloud regions. Some of these servers form the storage layer, called the brokers.

2. Clients: They allow you to write distributed applications and microservices that read, write, and process streams of events in parallel, at scale, and in a fault-tolerant manner even in the case of network problems or machine failures.

Thank you for reading. Wish you a good day! To connect with me on LinkedIn, click here.