Apache Kafka integration with Spring Boot

 

In my previous blog, we discussed the importance of inter-service communication and especially asynchronous communication in Microservices. In this blog, I will be covering the steps to integrate Apache Kafka with spring boot application.

What is Apache Kafka?

Apache Kafka is the widely used tool to implement asynchronous communication in Microservices based architecture.  Apache Kafka is a simple messaging system which works on a producer and consumer model. In this model, the producer will send data to one or more topics. All consumers who are subscribed to that particular topics will receive data. Apache Kafka can be run as a cluster on one or more servers. Apache  Kafka cluster stores multiple records in categories called topics. Each record consists of a key, a value, and a timestamp.

How Apache Kafka Works?

Apache Kafka has four core API’s as mentioned in the below diagram:

Credit: https://kafka.apache.org

  • Producer API is  responsible for sending the stream of data to one or more topics
  • Consumer API  is responsible for receiving data from one or more topics.
  • Stream API  is responsible for consuming input data from one or more topics and producing as an output stream for one or more topics.
  • Connector API is responsible for building and running as reusable producers or consumers that connect Kafka topics to existing applications or data sources.

Apache Kafka will internally use ZOOKEEPER for managing different Kafka instances as a cluster, maintaining leader for each partition to avoid a crash of data and it will track the messages and topics of each Kafka instance.

Basic building blocks in Apache Kafka

Topic: Topic is a unique representation or category to which producer publishes a stream of data. A topic will be subscribed by zero or multiple consumers for receiving data.

Producer: Producer is responsible for sending data to one or more topics and assigning data to partitions within the topic.

Consumer: Consumer is responsible for consuming data from one or more topics when the producer sends the data to topics.

Apache Kafka has a built-in system to resend the data if there is any failure while processing the data, with this inbuilt mechanism it is highly fault-tolerant.

Integration of Apache Kafka with Spring Boot Application

Apache Kafka Setup

  1. Download the Apache Kafka software from the following link and extract the folder from zip: https://kafka.apache.org/downloads
  2. If you want to change the port number of a zookeeper, open zookeeper properties file inside config folder and modify the client port property.
  3. Configure the zookeeper running URL in the server.properties file inside the config folder.
  4. Now start the zookeeper server by using the following command:

.\bin\windows\zookeeper-server-start.bat  .\config\zookeeper.properties (windows)

.\bin\zookeeper-server-start.sh  .\config\zookeeper.properties(linux)

5. Open new terminal and now start the kafka server using the following command:

.\bin\windows\kafka-server-start.bat  ./config/server.properties(windows)

.\bin\\kafka-server-start.sh  ./config/server.properties(linux)

Now Kafka server is running on port number 9092(default).

Setup in spring boot project

6. Add the following dependency in your pom.xml:
            

7. Create sender configuration class and add the properties of serialization and deserialization classes, kafka server URL.
Create Kafka template bean with these properties:

8. Now add the class for receiver configuration with the deserializer class, consumer group, and Kafka server URL.

9. Now add the listener for receiving data from the sender. In this example when a user sends a request for creating a new user, it will return a response immediately and the producer will send data to the consumer. Once the consumer receives the data it will save in the MySQL database.

10. Now add the topic name and Kafka server URL in your application.properties
              user.create=user-create
              kafka.bootstrap-servers=localhost:9092

              consumer.group=user

11. Now run the application and send the user object to the user creation endpoint.

Access to the source code:

You can download the source code from the below URL and find the folder springboot-kafka.

    https://github.com/walkingtree/microservices.git

Summary:

In this article, we have covered detailed steps of installing apache Kafka and steps we need to follow to integrate Apache Kafka with spring boot application.

I hope this article was informative and leaves you with a better understanding of Apache Kafka integration with Spring Boot. At Walking Tree, we are excited about the possibilities that Microservices ecosystems bring in. Stay tuned for more articles on this topic

 

Related Posts

Leave a comment

Translate »