Spring Boot | How to consume JSON messages using Apache Kafka

Apache Kafka is widely used for real-time data streaming, and Spring Boot makes it easy to integrate Kafka into Java applications. This article demonstrates how to consume JSON messages in a Spring Boot application and process them efficiently using Kafka consumers.

• Configure Kafka consumer properties and JSON deserialization in the application.
• Use @KafkaListener to receive and process incoming JSON messages.
• Map JSON data to Java objects (POJOs) for structured handling.

Step by step Implementation process To consume Jason Massage 

Follow these steps to implement JSON message consumption in Apache Kafka using a Spring Boot application.

Step 1: Create a Spring Boot Project

Create a Spring Boot project using Spring Initializr and

• Add the dependency: Spring for Apache Kafka
• Provides Kafka producer and consumer support.
• Enables Kafka listener annotations.
• Simplifies Kafka configuration in Spring Boot

Step 2: Create the Student Model Class

Since the application will consume student details in JSON format, create a Student model class.

• Represents the JSON structure.
• Stores incoming Kafka message data.

// Creating a student class
public class Student {

    // Data members of the class
    int id;
    String firstName;
    String lastName;

    // Constructor of the student
    // Class
    public Student()
    {
    }

    // Parameterized constructor of
    // the student class
    public Student(int id, String firstName,
                   String lastName)
    {
        this.id = id;
        this.firstName = firstName;
        this.lastName = lastName;
    }

    @Override
    public String toString()
    {
        return "Student{"
            + "id = " + id
            + ", firstName = '" + firstName + "'"
            + ", lastName = '" + lastName + "'"
            + "}";
    }
}

Step 3: Configure Kafka Consumer

Create a configuration class named Config and annotate it with @Configuration and @EnableKafka.

• Establishes communication between Spring Boot and Kafka.
• Defines consumer properties.
• Configures JSON deserialization.

@EnableKafka
@Configuration
public class Config {

    // Function to establish a connection
    // between Spring application
    // and Kafka server
    @Bean
    public ConsumerFactory<String, Student>
    studentConsumer()
    {

        // HashMap to store the configurations
        Map<String, Object> map
            = new HashMap<>();

        // put the host IP in the map
        map.put(ConsumerConfig
                    .BOOTSTRAP_SERVERS_CONFIG,
                "127.0.0.1:9092");

        // put the group ID of consumer in the map
        map.put(ConsumerConfig
                    .GROUP_ID_CONFIG,
                "id");
        map.put(ConsumerConfig
                    .KEY_DESERIALIZER_CLASS_CONFIG,
                StringDeserializer.class);
        map.put(ConsumerConfig
                    .VALUE_DESERIALIZER_CLASS_CONFIG,
                JsonDeserializer.class);

        // return message in JSON formate
        return new DefaultKafkaConsumerFactory<>(
            map, new StringDeserializer(),
            new JsonDeserializer<>(Student.class));
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String,
                                                   Student>
    studentListner()
    {
        ConcurrentKafkaListenerContainerFactory<String,
                                                Student>
            factory
            = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(studentConsumer());
        return factory;
    }
}

Step 4: Create Kafka Consumer Service 

Create a service class that listens for incoming messages from Kafka.

• Receives messages from Kafka topics.
• Converts JSON data into Student objects.

@Service
public class KafkaService {

    // Annotation required to listen
    // the message from Kafka server
    @KafkaListener(topics = "JsonTopic",
                   groupId = "id", containerFactory
                                   = "studentListner")
    public void
    publish(Student student)
    {
        System.out.println("New Entry: "
                           + student);
    }
}

Step 5: Start ZooKeeper and Kafka Server

Before running the Spring Boot application, start ZooKeeper and the Kafka broker.

• ZooKeeper manages Kafka cluster metadata.
• Kafka broker handles message storage and delivery.
• Consumers cannot receive messages unless the Kafka server is running.

Step 6: Create a Kafka Topic

Create a topic named JsonTopic.

For macOS/Linux

bin/kafka-topics.sh --create \

--zookeeper localhost:2181 \

--replication-factor 1 \

--partitions 1 \

--topic JsonTopic

For Windows

.\bin\windows\kafka-topics.bat --create ^

--zookeeper localhost:2181 ^

--replication-factor 1 ^

--partitions 1 ^

--topic JsonTopic

Step 7: Start Kafka Producer Console

Open a new terminal and run the Kafka producer console.

For macOS/Linux

bin/kafka-console-producer.sh \

--broker-list localhost:9092 \

--topic JsonTopic

For Windows

.\bin\windows\kafka-console-producer.bat ^

--broker-list localhost:9092 ^

--topic JsonTopic

Step 8: Run the Spring Boot Application

Start the Spring Boot application.

Now enter JSON data in the Kafka producer console:

{
"id": 101,
"firstName": "John",
"lastName": "Doe"
}

Press Enter to send the message.

Expected Output

New Entry: Student{
id=101,
firstName='John',
lastName='Doe'
}

The JSON message is automatically deserialized into a Student object and displayed on the console.