Questions –
- How do you create a topic in Kafka using the Confluent CLI?
- Command
- Explain the role of the Schema Registry in Kafka.
- How do you register a new schema in the Schema Registry?
- What is the importance of key-value messages in Kafka?
- Describe a scenario where using a random key for messages is beneficial.
- Provide an example where using a constant key for messages is necessary.
- Write a simple Kafka producer code that sends JSON messages to a topic.
- How do you serialize a custom object before sending it to a Kafka topic?
- Describe how you can handle serialization errors in Kafka producers.
- Write a Kafka consumer code that reads messages from a topic and deserializes them from JSON.
- How do you handle deserialization errors in Kafka consumers?
- Explain the process of deserializing messages into custom objects.
- What is a consumer group in Kafka, and why is it important?
- Describe a scenario where multiple consumer groups are used for a single topic.
- How does Kafka ensure load balancing among consumers in a group?
- How do you send JSON data to a Kafka topic and ensure it is properly serialized?
- Describe the process of consuming JSON data from a Kafka topic and converting it to a usable format.
- Explain how you can work with CSV data in Kafka, including serialization and deserialization.
- Write a Kafka producer code snippet that sends CSV data to a topic.
- Write a Kafka consumer code snippet that reads and processes CSV data from a topic.
- Different way to receive and ack the kafka
- What makes kafka fast?
2.Explain the role of the Schema Registry in Kafka.
The Schema Registry in Kafka plays a crucial role in managing schemas for data that is sent to and from Kafka topics.
Schema Management:
- Centralized Schema Repository: The Schema Registry acts as a centralized repository for schemas used in Kafka messages. It stores and manages schemas independently from the Kafka brokers.
- Schema Evolution: It facilitates schema evolution by allowing compatibility checks between different versions of schemas. This ensures that producers and consumers can evolve their schemas without causing disruptions.
Example:
- Suppose a producer wants to publish messages to a Kafka topic using Avro serialization. Before sending data, it registers the Avro schema with the Schema Registry, which assigns it an ID. When the producer sends a message, it includes the schema ID alongside the serialized data. Consumers retrieve the schema ID from the message, fetch the corresponding schema from the Schema Registry, and deserialize the data accordingly.
22.What makes kafka fast?
Zero-copy writes make Kafka fast, but how exactly? ⚡
Kafka is a message broker, and it accepts messages from the network and writes to the disk, and vice versa. The traditional way of moving data from network to disk involves `read` and `write` system calls, which require data to be moved to and from user space to kernel space.
Kafka leverages `sendfile` system call which copies data from one file descriptor to another within the kernel. Kafka uses this to directly transfer data from the network socket to the file on disk, bypassing unnecessary copies.
If you are interested, just read the man page of `sendfile` system call. In most cases, whenever you see something extracting extreme performance a major chunk of it comes from leveraging the right system call.
ps: I used this zero copy while building Remote Shuffle Service for Apache Spark. It proved pivotal in getting a great performance while moving multi-tb data across machines.
⚡ Admissions for my System Design June 2024 cohort are open, if you are SDE-2 and above and looking to build a rock-solid intuition to design any and every system, check out
UBer Usecase –
Techglot 