From the course: Apache Kafka Essential Training: Building Scalable Applications

Unlock the full course today

Join today to access over 24,200 courses taught by industry experts.

Managing consumer settings

Managing consumer settings

- [Instructor] Let's review some best practices for setting up Kafka consumers in this video. Consumer groups need to scale to match processing loads. Choose sizes based on the expected incoming messages and desired latency. We can also add new consumers to the group when we need additional capacity. Batching is an important consideration for consumers. If you're using it for batch processing, use larger batch sizes to reduce network round trips. On the other hand, if you're using it for real-time stream processing, use smaller batch sizes and smaller polling intervals to minimize latency. Unless the consumer operations are simple and quick, use manual commits to ensure reliability of data processing. Also, provide provisions for reprocessing messages when they are received again by another consumer in case of failures. Use non-blocking processing and multi-threading processing for higher throughput on the consumer side.…

Contents