This article is based on the talk Scale in / Scale out with Kafka-Streams and Kubernetes from Xebicon’18. Loïc Divad is the author of this article and he aims to present a few advantages that come with specific practices related to Kafka-Streams like containerization and orchestration, and especially autoscaling.
Kafka consumers subscribe to topics and then regularly poll messages from it. When it comes to Kafka-Streams, polling intervals are generally short enough to get a real time processing effect.
The article then explains:
- What is consumer lag, a reason to scale
- Message consumption
- The consumer protocol
- Kubernetes and the custom metrics support
- Expose the JMX metrics in a Prometheus format
- Building the Docker image of the streaming app
- Metrics aggregation to Stackdriver
- MetricServer Setup
- Horizontal Pod Autoscaler configuration
… and much more. With existing tools like Stackdriver and Kubernetes, it was simple to enrich the capabilities of streaming application. There is also a list of useful readings on the same subject. Excellent!
[Read More]