Learn how to analyze time-series data through RedisTimeSeries with Apache Kafka in this practical walkthrough. RedisTimeSeries is a Redis module that brings native time-series data structure to Redis. By Abhishek Gupta.
Generally speaking, time-series data is (relatively) simple. Having said that, we need to factor in other characteristics as well: Data velocity - e.g. Think hundreds of metrics from thousands of devices per second and volume (big data): Think data accumulation over months (even years)
Thus, databases such as RedisTimeSeries are just a part of the overall solution. You also need to think about how to collect (ingest), process, and send all your data to RedisTimeSeries. What you really need is a scalable data pipeline that can act as a buffer to decouple producers and consumers. That’s where Apache Kafka comes in! In addition to the core broker, it has a rich ecosystem of components, including Kafka Connect (which is a part of the solution architecture presented in this blog post), client libraries in multiple languages, Kafka Streams, Mirror Maker, etc.
The article also deals with:
- Scenario: Device monitoring
- Solution architecture
- Set up the infrastructure components
- Setup local services
- Deploy the device data processor application
- Start simulated device data generator
- Delete resources
- What about long term data retention?
Your time-series data volumes can only move one way—up! It’s critical for your solution to be scalable. Very nice!
[Read More]