Processing time-series data with Redis and Apache Kafka

Click for: original source

Learn how to analyze time-series data through RedisTimeSeries with Apache Kafka in this practical walkthrough. RedisTimeSeries is a Redis module that brings native time-series data structure to Redis. By Abhishek Gupta.

Generally speaking, time-series data is (relatively) simple. Having said that, we need to factor in other characteristics as well: Data velocity - e.g. Think hundreds of metrics from thousands of devices per second and volume (big data): Think data accumulation over months (even years)

Thus, databases such as RedisTimeSeries are just a part of the overall solution. You also need to think about how to collect (ingest), process, and send all your data to RedisTimeSeries. What you really need is a scalable data pipeline that can act as a buffer to decouple producers and consumers. That’s where Apache Kafka comes in! In addition to the core broker, it has a rich ecosystem of components, including Kafka Connect (which is a part of the solution architecture presented in this blog post), client libraries in multiple languages, Kafka Streams, Mirror Maker, etc.

The article also deals with:

  • Scenario: Device monitoring
  • Solution architecture
  • Set up the infrastructure components
  • Setup local services
  • Deploy the device data processor application
  • Start simulated device data generator
  • Delete resources
  • What about long term data retention?

Your time-series data volumes can only move one way—up! It’s critical for your solution to be scalable. Very nice!

[Read More]

Tags app-development apache microservices nosql event-driven messaging