Sherlock: Near real time search indexing for commerce site

Click for: original source

Prasanna Ranganathan from Flipkart published article about building a world-class e-commerce discovery experience through search. The dynamic nature of e-commerce poses unique challenges - stock units, availability, pricing, catalog data, etc. can all change at a very high rate and the system needs to keep up with the latest data lest the customer be disappointed.

Challenge: Processing all the data updates (10K per sec on average) on a continuous basis.

Stack they built their solution on:

  • HBASE
  • Storm + Kafka
  • Redis
  • Solr

Their real-time distributed stream processing framework of choice is Storm. Storm, coupled with Kafka, has a thriving user community not to mention the backing of the Apache foundation. Good read with supporting resources and charts.

[Read More]

Tags nosql software-architecture apache streaming