Usually, entities like order, order items etc tend to grow substantially year on year as we scale more and serve large customers. Generally, MySQL is the widely used datastore due to its durable & ACID guarantees. While MySQL is a brilliant tech stack, it comes with the overhead of data maintenance. By Manohar K.
Warm Store: This is a store “similar” to your transactional store. We will move all the entities which have completed their lifecycle to this store. Note that we will only keep the last X time frame data here. Also, the latency requirement is a little relaxed here as the rate of access is “relatively” less. Hint: Think MySQL like store, but horizontally scalable.
The article then takes on the journey and explains:
- MySQL scaling
- Phases of MySQL scaling
- Challenges with sharded MySQL
- Solution overview
At a high level, we need to come up with 3 different layers of data stores with the following characteristics:
- Hot Store
- Warm Store
- Cold Store
Now, there are multiple DSQL databases available, we chose Pingcap’s TiDB. Along with the excellent core DSQL features and being MySQL compatible, TiDB also provides a plethora of tooling. This is something which came in very handy in our solution. Following are the tools provided by TiDB.
When it comes to data management and especially when it is a technology like MySQL, the archival policies usually people follow is a one time big bang activity of data purge and movement. Good read!
[Read More]