Building an adaptive, multi-tenant stream bus with Kafka and Golang

Click for: original source

Back in the 2000s, SOAP/WSDL with ESB (Enterprise Service Bus) was the dominant server-side architecture for many companies. Since the 2010s, microservices and service mesh technologies have grown wildly and thus became the de-facto industry standards. By Xinyu Liu.

At Lyft, hundreds of microservices are backed by numerous databases. New tables are being created constantly to support features and demands of our fast-growing business. Online

The main sections of the article:

  • A use case
  • CDC pipeline remake
  • Architecture
    • Load balancing and auto scaling
    • A greedy algorithm
    • Threading model

During performance testing, the CDC adapter service routed 4 million messages per minute with an average message size of 1.5KB. Such a level of workload was easily handled by 5 Kubernetes vCPU and 10 GB memory in total.

Plenty of other resources are mentioned. And you will get charts explaining architecture etc. Good read!

[Read More]

Tags software-architecture apache streaming apis devops web-development programming