Uber has one of the largest deployments of Apache Kafka® in the world. It empowers a large number of real-time workflows at Uber, including pub-sub message buses for passing event data from the rider and driver apps, as well as financial transaction events between the backend services. By Prateek Agarwal, Ryan Turner, and KK Sriramadhesikan.
During our performance runs, we saw ~2.5X degradation in the p99 latencies when enabled security feature on the clusters. On further investigation, we discovered that Java 11 has significant SSL/TLS performance improvements over Java 8 because of faster cryptographic operation handling in Java 9+.
The article deep dives into:
- Kafka security concepts
- Encryption
- Authentication
- Authorization
- Security provider
- Uber PKI (uPKI) framework
- Architecture
- Charter IAM framework
- Key/Certificate retrieval flow
- Authorization flow
- Enabling Security Feature on Clusters
… and much more. In this blog we showed the essential components to enable security features on a Kafka cluster. We then showed how Kafka interacts with uPKI and Charter (IAM) systems together to attain security on the Kafka clusters. We discussed how we incrementally enabled security features on Kafka clusters and topics without any degradation. Nice one!
[Read More]