Bo Yang, Nan Zhu, Felix Cheung, Xu Ning from Uber Engineering team published blog post about JVM Profiles. Data is at the heart of strategic decision-making process at Uber. Right sizing the resources allocated to Spark applications and optimizing the operational efficiency of Uber data infrastructure requires fine-grained insights about these systems, namely their resource usage patterns.
The challenge they faced:
- Existing tools could only monitor server-level metrics
- Need to monitor large number of processes (e.g. thousands of executors) running across many servers
- The profiler needs to be launched automatically with each process (unknown when these processes will launch and how long they will take)
To address these challenges, Uber built and open sourced JVM Profiler. There are some existing open source tools, like Etsy’s statsd-jvm-profiler, which could collect metrics at the individual application level, but they do not provide the capability to dynamically inject code into existing Java binary to collect metrics. Inspired by some of these tools, we built our profiler with even more capabilities, such as arbitrary Java method/argument profiling.
What does the JVM Profiler do?
- A Java agent
- Advanced profiling capabilities
- Data analytics reporting:
JVM Profiler supports a variety of use cases, most notably making it possible to instrument arbitrary Java code. And it is easy to change via simple configuration change.
You will find more info in the article, together with explanation schemes on its architecture and guides for various use cases. Brilliant!
[Read More]