Code Analyzer for Apache Spark

Code Analyzer for Apache Spark performs runtime profiling of your Apache® Spark applications, which enables you to perform root cause analysis (RCA) of application failures and to identify performance bottlenecks. Code Analyzer exposes performance problems, identifies sections of your application code that need to be optimized, and identifies cluster weather conditions that affect your applications’ performance. With this information, you can improve execution scheduling and eliminate cluster conditions that adversely affect performance.

Code Analyzer for Apache Spark provides the complete picture of everything that contributes to your Spark applications’ performance:

• Code execution plan. Spark provides events during runtime to indicate which block of code is being executed. You can see its stage data—what code ran during which stage—and executor data—resource usage for each Spark executor.

• Time series view. Visual representation of an application’s performance-critical metrics such as resource consumption and garbage collection, as reported by Spark executors, which makes it very easy to detect outliers.

The combination of the code execution plan and time series view exposes performance problems. The correlation of a stage’s executed code and resource usage identifies which sections of code need to be optimized.