Follow

Apache Spark Plugin

Monitor, visualize and alert on your Apache Spark MetricsServlet sink in one-second intervals with this TrueSight Pulse Apache Spark API plugin.

You may also refer to this video for a recorded walk through of the install and configuration process.

Prerequisites
  • Supported OS: Linux
  • Apache Spark 1.2.1 or later
  • TrueSight Pulse meter 4.2 or later
To install new meter go to 'Settings-> Installation' or see instructions.
To upgrade the meter to the latest version see instructions
 
Metrics Collected:
  • SPARK_MASTER_WORKERS_COUNT
  • SPARK_MASTER_APPLICATIONS_RUNNING_COUNT
  • SPARK_MASTER_APPLICATIONS_WAITING_COUNT
  • SPARK_MASTER_JVM_MEMORY_USED
  • SPARK_MASTER_JVM_MEMORY_COMMITTED
  • SPARK_MASTER_JVM_HEAP_MEMORY_USED
  • SPARK_MASTER_JVM_HEAP_MEMORY_USAGE
  • SPARK_MASTER_JVM_NONHEAP_MEMORY_COMMITTED
  • SPARK_MASTER_JVM_NONHEAP_MEMORY_USED
  • SPARK_MASTER_JVM_NONHEAP_MEMORY_USAGE
  • SPARK_APP_JOBS_ACTIVE
  • SPARK_APP_JOBS_ALL
  • SPARK_APP_STAGES_FAILED
  • SPARK_APP_STAGES_RUNNING
  • SPARK_APP_STAGES_WAITING
  • SPARK_APP_BLKMGR_DISK_SPACE_USED
  • SPARK_APP_BLKMGR_MEMORY_USED
  • SPARK_APP_BLKMGR_MEMORY_FREE
  • SPARK_APP_JVM_MEMORY_COMMITTED
  • SPARK_APP_JVM_MEMORY_USED
  • SPARK_APP_JVM_HEAP_MEMORY_COMMITTED
  • SPARK_APP_JVM_HEAP_MEMORY_USED
  • SPARK_APP_JVM_HEAP_MEMORY_USAGE
 
How To Install Apache Spark Plugin
 
Plugin Setup

MetricsServlet
MetricsServlet is added by default as a sink in master, worker and client driver. See the /etc/conf/metrics.properties file on your Spark installation for more details.
 
JMV Source
You can also enable the jvm source for instance master, worker, driver and executor to get detailed metrics of the JVM uncommenting the following lines in your /etc/conf/metrics.properties
 
The plugin gathers metrics from the Master and an optional running application. So you need to configure the host and port for the WebUI of the master and application process.
 
WebUI Configuration
By default, the WebUI for the master runs on port 18080 and, for applications the its on 4040 port. These are the default values for this parameters but you can change them based on your configuration. 
 
Installing the plugin
 
1) Log into the BMC TrueSight Pulse console.
 
2) Go to Settings > Plugins
 
3) Click on 'Get Plugins'.
 
4) Scroll to 'spark' and click 'Install'.
 
 
5) Pulse provides a list of meters to associate to the plugin.  Please select a meter.  Now an "Add a plugin..." drop down box will display.  Please click and select 'spark':
 
6) You will be prompted to "Add an Instance".
 
7) You can select default settings or update per your configuration. The "Source" value is the name to display in the legend for the metrics data. It will default to the hostname of the server.  Click "OK".
 
 
8) The following dashboards will be created:
  • Spark Master
  • Spark Application
 
Have more questions? Submit a request