Follow

Hadoop Plugin

A plugin that collects metrics from Hadoop Nodes version 2.7.0 and later.

You may also refer to this video for a recorded walk through of the install and configuration process.

Prerequisites
  • Supported OS: Linux
  • TrueSight Pulse meter 4.5 or later
  • Hadoop 2.7.0 or later
To install new meter go to 'Settings-> Installation' or see instructions.
To upgrade the meter to the latest version see instructions
 
Metrics Collected
  • HADOOP_NAMENODE_FS_CAPACITY_REMAINING
  • HADOOP_NAMENODE_FS_CORRUPTED_BLOCKS
  • HADOOP_NAMENODE_FS_MISSINGED_BLOCKS
  • HADOOP_NAMENODE_FS_BLOCKSED_TOTAL
  • HADOOP_NAMENODE_FS_FILES_TOTAL
  • HADOOP_NAMENODE_FS_BLOCKED_CAPACITY
  • HADOOP_NAMENODE_FS_UNDER_REPLICATED_BLOCKS
  • HADOOP_NAMENODE_FS_CAPACITYED_USED
  • HADOOP_NAMENODE_FS_TOTAL_LOAD
  • HADOOP_NAMENODE_FSSTATE_VOLUME_FAILURES_TOTAL
  • HADOOP_NAMENODE_FSSTATE_NUMBER_OF_LIVE_DATA_NODES
  • HADOOP_NAMENODE_FSSTATE_NUMBER_OF_DEAD_DATA_NODES
  • HADOOP_NAMENODE_FSSTATE_NUMBER_OF_STALE_DATA_NODES
  • HADOOP_MAP_REDUCED_PROCESS_CPU_TIME
  • HADOOP_MAP_REDUCED_PROCESS_CPU_LOAD
  • HADOOP_MAP_REDUCED_SYSTEM_CPU_LOAD
  • HADOOP_MAP_REDUCED_AVAILABLE_PROCESSORS
  • HADOOP_MAP_REDUCED_TOTAL_SWAP_SPACE_SIZE
  • HADOOP_MAP_REDUCED_FREE_SWAP_SPACE_SIZE
  • HADOOP_MAP_REDUCED_FREE_PHYSICAL_MEMORY_SIZE
  • HADOOP_MAP_REDUCED_TOTAL_PHYSICAL_MEMORY_SIZE
  • HADOOP_YARN_APPLICATION_RUNNING
  • HADOOP_YARN_APPLICATION_FAILED
  • HADOOP_YARN_APPLICATION_KILLED
  • HADOOP_YARN_APPLICATION_PENDING
  • HADOOP_YARN_AVAILABLE_MEMORY
  • HADOOP_YARN_AVAILABLE_VCORES
  • HADOOP_YARN_NUMMBER_OF_UNHEALTHY_NODES
  • HADOOP_YARN_NUMMBER_OF_ACTIVE_NODES
  • HADOOP_YARN_NUMMBER_OF_LOST_NODES
  • HADOOP_YARN_USED_MEMORY
  • HADOOP_YARN_NUMBER_OF_CONTAINERS
  • HADOOP_DATANODE_HEAP_MEMORY_USED
  • HADOOP_DATANODE_HEAP_MEMORY_MAX
  • HADOOP_DATANODE_GC_COUNT
  • HADOOP_DATANODE_GC_TIME_MILLIS
  • HADOOP_DATANODE_GC_NUMBER_OF_WARN_THREADSHOLD_EXCEEDED
  • HADOOP_DATANODE_GC_NUMBER_OF_INFO_THREADSHOLD_EXCEEDED
  • HADOOP_DATANODE_GC_TOTAL_EXTRA_SLEEP_TIME
  • HADOOP_DATANODE_BLOCKS_READ
  • HADOOP_DATANODE_BLOCKS_WRITTEN
  • HADOOP_DATANODE_VOLUME_FAILURES
  • HADOOP_DATANODE_NUMBER_OF_FAILED_STORAGE_VOLUMES
  • HADOOP_DATANODE_REMAINING_CAPACITY
Installing the plugin
 
1) Log into the Pulse.
 
2) Go to Settings > Plugins
 
3) Click on 'Get Plugins'.
 
4) Scroll to 'hadoop' and click 'Install'.
 
5) Pulse provides a list of meters to associate to the plugin.  Please select a meter. Now an "Add a plugin..." drop down box will display.  Click 'hadoop'.
 
6) You will be prompted to "Add a hadoop".

7) You must enter details as per your hadoop configuration.
Field Description
Host The hadoop host name
Type Type of hadoop node (single node or multi node) to monitor
RM Port Resource manager hadoop port number
DataNode Port DataNode hadoop port number
NameNode Port NameNode hadoop port number
PollInterval The frequency (in milliseconds) used to poll for metrics. Default value is 1000 ms, we do not recommend a value smaller than the default value
 
8) Click 'OK'
 
9) You can add multiple Hadoop nodes as required.
    
10) Click 'Save'
 
11) The following dashboards will be created:
  • Hadoop NameNode
  • Hadoop CPU
  • Hadoop YARN
  • Hadoop DataNode
 
Have more questions? Submit a request