Follow

Using the meter for Windows performance counter collection

Overview

The Truesight meter collects a set of core metrics that are, for the most part, common across a variety of OS flavors. These include metrics such as CPU, memory, disk and network utilization. Only a few, such as Processor Queue Length are MS Windows specific. Prior to the meter's counter collection feature, access to any of the thousands of other metrics were either via BMC provided plugins or customer written software. Though the plugins provide a tenable solution, they are still very much application specific, such as IIS, or MS SQL, and do not provide a general, non-application specific capability.
The counter collection feature attempts to address a number of issues with the prior model and provide an extremely flexible approach in gaining access to any of the other performance counters. In addition to counter access, ease of use and deployment was another important consideration. The remainder of this documentation will provide all the necessary information to setup custom counter collection.

Requirements

The only requirement for using the performance counter collection feature is that the Truesight meter version must be 4.7.0 or greater. The officially supported platforms are Windows server 2008, 2012, or 2016.

Quick Start

For those with simple counter collection requirements, this section will provide the necessary information to quickly setup custom counter metrics. In this section, we will determine the counter(s) to be collected, modify the meter's counter configuration file, and add the new metric to the Pulse dashboard.

Meter Counter Configuration File

The meter parses a static counter configuration file in order to determine which, if any, counters are to be collected, and the metric names to which the counters are matched. The counter file is named - win_pdh.ini - and resides in the same location as the meter executable. The meter will monitor this file for changes, and update its counter specifications automatically. This means that when the file contents are modified, the meter will detect the changes and act appropriately without the need for restarting the service.
For example, assume a user wishes to monitor the counter - "System Calls/sec" This counter is contained in the "System" object so the counter path would be "\System\System Calls/sec".
See Appendix A for information on determining which counters are available and the path.
Once the counter path is determined, the user needs to specify the metric name mapping for the Pulse back end. It is recommended that a consistent prefix be used for Windows counters in order to avoid name collision with other metrics, and facilitate finding the metrics in the UI. In this particular case, we will use PDH_SYSTEM_CALLS_SEC as our metric name.
With this information, the user can modify the win_pdh.ini file and add the following section:

    [PDH_SYSTEM_CALLS_SEC]
    counter=\System\System Calls/sec

Shortly after these changes are saved, the meter will begin collecting and sending the new counter data to the back end.

Dashboard modification

The user simply needs to add the metric to the dashboard at this point. In the UI, the user can select "Edit Dashboard" widget and find the new metric via the PDH prefix. Drag and drop the metric onto the desired slot in the dashboard and save the changes.



Once the dashboard is saved, the new metric will be visible in the graph.


Any additional counters can be added in a similar fashion by editing the win_pdh.ini file and appending the definitions in the manner described above. If the user is running multiple meters on multiple hosts, this file can be pushed to the remote machines and the meter will begin reporting the metrics for those hosts.
For information on additional configuration options in the win_pdh.ini file, refer to Appendix B.

Performance Counter Plugin

Plugins may generate a win_pdh.ini file in their local directory which allows plugins to leverage the meters collection ability. If the meter detects a win_pdh.ini file in the plugin directory, and the plugin is currently enabled, the meter will begin collecting any counters specified in the file. BMC provides a general use performance counter plugin whereby a user can enter the counters to be collected in the UI. The required fields are the same as outlined in the Quick Start section above.
In following section, we will create the same counter metric as in the prior section. In this instance, we will be using the plugin rather than modifying the meters counter file. Note that the meter will continue to collect any metrics specified in its local counter file. It is assumed that the user is familiar with the plugin installation process.

Create the counter metric via the plugin UI

Select a meter which has the plugin installed. In the plugin configuration screen, select "Add a Windows Counter". The metric name will be PDH_SYSTEM_CALLS_SEC and the counter path is \System\System Calls/sec.

Save the plugin configuration changes, and within a few seconds the new metric will be collected. Any number of performance counters may be collected via the plugin. The metric may need to be added to the Dashboard to become visible. See the Quick Start Guide for the appropriate steps.

Advanced Counter Paths

In the previous examples, the counter we pulled was a single instance and provided a single metric value. The following sections will address two additional topics; parent instances, and the wildcard character.

Parent Instances in Counters

Some services, such as MS SQL, can run multiple instances. In order to pull the correct counter data, the instance needs to be specified in the path. This section will demonstrate pulling these types of counters from a MS SQL instance.
In this example, we will look at pulling the "Latches\Average Latch Wait Time (ms)" counter from MS SQL. Our parent instance name is 'MSSQL$SQLEXPRESS', giving the complete counter path of:
'MSSQL$SQLEXPRESS:Latches\Average Latch Wait Time (ms)'
In the case of multi-instance service deployment, the meter supports specifying a list of instance names and will generate the multiple counter paths for a single metric. This removes the need for the user to specify individual metric/counter definitions for each instance. This behavior is controlled by the "instances=" variable in the win_pdh.ini file. By way of example, let's assume we have three MSSQL instances - SQL_PROD, SQL_STAGING and SQL_DEV.
Our metric specification could be set up thusly:

    [PDH_LATCH_WAIT_TIME]
    instances=SQL_PROD, SQL_STAGING, SQL_DEV
    counter=Latches\Average Latch Wait Time (ms)

The meter will create and collect the counters:

    SQL_PROD:Latches\Average Latch Wait Time (ms)
    SQL_STAGING:Latches\Average Latch Wait Time (ms)
    SQL_DEV:Latches\Average Latch Wait Time (ms)

Since we now have three potentially different values for our single metric - PDH_LATCH_WAIT_TIME - the meter will append the instance name to the source in order to differentiate the values. From this example, the user would see the following sources graphed (assuming a hostname of "win-server")

    win-server.SQL_PROD
    win-server.SQL_STAGING
    win-server.SQL_DEV

This mimics the existing behavior of the BMC provided MSSQL plugin.

Wildcards in Counters

Many of the Windows performance counters resolve to an array of services or instances. For example, Windows provides a global counter for threads in the path \System\Threads. It always provides a counter on a per-process basis. The total value as a single metric is normally collected using the (total) specification. Eg: _\Process(_total)\Thread Count




It is possible to retrieve the values for each process in a single counter. The following would return an array of values for each running process - \Process(*)\Thread Count




The meter supports using a wildcard character and processing an array of responses. Since each counter element maps to the same metric name, the meter will differentiate the values by appending the instance name to the source. The following image illustrates specifying the collection of all processes thread counts via the wildcarded counter path.
Given the volume of sources created, this is not a recommended scenario but serves to illustrate the functionality.


Appendix A

Exploring available performance counters

Windows provides a tool which will dump all the currently available counters - TypePerf.exe One quick method to dump all the counters to a file would be using the following command in a PowerShell
TypePerf.exe -q > counters.txt
For complete details on using TypePerf and the various options, view the MS technical article at : {+}https://technet.microsoft.com/en-us/library/bb490960.aspx+

Appendix B

Meter Counter Template

The meter uses a simple .ini type file to specify the counters for collection. Upon startup, if the file it not present, the meter will create a template file. The only required fields for counter definitions are the 'metric name' as the section header, and the counter path. Below is a description of each section element and purpose.

Global Section

version
This is the version of the file format specification. It is currently unused.

Counter Section

[METRIC_NAME] *Required
    This is the section header for each counter to be collected. The section name is the name of the metric that will be pushed to the back end. It is recommend that the metric names for the performance counters be prefixed for easy location in the metrics UI.        eg: PDH_THREAD_COUNT 

counter *Required
    The full path to the desired counter. See Appendix A for determining which counters are available and the path. The specified path should contain the leading '\' character. For example: \System\Threads. A path may also specify a wildcard character. eg: \Process(*)\Thread Count


instance_name
    This field is an optional, comma delimited list of instance names. This is useful in the case of services such as MSSQL that allow multi-instance setups. If present, a counter will be created for each instance_name listed.

    eg: instance_name=a,b counter=\object\counter - the following counters would be generated:
    \a:object\counter
    \b:object\counter

push
    This is a boolean variable set to "true" or "false". If "true", the meter will push the counter metric to the back end. If "false", the meter will retain the value for querying by plugins, but will not send the values to the back end directly. This is useful for plugins that may wish to generate alarms on values, or created aggregated metrics. The default value is "true"

factor
    This is a multiplier value that will be applied to the counter value. The default value is 1.0

interval_sec 

     This is the counter collection time in seconds. Due to the method Windows uses to compute certain counters with history requirements (such as the "per second" counters), the lower limit for this value can be no faster than 2 seconds. 

include_tags
    A comma delimited list of meter tags. If present, the specified counter will only be collected by meters with tags matching one of the specified tags.

exclude_tags

    A comma delimited list of meter tags. If present, the specified counter not be collected by meters with tags matching one of the specified tags.
    {}Note{*}: In the event that a meter matches both include and exclude tags, the exclude tag will take precedence. A warning message will be emitted by the meter in this case.

Appendix C

Debugging Counters

get-counter
    get-counter is a standard Windows tool available in the PowerShell. It can be very useful for determining if the correct paths are being specified and which instances are available when using the wildcard specification.
    For more information on usage, "help get-counter" in a PowerShell will provide the available options.


truesight-meter
   The meter itself has an option build in to allow testing of ini specification files. When invoked with the --verify-pdh option, the meter will parse the ini file, and make one pass of counter collection. This can be useful for validation files before deployment.
   eg: truesight-meter.exe --verify-pdh win_pdh.ini

Meter status page
     The meter status page ({+}http://127.0.0.1:9193/info+) contains a section related to performance counters. A status of 0 indicates successful collection, while any other value denotes an issue. The status code is mapped to a PDH error response. Status codes can be found at {+}https://msdn.microsoft.com/en-us/library/windows/desktop/aa373046(v=vs.85).aspx+

meter.conf debug flag
    The meter.conf file has a debug flag specifically for performance counters. To enable debug, modify the win_per_counters.debug section and set the value to 'true'. The meter will provide much more information on the counter collection loop.

Have more questions? Submit a request