Actions allows you add multiple actions to an alarm. Now when an alarm triggers you can do anything:
- send an email
- post to your chatroom (Slack, CampFire, Flowdock, HipChat)
- trigger an issue with PagerDuty
- call a URL to add a new server to your infrastructure
- call a URL to restart an application,
- call a URL to turn on your coffee machine
We have handpicked the initial list of actions based on your feedback, if there are more you would like to see, let us know.
Being notified when something happens is fantastic. Opening and Resolving incidents in PagerDuty is great, but what if we could resolve the issue automatically? When an alarm triggers, let's action it.
Here is a video that covers alarms and actions as well:
In this example, we will create an alarm on the memory usage on a server, and if it goes beyond a certain threshold, we'll restart the service.
Open the Settings dialog and select the Actions tab. We do not have any actions installed yet, so lets create one. Click on the New Action button.
We see a list of the available actions. You can click on the action to read more about it, or click Add to configure an action of that type for your account. We want to add the URL action in this case, as we want BMC TrueSight Pulse to call us when something happens, so click on that one.
The URL action takes in a couple of parameters, so lets go through them.
- Name: what should we call this? This will show up in the alarms dialog, so we want something descriptive.
- Method: what HTTP Method should TrueSight Pulse use when calling our server? This example uses a POST, so TrueSight Pulse will send some helpful data in the body of the POST so we can adjust what action our server should take
- Protocol: HTTP or HTTPS? If you have a certificate, use HTTPS.
- URL: how do we contact your server? Make sure to include the port and path in the URL. HTTP will default to 80 and HTTPS will default to 443. Ports 8080 and 8443 are also supported but you have to specify them in your url.
- Username: does the URL require authentication? In this example our URL will check for the username and password.
- Password: same as above
- POST data: Are there any key values pairs that you want to add to the request? Enter these here key=value, one per line.
Enter in all of the values and click Save.
Now that we have an Action, we need to attach it to something. Lets create an Alarm that look at the Memory Usage on a server. When the memory usage goes above a certain threshold, restart the application.
Click on the Alarms tab on the left and then Add alarm.
We want to alarm the Memory Usage, so we select Memory Usage from the drop down, enter in our values and pick the new Restart My Service action from the drop down and hit Save.
With my new Alarm and Action in place, we need something on the server to listen to that POST. Below is a simple webserver in node.js that listens for the POST and runs a command. In this case, it's restarting a service called 'graphdat' using supervisord:
With the server up and running:
$ node restart-graphdat-on-the-local-machine.js
We can see the service is up and running.
Now how do we test that this works? We can either spike the memory on the server, or we can edit the alarm to a very low value so we know that it will trigger. That sounds much better. Lets set the alarm to a 1 second period and to a threshold of 1%.
With the alarm saved, we can see that it triggers a couple of seconds later.
If we have a look in our console, we can now see that the service was restarted.
Put the Alarm back to it original settings.
The alarm is resolved:
And the service ignores the resolution because we do not want to restart the service when everything is up and running.
With Plugins, Alarms and Actions you can be notified when things are happening on your server infrastructure and in your applications, and now, you can do something about it.