I noticed a way to reduce our alert noise by up to about 80%. This may seem very obvious but it wasn’t to me till after I realized it.

In my environment we are using a ton of different applications by various software vendors. Some of these applications automatically restart their windows services during the day and night. Other times we will get a heartbeat notification in the middle of the night.

With the majority of these notifications we will get an open and then a closed alert almost immediately. In my environment SCOM is setup to send an SMS alert to the on-call admin. This meant that the on-call admin would often be getting woken up in the middle of the night for an issue that automatically resolved itself.

I wanted to reduce the number of alerts that the on-call person (which is sometimes me). I came up with the idea to set alert aging on initial alert by 5 minutes.

We were already using alert aging for escalation but by aging the initial alert drastically reduced the number of alerts that the on-call person gets. The majority of the Open and Closed alerts are no longer bothering the on-call person in middle of the night

 

 

 

 

 

 

Being that SCOM is state based anything that is really down or causing an issue will still page the on-call person.

Categories:

Tags:

2 Responses

  1. I have implemented the alert aging in my environment, and it is working great with one exception. I still receive a “Closed” alert for all of the alerts that would normally alert for an “Open” alert if the alert aging was not in use. So, I am constantly receiving emails for “Closed” and nothing for “Open” and it throwing off the whole deal. Is there any way that I can prevent this from happening, or any steps that I can use to troubleshoot this.

    Thanks

  2. Hi Larry,

    Just set the filter rules to “New(0)” events only. If you need a overview of open or closed alerts use OpsMgr console or the reporting function.

    Peter Dahl

Leave a Reply

Your email address will not be published. Required fields are marked *