Some practical advice from ThePurpleBuffalo that might save your from succumbing to ‘cried wolf’ syndrome, "High alert volume will inevitably lead to 'Alert Fatigue,' which puts your environment at risk.
All alerts need to be:
- Relevant (I don't care if something in a dev environment goes down.)
- Timely (I don't care that something was down 8 months ago.)
- Actionable (If I can't do anything to fix it, why am I getting an alert for it?)
If you are receiving alerts that fail to meet any of these criteria, you need to spend time tweaking your alerts. Managing alerts is an ongoing process. If after you have done all of that cleanup and tweaking you are still experiencing high alert volumes, then you need to spend time improving your infrastructure to either be more reliable or self healing. See also [this resource]."