APM alerts and incidents
Which alerts the APM generates, how to adjust their thresholds, and how they open and close incidents in Core.
On this page
The APM watches each service continuously and, when something goes out of the ordinary, opens an incident in Core. From there, people are notified according to your notification rules, just like any other incident.
Available alerts
Alert | When it triggers | Incident severity | How it closes |
|---|---|---|---|
High p95 latency | The p95 exceeds the threshold, with a minimum number of requests in the window | Warning | Automatically, when latency returns to normal |
High error rate | The percentage of requests with errors exceeds the threshold, with a minimum number of requests | Critical | It doesn't close when it drops below the threshold. It closes after a new version without the error, after the days without recurring (14 by default), with Declare fix, or manually |
Stopped sending data | The service sends no telemetry for the specified minutes | Critical | Automatically, when it starts sending again |
New error | An exception that hadn't been seen before appears | Warning | After a new version without the error, after the days without recurring (14 by default), with Declare fix, or manually |
Error reappeared | An error that was resolved appears again (because its incident was closed or you marked it Resolved) | Warning | Same as a new error |
How error incidents close
Latency and a service that stops sending data close automatically when they return to normal. Error rate and errors stay open even if they stop occurring, and they close in one of these ways:
New version without the error: you deploy a new version of the service and the error doesn't appear again in the next 30 minutes. For the error rate, the service must also be below the threshold.
Days without recurring: the error doesn't appear again within the days set in Close errors that don't recur (days) (14 by default). For the error rate, the service must also be below the threshold at that moment.
Declare fix: you describe what was fixed and the incident closes.
Resolve manually from the incident.
When an error incident closes, in any of these ways, the error becomes Resolved in APM → Errors. If it appears again later, ARGUNIS sends an Error reappeared notification and opens a new incident.
When the error rate drops below the threshold, the incident stays open: it's logged as an event, with no recovery notification. That way you don't lose track of a problem that only calmed down because of low traffic.
View the alerts
Go to APM → Alerts.
Review:
Company thresholds: the values that apply to all services without their own thresholds.
Services in alert or with their own settings: each service with its active alerts and whether it's muted or has own thresholds.
APM incidents: open ones and those from the last 7 days, with their Ticket, Service, Title, Start, and Status. Click the ticket to open the incident.
Change the company thresholds
You need the Administer permission in the APM.
In APM → Alerts, click Change (or go to APM → Configuration and settings and look for Company alert thresholds).
Click Edit.
Adjust:
Maximum p95 (ms).
Maximum errors (%).
Minimum requests in the window: prevents alerts from a single error in a low-traffic service.
Window (min): how many minutes it's measured over.
Notify if a service sends no data for (min).
Close errors that don't recur (days): from 1 to 90, 14 by default. An error incident closes automatically if the error doesn't appear again within those days.
Click Save.
To give a service different thresholds or mute it, use Configure → Thresholds on its page. See Read a service's detail page.
What happens next
Each APM incident also appears in Core → Incidents and reaches the people defined by your notification rules and resolver groups. From the incident you can acknowledge it, snooze it, declare the fix, or resolve it.