Models
Check the state of the model for each component type, how it did in validation, when it was trained and when it trains again.
On this page
Cerebro has one model for each component type (servers, web/TCP services, cloud services and APM services). The models train themselves every night and also when new failures or verdicts arrive or when the data changes. A new model (the challenger) only replaces the current one (the champion) if it beats it.
Cerebro → Models shows the state of each one.
The table
Column | What it shows |
|---|---|
Type | The component type and how many have Cerebro turned on. |
Mode | The model's state (see below), its version and its date. If it is still learning, how many failures are still needed. |
Validation (24 h) | How the model did in its test with past data. |
Last training | When it happened, what triggered it (nightly, on activation, manual, new failures and verdicts, data drift, performance decline) and the result: new model in use or the current one was kept, with the reason. |
Next | When the next training is. |
Monitoring | Whether the data changed compared with what the model knows (Drift: yes or no) and how it is doing with real failures (In production). |
Model states
State | What it means |
|---|---|
No components | No component of that type has Cerebro turned on. |
Training | Learning from the history of the enabled components. |
Anomaly mode | There are not enough failures yet to estimate a percentage: it shows how unusual each component is and the disk forecast. |
Calibrated | Model with percentages, calibrated with your company's history. The screen shows the model type in parentheses. |
Training error | The last training failed; it will be retried in the next cycle. |
How to read the validation
PR-AUC: how well the model separates the moments before a failure from the rest. Higher is better; in parentheses, the value of the current rules for comparison.
Recall: out of every 100 failures in the test period, how many it anticipated.
False alarms/wk: false alarms per component per week.
Brier and ECE: how close the percentage shown is to what actually happened. Lower is better.
Below the table, What weighs most overall lists the variables with the most influence on the models, for example Disk, Memory, Latency or Errors in logs.
Retrain manually
If you administer Cerebro, click Retrain on a type's row. It is optional: Cerebro already trains itself. The button isn't available while the model is training.
If the table is empty
If you see No component has Cerebro turned on, turn it on from a component's page. See Turn on Cerebro for a component.
Related
Summary: model improvement over time
Predictive alerts: your verdicts train the model