Anomalies

An anomaly is a metric that is far outside its normal range. For example, query spend in one project is 4× its usual Tuesday level. SlashPeak checks for anomalies on every scan, on every plan.

Anomalies show on Dashboard → Anomalies, and as a bell on the charts.

What is checkedCopy Anchor Link

Every scan checks each project and region for these kinds of series:

• Query spend, by cost category
• Bytes billed (on-demand jobs) and slot-hours (capacity jobs)
• Failed jobs
• Storage
• Scheduled jobs: missed runs, late runs and unusual run sizes
• Table freshness: a table that usually updates but is now late ("stale table")

Block size: depends on your planCopy Anchor Link

Each scan looks at two time grains:

• Each day, in your account timezone.
• Each block of hours since the last scan. The block size is your scan schedule.

Smaller blocks find a short spike sooner. A daily check still catches a slow rise that lasts all day.

How it learns normalCopy Anchor Link

1. Each series is compared with its own last 42 days.
2. First it finds the pattern: steady, a daily cycle, a weekly cycle, bursty, or a scheduled job.
3. Then it uses the check that fits that pattern. A cycle is checked by two different methods, and both must agree.
4. A scheduled job is checked for missed runs, late runs and unusual run sizes.

A new series needs about 7 days of data before it is checked. A weekly cycle needs about 21 days to be learned.

What counts as an alertCopy Anchor Link

• Rises only. A spike up or a level shift up is an alert. A drop is not, because a drop costs you nothing.
• Scheduled jobs: a missed run or a late run is always an alert.
• The checks must agree, and the change must be big enough to matter in dollars or volume.
• Late data is never read as a drop or a missed run.

One row per anomalyCopy Anchor Link

An anomaly lasts from its first unusual block until the series is back to normal. Each row shows:

• The span (start and end, or "ongoing").
• The reason: spike, level shift up, missed run, late run, or stale table.
• The actual value and the expected value, and the normal range.
• The cost of the change in dollars.

When a short block and a full day flag the same rise, you see one anomaly, not two.

Bells on chartsCopy Anchor Link

A red bell on a chart marks an anomaly that sent an alert. A bell with a dashed ring sent no message: it was on the first scan, it repeats one already sent, or another alert already covered it.

Why? — the cause pageCopy Anchor Link

Each row and each bell has a Why? link. It opens a page in a new tab with two parts:

• Why — the history chart, the normal band, and how far the metric moved.
• Cause — what did it, in one sentence. For example:
  • a new query,
  • a user who ran a query many more times,
  • a query that reads more data per run,
  • a new error,
  • a table that grew.

Below the sentence: the queries, users, tables, errors or dbt models that changed most, with each one's share of the change, its SQL and its jobs, with a link to BigQuery.

While an anomaly is still open, its cause says "so far" and updates with each scan.

AlertsCopy Anchor Link

Every anomaly is an alert. It goes to the channels you turn on in Settings → Notifications:

• Email (each person picks for themselves)
• Slack
• Webhook (anomaly.detected)

Alert messages carry the same cause sentence as the Why page. Turn on Alert when back to normal to also get a message (and anomaly.recovered) when the metric is back inside its range.

A connection's first scan never alerts. It also checks the last 7 days, so you see recent anomalies at once, but those rows are never sent.

See Alerts and notifications.

Mark as "Not an anomaly"Copy Anchor Link

Owners can mark an anomaly as normal, for example a planned backfill.

• For 30 days, the same kind of change on the same series, up to 25% larger, is hidden and sends no message.
• Undo it from the row, or from Settings → Notifications → Muted anomalies.

CoverageCopy Anchor Link

A line above the feed shows:

• how many series are checked,
• how many are still learning,
• which data sources are late.

Health badgeCopy Anchor Link

An open anomaly alert turns the dashboard health badge Critical. An alert in the range that is now back to normal makes it Needs attention. See Dashboard.

Not split by cost centerCopy Anchor Link

Anomalies are checked per project and region. The cost-center filter on the dashboard does not split them.