Analytics and metrics partial outage

Resolved

Degraded performance

Started about 3 years agoJuly 20, 2023Lasted about 9 hoursJuly 20, 20238:42 PMJuly 21, 20236:05 AMUTC

Affected

API

API - HTTP REST API

API - Dashboard Browsing

Updates

Resolved
July 21, 2023 at 6:05 AMUTC
Resolved
July 21, 2023 at 6:05 AMUTC
This incident has been resolved.
Monitoring
July 20, 2023 at 11:43 PMUTC
Monitoring
July 20, 2023 at 11:43 PMUTC
The cluster rebalancing is complete, enabling all the ES queries, monitoring the performance and resolution.
Identified
July 20, 2023 at 10:55 PMUTC
Identified
July 20, 2023 at 10:55 PMUTC
The root cause was identified as a failed allocator for the ElasticSearch cluster.

A failure in one of the ElasticSearch nodes caused and increased CPU and load for the whole cluster, we are relocating the data to a different node and rebalancing the shards between the new nodes.

We turned off some search queries to temporarily reduce the cluster load and reduce the recovery time.
Investigating
July 20, 2023 at 8:42 PMUTC
Investigating
July 20, 2023 at 8:42 PMUTC
We are dealing with a partial outage with our analytics systems. Insights and performance metrics are temporarily unavailable.

Currents - Analytics and metrics partial outage – Incident details