Currents - Run timeouts end unresponsive results – Incident details

All systems operational

Run timeouts end unresponsive results

Resolved
Degraded performance
Started about 1 month agoLasted about 2 hours

Affected

API - Reporting and Orchestration

Degraded performance from 3:21 PM to 4:03 PM, Operational from 4:03 PM to 5:24 PM

API

Degraded performance from 3:21 PM to 4:03 PM, Operational from 4:03 PM to 5:24 PM

API - HTTP REST API

Degraded performance from 3:21 PM to 4:03 PM, Operational from 4:03 PM to 5:24 PM

API - Dashboard Browsing

Degraded performance from 3:21 PM to 4:03 PM, Operational from 4:03 PM to 5:24 PM

Data Pipeline

Degraded performance from 3:21 PM to 4:03 PM, Operational from 4:03 PM to 5:24 PM

Updates
  • Resolved
    Resolved

    Incident is now resolved. We've identified the poorly performing query that was the cause.

  • Monitoring
    Monitoring

    We've increased our task processing capacity, and have restored performance.

    We continue to investigate and monitor.

  • Investigating
    Investigating

    We are currently investigating this incident.

    We're seeing a spike in processing load, that has resulting in a slowdown in results processing. This is having other cascading effects on how results and run progress is being shown in our system, including causing run timeouts.