Currents - Issues with slow processing of results – Incident details

All systems operational

Issues with slow processing of results

Resolved
Major outage
Started 3 days agoLasted about 5 hours

Affected

API

Operational from 5:11 PM to 10:22 PM

API - HTTP REST API

Operational from 5:11 PM to 10:22 PM

API - Dashboard Browsing

Operational from 5:11 PM to 10:22 PM

Data Injestion

Partial outage from 5:11 PM to 5:38 PM, Operational from 5:38 PM to 10:22 PM

Data Pipeline

Degraded performance from 5:11 PM to 10:22 PM

Scheduler

Operational from 5:11 PM to 10:22 PM

Updates
  • Resolved
    Resolved

    This incident has been resolved. All queues are caught up.

  • Monitoring
    Monitoring

    We are almost fully recovered, though are still seeing a 10 minute delay on our analytics queues, that power some of the reports in the dashboard. These should finish catching up over the next half hour, with the delay improving steadily.

  • Update
    Update

    We are continuing work on full recovery. Initial results processing and notifications are back to normal, but some data queues that populate the dashboard charts and explorers are still delayed.

  • Identified
    Identified

    We are continuing to work on recovery. The endpoint errors have been resolved. But our queues are backed up, so results are taking a while to show in the dashboard.

  • Investigating
    Investigating

    We are currently investigating this incident. But processing results has gotten significantly behind.

    We are also seeing an increase of errors in the endpoints that receive client requests from the reports.