Currents - Slow reported run completions – Incident details

All systems operational

Slow reported run completions

Resolved
Degraded performance
Started 1 day agoLasted about 1 hour

Affected

Data Pipeline

Degraded performance from 4:42 PM to 5:23 PM, Operational from 5:23 PM to 6:00 PM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    The queue has now recovered, we will monitor the change as the extra processes we launched ramp back down to normal levels.

  • Identified
    Identified

    Our background tasks queues have gotten behind and as a result run status in the dashboard and event notifications are currently delayed.

    This was due to a low auto-scaling cap put in place during our last incident (to limit database connections). This scaling issue has now been resolved but we will expect it to take some time for the system to catch up. And result notifications will likely continue to be delayed for the next hour or two.