Currents - Slowness in test results ingestion rate, runs timing out – Incident details

All systems operational

Slowness in test results ingestion rate, runs timing out

Resolved
Partial outage 30 %
Started about 2 months agoLasted about 3 hours

Affected

Data Injestion

Partial outage from 2:30 AM to 5:20 AM

Data Pipeline

Partial outage from 2:30 AM to 5:20 AM

Playwright Integration

Degraded performance from 2:30 AM to 5:20 AM

Updates
  • Resolved
    Resolved

    This incident has been resolved. The preliminary root cause is a sudden spike in concurrent executions, which affected the DB capacity. After increasing the capacity and processing the backlog the system is back to normal. A more thorough investigation and mitigation are to follow to improve system stability due to surge in concurrent requests.

  • Investigating
    Investigating
    We are currently investigating this incident.