Currents - Intermittent parallelization errors – Incident details

Intermittent parallelization errors

Resolved
Degraded performance
Started about 1 year agoLasted about 24 hours

Affected

API - Reporting and Orchestration

Degraded performance from 11:22 PM to 11:15 PM

API

Degraded performance from 11:22 PM to 11:15 PM

API - HTTP REST API

Degraded performance from 11:22 PM to 11:15 PM

API - Dashboard Browsing

Degraded performance from 11:22 PM to 11:15 PM

Updates
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Scaling up Redis cluster completed. We are currently monitoring the system performance.

  • Identified
    Identified

    Reverting the status back to "Identified."

    Scaling up the cluster to accommodate the increased demand. The backed Redis cluster exceeded the allowed network bandwidth limits, causing some requests to drop.

  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Reconfiguring the Redis configuration was completed and the # of errors dropped to 0. Working on follow-up tasks to ensure better scalability. Monitoring the errors.

  • Identified
    Identified

    The root cause has been identified - the backed Redis cluster exceeded the allowed network bandwidth limits, causing some requests to drop. Performing cluster reconfiguration.

  • Investigating
    Investigating

    We are experiencing intermittent parallelization-related issues. Arbitrary can fail to report their results. https://currents.instatus.com