Currents - Increased errors and timeouts for reporting runs and uploading artifacts – Incident details

Increased errors and timeouts for reporting runs and uploading artifacts

Resolved
Partial outage 70 %
Started 14 days agoLasted about 3 hours

Affected

Ingest and Orchestration

Partial outage from 3:33 PM to 4:46 PM, Operational from 4:46 PM to 6:17 PM

Updates
  • Resolved
    Resolved

    Incident resolved.

    We had a partial outage with our object storage provider that resulting in some uploads and requests timing out.

    We also had issues with our retry logic triggering access limit. We will look into improving this area.

  • Monitoring
    Monitoring

    We are no longer experiencing the issue, but are still investigating the cause.

  • Investigating
    Investigating
    We are currently investigating this incident.