Currents - Notice history

We implemented a mitigation and are currently monitoring the result. As well as continuing to investigate the cause.

The system is back to normal.

Initial Technical Analysis

• A sudden increase in ingress traffic caused one of the OLAP DB to throttle new write and read requests

• That caused an accumulation of backlog tasks - an autoscaling kicked off but couldn't cope with the backlog

• Increasing the infrastructure capacity resolved the issue.

Impact

The issue affected test results reported between April 25 ~4:30pm-7:30pm GMT

• runs could be marked as timed out

• test results reporting is missing or delayed for affected runs

• CI jobs displaying warnings and connectivity error

The issue is caused by an increased memory pressure on an infrastructure component. We are performing an adjustment to it capacity and taking care of the pending tasks.

The error has been resolved, and the proper configuration restored. An internal issue has been logged to review the incident and look for future deployment improvements.