Duration: 5 hours and 11 minutes

Affected Components: Data Pipeline, API - HTTP REST API, Playwright Integration, Scheduler, Data Injestion, Cypress Integration, API - Dashboard Browsing, 3rd Party Integrations

Feb 11, 19:41:12 GMT+0
Identified - We are continuing work on full recovery. Initial results processing and notifications are back to normal, but some data queues that populate the dashboard charts and explorers are still delayed..

Feb 11, 22:22:54 GMT+0
Resolved - This incident has been resolved. All queues are caught up..

Feb 11, 22:15:58 GMT+0
Monitoring - We are almost fully recovered, though are still seeing a 10 minute delay on our analytics queues, that power some of the reports in the dashboard. These should finish catching up over the next half hour, with the delay improving steadily. .

Feb 11, 17:11:35 GMT+0
Investigating - We are currently investigating this incident. But processing results has gotten significantly behind. We are also seeing an increase of errors in the endpoints that receive client requests from the reports..

Feb 11, 17:38:30 GMT+0
Identified - We are continuing to work on recovery. The endpoint errors have been resolved. But our queues are backed up, so results are taking a while to show in the dashboard. .

Duration: 1 hour and 17 minutes

Affected Components: Data Pipeline

Dec 15, 17:23:24 GMT+0
Monitoring - The queue has now recovered, we will monitor the change as the extra processes we launched ramp back down to normal levels..

Dec 15, 18:00:16 GMT+0
Resolved - This incident has been resolved..

Dec 15, 16:42:53 GMT+0
Identified - Our background tasks queues have gotten behind and as a result run status in the dashboard and event notifications are currently delayed. This was due to a low auto-scaling cap put in place during our last incident (to limit database connections). This scaling issue has now been resolved but we will expect it to take some time for the system to catch up. And result notifications will likely continue to be delayed for the next hour or two..

Duration: 2 hours and 50 minutes

Affected Components: Data Pipeline, Playwright Integration, Data Injestion

Dec 13, 02:30:00 GMT+0
Investigating - We are currently investigating this incident..

Dec 13, 05:20:09 GMT+0
Resolved - This incident has been resolved. The preliminary root cause is a sudden spike in concurrent executions, which affected the DB capacity. After increasing the capacity and processing the backlog the system is back to normal. A more thorough investigation and mitigation are to follow to improve system stability due to surge in concurrent requests..

Duration: 12 hours and 22 minutes

Affected Components: Data Pipeline, API - HTTP REST API, , Data Injestion, API - Dashboard Browsing, API →

Oct 20, 10:00:00 GMT+0
Investigating - Today on 01:00 PT, AWS experience an outage (us-east-1). Most of AWS services have been restored, yet we are still facing issues with recovering all of our services. .

Oct 20, 10:00:00 GMT+0
Investigating - Our auto-scaling is currently impacted due to an AWS EC2 outage. We are waiting for new instances to be provisioned in order to resume and complete all pending tasks..

Oct 20, 12:44:55 GMT+0
Identified - Due to AWS us-east-1 outage we are still experiencing issues. There is a big backlog of data update tasks we are unable to process because of lack of EC2 resources. We are looking for alternative runtimes to restore the functionality. Relevant excerpt from AWS: \[04:48 AM PDT\] We continue to work to fully restore new EC2 launches in US-EAST-1\. We recommend EC2 Instance launches that are not targeted to a specific Availability Zone (AZ) so that EC2 has flexibility in selecting the appropriate AZ. The impairment in new EC2 launches also affects services such as RDS, ECS, and Glue. We also recommend that Auto Scaling Groups are configured to use multiple AZs so that Auto Scaling can manage EC2 instance launches automatically..

Oct 20, 22:21:31 GMT+0
Resolved - Our systems are back to normal. After AWS restored their services we were able to start processing the incoming data. There's still a backlog of non-processes event accumulated during the outage. * Our focus is on processing the newly created runs without any delay. * Due to nature of our service there's less value in real-time processing of the delayed events because the associated runs are already expired and timed out. * The backlog of post-process still to be process for analytics and performance analysis. This outage revealed a few performance and resilience related issues with our system. We will follow up with a more detailed analysis..

Duration: 2 hours and 1 minute

Affected Components: Data Injestion

Aug 21, 18:23:28 GMT+0
Identified - Due to a network issue, uploads may be slow. Cypress uploads of logs in all regions are impacted (because our ingest server for these is in the impacted area), and playwright artifact uploads from the us east coast..

Aug 21, 19:06:36 GMT+0
Monitoring - The issue looks to have been resolved. We are continuing to monitor and confirm..

Aug 21, 20:24:18 GMT+0
Resolved - This incident has been resolved..

Duration: 2 hours and 43 minutes

Affected Components: Data Injestion

Aug 18, 15:33:32 GMT+0
Investigating - We are currently investigating this incident..

Aug 18, 16:46:15 GMT+0
Monitoring - We are no longer experiencing the issue, but are still investigating the cause. .

Aug 18, 18:17:00 GMT+0
Resolved - Incident resolved. We had a partial outage with our object storage provider that resulting in some uploads and requests timing out. We also had issues with our retry logic triggering access limit. We will look into improving this area..

Duration: 13 minutes

Affected Components: Playwright Integration, Data Injestion

May 15, 19:17:38 GMT+0
Resolved - This incident has been resolved..

May 15, 19:05:00 GMT+0
Identified - A failed deployment caused a 10-minutes outage. During that outage test reporters were not able to upload artifacts. After receiving the alerts we rolled back to failed deployment and the system went back to normal. .

Duration: 1 hour and 4 minutes

Affected Components: API - Dashboard Browsing, API - HTTP REST API

May 9, 00:04:35 GMT+0
Investigating - We are currently investigating this incident, where our elasticsearch cluster has become unhealthy, and is impacting the processing of results..

May 9, 00:59:23 GMT+0
Monitoring - We implemented a mitigation and are currently monitoring the result. As well as continuing to investigate the cause..

May 9, 01:08:08 GMT+0
Resolved - This incident has been resolved..

Duration: 2 hours and 45 minutes

Affected Components: Data Pipeline, Data Injestion, API - HTTP REST API, API - Dashboard Browsing

Apr 28, 17:11:34 GMT+0
Investigating - We are currently investigating this incident..

Apr 28, 18:46:38 GMT+0
Identified - The issue is caused by an increased memory pressure on an infrastructure component. We are performing an adjustment to it capacity and taking care of the pending tasks..

Apr 28, 19:56:08 GMT+0
Resolved - The system is back to normal. ### Initial Technical Analysis * A sudden increase in ingress traffic caused one of the OLAP DB to throttle new write and read requests * That caused an accumulation of backlog tasks - an autoscaling kicked off but couldn't cope with the backlog * Increasing the infrastructure capacity resolved the issue. ### Impact The issue affected test results reported between April 25 \~4:30pm-7:30pm GMT * runs could be marked as timed out * test results reporting is missing or delayed for affected runs * CI jobs displaying warnings and connectivity error.

Duration: 3 hours

Affected Components: API - HTTP REST API

Mar 21, 17:30:00 GMT+0
Resolved - The error has been resolved, and the proper configuration restored. An internal issue has been logged to review the incident and look for future deployment improvements..

Mar 21, 14:30:00 GMT+0
Monitoring - Some incompatible settings from a failed deployment were left in place after a rollback, and our rest api endpoint was unreachable for the past 3hrs. Spot instance reset requests, and calls directly to our api endpoints were impacted. We implemented a fix and are currently monitoring the result..

Duration: 7 hours and 38 minutes

Affected Components: Data Pipeline, Data Injestion

Jan 11, 20:24:48 GMT+0
Investigating - We are currently investigating this incident. We are seeing an elevated number of errors, as well as load on our database. Run processing has been significantly delayed. We are investigating..

Jan 11, 23:15:30 GMT+0
Identified - We have identified a bottleneck in DB writes that caused slow processing and accumulation of tasks in processing queues. Scaling up the DB cluster and terminating long-running operations restored the system stability. Investigating the root case for the increase in DB resources consumption..

Jan 12, 01:17:32 GMT+0
Monitoring - The system is now once again working as normal. During the incident you may have failed to create runs, and run data from the incident window may be missing in the dashboard, leading to timeout out runs and missing results. New runs should now be working as expected as of Sat 11:30pm UTC We are still closely monitoring for further errors. And we will be doing a deeper investigation into the root cause on Monday. Several steps were taken to increase our database capacity during the incident, and canceling long running task. As well as a rollback of of some of our most recent deployed services. It's not yet clear whether the rollback or the cancelling of long running tasks resolved the incident. We will be looking into the root cause deeper during regular business hours. .

Jan 12, 04:02:50 GMT+0
Resolved - This incident has been resolved. The root cause was identified as a bug in telemetry configuration for background tasks queue. Adding telemetry data broke tasks deduplication, causing DB overload due to redundant write requests..

Duration: 3 hours and 1 minute

Affected Components: API - HTTP REST API, API - Dashboard Browsing, Data Injestion

Dec 19, 17:01:30 GMT+0
Investigating - We are currently investigating this incident..

Dec 19, 18:01:44 GMT+0
Investigating - We have reverted the most recent release, and the error rates have gone down. We are still experiencing some slowdowns due to the task backlog that grew during the incident, but the system is mostly recovered. Runs that were created during the outage will be marked as timed out, and you may see errors related to those runs in your local clients if they are still in progress. New runs should not be affected, and should now work as expected. We are still actively investigating the root cause..

Dec 19, 18:23:31 GMT+0
Identified - We are still seeing issues as a result of the backlog, we are scaling up to address the backlog of tasks. We are continuing to investigate the cause in the meantime..

Dec 19, 19:05:54 GMT+0
Monitoring - Our backlog has recovered, and the service is back to normal. The root cause appears to have been changes in our now reverted deployment. We will continue to investigate the cause and provide more details after our investigation..

Dec 19, 20:02:32 GMT+0
Resolved - The system has recovered. We will continue to investigate the root cause and will update the description of the incident with the details when we have them..

Duration: 4 hours and 48 minutes

Affected Components: Data Pipeline

Oct 29, 13:00:00 GMT+0
Monitoring - The service is now recovered, we are monitoring for additional errors. We experienced a scaling issue with our data processing service. This is resulting in run results and run start handling being delayed. Some delays may have resulted in failure to record runs. .

Oct 29, 17:47:55 GMT+0
Resolved - ## This incident has been resolved All systems are operational. ## Impact Assessment * customers run results were significantly delayed for \~5 hours * the delays caused other issues with services that expected the results like webhook and integrations. ### ## Root cause analysis At \~8:20pm PST (3:20am UTC) we had an internal cleanup task run that removed old docker images used to deploy our data processing service. An oversight in how this was configured vs our processes resulted in the currently deployed service's docker image being removed. Between \~4:00am PST (11:00am UTC) and \~9:00am PST (4pm UTC) the system was not able to successfully take any scaling action while it tried to deploy the deleted image. There were still a small number of instances still running and processing tasks, but not enough to deal with the load. Once we re-deployed a newer image, we were able to quickly recover to normal operating status. .

Duration: 4 hours and 10 minutes

Affected Components: Data Pipeline, Playwright Integration, Data Injestion

Aug 27, 20:33:08 GMT+0
Investigating - We are observing an increased errors rate for Playwright orchestration requests. We are currently investigating this incident. .

Aug 27, 22:19:31 GMT+0
Identified - We have identified the root cause and released a hot patch that prevent erroneous HTTP responses. The orchestration isn't distributing the tests in the optimal order at the moment, working on restoring the service to full capacity..

Aug 28, 00:42:54 GMT+0
Resolved - ## This incident has been resolved All systems are operational. ### ## Impact Assessment * a few customers weren't able to create orchestrated runs for \~1.5h * the orchestration didn't produce optimal results for \~4 hours ### ## Root cause analysis At \~12:15pm PST (7:15pm UTC) our database cluster performed a failover from primary node to one of its secondary nodes as part of applying a maintenance patch. ```log {"t":{"$date":"2024-08-27T19:12:21.629+00:00"},"s":"I", "c":"ACCESS", "id":20250, "ctx":"conn126158","msg":"Authentication succeeded","attr":{"mechanism":"SCRAM-SHA-1","speculative":true,"principalName":"mms-automation","authenticationDatabase":"admin","remote":"192.168.253.167:50284","extraInfo":{}}} ``` ```log {"t":{"$date":"2024-08-27T19:15:01.090+00:00"},"s":"I", "c":"COMMAND", "id":21579, "ctx":"conn126158","msg":"Attempting to step down in response to replSetStepDown command"} ``` The patched server caused one of the queries to return a malformed response, which resulted in a cascading series of failures. As a result some customer were not able to create orchestrated runs. After releasing a hot patch that restored system stability, we released a permanent fix that restored the full service capacity. ## Update 2024-08-28 12:25AM PST The root cause was further refined to an issue in MongoDB update. _The cluster underwent MongoDB Version Upgrade from v6.0.16 to v6.0.17\. The plan for the upgrade completed at_ `08/27/2024, at 07:13:33 PM UTC` _which corresponds with the onset of the query issue._ _​_ _MongoDB internal team identified aggregation issues introduced in version v6.0.17 and has scheduled a downgrade the cluster to v6.0.16._ _MongoDB is progressively reverting the 6.0.17 upgrades back to 6.0.16, but I do not have a timeline for when your clusters will be done._ _​_ _We sincerely apologize for the inconvenience. Please let me know if you have any further questions._.

Duration: 2 hours and 4 minutes

Affected Components: API - HTTP REST API, API - Dashboard Browsing, Data Pipeline, Data Injestion

Jul 29, 15:21:22 GMT+0
Investigating - We are currently investigating this incident. We're seeing a spike in processing load, that has resulting in a slowdown in results processing. This is having other cascading effects on how results and run progress is being shown in our system, including causing run timeouts..

Jul 29, 17:24:53 GMT+0
Resolved - Incident is now resolved. We've identified the poorly performing query that was the cause..

Jul 29, 16:03:28 GMT+0
Monitoring - We've increased our task processing capacity, and have restored performance. We continue to investigate and monitor..

Duration: 1 hour and 52 minutes

Affected Components: API - Dashboard Browsing

May 29, 15:31:13 GMT+0
Identified - Our in-app support chat is currently experiencing issues. If you need support, please reach out using our support email. Our chat provider's incident status can be followed here: <https://www.intercomstatus.com/incidents/lxsqvttdjj76>.

May 29, 16:35:44 GMT+0
Monitoring - Chat support should now be working, but may be a little slow. If you have trouble with it loading, please try refreshing the page at least once and trying again. Our email support is always available if you are running into troubles..

May 29, 17:23:04 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 45 minutes

Affected Components: Data Pipeline

Apr 11, 19:15:19 GMT+0
Resolved - The system has recovered.

Apr 11, 17:30:00 GMT+0
Monitoring - We have increased our processing capacity, and are monitoring as the system catches up with the load.

Duration: 1 hour and 3 minutes

Affected Components: Data Pipeline

Feb 9, 22:35:24 GMT+0
Investigating - As part of planned maintenance with no expected downtime, we've detected that a small percentage of test results missing. We are currently investigating this incident..

Feb 9, 23:38:23 GMT+0
Resolved - ✅ The root cause has been identified and classified as a false negative. Spec files with no tests were mistakenly identified as missing results. No real impact on customer data..

Duration: 23 hours and 53 minutes

Affected Components: API - HTTP REST API, API - Dashboard Browsing, Data Injestion

Aug 15, 17:39:35 GMT+0
Monitoring - Scaling up Redis cluster completed. We are currently monitoring the system performance..

Aug 14, 23:22:42 GMT+0
Investigating - We are experiencing intermittent parallelization-related issues. Arbitrary can fail to report their results. https://currents.instatus.com .

Aug 14, 23:36:33 GMT+0
Identified - The root cause has been identified - the backed Redis cluster exceeded the allowed network bandwidth limits, causing some requests to drop. Performing cluster reconfiguration..

Aug 15, 00:14:13 GMT+0
Monitoring - Reconfiguring the Redis configuration was completed and the # of errors dropped to 0. Working on follow-up tasks to ensure better scalability. Monitoring the errors. .

Aug 15, 06:36:09 GMT+0
Resolved - This incident has been resolved..

Aug 15, 16:38:51 GMT+0
Identified - Reverting the status back to "Identified." Scaling up the cluster to accommodate the increased demand. The backed Redis cluster exceeded the allowed network bandwidth limits, causing some requests to drop. .

Aug 15, 23:15:27 GMT+0
Resolved - This incident has been resolved..

Duration: 9 hours and 23 minutes

Affected Components: API - Dashboard Browsing, API - HTTP REST API, , API →

Jul 20, 23:43:33 GMT+0
Monitoring - The cluster rebalancing is complete, enabling all the ES queries, monitoring the performance and resolution..

Jul 21, 06:05:37 GMT+0
Resolved - This incident has been resolved..

Jul 20, 20:42:20 GMT+0
Investigating - We are dealing with a partial outage with our analytics systems. Insights and performance metrics are temporarily unavailable..

Jul 20, 22:55:30 GMT+0
Identified - The root cause was identified as a failed allocator for the ElasticSearch cluster. A failure in one of the ElasticSearch nodes caused and increased CPU and load for the whole cluster, we are relocating the data to a different node and rebalancing the shards between the new nodes. We turned off some search queries to temporarily reduce the cluster load and reduce the recovery time. .

Duration: 11 hours and 29 minutes

Affected Components: , 3rd Party Integrations, Data Injestion, API →

Jun 13, 18:19:00 GMT+0
Investigating - We are currently investigating this incident..

Jun 13, 19:25:07 GMT+0
Identified - The issue is caused by the widespread [AWS outage](https://health.aws.amazon.com/health/home#/account/dashboard/open-issues?eventID=arn:aws:health:us-east-1::event/MULTIPLE_SERVICES/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE_798F8_DDDD18AFADD&eventTab=details) Switching to an alternative runtime.....

Jun 13, 20:06:54 GMT+0
Identified - We are attempting to switch the workload from AWS Lambda to AWS ECS, however, the commands are failing with a timeout. Looking for alternative ways of changing the infrastructure setup. .

Jun 13, 21:17:39 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jun 14, 05:48:13 GMT+0
Resolved - This incident has been resolved..

Duration: 18 minutes

Affected Components: Data Injestion, Data Pipeline, Cypress Integration

Mar 29, 20:11:08 GMT+0
Identified - The root cause is identified, rolling back the faulty deployment.

Mar 29, 20:20:55 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 29, 20:29:29 GMT+0
Resolved - This incident has been resolved. The root cause was identified as an issue with downstream npm dependency.

Duration: 6 days and 20 minutes

Affected Components: Cypress Integration

Mar 8, 20:36:52 GMT+0
Identified - Cypress 12.6.0+ introduced a breaking change to Currents integration - you may see an error message `Unsupported recording service`. We are working on restoring the integration. Meanwhile, please downgrade cypress runner to version 12.5.0. We apologize for the inconvenience, a separate communication will follow..

Mar 10, 11:01:11 GMT+0
Resolved - Customers experiencing "Unsupported Recording Service" please consider upgrading to https://github.com/currents-dev/cypress-cloud package.

Duration: 2 days, 12 hours and 27 minutes

Affected Components: , API →

Mar 3, 18:40:22 GMT+0
Investigating - Slow response time when loading the dashboard and getting API responses. We are currently investigating this incident..

Mar 4, 07:49:58 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 6, 07:07:13 GMT+0
Resolved - This incident has been resolved..

Duration: 2 hours and 32 minutes

Affected Components: Data Pipeline, , API →

May 23, 16:05:48 GMT+0
Investigating - We are currently investigating this incident..

May 23, 18:38:12 GMT+0
Resolved - We just resolved the issue!.

May 23, 16:05:48 GMT+0
Identified - We are continuing to work on a fix for this incident. - The requests to a downstream Elasticsearch dependency have been timing out. - Initiated increase of Elasticsearch cluster capacity.

May 23, 17:42:47 GMT+0
Monitoring - - We implemented a fix and currently monitoring the result - Due to the outage some items take longer than expected to appear in the dashboard. - We are running a process to validate data integrity and backfilling the missing data in Elasticsearch cluster..