Spinnable - Notice history

Project creation has returned to baseline consistently.

Project creation is reporting as recovered across the platform. We will monitor for stability in the near term.

We are seeing the mitigation steps having the desired effect. Project creation is recovering.

While the first set of actions are taking effect, we are implementing a second set of likely mitigations to attempt to facilitate improvement in project creation.

We continue to work this issue. The steps we have taken are addressing the hypothesized cause of the issue. We are implementing further related mitigations.

We have a working hypothesis on the cause of the issue, and are implementing steps to try to improve project creation performance.

Project upgrades have remained stable, we are resolving this incident.

We have deployed a tested fix for the upgrade issue impacting some projects. The fix has been effective and we are monitoring for continued stability.

We are continuing to test the fix to project upgrades. We will update again when the fix is applied.

The team has identified a likely fix and is consolidating the details in order to move to thorough testing. We will update again when the fix is applied.

We are continuing to fix project upgrade issues and are investigating further remediation. We will provide our next update within the next 14 hours.

We are developing a fix for the upgrade issues.

We have identified the cause of the upgrade delays and are working on a fix.

We are continuing to investigate project upgrade issues.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The incident has been resolved.

The upstream DNS provider has implemented a fix and is monitoring the results. We are seeing signs of recovery; we will continue to monitor the situation and will provide further updates as more information becomes available.

We have identified that the issue is with an upstream DNS provider, causing DNS creation delays for newly created, unpaused, or updated databases. Our team is actively monitoring the situation and will provide further updates as more information becomes available.

This incident is resolved. DNS creation for new projects has been fixed since 20:35 UTC. Backfill of missing DNS records has been completed at 22:20 UTC. Please contact Support if your project seems impacted by missing DNS records.

All failed DNS creation has been backfilled, and customers should no longer see missing DNS records in their projects. DNS record creation and latency have returned to normal levels. We will continue to monitor before resolving this incident.

We have identified and mitigated the issue that was blocking DNS updates. DNS record creation is succeeding, and latency has returned to normal levels. A subset of projects that encountered provisioning failures during the incident will require DNS records to be backfilled, and we are actively remediating these affected projects.

We have identified the cause of delays in DNS records creation and are actively working on a fix. We will continue to provide updates as things progress.

Project actions have been sustained at normal performance. This incident is resolved.

The mitigation has returned project activity to normal performance. Project actions that failed during the outage are being requeued to run. We will continue to monitor to ensure project functionality.

We have identified a problem with a specific job in us-east-1, we have mitigated the job and are observing the results.

We are testing mitigation options to attempt to restore project action performance.

We have determined that there are four main project actions impacted in the identified regions: Pause/unpause failures, Restore/PITR failures, PITR configuration failures and Backup problems. We are pursuing root cause and mitigation options in parallel.

Project actions in us-east-1, us-east-2, sa-east-1, ca-central-1 are experiencing significant increases in error rates. We are exploring the best options for restoring performance.