Spinnable - Project Actions Failing Across Multiple Regions – Incident details

All systems operational

Project Actions Failing Across Multiple Regions

Resolved
Partial outage
Started 9 days agoLasted about 8 hours

Affected

Subprocessors

Operational from 5:49 PM to 6:16 PM, Degraded performance from 6:16 PM to 8:21 PM, Operational from 8:21 PM to 1:41 AM

Supabase → Database

Operational from 5:49 PM to 6:16 PM, Degraded performance from 6:16 PM to 8:21 PM, Operational from 8:21 PM to 1:41 AM

Updates
  • Resolved
    UTC
    Resolved

    Project actions have been sustained at normal performance. This incident is resolved.

  • Monitoring
    UTC
    Monitoring

    The mitigation has returned project activity to normal performance. Project actions that failed during the outage are being requeued to run. We will continue to monitor to ensure project functionality.

  • Identified
    UTC
    Identified

    We have identified a problem with a specific job in us-east-1, we have mitigated the job and are observing the results.

  • Update
    UTC
    Update

    We are testing mitigation options to attempt to restore project action performance.

  • Update
    UTC
    Update

    We have determined that there are four main project actions impacted in the identified regions: Pause/unpause failures, Restore/PITR failures, PITR configuration failures and Backup problems. We are pursuing root cause and mitigation options in parallel.

  • Investigating
    UTC
    Investigating

    Project actions in us-east-1, us-east-2, sa-east-1, ca-central-1 are experiencing significant increases in error rates. We are exploring the best options for restoring performance.