Google’s Cloud Storage Failure Started Earlier
Preliminary report moves the Cloud Storage incident start earlier.
Google’s 9 October report moves the 8 October us-central1 incident start from 05:19 to 04:00 US Pacific. The revised period is 11:00 to 23:30 UTC.
Google says a routing change exhausted selected nodes. It disabled that change and rolled back a separate database update linked to metadata errors.
Write requests saw 503s and timeouts; reads had minor impact. Other regions and services were unaffected.
The following response steps are BlackTree operational guidance, not Google instructions or completed tests.
Search the full revised window
Use the revised 11:00 UTC start.
Review queries should use that boundary consistently across alerts, support cases, application logs and retry jobs. Otherwise, different teams can appear to agree while examining different time ranges.
BlackTree recommends creating one incident inventory with these fields:
- project, bucket, object or operation and service owner;
- request, trace or job identifier where available;
- UTC timestamp and client or service location;
- response code, timeout or application error;
- retry count and the final state currently observed;
- downstream workflow, deadline and business consequence.
Preserve the original error evidence before a manual retry. A later successful request can restore a workflow while hiding the first request’s outcome.
Reconcile writes before repeating them
A failed client request does not, by itself, establish whether an object exists, whether the intended generation is current or whether a dependent process acted on an earlier result.
For each uncertain write, compare the intended object name with the current object generation, size, checksum and relevant timestamps. Match that state against application logs and any request identifier. Where an external workflow consumes the object, verify its state separately.
Use the application’s existing idempotency controls when a retry is required. If the workflow lacks them, choose an explicit reconciliation owner and record the decision before resubmitting work. Avoid a broad replay based only on provider closure.
Keep metadata and read checks scoped
Keep the metadata review evidence-led. Search for actual errors before deciding which bucket, object or policy checks belong in the incident review.
Limit read checks to callers and time ranges where logs show a problem. Do not turn a regional incident into a claim that every Cloud Storage request failed.
Close with evidence
Define closure for each affected workflow. Useful evidence includes a reconciled object state, a completed dependent job, an explained retry, restored monitoring and an owner for any unresolved request. Record the time range and query used so the check can be repeated when the provider record changes.
This preliminary report makes no data-loss finding. It covers 8 October, not an 11 October outage. Keep any security, loss or cross-region claim out of incident communications unless separate evidence supports it.
BlackTree’s DigitalOcean BLR1 recovery analysis uses the same operational principle for a distinct provider incident: reconcile dependent work against local evidence before closing the event.
