The Propagation Gap: Understanding the 23-Minute Revocation Window in Google Cloud API Keys

In a distributed cloud environment, speed and consistency often exist in a delicate balance. Recent security research has highlighted a critical friction point between these two principles: the time it takes for a credential revocation to propagate across Google Cloud’s global infrastructure.

Specifically, researchers have identified a significant security window where deleted Google API keys remain intermittently functional for up to 23 minutes, potentially allowing unauthorized actors to maintain access even after a breach has been “remediated.”

The Mechanics of Eventual Consistency in Authentication

The vulnerability was uncovered by security researchers at Aikido, led by Joe Leon. Their findings suggest that the issue is not a traditional “bug” in the sense of broken code, but rather a byproduct of eventual consistency. In large-scale distributed systems, updates to a database (like a key deletion) are propagated across various global nodes over time to ensure high availability and performance.

During testing across 10 separate trials, the researchers documented a highly inconsistent revocation timeline:

• Maximum Delay: Approximately 23 minutes
• Minimum Delay: Approximately 8 minutes
• Median Delay: Approximately 16 minutes

This delay creates a period of “non-deterministic authentication.” Because Google’s backend is composed of countless distributed servers, a request might be processed by a node that has received the deletion update (resulting in a 403 Forbidden), while a subsequent request might be routed to a node that has not yet synchronized (resulting in a 200 OK). This inconsistency provides a window of opportunity for an attacker to continue exfiltrating data or exploiting services like Gemini or BigQuery.

Regional Disparities and Logging Blind Spots

The research further indicated that revocation speed is influenced by geographic location and network topology. Tests conducted across various Google Cloud regions—including us-east1, europe-west1, and asia-southeast1—showed varying success rates. In several instances, nodes geographically distant from the U.S. appeared to revoke credentials faster, suggesting that internal routing patterns and caching layers play a significant role in how quickly a “delete” command is respected globally.

Beyond the access window, there is a secondary challenge for Incident Response (IR) teams: attribution. When a key is deleted, Google Cloud Platform (GCP) logs transition the associated request metadata to a generic apikey:UNKNOWN label. This makes it difficult for security analysts to perform post-mortem forensics, as they can no longer easily map specific malicious requests back to the original compromised credential once it has been purged from the system.

Credential Hierarchy: Not All Keys Are Created Equal

Crucially, the researchers noted that this 23-minute lag is not universal across all Google Cloud authentication methods. The disparity suggests that Google has implemented more robust, immediate revocation paths for newer or more sensitive credential types:

• Service Account Keys: Highly efficient, with revocation occurring in approximately 5 seconds.
• New Gemini API Keys (AQ-prefixed): Rapid revocation, typically within 1 minute.
• Legacy Google API Keys: Vulnerable to the full 23-minute propagation delay.

Google has reportedly classified this behavior as “won’t fix,” maintaining that the eventual consistency model is an architectural choice intended to preserve system performance. However, for security professionals, this creates a discrepancy between the user experience (where the console implies immediate deletion) and the actual security posture.

Technical Mitigation Strategies

Since the “revocation window” is an inherent part of the current architecture, organizations must adopt a defense-in-depth approach to credential management. We recommend the following adjustments to standard operating procedures:

• Assume Exposure: During an active incident, treat API key deletion as a “best effort” rather than an instant kill-switch. Assume the key remains active for at least 30 minutes.
• Prioritize Service Accounts: Move away from legacy API keys in favor of Service Accounts, which offer near-instantaneous revocation and better identity management.
• Proactive Rotation: Implement automated rotation schedules to minimize the lifespan of any single key, reducing the impact if one is leaked.
• Enhanced Monitoring: If a key is suspected of compromise, monitor Cloud Logging for any apikey:UNKNOWN activity immediately following the deletion to identify ongoing exploitation.

Ultimately, this research serves as a reminder that in distributed computing, “deleted” does not always mean “gone.”