HollowByte: The 11-Byte Attack That Broke OpenSSL

A newly disclosed vulnerability serves as a sobering reminder of how deeply our modern digital infrastructure relies on foundational cryptographic libraries. Even a tiny oversight in a low-level protocol implementation can ripple upward, threatening the stability of everything from web servers to global databases.

The Okta Red Team recently uncovered a sophisticated Denial of Service (DoS) flaw within OpenSSL, dubbed “HollowByte.” This vulnerability allows a remote, unauthenticated attacker to force a server to allocate disproportionate memory chunks before a single security handshake is even completed—all using a payload just 11 bytes long.

The Mechanics of the Handshake Vulnerability

To understand the exploit, we must look at the initial stages of the Transport Layer Security (TLS) handshake. The process begins with a ClientHello message encapsulated within a TLS record. Every handshake message includes a 4-byte header that explicitly declares the length of the incoming message body.

The vulnerability lies in how older versions of OpenSSL manage its receive buffers. Instead of validating the incoming data, the library allocates a buffer based on the attacker-declared length *before* any actual data has been received. When a malicious 11-byte payload hits the server, the TLS state machine processes the header and triggers an unvalidated pre-allocation sequence:

Read Header → grow_init_buf() → OPENSSL_clear_realloc() → malloc(attacker_size)

Because there is no payload validation at this stage, the malloc() function allocates up to 131 KB based solely on the untrusted claim within the packet. The worker thread then enters a blocked state, waiting indefinitely for the rest of the data that the attacker never intends to send.

From Thread Exhaustion to Heap Fragmentation

While holding connections open to exhaust thread pools is a well-known tactic (reminiscent of the Slowloris attack), HollowByte introduces a much more destructive secondary effect due to how glibc manages memory.

When a connection is eventually dropped, OpenSSL frees the allocated buffer. However, to optimize performance, glibc does not immediately return small-to-medium allocations back to the Operating System; it retains them in its own pool for potential reuse. By launching waves of connections with randomized claimed sizes, an attacker can prevent the allocator from successfully reusing freed chunks. This leads to severe heap fragmentation, causing the server’s Resident Set Size (RSS) to climb uncontrollably.

The impact is devastating. In testing conducted with NGINX, an unpatched server in a 1 GB RAM environment was hit by an Out-of-Memory (OOM) killer once it reached just 547 MB of frozen, fragmented memory. In a 16 GB environment, the attack was able to lock up 25% of total system memory while remaining well under typical connection-limiting thresholds, rendering standard rate-limiting defenses ineffective.

Scope of Impact and Remediation

Because OpenSSL is a ubiquitous dependency, the blast radius of this flaw is massive. It affects web servers like NGINX and Apache, various language runtimes (Node.js, Python, Ruby, PHP), and critical databases such as MySQL and PostgreSQL.

The Okta Red Team detailed the discovery in their technical breakdown: OpenSSL HollowByte: A DoS Hiding in 11 Bytes.

OpenSSL has addressed the issue by moving away from trusting declared header sizes. The library now utilizes incremental buffer growth, ensuring that buffers only expand as bytes actually arrive on the wire. This fix was merged via pull requests #30792, #30793, and #30794. The resolution is available in the following versions:

• OpenSSL v4.0.1 (Primary fix)
• Backports: 3.6.3, 3.5.7, 3.4.6, and 3.0.21

Interestingly, OpenSSL classified this as a “hardening fix” rather than issuing a formal CVE advisory. Despite this classification, the technical reality is clear: security teams should prioritize upgrading their OpenSSL packages immediately, particularly on internet-facing infrastructure where traditional connection-limiting defenses provide no protection against this memory-exhaustion vector.