A critical yet silently patched denial-of-service (DoS) vulnerability, dubbed "HollowByte" by Okta’s Red Team, has brought to light significant concerns regarding OpenSSL’s security disclosure practices and the subtle complexities of memory management in widely used software components. The flaw, which can lead to unpatched OpenSSL servers allocating and effectively losing up to 131 KB of memory per connection due to a mere eleven bytes of attacker-controlled data, poses a persistent threat, particularly on systems utilizing the glibc library, where the allocated memory remains unavailable until the affected process is restarted. This revelation has sparked debate within the cybersecurity community, primarily due to OpenSSL’s decision to classify the fix as a "bug or hardening" rather than a security vulnerability, thereby foregoing a public CVE identifier, official advisory, or even a changelog entry.
The Anatomy of HollowByte: A Deceptive DoS Attack
The core of the HollowByte vulnerability lies in a fundamental trust issue within older OpenSSL versions during the initial stages of a Transport Layer Security (TLS) handshake. TLS, the cryptographic protocol designed to provide secure communication over a computer network, relies on a series of messages exchanged between a client and a server. Each TLS handshake message is preceded by a four-byte header, three of which are designated to declare the expected length of the message body. Prior to the June 9 patches, OpenSSL servers would immediately allocate a receive buffer sized according to this declared length as soon as the header arrived. Crucially, this allocation occurred before any part of the message body was received and before internal integrity checks of the handshake could be performed.
An attacker could exploit this by sending a ClientHello message header with a maliciously inflated length declaration (up to 131 KB, the maximum permissible size for a ClientHello message) but then simply fail to send the corresponding message body. The server, having prematurely committed the memory, would then block the worker thread assigned to that connection, endlessly awaiting data that would never arrive. This effectively creates a hung connection, preventing any authentication, session establishment, or key exchange from proceeding.
The Persistent Drain: How Glibc Amplifies the Threat
While connection exhaustion attacks are a well-understood category of DoS, reminiscent of techniques like Slowloris which merely tie up server resources by keeping connections open, HollowByte introduces a more insidious dimension due to its interaction with the glibc memory allocator. Glibc (GNU C Library) is a pervasive component in many Linux-based systems, providing fundamental functions including memory management. When the attacker eventually drops the connection, OpenSSL attempts to free the allocated buffer. However, glibc’s default behavior for small and medium-sized memory chunks is to hold them for potential reuse by the application rather than immediately returning them to the operating system kernel.
Okta’s Red Team discovered that by subtly varying the claimed size on each malicious connection, the attacker could effectively bypass glibc’s internal reuse mechanisms. This constant allocation of slightly different-sized chunks, followed by their "freeing" into glibc’s internal pools, leads to severe heap fragmentation. The consequence is a steady climb in the process’s resident set size (RSS), consuming physical memory that is not truly in use by active processes. This memory remains fragmented and inaccessible for other legitimate uses, even long after the attacker has ceased their activity. The only way to reclaim this memory is to restart the affected OpenSSL process, which in many production environments, is a disruptive operation itself, essentially forcing a DoS through memory exhaustion.
Quantifying the Impact: Okta’s Testing Reveals Alarming Figures
Okta’s testing provided concrete figures illustrating the severity of HollowByte. In their NGINX server tests, a server provisioned with 1 GB of memory was OOM-killed (Out-Of-Memory) after just 547 MB of memory became "frozen" in these inaccessible fragments. On larger systems, such as a 16 GB server, the attack successfully locked up 25% of the total system memory without ever hitting traditional connection limits. This is a critical distinction, as it means conventional defenses designed to cap the number of concurrent connections will be ineffective against HollowByte. The attack doesn’t overwhelm the server with too many active connections; instead, it silently drains its memory resources through a relatively small number of highly damaging, ghost connections. Okta emphasized that "standard connection-limiting defenses won’t stop it," underscoring the novel challenge posed by this vulnerability. As of July 18, "The Hacker News" reported no public proof-of-concept exploit code for HollowByte on platforms like GitHub, suggesting that while the technical details are known, widespread exploitation tools are not yet readily available.
A Quiet Fix: OpenSSL’s Controversial Classification
The most contentious aspect of the HollowByte discovery is OpenSSL’s handling of its disclosure and remediation. The fix for this vulnerability was included in OpenSSL releases 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21, all dated June 9. Despite its significant impact, particularly when combined with glibc’s memory management, OpenSSL opted not to assign a Common Vulnerabilities and Exposures (CVE) identifier, publish a security advisory, or even explicitly document the fix in its changelog entries.
Matt Caswell, the developer who authored the patch, explicitly stated in the relevant pull request (e.g., #30792 for master and 4.0) that the security team chose to "handle this as a ‘bug or hardening’ only fix." This decision stands in stark contrast to OpenSSL’s own security policy, which defines four severity tiers (Critical, High, Moderate, Low), all of which typically warrant a CVE, a changelog note, and an entry on their vulnerabilities page. HollowByte received none of these customary acknowledgments. For instance, the release notes for OpenSSL 4.0.1 and its comprehensive 23-entry changelog made no mention of this significant memory issue.
OpenSSL has not publicly articulated its reasoning for this classification. One possible rationale, as inferred by "The Hacker News," could be that a bounded allocation (131 KB per connection) might not be considered a vulnerability in the traditional sense, as every TLS server allocates memory per connection. From this perspective, the issue might be viewed as an inefficient memory use rather than a security flaw. However, Okta’s counter-argument is compelling: the memory, once allocated, never returns, leading to a functional denial of service through resource exhaustion.
A Question of Policy: Inconsistencies in Vulnerability Disclosure
The decision to downplay HollowByte raises questions about the consistency of OpenSSL’s vulnerability disclosure policy, especially when compared to other recent issues. For example, in January, OpenSSL assigned CVE-2025-66199, rated "Low," to a TLS 1.3 certificate-compression bug. This flaw also involved a peer-supplied length growing a heap buffer before validation, potentially consuming around 22 MiB per connection. However, that vulnerability required four specific conditions to be met for exploitation: certificate compression compiled in, a compression algorithm available, the extension negotiated, and, on servers, client certificates requested. HollowByte, in contrast, requires none of these prerequisites, making it a more broadly applicable threat.
Furthermore, the same June 9 release that silently fixed HollowByte did assign CVE-2026-34183, rated "Moderate," to an unbounded memory growth vulnerability in the QUIC PATH_CHALLENGE handler. Both this and HollowByte are memory-exhaustion DoS issues, yet one received a CVE and the other did not. This apparent discrepancy complicates the understanding of OpenSSL’s internal criteria for vulnerability classification and public disclosure. The release also addressed 18 other CVEs, including a High-severity use-after-free bug in PKCS7_verify(), meaning users updating to these releases would receive fixes for these well-documented issues, while the HollowByte fix would remain largely unnoticed.
Patching Challenges and Recommendations for Downstream Users
The absence of a CVE and official advisory creates significant challenges for organizations, especially those relying on downstream Linux distributions or custom build processes. Distributions like Red Hat often employ a "backporting" strategy, where security fixes are applied to existing stable versions of software rather than moving to a newer upstream version. This means a patched package might still report the original, vulnerable version number. Normally, security advisories and OVAL feeds, keyed to CVE names, would guide users to these backported fixes. Without a CVE for HollowByte, this standard process is rendered ineffective.
Organizations are therefore left with a more arduous task: directly inquiring with their package maintainers or manually inspecting package changelogs to ascertain if their OpenSSL builds incorporate the specific patches. The relevant pull requests are #30792 for master and 4.0 branches, #30793 for 3.6, 3.5, and 3.4, and #30794 for the 3.0 branch. For those who build OpenSSL from source, the recommendation is clear: upgrade to one of the listed patched releases (4.0.1, 3.6.3, 3.5.7, 3.4.6, or 3.0.21) and ensure all services loading the older OpenSSL libraries are restarted.
The Unaddressed DTLS Vulnerability
Compounding these concerns is the fact that the fix for HollowByte currently applies only to TLS. Matt Caswell noted in the pull request discussions that addressing the issue in Datagram Transport Layer Security (DTLS) would be significantly more invasive and that the project decided against it for now. A comparison of OpenSSL’s source code between the 3.6.2 and 3.6.3 tags, performed by "The Hacker News," revealed that the DTLS handshake file remained byte-identical across the fix. Even in the newest 4.0.1 release, the DTLS path continues to size its buffer based on the peer-declared length, leaving it susceptible to a similar memory exhaustion attack.
OpenSSL has not classified this remaining DTLS vulnerability, nor has it committed to a timeline for its remediation. The official release notes, changelogs, and vulnerabilities page remain silent on this unaddressed risk, leaving users to glean this critical information only from the more obscure pull request comments.
Broader Implications and Ongoing Questions
The HollowByte incident underscores the intricate balance between security, performance, and transparency in open-source projects that form the bedrock of global internet infrastructure. The quiet patching of a significant DoS vulnerability, particularly one that bypasses common mitigation strategies and interacts negatively with ubiquitous system libraries like glibc, raises legitimate questions about how security-critical projects prioritize and communicate risks.
"The Hacker News" has formally reached out to OpenSSL for clarification on why HollowByte was triaged below the "Low" severity tier and whether the fix has been backported to the extended-support 1.1.1 and 1.0.2 branches. They have also inquired with Okta about whether the memory fragmentation issue observed with glibc persists with other memory allocators. The responses to these questions will be crucial in fully understanding the scope and implications of this stealthy threat.
The cybersecurity community now faces the challenge of identifying and patching a vulnerability that intentionally lacks the standard identifiers used for automated scanning and reporting. This situation highlights the potential for "shadow vulnerabilities" to persist in critical systems, hidden not by design flaws, but by ambiguous disclosure policies, posing an ongoing and difficult-to-track risk to internet-facing services worldwide.
