Nvidia GPUs Vulnerable to Exploit as Researchers Uncover New Memory Attack

Researchers from the University of Toronto have demonstrated a new memory attack technique, GPUThor, that notably advances the art of GPU exploitation. This method is capable of bypassing the error-correcting code (ECC) defenses employed on enterprise-grade Nvidia GPUs, potentially allowing attackers to gain root access to affected systems.

GPUThor is recognized as an evolution of the existing Rowhammer attack, which exploits the physical proximity of memory cells. Initially showcased against DDR3 and DDR4 memory in 2015, the original Rowhammer exploits how tightly packed memory cells can unintentionally leak electric charges, flipping bits between 0 and 1. This phenomenon—dubbed “row hammering”—can lead to grave security vulnerabilities, including privilege escalation and even manipulation of AI models.

Over the years, various adaptations of Rowhammer have emerged, extending their reach to memory types like DDR5, which was thought to be impervious, as well as GDDR chips used in graphics cards. The researchers describe GPUThor as the first Rowhammer variant targeting Nvidia GPUs that effectively defeats ECC, which had previously been considered a reliable countermeasure. They claim, “Our previous study, GPUHammer, showcased Rowhammer bit flips on the GDDR6 memory of Nvidia GPUs, while a follow-up, GPUBreach, demonstrated privilege escalation leading to root access.” Historically, Nvidia had suggested enabling ECC as a strategic defense mechanism.

Distinct Features of GPUThor

What makes GPUThor particularly concerning is its non-uniform bit-flipping approach. Unlike earlier attacks that executed uniform memory row hammering, GPUThor strategically targets rows to induce double and triple-bit flips, creating errors that ECC mechanisms aren’t equipped to manage. This attack is approximately 6.6 times more aggressive than its predecessors, resulting in an astounding 500 to 23,500 times the number of bit flips.

The practical implications are significant; GPUThor drastically shortens the time needed to identify an exploitable bit flip. For instance, utilizing GPUHammer takes about 21.9 hours on an Nvidia RTX A6000 GPU without ECC enabled. In contrast, GPUThor could identify vulnerabilities in just 1.1 minutes on the same hardware.

The research team confirmed the effectiveness of their method on various Nvidia Ampere GPUs equipped with GDDR6 memory, including the RTX A4000, A4500, A5000, and A6000—models frequently utilized in workstations and cloud services. However, they noted that their attack did not work on Nvidia A100 or H100 servers or newer RTX 5090 and RTX 6000 models, which utilize different memory, such as HBM or GDDR6X. The group intends to further investigate these newer models, as they do not rule out the presence of other potential vulnerabilities.

Implications of GPUThor

This vulnerability poses a serious risk in environments where Nvidia’s enterprise-class GPUs are leveraged, particularly for AI model training and inference. Much of today's sensitive operations rely on these GPUs, often shared across various virtual machines within data centers. During their studies, researchers found that their attacks could crash GPUs so frequently that the cards self-reported defects, necessitating replacement.

The consequences of such exploits extend beyond mere disruption. They can trigger denial-of-service scenarios that incapacitate all workloads on affected GPUs. Notably, the researchers were able to corrupt memory page tables, allowing unprivileged applications to escalate privileges to root. “In cloud environments where GPUs are shared, an attacker can flip bits in a target's data, leading to crashes,” the team explained. The threat escalates for any untrusted code running on these GPUs, as it opens doors for potential malware entry points.

Addressing the Threat

Since GPUThor exploits a fundamental design flaw, effectively mitigating the risk necessitates improvements in hardware defenses in future GPU designs. For the time being, users are advised to exercise caution regarding executing untrusted code on their GPUs and to keep a close eye on Nvidia's ECC error counters—an uptick could indicate an ongoing attack.

Following their findings, the University of Toronto team reported GPUThor to Nvidia in April 2023. In response, Nvidia published a security advisory outlining steps to mitigate the threat, including enabling host IOMMU/DMA isolation where feasible and utilizing tools like nvidia-smi to monitor ECC telemetry on GPUs.