Lateral HBM: The Revolution That Cools AI GPUs

What happened?

At the 2026 IEEE/JSAP Symposium on VLSI Technology and Circuits, researchers from Korea and Japan presented two independent approaches to redesign HBM (High-Bandwidth Memory) by placing DRAM chips on their side, instead of stacking them vertically as is currently done. The Korean proposal, called V-Die (Vertical-Die), was developed by the Ulsan National Institute of Science and Technology (UNIST). The Japanese proposal, MOSAIC, comes from the University of Tokyo. Both studies, reported by Tom's Hardware, address a critical bottleneck in AI accelerators: the thermal and bandwidth limitations of conventional stacked memory.

Why is it important?

Current AI GPUs, such as NVIDIA's H100 or B200, use vertically stacked HBM3 or HBM4. This stacking creates a serious thermal problem: heat gets trapped between layers, limiting density and performance. By rotating the chips 90 degrees, space is freed for liquid cooling channels between dies, and more input/output vias (TSVs) can be added, increasing bandwidth and capacity without raising temperature. This paradigm shift could double performance per watt in future accelerators, according to the simulations presented. Historically, vertical stacking has been the standard since HBM's inception in 2013, but increasing layers (from 8 in HBM2E to 16 in HBM4) has exacerbated thermal issues. These proposals break that trend by offering a radical solution.

Technical details

In the V-Die design, DRAM chips are placed vertically on a substrate, with TSVs traversing the bottom edge of each die. This allows each die to have its own I/O interface, increasing parallelism. Simulations show that a V-Die system achieves 540 tokens per second on a GPT-3-like workload, compared to 296 tokens/s for conventional HBM4, an 82% increase. Additionally, direct liquid cooling between dies reduces temperature by an estimated 30%. MOSAIC, on the other hand, addresses the interconnection problem: with many vertical dies, the number of physical connections becomes unfeasible. The solution is a contactless interface that uses tiny induction coils to transfer data at 4 Gbps per channel. MOSAIC doubles the equivalent capacity of HBM4 in a DRAM-on-GPU configuration, and its design allows scaling to over 100 dies without increasing interconnection complexity. Both proposals, though conceptual, are based on detailed simulations and small-scale prototypes presented at the symposium.

Consequences for the industry

These innovations could enable future AI GPUs to operate with higher bandwidth and lower energy consumption, alleviating the memory bottleneck that limits scaling of models like GPT-4 or Gemini. Manufacturers such as SK Hynix, Samsung, and Micron, which dominate the HBM market with a combined share of over 90%, will need to consider these architectures for their post-HBM4 roadmaps. The impact on the data center market is significant: according to IDC data, spending on AI servers will grow 35% annually until 2028, and HBM memory accounts for up to 40% of a GPU's cost. Reducing that cost through improvements in density and cooling could accelerate AI adoption. Additionally, companies like NVIDIA have already shown interest in advanced packaging technologies, as evidenced by their investment in TSMC CoWoS. However, transitioning to these new architectures will require changes in production lines and certification by integrators.

What readers should know

• These are academic research projects, not commercial products. Years of development are still needed for implementation, possibly until 2028-2030 according to the researchers' own estimates.
• Direct liquid cooling between dies is a paradigm shift in memory packaging, but it poses sealing and long-term reliability challenges.
• The technique could extend to other types of stacked memory, such as DDR5 or LPDDR, though the initial focus is HBM for AI.
• MOSAIC, by using inductive coupling, eliminates the need for dense physical connections, which could simplify assembly and reduce costs.

"The main challenge of AI today is not just computations, but moving data without melting the chip," says an analyst at TheVortiq. "These proposals tackle the problem at its root, but their commercial viability will depend on collaboration between memory manufacturers and chip designers."

In summary, V-Die and MOSAIC represent a significant conceptual advance that could redefine memory design for AI in the next decade. Investors and professionals should closely follow upcoming prototypes and statements from SK Hynix, Samsung, and Micron regarding their post-HBM4 roadmaps.