At SC25, WEKA—best known for bringing high-performance data architectures to AI infrastructure—announced something that feels less like an upgrade and more like a pressure-relief valve for the entire AI industry. The company has taken its Augmented Memory Grid technology from concept to full commercial availability on NeuralMesh. And the timing could not be more relevant.

AI builders everywhere are running into the same wall: GPU memory. It’s fast, it’s precious, and it’s nowhere near large enough for the sprawling long-context models and agentic AI workflows that now dominate the market. The industry has thrown compute, distributed clusters, and clever caching at the problem—yet the wall remains.

WEKA’s answer: eliminate the wall entirely.

Validated on Oracle Cloud Infrastructure (OCI) and other major AI clouds, Augmented Memory Grid expands the available GPU memory footprint by 1000x, turning gigabytes into petabytes, while cutting time-to-first-token by up to 20x. Long-context inference, reasoning agents, research copilots, and multi-turn systems suddenly behave like they’ve been freed from a decade-old hardware ceiling.

It’s not an incremental improvement—it’s a structural rewrite of how AI memory can work.

The AI Memory Wall: Why GPU HBM Can’t Keep Up

The bottleneck isn’t theoretical. High-bandwidth memory (HBM) on GPUs is blisteringly fast but extremely small. System DRAM offers more space but only a fraction of the bandwidth. Once both tiers fill, inference workloads begin dumping their key-value cache (KV cache), forcing GPUs to recompute previously processed tokens.

That recomputation is the silent killer: it burns GPU cycles, slows inference speeds, drives up power consumption, and breaks the economics of long-context AI.

As large language models move toward 100K-token, 1M-token, and agentic, continuously-running interactions, the HBM-DRAM hierarchy collapses under its own constraints. And so far, no amount of clever software trickery has truly solved it.

WEKA’s approach: change the architecture.

Augmented Memory Grid: A New Memory Layer for AI

Instead of forcing GPUs to live inside the rigid boundaries of HBM, Augmented Memory Grid creates a high-speed bridge between GPU memory and flash-based storage. It continuously streams KV cache to and from WEKA’s “token warehouse,” a storage layer built for memory-speed access.

The important detail:
It behaves like memory, not storage.

Using RDMA and NVIDIA Magnum IO GPUDirect Storage, WEKA maintains near-HBM performance while letting models access petabytes of extended memory.

The result is that LLMs and reasoning agents can keep enormous context windows alive—no recomputation, no token wastage, and no cost explosions.

“We’re bringing a proven solution validated with OCI and other leading platforms,” said WEKA CEO and co-founder Liran Zvibel. “Scaling agentic AI isn’t just compute—it’s about smashing the memory wall with smarter data paths. Augmented Memory Grid lets customers run more tokens per GPU, support more users, and enable entirely new service models.”

This isn’t “HBM someday.” It’s HBM-scale capacity today.

OCI Validation: The Numbers That Matter

The technology didn’t just run in a lab. OCI testing confirmed the kind of performance that turns heads:

• 1000x KV cache expansion with near-memory speeds

• 20x faster time-to-first-token when processing 128K tokens

• 7.5M read IOPs and 1M write IOPs across an eight-node cluster

These aren’t modest deltas—they fundamentally change how inference clusters scale.

Nathan Thomas, VP of Multicloud at OCI, put it bluntly:
“The 20x improvement in time-to-first-token isn’t just performance—it changes the cost structure of running AI at scale.”

Cloud GPU economics have become one of the industry’s greatest pain points. Reducing idle cycles, avoiding prefill recomputations, and achieving consistent cache hits directly translate into higher tenant density and lower dollar-per-token costs.

For model providers deploying long-context systems, this is the difference between a business model that breaks even and one that thrives.

Why Long-Context AI Needed This Yesterday

As LLMs evolve from text generators into autonomous problem-solvers, the context window becomes the brain’s working memory. Coding copilots, research assistants, enterprise knowledge engines, and agentic workflows depend on holding vast amounts of information active simultaneously.

Until now, supporting those windows meant trading off between:

• astronomical compute bills

• degraded performance

• artificially short interactions

• forced summarization that loses fidelity

With Augmented Memory Grid, the trade-offs shrink dramatically. AI agents can maintain state, continuity, and long-running memory without burning GPU cycles on re-prefill phases.

Put differently:
LLMs get to think bigger, remember longer, and respond faster—without crushing infrastructure budgets.

A Broader Shift: AI Architecture Moves Beyond Compute

For the last five years, AI scaling strategies have focused overwhelmingly on compute—bigger GPUs, faster interconnects, more parallelization. Memory, by contrast, has been the quiet constraint no one could fix.

WEKA’s move highlights a turning point:
AI’s next leap forward won’t come from more FLOPs. It will come from smarter memory architectures.

NVIDIA’s ecosystem support—Magnum IO GPUDirect Storage, NVIDIA NIXL, and NVIDIA Dynamo—signals that silicon vendors recognize the same shift. Open-sourcing a plugin for the NVIDIA Inference Transfer Library shows WEKA wants widespread adoption, not a walled garden.

OCI’s bare-metal infrastructure with RDMA networking makes it one of the first clouds capable of showcasing the technology without bottlenecks.

This ecosystem convergence—cloud, GPU, and storage—suggests that memory-scaling tech will become a foundational layer of next-gen inference stacks.

Commercial Rollout and What Comes Next

Augmented Memory Grid is now available as a feature for NeuralMesh deployments and listed on the Oracle Cloud Marketplace. Support for additional clouds is coming, though the company hasn’t yet named which.

The implications for AI providers are straightforward:

• Long-context models become affordable to run

• Agentic AI becomes easier to scale and commercialize

• GPU clusters become more efficient

• New monetization models become viable (persistent assistants, multi-user agents, continuous reasoning systems)

WEKA has effectively repositioned memory—from hardware limitation to software-defined superpower.

If compute defined AI’s last decade, memory may define its next one.

Get in touch with our MarTech Experts.