The 2024 AI HW Summit: Here’s What Caught My Attention (forbes.com)

The 2024 AI HW Summit: Here’s What Caught My Attention

The Summit drew over 1000 attendees this year, with scores of presentations and hundreds of AI leaders from large companies as well as many startups.

Every September since 2019, the AI HW Summit in the Bay Area has been the focal point for new technologies around AI. While the event, hosted by UK-based Kisaco, started with semiconductors, it has continually expanded its focus to include software, models, networking, and full data center optimization. Next year it will return as the AI Infra Summit, acknowledging that AI has become a full-stack endeavor that consumes entire data centers.

It may surprise some that Nvidia did not present at the event. They don’t see the need, since everyone knows who they are and how fast their GPUs are.

Here’s a few insights from the event.

A Food Fight Erupted Over the Claim, “Fastest Inference on the Planet”

Seems like the battle over inference services is really heating up, with Cerebras, Groq, and Samba Nova all claiming to be the fastest available tokens-as-a-service. Now, I am fairly confident that nobody is lying here, but let’s just say each company is cherry-picking the size of the Llama3.1 model they want to tout. And they are mostly referencing tests run by Artificial Analysis, which has results on its website.

(an excerpt on Celestial AI )

Optical is the Next Big Thing

How many times have we heard that? Its always coming “soon”. Yes, optical interconnects are widely used for rack-to-rack connectivity in modern data centers to get around the length limitations of copper and the need for retimers. But optical is rarely used within a rack, where the cable lengths are not a problem for the cheaper copper solutions.

The Celestial optical fabric can be used for system to HBM memory and System to System connectivity. ... [+]

But that may be about to change. Celestial AI is developing an elegant and performant design they were touting at the conference. Their approach could help solve the “memory wall” GPUs contend with today, by providing access to over 33 TB of shared HBM memory space. They claim they can lower costs by over 25 times, power by 8 times, and RDMA latency by 5 times, all while providing over 4 times better bandwidth. We will be watching these guys closely as they finish engineering their 1st generation.

Celestial AI's Fabric can reduce costs by over 20-fold, power by 8-fold, and latency by 5-fold ... [+]