If there's one company and one person that has garnered the attention of the entire tech sector in 2023, it's undoubtedly Nvidia and its CEO Jensen Huang. Back in February, Huang declared that "AI's iPhone moment" had arrived. Six months later, the market examined Nvidia's financial performance over the last two quarters, and the company responded with profits that exceeded all expectations.
The number of times "AI" is mentioned during companies' 2Q23 earnings call
Source: Earnings calls, compiled by DIGITIMES, 2023/08
Nvidia's financial data for the second quarter of fiscal year 2024 (2QFY24) presented an outstanding performance: a 101% on-year increase in revenue, a 422% on-year increase in net profit, and a gross profit margin of 71.2%. All these matrics far exceeded projections. The H100 series chips are in very high demand in the AI-trend-driven LLM market.
The explosive growth in AI computing power demand meant that for cloud providers, their only solution is to constantly procure more AI servers. However, for LLM developers, Nvidia's data center business is thriving and raking in substantial revenue from customers that have yet to achieve profitability. The LLM wave has not fully arrived, but Nvidia is already taking all the money.
Before the financial reports were released, there were continuous rumors about price increases for Nvidia's high-end AI GPUs. Citing analyst reports, Barron's pointed out that the manufacturing cost for H100 is US$3,320 with a selling price ranging from US$25,000-30,000. The profits are nearly 10 times the cost, with a gross profit margin of over 70%, which is quite astonishing for chip products.
Regarding this, Huang emphasized that Nvidia provides a software ecosystem and a hardware platform. It offers the flexibility and versatility of software architecture. Coupled with a broad installation base and coverage, the sheer volume of code and application combination types is astounding. Simply put, Nvidia is selling a "quasi-system" rather than a "single chip," something that took Nvidia 20 years to achieve.
Analysts pointed out that the CUDA software ecosystem Nvidia constructed not only made its GPU favored but is also a key reason why its customers couldn't easily switch to competitors like AMD. This is because AI computing power configuration is much more than just stacking hardware. You can stack multiple Nvidia HGX platforms, each weighing more than 30kg and having over 35,000 components, and still not get the AI computing power required for LLMs.
Running a language model (LM) training or inference task with hundreds of acceleration cards and AI servers requires complementary software and communication networks. This doesn't even include the massive amount of customer data. This series of software-hardware integration is a complex process. Huang referred to AI GPUs like the H100 as "technological miracles," and it's for good reason.
The H100 began mass production and was introduced to the market in September 2022. It has been in high demand ever since. Even its predecessor, the A100, remains a highly sought-after product. Nvidia's focus is on system-level server products. Its latest DGX product GH200 contains 256 H100 chips and the Grace CPU and is expected to become the new revenue generator after mass production in the second half of 2023.
On the other hand, why is H100 so expensive? On the hardware side, H100 incorporates four advanced technologies. The first is TSMC's 4nm process (N4). Considering the H100's size (26.8cm by 11.1cm) and the 80 billion transistors it contains, TSMC's manufacturing technology is a necessity.
The second is the connection technology. Each H100 includes 3 NVLink connections. This is Nvidia's exclusive data transmission technology. The fourth generation NVLink can provide a 900GB/s GPU-to-GPU interconnectivity bandwidth. This allows multiple H100 to be jointly used, thus doubling their performance.
The third is the 80GB HBM2e high bandwidth memory (HBM), which SK Hynix and Samsung Electronics are the only suppliers right now. It's worth noting that Nvidia recently announced that the HBM3e will be installed in its latest GH200, making it the first GPU product in the world to support HBM3e.
Last but not least is TSMC's CoWoS packaging technology. While TSMC is actively expanding its CoWoS capacity, the supply shortage is likely to continue until 2024.
Each of these four technologies is cutting-edge in their respective fields, so they naturally come at a considerable cost. If LLM is the force behind the arrival of the AI GPU accelerated computing era, Nvidia's massive leading position right now is a result of the long-term investments and accumulated efforts of Jensen Huang's leadership team.
Article translated by Jack Wu
