Nvidia Packs Optical Components into Switch Chip Packages to Target Data Transfer Bottlenecks in AI Clusters
This in-depth report from Scientific American explains why Nvidia is replacing pluggable optical modules with a co-packaged optics solution: human operations in data centers are responsible for 5% of downtime, old optical modules consume about 25 watts each, and integrated optical engines are claimed to increase mean time between interruptions by at least 10 times. The article also reveals Nvidia's investment layout in the silicon photonics ecosystem and its strategic intent to control the communication layer of AI infrastructure.
The bottleneck in large-scale AI clusters is shifting from computation to data transfer. Gilad Shainer, Senior Vice President of Networking at Nvidia, said in an interview with Scientific American that the rule is simple: use copper whenever you can. But copper is only suitable for short-distance, low-bandwidth scenarios. When tens or even hundreds of thousands of compute nodes and memory units in a data center need to communicate with each other at extremely high speeds, the problems with copper become apparent: electrons carry charge and repel each other, requiring power to drive them; copper itself acts like a filter, smearing clear electrical pulses into blurred noise.
Photons do not have this problem. They carry no charge and hardly interact with one another. A laser beam passing through glass fiber optic cables experiences negligible signal attenuation over several kilometers. However, data inside chips is in electronic form. To hand it off to photons for transmission and then back to electronics for processing, a conversion device is required—a transceiver.
The traditional approach is to make transceivers into pluggable modules inserted into the front panel of switches. Craig Thompson, Vice President of Interconnect Products at Nvidia, explained the fatal weakness of this method: physical connectors can loosen and move, having pins, solder joints, and epoxy, and every insertion or removal affects adjacent modules. Shainer provided a figure: the presence of humans in data centers is itself responsible for 5% of downtime.