shards.si

No frontier model runs on one machine. Training splits a model's parameters and optimizer states across thousands of GPUs — sharded, by name, in every major framework: FSDP, DeepSpeed, tensor and data parallelism. Inference does the same across model and pipeline shards to serve anything too large for a single device.

This isn't a phase superintelligence-scale computing grows out of. It's the opposite — as models get larger and workloads spread further across hardware, the amount of computing that happens inside a shard, rather than on a whole machine, only grows. The term describes the substrate itself, not a workaround.