Parallelism

A technical walkthrough of the transformer attention block, the parallelism strategies used to distribute it (DP, TP, CP, PP), mixture-of-experts dispatch and combine, and the collective communication primitives and cost models underneath all of it.