Machine Learning

A question-driven, ground-up walkthrough of transformer internals, DP/TP/CP/PP parallelism, MoE dispatch and combine, and the collective operations underneath it all, written while reading the NanoCP and UltraEP papers.