Session 01 — GPU Programming Model
Why can a GPU solve large parallel problems even though an individual GPU thread is not faster than a CPU thread?
Why does CUDA organize threads into threads, blocks, and grids? What is a warp?
When moving from CUDA to Triton or TileLang, why does the programming model shift from thinking about one thread to thinking about a tile of data?
- ML systems on GPUs: SIMD and SIMT, throughput and latency, and why GPUs suit massively parallel workloads
- CUDA programming model: host and device, kernel launches, threads, blocks, grids, warps, and coordination between threads
- Triton and TileLang: program-, block-, and tile-level programming
This session introduces the basic viewpoint used throughout the rest of the seminar.