Seminar Schedule

Topic 1, Session 1.0 | From HPC to AI Infra: Parallel Computing and Parallel Programming Topic 1, Session 1.1 | CUDA Programming Model Topic 1, Session 1.2 | Triton/TileLang Tile Level Programming Seminar overview, curriculum design, evaluation, and computing resources.

This session follows CUDA Core FP32 GEMM to develop an intuition for data reuse and the abstraction of the memory hierarchy. We start from a strawman GEMM, evolve it through tiling, and use the Roofline Model to analyze data reuse. From the hardware perspective, we then study SMs, warp execution, coalescing, latency hiding, shared-memory bank conflicts, padding, swizzle, and per-thread microtiles. Finally, we write and tune a kernel with Nsight Compute.

In this workshop, we introduce TileLang syntax, writing kernels with TileLang, CTA-to-thread mapping and layout inference, latency hiding, DSL JIT compilation, and host-side integration. We also cover operator optimization in TileLang and several related tools.

This session explores the hardware and programming model of Tensor Cores, tracing the evolution of mma.sync on Ampere, wgmma on Hopper, and tcgen05 on Blackwell. We derive the design logic of Tensor Cores from data movement and issue bandwidth, study fragment layouts, and learn how layouts and swizzles can resolve bank conflicts. We conclude with the use of low-precision Tensor Cores through FP8 fine-grained scaling.

As hardware has evolved, compute capability and memory bandwidth have grown rapidly while clock frequency has improved more slowly. Improvements in the latency of individual operations have lagged behind system throughput, so modern architectures increasingly rely on parallelism to tolerate and hide latency. We trace this evolution from instruction-level parallelism and out-of-order execution to memory-level parallelism, warp scheduling, and TLP. As specialized units such as Tensor Cores continue to raise throughput, the gap between computation and data supply grows wider. Mechanisms such as cp.async and TMA expose latency hiding as an explicit pipeline that must be organized by programmers and compilers. We will synthesize these ideas as Pipeline Ordering: Data Orchestration, a key to reaching the performance limits of modern hardware and optimized operators.

As computational demands continue to grow, a single compute unit is no longer enough; we need ways to connect compute units together. Historically, buses, PCIe, TCP/IP, and Ethernet have answered key interconnect questions at different layers. In the AI era, large-scale HPC/AI workloads demand higher bandwidth, lower latency, and greater reliability, driving the evolution of network systems. Starting from the origin of communication requirements, this session explains the composition and design of modern network systems from first principles. No pre-reading is required.

The central question behind a pipeline is: who decides what work runs, when it runs, and on which resource? Who maintains dependencies, advances execution, and determines when a stage has truly completed? These questions involve trade-offs between hardware and software responsibility, programming and hardware complexity, static and dynamic scheduling, explicit control and automation, flexibility and predictability, and the performance gains of fine-grained overlap versus its state and synchronization costs. Inside a single GPU, Compiler, Warp Scheduler, Scoreboard, Pipeline, Barrier, Copy Engine, and Memory Hierarchy share these responsibilities. We study how data moves among Registers, Shared Memory, on-chip specialized storage, and HBM while computation progresses correctly. We then extend the same problem to high-speed GPU-to-GPU interconnects, where Producer and Consumer may reside on different GPUs and data may cross GPU Fabric, NVLink / NVSwitch, PCIe, RNIC, RDMA Transport, and switching networks. This session examines where Readiness, Ordering, Completion, and Visibility responsibilities should live across devices, address domains, queues, protocols, and failure domains, and introduces communication requirements in workloads and classic industry solutions. No pre-reading is required.