Objectives
To learn parallel programming with graphics processing units (GPUs).
Outcomes: Students would learn concepts in parallel programming, implementation of programs on GPUs, debugging and profiling parallel programs.
Course Contents:
Topic (number of lectures + number of tutorials)
- Introduction (2 + 1):
History, graphics processors, graphics processing units, GPGPUs. Clock speeds, CPU / GPU comparisons, heterogeneity. Accelerators, parallel programming, CUDA / OpenCL / OpenACC,
- Hello World Computation (3 + 1)
Kernels, launch parameters, thread hierarchy, warps / wavefronts, thread blocks / workgroups, streaming multiprocessors, 1D / 2D / 3D thread mapping, device properties, simple programs
- Memory (8 + 2) :
Memory hierarchy, DRAM / global, local / shared, private / local, textures, constant memory. Pointers, parameter passing, arrays and dynamic memory, multi-dimensional arrays. Memory allocation, memory copying across devices. Programs with matrices, performance evaluation with different memories
- Synchronization (6 + 2):
Memory consistency. Barriers (local versus global), atomics, memory fence.
Prefix sum, reduction. Programs for concurrent data structures such as worklists, linked-lists. Synchronization across CPU and GPU
- Functions (3 + 1):
Device functions, host functions, kernels, functors. Using libraries (such as Thrust), developing libraries.
- Support (1 + 2):
Debugging GPU programs. Profiling, profile tools, performance aspects
- Streams (3 + 1):
Asynchronous processing, tasks, task-dependence. Overlapped data transfers, default stream, synchronization with streams. Events, event-based-synchronization
- overlapping data transfer and kernel execution, pitfalls.
- Case studies (3 + 2) :
Image processing. Graph algorithms. Simulations. Deep learning.
- Advanced topics (8 + 2) : Dynamic parallelism. Unified virtual memory. Multi-GPU processing. Peer access. Heterogeneous processing
Course evaluation would involve programming assignments.
Text Books:
Programming Massively Parallel Processors: A Hands-on Approach; David Kirk, Wen-mei Hwu; Morgan Kaufman; 2010 (ISBN: 978-0123814722)
Reference Books:
CUDA Programming: A Developer's Guide to Parallel Computing with GPUs; Shane Cook; Morgan Kaufman; 2012 (ISBN: 978-0124159334)
