Course Syllabus
Welcome to ECE408 / CS483 / CSE408
The aim of this course is to provide knowledge and hands-on experience in developing software for processors with massively parallel computing resources, such as Graphics Processing Units, or GPUs. Effectively programming these processors requires in-depth knowledge about parallel programming principles, as well as the parallelism models, communication models, and resource limitations of these processors. In addition, this course also prepares the students to use the parallelism models of modern programming environments for high-performance computing and AI. The target audiences of the course are students who want to develop applications for these processors for fun, for work, and/or for research, as well as those who want to develop programming tools and future processors.
The course involves 8 programming lab assignments of steadily growing complexity. All assignments will involve programming a massively parallel GPU system using CUDA, which is a popular commercial language extension of C/C++ for GPU programming. Assignments involve tasks such as matrix multiplication, convolution, reduction, histogram calculation, and sparse matrix-vector multiplication. During the final third of the semester, students work on a larger, more complex project/competition.
Course Staff
Instructors
Prof. Volodymyr Kindratenko.
Graduate Teaching AssistantsHowie Liu (hanwenl4@illinois.edu), |
Undergraduate Course AssistantsHrishi Shah (hrishis2@illinois.edu), |
Class Time and Location
AL: 1002 ECEB, Tuesdays and Thursdays, 9:30-10:50am US Central Time by Prof. Volodymyr Kindratenko. Lectures will be recorded and posted on Mediaspace.
ALL other sections: Recorded lectures posted on Mediaspace.
Reading Materials
Textbook
D. Kirk and W. Hwu, “Programming Massively Parallel Processors – A Hands-on Approach,” Morgan Kaufman Publisher, 3rd edition, 2016, ISBN 978-0123814722. Relevant to this course chapters can be found under Files/Textbook.
NVIDIA documentation
NVIDIA, CUDA C Programming Guide (CUDA PG)
https://docs.nvidia.com/cuda/cuda-c-programming-guide
https://docs.nvidia.com/cuda/pdf/CUDA_C_Programming_Guide.pdf
NVIDIA, CUDA C++ Best Practices Guide (CUDA BPG)
https://docs.nvidia.com/cuda/cuda-c-best-practices-guide
https://docs.nvidia.com/cuda/pdf/CUDA_C_Best_Practices_Guide.pdf
NVIDIA Developer Blog (NVIDIA DB)
https://developer.nvidia.com/blog
Computer System Setup
We will use NCSA's Delta supercomputer for programming assignments (labs and project) in this course. We will distribute course assignments via a GitHub repository and will collect students' solutions via GitHub as well. Below are instructions on how to get started with Delta and GitHub.
Delta Access
To get started with Delta, follow these steps:
- Go to https://identity.access-ci.org/new-user and register yourself as a new user. If you already have an ACCESS account (some of you might already have it due to a research project, etc.) do not register as new user, just use the same username. When registering for new account, use "University of Illinois Urbana-Champaign" as your Identity Provider. This way, you can just use your UofI credentials.
- Once your account is created, note your ACCESS user ID and submit this user ID to the course staff via this form: https://forms.illinois.edu/sec/1378190679.
- Wait for an email from NCSA Delta team confirming completion of your account registration. This may take several days, so please be patient. After receiving the confirmation email, you should be able to login into Delta via ssh and clone the GitHub repository to get started with the labs.
- While you are waiting for a confirmation that your account has been setup on Delta, go ahead and familiarize yourself with Delta's documentation and also setup your class GitHub repository.
GitHub setup
To get started with GitHub:
- First, follow this link to establish a class repository: https://edu.cs.illinois.edu/create-gh-repo/sp25_ece408. This needs to be done only once. After this step is complete, you should be able to visit https://github.com/illinois-cs-coursework/sp25_ece408_NetID page (where NetID is your NetID) that shows the contents of your repository. Initially, this repository will be empty.
- After completing step 1, you should be able to access the class release repository at https://github.com/illinois-cs-coursework/sp25_ece408_.release. Please read
README.release.md file for further instructions about what to do next. You can proceed with these steps once your Delta account is setup.
Getting Help
Office hours
Course instructor and TAs will have office hours offered in-person and on-line starting on the second week of the semester (week of 1/27). Watch this space for updates.
| US Central Time | Monday | Tuesday | Wednesday | Thursday | Friday |
| 8 - 9 am | |||||
| 9 - 10am | |||||
| 10 - 11am | |||||
| 11am - noon | Hrishi ECEB 4036 |
||||
| noon - 1pm | Charles ECEB 4034 |
||||
| 1 - 2pm | Shengjie ECEB 4036 |
Hrishi ECEB 4036 |
|||
| 2 - 3pm | Yue ECEB 3036 |
||||
| 3 - 4pm | Xiyue Zoom |
Yifei Zoom |
Colin & Howie ECEB 4034 |
||
| 4 - 5pm | |||||
| 5 - 6pm | |||||
| 6 - 7pm | Vijay ECEB 4036 |
||||
| 7 - 8pm | |||||
| 8 - 9pm |
Discussion board
We will use Campuswire for Q&A. Sign up here if you are not yet enrolled. (You will be asked for a verification code, which is 1698.)
Course chatbot
Yes, we are providing a GPT-4 powered AI chatbot trained on ECE 408 course materials! You are welcome to use it to learn, not to cheat! You can start by asking the chatbot how to succeed in this class. :)
Please read this disclaimer before using the chatbot, this a very experimental technology and you must understand its limitations and agree to the provisions outlined in the disclaimer.
You can use the chatbot in a couple of ways, just follow the corresponding links:
- Q&A chatbot - in this mode the chatbot gives direct answers to the questions based on the course-related documentation.
- Guided learning chatbot - in this mode the chatbot attempts to guide you toward the answer. This is a good way to get help so you can learn.
Grading
Grads and undergrads in the same grading pool. All grades will be posted in Canvas.
- Exam 1: 25%
- Exam 2: 25%
- Lab Assignments: 25%
- Correctness (determined by the number of data sets passed and the functionality) 90%
- Responses to the quiz 10%
- Lab with the lowest score will be dropped from the final grade, therefore no late submissions beyond grade period will be allowed.
- Lab 0 and Lab 1 grades will not be counted towards the final grade.
- Final Project: 25%
- Detailed Rubric will be posted with the project description
Grace Period for Assignments
All assignments are due by Friday at 8pm. However, recognizing potential issues such as Delta system maintenance, interviews, or illness, we provide a three-day grace period. You can submit an assignment by Monday at 8pm and still receive full credit without penalty. You do not need to notify us if you intend to use this default 3-day grace period. However, to extend your grace period beyond 3 days, you must notify us by Friday 8pm via email to the TA(s) responsible for the assignment you are asking about. Extensions to the grace period are given out for extenuating circumstances at our discretion. Requests after this time will not be considered. Please also note that the code in your GitHub repository at 8pm on Monday will be considered the final version, even if you had a correct version earlier. We will retrieve and grade the code at that time.
Lecture Schedule
Lecture slides can be found here. Reading materials can be found here and here. Lecture recordings can be found here.
| Week | Date | Activity | Topic | Reading |
| 1 | 1/21/25 | Lecture 1 | Introduction to the Course | Kirk & Hwu: Chapter 1 CUDA PG: Introduction |
| 1/23/25 | Lecture 2 | Introduction to Parallel Computing and CUDA | Kirk & Hwu: Chapter 2 CUDA PG: Programming Model |
|
| 1/24/25 | Lab 0 due | ACCESS and GitHub setup; Delta access and Device Query example |
|
|
| 2 | 1/28/25 | Lecture 3 | CUDA Parallel Execution Model | Kirk & Hwu: Chapter 3 |
| 1/30/25 | Lecture 4 | CUDA Memory Model | Kirk & Hwu: Chapter 4 | |
| 1/31/25 | Lab 1 due | Vector Addition | ||
| 3 | 2/4/25 | Lecture 5 | Locality and Tiled Matrix Multiply | Kirk & Hwu: Chapter 4 CUDA PG: Shared Memory NVIDIA DB: Shared Memory |
| 2/6/25 | Lecture 6 | Generalized Tiling and DRAM Bandwidth | Kirk & Hwu: Chapter 5 CUDA BPG: Device Memory Spaces NVIDIA BD: Global Memory |
|
| 2/7/25 | Lab 2 due | Simple Matrix Multiplication | ||
| 4 | 2/11/25 | Lecture 7 | Convolution concept (1D and 2D); 1D Basic Convolution Kernel, and constant cache | Kirk & Hwu: Chapter 7 CUDA BPG: Device Memory Spaces |
| 2/13/25 | Lecture 8 | 2D Tiled Convolution Kernel | Kirk & Hwu: Chapter 7 NVIDIA DB: Shared Memory |
|
| 2/14/25 | Lab 3 due | Tiled Matrix Multiplication | ||
| 5 | 2/18/25 | Lecture 9 | Shared Memory Reuse Analysis | Kirk & Hwu: Chapter 7 |
| 2/20/25 | Lecture 10 | Introduction to Machine Learning |
|
|
| 2/21/25 | Lab 4 due | 3D Convolution | ||
| 6 | 2/25/25 | Lecture 11 | Deep Networks: Inference and Training | Kirk & Hwu: Chapter 16 |
| 2/27/25 | Lecture 12 | Computation in CNNs and Transformers | Kirk & Hwu: Chapter 16 | |
| 2/28/25 | ||||
| 7 | 3/4/25 | Midterm 1 | Lectures 1-12 and Labs 0-4 | |
| 3/6/25 | Lecture 13 | Reductions | Kirk & Hwu: Chapter 5 | |
| 3/7/25 | CNN-PM1 due | Project Milestone 1 Code | ||
| 8 | 3/11/25 | TA lecture | Overview of Project | |
| 3/13/25 | Lecture 14 | Parallel Scan: Kogge-Stone | Kirk & Hwu: Chapter 8 | |
| 3/14/25 | Lab 5 due | List Reduction | ||
| Spring Break | ||||
| 9 | 3/25/25 | Lecture 15 | Parallel Scan: Brent-Kung | Kirk & Hwu: Chapter 8 |
| 3/27/25 | Lecture 16 | Histogramming and Atomic Operations | Kirk & Hwu: Chapter 9 CUDA PG: Atomic Functions |
|
| 3/28/25 | Lab 6 due | Scan | ||
| 10 | 4/1/25 | Recorded lecture | Application Profiling with Nsight | C. Pearson's slides and videos (part1, part2, part3, part4) Nsight Systems, Nsight Compute |
| 4/3/25 | Lecture 17 | Sparse Matrix I | Kirk & Hwu: Chapter 10 | |
| 4/4/25 | Lab 7 due | Histogram | ||
| 11 | 4/8/25 | Lecture 18 | Sparse Matrix II | Kirk & Hwu: Chapter 10 |
| 4/10/25 | Lecture 19 | GPU Systems Architecture | ||
| 4/11/25 | CNN-PM2 due | Project Milestone 2 Code & Report | ||
| 12 | 4/15/25 | Lecture 20 | Tasks Parallelism and Asynchronous Data Transfer | Kirk & Hwu: Chapter 18 |
| 4/17/25 | Lecture 21 | Other Accelerations APIs | ||
| 4/18/25 | Lab 8 due | Sparse Matrix Multiplication | ||
| 13 | 4/22/25 | Lecture 22 | Tensor Processing APIs | NVIDIA Technical Blog CUDA PG: Warp Matrix Functions |
| 4/24/25 | Guest Lecture | TBD | ||
| 14 | 4/29/25 | Guest Lecture | TBD | |
| 5/1/25 | Lecture 23 | Course Retrospective | ||
|
5/2/25 |
CNN-PM3 due | Project Milestone 3 Code & Report | ||
| 15 | 5/6/25 | Midterm 2 | ||
| 5/9/25 | Project Competition due | Project Competition | ||
Labs (aka MPs)
Lab (aka MP) assignments will be done using NCSA's Delta system for coding and PrairieLearn for quiz submission. Programming part of each lab will be distributed via GitHub repository. Q&A part will be done via PrairieLearn. Do not start working on the quiz part before completing the programming part as the quiz will require providing info from the actual working lab. Labs are individual assignments; they must be carried out and submitted individually.
Lab assignments are due on Fridays at 8:00pm US Central Time, unless otherwise noted or explicitly extended by the teaching staff.
| Date released | Due Date | Notes | |
| Lab 0: Device Query | 1/21/25 | 1/24/25 | not counted toward the final grade |
| Lab 1: Vector Addition | 1/24/25 | 1/31/25 | not counted toward the final grade |
| Lab 2: Simple Matrix Multiplication | 1/31/25 | 2/7/25 | |
| Lab 3: Tiled Matrix Multiplication | 2/7/25 | 2/14/25 | |
| Lab 4: 3D Convolution | 2/14/25 | 2/21/25 | |
| Lab 5: List Reduction | 3/7/25 | 3/14/25 | |
| Lab 6: Scan | 3/14/25 | 3/28/25 | |
| Lab 7: Histogram | 3/28/25 | 4/4/25 | |
| Lab 8: Sparse Matrix Multiplication | 4/11/25 | 4/18/25 |
Lab TA: Yue Yuan and Xiyue Zhu
Project-CNN
We release the project in the same GitHub repository as the labs. Use "git pull" if you have checked out the project before. We will update instructions one milestone at a time. Projects are individual assignments, they must be carried out and submitted individually. We will use Delta for project as well.
Milestones are due on Fridays at 8:00pm US Central Time, unless otherwise noted or explicitly extended by the teaching staff.
| Date Released | Due Date | Notes | |
| Milestone 1 | 2/21/25 | 3/7/25 | |
| Milestone 2 | 3/12/25 | 4/11/25 | |
| Milestone 3 | 4/15/25 | 5/2/25 | |
| Final Competition | 5/5/25 | 5/9/25 | optional |
Project-CNN TAs: Shengjie Ma and Yifei Song
Project-GPT
This semester, we are excited to launch a new project where we implement GPT-2 from scratch! Our goal is to build a fully functional version of the model, diving deep into the transformer architecture.
Stay tuned for updates, behind-the-scenes insights, and technical deep dives as we publish this project! 🚀✨
Project-GPT TA: Howie Liu
Exams
There are two midterm exams, there is no final exam.
| Midterm 1 | Midterm 2 |
|
When: 3/4/25, 7-10pm Where: TBD What: Lectures 1-12 and Labs 0-4 How: paper-based Alternative Exam Time: as arranged Study Materials: see Files/MT1 Review Session: TBD Last-minute Office Hours: TBD The cover page and the reference sheet will be provided. |
When: 5/6/25, 7-10pm Where: TBD What: All lectures, including all guest lectures, all Labs, all project milestones. With a focus on everything after Exam 1. How: paper-based Alternative Exam Time: as arranged Study Materials: see Files/MT2 Review Session: TBD Last-minute Office Hours: TBD The cover page and the reference sheet will be provided. |
Cheating and Academic Integrity
We take cheating very seriously. If we catch you cheating, or helping someone cheat, the penalty will not be less than a zero on the applicable assignment and the incident will be reported to the College.
The Student Code defines cheating as:
Using or attempting to use in any academic exercise materials, information, study aids, or electronic data that the student knows or should know is unauthorized (§1-402(a))
and notes that:
Substantial portions of the same academic work may not be submitted for credit more than once or by more than one student without authorization. (§1-402(a)(4))
The student code also disallows facilitation of cheating, prohibiting
Helping or attempting to help another to commit an infraction of academic integrity, where one knows or should know that through one’s acts or omissions such an infraction may be facilitated. (§1-402(c))
Cheating is an offense to the entire academic ecosystem: you, your classmates, and the course staff. It creates extra work for the course staff, it creates grading imbalances for your classmates, and it prevents you from learning the material that the rest of your academic career is built on.