UIUC CS 498: Machine Perception, Fall 2026

Machine perception applications: 3D scene understanding, autonomous navigation, robotics, and spatial mapping

Instructor: Prof. Shenlong Wang, TA: Hongchi Xia

Time: Wed/Fri 9:30 a.m.-10:45 a.m. Fall 2026, Location: Siebel 0216

First lecture: Wednesday, August 26, 2026, 9:30–10:45 a.m., Siebel 0216.

Office Hour: Friday, 10:45 a.m.-noon (Siebel 2407)

Schedule, Canvas, GitHub repository

1. Introduction

Overview: Machine perception enables embodied systems to sense, interpret, and reason about the world around them. The course develops geometric, probabilistic, and learning-based approaches across vision, LiDAR, radar, sonar, GPS, inertial sensing, odometry, touch, force, and torque. Topics include poses and transforms, camera models and calibration, deep learning, 3D representations, motion and correspondence, reconstruction, state estimation, localization and mapping, segmentation, detection and tracking, occupancy, neural rendering, multimodal foundation models, and perception for action. Lectures, tutorials, implementation assignments, and role-playing paper discussions connect foundational methods to modern systems. The course culminates in a two-track project competition with a proposal, implementation, analysis, report, and poster/demo presentation.

Course format: Students learn through lectures, paper discussions, hands-on implementation, quizzes, and a substantial final project. The emphasis is on connecting foundational ideas to modern machine-perception systems and research practice.

Topic progression: The semester moves from sensors, poses, and camera models through reconstruction and state estimation, then to scene understanding, neural rendering, multimodal models, and perception-action systems.

Prerequisites: You should be comfortable with multivariable calculus, linear algebra, data structures (CS 225 or equivalent), and probability and statistics (CS 361, STAT 400, or equivalent). Prior coursework in machine learning, robotics, or computer vision is not required. However, completing at least one of CS 444, CS 445, or CS 446 is recommended because the course moves quickly from mathematical foundations to implementation in Python and PyTorch.

Prerequisite check: If you are unsure whether your background is sufficient, speak with the instructor or TA during office hours before committing to the course.

Course Objective: By the end of the course, you will be able to explain how common sensors produce measurements and how those measurements support estimation and decision-making; apply geometric, probabilistic, and learning-based methods to perception problems; implement, debug, and test representative machine-perception algorithms in Python and PyTorch; critically evaluate research papers from multiple roles; design controlled experiments and interpret their results; and communicate technical work through presentations, visualizations, and writing.

Applied outcomes: The course develops practical experience with tasks such as calibrating sensors, estimating camera and robot poses, fusing motion measurements, reconstructing scenes, localizing and mapping, detecting and tracking objects, building semantic occupancy maps, and creating simulation-ready representations. These skills support research and applications in robotics, autonomous systems, AR/VR, and embodied AI.

Requirements: This is a fast-paced, technically demanding course. Students are expected to attend lectures and discussions in person, complete the assigned readings before class, contribute actively to role-playing discussions, and submit all implementation assignments, write-ups, quizzes, and final-project deliverables on time. The course will introduce the required machine-perception concepts, but students are responsible for keeping up with the mathematical and programming workload. Attendance and preparation: Regular, prepared attendance is required because each class depends on live discussion, peer feedback, and assigned roles.

Recommended references: The course does not follow a single textbook. Useful references include:

Richard Szeliski, Computer Vision: Algorithms and Applications (2nd ed.);
Richard Hartley and Andrew Zisserman, Multiple View Geometry in Computer Vision;
Sebastian Thrun, Wolfram Burgard, and Dieter Fox, Probabilistic Robotics;
Tim Barfoot, State Estimation for Robotics;
Frank Dellaert and Michael Kaess, Factor Graphs for Robot Perception;
Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning;
Kris Hauser, Robotic Systems.

Several drafts are available online from their authors.

2. Grading & Deliverables

Course grades are based on 100 total points.

HW0 (10%): Perspective, Homography, Stereo
HW1 (15%): SLAM, Mapping
HW2 (10%): Point Cloud Segmentation
HW3 (15%): Detection and Tracking
HW4 (13%): NeRF
HW5 (13%): Diffusion Policy
Final Project (20%)
Attendance (4%)

Late submission policy:

Free late days for homework assignments: Each student gets a total of five free late days that apply to homework assignments throughout the whole semester. As long as you stay within your total late days budget, there is no need to request an extension and no late penalty will be assessed.
Late penalty: If you are out of late days, for every day that your assignment is late, your score is multiplied by 0.6. Submissions that are more than five days late (beyond any free days) will not be accepted. You are not allowed to submit different parts of the assignment at different times to receive a late penalty on only part of the assignment.
Medical accommodations: Requests beyond the five free late days will be considered only when supported by a doctor's note.

3. Final Project

The final project is a two-track competition, tentatively hosted on Kaggle, built around two real-world 3D perception applications. For both tracks, the teaching staff will provide training and validation data and will reserve a held-out test set for final evaluation on the Kaggle server. Use the validation set for development and tuning. Each team may submit to the test benchmark at most three times per week, and leaderboard rank is determined by held-out test performance. Everyone only needs to choose to participate in one track.

Track I - Semantic Occupancy (no AI track): Build a model that converts an online RGB-D sensor stream into a semantic occupancy map of the robot's surroundings. The staff will provide helper functions and a baseline model. AI coding or problem-solving tools are not permitted for any deliverable in this track. The purpose is to demonstrate independent understanding of the complete modeling and debugging process; a competitive solution is expected to require roughly 100-400 lines of student-written code. Each student should be able to get reasonable performance on semantic occupancy with reasonable free GPU resources on Google Colab or Kaggle Notebook

Track II - Simulation-Ready Reconstruction (AI-assisted): Build an end-to-end system that reconstructs a complete 3D scene from input videos, including complete object geometry, texture, and physical properties needed for simulation. AI assistants such as Copilot, Claude, and Codex are permitted. Beyond a baseline data loader and evaluation code, the staff will provide limited scaffolding, and this track may require more computation resources (e.g. personal GPU, paid Colab Pro plan, pre-emptable GPU on Campus Cluster or Delta / Delta AI). Save your pipeline and document how AI contributed: prompts, feedback, failed attempts, revisions, and the iterations that improved validation and test performance. Include clear visual results and a video showing the reconstruction loaded in MuJoCo or Isaac Sim. Note: CS498 instructor team cannot provide tech support for any of the GPU resources.

Final-project grading:

Model and system design (40%): evaluates technical soundness and correctness, rationale, implementation quality, and appropriate use of validation data.
Report writing (40%) evaluates clarity, reproducibility, related-work context, experimental design, limitations, and, for Track II, the AI-human collaboration record.
Analysis and visualization (20%) evaluates quantitative metrics, ablations or controlled comparisons, error analysis, qualitative results, and the required simulator demonstration.
Bonus (0-10%) Competition winners may earn a 1-10 percentage-point bonus; the exact award will depend on final leaderboard performance.

4. Participation

In-class and discussion participation (4%): We aim to maintain a friendly, rigorous, and interactive classroom. Ask questions during lectures, discussions, and office hours, and help classmates examine both strengths and limitations of the work under discussion. Attendance may be checked through short in-class forms, but participation credit reflects both presence and substantive engagement.

Canvas and Slack:

Canvas is the official source for announcements, course materials, submissions, grades, and formal course updates.
Use the appropriate public Canvas channel for conceptual and technical questions whenever possible so that classmates can benefit from the discussion. Be respectful and do not share private or restricted material.
Canvas is moderated by the teaching staff; follow any channel-specific posting rules.
For private matters, you can choose to post Canvas questions that are not public (hence will only be visible to Shenlong and Hongchi).

Email:

Only email both Hongchi / Shenlong when Canvas is unavailable. When sending email, start with [CS498 Fall] in title (otherwise will be filtered).

DRES Accommodations:

To obtain disability-related academic adjustments and/or auxiliary aids, students with disabilities must contact the course instructor and the Disability Resources and Educational Services (DRES) as soon as possible. To contact DRES you may visit 1207 S. Oak St., Champaign, call 333-4603 (V/TDD), or e-mail a message to disability@uiuc.edu.

Academic Integrity

Submit work produced by you and your group, and cite all significant sources of ideas, code, data, and prior results. Follow the AI-use policy stated for each assignment and project track: Track I prohibits AI assistance, while Track II permits AI tools only with the required collaboration record and disclosure. If any submission builds on work completed before the semester, discuss it with the instructor in advance. Academic-integrity violations will be handled under university policy and may receive a zero for the affected assignment or a more serious sanction.

Anti-racism and Inclusivity Statement

The intent of this section is to raise student and instructor awareness of the ongoing threat of bias and racism and of the need to take personal responsibility in creating an inclusive learning environment. The Grainger College of Engineering is committed to the creation of an anti-racist, inclusive community that welcomes diversity along a number of dimensions, including, but not limited to, race, ethnicity and national origins, gender and gender identity, sexuality, disability status, class, age, or religious beliefs. The College recognizes that we are learning together in the midst of the Black Lives Matter movement, that Black, Hispanic, and Indigenous voices and contributions have largely either been excluded from, or not recognized in, science and engineering, and that both overt racism and micro-aggressions threaten the well-being of our students and our university community. The effectiveness of this course is dependent upon each of us to create a safe and encouraging learning environment that allows for the open exchange of ideas while also ensuring equitable opportunities and respect for all of us. Everyone is expected to help establish and maintain an environment where students, staff, and faculty can contribute without fear of personal ridicule, or intolerant or offensive language. If you witness or experience racism, discrimination, micro-aggressions, or other offensive behavior, you are encouraged to bring this to the attention of the course director if you feel comfortable. You can also report these behaviors to the Bias Assessment and Response Team (BART) (https://bart.illinois.edu/). Based on your report, BART members will follow up and reach out to students to make sure they have the support they need to be healthy and safe. If the reported behavior also violates university policy, staff in the Office for Student Conflict Resolution may respond as well and will take appropriate action.

Appendix 1: Final Report Template

Title: A concise, descriptive project title

Authors: List each team member's name, email address, and NetID

1. Introduction and Related Work

Define the problem, its motivation, and the setting in which it matters. State the research question clearly, explain why existing solutions are insufficient, and summarize the practical or scientific significance of solving it.
Summarize the most relevant prior work and cite the original sources. Explain what those methods accomplish, where they fall short for your setting, and how your project differs. Include a complete reference list at the end of the report.
State your main contributions and the approach you propose. Clearly distinguish your original work from methods, models, code, data, or design choices borrowed from prior work, and cite every borrowed component.
End the introduction with a brief preview of the strongest quantitative and qualitative results. Report the final competition test metric and rank when applicable, without using the held-out test set for model selection.

2. Approach

Problem formulation: Define the task mathematically. Specify the inputs, expected outputs, notation, assumptions, constraints, and objective or prediction target so that the evaluation problem is unambiguous.
Approach overview: Explain the central idea of the method and why it should address the problem. Provide a clear system diagram or algorithm summary when it improves understanding, and identify the key design decisions.
Inference: Describe the complete path from input to output, including preprocessing, model architecture, post-processing, test-time optimization, and any iterative inference procedure such as test-time optimization. State what runs at inference time and its expected runtime or resource needs.
Training procedure: Identify all learnable parameters, loss terms, optimizer, learning-rate schedule, data augmentation, initialization, stopping criteria, and important hyperparameters. Explain any special training arrangement and how hyperparameters were chosen.

3. Experiments and Results

Experimental setup: State the research questions or hypotheses tested, the baselines and ablations used, the hardware and software environment, and any random seeds or repeated trials. Explain how comparisons are kept fair.
Datasets and splits: Describe the data source, licensing or access restrictions, train/validation/test splits, preprocessing, filtering, and any synthetic or augmented data. State clearly which split was used for each decision.
Baselines: Describe all the competing algorithms and alternative baseline methods you’ve tried.
Metrics: Define every reported metric, including units and whether higher or lower is better. Justify why the metrics measure the intended behavior, and report uncertainty or variation when multiple runs are available.
Validation results: Use clearly labeled tables and figures to compare the proposed method with baselines and relevant ablations. Discuss which changes helped, which did not, and how the validation set guided development.
Final test and competition results: Report the final held-out test metrics and competition rank when applicable. Identify the submitted model or checkpoint, and do not use test results to tune the method.
Note: remember to provide sufficient and well-organized quantitative and qualitative results to support your analysis.

4. Analysis, Limitations, and Conclusion

Analysis: Interpret the results rather than only restating numbers. Explain the main improvements and tradeoffs, connect evidence back to the research question, and use qualitative examples to clarify model behavior.
Failure cases, limitations, and conclusion: Present representative failures and diagnose likely causes. Discuss limitations, safety or generalization concerns, and cases where the method should not be trusted. Conclude with the main findings and the most valuable next steps.

5. Reproducibility and Implementation Details

Reproducibility: List dependency versions, hardware, training and inference time, compute requirements, key configuration values, and the command or procedure needed to reproduce the main results. Identify all borrowed code or pretrained models and cite their sources and licenses.
AI-use record: Identify every AI tool used and document the human-AI workflow, including representative prompts, feedback, iterations, failures, revisions, and insights. Track I reports must explicitly state that no AI tools were used. Track II reports must make the collaboration record detailed enough for a reader to understand what the team decided, verified, and changed.

6. Supplementary Materials

Provide links to code, checkpoints, configuration files, complete quantitative results, and inference images or videos that support the report. For semantic occupancy, include side-by-side RGB input, ground-truth semantic occupancy rendering, and predicted rendering when available. For simulation-ready reconstruction, include input views, reconstructed geometry and appearance, and a video of the inferred scene running in MuJoCo or Isaac Sim. Caption every figure and identify the example, model, and setting shown.

Course Schedule

Fall 2026 course schedule (tentative)
Date	Day	Topic	Lab / Tutorial	Homework	Final Project
2026-08-26	Wed	L1: Intro, Logistics, History and Overview
2026-08-28	Fri	L2: Poses & Transforms
2026-09-02	Wed	Camera I
2026-09-04	Fri	Camera II	Tutorial: Linear Algebra & Pytorch	HW0 Out
2026-09-09	Wed	LiDAR, Radar, Sonar
2026-09-11	Fri	GPS, IMU, Odometer
2026-09-16	Wed	Touch, Force, Torque, etc.
2026-09-18	Fri	Sensor Calibration		HW0 Due / HW1 Out
2026-09-23	Wed	Deep Learning I
2026-09-25	Fri	Deep Learning II
2026-09-30	Wed	3D Representations
2026-10-02	Fri	Motion & Correspondence
2026-10-07	Wed	Structure from Motion & 3D Reconstruction		HW1 Due / HW2 Out
2026-10-09	Fri	Localization & State Estimation
2026-10-14	Wed	SLAM
2026-10-16	Fri	Segmentation
2026-10-21	Wed	Detection		HW2 Due
2026-10-23	Fri	Object Tracking		HW3 Out
2026-10-28	Wed	Pose Tracking
2026-10-30	Fri	Occupancy
2026-11-04	Wed	NeRFs / 3DGS		HW4 Out	Final Competition Starts
2026-11-06	Fri	CLIP and DINO		HW3 Due
2026-11-11	Wed	VLM & Multimodal Foundation Models
2026-11-13	Fri	Perception & Action	Tutorial: Diffusion Models
2026-11-18	Wed	Imitation Learning & Diffusion Policy		HW4 Due / HW5 Out
2026-11-20	Fri	Reinforcement Learning
2026-11-25	Wed	No class -- Thanksgiving
2026-11-27	Fri	No class -- Thanksgiving
2026-12-02	Wed	World Models / WAMs
2026-12-04	Fri	VLA		HW5 Due
2026-12-09	Wed	Perception in Industry			Final Competition Due