DA5453: Learning from Human Preferences
This course covers the fundamental concepts of learning human preferences from comparisons and choices, with modern applications, primarily LLM alignment. The course has four modules, listed below. Note that the course is mathematically heavy. To have some idea about the flavour of the class, please go through the take-home assignment for module 1, the coding assignment for module 2, and the list of papers suggested for projects. Note that these files may undergo changes as the course proceeds; they have been released in advance for students to preview the course.
Being an elective, students taking this course are expected to a have a research mindset–to go deeper into the concepts covered in class, and to self-study topics that one might be weak in.
Prerequisites: one basic ML course (DA5400 or equivalent); comfort with probability and optimisation (DA5000 or equivalent). Basic familiarity with LLMs and PyTorch would help. Multi-armed bandits, reinforcement learning, and LLM architecture details are not pre-requisites.
The course is appropriate for the following students: B.Tech 3rd year and above (any department), M.Tech 2nd year and above, and all research students (MS/PhD).
Logistics
Slot J · Venue: NAC 2-633 · 9 Credits
| Slot | Day & time | Use |
|---|---|---|
| J1 | Wednesday 2:00–3:15 PM | Lecture |
| J2 | Thursday 3:30–4:45 PM | Lecture |
| J3 | Monday 5:00–5:50 PM | Tutorials, vivas |
First lecture: Wed 29 Jul 2026 · Last lecture: Thu 5 Nov 2026.
Office hours after class (Wed/Thu), in the classroom or my office (NAC 2, 655).
Modules
The course will consist of four modules, taught in the following order.
| Module | Topic | Lectures |
|---|---|---|
| M1 | Choice modeling foundations (Bradley-Terry and Multinomial Logit, estimation and guarantees, limitations) | 8 |
| M2 | LLM alignment–reinforcement learning from human feedback (RLHF) and Direct Preference Optimisation (DPO) | 6 |
| M3 | Online learning from comparisons (active ranking, dueling bandits) | 7 |
| M4 | Learning from implicit preferences (learning from clicks) | 2 |
Assessment
No end-semester exam.
| Component | Marks | When |
|---|---|---|
| Quiz-1 (Module 1) | 20 | Thu 27 Aug, in class |
| Coding Assignment (Module 2) | 20 | Due Fri 2 Oct |
| Quiz-2 (Module 3) | 20 | Thu 15 Oct, in class |
| Project | 40 | Proposal 11 Sep; presentations 28 Oct–5 Nov; report 27 Nov |
Modules 1 and 3 each come with a long, ungraded take-home assignment, released at the start of the module. Quizzes are closed-book, 2 hours, and based largely on these take-home assignments.
The project is to be done in pairs: master a paper from a curated list, teach it to the class in ~15 minutes, then implement the paper’s main idea in code (or dive deeper into the theorems) and write a report. More instructions will be given in class.
Materials
NEW — Module 1 course notes (Lectures 1–6) — PDF. These notes form a consolidated reader on probabilistic choice models, Bradley–Terry estimation, Rank Centrality, and finite-sample recovery, with exercises, hints, and selected model solutions.
Module 1 take-home assignment — PDF (ungraded; use as quiz 1 prep).
Module 2 coding assignment — preview notebook and data + files (zip) (provisional preview; the final assignment will be revised before release).
Module 3 take-home assignment — to be released.
Capstone paper list — suggested papers.
Suggested reading: Machine Learning from Human Preferences, Truong, Haupt and Koyejo (Stanford, 2025).
Note: we will not be following this course too closely.