DA5453: Suggested Papers for the Capstone Project

Back to the course webpage

This is a broad catalogue of papers related to the course. You are not expected to read all of them. For the capstone project, each pair of students will normally select one paper, understand its main ideas and results, present it to the class, and carry out a computational study based on it. A project centred on a theoretical paper must still contain an implementation of an estimator or algorithm, at least on synthetic data.

The labels below are meant to help you begin:

• Highly recommended papers are particularly close to the course and lend themselves to a well-scoped project.
• Recommended papers are also strong project choices.
• Course background papers are substantially covered in class. They are included for reference, but will not normally be assigned as standalone project papers.
• Companion reading is useful in support of another paper, but is usually too broad, too specialised, or not sufficiently self-contained for a project by itself.

Unmarked papers are also possible choices, but their scope should be discussed with the instructor. You may propose a paper outside this list, subject to approval. In all cases, the precise project scope will be fixed separately.

Last updated: 25 July 2026.

1. Estimation and ranking from comparisons

• Rank Centrality: Ranking from Pairwise Comparisons
  Sahand Negahban, Sewoong Oh and Devavrat Shah. Operations Research, 2017.
  Course background.

• Fast and Accurate Inference of Plackett–Luce Models
  Lucas Maystre and Matthias Grossglauser. NeurIPS, 2015.
  Highly recommended. The paper extends the spectral viewpoint from pairwise comparisons to Plackett–Luce data and develops LSR and iterative LSR. A suitable project can implement these algorithms and compare their accuracy and convergence with Rank Centrality and likelihood-based methods.

• Simple, Robust and Optimal Ranking from Pairwise Comparisons
  Nihar B. Shah and Martin J. Wainwright. JMLR, 2018.
  Recommended. This paper gives a particularly clean example in which a simple Borda-type procedure is both robust to model misspecification and statistically optimal. One can compare Borda, BTL maximum likelihood and Rank Centrality under several synthetic comparison models.

• Stochastically Transitive Models for Pairwise Comparisons: Statistical and Computational Issues
  Nihar B. Shah, Sivaraman Balakrishnan, Adityanand Guntuboyina and Martin J. Wainwright. ICML, 2016; IEEE Transactions on Information Theory, 2017.
  Recommended. The paper replaces the parametric BTL assumption by stochastic transitivity and studies the resulting statistical–computational trade-off. A project can implement the tractable estimators and examine when their additional flexibility is useful.

• Efficient Computation of Rankings from Pairwise Comparisons
  M. E. J. Newman. JMLR, 2023.
  Recommended. The proposed rearrangement of the classical Zermelo iteration is simple to implement and can be much faster. This makes possible a focused project on convergence, numerical stability and running-time comparisons.

• Accelerated Spectral Ranking
  Arpit Agarwal, Prathamesh Patil and Shivani Agarwal. ICML, 2018.

• Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence
  Nihar B. Shah, Sivaraman Balakrishnan, Joseph Bradley, Abhay Parekh, Kannan Ramchandran and Martin J. Wainwright. JMLR, 2016.

• A Statistical Convergence Perspective of Algorithms for Rank Aggregation from Pairwise Data
  Arun Rajkumar and Shivani Agarwal. ICML, 2014.

2. Beyond BTL and MNL: heterogeneity, context and intransitivity

• Learning a Mixture of Two Multinomial Logits
  Flavio Chierichetti, Ravi Kumar and Andrew Tomkins. ICML, 2018.
  Recommended. Mixtures provide a natural way to represent heterogeneous populations while retaining a clear probabilistic model. The learning algorithm can be implemented and tested on synthetic mixtures, including regimes in which a single MNL model fails.

• Pairwise Choice Markov Chains
  Stephen Ragain and Johan Ugander. NeurIPS, 2016.
  Recommended. PCMC replaces the fixed MNL weights by a Markov-chain model of choice and thereby permits several violations of IIA. A project can fit MNL and PCMC models to the same data and study predictive fit, regularity and computation.

• Modeling Intransitivity in Matchup and Comparison Data
  Shuo Chen and Thorsten Joachims. WSDM, 2016.
  Recommended. The blade–chest model represents cyclic effects that a one-dimensional utility cannot capture. It supports a concrete project using synthetic rock–paper–scissors structures and one of the public matchup datasets studied in the paper.

• Discovering Context Effects from Raw Choice Data
  Arjun Seshadri, Alexander Peysakhovich and Johan Ugander. ICML, 2019.
  Highly recommended. The context-dependent random-utility model is a direct and elegant extension of the MNL model studied in Module 1. A project can implement its likelihood, compare it with MNL, and investigate which context effects can be recovered from finite data.

• Learning Interpretable Feature Context Effects in Discrete Choice
  Kiran Tomlinson and Austin R. Benson. KDD, 2021.
  Highly recommended. The linear context logit model uses observable features to obtain interpretable context effects. It is a natural follow-up to the Module 1 discussion and admits both synthetic experiments and studies on the datasets used by the authors.

• Choice Set Confounding in Discrete Choice
  Kiran Tomlinson, Johan Ugander and Austin R. Benson. KDD, 2021.
  Recommended. This paper asks what happens when the set of alternatives offered to a user is itself preference-dependent. A project can construct a confounded data-generating process and compare naive estimation with the proposed causal corrections.

3. Personalised preferences, implicit feedback and learning to rank

• Preference Completion: Large-scale Collaborative Ranking from Pairwise Comparisons
  Dohyung Park, Joe Neeman, Jin Zhang, Sujay Sanghavi and Inderjit S. Dhillon. ICML, 2015.
  Recommended. AltSVM gives a scalable non-convex method for learning a low-rank user–item score matrix from pairwise preferences. It can be compared with BPR and ordinary matrix factorisation on MovieLens-derived comparisons.

• Collaboratively Learning Preferences from Ordinal Data
  Sewoong Oh, Kiran K. Thekumparampil and Jiaming Xu. NeurIPS, 2015.

• Learning from Comparisons and Choices
  Sahand Negahban, Sewoong Oh, Kiran K. Thekumparampil and Jiaming Xu. JMLR, 2018.
  Companion reading. This is a comprehensive treatment of the low-rank preference-learning framework and is best used to support a more narrowly scoped paper.

• BPR: Bayesian Personalized Ranking from Implicit Feedback
  Steffen Rendle, Christoph Freudenthaler, Zeno Gantner and Lars Schmidt-Thieme. UAI, 2009.
  Course background.

• Optimizing Search Engines using Clickthrough Data
  Thorsten Joachims. KDD, 2002.
  Course background.

• Accurately Interpreting Clickthrough Data as Implicit Feedback
  Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke and Geri Gay. SIGIR, 2005.
  Course background.

• Unbiased Learning-to-Rank with Biased Feedback
  Thorsten Joachims, Adith Swaminathan and Tobias Schnabel. WSDM, 2017.
  Course background.

• Position Bias Estimation for Unbiased Learning to Rank in Personal Search
  Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler and Marc Najork. WSDM, 2018.
  Recommended. The regression-EM method estimates position propensities without requiring fully randomised rankings. A project can simulate a click model, recover the propensities and measure the downstream effect on an IPS learning-to-rank estimator.

• Recommendations as Treatments: Debiasing Learning and Evaluation
  Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak and Thorsten Joachims. ICML, 2016.
  Recommended. This is a clean bridge from inverse-propensity weighting to matrix factorisation. The proposed estimator can be studied on semi-synthetic data where exposure propensities and the true prediction risk are known.

• Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feedback
  Yuta Saito, Suguru Yaginuma, Yuta Nishino, Hayato Sakata and Kazuhide Nakata. WSDM, 2020.
  Recommended. The paper derives unbiased and clipped estimators for recommendation from non-randomly missing feedback. It offers a manageable bias–variance study using synthetic exposure and relevance models.

4. Ordinal and triplet embedding

• Adaptively Learning the Crowd Kernel
  Omer Tamuz, Ce Liu, Serge Belongie, Ohad Shamir and Adam Tauman Kalai. ICML, 2011.
  Recommended. This paper combines triplet-based similarity judgements, kernel learning and adaptive query selection. A project can implement the basic estimator and compare adaptive and random triplet collection.

• Stochastic Triplet Embedding
  Laurens van der Maaten and Kilian Q. Weinberger. MLSP, 2012.
  Recommended. STE and t-STE give a direct probabilistic route from triplet comparisons to a visual embedding. The methods are straightforward to implement and permit clear comparisons with standard triplet-loss baselines.

• Local Ordinal Embedding
  Yoshikazu Terada and Ulrike von Luxburg. ICML, 2014.

• Finite Sample Prediction and Recovery Bounds for Ordinal Embedding
  Lalit Jain, Kevin Jamieson and Robert Nowak. NeurIPS, 2016.
  Recommended. The paper connects noisy triplets, low-rank distance matrices and finite-sample recovery, while also proposing projected-gradient algorithms. A project can reproduce its synthetic recovery experiments and examine the effect of dimension, noise and triplet sampling.

• Cost-Effective HITs for Relative Similarity Comparisons
  Michael J. Wilber, Iljung S. Kwak and Serge J. Belongie. HCOMP, 2014.
  Companion reading and dataset.

5. Preference-based alignment: foundations and training objectives

• Deep Reinforcement Learning from Human Preferences
  Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg and Dario Amodei. NeurIPS, 2017.
  Course background.

• Training Language Models to Follow Instructions with Human Feedback
  Long Ouyang et al. NeurIPS, 2022.
  Course background.

• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon and Chelsea Finn. NeurIPS, 2023.
  Course background.

• Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons
  Banghua Zhu, Michael I. Jordan and Jiantao Jiao. ICML, 2023.
  Recommended. The paper joins the BTL/Plackett–Luce estimation problem to offline policy optimisation through pessimism. Its key ideas can be studied in a finite-action simulator without training a large language model.

• A General Theoretical Paradigm to Understand Learning from Human Preferences
  Mohammad Gheshlaghi Azar et al. AISTATS, 2024.
  Highly recommended. The paper places RLHF, DPO and related objectives in a common framework and motivates Identity Preference Optimisation (IPO). A project can derive and implement the objectives in a tabular or small-model setting and compare their behaviour under noisy preferences.

• Model Alignment as Prospect Theoretic Optimization
  Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky and Douwe Kiela. ICML, 2024.
  Recommended. KTO replaces paired comparisons by desirable and undesirable examples and connects the objective to prospect theory. A scaled-down project can compare KTO and DPO under controlled pairing and label-noise conditions.

• Provably Robust DPO: Aligning Language Models with Noisy Feedback
  Sayak Ray Chowdhury, Anush Kini and Nagarajan Natarajan. ICML, 2024.
  Recommended. The proposed correction gives a precise way to study random preference-label flips. It can be implemented with a small policy model and evaluated while varying the noise rate and its misspecification.

• SimPO: Simple Preference Optimization with a Reference-Free Reward
  Yu Meng, Mengzhou Xia and Danqi Chen. NeurIPS, 2024.

6. Alignment diagnostics, data, inference and personalisation

• Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization
  Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora and Boris Hanin. ICLR, 2025.
  Highly recommended. The paper isolates a surprising training-dynamics failure of DPO and provides code and diagnostics. This supports a compute-conscious project based on reproducing likelihood displacement and testing the proposed CHES score. (Code)

• Preference Learning Algorithms Do Not Learn Preference Rankings
  Angelica Chen et al. NeurIPS, 2024.
  Recommended. The paper separates pairwise training accuracy from recovery of an entire preference ranking. Much of the project can be inference-only: evaluate several learned or synthetic reward functions under ordinary and ranking-aware metrics.

• Theoretical Guarantees on the Best-of-n Alignment Policy
  Ahmad Beirami et al. ICML, 2025.

• Scaling Laws for Reward Model Overoptimization
  Leo Gao, John Schulman and Jacob Hilton. ICML, 2023.
  Recommended. The paper gives an empirical account of Goodhart-like behaviour when a proxy reward is optimised too strongly. A scaled-down project can reproduce the phenomenon with synthetic gold and proxy reward models.

• FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
  Rohan Deb et al. ICML, 2025.

• LoRe: Personalizing LLMs via Low-Rank Reward Modeling
  Avinandan Bose, Zhihan Xiong, Yuejie Chi, Simon S. Du, Lin Xiao and Maryam Fazel. COLM, 2025.
  Recommended. LoRe treats variation across users as low-rank structure in the reward matrix, making a useful bridge to collaborative preference learning. Its frozen-embedding implementation permits experiments without full LLM fine-tuning. (Code)

• Language Model Personalization via Reward Factorization
  Idan Shenfeld, Felix Faltings, Pulkit Agrawal and Aldo Pacchiano. COLM, 2025.
  Companion reading for a project on low-rank personalised rewards.

• Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF
  Anand Siththaranjan, Cassidy Laidlaw and Dylan Hadfield-Menell. ICLR, 2024.
  Recommended. The paper treats annotator disagreement as information about hidden context rather than as independent noise. A project can construct a heterogeneous annotator population and compare scalar aggregation with the proposed distributional model.

• RewardBench: Evaluating Reward Models for Language Modeling
  Nathan Lambert et al. Findings of NAACL, 2025.
  Highly recommended. RewardBench turns reward-model evaluation into a modular and reproducible problem covering chat, reasoning and safety. A project can evaluate small open reward models, add a controlled stress-test subset, and analyse failure patterns. (Code)

• AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
  Yann Dubois et al. NeurIPS, 2023.
  Companion reading and software framework. (Code)

7. Active ranking and dueling bandits

• The K-armed Dueling Bandits Problem
  Yisong Yue, Josef Broder, Robert Kleinberg and Thorsten Joachims. JCSS, 2012; conference version at COLT, 2009.
  Recommended. This foundational paper develops Interleaved Filter, which is not covered in detail in class. It provides a natural baseline implementation and a starting point for comparing later dueling-bandit algorithms.

• Relative Upper Confidence Bound for the K-Armed Dueling Bandit Problem
  Masrour Zoghi, Shimon Whiteson, Rémi Munos and Maarten de Rijke. ICML, 2014.
  Course background.

• Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problem
  Junpei Komiyama, Junya Honda, Hisashi Kashima and Hiroshi Nakagawa. COLT, 2015.

• Double Thompson Sampling for Dueling Bandits
  Huasen Wu and Xin Liu. NeurIPS, 2016.
  Recommended. Double Thompson Sampling has a simple posterior-sampling implementation and works for both Condorcet and Copeland settings. It can be compared directly with RUCB on synthetic and public preference matrices.

• Copeland Dueling Bandits
  Masrour Zoghi, Zohar Karnin, Shimon Whiteson and Maarten de Rijke. NeurIPS, 2015.
  Recommended. This paper removes the assumption that a Condorcet winner exists. A project can generate cyclic preference matrices and compare Condorcet-based, Copeland-based and Thompson-sampling approaches.

• Reducing Dueling Bandits to Cardinal Bandits
  Nir Ailon, Zohar Karnin and Thorsten Joachims. ICML, 2014.

• Active Ranking using Pairwise Comparisons
  Kevin G. Jamieson and Robert Nowak. NeurIPS, 2011.
  Recommended. The paper exploits a low-dimensional geometric structure to select informative comparisons. A project can implement the noiseless and robust procedures and measure the gain over random querying as the dimension changes.

• Just Sort It! A Simple and Effective Approach to Active Preference Learning
  Lucas Maystre and Matthias Grossglauser. ICML, 2017.
  Highly recommended. The central idea—repeated noisy sorting—is simple, surprising and very easy to implement, while still admitting useful theory. A project can compare repeated Quicksort with random sampling and more elaborate active-ranking rules under BTL and misspecified models.

• Active Ranking from Pairwise Comparisons and When Parametric Assumptions Do Not Help
  Reinhard Heckel, Nihar B. Shah, Kannan Ramchandran and Martin J. Wainwright. Annals of Statistics, 2019.

• Maximum Selection and Ranking under Noisy Comparisons
  Moein Falahatgar, Alon Orlitsky, Venkatadheeraj Pichapati and Ananda Theertha Suresh. ICML, 2017.
  Course background.

8. Contextual, multiway and assortment bandits

• Thompson Sampling for the MNL-Bandit
  Shipra Agrawal, Vashist Avadhanula, Vineet Goyal and Assaf Zeevi. COLT, 2017.
  Recommended. This paper complements the UCB method discussed in class with a posterior-sampling algorithm for dynamic assortment selection. A project can implement both approaches and compare their regret across assortment sizes and parameter regimes.

• MNL-Bandit: A Dynamic Learning Approach to Assortment Selection
  Shipra Agrawal, Vashist Avadhanula, Vineet Goyal and Assaf Zeevi. Operations Research, 2019.
  Course background.

• Thompson Sampling for Multinomial Logit Contextual Bandits
  Min-hwan Oh and Garud Iyengar. NeurIPS, 2019.

• Contextual Dueling Bandits
  Miroslav Dudík, Katja Hofmann, Robert E. Schapire, Aleksandrs Slivkins and Masrour Zoghi. COLT, 2015.
  Recommended. The von Neumann winner provides a game-theoretic alternative to assuming a Condorcet winner. A project can implement the finite-policy version and compare the two solution concepts on contextual preference matrices.

• Optimal Algorithms for Stochastic Contextual Preference Bandits
  Aadirupa Saha. NeurIPS, 2021.

• PAC Battling Bandits in the Plackett–Luce Model
  Aadirupa Saha and Aditya Gopalan. ALT, 2019.

• Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Models
  Viktor Bengs, Aadirupa Saha and Eyke Hüllermeier. ICML, 2022.
  Recommended. CoLSTIM connects the linear/logistic viewpoint of Module 3 with contextual duels through perturbed utility estimates. Its synthetic experiments can be reproduced and extended to test model misspecification.

• Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizability
  Aadirupa Saha and Akshay Krishnamurthy. ALT, 2022.

• Battle of Bandits
  Aadirupa Saha and Aditya Gopalan. UAI, 2018.