Bi-Level Reinforcement Learning and LLM Alignment
Principal Investigator: Vaneet Aggarwal
Large language models (LLMs) are tuned to behave the way people want through "alignment". Usually a model is trained to score answers by human preference, and the LLM is then trained to produce high-scoring answers, for example by reinforcement learning from human feedback (RLHF). This is a nested, or "bilevel", learning problem: one learning loop sits inside another. In practice it is often done with heuristics and little understanding of when it works.
This project puts alignment on a firmer footing. Our outcomes include:
- Bounds on how much data nested (bilevel) RL needs, together with faster algorithms that meet them.
- Alignment methods that still work when human or AI feedback is noisy or imperfect, including methods robust to shifts in the preference data.
- Proofs of when "self-improving" alignment, where a model learns from its own outputs, actually converges.
- BalancedDPO, which aligns a model to several quality measures at once, and Align-Pro, a principled way to optimize prompts instead of retraining the model.
- More honest evaluation: a statistical correction for the "winner's curse" (the best-looking model on a benchmark is usually overrated) and stress tests for interactive AI agents.
A survey of RL techniques for LLMs (ACM TIST) gives an overview of the field.
Representative Publications
Prashant Trivedi, Souradip Chakraborty, Avinash Reddy, Vaneet Aggarwal, Amrit Singh Bedi, and George K. Atia, "Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment," in Proc. AAAI, Feb-Mar, 2025.
Mudit Gaur, Utsav Singh, Amrit Singh Bedi, Raghu Pasupathy, and Vaneet Aggarwal, "On The Sample Complexity Bounds In Bilevel Reinforcement Learning," in Proc. NeurIPS, Dec 2025.
Ruihan Li and Vaneet Aggarwal, "Preference-Aligned Ride-Sharing Repositioning via a Two-Stage Bilevel RLHF Framework," Electronics, Feb 2026.
Dipesh Tamboli, Souradip Chakraborty, Aditya Malusare, Biplab Banerjee, Amrit Singh Bedi, and Vaneet Aggarwal, "BalancedDPO: Adaptive Multi-Metric Alignment," Transactions on Machine Learning Research, Apr 2026.
Naman Saxena, Mudit Gaur, Vaneet Aggarwal, "Hypergradient-based Bilevel Reinforcement Learning with Improved Sample Complexity," arXiv, Jul 2026.
Yang Xu, Chenang Li, Jiefu Zhang, Haixiang Sun, Zhou Li, and Vaneet Aggarwal, "Selection-Aware Stress Testing for Interactive Agents," arXiv, Aug 2026.
Mudit Gaur, Naman Saxena, and Vaneet Aggarwal, "Improved Sample Complexity Analysis For First Order Bilevel Reinforcement Learning," arXiv, Oct 2026.
Xudong Wu, Pangpang Liu, Vaneet Aggarwal, and Jiayu Chen, "On the Convergence of Self-Improving Online LLM Alignment, " in Proc. UAI, Aug 2026.
Zimeng Li, Mudit Gaur, and Vaneet Aggarwal, "Oracle-Robust Online Alignment for Large Language Models," in Proc. NeurIPS, Dec 2026.
Yang Xu, Jiefu Zhang, Haixiang Sun, Zihan Zhou, Tianyu Cao, and Vaneet Aggarwal, "Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking," in Proc. NeurIPS, Dec 2026.
Xudong Wu, Jian Qian, Pangpang Liu, Vaneet Aggarwal, and Jiayu Chen, "Distributionally Robust Listwise Preference Optimization," in Proc. NeurIPS, Dec 2026.
Saksham Sahai Srivastava and Vaneet Aggarwal, "A Technical Survey of Reinforcement Learning Techniques for Large Language Models," Accepted to ACM Transactions on Intelligent Systems and Technology, Jun 2026.
Keywords: AI evaluation, bilevel optimization, LLM alignment, preference optimization, RLHF, robustness, trustworthy AI

