The Center for Education and Research in Information Assurance and Security (CERIAS)

The Center for Education and Research in
Information Assurance and Security (CERIAS)

Bi-Level Reinforcement Learning and LLM Alignment

Principal Investigator: Vaneet Aggarwal

Large language models (LLMs) are tuned to behave the way people want through "alignment". Usually a model is trained to score answers by human preference, and the LLM is then trained to produce high-scoring answers, for example by reinforcement learning from human feedback (RLHF). This is a nested, or "bilevel", learning problem: one learning loop sits inside another. In practice it is often done with heuristics and little understanding of when it works.

This project puts alignment on a firmer footing. Our outcomes include:

  • Bounds on how much data nested (bilevel) RL needs, together with faster algorithms that meet them.
  • Alignment methods that still work when human or AI feedback is noisy or imperfect, including methods robust to shifts in the preference data.
  • Proofs of when "self-improving" alignment, where a model learns from its own outputs, actually converges.
  • BalancedDPO, which aligns a model to several quality measures at once, and Align-Pro, a principled way to optimize prompts instead of retraining the model.
  • More honest evaluation: a statistical correction for the "winner's curse" (the best-looking model on a benchmark is usually overrated) and stress tests for interactive AI agents.

A survey of RL techniques for LLMs (ACM TIST) gives an overview of the field.

Representative Publications

  • Prashant Trivedi, Souradip Chakraborty, Avinash Reddy, Vaneet Aggarwal, Amrit Singh Bedi, and George K. Atia, "Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment," in Proc. AAAI, Feb-Mar, 2025.

  • Mudit Gaur, Utsav Singh, Amrit Singh Bedi, Raghu Pasupathy, and Vaneet Aggarwal, "On The Sample Complexity Bounds In Bilevel Reinforcement Learning," in Proc. NeurIPS, Dec 2025.

  • Ruihan Li and Vaneet Aggarwal, "Preference-Aligned Ride-Sharing Repositioning via a Two-Stage Bilevel RLHF Framework," Electronics, Feb 2026.

  • Dipesh Tamboli, Souradip Chakraborty, Aditya Malusare, Biplab Banerjee, Amrit Singh Bedi, and Vaneet Aggarwal, "BalancedDPO: Adaptive Multi-Metric Alignment," Transactions on Machine Learning Research, Apr 2026.

  • Naman Saxena, Mudit Gaur, Vaneet Aggarwal, "Hypergradient-based Bilevel Reinforcement Learning with Improved Sample Complexity," arXiv, Jul 2026.

  • Yang Xu, Chenang Li, Jiefu Zhang, Haixiang Sun, Zhou Li, and Vaneet Aggarwal, "Selection-Aware Stress Testing for Interactive Agents," arXiv, Aug 2026.

  • Mudit Gaur, Naman Saxena, and Vaneet Aggarwal, "Improved Sample Complexity Analysis For First Order Bilevel Reinforcement Learning," arXiv, Oct 2026.

  • Xudong Wu, Pangpang Liu, Vaneet Aggarwal, and Jiayu Chen, "On the Convergence of Self-Improving Online LLM Alignment, " in Proc. UAI, Aug 2026.

  • Zimeng Li, Mudit Gaur, and Vaneet Aggarwal, "Oracle-Robust Online Alignment for Large Language Models," in Proc. NeurIPS, Dec 2026.

  • Yang Xu, Jiefu Zhang, Haixiang Sun, Zihan Zhou, Tianyu Cao, and Vaneet Aggarwal, "Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking," in Proc. NeurIPS, Dec 2026.

  • Xudong Wu, Jian Qian, Pangpang Liu, Vaneet Aggarwal, and Jiayu Chen, "Distributionally Robust Listwise Preference Optimization," in Proc. NeurIPS, Dec 2026.

  • Saksham Sahai Srivastava and Vaneet Aggarwal, "A Technical Survey of Reinforcement Learning Techniques for Large Language Models," Accepted to ACM Transactions on Intelligent Systems and Technology, Jun 2026.

Keywords: AI evaluation, bilevel optimization, LLM alignment, preference optimization, RLHF, robustness, trustworthy AI