Publications
Comments on Publication Venues.
The category “Selective ML/CS Conferences (M)” refers to papers published in highly selective machine learning and computer science venues, such as NeurIPS, ICML, ICLR, and AISTATS, typically with acceptance rates below 25%. These tend to be long papers of at least the same quality as typical journal papers in quality and rigor. In control theory, the premier journals (J) are IEEE Transactions on Automatic Control and Automatica, while the main flagship conferences (C) are the IEEE Conference on Decision and Control and the American Control Conference.
♦️ Selected First-Authored Publications
[\(M_2\)] Corruption-Tolerant Optimal Asynchronous Q-Learning
International Conference on Machine Learning, ICML 2026.
Summary
We study the problem of learning the optimal policy in a discounted, infinite-horizon reinforcement learning (RL) setting in the presence of adversarially corrupted rewards. To address this problem, we develop a novel robust variant of the Q-learning algorithm and analyze it under the challenging asynchronous sampling model with time-correlated data. Despite corruption, we prove that the finite-time guarantees of our approach match existing bounds, up to an additive term that scales with the fraction of corrupted samples. We also establish an information-theoretic lower bound, revealing that our guarantees are near-optimal. Notably, our algorithm is agnostic to the underlying reward distribution and provides the first finite-time robustness guarantees for asynchronous Q-learning. A key element of our analysis is a refined Azuma-Hoeffding inequality for almost-martingales, which may have broader applicability in the study of RL algorithms.
[\(M_1\)] Adversarially-Robust TD Learning with Markovian Data: Finite-Time Rates and Fundamental Limits
International Conference on Artificial Intelligence and Statistics, AISTATS 2025.
Summary
One of the most basic problems in reinforcement learning (RL) is policy evaluation: estimating the long-term return, i.e., value function, corresponding to a given fixed policy. The celebrated Temporal Difference (TD) learning algorithm addresses this problem, and recent work has investigated finite-time convergence guarantees for this algorithm and variants thereof. However, these guarantees hinge on the reward observations being always generated from a well-behaved (e.g., sub-Gaussian) true reward distribution. Motivated by harsh, real-world environments where such an idealistic assumption may no longer hold, we revisit the policy evaluation problem from the perspective of adversarial robustness. In particular, we consider a Huber-contaminated reward model where an adversary can arbitrarily corrupt each reward sample with a small probability ϵ. Under this observation model, we first show that the adversary can cause the vanilla TD algorithm to converge to any arbitrary value function. We then develop a novel algorithm called Robust-TD and prove that its finite-time guarantees match that of vanilla TD with linear function approximation up to a small O(ϵ) term that captures the effect of corruption. We complement this result with a minimax lower bound, revealing that such an additive corruption-induced term is unavoidable. To our knowledge, these results are the first of their kind in the context of adversarial robustness of stochastic approximation schemes driven by Markov noise. The key new technical tool that enables our results is an analysis of the Median-of-Means estimator with corrupted, time-correlated data that might be of independent interest to the literature on robust statistics.
[\(C_4\)] Robust Asynchronous Q-Learning under Reward and State Corruption via Batching
IEEE Conference on Decision and Control, CDC 2026.
Summary
Motivated by reinforcement learning in harsh environments, we consider the problem of learning an optimal policy subject to adversarially corrupted feedback. Specifically, at each time-step, an adversary can perturb both the reward and state observations of the learner following the Huber contamination model. To defend against such data corruption, we propose BR-Async-Q: a novel, epoch-based, robust Q-learning algorithm built upon two key ideas: (i) partitioning the online data stream into batches to reduce variance, and (ii) constructing robust estimates of the Bellman optimality operator using such batched data. We prove a high-probability error bound for BR-Async-Q that matches that for vanilla Q-learning, up to a small additive term that scales with the fraction of corrupted samples. To our knowledge, this provides the first robustness guarantee for asynchronous Q-learning subject to both reward and state corruption. Furthermore, when only rewards are corrupted, the dependence of our algorithm’s bound on the corruption fraction is minimax optimal.
[\(C_3\)] Robust Q-Learning under Corrupted Rewards
IEEE Conference on Decision and Control, CDC 2024.
Summary
Recently, there has been a surge of interest in analyzing the non-asymptotic behavior of model-free reinforcement learning algorithms. However, the performance of such algorithms in non-ideal environments - such as in the presence of corrupted rewards - is poorly understood. Motivated by this gap, we investigate the robustness of the celebrated Q-learning algorithm to a strong-contamination attack model, where an adversary can arbitrarily perturb a small fraction of the observed rewards. We start by proving that such an attack can cause the vanilla Q-learning algorithm to incur arbitrarily large errors. We then develop a novel robust synchronous Qlearning algorithm that uses historical reward data to construct robust empirical Bellman operators at each time step. Finally, we prove a finite-time convergence rate for our algorithm that matches known state-of-the-art bounds (in the absence of attacks) up to a small inevitable error term that scales with the adversarial corruption fraction.
[\(C_2\)] Robust Federated Q-Learning with Almost No Communication
American Control Conference, ACC 2026.
Summary
We consider a federated reinforcement learning setting involving M agents, all of whom interact with a common Markov Decision Process (MDP). The agents exchange information via a central server to learn the optimal value function. Our goal is to understand to what extent one can hope for collaborative sample-complexity speedups in such a setting, when a small fraction of the agents are adversarial and can act arbitrarily. To that end, we propose Robust Fed-Q, a federated Q-learning algorithm that blends ideas from both model-based and model-free RL, along with the median-of-means device from robust statistics. We prove that despite corruption, with high-probability, Robust Fed-Q (i) guarantees exact convergence to the optimal value function in the limit of infinite samples, and (ii) enjoys near-optimal finite-time rates that benefit from collaboration. In addition, our approach requires just constant rounds of communication to achieve each of the above guarantees, a feature of independent interest in FL where communication is the major bottleneck.
[\(C_1\)] Variance-Reduced Q-Learning over Static and Time-Varying Networks
American Control Conference, ACC 2026.
Summary
We investigate a decentralized reinforcement learning problem involving multiple agents that interact with the same Markov Decision Process (MDP). The agents can exchange information over a network to collectively learn the optimal state-action value function. For this setting, we introduce a novel epoch-based distributed Q-learning algorithm called VRDQ, where within each epoch, agents locally estimaten the Bellman optimality operator and diffuse information using a consensus-based protocol. For both static and time-varying networks, we establish high-probability finite-time convergence rates for VRDQ that enjoy linear speedups from collaboration. Crucially, we prove that such speedups in sample-complexity require only \(\tilde{O}(1)\) communication, substantially improving upon the communication costs in prior work.
♠️ Accepted Workshop Presentations
- Multi-Agent Robust FRL with Sparse Communication, NCSU Robotics Symposium 2026.
- Agnostic Q-Learning under corrupted Feedback, NeurIPS 2025 Reliable ML Workshop.
- Robust Federated RL with Byzantine Agents, Applied AI Symposium 2025, NESCW 2026.
- Theoretical Limits of Robust TD Learning, New York RL Workshop, NYRL 2025, Amazon.
- Towards Finite-Time Theory for Adversarially-Robust RL, NESCW 2025.
- Adversarially-Robust Deep Q-Network for Algorithmic Trading, MLSS 2025, NCSU.
- Robust Algorithms for Adversarial Reinforcement Learning, Applied AI Symposium 2024.
♣️ Invited Talks / Seminars
- Finite-Time Rates for Adversarially-Robust Reinforcement Learning, CORAL Seminar, North Carolina State University.