Publications
Comments on Publication Venues.
The category βSelective ML/CS Conferences [\(C^{*}\)]β refers to papers published in highly selective machine learning and computer science venues, such as NeurIPS, ICML, ICLR, and AISTATS, typically with acceptance rates below 25%. These tend to be long papers of at least the same quality as typical journal papers in quality and rigor. [\(C\)] denotes flagship control conferences, such as the IEEE Conference on Decision and Control (CDC), the American Control Conference (ACC), Learning for Dynamics and Control (L4DC), and the European Control Conference (ECC), while [\(J\)] denotes journals, such as IEEE Transactions on Automatic Control (TAC), Automatica, the Journal of Machine Learning Research (JMLR), Transactions on Machine Learning Research (TMLR), and IEEE Control Systems Letters (L-CSS).
β¦οΈPeer-Reviewed Conference Publications
[\(C_{12}^{*}\)]
Sreejeet Maity and Aritra Mitra,
βCorruption-Tolerant Optimal Asynchronous Q-Learningβ, in
International Conference on Machine Learning (ICML), 2026.
Summary
We study the problem of learning the optimal policy in a discounted, infinite-horizon reinforcement learning (RL) setting in the presence of adversarially corrupted rewards. To address this problem, we develop a novel robust variant of the Q-learning algorithm and analyze it under the challenging asynchronous sampling model with time-correlated data. Despite corruption, we prove that the finite-time guarantees of our approach match existing bounds, up to an additive term that scales with the fraction of corrupted samples. We also establish an information-theoretic lower bound, revealing that our guarantees are near-optimal. Notably, our algorithm is agnostic to the underlying reward distribution and provides the first finite-time robustness guarantees for asynchronous Q-learning. A key element of our analysis is a refined Azuma-Hoeffding inequality for almost-martingales, which may have broader applicability in the study of RL algorithms.
[\(C_{11}^{*}\)]
Sreejeet Maity and Aritra Mitra,
βAdversarially-Robust TD Learning with Markovian Data: Finite-Time Rates and Fundamental Limitsβ, in
International Conference on Artificial Intelligence and Statistics (AISTATS), 2025.
Summary
One of the most basic problems in reinforcement learning (RL) is policy evaluation: estimating the long-term return, i.e., value function, corresponding to a given fixed policy. The celebrated Temporal Difference (TD) learning algorithm addresses this problem, and recent work has investigated finite-time convergence guarantees for this algorithm and variants thereof. However, these guarantees hinge on the reward observations being always generated from a well-behaved (e.g., sub-Gaussian) true reward distribution. Motivated by harsh, real-world environments where such an idealistic assumption may no longer hold, we revisit the policy evaluation problem from the perspective of adversarial robustness. In particular, we consider a Huber-contaminated reward model where an adversary can arbitrarily corrupt each reward sample with a small probability Ο΅. Under this observation model, we first show that the adversary can cause the vanilla TD algorithm to converge to any arbitrary value function. We then develop a novel algorithm called Robust-TD and prove that its finite-time guarantees match that of vanilla TD with linear function approximation up to a small O(Ο΅) term that captures the effect of corruption. We complement this result with a minimax lower bound, revealing that such an additive corruption-induced term is unavoidable. To our knowledge, these results are the first of their kind in the context of adversarial robustness of stochastic approximation schemes driven by Markov noise. The key new technical tool that enables our results is an analysis of the Median-of-Means estimator with corrupted, time-correlated data that might be of independent interest to the literature on robust statistics.
[\(C_{10}\)]
Sreejeet Maity and Aritra Mitra,
βRobust Asynchronous Q-Learning under Reward and State Corruption via Batchingβ, in
IEEE Conference on Decision and Control (CDC), 2026.
Summary
Motivated by reinforcement learning in harsh environments, we consider the problem of learning an optimal policy subject to adversarially corrupted feedback. Specifically, at each time-step, an adversary can perturb both the reward and state observations of the learner following the Huber contamination model. To defend against such data corruption, we propose BR-Async-Q: a novel, epoch-based, robust Q-learning algorithm built upon two key ideas: (i) partitioning the online data stream into batches to reduce variance, and (ii) constructing robust estimates of the Bellman optimality operator using such batched data. We prove a high-probability error bound for BR-Async-Q that matches that for vanilla Q-learning, up to a small additive term that scales with the fraction of corrupted samples. To our knowledge, this provides the first robustness guarantee for asynchronous Q-learning subject to both reward and state corruption. Furthermore, when only rewards are corrupted, the dependence of our algorithmβs bound on the corruption fraction is minimax optimal.
[\(C_9\)]
Sreejeet Maity and Aritra Mitra,
βRobust Q-Learning under Corrupted Rewardsβ, in
IEEE Conference on Decision and Control (CDC), 2024.
Summary
Recently, there has been a surge of interest in analyzing the non-asymptotic behavior of model-free reinforcement learning algorithms. However, the performance of such algorithms in non-ideal environments - such as in the presence of corrupted rewards - is poorly understood. Motivated by this gap, we investigate the robustness of the celebrated Q-learning algorithm to a strong-contamination attack model, where an adversary can arbitrarily perturb a small fraction of the observed rewards. We start by proving that such an attack can cause the vanilla Q-learning algorithm to incur arbitrarily large errors. We then develop a novel robust synchronous Q-learning algorithm that uses historical reward data to construct robust empirical Bellman operators at each time step. Finally, we prove a finite-time convergence rate for our algorithm that matches known state-of-the-art bounds (in the absence of attacks) up to a small inevitable error term that scales with the adversarial corruption fraction.
[\(C_8\)]
Sreejeet Maity and Aritra Mitra,
βRobust Federated Q-Learning with Almost No Communicationβ, in
American Control Conference (ACC), 2026.
Summary
We consider a federated reinforcement learning setting involving M agents, all of whom interact with a common Markov Decision Process (MDP). The agents exchange information via a central server to learn the optimal value function. Our goal is to understand to what extent one can hope for collaborative sample-complexity speedups in such a setting, when a small fraction of the agents are adversarial and can act arbitrarily. To that end, we propose Robust Fed-Q, a federated Q-learning algorithm that blends ideas from both model-based and model-free RL, along with the median-of-means device from robust statistics. We prove that despite corruption, with high-probability, Robust Fed-Q (i) guarantees exact convergence to the optimal value function in the limit of infinite samples, and (ii) enjoys near-optimal finite-time rates that benefit from collaboration. In addition, our approach requires just constant rounds of communication to achieve each of the above guarantees, a feature of independent interest in FL where communication is the major bottleneck.
[\(C_7\)]
Sreejeet Maityβ , Feng Zhu, Robert Heath, and Aritra Mitra,
βVariance-Reduced Q-Learning over Static and Time-Varying Networksβ, in
American Control Conference (ACC), 2026.
Summary
We investigate a decentralized reinforcement learning problem involving multiple agents that interact with the same Markov Decision Process (MDP). The agents can exchange information over a network to collectively learn the optimal state-action value function. For this setting, we introduce a novel epoch-based distributed Q-learning algorithm called VRDQ, where within each epoch, agents locally estimate the Bellman optimality operator and diffuse information using a consensus-based protocol. For both static and time-varying networks, we establish high-probability finite-time convergence rates for VRDQ that enjoy linear speedups from collaboration. Crucially, we prove that such speedups in sample-complexity require only \(\tilde{O}(1)\) communication, substantially improving upon the communication costs in prior work.
[\(C_6\)]
Sreejeet Maity, Aritra Mitra,
βTowards Finite-Time Robust Policy Evaluation under Complete Feedback Corruption: Near-Optimality via Batchingβ,
Under Review at Learning for Dynamics and Control (L4DC), 2027.
Summary
[\(C_5\)]
Sreejeet Maity, Aritra Mitra,
βAdaptive Quantization Preserves Fast Convergence in Temporal-Difference Learningβ,
Under Review at European Control Conference (ECC), 2027.
Summary
β¦οΈSelected Journals
[\(J_4\)]
Sreejeet Maity and Aritra Mitra,
βLearning from Unreliable Trajectories: Adversarially-Robust Federated Q-Learningβ,
Under Review at Journal of Machine Learning Research (JMLR).
Summary
We study federated reinforcement learning in which multiple agents interact with a common Markov decision process and communicate through a central server to collaboratively learn the optimal state-action value function. Our goal is to understand whether the sample-efficiency benefits of collaboration can be retained when a fraction of the agents behave adversarially and transmit arbitrarily corrupted
information. To address this problem, we introduce Robust Async-Fed-Q, an epoch-based federated learning algorithm that combines variance-reduced estimation of the Bellman optimality operator at the agents with robust aggregation at the server. We establish high-probability finite-time guarantees showing that the proposed method preserves the statistical gains of collaboration among the honest agents while tolerating adversarial corruption. In particular, the effect of the adversarial agents decreases as the amount of data collected by each honest agent grows and eventually vanishes in the infinite-sample
limit. We complement these guarantees with information-theoretic lower bounds that characterize the unavoidable statistical cost of adversarial corruption, leading to the first nearly matching upper and lower bounds for adversarially robust federated reinforcement learning. We further extend our framework to accommodate single-trajectory Markovian sampling and heterogeneous partial coverage, where different agents may explore different regions of the state-action space and learning relies on their collective coverage. Finally, our epoch-based design substantially improves the best known communication complexity for federated Q-learning under asynchronous sampling.
[\(J_3\)]
Sreejeet Maityβ , Feng Zhu, Robert Heath, Aritra Mitra,
βDecentralized Q-learning with Asynchronous Sampling and Partial Coverageβ,
Manuscript in preparation for IEEE Transactions in Signal Processing (TSP).
Summary
We investigate a decentralized reinforcement learning problem where multiple agents interact with the same Markov decision process (MDP). The goal is to collectively learn the optimal state-action value function by exchanging information over a network. Motivated by practical considerations, we consider an asynchronous sampling model, where each agent observes only a single state-action pair at every time-step. For this setting, we introduce a novel epoch-based decentralized Q-learning algorithm named AVRDQ, in which agents locally estimate the Bellman optimality operator and diffuse information using a consensus-based protocol. We further extend our framework to partial state-action coverage, where different agents may observe different subsets of the stateaction space. In both settings, we establish high-probability finite-time convergence guarantees for AVRDQ that exhibit linear collaborative speedups. To the best of our knowledge, this is the first work to simultaneously achieve linear collaborative speedups and poly-logarithmic communication complexity for decentralized Q-learning under asynchronous sampling, with guarantees further extending to partial state-action coverage.
[\(J_2\)]
Sreejeet Maity, Leonardo Felipe Toso, James Anderson, Aritra Mitra,
βValue Function Representation for Adversarially Robust Federated TD Learningβ,
Under Review at Transactions on Machine Learning Research (TMLR).
Summary
[\(J_1\)]
Sreejeet Maity, Aritra Mitra,
βTowards Minimax-Optimal Asynchronous Q-Learning from Corrupted Markovian Dataβ,
Under Review at Automatica.
Summary
β οΈ Workshop and Symposium Presentations
[\(W_8\)]
Learning from Unreliable Data: Promise and Fundamental Limits,
Applied AI Symposium 2026.
[\(W_7\)]
Multi-Agent Robust FRL with Sparse Communication,
NCSU Robotics Symposium 2026.
[\(W_6\)]
Agnostic Q-Learning under corrupted Feedback,
NeurIPS 2025 Reliable ML Workshop.
[\(W_4\)]
Theoretical Limits of Robust TD Learning,
New York RL Workshop, NYRL 2025, Amazon.
[\(W_3\)]
Towards Finite-Time Theory for Adversarially-Robust RL,
NESCW 2025.
[\(W_2\)]
Adversarially-Robust Deep Q-Network for Algorithmic Trading,
MLSS 2025, NCSU.
[\(W_1\)]
Robust Algorithms for Adversarial Reinforcement Learning,
Applied AI Symposium 2024.
β£οΈ Selected Invited Talks or Seminars
[\(S_3\)]
Finite-Time Rates for Adversarially-Robust Reinforcement Learning
Venue:
CORAL Seminar, North Carolina State University (Spring 2026)
Host: ECE Department and Control Faculty, North Carolina State University.
[\(S_2\)]
Unreliable Collaboration at Scale: Near-Optimal Adversarial Federated RL
Venue:
Computer Science Departmental Talk, North Carolina State University (Fall 2026)
Host:
Prof. Min Chi
, Department of Computer Science, North Carolina State University.
[\(S_1\)]
Learning When the Data Lies: Promise and Fundamental Limits
Venue:
Columbia University, New York (Spring 2027)
Host:
Prof. James Anderson
, Department of Electrical Engineering, Columbia University.
β₯οΈ Ph.D. Thesis
[\(T\)]
Towards a Finite-Time Theory of Adversarially Robust Reinforcement Learning: Algorithms, Guarantees, and Fundamental Limits,
(Ongoing; Expected Completion: December 2027)