Neural Exploitation and Exploration of Contextual Bandits

In this paper, we study the neural exploration strategy for contextual bandits. The dilemma of exploitation and exploration widely exists in real-world applications such as recommender systems, online advertising, and clinical trials. Contextual bandits provide principled methods to solve this dilemma, including two prevalent techniques: Thompson Sampling (TS), and Upper Confidence Bound (UCB). Neural contextual bandits have been studied to adapt to the non-linear reward function, combined with TS or UCB strategies for exploration. In this paper, we introduce, EE-Net, which is a novel framework to utilize another neural network to learn the potential gain of exploitation neural network for exploration, different from UCB-based and TS-based approaches that rely on the large-deviation-based statistical confidence bound. In addition, we provide an instance-based $\widetilde{\mathcal{O}}(\sqrt{T})$ regret upper bound for EE-Net with a new proof workflow. Empirically, we show that EE-Net outperforms related linear and neural contextual bandit baselines on real-world datasets.