Attention-based Curiosity-driven Exploration in Deep Reinforcement Learning

Abstract

Reinforcement Learning enables to train an agent via interaction with the environment. However, in the majority of real-world scenarios, the extrinsic feedback is sparse or not sufficient, thus intrinsic reward formulations are needed to successfully train the agent. This work investigates and extends the paradigm of curiosity-driven exploration. Our aim is to develop means for the better incorporation of state-and/or action-dependent information into existing intrinsic reward formulations. First, a probabilistic approach is taken to exploit the advantages of the attention mechanism, which is successfully applied in other domains of Deep Learning. Combining them, we propose new methods, such as Attention-aided Advantage Actor-Critic, an extension of the Actor-Critic framework. Second, another curiosity-based approach - Intrinsic Curiosity Module - is extended. The proposed model utilizes attention to emphasize features for the dynamic models within Intrinsic Curiosity Module, moreover, we also modify the loss function, resulting in a new curiosity formulation, which we call rational curiosity (RCM). The corresponding implementation can be found at https://github.com/rpatrik96/AttA2C/.

Citation

P. Reizinger and M. Szemenyei, “Attention-Based Curiosity-Driven Exploration in Deep Reinforcement Learning,” ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 3542-3546, doi: 10.1109/ICASSP40776.2020.9054546.

@INPROCEEDINGS{reizinger2020atta2c,  
    author={Reizinger, Patrik and Szemenyei, Márton},
    booktitle={ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
    title={Attention-Based Curiosity-Driven Exploration in Deep Reinforcement Learning},
    year={2020},
    volume={},
    number={}, 
    pages={3542-3546},
    doi={10.1109/ICASSP40776.2020.9054546}
}