MURAL - Maynooth University Research Archive Library

    Hierarchical reinforcement learning for efficient and effective automated penetration testing of large networks

    Ghanem, Mohamed C. and Chen, Thomas M. and Nepomuceno, Erivelton (2022) Hierarchical reinforcement learning for efficient and effective automated penetration testing of large networks. Journal of Intelligent Information Systems. ISSN 0925-9902

    Download (2MB) | Preview

    Share your research

    Twitter Facebook LinkedIn GooglePlus Email more...

    Add this article to your Mendeley library


    Penetration testing (PT) is a method for assessing and evaluating the security of digital assets by planning, generating, and executing possible attacks that aim to discover and exploit vulnerabilities. In large networks, penetration testing becomes repetitive, complex and resource consuming despite the use of automated tools. This paper investigates reinforcement learning (RL) to make penetration testing more intelligent, targeted, and efficient. The proposed approach called Intelligent Automated Penetration Testing Framework (IAPTF) utilizes model-based RL to automate sequential decision making. Penetration testing tasks are treated as a partially observed Markov decision process (POMDP) which is solved with an external POMDP-solver using different algorithms to identify the most efficient options. A major difficulty encountered was solving large POMDPs resulting from large networks. This was overcome by representing networks hierarchically as a group of clusters and treating each cluster separately. This approach is tested through simulations of networks of various sizes. The results show that IAPTF with hierarchical network modeling outperforms previous approaches as well as human performance in terms of time, number of tested vectors and accuracy, and the advantage increases with the network size. Another advantage of IAPTF is the ease of repetition for retesting similar networks, which is often encountered in real PT. The results suggest that IAPTF is a promising approach to offload work from and ultimately replace human pen testing.

    Item Type: Article
    Keywords: Penetration testing · Artifcial intelligence; Machine learning; Reinforcement learning; Hierarchical reinforcement learning; Markov decision process; Vulnerability; assessment;
    Academic Unit: Faculty of Science and Engineering > Electronic Engineering
    Faculty of Science and Engineering > Research Institutes > Hamilton Institute
    Item ID: 16842
    Identification Number:
    Depositing User: Erivelton Nepomuceno
    Date Deposited: 10 Jan 2023 16:51
    Journal or Publication Title: Journal of Intelligent Information Systems
    Publisher: Springer
    Refereed: Yes
    Use Licence: This item is available under a Creative Commons Attribution Non Commercial Share Alike Licence (CC BY-NC-SA). Details of this licence are available here

    Repository Staff Only(login required)

    View Item Item control page


    Downloads per month over past year

    Origin of downloads