Ghanem, Mohamed C. and Chen, Thomas M. and Nepomuceno, Erivelton (2022) Hierarchical reinforcement learning for efficient and effective automated penetration testing of large networks. Journal of Intelligent Information Systems. ISSN 0925-9902

Preview

Download (2MB) | Preview

Abstract

Penetration testing (PT) is a method for assessing and evaluating the security of digital assets by planning, generating, and executing possible attacks that aim to discover and exploit vulnerabilities. In large networks, penetration testing becomes repetitive, complex and resource consuming despite the use of automated tools. This paper investigates reinforcement learning (RL) to make penetration testing more intelligent, targeted, and efficient. The proposed approach called Intelligent Automated Penetration Testing Framework (IAPTF) utilizes model-based RL to automate sequential decision making. Penetration testing tasks are treated as a partially observed Markov decision process (POMDP) which is solved with an external POMDP-solver using different algorithms to identify the most efficient options. A major difficulty encountered was solving large POMDPs resulting from large networks. This was overcome by representing networks hierarchically as a group of clusters and treating each cluster separately. This approach is tested through simulations of networks of various sizes. The results show that IAPTF with hierarchical network modeling outperforms previous approaches as well as human performance in terms of time, number of tested vectors and accuracy, and the advantage increases with the network size. Another advantage of IAPTF is the ease of repetition for retesting similar networks, which is often encountered in real PT. The results suggest that IAPTF is a promising approach to offload work from and ultimately replace human pen testing.

Item Type:	Article
Keywords:	Penetration testing · Artifcial intelligence; Machine learning; Reinforcement learning; Hierarchical reinforcement learning; Markov decision process; Vulnerability; assessment;
Academic Unit:	Faculty of Science and Engineering > Electronic Engineering Faculty of Science and Engineering > Research Institutes > Hamilton Institute
Item ID:	16842
Identification Number:	https://doi.org/10.1007/s10844-022-00738-0
Depositing User:	Erivelton Nepomuceno
Date Deposited:	10 Jan 2023 16:51
Journal or Publication Title:	Journal of Intelligent Information Systems
Publisher:	Springer
Refereed:	Yes
URI:	Publisher
Use Licence:	This item is available under a Creative Commons Attribution Non Commercial Share Alike Licence (CC BY-NC-SA). Details of this licence are available here

Repository Staff Only(login required)

Item control page

Downloads

Origin of downloads