Making JavaScript Render Decisions to Optimize Security-Oriented Crawler Process


Aktas O., CAN A. B.

IEEE ACCESS, vol.12, pp.161688-161696, 2024 (SCI-Expanded) identifier identifier

  • Publication Type: Article / Article
  • Volume: 12
  • Publication Date: 2024
  • Doi Number: 10.1109/access.2024.3481646
  • Journal Name: IEEE ACCESS
  • Journal Indexes: Science Citation Index Expanded (SCI-EXPANDED), Scopus, Compendex, INSPEC, Directory of Open Access Journals
  • Page Numbers: pp.161688-161696
  • Hacettepe University Affiliated: Yes

Abstract

The widespread use of web applications requires important changes in cybersecurity to protect online services and data. In the process of identifying security vulnerabilities in web applications, a systematic approach is employed to detect and mitigate cybersecurity risks. This approach utilizes web crawlers to identify attack vectors. Traditional web crawling methods are resource-intensive and often need to be more efficient in handling dynamic JavaScript-rich content. Addressing this crucial gap, our study introduces an innovative approach to predict the necessity of JavaScript rendering, thereby enhancing the effectiveness and efficiency of security-oriented web crawlers. This approach seeks to reduce computational requirements and quicken the security evaluation process through the use of machine learning algorithms. By utilizing a dataset containing the source code from the main pages of 17,160 websites, our experimental results demonstrate a 20% reduction in execution time compared to full JavaScript rendering, indicating an improvement in resource usage without any significant reduction in coverage. Our methodology significantly improves the efficiency of security-focused web crawlers and helps security scanners to detect security risks of web applications with fewer resources.