An Optimized Lightweight CNN with Randomized Hyperparameter Search for Real-Time Image-Based Malware Detection

Abstract

While image-based malware detection using deep learning has shown promise, existing methodologies predominantly rely on computationally expensive pre-trained architectures (e.g., VGG, ResNet) that create significant bottlenecks for real-time deployment on resource-constrained gateways. This research addresses this critical gap by proposing a streamlined, lightweight custom Convolutional Neural Network (CNN) specifically optimized for real-time operation. The novelty of this work lies in the strategic integration of Randomized Search Cross-Validation (RS-CV) to automate the discovery of an optimal configuration of filters, dense units, and dropout rates, eliminating the inefficiencies and biases of manual hyperparameter tuning. The proposed method transforms binary files into 64x64 grayscale images—reducing computational input by over 90% compared to standard architectures—which are then processed by the optimized custom network. Experimental results demonstrate the scientific significance of this approach, as the model achieved a near-perfect Area Under the Curve (AUC) of 0.9996 and identified threats with an average inference time of only 12–15 milliseconds. Out of 1,068 test samples, only 10 misclassifications were recorded, proving that a mathematically optimized lightweight model can outperform heavy ensemble frameworks in both accuracy and speed. These findings provide a reproducible framework for high-speed, front-line cybersecurity systems capable of detecting obfuscated threats in live network environments.