NSF Award
2211905
Obfuscation technology has been widely adopted by writers of malicious code (malware) to circumvent defense solutions. The goal of obfuscation is to transform malware into an equivalent, but highly complex form that hides the malware's structure and hinders automatic detection solutions and even manual inspection by security analysts. Rapid analysis of obfuscated malware is vital for a swift response to emerging threats, such as ransomware. This research project advances human knowledge on defeating obfuscated malware. The project's novelties are the new knowledge revealed from the state-of-the-art obfuscation, the new techniques designed for extracting the knowledge, and the new deobfuscation methods for understanding such type of stealthy malware. The project's broader significance and importance are new cybersecurity learning experiences for K-12/undergraduate/graduate students, and new technologies for national cybersecurity against a wide range of emerging malware threats, with high potential for transition to practice. <br/><br/>The insight of this project is to leverage the virtualization technique itself to beat modern virtualization obfuscators. Two major features play a crucial role in the success of virtualization obfuscation: sophistication (the obfuscated form is very complex and different from the original program) and diversification (multiple obfuscated forms of the same program strikingly vary). These features heavily impede existing deobfuscation techniques that rely on recognizing special virtual machine patterns or treating the whole virtualization as a black box. This project invents and implements a series of novel methods to: (1) comprehensively probe the sophisticated structures inside virtual machines, such as interpretation architecture, virtual instructions, and handler encryption, (2) reveal the core techniques to combine, mutate, and randomize diverse virtual machines, and (3) build a new, interpretable virtual machine specifically for deobfuscation, which can be stitched into a simple, executable program as the deobfuscation result. The new techniques developed from this project effectively free security professionals from the painful, tedious deobfuscation steps incurred in malware analysis.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
