Enabling Detection of Elusive Malware by Going Out of the Box with Semantically Reconstructed View (OBSERV)

Research Areas: Network Security

Principal Investigator: Dongyan Xu

There is an alarming trend that elusive malware is armed with techniques that detect, evade, and subvert malware detection facilities of the victim. On the defensive side, a fundamental limitation of traditional host-based anti-malware systems is that they run inside the very hosts they are protecting, making them vulnerable to malware's counter-detection and subversion. To address this limitation, solutions using virtual machine (VM) technologies advocate placing the malware detection facility outside of the protected VM. However, a dilemma exists between these two approaches: The "out of the box" approach gains tamper resistance at the cost of losing the native, semantic view of the host enjoyed by the "in the box" approach. To resolve the above dilemma, a new approach called OBSERV ("Out of the Box with Semantically Reconstructed View") is introduced to achieve the advantages of both camps by reconstructing the semantic internal view of a VM from external, low-level observations. OBSERV enables two exciting malware defense opportunities: (1) malware detection by view comparison and (2) real-time detection and stoppage of kernel-level rootkits. The broader impact of this research is two-fold: (1) It will enhance the trustworthiness and effectiveness of widely deployed anti-malware systems. Moreover, OBSERV is expected to be viewed favorably by the anti-virus software industry because of its support for existing off-the-shelf anti-virus software. (2) Results from this research will lead to the development of education materials for undergraduate and graduate courses and for professional training sessions.

Personnel

Other PIs: Xuxian Jiang, George Mason University

Students: Zhiqiang Lin

Representative Publications

Xuxian Jiang, Xinyuan Wang, Dongyan Xu, "Stealthy Malware Detection Through VMM-Based Out-of-the-Box Semantic View Reconstruction", Proceedings of ACM Conference on Computer and Communications Security (CCS 2007), Alexandria, VA, November 2007.

Keywords: malware detection, OBSERV, semantics