Virtualization-Enabled Malware Research

Research Areas: Network Security

Principal Investigator: Dongyan Xu

Funding Source: Microsoft Research and National Science Foundation through the NMI program under grant number OCI-0504261.

In the battle against Internet malware, we have witnessed increasingly novel features of emerging malware in their infection, propagation, and contamination strategies – examples include polymorphic appearance, multi-vector infection, self-destruction, and intelligent payloads such as self-organized attack networks or mass-mailing. Furthermore, the damages caused by a malware incident can be detrimental and hard to recover (e.g., the installation of kernel-level rootkits). Our research goal is to thoroughly understand key malware behavior such as probing, propagation, exploitation, contamination, and "value-added" payloads. These results will be used to design effective malware detection and defense solutions. To reach this goal, we realize that effective malware experimentation tools and environments are lacking in current malware research. By leveraging and extending virtualization technology, we propose to develop a virtualization-based integrated platform for the capture, observation, and analysis of malware. The platform consists of two parts: The front-end of the platform is a virtual honey farm system called Collapsar, which captures and contains malware instances from the real Internet. The back-end of the platform is a virtual playground environment called vGround, where the captured malware instances are unleashed to run while remaining completely isolated from the real Internet. Using this integrated platform, security researchers will be able to observe and analyze various aspects of malware behavior as well as to evaluate corresponding malware defense solutions, with high fidelity and efficiency.


Students: Ryan D. Riley Junghwan Rhee

Representative Publications

  • Ryan Riley, Xuxian Jiang, Dongyan Xu, "An Architectural Approach to Preventing Code Injection Attacks", Proceedings of IEEE/IFIP International Conference on Dependable Systems and Networks (DSN-DCCS 2007), Edinburgh, UK, June 2007.

  • Xuxian Jiang, Dongyan Xu, "Collapsar: A VM-Based Architecture for Network Attack Detention Center," Proceedings of the 13th USENIXSecurity Symposium, San Diego, CA, August 2004.

Keywords: contamination, defense solutions, exploitation, malware detection, probing, propagation