VMHunt: A Verifiable Approach to Partially-Virtualized Binary Code Simplification
VMHunt: A Verifiable Approach to Partially-Virtualized Binary Code Simplification
复制标题
DOI：
10.1145/3243734.3243827
复制
发表时间：
2018-10
期刊：
影响因子：
--
通讯作者：
Dongpeng Xu;Jiang Ming;Yu Fu;Dinghao Wu
中科院分区：
文献类型：
--
作者：
Dongpeng Xu;Jiang Ming;Yu Fu;Dinghao Wu
代码虚拟化是恶意软件作者为了躲避雷达而采用的一种高度复杂的混淆技术。然而，代码虚拟化不断增加的复杂性也成为实际应用的一把“双刃剑”。由于其性能限制和兼容性问题，代码虚拟化很少在整个程序上使用。相反，它主要仅用于保护代码的关键部分，例如安全检查和加密密钥。人们已经提出了许多技术来对虚拟化代码进行逆向工程，但它们都有一些共同的局限性。他们假设虚拟化代码的范围是预先已知的，并且主要关注代码模拟器的经典结构。此外，很少有工作验证其反混淆结果的正确性。在本文中，通过对代码虚拟化的类型和范围的假设较少，我们提出了一种可验证的方法来解决部分虚拟化二进制代码简化的挑战。我们的主要见解是，代码虚拟化是一种进程级虚拟机（VM），进入和退出VM时的上下文切换模式可用于检测VM边界。基于VM边界的范围，我们简化了虚拟化代码。我们首先忽略给定虚拟化片段中不影响该片段最终结果的所有指令。为了更好地恢复通过按位运算对变量进行编码的数据混淆效果，我们随后运行一种称为多粒度符号执行的新符号执行，以进一步简化跟踪片段。生成的简洁符号公式有利于我们简化结果的正确性测试。我们已将我们的想法实现为开源工具 VMHunt，并使用实际应用程序和恶意软件对其进行了评估。令人鼓舞的实验结果表明，VMHunt 比现有技术有了显着改进。
Code virtualization is a highly sophisticated obfuscation technique adopted by malware authors to stay under the radar. However, the increasing complexity of code virtualization also becomes a "double-edged sword" for practical application. Due to its performance limitations and compatibility problems, code virtualization is seldom used on an entire program. Rather, it is mainly used only to safeguard the key parts of code such as security checks and encryption keys. Many techniques have been proposed to reverse engineer the virtualized code, but they share some common limitations. They assume the scope of virtualized code is known in advance and mainly focus on the classic structure of code emulator. Also, few work verifies the correctness of their deobfuscation results. In this paper, with fewer assumptions on the type and scope of code virtualization, we present a verifiable method to address the challenge of partially-virtualized binary code simplification. Our key insight is that code virtualization is a kind of process-level virtual machine (VM), and the context switch patterns when entering and exiting the VM can be used to detect the VM boundaries. Based on the scope of VM boundary, we simplify the virtualized code. We first ignore all the instructions in a given virtualized snippet that do not affect the final result of that snippet. To better revert the data obfuscation effect that encodes a variable through bitwise operations, we then run a new symbolic execution called multiple granularity symbolic execution to further simplify the trace snippet. The generated concise symbolic formulas facilitate the correctness testing of our simplification results. We have implemented our idea as an open source tool, VMHunt, and evaluated it with real-world applications and malware. The encouraging experimental results demonstrate that VMHunt is a significant improvement over the state of the art.