Anti -Reverse Engineering Techniques employed by Malware

3310 words (13 pages) Essay

8th Feb 2020 Computer Science Reference this


Disclaimer: This work has been submitted by a university student. This is not an example of the work produced by our Essay Writing Service. You can view samples of our professional work here.

Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of

Abstract— In Modern days, analyzing the malware samples through reverse engineering methodologies became more difficult because the malware authors are using Anti-reverse engineering techniques to evade the detection of malware. Previously these techniques are used by the software developers to prevent the software cracking, however such protection schemes are massively applied to malicious software these days. By identifying the functionality of these techniques, the malware analyst can easily detect the presence of Anti-Reverse Engineering methods. This paper proposes the proof of concept about the functionality of various anti-reverse engineering techniques used in malware samples.

Keywords: Malware, Reverse Engineering, Anti-reverse Engineering, Anti-debugging, Anti-VM.

I.     Introduction

In recent years, the growth of malware is rapidly increasing. Over 350,000 malicious programs are being registered by Anti-virus softwares every day. In 2018, the most trending malwares are emotet, kovter, trickbot, zeus etc. The complexity of malwares has been increasing considerably with the emerging of technology. To analyze these malicious softwares effectively, the first step is to extract the malicious behavior which is the process of Reverse engineering. It is a structured approach for analyzing the malware.  The behavior and functionality of the malwares can be detected through reverse engineering methodologies. Reverse-engineering a malware involves two key phases i.e Static and Dynamic analysis. On one hand static analysis describes about process of analyzing the code to determine its function, on other hand the dynamic analysis observes behavior of malware by executing it in a controlled and monitored environment.

In static analysis, the examining of the malware is done through two methods: Basic static and advanced static analysis. The Basic static phase involves analyzing the malware sample without observing the actual code. It is good to run the program in different anti-virus softwares to determine if the file is malicious or not. These softwares mainly detect the malicious files with the help of pattern matching as well as the file signatures analysis. For unique identification of the malware, hashing is the most commonly used method. By those hash values, the identification of the malware can be done easily. Mostly used hash algorithm for malware analysis is MD5 [1]. The Advanced static phase involves in analyzing executable (malware) through linked libraries and loading the sample in IDA disassembler.

Unlike static analysis, the Dynamic analysis helps us to understand the malware functionality. It will be done through two approaches, Basic dynamic and advanced dynamic analysis. The Basic dynamic analysis involves observing the functionality of the malware by building it in a sandbox or in a virtual machine (safe environment). The analysis of the malware can be performed by using the sandboxes and monitoring the process of malware. In Advanced dynamic analysis, the malware is loaded into the debugger for examining the internal state [2]. By using these methodologies, functionality and behavioral detection of the malware will be done.

When compared to static analysis, the dynamic analysis gives much information about runtime behavior of the malicious program. The security researcher can analyze behavior and functionality of the malware with the help of dynamic analysis. In order to evade the detection of malware through this analysis, the malware authors are using the Anti-reverse Engineering techniques. The malware analyst will find difficulties in analyzing those malware samples which are using these techniques. Detecting and counteracting these types of techniques play a vital role in the analysis. This paper mainly focuses on Proof Of Concept (POC) about the functionality of the Anti-Reverse engineering techniques.

II.    Related Work

Malware authors use Anti-Reverse Engineering Techniques widely to impede the reverse engineering process of a malware. In reverse engineering, the malware authors will run the malware samples in debuggers, disassemblers or in virtual environments to analyze the functionality and behavior. The recent malware sample plays a lot of tricks to recognize the environment in which they are running with the help of Anti-reverse engineering techniques. Whenever malware recognizes the environment, it may hide the malicious functionality or it may terminate. In most of the cases, malware uses these techniques to make the analysis more difficult. So, the malware analyst should know the functionality of those techniques which were used by the malware. These techniques include the following,

Before you begin to format your paper, first write and save the content as a separate text file. Complete all content and organizational editing before formatting. Please note sections A-D below for more information on proofreading, spelling and grammar.

Keep your text and graphic files separate until after the text has been formatted and styled. Do not use hard tabs, and limit use of hard returns to only one return at the end of a paragraph. Do not add any kind of pagination anywhere in

Without having awareness on these Anti-Reverse engineering techniques one can’t detect their presence in the malware. For example, the security researcher executes the malware in the debugger to know its functionality which is having the Anti-debugging technique. Then the malware hides its malicious functions and simply runs as legitimate code this is done by using the function named IsDebuggerPresent. Likewise the malware sample also checks for the time checks, driver space etc., to detect if it is running in the virtual environment. So, this research paper creates the awareness of these Anti-Reverse engineering techniques.

III.   Related Work

Malwares are often executed in an isolated environment such as virtual machines and sandboxes and also authors make use of debuggers to analyze the malicious software. All these techniques speedup the analysis, this is the reason why the malware authors thinking about the detection techniques to make the analysis difficult when it is being executed in these environments. So, the malware analysts invented the new techniques named Anti-Reverse Engineering. In this paper, Proof of concept is developed to describe the functionality of Anti-Reverse engineering techniques. Let’s have a brief explanation of each technique used by the malwares.


Anti-debugging is one of the techniques in Anti-reverse engineering which was used to thwart the debuggers. The malware analyst makes use of debuggers to find out the functionality of malware, so to hinder this process malware authors uses the anti-debugging technique. With the help of windows API the malware will recognize that it was running in a debugger [3].  The Windows API which was used by the malware to detect the presence of debugger is IsDebuggerPresent.

Based on this function, the malware sample checks if it is running in the debugger or not. The function IsDebuggerPresent returns zero then it represents that malicious sample is not running in the debugger, if it returns the non-zero value then it represents that it is not running in the debugger.

Figure 1

In the figure1 we can observe the exe running in the Ollydbg. Now it checks for the function IsDebuggerPresent.

Figure 2

In figure 2 it states that the debugger is present. The malware author will embed these types of API’s in the malicious code so it can recognize the presence of a debugger.

Virtual Environment /Isolated Environment:


In order to study the techniques of an attacker the analyst often use the isolated environment to run the malware i.e., virtual machines or sandboxes. By viewing some of the artifacts we can differentiate virtual machine and physical machine. With the help of these malware authors acknowledge the virtual machines. Below mentioned are the techniques how the malwares recognize the isolated environments.


VMware is one of the virtual environments in which we can run many OS on it. The malware author makes use of these functions to recognize the VM.

  • Checking for Mac Addresses:

Malware sample checks for the Prefixes of MAC addresses like

1)       00:05:69

2)       00:0C:29

3)       00:1C:14 and

4)       00:50:56.

  • Checking for Registry Keys:

It checks for some of the registry keys like,

1)       HKLMSOFTWAREVmware Inc.\Vmware Tools

2)       HKEY_LOCAL_MACHINEHARDWAREDEVICEMAPScsiScsiPort2ScsiBus               0Target Id 0Logical Unit Id 0Identifier

  • Checking for Existence of Files

1)       C:windowsSystem32DriversVmmouse.sys

2)       C:windowsSystem32Driversvm3dgl.dll

Figure 3

When it was executed in the VM it checks for the above mentioned functions. As the above figure clearly shows that all the functions Is Present, it means that we are running the exe in Virtual Machine.


       Anti-Virtual Box:

It is one of the techniques in Anti-reverse engineering which hinder the process of reverse engineering. The virtual Box detection can be done by checking the following functions,

  • Checking for MAC address

1)       08:00:27

  • Checking for Processes Indicating a VBox

1)       vboxservice.exe

2)       vboxtray.exe

  • Checking for existing files

1)       C:windowsSystem32DriversVBoxMouse.sys

2)       C:windowsSystem32DriversVBoxGuest.sys


4)       C:windowsSystem32DriversVBoxVideo.sys

5)       C:windowsSystem32vboxdisp.dll

6)       C:windowsSystem32vboxhook.dll [4].

Figure 4

From the above figure we can observe that, the exe is not running in the virtual box so it is showing the functions are not presented.

       Generic Sandbox:

Whenever we run the malware in the generic sandbox, it checks for,

Mouse Activity:

   The function GetCursorPros calls twice to compare the recorded coordinates of the cursor, if the values are not equal then we can state that it was running in a simulated environment.



The function GetUserName retrieves the username and the function strstr() checks for the strings like Sandbox, Virus, malware. If any of these strings are detected then the malware confirms that it was running in a sandbox.

File Path:

GetModuleFileName() function, used for retrieving the path of the file which contains the specified module. Here strstr() function checks for the strings like \SAMPLE, \VIRUS, Sandbox.

Drive Space:

The functions DeviceIoControl() and CreateFile() checks the physical drive space.

Time checks:

Gensandbox_uptime() uses GetTickCount() function to get the time elapsed.

Figure 5

As the above figure clearly shows that we are not running in the Generic Sandbox.

       Qemu Detection:


 The malware will detect the Qemu with the help of,

  • Timing attacks
  • OS implementation
  • Hardware implementation
  • Emulator-related software inside the OS, for example VMware tools[7]

Registry Keys:

1)      Scsi port: The Scsi identifier is detected when we enable the scsi option in the qemu.

2)      SystemBiosVersion: Checks for the string named Qemu.


       It will check for the string “Qemu Virtual CPU”.

                                                        Figure 6

As the figure 6 describes that we are not running the exe in Qemu environment.


The Sandboxie is one of the isolated environment, which is used by the malware analysts to run the malicious samples. It uses the module named sbiedll.dll. With the help of this process name the malicious program will detect that it was running in Sandboxie.

Sbiedll.dll is an executable file which contains the machine code. Whenever we start the Sandboxie application, then commands in the sbiedll.dll will be executed [5].

Figure 7

The module sbiedll.dll is not existing so, from the above figure we can understand that the Sandboxie is not detected.

Wine Detection:


The wine is the software used to run the windows executable files in Linux. Many of the malwares fail in compromising the linux systems. When the malicious programming is running in linux environment, it fails to change the registry key values like which were done in Windows systems. So the malwares can’t affect the linux systems. It checks for the functions like GetProcAddress and Registry key for wine detection [6].

GetProcAddress: The function named GetProcAddress() will be called to get the wine_get_unix_file_ name function address which was exported and is available in kernel32.dll. If this function doesn’t return NULL then the wine has been detected.

Registry Key: It just looks for the windows registry key named wine_reg_key1.

Figure 8

Here we are using the Wine software to run the exe in the linux environment. 

Due to the lack of awareness on these techniques malware analyst thinks that it is the legitimate file and runs in the host systems which lead to infection. So, it is important to know about Anti-Reverse Engineering Techniques.

IV.   Conclusion

This paper discusses about the functionality of Anti-reverse engineering techniques. Malware analysts find the difficulties in analyzing the malware samples because the malware authors embed the Anti-reverse engineering technique. These techniques will thwart the detection of the malware from static and dynamic analysis. So, this paper gives a brief explanation about the functions used by the malware to evade the detection. With the help of these techniques the analysts can detect the presence of Anti-reverse engineering techniques in the malware sample.


[1]      Jeremy Scott “Detecting malware through static and dynamic techniques” blog on NTT Security

[2]      Syarif Yusirwan S, Yudi Prayudi, Imam Riadi “Implementation of Malware Analysis using Static and Dynamic Analysis Method” International Journal of Computer Applications (0975 – 8887) Volume 117 – No. 6, May 2015

[3]      Michael Sikorski and Andrew Honig “PRACTICAL MALWARE ANALYSIS” venom630

[4]      Yaniv Assor “Anti-VM and Anti-Sandbox Explained” blog on cyberbit.


[6]      Rory DuncanZ. Cliffe Schreuders “Security implications of running windows software on a Linux system using Wine: a malware analysis study” springerlink


Cite This Work

To export a reference to this article please select a referencing stye below:

Reference Copied to Clipboard.
Reference Copied to Clipboard.
Reference Copied to Clipboard.
Reference Copied to Clipboard.
Reference Copied to Clipboard.
Reference Copied to Clipboard.
Reference Copied to Clipboard.

Related Services

View all

DMCA / Removal Request

If you are the original writer of this essay and no longer wish to have your work published on the website then please:

Related Lectures

Study for free with our range of university lectures!