Malicious Code Prevention In Kernel Mode Computer Science Essay


As the window system are affected by the malicious code, the vulnerability on the system are more. The behavior based monitoring is used to avoid the polymorphic malicious code. Signature based detection system or fixing these vulnerabilities are not the efficient method. Thus the behavior based monitoring is used to avoid malicious code in window system. Some of them hook high level system APIs to detect the suspicious behavior of the code. Thus this cannot detect malicious code that directly invokes the native APIs. Thus a security scheme is used, that hooks native APIs in the kernel mode. It provides authentication of the system service caller in the kernel mode. So it prevents the malicious code calling native API directly. To provide extra authentication the dispatch ID is scrambled. For the scrambling to take place first the dispatch ID is distinguished as local ID and remote ID and then the filter is used to find the legitimate user. Next the scrambling is done with that legitimate user ID. The unscrambling is done to get the original dispatch ID only if it is a legitimate user. It introduces an average eight percent computation overhead into the system.

Lady using a tablet
Lady using a tablet


Essay Writers

Lady Using Tablet

Get your grade
or your money back

using our Essay Writing Service!

Essay Writing Service

Index Terms- Malicious code, Native API, Scrambling


The attackers inject the malicious code into the system. Thus by attacking the system they get the privilege to access the system. Many computers can run the same software so the code injection on one system will affect all other easily.API hooking is the technique used in operating system to prevent the code injection. Thus in API hooking the legitimate user can get the system service easily but for the illegal user it disallow the injected code to get the system service.API hooking technique in which you temporarily alter the command or call command when the application calls the function and divert the process to an alternative function. The main aim of the illegal user is to get the system service. Getting system service is not a simple task. The injected code needs the following to get the system service.

They are

1. Large buffer sizes are needed.

2. Large instructions given by the injected code are detected by intrusion detection system (IDS).

The API hooking is of two type's user mode API hooking in this method it forbid the injected code calling the user mode api this mechanism is notified then reviewed and justified. the main disadvantage of this hooking is that if the injected code does not call the user mode api, it cannot stop the illegal system request

For kernel mode API hooking it inspect the request and decide whether to allow the request. in this the disadvantage is that the system service request is called from the user mode api or from the infected code is not known.


In this first we are going to see about how the system service is provided by the window system. The system service is getting by initially calling the higher level API. The ntdll.dll present in the higher level API is processed. The dispatch ID of the process is moved to an register. Then by sysenter it enters into the higher level API. This higher level API contains the system service dispatch table (SSDT). The table contains the services and the ID for various process. Based upon the dispatch ID present in the register the service is provided by the native API.

Fig.1. System call in the operating system


Windows operating system are affected by the malicious code. The attackers inject their code into this software and force them to abnormally execute the injected code. Thus the victim computers will be controlled by the attackers and spread the injection to other computer. The injected code intends to store itself into the file system; it needs to create a file. Thus without invoking the system service, it may require over thousands lines of machine code to re implement file I/O. So control of the system is got by the attackers only by getting the system service. The system service is provided by the native API. So the attackers infect the native API in the kernel mode.

Lady using a tablet
Lady using a tablet


Writing Services

Lady Using Tablet

Always on Time

Marked to Standard

Order Now

Fig.2. Injected code to invoke the system service

Thus the injected code may affect the library function like usr.dll, kernel32.dll, and ntdll.dll in the higher level API in the user mode. It may attack the system service directly in the lower level API in the kernel mode.


Attacks that attempt to compromise a computer system using the system call interface are an important threat. In 2006 M. Rajagopalan et al proposed an approach called authenticated system calls [1]. An authenticated system call is a method used to verify the system call at the kernel. Thus it protect against the attack at low cost. In authenticated system the policy is limited; we can increase the effectiveness of the system by improving the expressiveness of policies.

Consumer electronic products are equipped with diverse device, the device drivers for those devices cause system failure. Jeaho Hwang et al in 2010 proposed a prototype called AppWatch [2], which protects core application software from invalid writing of device drivers to the application's memory by exploiting a memory management unit (MMU). it does not cause any modification on kernel behavior resulting in high portability at an extremely low engineering cost. Overhead is more for traversing a page table.

Drive-by-Download attacks are one of the most severe security threats to computer and network system. The browser who using the web page are attacked by the attacked web page the hijacker can download and execute any code on the victim's host. Thus in 2011 Fu-Hau Hsu et al proposed a scheme Browser Guard [3] a runtime, behavior based solution to drive by download attack. Browser Guard analyzes the download scenario of every downloaded object. Based on the download scenario, Browser Guard blocks the execution of any executable file that is downloaded to the host machine without the consent of a user. Browser Guard does not need to analyze the source file of any web page or the run-time states of any script code. It has low performance overhead. Browser Guard only supports IE 7.0 on a Windows system; we believe the defense model of Browser Guard can serve as a guide to develop similar tools for other browsers.

Rootkit affects the system security by modifying kernel control and non control data structure to achieve a variety of malicious goals. Arati Baliga et al developed Gibraltar [4], a tool which is used to detect the kernel level rootkits. The data structure invariants is used for this purpose. The anomaly based detection is used. Thus an automatic technique is used to infer invariant which is present in the kernel data structure, and use these invariants to detect rootkits. The overhead is of 0.5 percent. The automatically inferred invariants where quite precise. Gibraltar tool detect only 23 rootkits, the other type of rootkits may compromise the kernel.

Rootkit affect the kernel control data Rootkit authors only need to compromise one piece of control data to launch their attacks, while defenders need to "protect thousands of such values widely scattered across kernel memory space. In 2011 Jinker li et al proposed a scheme called indexed hooks [5] that comprehensively and efficiently transform kernel control data into indexed books, which allows us to effectively limit them to take konly the legal jump targets allowed by the CFG. Advantages of our approach are generic, effective and can be implemented on commodity hardware with low performance overhead

Attacks on window system are reported endlessly. Attackers inject the code in the software and force them to abnormally execute the injected code. Hung-min sun et al proposes a method API hooking [6] in both windows and Linux system in 2011, to stop injected code through protecting system service. By inspecting the behaviors' of how API are called these method intend to allow all legitimate code to call system services, at the same time disallow all injected code to call system services It introduce an average eight percent computation overhead into the system. API hooking does not provide complete for window system. Malicious code can access kernel API's by calling higher level user mode API's.

Rootkit, which unnoticeably reside in your computer, stealthily carry on remote control and software eavesdropping, are a threat to network and computer security. In 2006 LI xianghe et al proposed a technology called kernel hook [7] , is focused on detecting hook and hidden processes. By identifying hooks, we will detect the rootkit. No detection solution is complete or faultless and unbreakable.

Lady using a tablet
Lady using a tablet

This Essay is

a Student's Work

Lady Using Tablet

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.

Examples of our work

Kernel rootkit, is the most typical set of rootkit it has the significant challenge for the investigation and defence. Persistent kernel rootkits, it is the most challenging type of rootkit. It place the kernel hook that tamper with the kernel execution and it hide their presence. Zhi wang et al in 2008, proposed a scheme in that the related set o hooks which are present in the kernel are identified by monitoring and analyze its kernel side execution path. This hooks are hijacked to get the service. Hoop map [8] is building a proof of concept. To identify hook in the kernel side execution have challenges, for monitoring the kernel side the right side execution of kernel is difficult to get. The relevant run time information is difficult to obtain in the identified kernel execution path.

Personal digital assistance, mobile phones and smart phones which connect to internet are susceptible to malware thread such as viruses, torjan horses, worms. Asaf sabtai et al proposed Andromaly [9] a framework in 2010 realizes on host based malware detection which continuously monitors various features and event oriented from the mobile device. The detection approach and algorithm are light weight and run on the device itself. Anomaly detection in android platform has challenges, malicious activities can be very short thus not providing sufficient data to learn from or detect. Not enough malicious application to use in the training phase.

Dissimilarity in the kernel space is a serious problem. Gyorgy kovacs et al proposed the use of negative centered correlation and centered euctidation dissimilarity [10] in kernel space and dissimilarity measure. It improves accuracy and efficiency of the system.

Insider attack is a serious problem in computer network. Masqueraders and traitors as two distinct cases of insider attack. Malek ben salem et al proposed two approach profile user behavior for Masqueraders prevention and decays and trap based defense by honeypots and honey tokens [11] for traitors prevention. Prevent from insider attack.


Our goal is to authenticate a system service caller in the kernel mode. Our system has four components: the monitor, the preprocessing function, the processing function, and the driver.


The protection process is started by the monitor. The process is created by the monitor using CreateProcess. After the process is loaded and initialized the main thread is suspended. The monitor injects the DLL by using Remote DLL. The CreateRemoteThread issued to create the remote thread to initialize the DLL. The monitor contains the PID into the monitor list. The DLL customize the ntdll.dll and it injects the hook.dll into the process.

Steps in the DLL:

The ntdll.dll address is located in the memory.

WriteProcessMemory instruction is used to attach the processing and preprocessing function in the ntdll.dll

Replace the sysenter command by jump preprocessing command.

VirtualProtect is used to create read-only page for validation function.

Fig.3. System architecture

The Preprocessing function

It is a segment of code placed in the ntdll.dll for the process. It function in the user mode to do the following

The register might be modified and cause the data integrity problem so the registers are needs to back up in advance in the stack. The register value will be later restored in the processing function.

We also store password P into the stack.

P= protected API number (dispatch ID) [9 bits] and a random key K [23 bits].

The password is stored in the stack pointer esp, the esp will be stored at the fixed unused memory address S.


The driver does the protection mechanism in the kernel mode. The Windows Driver Kit is used to develop the device driver. The OSR Driver loader is used to load the device driver. It does two functions.

The request is first check for the authentication, if it does not find the authentication. The driver records the dispatchID, the processID and the threadID and change the address where the processing function located.

If the process is already authenticated the driver retrieve the record by matching the service number, the processID, and the threadID. The record is removed and the system service is provided.

The Processing function

It is a segment of code placed in the ntdll.dll for the process. The functions restore the stack pointer esp from the addresses saved at S. Thus the password is popped. Then it validate the password P which contain the value K, is compared with the key registered by the monitor at load time. If it is equals then the process is authenticated and executes the sysenter to reenter the kernel mode. If the K value is not equal, then the process is suggested to be hijacked by some malicious code. Thus it alerts the system administrator about the malicious code. Finally the process is terminated and the password P is erased from the memory.


These methods effectively prevent the native API to give the system service to the illegal user. Thus in the kernel mode it effectively prevent malicious code to access the native APIs. This method does not provide the solution if the malicious code affects the user mode in the operating system. Thus the malicious code can access the lower level kernel API by calling the higher level user API.