Api Hooking In Windows Computer Science Essay

Published: Last Edited:

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.

The main aim of this report is to show the techniques used for Hooking applications in windows platform. Hooking enables us to alter the behaviour of application or software components by intercepting function calls in the Portable Executable (PE) files of application. Hooking enables us to open a new world for us to communicate and interfere the applications by intercepting function calls or message or events passed between software components. In this report we can see the various ways of achieving user mode hooking by loading the required DLL into the memory space of the target application which we want to intercept.


A "Hook" function is a basically a callback function that is inserted in the windows message system which acts on events. The main duty of callback function is to send a function as parameter into another function to modify or for discarding them. After completion of the callback function, the control is passed back to the calling function. Software technology has grown in the world we live today making security issues and extended functionality our first priority. Hooking plays a special role in developing application with extended functionality and to include debugging. With this technology the old boundaries between users and applications are going to be erased. The term Rootkits is often used in Operating systems, which refers to a set of programs that uses system hooking to modify or alter system files, process or other objects.

The Microsoft Windows allows us to insert hooks to process or modify keyboard and mouse events. We can intercept keyboard messages/events in windows before they reach an application using "Hooks". This function is called by the windows when the specified event happens (i.e. when ever there is a keyboard strokes ). A hook for any event will be the same. In this case the hook is called when ever the keyboard application calls. The only difference will be in the value/parameter passed to it which depends on the key pressed.

The Protection Ring Architecture used by Windows.

The intel i386 architecture supports four modes of security privileges from ring 0 - ring 3 to protect system code & data from being intentionally or unintentionally modified by lower privileges code. In these ring levels of protection the ring 0 has the highest priority & they are reserved for the operating system. The ring 3 has the lowest level priority and they are used by the applications. The Microsoft windows use two privilege levels ring 0 & ring 3 for process & data security. Microsoft Windows uses these two privilege mode to run on any CPU architecture that do not follow the four ring architecture.

Many of the Microsoft Windows application runs on the ring 3, like Internet explorer, Microsoft word, Session Manager, etc. the application codes for Internet Explorer & Microsoft word executes within the ring 3.

The high priority kernel level codes runs on the ring 0 & they are used in device drivers & windows kernel system components such as cache , I/O, object, plug & play, N/W protocol, anti-virus, etc. the operating system manages the interaction between system kernel & hardware resources with the user applications. The user mode applications use win32 API calls to interact with the kernel. The ring 0 level applications can be modified by Kernel Mode hooking & ring 3 can be modified by User Mode Hooking.

User Mode Hooking

The User Mode Hooking involves system hooking in application space which runs in ring 3. When ever the windows application makes a system call it follows a predetermined path of processing. User Mode hooking are mainly done by modifying API whose code resides in DLL. The application will call the required DLL when ever the application is in run-time. We can modify the DLL according to our need. So when ever the API is called the execution path will be redirected to our code and we can modify the return result. There are some issues that should be taken care off like Patching, unhooking the procedures and passing the system control to the next system message which we will discuss later in the sections.

Kernel Mode Hooking

The kernel resides in the lowest level in the ring architecture and is the most reliable & robust method of system hooking. Only authorised users can view or modify kernel memory. In kernel there is a proxy gate between the user mode & the kernel mode. This gate monitors all the purpose of the incoming system call & after execution of the code in kernel space it sends back the results to the system. The Kernel Rootkit can call the original system call & then alter the results before sending it back.

Another way to achieve kernel mode hooking is by modifying System Service Descriptor Table (SSDT) also known as system table. The SSDT is a function pointer table which holds all the addresses of the system call functions in kernel memory. By changing the address in this table to our hook procedure we can redirect the execution path of the original system call to our hook procedure.

How User-Mode Hooking is Achieved in Windows.

The internet explorer or Microsoft Word which are Win32 applications runs in user space as a user application or as a service. Whenever these application are executed the program/ application is loaded into the application import table. We can over write the function in the application import table to our hand crafted machine code this is called API Hooking. By Hooking API's we can alter or modify the process to our imagination. We can achieve this either by an IAT (Import Address Table) hooking, Inline Function Hooking and by injecting a DLL into user-mode process.

Import Address Table(IAT) Hooking

The Win32 application import the address of the application functions. All the DLL the applications uses are imported by the application when it is called by the Operating System. The OS then locates each imported function in memory & over writes it with the actual address of the function. Once our hook function is in the application address space, we can replace the target function address in the IAT with our hook function address. Thus enables us to execute hook function instead of the original function. This types of hooks are really easy to discover and they are powerful simple technique.

One of the main problem with this type of hooking is with the binding time. Some of the Win32 application do late demand-binding, the function address will not be known until the function is called. Thus reduces the memory the application uses. The function will not have these address in IAT when we try to hook them.

Inline function hooking


The Inline Function Hooking is more powerful & advanced than the IAT hooking. It does not have problem with binding time of the DLL. Instead of replacing pointers in the import address table , the Inline Function Hook replaces several bytes in the original function. The Inline Function Hook adds an unconditional jump from the target function to the hook code. It saves the original function which is going to be over written to preserve the same functional behaviour. After that it changes a portion of the target function code to the hook code.

The main location to use the Inline hook is within the first five bytes of the target function which we want to hook. In Windows Operating system most of the functions in the Win32 API begin the same way. This is called preamble. The following block of codes is the Assembly language for typical function preambles.

Pre-XP SP2 Code Bytes Assembly

55 push ebp

8bec mov ebp, esp

…… ……

Post-XP SP2 Code Bytes Assembly

8bff mov edi, edi

55 push ebp

8bec mov ebp, esp

Our patching function must be able to determine the length of the function instruction & to disassemble it to preserve the original functions opcodes. The Microsoft XP SP2 operating system allow us to insert new code without rebooting the system which is called Hot Patching. The preamble in windows is exactly five bytes. An unconditional jump to the hook code on the X86 architecture usually requires five bytes. The first byte will be used for the jmp opcode & the remaining four bytes are the address of our hook.

The other reason for over writing the beginning of the target function is because, if we place our hook function deeper inside the target function we will have to worry about the code entry & it may be called by the target function many times resulting in undesired result. The original function which is saved before hooking is called trampoline. And the unconditional jump which we places in the target function is called the detour. The detour calls the trampoline which jumps to the target function plus five bytes roughly.

Injecting a DLL into User-Mode Process

The other way to achieve Hooking is by injecting a DLL into the target process, which enables us to alter the execution path of commonly used API in win32 applications. We can inject a DLL into the target process by three ways they are

Injecting a DLL using Windows Hooks.

Injecting a DLL using the registry.

Injecting a DLL using Remote Threads.

Injecting a DLL using Windows Hooks

The windows Applications receive many event messages for all the events in the system which is related to the application. For eg: an application can receive events when a key is pressed in its window or when the mouse is in focus. The Microsoft defines a function that makes it possible to hook Windows messages in other process. SetWindowHookEx is a function of Microsoft which enables us to load our DLL into the address space of other process. The prototype of SetWindowsHookEx defined by the Microsoft is listed below:

HHOOK SetWindowsHookEx( int idHook, HOOKPROC lpfn, HINSTANCE hMod, DWORD dwThreadID );

The first parameter in the function is type of event message that will trigger the hook. An example for this is the WH_KEYBOARD, which installs a hook procedure to monitor the keystrokes messages from the keyboard. The second parameter identifies the address the system should call when a window is about to process the specified message. The third parameter is the virtual memory address of the DLL that contains this function. The last parameter is the thread to hook. The following is a template on how DLL would be implemented.

BOOL APIENTRY DllMain(HANDLE hModule, DWORD dwMode, LPVOID lpReserved) // main function


if(dwMode == DLL_PROCESS_ATTACH) //dwMode is the thread to be hooked. If the value of dwMode is 0, the system will hooks all threads in the current windows desktop.


// code here


return TRUE;


__declspec (dllexport) LRESULT myKBFn (int code, WPARM wParam, LPARAM lParam)


return CallNextHookEx(_hook, code, wParam, lParam); // to pass to the next message in the message list.


The DllMain() is automatically called by every application that uses User32.dll and it is in every application address space. The system maintains a seprate hook chain for each type of hook in the system. The hook chain contains a list of poiters to the hook procedures. When a message or event occurs the system passes it to the hook procedure in the hook chain. The SetWindowsHookEx() function is mainly used to installs a hook procedure at the beginning of a hook chain. For example we can use the WH_KEYBOARD as the first parameter of the SetWindowsHookEx(), which defines the type of event message that will trigger the hook. It will allow us to monitor or modify all the keyboard keystroke messages. So when ever an keyboard event occurs the system calls the procedure from the hook chain associated with the hook. The return value of the SetWindowsHookEx() is a pointer to the hook. We must save this value to perform a number of task. One of the most important task is to call the CallNextHookEx() which calls the next message in the message queue. If we don't call the CallNextHookEx() then the hook procedure will exit without processing the other messages in the message list, making the windows crash or causing errors.

In the DllMain() function we can write our own code to alter or to change the keyboard messages. For each key pressed in the keyboard will be passed to the message list for processing. If we want to alter all the messages in the message list from lower case to upper case characters, we can write the code in the hook procedure in such a way that the ascii value of a particular lower case character can be changed to its corresponding upper case ascii value. For example the ascii value of 'a' is 97 and its corresponding upper case ascii value is 65.


In this report you can see the various ways in which user mode hooks are implemented and the ways to get access to the target application address space. User Mode Hooks compared to Kernel Mode Hooks are simple and also they got some limitations. The User Mode Hooks can be easily detected by enumerating the hooks installed. Hooking API function calls enables us to track invisible actions that occur during the API call. It also gives us extended functionality by allowing us to add pre- & post processing in the original API functions.