Buffer Overflows In C C Programming Vulnerability Attacks And Defenses Computer Science Essay

Published: Last Edited:

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.

For the past four decades, C and C++ have retained their status as the most widely used high level programming languages. Both these languages are vital in the development of applications that require high computational efficiency as well as the capability to more closely interact with the operating system. To achieve this, easy access to memory is facilitated by the inherent direct memory addressing methods of C/C++. However this ease of directly accessing memory poses several security vulnerabilities. Of these, buffer overflows are the most significant. This report discusses in detail the various buffer overflow vulnerabilities that exist, possible attacks that exploit these vulnerabilities and finally the security measures that can help alleviate these vulnerabilities without affecting the performance of software applications.

According to Microsoft Corporation, a buffer overflow "is an attack in which a malicious user exploits an unchecked buffer in a program and overwrites the program code with their own data. If the program code is overwritten with new executable code, the effect is to change the program's operation as dictated by the attacker. If overwritten with other data, the likely effect is to cause the program to crash." [1]

The vulnerabilities posed by buffer overflows in C language programming were identified for the first time in 1973. However the very first instance of buffer overflow exploitation was observed in 1988, when the notorious Morris Worm penetrated and subsequently crashed around six thousand systems in less than 24 hours. This worm operated, in part, by overflowing an unchecked buffer, initialized by the gets( ) C language function call in the UNIX fingerd daemon process. In 1996, a hacking enthusiast named Elias Levy wrote a highly controversial article titled "Smashing the Stack for Fun and Profit" which offered a thorough insight into methods for exploiting buffer-overflow. Later in 2001, the Red Code worm exploited a buffer overflow vulnerability to infect systems running Microsoft's Internet Information Services web server. Of lately, the buffer overflow vulnerabilities in Microsoft Xbox gaming console have reportedly been exploited in a bid to allow pirated gaming software to run on these systems.

Figure : CERT Advisory Statistics (Src: CERT Report dated Oct 16, 2003)

During the last four decades, buffer overflow has emerged as most common mode of mounting security attacks. This is because such vulnerabilities are not only widespread, relatively easy to exploit, but chiefly because most system and application software are based on C/C++ languages which have frameworks inherently susceptible to buffer overflows. This is so because these languages do not perform implicit bounds checking, provide standard libraries function calls that do not enforce bound checking and define strings as null-terminated arrays of characters [2]. The relative ease of exploiting this vulnerability makes it the method of choice in mounting a remote penetration attack wherein the objective is to gain root level privileged access to a system. Once this materializes, the attacker can manipulate files, alter registry data, install new code or crash the system itself.

In fact, statistics from CERT show that from 1999 till the end of 2003, buffer overflows accounted for almost half the vulnerabilities cited in their advisory reports, peaking at 74.1% in 2003, followed by 54.1% in the year 2002.

Buffer Overflow Vulnerability

A vulnerability is formally defined as a set of conditions that allow an attacker to violate an explicit or implicit security policy [2].

The buffer overflow condition arises when a programme attempts to store more data in a buffer than its actual storage capacity or in other words, when a programme tries to store data in the memory region beyond the space allocated to the buffer itself. A buffer is in fact a continuous region of memory set aside to store finite amount of any type of data.

Programmes compiled in languages such as Java, Ada, Pascal and C# not only identify buffer overflows, but also generate exceptions in such cases. On the contrary in C and C++, there is no automated mechanism to monitor bounds on a buffer. This means that the user can write past a buffer which may result in corrupting the data in adjacent memory block, crashing the programme or initiating execution of some malicious code instead of process execution code.

Consider for instance the following simple C++ language code.

void main () {

int buffer[4];

buffer[10] = 5;


Ostensibly, it is a valid C code and therefore any compiler should compile it without generating an error. However, it can be observed that the third statement attempts to write a value to a location that is outside the bound of the buffer array. This causes the programme to behave in an unpredictable manner.

In order to understand how this vulnerability can be exploited, it is essential to have an overview of process memory organization as well as a typical function call.

An overview of process memory

A process is any program that has been loaded into memory and managed by the operating system [3]. Whenever a program is started, its binary code is loaded into memory and executed instruction by instruction. This program in execution including up to date values of the program counter, registers, and variables, is termed as a process [4]. The various micro-kernel operating systems of present day such as Windows, Linux, Solaris, Mac OS etc allocate virtual memory for each new process. This virtual memory is translated onto physical memory addresses by the Memory Management Unit of respective system. The process memory is organized into five distinct segments as depicted below.

Figure : Process Memory Organization [4]

At the top of the process memory organization is the Code Segment. Often marked as a read-only [4] region, this segment holds programme instructions in binary format. An attempted modification of its contents causes a Segmentation Fault to occur.

The Data Segment contains static and global variables that have been initialized to some value. Whereas uninitialized global variables reside in the adjacent BSS (Block Started by Symbol) Segment.

The Heap Segment contains any variables that have been assigned memory dynamically during process execution. In C, dynamic memory is allocated using the malloc() function while de-allocation is undertaken using the free() function. In C++, new and delete operators are used for these purposes. A heap grows from lower memory addresses towards higher memory addresses.

The Stack Segment contains automatic variables that are created as a result of a function call. It also stores the function arguments as well as saved programme status. This information pushed onto a stack is collectively referred to as a frame. In contrast to a heap (a FIFO data structure), a stack is a Last in First Out data structure and grows from higher towards lower memory addresses.

A typical function call

Functions or subroutines are an integral feature of any modern programming language. A function is essentially a section of a programme code that facilitates structured programming. Whenever a function is called, certain information collectively known as a frame is pushed onto the process's stack segment. After the function completes its task, control is returned to the calling programme while simultaneously, the frame is popped off the stack.

A stack frame contains function parameters, automatic variables, as well as process status information. The process status information comprises the Return Address (RA)/Return Instruction Pointer (RIP) and the Saved Frame Pointer. RIP is the address of the next instruction in the calling routine that will be executed once the current subroutine exits. SFP is the starting address of the previous stack frame. In Intel's 32 bit architecture, the registers %eip (extended instruction pointer) and %ebp (extended base pointer) are respectively used to store RA and SFP.

void foo(int a, int b, int c)


int bar[2];

char qux[3];

bar[0] = 'A';

qux[0] = 0x2a;


int main(void)


int i = 1;

foo(1, 2, 3);

return 0;


main: foo:

pushl %ebp pushl %ebp

movl %esp,%ebp movl %esp,%ebp

subl $4,%esp subl $12,%esp

movl $1,-4(%ebp) movl $65,-8(%ebp)

pushl $3 movb $66,-12(%ebp)

pushl $2 leave

pushl $1 ret

call foo

addl $12,%esp

xorl %eax,%eax



Consider the above C language programme as an example, accompanied by its corresponding assembly language code[4].

First we consider the main( ) function execution. As part of main( ) prologue, the present frame pointer is pushed onto the stack (line 1) and the new frame pointer is set to the current stack pointer %esp (line 2). It is noteworthy that the stack pointer always points to the top of the stack. In line 3, memory is assigned for the local variable i while in line 4, i is assigned the value 1. In the next three steps, arguments for the function foo are pushed onto the stack. Thereafter the function foo() is called and the return address of the calling function (main() in this case) contained in register %ebp is pushed onto the stack. As part of foo() prologue, 12 bytes of memory is allocated on stack for (line 3). Ins (8 bytes for int array variable bar, and 3 bytes for character array qux). Memory is allocated on the stack in multiple of 4 bytes so 12 bytes are allocated instead of 11.In the subsequent next two lines, memory is allocated to the variables. The epilogue of the function foo() or for general purpose any function comprises the leave and ret instructions. The leave instruction entails restoring the value of stack pointer to the old frame pointer (pfp). Thereafter, the ret instruction executes and popping the return address in %eip into the programme counter. Once back in the frame of main() function, the memory allocated to the three arguments of foo() before calling it is cleared up, by adding 12 bytes to %esp.

Figure : Status of Stack for the example code (Prologue execution)

Then the main() routine's return value of 0 is saved in the %eax register before main()'s epilogue which entails the same operations discussed before.

Figure :Figure 3: Status of Stack for the example code (Epilogue execution)

Buffer Overflow Attacks

All known attacks exploiting the buffer overflow vulnerability generally take place in two steps

Altering the flow control of the programme

Execution of some malicious code (provided by the attacker) that operates on data.

The most well known attacks that exploit the buffer overflow vulnerability in C/C++ are discussed in detail.

Stack Smashing

In this type of attack, a buffer is overrun to overwrite data in the stack segment. In this way, the values of automatic variables stored in the stack segment (as discusses previously) can be modified. This causes a loss of integrity, eliciting unintended programme behaviour.

An attacker may also overwrite the return address in a stack with the consequence that once the current function returns, process execution starts from the modified return address supplied by the attacker. Consider the following C language code [4] that highlights this vulnerability.

void foo(char *args)


char buf[256];

strcpy(buf, args);


int main(int argc, char *argv[])


if (argc > 1)


return 0;


Figure : Stack Smashing Buffer Overflow Vulnerability Exploitation [4]

The function foo is supposed to copy the string arg passed as its argument to the fixed length string buf of length 256 bytes. This is accomplished using the strcpy function which copies characters into the destination string till it encounters a null character in the source string. An attacker can provide a source string so as to overflow the buffer, thereby modifying adjacent memory space. As shown in figure b, the buffer's adjacent memory chunks in the stack are the 4 byte SFP followed by the Return Address (RIP: Return Instruction Pointer) of calling function ( main() ). If the buffer is overflowed by 8 bytes, (i.e. input is of 264 bytes), then the Return Address is overwritten by the last 4 byte of attacker's input string. Consequently when the function foo( ) returns, the overwritten value of Return Address is popped into the IP register and the code at that address is executed. A plausible instance, the attacker can fill the 256 byte buffer with shell code, while the return address is over written with the starting address of the buffer in the stack. Hence when the function foo() returns, and RA is popped into the IP register, the attacker's shell code is executed.

Arc Injection

This technique, sometimes also referred to as return-into-libc involves transferring control to a piece of code that already exists in the memory. No executable instructions are inserted in the buffer as was previously seen in case of stack Smashing code injection. In contrast arc injection merely modifies the program's control flow graph by inserting a new arc as opposed to inserting also a new node which is the case in Code Injection [4].

A possible scenario involves overwriting the return address on the stack with that of some function existing in c library such as system () or exec, which can subsequently be exploited to execute other programmes already in the system. In other words, control is passed onto a C library function. Consider the scenario depicted in the following figure. By overflowing a fixed length buffer, crafted the return address is overwritten with the address of the c library function system(). The address of a possible argument "/bin/sh" to this function is also overwritten in the adjacent memory chunk in the stack. As a result, when the current function returns, control is transferred to the system() function which on the basis of the argument "/bin/sh" opens the shell programme. Here a question arises as to which address should the system function return, when it accomplishes its task. In all probability, this possibility doesn't exist. Therefore an attacker can provide a spurious return address for the system function in the 'crafted' stack frame.

A significantly powerful arc injection based attack can be launched by chaining together calls to functions. This requires the attacker to provide a valid return address for the C library function which can result in consecutively calling functions one after the other. An example of this is depicted in figure 6. wherein, the attacker first overwrites the return address with the address of the system function setuid(). The immediate block in stack , supposed to contain the return address to which execution should jump to when setuid() returns. In this case, the address of system function is placed here. The next adjacent block contains the address of argument ('0') meant for setuid() while the argument for system function ('/bin/sh') is conveniently placed in the next block. Through this chaining methodology, the attacker first elevates his privilege level and then opens up the shell command line programme.

In yet another possibility, an attacker can manipulate chaining of C library function calls to install some shell code in the data segment and ultimately transfer control to it. For instance depicted in figure 7. the return address RA is overwritten with the address of the standard function strcpy() with addresses of two arguments (src and destination string) in the data segment overwritten in the next adjacent stack blocks. Copies the shell code in data segment and when strcypy returns, control is transferred to this shell code now resting in the data segment.

Figure : Illustration of Arc Injection Attack (Simple case) [4]

Figure : Illustration of Arc Injection Attack (Chaining of Function Calls) [4]

Arc injection attacks involving chaining are sufficiently complex to mount. In general, such attacks are effective even if the stack segment is made non executable.

Pointer Subterfuge

The term Pointer subterfuge refers to exploits through which the value of a pointer is modified. A pointer is a variable type that stores address of a data structure. These attacks are divided into three main categories

Function Pointer Clobbering

This technique involves overflowing a buffer in order to overwrite the value of function pointers in a code. Consider the function definition for foo() [4].

void foo(void *arg, size_t len)


char buf[256];

void (*f)() = ...;

memcpy(buf, arg, len);




wherein a buffer/character array of 256 byte buf is declared, followed immediately by the declaration for function pointer *f. In the third line, memcpy() function performs the copy operation into buf without any implicit bound checking. Thus there's a possibility of a buffer overflow here which can be exploited to overwrite the value of *f with some attacker provided address, so that when memcpy returns the control is transferred to the attacker's code when f is called in the next line

Data Pointer Modification

This technique is elaborated by the example C language code [4] given below.

void bar(void *arg, size_t len)


char buf[256];

long val = ...;

long *ptr = ...;

extern void (*f)();

memcpy(buf, arg, len);

*ptr = val;




The variable val and the pointer *ptr that is later used are both declared after the fixed length buffer buf. Exploiting the buffer overflow vulnerability in the first line of the bar function definition, an attacker can overwrite both the l-value [1] and the r-value of the assignment statement in line 6, allowing him the possibility of arbitrarily writing four bytes in memory.

Virtual Pointer Overwriting

In C++, virtual functions are widely used as part of the object oriented programming methodology. A virtual function is a function whose behaviour can be overridden within an inheriting class by a function with the same signature [7]. Each class in a C++ program has a corresponding virtual function table providing for the implementation of virtual functions. Every object of a class holds a virtual pointer which refers to the virtual function table of its parent class. Overwriting of an object's virtual pointer can result in modification of program flow control. In most attack scenarios, virtual pointer value is overwritten by some spurious value pointing to a spurious virtual function table that itself may contain several entries. When a virtual function is called, the execution jumps to the value pointed to by the virtual pointer, which is in essence an entry in the associated virtual function table that itself may have been overwritten by the attacker. Hence overwriting the virtual pointer value by a fake value that points to some fake virtual function table address can lead to the transfer of process control to the code in that particular location.

Heap Buffer Overflow

This technique makes use of the inherent flaws in C language's dynamic memory allocation mechanism. The standard C library employs malloc(), calloc(), realloc() and free() functions to support dynamic memory management mechanisms. The memory on the heap is allocated in arbitrary sized blocks known as chunks which can be allocated, freed, subdivided, or consolidated with other chunks. Two free chunks can never be adjacent to one another; instead they're merged together to form bigger free chunks. The structure of each chunk is depicted as follows.

Figure : Structure of a Free Chunk [4]

Figure : Structure of an Allocated Chunk [4]

In addition to holding the payload data, each chunk has fields containing other important information, referred to as boundary tags. As shown in the figures above, each boundary tag contains three pointers (i) the chunk pointer that points to the beginning of the present chunk, (ii) the mem pointer, which is returned to a process that has requested memory to be allocated though the malloc() method, and (iii) the next_chunk pointer which points to the start of the next chunk[4]. The prev_size field indicates size of the previous chunk in case it is free. Otherwise, it contains last four bytes from the preceeding chunk. The size field occupies fifteen bits and holds size information for the present chunk. The PREV_INUSE bit indicates whether the previous chunk is free or in use. The fd pointer field contains address of the next chunk while the bk points to the start of the previous chunk in the heap.

Figure : A Double Linked Structure [4]

Free chunks are arranged in circular doubly linked list configuration, also referred to as a bin. A bin or a double linked list comprising of three chunks is as show below. In normal circumstances bins contain free chunks having uniform size. For bins with constituent chunks of different sizes, the chunks are arranged in descending order with respect to the size.

Whenever a chunk is removed from the double linked list/bin the unlink() macro is called[2].

#define unlink(P, BK, FD)


BK = P->bk;

FD = P->fd;

FD->bk = BK;

BK->fd = FD;


Here, P is the chunk being unlinked, BK the back pointer, and FD is the forward pointer. With reference to the figures 2 and 3 it can observed that FD->bk is same as FD + 12 and BK->fd is equivalent to BK +8.

To highlight the potential exploitation of the unlink() macro consider the following piece of code[4].


char *buf1 = malloc(0);

char *buf2 = malloc(256);

char *buf3 = malloc(0);






In the first three lines, memory is dynamically allocated to buf1, buf2 and buf3 string buffers. In the fourth line, user entered string is stored in the buf2 buffer. Finally, the next 3 statements de-allocate memory using the free() function. The structure of the chunk for buf2 is shown for the case when it is created/allocated using malloc() and for the case when its data-payload field gets filled after the gets(buf2) statement execution. In a manner similar to overwriting the buffer to smash a stack, a malicious user can provide an input to overflow the 256 bytes allocated to buf2 and thus overwrite the fd and bk fields in buf2 chunk.

Figure : Status of Chunk's fields before gets(buf2) called [4] Figure : Status of Chunk's fields after gets(buf2) called [4]

When free(buf1) is called, it is checked whether the next chunk is free or allocated. In the latter case (i.e. if free) the unlink() is called in order to consolidate it with the chunk that is to be freed at present. So, to check whether the second chunk is free, PREV_INUSE bit of the third chunk is checked. The unlink() macro shouldn't be called on an already allocated chunk, as it will result in a segmentation fault. In this particular example, when the unlink() macro is called it dereferences (fd + 12) in order to write at that address the value bk (overwritten value of the back pointer). Therefore an attacker can write 4-bytes at an arbitrary memory location.

In this particular example, the fd field is overwritten with &free() - 12, where &free() implies the address of free() in the Global Offset Table [2] . As the unlink() macro when called, adds an offset of 12 in order to access the bk field of the next chunk, this offset is subtracted. In the bk field, the value buf2+8 is overwritten, since the shell code in the data payload field is at a distance of eight bytes from the beginning of buf2. After the data payload field, 4 bytes (in this case four purposeless values of 0x42) have been used to fill the prev_size field. Next, comes the fifteen bit current chunk's size field and then the one bit PREV_INUSE field which is overwritten by the value 0. Therefore when the free() function is called, the execution jumps to the shell code instead of de-allocating the relevant chunk.

Defense Strategies

Secure Programming

High level languages such as C or C++ facilitate flexibility and allow for high performance by providing mechanisms for direct memory access. But this ease of accessing memory directly poses security vulnerabilities that have previously been discussed in detail. Use of inherently unsafe C/C++ library functions such as gets(), memcpy() , strcpy() should always be avoided by the programmer, as none of these undertake implicit checks on bounds of fixed length buffers.

Source Code Auditing

A detailed analysis of a source code can be instrumental in highlighting security vulnerabilities. Code auditing on a line by line basis uncovers many software flaws. But this manual procedure can prove to be tedious and prone to human errors. Therefore, automated code auditing is a more viable option when it comes to analyzing complex programs.

Automated Source code analyzers can be categorized as being static or dynamic. Static tools only analyze a program's source code whereas dynamic analyzers also evaluate run-time errors that may be generated. Static analysis tools can either be lexical or semantic. The lexical analyzers perceive a program's source code as composed of individual tokens. Every token is analyzed individually. Examples of Static Lexical analysis tools are grep, flawfinder and RATS [4]. Static analyzers belonging to semantic class utilize semantic interpretation from a program's control flow analysis and make conclusions regarding the existence of potential vulnerabilities. For example when a program is executed, compiler warnings give semantic errors. Static Semantic Analysis tools like splint feature a rich set of functions for contextual error analysis [4].

Dynamic analysis tools execute the program and evaluate run-time errors that are caused by buffer overflows. Tracer tools are of particular importance in this category. These tools generate logs about flow control and analyze the 'audit trail' [4] . Examples of tools used for dynamic analysis are Electric Fence, Purify, and valgrind.

Binary Auditing

If the high level source code of a program is not available and we only have access to binary code, then performing code audit through conventional means becomes an extremely complicated process. In such a scenario, techniques such as fault injection or reverse engineering are made use of.

Fault injection involves tampering with the program execution environment in order to elicit program flaws that give clues about internal errors. To achieve this, the program is fed with inputs generated by modules known as fuzzers[5], that generate random data to find buffer overflows by chance.

In the reverse engineering approach the binary program itself is examined, but its execution environment isn't manipulated. Disassemblers such as IDA Proand SoftICE etc visualize assembler instructions in an interactive GUI. This allows the analyst to reconstruct a detailed picture from the program's internals[4].

Dealing with unsafe library functions

The inherently unsafe C/C++ library functions can be wrapped by additional library packages. For instance, the approach used by libsafe project involves detecting calls to vulnerable C/C++ standard library functions and substituting them with their 'safe' counterparts in the libsafe library. This wrapping approach is effective against several buffer overflow attacks. But itis limited in scope since it is not feasible to cover all the unsafe C/C++ standard library functions because doing so hampers software performance in some cases.

Use of Compiler Extensions

This approach seeks to extend capabilities of the compiler that in order to provide for measures such as protecting the return address (in a function's stack call frame) from being overwritten, performing bound checks on buffers etc. StackGuard extension to GCC provides a protection method that involves placing a so called 'canary1 value' between the Saved Frame Pointer and Return Address in a stack. Whenever a function returns after completing execution, the 'canary value' is compared to its originally assigned value. In case it differs from the original value, it is established that the Return Address has been modified. Consequently StackGuard causes the program execution to terminate. This method is however limited to only protecting against any overwriting of return address.

Execution Environment Modification

The most well known Execution Environment Modification technique involves making the stack non-executable. As we have observed, most buffer overflow vulnerabilities involve overwriting return address of a function in the stack and replacing it by the address of some injected malicious code, which in most cases is placed on the stack. Rendering a stack non-executable makes it extremely difficult, if not impossible to exploit buffer overflow vulnerabilities. In this perspective, the PaX patch for the Linux kernel offers a plausible solution. PaX sets data memory non-executable and program memory as non-writable. This effectively prevents direct code injection and execution. The success of most buffer overflow exploitation techniques depends on knowledge of certain addresses in the target program. PaX addresses this vulnerability by randomly arranging the program memory, employing a technique known as Address Space Layout Randomization (ASLR). Thus the lack of knowledge of the address space leaves the attacker with no option other than to rely on guessing the addresses. PaX however also has its limitations, discussion of which is outside the scope of this report. Most notably, it doesn't prevent the overwriting of pointers[5].


Even four decades since the introduction of C/C++ programming languages, buffer overflows remain one of the biggest software vulnerabilities. With unprecedented development and expansion of the internet in past 20 years, these vulnerabilities have allowed for the possibility of mounting attacks on remote systems, wherein the objective of an attacker is mostly to inject and execute malicious code on a remote system which in turn allows him privileged access to such a system.

This report offered an in-depth overview of the most well known buffer overflow attack scenarios. A number of defence strategies to deal with these attacks have been discussed. It is noteworthy however, that despite the wide range of mitigation measures that have been devised, attack methodologies capitalizing on buffer overflow vulnerabilities have evolved continuously. One can conclude for sure that even the best and most sophisticated defence measures are rendered useless by the ever morphing exploits. A single defence measure is therefore sufficient to counter all attack scenarios. Therefore most effective defence against this ubiquitous threat should be based on a combination of both static and runtime solutions.