The Buffer Overflow And Nidps Computer Science Essay

Published: Last Edited:

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.

Currently, the information security has become a serious issue, the hackers though variety kind of means to attack the network and system. One of the most common attacks is the buffer overflow vulnerabilities. In many buffer overflows, the most representative one is the stack overflow issues [4]. At present, the buffer overflow vulnerability attacks have accounted for more than half of the number of attacks of the entire network [8].

Intrusion Detection and Prevention system is a network security technology initiative to protect themselves from attacks. As a reasonable complement to firewalls, Intrusion Detection and Prevention technology can help the system to deal with network attacks, extend the security management ability of the system administrator (including security auditing, monitoring, attack recognition and response), to improve the integrity of the information security infrastructure. The Intrusion Detection and Prevention system has been developed in order to deal with various network attacks such as buffer overflows [15]. There are variety kinds of Intrusion Detection and Prevention system has been employed to protect the network and system.

Buffer Overflow

Buffers is the data where can temporarily store, the space of the buffer is limited, generally the data stored in the buffer could not exceed the maximum capacity of the buffer space (Wikipedia, 2012). The buffer overflow occurs the program store the data larger than the maximum of the buffer. The capacity of the buffer cannot accommodate the redundant data, the data are lost, the buffer overflow occurred [2].

In the computer, when the data exceeds the maximum capacity of the buffer, the last data can overflow into nearby memory to corrupt valid data even to change the path of execution files and instructions. The vulnerabilities of exploit buffer overflow you can inject arbitrary code in the program, allows remote access, the program data become extremely unsafe [4]. In recent years, the buffer overflow is extensive used by hackers. Generally, the target of buffer overflow attacks is to get the privileges of a program, the attacker can control the program, if the enough privilege in a program, the attacker could achieve control of the host.

Most buffer overflow attacks try through the process to undermine the logic of the program [7]. For example, an attacker might try to execute the process of executing instructions, but the instructions are not belong to the originally program code or try to change the order instruction process. The attacker could use the code which is a part of the existing computer system, or inserting a new code into the computer system.

Generally, it is not feasible to the attacker to position the code in a system. In order to be able to achieve the attack, the attacker may use static buffer overflow injected code into the system [14]. The attacker can use the code in the system, then use of the system. The attacker can take advantage of the library used by the program, and re-directed the execution of the program and function, but this is not part of the actual program logic, we call this code redirection [15].

A typical buffer overflow virus is the Code Red. In 2001, the "Code Red" warm began to spread in the network. The worm relies on the "buffer overflow" bug, and it was depend on the Microsoft IIS server on the network to spread. This worm through the server's port 80, port 80 is also the Web server and browser information exchange channels (Wikipedia, 2012).

There are variety kinds of technologies to address the buffer overflow.


DEP (Data Execution Prevention) developed by Microsoft, it is a Windows security mechanisms, mainly used to prevent viruses and other threats to cause the damage to Microsoft platform [4]. The DEP first applied on Microsoft Windows XP SP2, now it has been extended to Windows 7. Unlike the anti-virus software, the purpose of hardware and software DEP is not to prevent unwanted programs installed on your computer, but monitor the software already installed in your computer, help determine whether the software safe to use system memory [4].

There are some limitations of the Data Execution Prevention:

DEP only able to prevent an intruder malicious code push into the stack, but in return libc or dll, the program execute an attack, the attacker to use the instruction has been already loaded, the DEP will not be able to protect the system from attack.

The Default of the DEP only to protect the Windows system components and services, other programs will not be protected.

The DEP requires the CPU to support this technology, not all of the CPU compatibles the DEP technology, especially some of the older CPU could not support the DEP.

There are some compatibility reasons, not all the software be able to support the DEP, Windows operating system cannot open the DEP protection for all processes, especially may cause the abnormal to the third-party software.

ASLR (Address space layout randomization) is a buffer overflow security protection technology. It is through the linear region of the heap, stack, and shared library mapping layout randomization to increases the difficulty to predict the destination address of the program [3]. It is also prevent the attacker directly locate the attack code. The ASLR can effectively reduce the success rate of buffer overflow attacks, now the Windows, Linux, FreeBSD, IOS, Android and other major operating platforms have adopted the technology (Wikipedia, 2012).

The limitations of the Address space Layout Randomization:

For Microsoft windows platform, only in Windows Vista and subsequent versions of Windows (such as Windows 7) can support the ASLR security features.

When use the ASLR, we need to combine the ASLR and DEP together. If the CPU does not provide the hardware support of the DEP or application no choice DEP protection, malicious code can be easily executed.

Same to the DEP, not all the software is able to support the ASLR technology.


Canaries are also called Stack Guard or stack canary system, the key of the stack canary system is the system randomly generates a value. This value is referred to as Canaries, placed in the stack top of the return address, when a function is ready to return it will make the comparison of the randomly generated value and which the value already stored in the local [1]. If the two values different, the return address will be invalid, and the system will not use this value corrupted. For most situations, stack canary system is able to play a protective role for many existing viruses. This is mainly because the almost of the viruses are depend on the over write to get the return address then achieve the control of execution right [10].

There are three different types of the canary: a random canary, a random xor canary and a terminator canary [11].

A random canary is to randomly select a value at a runtime. The random value ​​is stored in global variables; it is applied to each function of the program. In the same way save as Terminator Canary. To be assumed that, an attacker cannot overwrite the global value or lead to the disclosure of the value. In some cases, this is feasible. However, use the format string mandatory program leaked random value to overwrite global variables is not necessary, because an attacker can just as easily overwrite a function pointer [11].

The random XOR canary: in the latter attack, the data provided by the attacker as a string or storage replication destination address, the attacker can overwrite the data pointer, you can modify the pointer to point directly to the saved instruction pointer. When the attacker's data copied to the saved instruction pointer, the saved instruction pointer will be over write without modify the canary. When the system is running again, it will generate other random values, stored in a global variable; a random value xored with the saved instruction pointer the result will be stored in the stack. At the end of the function, the saved canary xored with the random values, the result also compared with the saved instruction pointer. If the two values ​​are different, the handler will be called and the program is terminated.

A terminator canary contain multiple the terminator NULL bytes or wrap used to indicate the end of the string in the C library string functions [11]. Generally, these values ​​are used to terminate a string; the attacker could exploit the buffer overflow to modify them. If an attacker run multi overruns in a function¼Œit is possible to repair the terminator canary. The first one will be used to change the instruction pointer and others overruns could be used to address canary by lining up the terminator in the string with the corresponding value in the terminator canary.

The limitation of the Canaries:

The canaries are also can be attacked. In terminator canary, the attack which can destroy the canary and guess the value of its storage, then the attacker could misuse of the canaries.

Host-based intrusion detection and prevention system

The Host-based Intrusion Detection and Prevention System usually treated as exclusively to a host system application [9]. On the host, the logs of the system, procedures of the system, or other resources of the system almost to be monitored by the HIDPS. The Host-based Intrusion Detection and Prevention System totally rely on the signature database; all the attribute information of the files and programs will be stored in the database [6]. The intrusion detection system make the comparison of the current file signature and the saved signature, if not the same then the current file will be illegal.

Compare with the Network based Intrusion Detection and Prevention System. There are some weaknesses on the HIDPS. For the majority of network, there will be variety kinds of host with the same level in the network, each host need the independently HIDPS [16]. It will be a high cost-consuming, the network manager also required for maintains each host of HIDPS, and it will take a very long time. The Network-based IDSs are mostly passive devices that monitor ongoing network activity without adding significant overhead or interfering with network operation 16.


NIDPS is based on NIDS, usually placed it between the edge of the network, in the middle of the internal network and external network [16]. NIDPS monitor, detect all packets from the external network and internal network to prevent intrusion. For the good organized network and good layout of the network environment the NIDPS can enable an enterprise or organization to use a few devices to monitor the large network. It also captures the flow of data from the network to the host, to analysis a specific signature or unusual and abnormal behavior. Once any suspicious or unusual behavior could be trigger the alarm and messaging to the central computer system or administrator (monitoring IDPS), and then automatically generate a response.

There are variety kinds of intrusion detection and prevention systems.

Signature Detection

Snort is an open source network intrusion detection and prevention systems, it is the core of NIDPS [3]. At the first time, Snort was written by Martin Roesch, but now it is developed by Sourcefire. Snort is an IPS tool, it is relying on the signature-based technology, all the signature stored in the database, snort can conduct real-time analysis of network traffic and data flow [3]. It is involve the protocol inspection and analysis, to detect different types of attacks. The snort typically follows the user-defined rules to checks the data packets. Snort rules can be written in any language, the structure of the snort rules are good, the administrator it can be easily read and modify the rules. In buffer overflow, it takes the appropriate measures to prevent buffer overflow, attacks can be detected with the match before the attack mode.

There are several advantages and disadvantages of the Signature Detection.

The advantage:

1. The signatures are easy to understand and develop.

2. Accurate to analysis and judgment of the known intrusion behavior.

The disadvantage:

1. Analysis of the data is completely dependent on the hardware system, the more data traffic the more system resources consumed; it is also affect the system speed.

2. Intrusion detection is totally depending on set of rules in the system, it cannot be able to take the initiative to learn and upgrade the system rules. Generally it will lead to a miscarriage of justice and does not detect new intrusions.

Anomaly Detection

Anomaly detection is to store the normal behavior of the storage system, such as kernel information, the log of the system, network packet information, software running information, and operating system information stored in the database [12][13]. When there is any abnormal behavior or intrusive activity, the system will deviate from the normal behavior of an alarm, marked as active activities exception.

The advantage of Anomaly Detection:

Be able to detect new threats and intrusion

The Anomaly Detection can also monitor the unknown attacks

Try to detect overflow attacks from the network, if the attack is existing, Anomaly Detection will generate a warning, and discarded the packet.

The disadvantage of Anomaly Detection:

It is difficult for the administrator to defining rules of the network.

Depends on the behavioral characteristics, it need to collect and store large amounts of behavioral characteristics to make the recognition rate and accuracy rate.

Additionally content

The buffer-overflow stills a serious issue for all operating system platforms, a large number of computer viruses rely on the buffer over-flow to control over the system [17]. There are varieties kinds of technology try to reduce the problem or even attempt to address the issue. However, it is still not effective enough to address the issue. Currently, with the rapid development of the cloud computing, many security vendors rely on the cloud computing technology to provide the cloud security service to their customers.

Generally, the NIDPS is effectively to address the buffer overflow and to detect the worms. In my point of view, apply the cloud computing and cloud security technology on the NIDPS. With the support of the cloud computing and cloud security, the NIDPS greatly enhance the speed and accuracy of the data analysis and process; reduce the misjudgment of legitimate data and behaviors make the detection of buffer overflow and worms or other types of treats more convenient. The NIDPS is weak to work at large amount of the network data traffic transferred on the network or the network is running at full capacity state, the NIDPS often not able to monitoring and analysis of the data in a real-time, even worse with the error of NIDPS it may stop working, the network lose protection. Combine the cloud computing and NIDPS together, the data will be dispersed to different hosts of the cloud, rather than just to rely only NIDPS equipment to handle these data.

The greatest drawback of the NIDPS is the new attacks and some variants of known threats still not detected and intercepted by the NIDPS. The combination of the cloud computing and NIDPS will provide more signature and more details to the NIDPS. The NIDPS also get a real-time upgrade, timely response to the new buffer overflow attack and intelligent identify the variants of the known attacks, the NIDPS become more and more smart.


This research to analysis the hazards of the buffer-overflow, in the future the buffer over-flow will be more dangerous, the attacker through a buffer overflow could launch a larger attack to the system or network. NIDPSs are widely used in the large company or some enterprises. The Signature Detection and Anomaly Detection are the most commonly technology of the NIDPS, the NIDPS can monitor and detect the almost common attack such as buffer overflow and worms. However, nothing is perfect, the attackers are becomes more and more intelligent, the attack also become more and more smart. Currently, the attacker not just to make use of a single attack, but combine a variety means of attack together to give more powerful attacks to the network.

The cloud computing play a significant role in network security. In this research, I suggest that, combine the cloud computing and NIDPS together to make the analysis and detect of the network behaviors more accurate and more efficient. Perhaps this is not the best solution of buffer-overflow and network attacks. We need more advanced technology to solve these problems.