Odt Enhanced Agent Based Intrusion Detection System Computer Science Essay

Published: Last Edited:

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.


This coursework covers the different some of the most common attacks perform by hackers like Tenel, FTP attack, some Reconnaissance techniques and Denial of service attacks. The lab scenario is build by using vmware to create virtual machines acting as target systems, Wire shark as sniffing tools and some other hacking tools like namp are used to perform different attacks, Snort rules are applied in implementation section to tune Snort to detect the attacks and generate the alerts which are then logged into the log file. In Evaluation section the output/alerts generated by snort rules are explained.


With current expansion of networks in term of the design and complexity various factors need to keep in mind while designing the network in terms of security. The security threats are not only external but can be internal also, often organisations are concern about the security threats from user sitting inside the network, now a days the interest is not limited to mitigate the threat but also to detect it so that situation can be analysed and further actions can be taken. This detection system is often called intruder detection system or IDS. IDS provide various strategies to detect these threats and attacks. Threats can be attacker, hacker, virus, spy ware, worms and many more. This article covers some of the most common ones and how Snort can be configured/tuned as IDS to detect and report them to network administrator.

IDS can be categorised into to main categories HIDS and NIDS

Host Based Intrusion Detection System (HIDS): A special agent is installed on the potential victim system to detect the attacks performed onto it.

Network-Based Intrusion Detection Systems (NIDS): A special agent is installed on a server in the network which sniffs all the packets entering the network and then look for particular pattern matching to a threat profile, once a threat is detected it is then reported to network. Although NIDS is easy to implement and much scalable but still it cannot sense the threats entering the network via encrypted tunnels targeting the victim host sitting inside the network.

The lab scenario used in this article is build to use Snort as HIDS, Snot is installed and configured on each of the host systems to detect the threats.


Snort can work as both NIDS to monitor your network and HIDS to monitor your system. Snort is easily configurable; Configuration and rules files can be easily opened in notepad and edited according to the requirements. The advantages of using Snort as IDS are listed below.

Snort is open source software and it's free.

Snort is widely in use by open source community for testing, development and in use by enterprises as IDS, This can be checked on www.snort.org showing thousands of downloads.

Snort is constantly updated.

Snort rules are regularly updated and new signatures are available for download

Snort can be installed on multiple environments Linux as well Windows

Requirements Analysis and Design

The below diagram shows the lab design, here virtual machine software called vmware has been used to run Windows 2003 and Ubuntu server as potential target machine and both servers are running Snort as IDS.

The main machine is physical machine setting on the same network of target machine from here intruder try to hack into its targets, on vmware virtual Ethernet connects to the main machine on external network and targets on internal. In window server FTP service is not installed by default so it has been enabled via add/remove program in control panel. Wireshark is used as sniffing in this scenario on victim windows server virtual machine to give to capture the activities of the intruder.

Snort Requirements

Snort can also be used like a sniffing tool like wireshark to sniff the packet (capturing network traffic) of the system interface, Snort needs application programming interface (API) called pcap .The pcap library is implemented in the unix system called Libpcap. On window operating system winPcap must be installed before snort, The WinPcap library supports the saving of the captured packet to a file and one can read these saved files for analysis.


In this sections different attacks like telnet, ftp, host scanning and Denial of service attacks are discussed in this section, these attacks are first generated from a virtual machine and then some rules are discussed here which are used in Snort to detect these attacks so that they can be prevented.

FTP Attacks

FTP is one of the most common, simple and powerful protocol which is used to transfer files over the network. FTP is based on Client / Server architecture where a client starts an FTP connection on TCP port and first sends the control information which contains userid, password, target file and the action needs to be done, and then in second TCP connection the data is downloaded, Although Snort simple, strong and with no overheads but at the same time its quite venerable because all data including control (userid, password) are sent in clear text with no encryption.

This are easily readable from the packets captured using any sniffer tool (here used wire shark) when the topology machine tries to ftp the virtual machine.

So we need some encryption mechanism to be in place so that the sniffed information can not be used even in it is hacked.


When using rules for Snort as IDS they can not prevent the hacker to see the information from the sniffed packet, everything in control and data will go as it is in clear text but using the rules discussed below an alert is generated in event when hacker is trying different passwords to login (which is called Brute force attack)

alert tcp any 21 -> any any (msg:"someone has tried entering five times in 60 seconds";content:"530 Login incorrect.";threshold: type threshold, track by_dst, count 5, seconds 30;sid:970;)


Brute force Attack - In this when an attacker tries to get into the system by trying 5 times wrong password the rule generates the following alert.

Using these logs the network administrator can keep an eye on hacker activity and can take the necessary actions.

Telnet Attacks

Telnet is a protocol very similar to FTP but instead of giving access to a file or folder to download (or delete or anything) it gives user ability to login remotely to the system and give access to all the data and programs installed there. The main security issue with telnet is also the same i.e. it sends the userid and password information in clear text which can be read by the intruder using any sniffing tool. That's why now a days SSH is mainly used as an alternative as it is much secure than telnet.

By analysing sniffed packets when a talent request is made the intruder can easily know all the characters in the password as shown.


This section explains both types of TELNET attacks and Snort rules to detect them

Brute force attack - In this arrack the intruder try different passwords from the dictionary (same as in FTP). Below Snort rules alerts/ logs can be genrated to give warning to the network administrator about this activity.

alert tcp any 23 -> any any (msg:"somebody tries to connect five times in 60 seconds";content:"Login incorrect";threshold: type threshold, track by_dst, count 5, seconds 30;sid:991;)

Access of the root user to the system - Whenever anybody logs on to the system using root as password the rule below will generate the message to inform the network administrator about the same.


Brute force attack- Below alert is generated in the log file of Snort for 5 telnet attempts in 30 seconds

Access of the root user to the system - Once the intruder has hacked into the target system and then try to login as super user the below alert is generated in the log file.


Reconnaissance techniques are widely used by hackers to gather the information about the live IP addresses, their active and running ports and the OS running on them and their related services. These are broadly classified into Host scanning and Port scanning categories in this section both attacks are discussed, scanning tool called nmap is used to perform them.

Host Scanning - Gathering information about the live systems on the network ad is done using process of ping sweep, here the intruder pings the network IP address so that ICMP request is sent to multiple hosts and all the live systems will return the with ICMP ECO reply hnce giving intruder information about the live hosts.


Using nmap the intruder can easily discover the live systems on the network

Below rule is applied on Snort to detect a host scans, the ICMP ECO reply type is 8 and content is "abcdefghikklmnop", when Snort matches this content an alert is then generated

alert icmp any any -> any any (

msg:" ICMP Echo Reply "; itype:8; content:"abcdefghijklmnop";

depth:16; sid:999;)


The alert generated by Snort in the log file when an ICMP ping is detected is shown below

[**] ICMP Echo Reply [**]

04/15-00:14:03.821509 ->

ICMP TTL:128 TOS:0x0 ID:485 IpLen:20 DgmLen:60

Type:0 Code:0 ID:1 Seq:62 ECHO REPLY


These alerts can inform network administrator about the host scan done on the network

Port scanning

Port scanning is a techniques used by hackers to check for all the open ports on the target systems and hence they can know about the running services, the port scan software scans for all the ports by sending series of messages and get the following response from the target system

Open or Accepted -the target host tell that these ports are open and in use

Closed or Denied or Not Listening -the target host tell that these ports are not in use

Filtered, Dropped or Blocked -when there is no reply from the host from these ports

Although the most common practice for port scanning is TCP scan where the intruder uses any TCP scanning tool to send series of request messages to the target system and the target system will complete the three way TCP handshake and then drop the connection, giving the intruder about all the information about the open ports.


When a port scan is applied to the target system using nmap, the intruder can get information about all the open ports and services running on it, as shown

In Sort there are inbuilt mechanisms called softportscan preprocessor and stream5 to protect the system from TCP and UDP port scans shown above. Preprocessor has got three components which can be configured to detect port scan attacks

Here the preprocessor look for all specified the protocols which Snort needs to deduct with a sense level and then result is logged into the portscan.log file


When a port scan is detected on the host a prtscan.log file is genrated in Snot by preprocessor.

Denial of service attack (DOS)

In DOS attack the intruder can make the system unusable by overloading the resources and eventually slowing it down, Here intruder aim is not to get access to the system but to crash it, Although there are different types of DOS attacks commonly used by hacker but only SYN flooding is discussed in this section

SYN flooding Attack

A TCP session is established using three way handshake, as shown in below diagram in red

Host X send the SYN packet to request a TCP session,

Host A sends back SYN/ACK to acknowledge this request

This is then acknowledged back by Host X and TCP session is established

When Host A receives the SYN request from X, it keeps track of the partially opened connection in a "listen queue" for at least listen queue for at least 75 second, the intruder can exploit this small size of listen queue sending multiple SYN requests to a host, but never replying to the SYN&ACK. Since victim can handle a limited number of TCP connection requests in one time its queue is then quickly filled up and it will start dropping the further TCP connection requests which can be of legitimate user.


The SYN flood is applied to the target system using a tool called Longcat, when this tool run it ask for the target ip address and the port no of the victum machine and then it ask the no of threats (syn floods) to be generated as this tools starts it is observed that the cpu utilization of the virtual machine goes on increasing to 90%

The packets are analyzed in wireshark as shown, it can be seen that there is a series of SYN and SYN, ACK going on with no final ACK coming back to establish the TCP session on port 80.

To detect the SYN flood (dos attack) the snort is configured with the rule to match

tcp flood packet more than 20 times in one second and if it detects as syn flood DOS attack raise a alert is generated

Alert tcp any any -> any any (msg" TCP SYN floodS (DOS)";flags:s,12;threshold:type threshold,track by_dst,count 20,seconds 1;sid:989;)


Log below shows how Snort has detected the SYN flood in the log files


This course work no only provides exposure on Snort working and dealing with the attacks but also shows in detail how these attacks can be performed and threat to any organization. Although it can be bit tedious to run Snort for the first time as it need some tweaking in .config file to make it work properly still its one of the most simple and easily configurable IDS around. This article explains how Snort can act as HIDS can easily detect even the attacks performed by intruders via encrypted tunnels. All the different types of attacks are explained, performed, analyzed and then Snort is configured to detect them.

This article uncovers the vulnerabilities of FTP and Telnet protocols being sending everything in clear text. Using nmap for showing Reconnaissance techniques has also shed some light on how an intruder can gather the information about live network systems and their ports. Apart from that the wireshark logs has proved to be quite handy to analyze TCP three way handshake, clear text data in FTP and telnet, looking into these logs make it easy to understand network communications and how rules can be developed to detect different kind of threats. Practically installing running these attacks on virtual machine can prove to make the system bit slowl specially while performing DOS attacks.

All these tools can help in understanding threats in a better way and eventually can help in developing rules for Snort making it even more strong IDS, In the end it can be concluded that Snort configured properly could prove to be a great IDS making it much easier for network administrator dealing with security threats.




E_Security course book by Module Leader: McCarra Greg, Merchiston C.38,

Module Author: Prof. Buchanan William J.

Module Number: CSN11102

Version: Semester 2, 2009/2010