6. Intrusion Detection

6.1. Intrusion Detection Systems (IDS)

Intrusion Detection Systems (IDS for short) are designed to catch what might have gotten past the firewall. They can either be designed to catch an active break-in attempt in progress, or to detect a successful break-in after the fact. In the latter case, it is too late to prevent any damage, but at least we have early awareness of a problem. There are two basic types of IDS: those protecting networks, and those protecting individual hosts.

For host based IDS, this is done with utilities that monitor the filesystem for changes. System files that have changed in some way, but should not change -- unless we did it -- are a dead give away that something is amiss. Anyone who gets in, and gets root, will presumably make changes to the system somewhere. This is usually the very first thing done. Either so he can get back in through a backdoor, or to launch an attack against someone else. In which case, he has to change or add files to the system.

This is where tools like tripwire (http://www.tripwire.org) play a role. Tripwire is included beginning with Red Hat 7.0. Such tools monitor various aspects of the filesystem, and compare them against a stored database. And can be configured to send an alert if any changes are detected. Such tools should only be installed on a known "clean" system.

For home desktops and home LANs, this is probably not an absolutely necessary component of an overall security strategy. But it does give peace of mind, and certainly does have its place. So as to priorities, make sure the Steps 1, 2 and 3 above are implemented and verified to be sound, before delving into this.

We can get somewhat the same results with rpm -Va, which will verify all packages, but without all the same functionality. For instance, it will not notice new files added to most directories. Nor will it detect files that have had the extended attributes changed (e.g. chattr +i, man chattr and man lsattr). For this to be helpful, it needs to be done after a clean install, and then each time any packages are upgraded or added. Example:

# rpm -Va > /root/system.checked

Then we have a stored system snapshot that we can refer back to.

Another idea is to run chkrootkit (http://www.chkrootkit.org/) as a weekly cron job. This will detect common "rootkits".

6.2. Have I Been Hacked?

Maybe you are reading this because you've noticed something "odd" about your system, and are suspicious that someone was gotten in? This can be a clue.

The first thing an intruder typically does is install a "rootkit". There are many prepackaged rootkits available on the Internet. The rootkit is essentially a script, or set of scripts, that makes quick work of modifying the system so the intruder is in control, and he is well hidden. He does this by installing modified binaries of common system utilities and tampering with log files. Or by using special kernel modules that achieve similar results. So common commands like ls may be modified so as to not show where he has his files stored. Clever!

A well designed rootkit can be quite effective. Nothing on the system can really be trusted to provide accurate feedback. Nothing! But sometimes the modifications are not as smooth as intended and give hints that something is not right. Some things that might be warning signs:

• Login acts weird. Maybe no one can login. Or only root can login. Any login weirdness at all should be suspicious. Similarly, any weirdness with adding or changing passwords.

  Wierdness with other system commands (e.g. top or ps) should be cause for concern as well.

• System utilities are slower, or awkward, or show strange and unexpected results. Common utilities that might be modified are: ls, find, who, w, last, netstat, login, ps, top. This is not a definitive list!

• Files or directories named "..." or ".. " (dot dot space). A sure bet in this case. Files with haxor looking names like "r00t-something".

• Unexplained bandwidth usage, or connections. Script kiddies have a fondness for IRC, so such connections should raise a red flag.

• Logs that are missing completely, or missing large sections. Or a sudden change in syslog behavior.

• Mysterious open ports, or processes.

• Files that cannot be deleted or moved. Some rootkits use chattr to make files "immutable", or not changable. This kind of change will not show up with ls, or rpm -V, so the files look normal at first glance. See the man pages for chattr and lsattr on how to reverse this. Then see the next section below on restoring your system as the jig is up at this point.

  This is becoming a more and more common script kiddie trick. In fact, one quick test to run on a suspected system (as root):

  /usr/bin/lsattr `echo $PATH | tr ':' ' '` | grep i--

  This will look for any "immutable" files in root's PATH, which is almost surely a sign of trouble since no standard distributions ship files in this state. If the above command turns up anything at all, then plan on completely restoring the system (see below). A quick sanity check:

  # chattr +i /bin/ps # /usr/bin/lsattr `echo $PATH | tr ':' ' '` | grep "i--" ---i---------- /bin/ps # chattr -i /bin/ps

  This is just to verify the system is not tampered with to the point that lsattr is completely unreliable. The third line is exactly what you should see.

• Indications of a "sniffer", such as log messages of an interface entering "promiscuous" mode.

• Modifications to /etc/inetd.conf, rc.local, rc.sysint or /etc/passwd. Especially, any additions. Try using cat or tail to view these files. Additions will most likely be appended to the end. Remember though such changes may not be "visible" to any system tools.

Sometimes the intruder is not so smart and forgets about root's .bash_history, or cleaning up log entries, or even leaves strange, leftover files in /tmp. So these should always be checked too. Just don't necessarily expect them to be accurate. Often such left behind files, or log entries, will have obvious script kiddie sounding names, e.g. "r00t.sh".

Packet sniffers, like tcpdump (http://www.tcpdump.org), might be useful in finding any uninvited traffic. Interpreting sniffer output is probably beyond the grasp of the average new user. snort (http://www.snort.org), and ethereal (http://www.ethereal.com), are also good. Ethereal has a GUI.

As mentioned, a compromised system will undoubtedly have altered system binaries, and the output of system utilities is not to be trusted. Nothing on the system can be relied upon to be telling you the whole truth. Re-installing individual packages may or may not help since it could be system libraries or kernel modules that are doing the dirty work. The point here is that there is no way to know with absolute certainty exactly what components have been altered.

We can use rpm -Va |less to attempt to verify the integrity all packages. But again there is no assurance that rpm itself has not been tampered with, or the system components that RPM relies on.

If you have pstree on your system, try this instead of the standard ps. Sometimes the script kiddies forget about this one. No guarantees though that this is accurate either.

You can also try querying the /proc filesystem, which contains everything the kernel knows about processes that are running:

# cat /proc/*/stat | awk '{print $1,$2}'

This will provide a list of all processes and PID numbers (assuming a malicious kernel module is not hiding this).

Another approach is to visit http://www.chkrootkit.org, download their rootkit checker, and see what it says.

Some interesting discussions on issues surrounding forensics can be found at http://www.fish.com/security/. There is also a collection of tools available, aptly called "The Coroner's Toolkit" (TCT).

Read below for steps on recovering from an intrusion.

6.3. Reclaiming a Compromised System

So now you've confirmed a break-in, and know that someone else has root access, and quite likely one or more hidden backdoors to your system. You've lost control. How to clean up and regain control?

There is no sure fire way of doing this short of a complete re-install. There is no way to find with assurance all the modified files and backdoors that may have been left. Trying to patch up a compromised system risks a false sense of security and may actually aggravate an already bad situation.

The steps to take, in this order:

• Pull the plug and disconnect the machine. You may be unwittingly participating in criminal activity, and doing to others what has been done to you.

• Depending on the needs of the situation and time available to restore the system, it is advantageous to learn as much as you can about how the attacker got in, and what was done in order to plug the hole and avoid a recurrence. This could conceivably be time consuming, and is not always feasible. And it may require more expertise than the typical user possesses.

• Backup important data. Do not include any system files in the backup, and system configuration files like inetd.conf. Limit the backup to personal data files only! You don't want to backup, then restore something that might open a backdoor or other hole.

• Re-install from scratch, and reformat the drive during the installation (mke2fs) to make sure no remnants are hiding. Actually, replacing the drive is not a bad idea. Especially, if you want to keep the compromised data available for further analysis.

• Restore from backups. After a clean install is the best time to install an IDS (Intrusion Detection System) such as tripwire (http://www.tripewire.org).

• Apply all updates from ftp://updates.redhat.com.

• Re-examine your system for unnecessary services. Re-examine your firewall and access policies, and tighten all holes. Use new passwords, as these were stolen in all likelihood.

• Re-connect system ;-)

At this time, any rootkit cleanup tools that may be available on-line are not recommended. They probably do work just fine most of the time. But again, how to be absolutely sure that all is well and all vestiges of the intrusion are gone?