Information is power, or so many of us think. As an I.T. administrator, no matter what the level, it is of my opinion that knowing what your network is doing is important. This includes regular operation, what it could do in the event of a disaster, or when it is just slightly not functioning.

Jeez, I’m personifying a computer network.

As I wrote in another post, I setup network monitoring for several nodes. But, what happens when an anomaly occurs? This is the story of hunting down a worm/virus, from far, far away.

While I can’t go into too many details, for obvious reasons, I’ll try and tell the story as-it-The overall process took for cleaning took several weeks to resolve, although minimizing the effect occurred the second the anomaly was discovered. The lengthy time to resolve is mainly due to the time to request computers to be patched/updated/scanned (like I said, this anomaly was with a large branch office in another country).

I’ve blurred out any relevant information for obvious reasons.

Almost immediately after setting up monitoring I noticed something strange occurring. As you can notice from the below graph (from nfsen), something is obviously wrong. If you can’t notice it, that big spike, in what we’ll call Network Green, should give you a clue. Now, I can expect a spike during lunch when people watch movies or send their 50mb picture email attachments, but this spike isn’t always at noon, and as we’ll see, it sure ain’t emails.

Find the anomaly:

So, let’s try to find out what is going on. Indeed, there are some analyzers of flow traffic that can display the types of traffic, and are useful if the issue isn’t apparent, but NFSEN’s filtering capabilities haven’t failed me yet (although seeing a graph with types of traffic is useful).

We can now take a look at what is going on. Let’s next just take a look at the statistics offered:

I’ve left the dates in there for you to see something interesting: Network Green transferred over 15 GIGS of UDP traffic. If we think back to networking basics, we can remember that UDP is a connectionless protocol. So, what uses UDP? DNS/TFTP, some streaming media, VoIP, and several types of encryption. That said, there isn’t any legitimate reason for any of the employees to use that much bandwidth, of this type. Only one thing seemed apparent: somebody was massively downloading from some p2p source, or we had a worm/virus.

Now that we know it is UDP related, let’s find out more. I’ve filtered by UDP in the graph just so you can see how much traffic that really is. Let’s select the anomaly. We first select the left side…

Then we select the right side, and we have a pretty green highlighted section.

Now it is time to hunt the person down. Using NFSEN’s built-in filtering capabilities, I was able to find which computer was being naughty. I first created the filter rule for the appropriate network (IP obscured for confidentiality):

Then could see the obvious place to look next:

As is highlighted, you can see that one IP is continually transferring UDP traffic over port 14857. Well, it certainly isn’t DNS or TFTP! (But we knew that when 15+ gigs was transferred:) When we look at the Top 10 Src IP Addr, ordered by bytes, we can see that one IP transferred a whole lot of traffic.

What now? I pulled out the nmap/nessus combo and tried to see what was running. Telneting to port 14857 didn’t return any form of hello message, and nmap didn’t return any known services for whatever was on port 14857. At this point I sent out the emails/documentation to managers in the remote office requesting for anti-virus to be checked and loaded onto any computers. Next, I blocked the port, from that computer, from sending outgoing traffic. Occasionally, as you can see in the graph, another computer or two would show the same symptoms, but within two weeks the oddity had disappeared. Thanks you NFSEN and Cisco.

Status: ✅

One of the greatest benefits, in my opinion, of Cisco routers is the ability to generate netflows. In a lot of ways, I would prefer to do this than implement some appliance (say, using ntop). The ability to analyse the amount of traffic becomes extremely valuable. Not only can one measure the amount of traffic, but the type of traffic that is being generated through the network.

Using a similar configuration, I setup all four Ciscos to export netflows that stream back to a server in the States. I decided to use nfdump as a collector. After the dumps are collected, it is simple to setup nfsen to parse and analyse the received flows. It even allows you to generate really pretty graphs.

So, why do this? For starters, collecting netflows allows the basic analysis of data, which can tell you several things. You can know instantly how saturated your connection is, if there are any anomalies, if there is any file sharing going on or when heavy traffic usage is. For instance, if the connection becomes slow during the end of the day, you can analyse what protocol is used the most during that time. Or, as was my case, hunting down virus infected computers that were fully saturating a 10mbit pipe.

A week in the life of NFSEN:

Abstract

The following is a Quick n’ Dirty method at implementing a very simple firewall.

Locate IPTables

Depending on your server, first locate iptables:

 [root@vps /]# which iptables 

Create IP Based Accept/Deny

Create a whitelist (ignored by firewall) or blacklist (packet dropped) if you wish:

 [root@vps /]# vi /usr/local/etc/whitelist.txt 

And/Or…

[root@vps /]# vi /usr/local/etc/blacklist.txt 

In each file, add each IP per line, for instance:

 4.2.2.2 66.35.15.20 

firewall.sh Script

Then put the following in /etc/init.d/firewall.sh, and edit to fit your needs:

#!/bin/sh
#
## Quick n Dirty Firewall
#
## List Locations
#

WHITELIST=/usr/local/etc/whitelist.txt
BLACKLIST=/usr/local/etc/blacklist.txt

#
## Specify ports you wish to use.
#

ALLOWED="22 25 53 80 443 465 587 993"

#
## Specify where IP Tables is located
#

IPTABLES=/sbin/iptables

#
## Clear current rules
#

$IPTABLES -F
echo 'Clearing Tables F'
$IPTABLES -X
echo 'Clearing Tables X'
$IPTABLES -Z
echo 'Clearing Tables Z'
echo 'Allowing Localhost'

#Allow localhost.
$IPTABLES -A INPUT -t filter -s 127.0.0.1 -j ACCEPT

#
## Whitelist
#

for x in `grep -v ^# $WHITELIST | awk \'{print $1}\'`; do
        echo "Permitting $x..."
        $IPTABLES -A INPUT -t filter -s $x -j ACCEPT
done

#
## Blacklist
#

for x in `grep -v ^# $BLACKLIST | awk \'{print $1}\'`; do
        echo "Denying $x..."
        $IPTABLES -A INPUT -t filter -s $x -j DROP
done

#
## Permitted Ports
#

for port in $ALLOWED; do
        echo "Accepting port TCP $port..."
        $IPTABLES -A INPUT -t filter -p tcp --dport $port -j ACCEPT
done

for port in $ALLOWED; do
        echo "Accepting port UDP $port..."
        $IPTABLES -A INPUT -t filter -p udp --dport $port -j ACCEPT
done

#
## Drop anything else
#

$IPTABLES -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
$IPTABLES -A INPUT -p udp -j DROP
$IPTABLES -A INPUT -p tcp --syn -j DROP

Start Firewall

[root@vps /]# chmod 700 /etc/init.d/firewall.sh
[root@vps /]# /etc/init.d/firewall.sh 

As you may know, I’ve been taking a few of the MCSE:Security tests when I have time.  This afternoon I passed the 70-291, which is claimed to be the hardest of all the tests (and of the one’s I’ve taken so far, I would concur).

 I’m taking my tests at the Geego Systems center.  If you are in Taipei for some reason, and needing a place to take some tests, this place is a safe bet.  Their staff is pretty darn cool, and they let me drink as much tea as I want.

 Anyways, the test.  I can’t talk about any of the questions, ‘cause I’ve “signed” an NDA, and Microsoft will take away my certification(s).  But, I’ll tell you what I did to study.

I took three approaches, as I do to all the tests.

1. Get an Exam Cram2 70-291 book, and read it through, cover-to-cover.  This, in my opinion, is a great way to get to know the language used, and some of the concepts.  Exam Cram 2 books do a pretty good job of cutting out the fluff.
2. I have a copy of Mastering Windows Server 2003 that I picked up in Hong Kong last year.  It doesn’t do a good job of cutting out the fluff, which makes it a great resource to flip through.  It is seems to have almost every tutorial you could imagine.  The one downside though, as there always is one, is that the book is starting to get a little dated.  Either way, this has to be one of the most complete books on 2003 I have ever flipped through.
3. It is all fine and dandy to read, but it is my belief that everything you read, you should do.  My little lab (see end of post for my cool diagram) seems to suite me quite well.  If you have read this far, and want some things to do in your test lab, this is what I would recommend:

• DNS - Know how to add CNAME’s, A records and play with MX records.  Know how to prioritize mail servers.  Know how to create aliases.
• WSUS - Sadly, many of the books I’ve seen (or the two listed above), don’t talk about WSUS.  I believe Microsoft just made the change on their tests, but it is on there.  And they hit this topic pretty hard, so be prepared.
• Security Templates  - Creating baselines and doing audits
• Network Security - Configuring firewalls, configure VPN between two networks (RRAS)
• DHCP - Creating scopes, relay agents and reserving client addresses
• TCP/IP -This test has a pretty big backbone of TCP/IP.  Know how to subnet, and general networking terms/methods.  Picking up a book on the Network+ certification, such as the Exam Cram2 Network+ book (which I read to refresh myself, and pass the Network+ test), could be useful.

Luckily I was able to get a few years of some real life hands-on experience with installing, implementing and managing AD through a previous I.T. position.   Practice always helps though.  Good luck!

Status: ✅

Users don’t like to remember passwords, heck, I don’t like to remember to use passwords. I decided to upgrade all the webapps to authenticate off the domain, welcome a start to SSO. To do this I implemented the adldap php class to control authentication to each webapp. Thus, a simple GPO can control who has access to the app or not. A simple solution to a rather simple problem.

Status: ✅

Backing up across the states has worked decently well, but due to several changes a more dedicated backup solution is in order. Desiring something quick, simple and inexpensive, research revealed a company that would perfectly fit the requirements. iBackup was a perfect substitute - instead of SSH+rsync to another office, iBackup provides rsync over ssl to their data center. A few simple changes to the cron job, and backup location is thus changed.

Status: ✅

Time has come to bring another network on the VPN, and perform some more upgrades. The usual by now, I guess.

• Get China on VPN
• Limit access to other locations
• Update all systems
• Perform security audits
• Upgrade wifi
• Setup video conferencing

Ian and I set off for our China office out of Hong Kong, and the next day started working. Total preparation was around a month, maybe a little large, mainly due to red tape. We first acquired assistance of IBM China, who were of a great help aiding us in finding our desired Cisco. One of the most important factors, which we couldn’t resolve by purchasing the Cisco in the States, is support/warranty contracts (if the Cisco totally dies, what then). Through our contact we were also able to find some local vendors that would support Wifi and the Cisco, in case of an emergency.

Before leaving I prepared the necessary configurations for the Cisco, or at least a good guide to start from. The technician who came out tried to get things going through the built in GUI, however wasn’t have so much luck. I took over using my pre-built configuration and soon (we swapped out the old router with the Cisco during lunch) everything, including overloaded NAT, was working fine. By the time employees came back from lunch, they couldn’t notice any difference.

While the Cisco tech (who I believe is a good guy, even though I did the Cisco install) was waiting for some paper work went through and upgraded the way obsolete wifi point from WEP (which wasn’t even turned on anyways) to WPA. The reasons for this, especially connected to the VPN, are very obvious. Technically the AP wasn’t supposed to support WPA, but he found the correct Chinese firmware and it worked. This is good, as the new AP wouldn’t be coming for a little while.

Next on the list was video conferencing. The solution was the path of least resistance: Skype on a laptop. Ian took this one, setup the laptop, and tested conferencing back to the States.

On the agenda for that night was VPN. The problem with bringing the China office on the VPN is one of security. Virus’ were quite prevalent (e.g. my shared drive on my Linux laptop, to use as a sandbox, had a couple .exe files dropped into it. All with rather odd names…) – so we first ran some security audits. Nessus was a great help, as always, and we tracked down over [an UNFATHOMABLE amount of] critical holes. Picking the biggest culprits we started patching computers, removing spyware and running anti-virus. Slowly (a few days) we got the number knocked down significantly.

Lastly I hooked China up to the VPN. In order to do this safely I created some very strict access lists, to only allow outgoing communication over ports 80 and 443 (since that is all they needed at that point). Previously setup we had a webshare website (auth linked to the PDC), so no need to open any other ports.

Overall we completed what we set out to do. We made a few good contacts, achieved our goals, and once again learned more about doing I.T. overseas.

Status: ✅

Before we grow any further, a new DNS scheme is in order. Following the pattern of: citycode.domain.com - shouldn’t be too hard. A slightly stressful rename of the PDC (just one so far, still small) was in order. After that (and client computers re-associated), the routers were updated, the DNS server updated, and everything worked peachie. Not bad for a weekends worth of work.

The following sites might be helpful to configure IPSec again, or on a more complex basis:

http://www.cisco.com/univercd/cc/td/doc/product/iaabu/pix/pix_60/ipsec/conipsec.htm

http://www.cisco.com/warp/public/707/overload_private.shtml

Logs spiraling crazy, we run out of disk space all the time. A nifty trick to find where the disk went is to issue:

du -cks * |sort -rn |head -11

This returns where the disk usage is, and makes finding the bloated log a lot easier.