Back in 2000 I managed to acquire several retired systems to bring to Uni: this included 4-5 cheap P120 machines. At the time, I thought this was great; I had an OpenBSD box as my gateway, a FreeBSD box, a few Linux boxes, and likely something else that doesn’t even exist now. The school has a superfast connection, unlimited bandwidth, and I was curious. Although I didn’t really have time, I still managed to install and have all these servers running from my room.

I realized I was doing at home what I was being paid to do at work.

Fast forward to 2007, and my mindset has changed. In 2007 I didn’t want to have 6 servers running at once, I wanted to have one server running 12 servers at once! Thanks to Xen and VMware this was easily obtained. Initially using Xen, and then ESXi, I had the freedom to setup Domains, tear them down, and start over. Eventually, however, I realized I was doing at home what I was being paid to do at work. That doesn’t sound like fun. I also realized that, despite picking a motherboard and processor that could shift into low power usage, I was still using more watts than I needed to. I was also spending way too much time mucking around with things - I want to focus on just one or two projects at a time, and I really want to start programming more.

Last month I finally finished the ultimate ‘consolidation’: I moved everything to a tiny embedded Linux box. While back in the U.S. I contacted WDL Systems and requested for shipping costs on a tiny embedded box. I bought the eBox-3300, with an embedded board from ICOP, and it was promptly shipped out. After returning home to Sydney I migrated all my apps from the various virtual servers to my little box running Debian 5.0: OSSEC, Samba, Lighttpd, Asterisk and flow-tools. The little box is just perfect for what I need - a tiny home server. I still get around 8MB/sec transferring files, which indicates the network is still the bottleneck, and VOIP calls with Asterisk are still clear.

Overall, I’ve been happy with this little box. My ‘playing time’ with IT has gone down significantly, my energy usage has gone down, and I now have a server I can take with me wherever I go.

I like ESX. I like Linux. It is absurdly easy to configure Linux as an NFS server and mount it in ESXIi).

Installed NFS

I currently use Ubuntu Server for my home lab, but the process is basically the same for Red Hat and derivatives.

sudo apt-get install nfs-common
sudo apt-get install nfs-kernel-server

Next, configure NFS so it can server your local LAN. Normally you would list only specific servers, but, well, we’re being cheap and dirty today. Open /etc/exports in VI or your editor of choice.

/etc/exports

/media/disk/Images 192.168.0.0/24(rw,no_root_squash,async

Restart NFS.

sudo /etc/init.d/nfs-common

Go to Configuration -> Storage -> Add Storage.

Select NFS

Fill in the info, see screenshot.

Wait a minute. Voila! New datastore.

Images to come shortly.

I realized a few of my log files were growing unusually large, and even worse, logrotate was skipping them. I took a look in logrotate.d and straight away realized why: I had created silly names for the log file. logrotate look for .log files, but I had specified mine as .log – e.g. kelvinism_access_log. I was as familiar with logrotate when I set up the domains, so set forth to get them in the rotation.

Firstly, I had to rename the actual log files. So, to rename kelvinism_access_log to kelvinism_access.log, a one-liner:

for x in *_log; do mv $x `basename $x _log`.log; done;

Next, I needed to rename the log location inside each of the Apache config files. While a one-liner might be possible, I used the following tiny script:

#!/bin/sh
 
for x in *
do
sed 's/_log/\.log/' $x > /tmp/tmpfile.tmp
mv /tmp/tmpfile.tmp $x
done

I’m not impressed too often with much software, especially the closed source kind. I find a leaning preference to all things FOSS. If I had a million dollars, I’d likely spend all day contributing to all the projects I wish I had time to contribute to. Regardless, there are a select few closed-source products that I believe are truly excellent. I mean, the type of software where you aren’t asking “I wish this could do this” and start asking “I wonder what else this can do.”

While I’ve played around with most types of virtualization out there (OpenVZ, Xen, V-Server, qemu…), I’ve really found a soft spot for VMWare.

Don’t get me wrong, if I was going to host a heap of Linux web servers I would absolutely use Xen, but for a heterogeneous environment, I haven’t used anything as easy as VMWare’s products. Not that I judge a product by how easy it is to use, not by a long shot, but ease of use sure makes judging other factors easier.

Regardless, this isn’t a post trumpeting VMWare. I just realized tonight that some of the VMs I have running don’t need to be except for certain hours of the day, or if condition A is true. The first example is my backup mail server; I really don’t need it even powered on unless my main server is down. The second example is my Server 2003 instance, which has VI3 on it; I don’t need this running unless I’m asleep. One of the most useful resources I’ve seen for the vmrun command is over at VirtualTopia – loaded with examples.

Turn off via time

On my “monitoring” instance, which is always up, I’ve decided to install the script that controls my VM. I’ve opted to use a soft shutdown.

192.168.0.10 = ESXi box

datastore1 = name of datastore that hosts VMs

#!/bin/sh
 
vmrun -t esx -h https://192.168.0.10/sdk -u root -p root_password stop "[datastore1] Server 2003 R2/Server 2003 R2.vmx" soft

I have that saved in a file called stop_2003.sh in /opt/vmware/bin; make sure it isn’t world readable. I also have a start_2003.sh:

#!/bin/sh
 
vmrun -t esx -h https://192.168.0.10/sdk -u root -p root_password start "[datastore1] Server 2003 R2/Server 2003 R2.vmx"

Next, edit root’s crontab (crontab -e):

# m h  dom mon dow   command
0 8 * * * /opt/vmware/bin/start_2003.sh
0 23 * * * /opt/vmware/bin/stop_2003.sh

The conditional task is a tad bit more tricky, but just a tad. Ping won’t do, since the mailserver could go down itself, so install nmap. Create a script:

#!/bin/bash

if nmap -p25 -PN -sT -oG - mail.kelvinism.com | grep 'Ports:.*/open/' >/dev/null ; then
echo \`time\` >> mailserver.log
else
/opt/vmware/bin/start_mail.sh
fi

And sticking with our theme, start_mail.sh:

#!/bin/sh

vmrun -t esx -h https://192.168.0.10/sdk -u root -p root_password start "[datastore1] Mail Server/Mail Server.vmx"

This of course changes the crontab entry to:

#!/bin/bash
 
if nmap -p25 -PN -sT -oG - mail.kelvinism.com | grep 'Ports:.*/open/' >/dev/null ; then
echo `time` >> mailserver.log
else
/opt/vmware/bin/start_mail.sh
fi

So, that’s it. detect_port.sh is lacking any type of error detection or redundancy - if one packet/scan is dropped, the mail server will turn on. I’ll re-work this at some point, but it works for now.

Update: Vmware has also released a decent blog entry about using vmrun: on their blog.

Amazon offers some pretty cool services: S3, EC2, Alexa Site Thumbnail, and others. A while back I wanted to use AST with Django, so ended up writing the Python bindings to the REST API (they didn’t previously exist. I even wrote up a quick tutorial.

Update: Amazon no longer maintains AST. I’ve decided to archive a few of the old sites, so no longer need to take thumbnails. However, a few other thumbnail services seem to have crept up, including SnapCasa", and WebSnapr.

A normal internet connection gets attacked, a lot. The majority of attacks are of the form “hello, anybody there?” – where most people just don’t answer. But sometimes, just sometimes, the question gets an answer. Depending on the answer, the attacker will start to explore.

A few weeks back I was a little bored and started fiddling. I wanted to play with my Cisco, but also wanted to play with OSSEC, but also has a GIS craving. In the end I decided to create a map of the people who ask, “hello”.

Take a look at the map and explanation if that sort of thing is your cup of tea.

I’ve been side-tracked on another little project, and keep coming back to NetFlow. For this project I’ll need to access NetFlow data with Django, but this is a bit tricky. First, I’m sort of lazy when it comes to my own project; maybe not lazy, I just like taking the most direct route. The most up-to-date NetFlow collector I noticed was flow-tools, and there is even a switch to export the information into MySQL. Sweet! However, I wanted to insert the flows into MySQL automatically, or at least on a regular basis. I first started writing a python script that would do the job, but after a few minutes noticed flow-capture had a rotate_program switch, and started investigating. Since I somehow couldn’t find anywhere instructions how to insert the data automatically, here’s what I came up with:

1. Download flow-tools; make sure to configure with –with-mysql (and you’ll have to make sure you have the needed libraries).
2. Create a new database, I called mine ’netflow'.
3. Create a table that can contain all the netflow fields, a sample is below. I added a “flow_id” field that I used as a primary key, but you don’t necessarily need this.

CREATE TABLE `flows` (
`FLOW_ID` int(32) NOT NULL AUTO_INCREMENT,
`UNIX_SECS` int(32) unsigned NOT NULL default '0',
`UNIX_NSECS` int(32) unsigned NOT NULL default '0',
`SYSUPTIME` int(20) NOT NULL,
`EXADDR` varchar(16) NOT NULL,
`DPKTS` int(32) unsigned NOT NULL default '0',
`DOCTETS` int(32) unsigned NOT NULL default '0',
`FIRST` int(32) unsigned NOT NULL default '0',
`LAST` int(32) unsigned NOT NULL default '0',
`ENGINE_TYPE` int(10) NOT NULL,
`ENGINE_ID` int(15) NOT NULL,
`SRCADDR` varchar(16) NOT NULL default '0',
`DSTADDR` varchar(16) NOT NULL default '0',
`NEXTHOP` varchar(16) NOT NULL default '0',
`INPUT` int(16) unsigned NOT NULL default '0',
`OUTPUT` int(16) unsigned NOT NULL default '0',
`SRCPORT` int(16) unsigned NOT NULL default '0',
`DSTPORT` int(16) unsigned NOT NULL default '0',
`PROT` int(8) unsigned NOT NULL default '0',
`TOS` int(2) NOT NULL,
`TCP_FLAGS` int(8) unsigned NOT NULL default '0',
`SRC_MASK` int(8) unsigned NOT NULL default '0',
`DST_MASK` int(8) unsigned NOT NULL default '0',
`SRC_AS` int(16) unsigned NOT NULL default '0',
`DST_AS` int(16) unsigned NOT NULL default '0',
PRIMARY KEY (FLOW_ID)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

4. Setup your router so it sends netflow packets to your linux box (see README/INSTALL)
5. Create a “rotate program” that will actually enter in the information into mysql.

kelvin@monitor:/usr/bin$ cat flow-mysql-export 
#!/bin/bash

flow-export -f3 -u "username:password:localhost:3306:netflow:flows" < /flows/router/$1

6. Create the /flows/router directory
7. Start flow-capture (9801 is the port netflow traffic is being directed to); all done.

flow-capture -w /flows/router -E5G 0/0/9801 -R /usr/bin/flow-mysql-export

I’ve evaluated Zenoss before, but forgot the default password, and searching for it didn’t come up with anything quickly. I tried everything under the sun: password, 1234, admin, God, Sex, but alas, grep to the rescue:

kelvin@monitor:/usr/local/zenoss/zenoss/etc$ grep admin *
hubpasswd:admin:zenoss

Update: it is listed on page 4 of the Admin PDF :)

My little server doesn’t have a cdrom, but I didn’t want to actually run ESX from a USB (i.e. esx-on-a-stick). Here are my notes of configuring a flash disk to boot the ESX installer (so you can install it onto a local disk). For this demo, my USB is /dev/sdb

1. Install the syslinux utils to your computer (apt-get install syslinux mboot)
2. Install the MBR

sudo install-mbr /dev/sdb

3. Copy all the files from the ISO to your fat32 formated partition
4. Install syslinux

sudo syslinux /dev/sdb1

5. Move isolinux.cfg to syslinux.cfg, and try booting. If it doesn’t work, edit syslinux.cfg says something like:

default menu.c32
menu title ESXi Boot
timeout 100

label ESXi
menu label Boot VMware ESXi
kernel mboot.c32
append vmkernel.gz --- binmod.tgz --- environ.tgz --- cim.tgz
ipappend 2

6. Unplug your USB, put it in your server, reboot, boot to USB-HDD (or select the USB disk), and install ESX to the local disk. You will likely be greeted with a sign saying “MBR FA:”, where you need to press “A” and then “1”.

I recently had the need to move a rather large (450GB) VMDK file from an external hard drive into ESXi. Since ESXi doesn’t support external hard drives, this makes things quite a bit more difficult. At first I tried using SCP to copy the file over (after enabling SSH access for ESXi). However, when I tried to do this the time left was almost 20 hours – a tad too long!

I rethought my idea and decided to use this process:

1. Create an NFS share on my laptop, using the external hard drive (with the VMDK) as a mount point.
2. Use vmkfstools to move the image over.
3. Update any bugs I encountered.

Creating the NFS share on Linux is extermily easy. After install nfs via whatever package management tool you choose, put this entry into your /etc/exports file:

  
/media/disk-1 192.168.1.0/24(ro,no_root_squash,async)  

This assumes your USB disk is mounted as /media/disk-1, and your local subnet is 192.168.1.0/24. In OpenFiler, add a new storage with type NFS and use your laptops IP as the hose, and /media/disk-1 as the mount point. For safey, tick read-only.

Next, unlock SSH if you haven’t already. Once you are in, browse to /vmfs/volumes and you can see your nfs share and your other datastores. Let’s say you USB virtual disk is located at /vmfs/volumes/nfs/bigdisk.vmdk, and you want to import it into your normal datastore, under a folder called ‘NAS’. Using vmware specific tools, you can import the file as so:

  
# vmkfstools -i /vmfs/volumes/nfs/bigdisk.vmdk /vmfs/volumes/datastore1/NAS/bigdisk.vmdk  

I needed to update the hardware version of my imported disk. To do this, open up the .vmdk file (you should also have a -flat.vmdk file), and update the virtualHWVersion entry from 7 to 4. With that, join your disk to an image, and you should be good to go.

An addition result I noticed was the speed at which it came over. By using SCP, the entire file was going to take 20hr. By using NFS and vmkfstools, the files was migrated in under 10 hours.