For various reasons I had the need to open a raw disk inside VMware Server 2. The reports from the field say that this just isn’t supported. Although I don’t need to actually run a raw disk, I needed to get some data off it – 400GB worth. It turns out ’not supported’ really means ’not in the UI.’ I don’t know the reason why it isn’t in the UI, maybe marketing wants people to use ESX, or maybe the UI guys fell behind with their workload.

Alas, it is possible. And here’s how.

1. Take out your ‘raw disk’ and put it into another machine.
2. Fire up Server 1.0.x or Workstation and open a virtual machine (or create a new one). Edit the preferences and add a new hard disk. Select ‘use a physical disk’, and select the disk you put in above. Select use entire disk. You may want to change the SCSI LUN to SCSI1:0 (depending how many disks are in your ‘proper’ server).
3. Save it as something like 500GB.vmdk
   3b. Copy out the relevant bit from the vmx file, e.g.

# Test VM.vmx
scsi1.present = "TRUE"
scsi1:0.present = "TRUE"
scsi1:0.fileName = "500GB.vmdk"
scsi1:0.deviceType = "rawDisk"

And of course, the entire 500GB.vmdk file

# 500GB.vmdk
# Disk DescriptorFile
version=1
CID=7e245252
parentCID=ffffffff
createType="fullDevice"

# Extent description
RW 976773168 FLAT "/dev/sdb" 0

# The Disk Data Base 
#DDB

ddb.virtualHWVersion = "6"
ddb.geometry.cylinders = "60801"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.geometry.biosCylinders = "60801"
ddb.geometry.biosHeads = "255"
ddb.geometry.biosSectors = "63"
ddb.adapterType = "buslogic"

Note: If your guest OS is 64-bit, you won’t be able to use buslogic. Switch the last entry above to ’lsilogic’.

While you could likely create the vmdk file by hand, the only number I’m not certain about is the part after the RW. (UPDATE: Note added to page). The Disk Data Base you can just see by typing in ‘fdisk /dev/sdb’

4. Move the disk back to the ‘server’ and turn the server back on.
5. Edit the vmx file of whatever virtual machine you want to use and put in the part copied from the vmx file of your other machine. Alternatively, if you did an upgrade, you could just copy it across now. Create a new 500GB.vmdk file in the same directory, paste in the bit you copied out from the test virtual machine. Double check that the ‘raw disk’ comes up as the same node in /dev.
6. Boot up the virtual machine. You will notice in the WebUI that a new scsi controller is inserted. You should also noticed a new disk accessible inside your virtual machine, e.g.

[root@files dev]# ls sd*
sda  sda1  sda2  sda3  sdb  sdb1
[root@files dev]# ls /mnt
cdrom  floppy
[root@files dev]# mkdir /mnt/disk
[root@files dev]# mount /dev/sdb1 /mnt/disk
[root@files dev]# ls /mnt/disk
Files  lost+found  Movies  Music  Personal  VMWare
[root@files dev]#

Update: Peter Jonsson kindly sent in the answer to “I don’t know what to put after the RW.” Below is the description of how to find the correct number. Thanks Peter!

The magic formula is:

ThePartAfterTheRW  =  TOTAL AMMOUNT OF DISKBYTES   /   512


This is my Western Digital 500 GB drive: 

fdisk -l

Disk /dev/sdc: 500.1 GB, 500107862016 bytes

256 heads, 63 sectors/track, 60563 cylinders

Units = cylinders of 16128 * 512 = 8257536 bytes

Disk identifier: 0x00000000

And using the formula I got the "RW" stuff:

500107862016 / 512  = 976773168

I think possibly one of the most practical upgrades in VMware Server 2 appears to be the ability to automatically turn on virtual machines in a stagnated order. I have fond memories of turning on a server with 10 virtual machines, and when they all turn on at once, the hard disk grinds to a halt. This forced me to turn on the machines manually afterwards.

+1 VMware in my books.

I now have successful installation of VMware Server 2 (Beta RC1) on top of Ubuntu 8.04 64-bit. I have been using various virtualization technologies for years, and VMware is usually the easiest to install and configure. So far, VMware Server 2 RC1, has proven to be the exception to the rule.

That said, I am very excited by the direction VMware is taking – this new server version looks to have great potential.

The ‘server’ this is on is a mATX motherboard from Gigabyte (GA-G33M-DS2R), with 4GB (2x2GB) of Transcend DDR2-800 memory, topped off with the E8200. I have been nothing but impressed with this combination of hardware.

However, although I was thinking VMware Server 2 would install seamlessly over Ubuntu, I was wrong. There were a few things I had to tweak to get everything working correctly.

The first thing I had major issues with was VMware choking on the parallel port. Normally the parport is the first thing I would turn off, but in this instance, I guess excitement overtook me. My tip is to first remove the lp module from inside /etc/modules, and then disable the parallel port inside the BIOS. The symptoms I was having involved VMware halting/freezing on either startup or shutdown. This occurred for both RC1 as well as 1.0.6.

My second tip, if VMware freezes half way through starting up or shutting down, is to go through the vmware startup script, /etc/init.d/vmware, and comment out anything refering to the parport_pc. In particular, I looked for this line and made sure to comment it out:

/sbin/modprobe -r parport_pc >/dev/null 2>&1

I commented out lines 974 and 1076. After doing this, VMware loaded perfectly.

The second major issue I had occurred after actually installing VMware. I opened Firefox and went to the IP of my virtual server, logged in just fine, and loaded up my first virtual machine. However, after booting the virtual machine, I was unable to open up the remote console. It turns out I had just upgraded to Firefox 3.0.1, and the Remote Console is set to fail on anything above 3.0.0.1. The fix is quite easy.

First click where it says “click anywhere to open the virtual machine”. Copy the address of the XPI and use something like wget to download the file. This is an example:

wget --no-check-certificate https://192.168.50.10/ui/plugin/vmware-vmrc-linux-x86.xpi

If you are using Gnome, right click the file you just downloaded and say Open With then Archive Manager. Do the same for the ‘install.rdf’ file inside, specifying gedit as the application if need be. Next, edit line 20 so it reads as follows:

3.0.*

Save the file, open the XPI with Firefox, and you should be good to go.

I’ve seen a lot of other suggestions on the ’net on how to fix VMware RC1 when booting – including disabling ipv6, checking the hosts file, and running the any-any patches. None of these approaches helped me at all, but maybe it is exactly what you need. My biggest tip is that if VMware isn’t starting up or stopping correctly, open up /etc/init.d/vmware and find out exactly where it is faulting (add things like ’echo “fail”’ inside the IF statements).

Installing the tools in VMware Server 2 is a little different than Workstation or the previous versions of VMware Server. Under the Summary tab of your Virtual Machine, look for a link that says “Install VMware Tools” – click it.

Wait for ‘Success’ to show up on the bottom, and jump into your virtual machine. Mount the tools as so:

mount /dev/cdrom /media/cdrom

And install as normal (copy the .tar.gz to /usr/src, extract it, install it). Easy peasy.

I needed another Python fix, and I need one pretty badly. I spent the weekend wondering why it appears to be impossible to edit the GUIDs inside an Exchange mailbox store (read: NOT the GUIDs stored in AD for Exchange). Anyways, I digress.

My goals were simple. I wanted to use Python, wanted something to do with traffic, and wanted to play around with Glade/PyGTK and graphing stuff. My end result was a little app that allows you to specify a GPX file, and it plots the waypoints (and calculates the moving average!). Pretty simple, pretty useless, but pretty fun. I really do like pretty pictures.

I ended up using matplotlab for the graphing part, but I don’t really like how the graphs look. I will likely use Pycha (which dips into Cairo) for my future projects – but we’ll see when that point comes. (If data sensitivity wasn’t an issue, I would totally use Google Charts, since I’m a sucker for APIs).

I showed a few of my co-workers my graph and one replied – oh! that’s really cool. (I think only two of my co-workers are actually interested in my geekyness). He then emailed me tonight a .kmz file containing a colorized file of his speed. I looked at the kml and noticed it appeared to be dynamically allocated judging by the top speed. Well, as you could guess, I surely had to modify my code to include colors.

Within an hour I had a semi-working example, and within two hours will easily be done with this blog post. The code might not be perfect, but it first parses the xml and returns the max speed for the trip. Next, it colorizes the speeds based on a scale of 0-255, with 0 being blue for fast and 255 for being yellow, or slow. I was going to study for the CCNA tonight, but it looks like writing Python is just too much fun.

So what, you might ask, are those dips? Good question. They are huge speed bumps (and the tall blue mound in the middle is a really steep hill).

I recently have been playing with parsing GPX files and spitting out the results into a special KML file. I initially wrote a parser using minidom, yet after running this the first time – and my Core2Duo laptop reaching 100% utilization for 10 seconds – I realized I needed to re-write it using something else.

I spent a little time reading the different parsers for XML and eventually read more about cElementTree. And it is included with Python2.5, sweet.

I quickly rewrote the code and did some tests. First, the two bits of code for parsing my GPX file:

minidom-speed.py

#!/usr/bin/python

from xml.dom import minidom
from genshi.template import TemplateLoader

def collect_info():
dom = minidom.parse('airport.gpx')
for node in dom.getElementsByTagName('trkpt'):
lat = node.getAttribute('lat')
lon = node.getAttribute('lon')
speed = node.getElementsByTagName('speed')[0].firstChild.data
speed = float(speed) * 10
coords = '%s,%s' % (lon, lat)
coords_speed = '%s,%s' % (coords, speed)
yield {
'coordinates': coords_speed
}

loader = TemplateLoader(['.'])
template = loader.load('template-speed.kml')
stream = template.generate(collection=collect_info())

f = open('minidom.kml', 'w')
f.write(stream.render())

cet-speed.py

#!/usr/bin/python

import sys,os
import xml.etree.cElementTree as ET
import string
from genshi.template import TemplateLoader

def collect_info():
mainNS=string.Template("{http://www.topografix.com/GPX/1/0}$tag")

wptTag=mainNS.substitute(tag="trkpt")
nameTag=mainNS.substitute(tag="speed")

et=ET.parse(open("airport.gpx"))
for wpt in et.findall("//"+wptTag):
wptinfo=[]
wptinfo.append(wpt.get("lon"))
wptinfo.append(wpt.get("lat"))
wptinfo.append(str(float(wpt.findtext(nameTag)) * 10))
coords_speed = ",".join(wptinfo)
yield {
'coordinates': coords_speed,
}

loader = TemplateLoader(['.'])
template = loader.load('template-speed.kml')
stream = template.generate(collection=collect_info())

f = open('cet.kml', 'w')
f.write(stream.render())

The speed difference is not just noticeable, but very noticeable.

minidom-speed.py

$ python -m cProfile minidom-speed.py
4405376 function calls (3787047 primitive calls) in 32.142 CPU seconds

cet-speed.py

$ python -m cProfile cet-speed.py
1082061 function calls (904167 primitive calls) in 6.736 CPU seconds

A quarter as many calls and almost 5x faster – at least that’s how I interpret the results. Much better!

I have had to search for the commands to setup a Windows 2003 box as an ntp client a few times now, so have decided to finally write them down here for my own good measure. Funny thing is, I’m pretty sure there are three ways to setup a 2003 box as an ntp client.

1) Via the CLI

Open up the cmd prompt and type in:

w32tm /config /manualpeerlist:"0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org 3.pool.ntp.org" 
/syncfromflags:MANUAL /reliable:YES /update

2) Via the CLI, option 2

net time setsntp: "0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org 3.pool.ntp.org"

3) Via GUI

Type in gpedit.msc and your local GPO editor will pop up. Go to the folder as indicated in the below screenshot and Enable the “Enable Windows NTP Client” option. Next set the “Configure Windows NTP Client” option to whatever time servers you so choose. As always, make sure to keep the 0x1 at the end.

Status: ✅

You can likely tell that I’ve been having some fun with graphing and mapping recently. I was reading a few articles about GIS and stumbled upon a pretty darn cool project at Webopticon, which included cool pictures. I showed it to a friend thinking they would find it interesting, and then realized: oh! KML has an altitude attribute. That could be interesting.
One of my projects is to create maps of Sydney’s traffic, so I have been experimenting heavily with Mapnik and OSM. I figured I could have some fun and finally parse some gps tracks and display the data.

I first started off trying to play around with the KML files my gps logger natively stores. After a while I realized it shouldn’t be this hard to parse the XML, and realized it also stores data in gpx format. I opened up one of the gpx files and immediately saw how much easier it would be to work with. I quickly created a parser for the xml in Python (using the dom method, yet I think I’m going to rewrite it using sax), and then with the aid of an article by Sean Gillies, converted the needed objects into KML. I used the speed attribute (with some magnification) as the altitude, and voila, a pretty picture.
This picture is as Victoria Road crosses James Rouse Drive – a spot that is always congested in the morning.
I’ll likely post some code shortly, I would like to rewrite the parsing section to use something event-driven – hopefully it will be a little faster.

First, we go ahead and install the needed packages. I’ve tried to include “my” list of packages that were needed to get a vanilla 7.10 image up to steam.

apt-get install build-essential libltdl3-dev autoconf libtool automake \
postgresql postgresql-8.2-postgis postgresql-server-dev-8.2 \
wget subversion libboost-python1.34.1 libboost-thread-dev \
libboost-program-options-dev libboost-regex-dev \
libboost-python-dev libboost-serialization-dev \
libboost-filesystem-dev libpng12-dev libjpeg62-dev \
libtiff4-dev zlib1g-dev libfreetype6-dev libgeos-dev \
unzip apache2-prefork-dev

Next we start to download a few components. I did this in my home directory, /home/kelvin

mod_tile - this is the apache module and rendering daemon that uses mapnik to render the maps.

svn co http://svn.openstreetmap.org/applications/utils/mod_tile

Mapnik - this will help us create the maps.

wget http://download.berlios.de/mapnik/mapnik_src-0.5.1.tar.gz

Now we start to install things.

tar -xpzf mapnik_src-0.5.1.tar.gz
cd mapnik-0.5.1

Build mapnik as per: http://wiki.openstreetmap.org/index.php/Mapnik – make sure to use scons as follows:

python scons/scons.py PYTHON=/usr/bin/python \
PGSQL_INCLUDES=/usr/include/postgresql \
PGSQL_LIBS=/usr/lib/postgresql BOOST_INCLUDES=/usr/include/boost BOOST_LIBS=/usr/lib

Now, I’m temporarily serving/rendering my tiles from an old Thinkpad “server” (PIII with 512MB RAM, of which only 128MB goes to the Xen instance that hosts all of this). So, I am using osm2pgsql on my laptop (a new Thinkpad), and pushing it into the postgres database on my “server”. So, I built osm2pgsql on my new Thinkpad, and setup postgres on the “server” to accept connections from my new Thinkpad.

pg_hba.conf – Set these lines should be added, assuming your computer is 192.168.10.100:

host    all     all     192.168.10.100/32  trust

Then I do the actual import, assuming my “server” has an IP of 192.168.10.10:

./osm2pgsql -H 192.168.10.10 -U username -l -m -d gis -W /home/location/to/osm/australia.osm

Make sure generate_image works before installing mod_tile!

Install mod_tile as per the modifications needed: http://www.kelvinism.com/howtos/notes-installing-mod_tile-mapnik/