Ubuntu 10.04, Django and GAE - Part 1

I’ve started to get into Google’s App Engine again, and have started developing a simple product that I had a use for. The initial first draft was a quick 200 lines in webapp, and it worked great. However, I’m starting to find certain things quite cumbersome. I’m a huge fan of Django, and but also about keeping things as simple as possible, which is why I picked webapp to begin with.
I’m now considering making a swap to Django, but there are some development issues; namely, I’m using Ubuntu 10.04, Python 2.6, and Django 1.2. This setup presents several setbacks, as GAE has the requirement of Django 1.1 and Python 2.5. There are two solutions that I found: a) use virtualenv, which I’ve detailed, or b) chroot. This document will hopefully show how to configure a chroot environment of Ubuntu 9.10 and prepare it for Django on GAE. Using a jailed environment should allow you to edit your code with your normal IDE and VCS, but use Django 1.1 and Python 2.5.
First, I installed schroot and debootstrap.

$ sudo apt-get install schroot debootstrap

Second, I edited /etc/schroot/schroot.conf and added the following section to the end.

[karmic]
description=karmic
type=directory
location=/var/chroot/karmic
priority=3
users=kelvinn #your username goes here
groups=admin
root-groups=root
run-setup-scripts=true
run-exec-scripts=true

Third, I created the directories needed for the jailed environment and installed karmic.

$ sudo mkdir -p /var/chroot/karmic
$ sudo debootstrap --arch i386 karmic /var/chroot/karmic

Forth, I logged into the jailed environment and updated packages, installed Python 2.5 / Django 1.1. Make sure to note that I don’t call ‘python’, I call ‘python2.5’.

$ sudo schroot -c karmic
(karmic)root@kelvinn-laptop:~# apt-get update
(karmic)root@kelvinn-laptop:~# apt-get install python2.5
(karmic)root@kelvinn-laptop:~# cd /usr/src
(karmic)root@kelvinn-laptop:~# apt-get install wget
(karmic)root@kelvinn-laptop:/usr/src# wget http://www.djangoproject.com/download/1.1.2/tarball/
(karmic)root@kelvinn-laptop:/usr/src# tar -xpzf Django-1.1.2.tar.gz
(karmic)root@kelvinn-laptop:/usr/src/Django-1.1.2# python2.5 setup install
(karmic)root@kelvinn-laptop:/usr/src/Django-1.1.2# exit

Lastly, I log in as my normal user, and start the app. Let’s say I have a folder called ‘~/gaeapps’ for my GAE stuff, and that’s where I put the SDK.

$ scroot -c karmic
(karmic)kelvinn@kelvinn-laptop:~/gaeapps$ ls
google_appengine  myproject
(karmic)kelvinn@kelvinn-laptop:~/gaeapps$ google_appengine/dev_appserver.py myproject

Ubuntu 10.04, Django and GAE - Part 2

All my Django sites are running 1.2, which poses a conflict with writing apps for Google’s App Engine, as use_library currently only supports < Django 1.1. There are two solutions that I found: a) use virtualenv, or b) chroot, which I’ve already detailed. This document will hopefully show you how to create a virtual environment to use a secondary django version, especially for GAE. Of the two options, I think this one is a bit quicker, but there will likely be tradeoffs that a chroot environment can deal with better, e.g. python imaging (I don’t use it for GAE).
First, install PIP and virtualenv:

kelvinn@kelvinn-laptop:~/workspace$ sudo easy_install -U pip
kelvinn@kelvinn-laptop:~/workspace$ sudo pip install -U virtualenv

Second, configure an environment for any app that will use Django 1.1:

kelvinn@kelvinn-laptop:~/workspace$ virtualenv --python=python2.5 --no-site-packages django-1.1
New python executable in django-1.1/bin/python
Installing setuptools............done.
kelvinn@kelvinn-laptop:~/workspace$ pip install -E django-1.1 yolk
kelvinn@kelvinn-laptop:~/workspace$ pip install -E django-1.1 Django==1.1

Now, download the python GAE sdk and put it in the django-1.1 folder. I also just dump any project directory requiring Django 1.1 into this django-1.1 folder, although I guess you could create a virtualenv for each project. The last thing to do is start the virtual environment, and run the GAE app.

kelvinn@kelvinn-laptop:~/workspace$ source django-1.1/bin/activate
(django-1.1)kelvinn@kelvinn-laptop:~/workspace$ yolk -l
(django-1.1)kelvinn@kelvinn-laptop:~/workspace$ cd django-1.1
(django-1.1)kelvinn@kelvinn-laptop:~/workspace/django-1.1$ ls
bin  google_appengine  include  lib  myproject1 myproject2
(django-1.1)kelvinn@kelvinn-laptop:~/workspace/django-1.1$ google_appengine/dev_appserver.py myproject1

When you’re all finished, you can jump out of virtualenv:

(django-1.1)kelvinn@kelvinn-laptop:~/workspace/django-1.1$ deactivate

Update: You’ll find this article especially interesting if you get an error such as the following:

UnacceptableVersionError: django 1.1 was requested, but 1.2.0.beta.1 is already in use

Debug Postfix Tip

One thing I love about open source stuff is that the developers usually take great care to allow awesome debug messages. There’s a catch-22, however: how much logging to enable? Today I was creating a Postfix/Dovecot/Postgresql install and I kept getting an error message in mail.log, but it wasn’t very helpful.

Luckily you can turn up the verbosity in Postfix for error messages. You’ll need to find out what component is in error, e.g. “postfix/virtual[4467]: warning:”, and then open master.cf. Add a -v to the end of the daemon that’s faulting, and you’ll get more logging than you know what to do with.

virtual   unix  -       n       n       -       -       virtual -v

I hope this helps somebody. You can read more debugging tips on the Postfix DEBUG readme.

Retiring Another Site

After much consideration, I’ve decided to retire yet another website: Colddirt. This site was created as an example of how to use the different parts of Django, mostly in just one afternoon. I’ve decided it is time to just take a few screenshots and retire the site - the domain is expiring, and honestly, I don’t want to spend $10 to renew the domain. So, here are the screenshots.

Retiring Old Websites

Sometimes all good things come to an end. There aren’t too many links going into either of these sites, but I’d like to redirect all of them to a page on my blog saying the website doesn’t exist anymore. I’ve already created simple screendumps for nostalgic purposes with the Firefox plugin Screengrab!, so the remaining simple server-side steps are:

  1. Edit the DNS records from the live server to the server with the notice page is on.
  2. Create a new vhost on the server the notice exists on, add old websites as ServerAlias.
  3. Add a redirect in the vhost to the notice about the retired sites.
  4. Reload apache config.

The below vhost entry will redirect any link to the retired sites to the notice page.

ServerName ducktracker.com
ServerAlias www.ducktracker.com blogmozaic.com www.blogmozaic.com
RedirectMatch permanent (.*)$ http://www.kelvinism.com/tech-blog/rip-old-sites/ 

RIP Old Sites

Over the years I’ve created several websites, some with the hope of becoming big and popular. Naturally, I’m still waiting to create a site that becomes big and popular. In the meantime, it seems appropriate to retire some of the older sites and ideas. The truth is, I’m moving all my little sites from MySQL on a mediocre VPS to Postgresql on EC2.

Because I put some sweat and blood into these sites, I thought it would be only appropriate to record how the sites looked for future sentimental value. I downloaded a small plug-in for Firefox to take full-length pictures, and I thus post them here.

Back in Taiwan I created a site to demo blogs. The idea certainly wasn’t revolutionary, but I thought that somebody would find it useful. Nope.

The second idea was to send a bunch of little ducks around the world and track their progress. My friend Jamie took one to Europe, but besides that, I don’t think most people really understood that they were supposed to hand the duck off to somebody else. I’ll re-brand and launch it again if I can find a little twitter logo to send around, because hopefully people on twitter will understand to pass the duck onwards.

So photos for nostalgic sake.

It is always a bit rough retiring old sites, but looking back I learned a little, had some hope, and that’s all I can ask for.

Fun with OSM

I have to admit, to me, editing OpenStreetMap is actually a little therapeutic. Sort of like gardening.

My first major contribution was when I brought my little QStarz GPS unit across Indonesia, by train, sitting against the window.

The most recent contribution was our trip to Dubbo, where I helped fill in a few missing roads, and added an initial outlay of Dubbo Zoo.

If you have a GPS or have $60 to spend on one, and like anything CAD-drawing like, give editing OSM a try!

Reduce Bandwidth By Shrinking Images

A friend has a simple WordPress-based website that I set up for him a few year ago - and he likes to post a lot of images. For various reasons I have it hosted on my VPS in Australia, which only comes with 15GB of bandwidth, most likely because I got a crazy good deal on it.

The problem is, that isn’t much bandwidth. Each page of his site was previously using almost 10MB of bandwidth per view - which meant the 15GB was running out. I deduced the high usage in Firebug and noticed it was gasp from large images, some almost 1MB large. The images on the server are contained in subdirectories under a gallery folder - and I wanted to resize them. What I actually wanted to do was:

  1. If the image was over 1024x900, resize it.
  2. If the image size (bytes) was over X, lower the quality.

I ended up using ImageMagick, which is what first obviously came to mind for image manipulation. While there is likely a better way to resize all the images, this is what I came up with. Maybe somebody else will find it useful too.

for dir in /home/vhosts/website.com/html/wp-content/gallery/*
do
cd $dir
for filename in *.jpg
do
echo Filename: "$filename"
width=`identify -ping -format %w "$filename"`
height=`identify -ping -format %h "$filename"`
if [ $width -ge 1024 -a $height -ge 900 ]; then
convert "$filename" -resize "%80>" "$filename"
echo New Dimensions: $width x $height
fi
filesize=`stat -c%s "$filename"`
if [ $filesize -ge 414981 ]; then
convert "$filename" -quality 90 "$filename"
echo Old Filesize: $filesize
echo New Filesize: `stat -c%s "$filename"`
fi
done
done

Save it as “resize.sh” and run it with “bash resize.sh”. The output:

Filename: img_5364.jpg
Old Filesize: 950748
New Filesize: 331426
Filename: img_5366.jpg
Filename: img_5367.jpg
Filename: 1c10_1.jpg
Filename: 1.jpg
Filename: 2752_1.jpg
Filename: 42920014.jpg
New Dimensions: 1544 x 1024
Filename: 42920017.jpg
New Dimensions: 1544 x 1024

French Word Collage

For the last year YS and I have been studying French once a week (4.5hr on Saturday!) A few months ago I wrote some Python code to read the RSS feeds of the most popular French newspaper site, and then kept track of the count of each word. I repeated this every week for a few months.
In the end I had a database of the top ~3000 words used on the site. Today I decided I wanted to play with Processing, so created this little word collage. It is 1680x1050, my monitor’s resolution. Two thumbs up for Processing!

Vortex86DX Instructions from ICOP

Alexandru T. and I have exchanged a few emails, and he sent through a few helpful suggestions that were provided from ICOP. I have included them below. Thanks Alexandru!

  1. Install Debian 5.0 on a normal PC (using a netinst image, for minimal install)
  2. After installation boot normally from the same PC
  3. Then, take the kernel from ftp://icop.com.tw/DIS_info/VDX/operating_system/VDX_Linux/linux-image-2.6.30-vortex86mx_1.0_i386.deb and then issue the following commands :
# dpkg -i  linux-image-2.6.30-vortex86mx_1.0_i386.deb
# update-initramfs -k 2.6.30-vortex86mx -c
# update-grub
# restart
  1. Then take the hard-drive and install it on the Vortex86DX
  2. When GRUB menu appears, press “e” and modify the boot loader as follows :
root        (hd0,0)
kernel        /boot/vmlinuz-2.6.30-vortex86mx root=/dev/hdb1 ro         --> if hdb1 does not work you can try (hda1= Primary Master or hdc1=Secondary Master)
initrd        /boot/initrd.img-2.6.30-vortex86mx

Then press b to boot

  1. After booting, go to /boot/grub/menu.lst and make modifications from above permanently, so you will boot without any intervention ;)

**Edit:**Bob A. has also sent through some additional resources for your eboxing pleasures.

FYI - this Swedish company, http://www.lweb.se/tag/ubuntu/, has a pre-made ISO for Ubuntu 8.04LTS with the correct kernel for the eBox 3300/3310. It even supports the new (1011) IDE controller on the recent models. You can just put the ISO on a thumb drive, stick it in your eBox, and install normally. No need to install first on another machine, and no need to update the kernel after you’re done. If you’re happy with running 8.04 then this is way easier than any other install option that I’ve found
so far.