Sydney's Driving Habits

Published on Sunday, December 30, 2007

Status:

I've heard that Sydney is one of the most liveable cities in the entire world -- a statement I would generally agree with. However. whoever made this statement obviously didn't drive to work every day. I've reached the conclusion that one of the biggest (and only) drawbacks to Sydney, is the traffic.
So why is the traffic this bad? I have a few theories. The first theory is that Sydney drivers aren't particularly cordial. I mean, the road could be ending on the merging lane and drivers still won't let you in. I've literally sat for two blocks in near deadlock traffic with my blinker on, and nobody would let me in. Combine this with the fact that whoever designed Sydney's roads was drunk at the time (roads that suddenly veer off to one side, other roads that go from three lanes to one, roads that steer around a park causing 20 minute delays, or roads always having a bottleneck whilst crossing the rail tracks, etc.,). This plays a role in how everybody typically drives -- they drive very close to each other. I don't just mean tailgating, it is something else. I've been to quite a few cities in the world, and I've never seen people drive so close to each other.
This close driving has one serious repercussion: there are accidents every morning. When I list to the radio, I continually hear of several major accidents every day. Now, I can't stop people from tailgating, nor can I stop the accidents -- but what I can do is at least try to plot the bottlenecks around the city. I mean, after driving a route several times you'll already know the bottlenecks -- but I think it would be fun to visualize them.
I've already done a fair bit of the research how to technically graph it, although there is still quite a bit more to do. My plan is to use OpenLayersOpenStreetMapMapnik and of course Django to glue everything together. I'll let you know when I have a prototype working.

Small Business Server 2003

Published on Tuesday, December 11, 2007

Status:


I've recently been hired for a company that does consulting for small businesses. Naturally, I haven't had a huge exposure to SBS. My employer asked me to take the SBS MCP Certification, and this afternoon, I passed. The exam was pretty easy -- I think Microsoft basically took the easy questions spread out across the MCSE and dumped them into the SBS exam (some of the questions were very similar). My tips to anybody taking the exam:

  • Really know the benefits and drawbacks of SBS; for instance, it has to be the root of the forest, Premium vs. Standard, it comes with Exchange/SQL/ISA, always use the wizards first...
  • The exam is a showcase for SBS, and then a showcase for 2003.
  • There is usually a logical answer.


Well, I know that isn't helpful at all. But a start.

Exam 70-282: Designing, Deploying, and Managing a Network Solution for a Small- and Medium-Sized Business

Zenity GUI to a Shell Script

Published on Sunday, October 21, 2007




I have to admit, I'm pretty lazy. I don't (ironically) like to type, and I really don't like typing the same command over and over. I found myself switching between my external monitor and laptop quite frequently, and decided to somewhat automate the task. Although I know there are other programs out there that allow this, they either had too many features, or crashed. Xrandr works just fine, but like I said, I'm lazy...

Enter Zenity. Initially I created a PyGTK monitor switcher, yet wanted something even simpler. If you aren't in the know, Zenity allows you to create super fast, super simple dialogs to regular commands. After you click "ok", the command is executed, and the dialog disappears. Perfect for switching displays.

And here is the simplistic code behind it:

#!/bin/sh

ans=$(zenity  --list  --text "How do you want to switch your monitor?" \
--radiolist  --column "Pick" --column "Output Type" TRUE LCD FALSE VGA \
FALSE Both);

if [ "$ans" = "LCD" ]
then
xrandr --output VGA --off
xrandr --output LVDS --auto
elif [ "$ans" = "VGA" ]
then
xrandr --output LVDS --off
xrandr --output VGA --auto
elif [ "$ans" = "Both" ]
then
xrandr --output VGA --auto
xrandr --output LVDS --mode 1024x768
fi




I send a big cheers and thanks to the Zenity guys, I'll surely use this quick language more frequently.

Mass Spam Delete Django

Published on Tuesday, October 2, 2007

As you can read, I've been traveling around quite a bit lately. This means I haven't been checking the comments on my blog, which means quite a bit of spam has been entered. I am blocking the spam via akismet, however, it is still recorded in the database. Being somebody who hates cluttered desktops, you can imagine how I feel about having a lot (447) of spam. Well, since akismet flips the is_public switch True for good comments and False for bad comments, that makes a really easy query in mysql.

mysql> delete from comments_freecomment where is_public = False

Of course, make sure you have backed up your database first.

Works With Windows Vista

Published on Saturday, July 14, 2007

I'm in Tibet right now, so will keep this somewhat brief.

For the next few hours I'll be sitting in this dingy internet cafe in Lhasa updating my blog with my travels. I happened to look down and on the monitor it says "works with windows vista". I find this quite humorous, of course it is going to work with vista, it is a normal generic monitor.

Anyhow, I guess there is a reason for marketing. Back to writing our travels.

OpenMoko In Person

Published on Friday, June 29, 2007

Two weeks ago I had the privilege of meeting up with some of the OpenMoko guys. You've seen me write a little bit about this project, but meeting them and actually getting to play with a device was quite enlightening.

Most foreigners in Taipei take taxis, but not I. My girlfriend gave me a rough estimate how to go to FIC headquarters, yet it was -- rough. I jumped on the right bus, going the right direction, yet didn't really know where to get off. In my broken Chinese I started talking to a group of high school kids, who somewhat pointed me in the right direction. At one point we were coming up to a light and a lady just said: "get off, go down that road." I grabbed my bag and jumped off the bus. The street signs started to resemble the characters I had written down, but I didn't know which direction to go in. I asked a travel agency, and they pointed me in the right direction -- and then it started to rain. I luckily found FIC without getting too wet, and quickly met Sean, the project leader for OpenMoko.

Sean, myself, and several other OpenMoko developers went out to lunch and talked shop. Mickey pulled out a Neo, the hardware platform OpenMoko is going to be based off of. I'll admit, when I saw photos I wasn't really impressed -- but in person, it is pretty sexy. Not too big, but big enough to be functional. In one of my previous posts I mentioned learning C++ back in university, and having basically not touched it since. In a way, I haven't had to, and I've recently found Python, which is just fun. However, I'm so excited about the OpenMoko, I've picked up the K & R bible to read while we are traveling. Not the ideal way to learn a language, but the concepts are somewhat a refresher. I can't wait to play with SIP/IAX on this thing, it will then be my dream phone.

Well, I have one more post to make to get some thoughts off my chest. If you want a postcard, send me an email.

Last Xenful Comments

Published on

One of the biggest things I regret is not utilizing Xen more. I've finally been admitted to Amazon's EC2 Limited Beta, just two days before I leave, so not enough time to actually do anything fun. However, I think Xen is an ideal infrastructure aid for SMEs in particular. The cost of technology is continuing to decrease, which means bigger servers cost less. This is great for the small/branch office. Let me explain.

One of the themes I noticed while studying and taking the MCSE was that the solution to the majority of the problems was to just buy more servers. Even for simple things like DHCP, buy another server. I've always operated on a limited budget, and anyways, I don't think money should be wasted on resources when it isn't needed. With a VT chipset, you aren't tied to any OS in particular.

My friend Ian and I were talking and he illustrated a great usage of Xen through his work. What he's ended up doing is installing the Small Business Edition of Server 2003 in a Xen node. The reasoning is that SBE is, apparently extremely difficult to create backups of -- mainly due to odd file locking behavior. I've had similar thoughts, but mainly taking advantage of Xen's migration feature. The idea of taking a small branch office and putting everything on a Xen server is quite appealing to me, especially considering a second server could be used to create virtual hot spare.

As you can see, I like Xen. I've found it relatively easy to install, and the fact that it is starting to come bundled with recent distributions is pretty darn, sweet.

My Portable Travel Stick

Published on

This will be my last post from Taiwan, and I'm placing it in my tech section. Shortly I will be flying to Hong Kong, and then traveling into China. I'm not bringing my laptop with me. I'm always a little wary of using public computers, especially in many of the poorly run internet cafes. Often the logged in user is the administrator, and we all know the computers are obviously crawling with worms and keyloggers. What can I do?

My partner kindly gave me a 128MB flash disk, which is perfect for what I want to do. I've installed the following applications to run directly from it:

  • Portable Clamwin - I plan to fire it up and do a memory scan before I start typing any passwords.

  • Portable Putty - This is useful for two reasons. Firstly, in case my server (or any server with SSH) needs help, I'm on it. Secondly, and more importantly, for security. Putty can easily be used as a SOCKS5 proxy over SSH, so I can tunnel Firefox and IM securely. Password sniffers, be gone! A side benefit is the ability to bypass the "Great Firewall", if needed (e.g. the block my Google account).

  • Portable Miranda - In case I'm feeling home sick, or have some crazy desire to talk on IRC. Don't count on it.

  • Firefox - I tried the Portable Apps package, yet it didn't work.

I noticed in the "known issues" that it doesn't work if loaded on a drive with a non-asci path, which this machine (and those in China) usually have. The "resolution" is to run it in Win98 compat mode, but this didn't work for me. To get around this, I downloaded the normal Firefox, installed it, copied the contents of "Mozilla Firefox" and dumped it in /Firefox. Then I created a profile directory called /FFProfile, and created a bat file called "firefox.bat":
start \Firefox\firefox.exe -profile \FFProfile


Double click the bat file, and you have FF running on your usb drive.
I'm in a search for a better keylogger detector, as I don't know how complete ClamAV will be. If you know of one, let me know. Until then, I'm going to pretend I have the perfect traveling USB companion.

IIS7 GoLive Program

Published on Monday, June 18, 2007

You can easily see that I am a big Linux fan, I've mentioned this over and over. I've also mentioned that I'm finishing up my MCSE, which started as a dare with my buddy, Ian. In the process of studying for my MCSE I've developed a certain amount of respect for Windows Server 2003, yet I'll comment more on this later.

What I really wanted to write two paragraphs about was an interesting discovery I made today. I happened to stumble over to Microsoft's hardware site, and noticed in my little Server Spy monitor that it was running IIS/7 -- the first time I have seen this anywhere. Upon further investigation I perused their GoLive program, which details several ways to test drive IIS7. There are some details about Microsoft's setup, which one would expect to be impressive. And it is. So, if you are a Microsoft person, you may want to grab a test drive of Beta3 while you still can.

Liferea RSS in Liferea

Published on Thursday, June 14, 2007

Today on Linux.com, Liferea was mentioned I find this particularly because I've been using Liferea for quite a few months, and haven't been able to find anything that better suits my needs. It sits quietly in the corner until a new RSS is available, and slightly changes color. Plus, it integrates wonderfully into GNOME.

Well, the Linux.com article does a great job summarizing the reader, so if you are in the market, I'd take a look. For posterities sake, I took a screenshot of the Linux.com feed of Liferea, in Liferea.

Computex 2007 Review

Published on

The buzz of Computex is finally wearing off, so I'll finally scribe my account of the event. YS was kind enough to go with me, we actually called it a date. The scene was quite similar to the trade shows we saw last year, however there seemed to be fewer people. This might be because we went on the final day, although we suspect there weren't that many people because there aren't "sweet" deals to be had. Regardless, there were several main "halls" filled full of stuff. The first three (two were in one building) weren't particularly interesting. I mean, there were some cool displays and lots of technology, but nothing that left me thinking "oh, that's cool" or "oh, me wanna." Until we hit the last hall.

The first place we stumbled upon was the SunComm booth, which seemed to have more "traffic" than any of the other VoIP booths. Let me say this: I think VoIP stuff is awesome, that it is going to be the future, and I dig looking at demos. Even though the SunComm website pretty much looks like crap, their products are way cool. I was lucky enough to purchase a phone directly from them, and have it mailed directly to my house in Taipei (which was just around the corner from their office). They don't normally do this.




Next up was one of the main reasons I even wanted to go to Computex: to see the OpenMoko phone. To be honest with you, if I had a million dollars, this would be one of the first things I would buy. The other would be a new ThinkPad X61s. So why is this phone so cool? First off, it is based on Linux, which if you haven't gathered, I like. The entire stack is open, or nearly open. I've always purchased pretty crap phones, or been given them for free. The reason I do this is because my requirements have never been met: I want a SIP/IAX client on a wifi tri (or preferably quad) band phone. GPS is a plus, but not necessarily required. So far, I haven't been able to fine anything that has suited my needs -- the HTC Universal almost would work, apparently. The slogan is "Free Your Phone", but I think of this project more to "make your phone never become obsolete." Indeed, the hardware may become obsolete, but the software never will. I keep being reminded that this project could *really* put a dent in the future iPhone sales. I can't wait to get one with WiFi and start hacking on it -- in the meantime, I'll need to refresh my C.







Up next, directly across the aisle, was Via's new line of motherboards. At Computex there were quite a few small sets of motherboards, but nothing quite like this. Smaller than mini-itx. Smaller than nano-itx. We have pico.

This thing is freaking small. Seeing this little board made me realize my desire to minimize how much (and the size) of the technology I own. Currently I'm selling my AMD Athlong64 3000 (it has two gigs of RAM), and will at some point attempt to switch to a single laptop solution (with Xen, of course). At the same time, I'm considering how useful one of these little boards could be. Many have two NICs, which would make perfect routers. I've seen some boards for as inexpensive as $65usd.

Tunneling over SSH

Published on Thursday, June 7, 2007

As a rule, whenever I'm online I'm logged into my server back in the States. I'm also usually wireless, which we all know is beyond insecure -- I've found it especially useful to tunnel firefox over SSH. I try my best to tunnel stuff over SSH back, and if you want to also, this is how.

Setup the SSH/SOCKS tunnel

I'm on Linux, so this is pretty darn easy.
ssh user@domain.com -D 1080

If the SSH daemon runs on a different port, you'd do something like this:
ssh -oPort=1234 user@damon.com -D 1080

Remember ports below 1024 are reserved, and you would need root access. Now it is time to configure the different programs to use the newly created tunnel.

Setting up Gnome (optional)

Tunneling Pidgen

Tunneling XChat

Tunneling Firefox
Note: I'm going to list two examples, one is with FoxyProxy and the other is with the ordinary proxy settings.

FoxyProxy

Normal Proxy

Make sure the other fields or empty, or you won't connect.

So, there you have it. There are quite a few unix shell providers out there, I'm sure it wouldn't be too hard to spot a link for one. I've seen QuadSpeedInternet having SSH access for $3/month, and JVDS or Lonestar offering possible free shells. Alternatively, you could just get a really inexpensive VPS at VPSLink ($6-$8/month, but they often have 25% off discounts).

Turn Off Google Suggest in Firefox

Published on

I generally like Firefox, and generally like Google. But having Google suggest enabled when I search for things is annoying, at least on my rather small laptop screen. Mostly it is because my internet right now is pretty crap, so I try to cut down on any extra traffic. Anyhow, I didn't even both Googling how to do this:

Type in 'about:config' into your address bar and type 'search' in the top filter bar. Look about half-way down for this:

browser.search.suggest.enabled


Double click it to turn it to false. All done.

Alexa Site Thumbnail with Python

Published on Wednesday, June 6, 2007

For one of my sites I needed to get thumbnails, yet Alexa Site Thumbnail didn't have any code snippets for Python. Well, no they/you do.

ThumbnailUtility.py



import base64
import datetime
import hmac
import sha
import sys
import re
import urllib
import xml.dom.minidom

AWS_ACCESS_KEY_ID = 'your-access-key-id'
AWS_SECRET_ACCESS_KEY = 'your-super-secret-key'

# This one is for an individual thumbnail...
def create_thumbnail(site_url, img_size):
    def generate_timestamp(dtime):
        return dtime.strftime("%Y-%m-%dT%H:%M:%SZ")
    def generate_signature(operation, timestamp, secret_access_key):
        my_sha_hmac = hmac.new(secret_access_key, operation + timestamp, sha)
        my_b64_hmac_digest = base64.encodestring(my_sha_hmac.digest()).strip()
        return my_b64_hmac_digest
    timestamp_datetime = datetime.datetime.utcnow()
    timestamp_list = list(timestamp_datetime.timetuple())
    timestamp_list[6] = 0
    timestamp_tuple = tuple(timestamp_list)
    timestamp = generate_timestamp(timestamp_datetime)
    signature = generate_signature('Thumbnail', timestamp, AWS_SECRET_ACCESS_KEY)
    parameters = {
        'AWSAccessKeyId': AWS_ACCESS_KEY_ID,
        'Timestamp': timestamp,
        'Signature': signature,
        'Url': site_url,
        'Action': 'Thumbnail',
        'Size': img_size,
        }
    url = 'http://ast.amazonaws.com/?'
    result_xmlstr = urllib.urlopen(url, urllib.urlencode(parameters)).read()
    result_xml = xml.dom.minidom.parseString(result_xmlstr)
    image_url = result_xml.childNodes[0].getElementsByTagName('aws:Thumbnail')[0].firstChild.data
    return image_url
  
# And this one is for a list
def create_thumbnail_list(all_sites, img_size):
    def generate_timestamp(dtime):
        return dtime.strftime("%Y-%m-%dT%H:%M:%SZ")
    
    def generate_signature(operation, timestamp, secret_access_key):
        my_sha_hmac = hmac.new(secret_access_key, operation + timestamp, sha)
        my_b64_hmac_digest = base64.encodestring(my_sha_hmac.digest()).strip()
        return my_b64_hmac_digest
    
    timestamp_datetime = datetime.datetime.utcnow()
    timestamp_list = list(timestamp_datetime.timetuple())
    timestamp_list[6] = 0
    timestamp_tuple = tuple(timestamp_list)
    timestamp = generate_timestamp(timestamp_datetime)
    
    signature = generate_signature('Thumbnail', timestamp, AWS_SECRET_ACCESS_KEY)
    
    image_loc = {}
    image_num = []
    image_size = {}
    
    count = 1   
    for s in all_sites:
        image_num = 'Thumbnail.%s.Url' % count
        image_loc[image_num] = s
        count += 1
        
    parameters = {
        'AWSAccessKeyId': AWS_ACCESS_KEY_ID,
        'Timestamp': timestamp,
        'Signature': signature,
        'Action': 'Thumbnail',
        'Thumbnail.Shared.Size': img_size,
        }
        
    parameters.update(image_loc)
    
    ast_url = 'http://ast.amazonaws.com/?'
        
    result_xmlstr = urllib.urlopen(ast_url, urllib.urlencode(parameters)).read()
    result_xml = xml.dom.minidom.parseString(result_xmlstr)
    
    image_urls = []
    count = 0
    for s in all_sites:
        image_urls.append(result_xml.childNodes[0].getElementsByTagName('aws:Thumbnail')[count].firstChild.data)
        count += 1
    return image_urls



This is how you interact with this code for a single thumbnail:
>>> from ThumbnailUtility import *
>>> create_thumbnail('kelvinism.com', 'Large')
u'http://s3-external-1.amazonaws.com/alexa-thumbnails/A46FF6A30BECB0730455F2AB306EDC28605BC19Cl?Signature=XpsxgPey4b0JgreZA46XnvHVVLo%3D&Expires=1181110547&AWSAccessKeyId=1FVZ0JNEJDA5TK457CR2'

And for a list:
>>> from ThumbnailUtility import *
>>> all_sites = ['kelvinism.com', 'alexa.com', 'vpslink.com']
>>> create_thumbnail_list(all_sites, 'Small')
[u'http://s3-external-1.amazonaws.com/alexa-thumbnails/A46FF6A30BECB0730455F2AB306EDC28605BC19Cs?Signature=%2BfcOUKwH4xD9IH9o1vfto%2FMoALU%3D&Expires=1181110698&AWSAccessKeyId=1FVZ0JNEJDA5TK457CR2', u'http://s3-external-1.amazonaws.com/alexa-thumbnails/D798D8CE8F821FCC63159C92C85B70319E44D0EFs?Signature=6jriChrGM%2F8DoejN9dn9Dv3Lc5w%3D&Expires=1181110698&AWSAccessKeyId=1FVZ0JNEJDA5TK457CR2', u'http://s3-external-1.amazonaws.com/alexa-thumbnails/23529C34E0518AA9C2577653AC237D3647BA8D2Ds?Signature=5ksuwZx0I5TqXWL3Kt%2BWP6r2LQk%3D&Expires=1181110698&AWSAccessKeyId=1FVZ0JNEJDA5TK457CR2']


This is just a simple example to get your feet wet, maybe you'll find it useful. If you are wondering how to integrate this with Django, don't worry, I've got you covered.

Alexa Site Thumbnail with Python II

Published on

This is how I actually use Alexa Site Thumbnail, and since I'min a sharing mood, I'll extend the code your way. In short, this takes the url and searches in STORELOC first, then any urls not already in STORELOC are retrieved and named via a slug. You need to pass two variables to either of these: blog_site.url and blot_site.slug -- since I'm using Django, this is naturally how sites are returned after I filter a queryset. What I do is place the call to Alexa as high up the page as I can, and because I've threaded this, the page can continue to load without waiting for Alexa's response. For instance, let's say you have some model with cool sites, and you want to return the sites filtered by owner...

views.py


from getAST import create_thumbnail_list
blog_sites = CoolSiteListing.objects.filter(owner__username__iexact=user_name, is_active=True)
create_thumbnail_list(blog_sites).start()


Notice the .start() on the create_thumbnail_list function? That starts the thread.

getAST.py



import base64
import datetime
import hmac
import sha
import sys
import re
import urllib
import xml.dom.minidom
import os
import threading

AWS_ACCESS_KEY_ID = 'your-access-key-id'
AWS_SECRET_ACCESS_KEY = 'your-super-secret-key'
STORELOC = "/path/to/store/thumbs/"

# This one is for an individual thumbnail...
class create_thumbnail(threading.Thread):
   # Override Thread's __init__ method to accept the parameters needed:
    def __init__(self, site_url, site_slug):
        self.site_url = site_url
        self.site_slug = site_slug
        threading.Thread.__init__(self)
        
    def run(self):
        # First check if the thumbnail exists already
        # site_slug is the name of thumbnail, for instance
        # I would generate the slug of my site as kelvinism_com,
        # and the entire image would be kelvinism_com.jpg 
        if not os.path.isfile(STORELOC+self.site_slug+".jpg"):
            def generate_timestamp(dtime):
                return dtime.strftime("%Y-%m-%dT%H:%M:%SZ")
            def generate_signature(operation, timestamp, secret_access_key):
                my_sha_hmac = hmac.new(secret_access_key, operation + timestamp, sha)
                my_b64_hmac_digest = base64.encodestring(my_sha_hmac.digest()).strip()
                return my_b64_hmac_digest
            timestamp_datetime = datetime.datetime.utcnow()
            timestamp_list = list(timestamp_datetime.timetuple())
            timestamp_list[6] = 0
            timestamp_tuple = tuple(timestamp_list)
            timestamp = generate_timestamp(timestamp_datetime)
            signature = generate_signature('Thumbnail', timestamp, AWS_SECRET_ACCESS_KEY)
            parameters = {
                'AWSAccessKeyId': AWS_ACCESS_KEY_ID,
                'Timestamp': timestamp,
                'Signature': signature,
                'Url': self.site_url,
                'Action': 'Thumbnail',
                }
            url = 'http://ast.amazonaws.com/?'
            result_xmlstr = urllib.urlopen(url, urllib.urlencode(parameters)).read()
            result_xml = xml.dom.minidom.parseString(result_xmlstr)
            image_urls = result_xml.childNodes[0].getElementsByTagName('aws:Thumbnail')[0].firstChild.data
            #image_name = re.sub("\.|\/", "_", result_xml.childNodes[0].getElementsByTagName('aws:RequestUrl')[0].firstChild.data) + ".jpg"
            image_name = self.site_slug + ".jpg"
            store_name = STORELOC + image_name
            urllib.urlretrieve(image_urls, store_name)
            return image_name
  
# And this one is for a list
class create_thumbnail_list(threading.Thread):
   # Override Thread's __init__ method to accept the parameters needed:
   def __init__(self, all_sites):
      self.all_sites = all_sites
      threading.Thread.__init__(self)
   def run(self):     
        SITES = []
        # go through the sites and only request the ones that don't
        # exist yet
        for s in self.all_sites:
            if not os.path.isfile(STORELOC+s.slug+"SM.jpg"):
                SITES.append(s)
                       
        if SITES: 
            def generate_timestamp(dtime):
                return dtime.strftime("%Y-%m-%dT%H:%M:%SZ")
            
            def generate_signature(operation, timestamp, secret_access_key):
                my_sha_hmac = hmac.new(secret_access_key, operation + timestamp, sha)
                my_b64_hmac_digest = base64.encodestring(my_sha_hmac.digest()).strip()
                return my_b64_hmac_digest
            
            timestamp_datetime = datetime.datetime.utcnow()
            timestamp_list = list(timestamp_datetime.timetuple())
            timestamp_list[6] = 0
            timestamp_tuple = tuple(timestamp_list)
            timestamp = generate_timestamp(timestamp_datetime)
            
            signature = generate_signature('Thumbnail', timestamp, AWS_SECRET_ACCESS_KEY)
            
            image_loc = {}
            image_num = []
            image_size = {}
            
            count = 1   
            for s in SITES:
                image_num = 'Thumbnail.%s.Url' % count
                image_loc[image_num] = s.url
                count += 1
                
            parameters = {
                'AWSAccessKeyId': AWS_ACCESS_KEY_ID,
                'Timestamp': timestamp,
                'Signature': signature,
                'Action': 'Thumbnail',
                'Thumbnail.Shared.Size': 'Small',
                }
                
            parameters.update(image_loc)
            
            ast_url = 'http://ast.amazonaws.com/?'
                
            result_xmlstr = urllib.urlopen(ast_url, urllib.urlencode(parameters)).read()
            result_xml = xml.dom.minidom.parseString(result_xmlstr)
    
            count = 0
            for s in SITES:
                image_urls = result_xml.childNodes[0].getElementsByTagName('aws:Thumbnail')[count].firstChild.data
                image_name = s.slug + "SM.jpg"
                store_name = STORELOC + image_name
                urllib.urlretrieve(image_urls, store_name)
                count += 1

Alexa Site Thumbnail And Django

Published on

So, you've seen how to look up thumbnails via python, but wonder how to integrate this with Django? I created a sample app to demonstrate. One thing to note about this app is it is slightly more complex than just using the previously mentioned ThumbnailUtility. For starters, the thumbnail is downloaded from Alexa onto the server. Another part is first searching if the thumbnail exists already, and if it does, serving that instead of querying Alexa. Let's just start with some code.

getAST.py





#!/usr/bin/python
import base64
import datetime
import hmac
import sha
import sys
import re
import urllib
import xml.dom.minidom
import os
from urlparse import urlsplit

AWS_ACCESS_KEY_ID = 'your-access-key-id'
AWS_SECRET_ACCESS_KEY = 'your-secret-key'
STORELOC = "/path/to/store/webthumbs/"

def create_thumbnail(site_url):
image_name = re.sub("\.|\/", "_", '.'.join(urlsplit(site_url)[1].rsplit('.', 2)[-2:])) + ".jpg"
if not os.path.isfile(STORELOC+image_name):
def generate_timestamp(dtime):
return dtime.strftime("%Y-%m-%dT%H:%M:%SZ")
def generate_signature(operation, timestamp, secret_access_key):
my_sha_hmac = hmac.new(secret_access_key, operation + timestamp, sha)
my_b64_hmac_digest = base64.encodestring(my_sha_hmac.digest()).strip()
return my_b64_hmac_digest
timestamp_datetime = datetime.datetime.utcnow()
timestamp_list = list(timestamp_datetime.timetuple())
timestamp_list[6] = 0
timestamp_tuple = tuple(timestamp_list)
timestamp = generate_timestamp(timestamp_datetime)
signature = generate_signature('Thumbnail', timestamp, AWS_SECRET_ACCESS_KEY)
parameters = {
'AWSAccessKeyId': AWS_ACCESS_KEY_ID,
'Timestamp': timestamp,
'Signature': signature,
'Url': site_url,
'Action': 'Thumbnail',
}
url = 'http://ast.amazonaws.com/?'
result_xmlstr = urllib.urlopen(url, urllib.urlencode(parameters)).read()
result_xml = xml.dom.minidom.parseString(result_xmlstr)
image_urls = result_xml.childNodes[0].getElementsByTagName('aws:Thumbnail')[0].firstChild.data
store_name = STORELOC + image_name
urllib.urlretrieve(image_urls, store_name)
return image_name





Not too much to mention here, basically just an extended version of the ThumbnailUtility. The only difference is the test at the beginning, and actually downloading the thumbnail.

views.py



# Create your views here.
from django.shortcuts import render_to_response, get_object_or_404
from django.http import HttpResponseRedirect, HttpResponse
from webthumbs.models import *
from django import newforms as forms
from getAST import create_thumbnail

attrs_dict = { 'class': 'form-highlight' }

class imageForm(forms.Form):
url = forms.URLField(max_length=100, verify_exists=True, widget=forms.TextInput(attrs=attrs_dict), initial='http://', label='Site URL')

def index(request):
disp_img = ''
# generate default form
f = imageForm()
# handle add events
if request.method == 'POST':
if request.POST['submit_action'] == 'Submit':
# attempt to do add
add_f = imageForm(request.POST)
if add_f.is_valid():
site_url = request.POST['url']
disp_img = create_thumbnail(site_url)
else:
f = add_f
return render_to_response(
'webthumbs/index.html',
{'printform': f,
'disp_img': disp_img
}
)




The key thing to look at here is how getAST is called:



site_url = request.POST['url']
disp_img = create_thumbnail(site_url)



index.html



{% extends "base.html" %}

{% block title %}Kelvinism.com - Blog{% endblock %}

{% block content %}

What thumbnail do you want?


{{ printform }}







{% endblock %}


So, there you have it, the code to take a url via form and display it right away.

Adding Search to Django

Published on Sunday, June 3, 2007

This is fairly well documented in the Django docs, so I'll be brief. This is the the bit of search code I use in almost all of my Django sites, and it works great:

def search(request):
    from django.db.models import Q
    q = request.GET.get("q", "")
    if q and len(q) >= 3:
        clause = Q(dirtword__icontains=q)               \
               | Q(description__icontains=q)       \
               | Q(tags__name__icontains=q)        
        site_search = Dirt.objects.filter(clause).distinct()
    else:
        site_search = Dirt.objects.order_by('?')[:100]
    return list_detail.object_list(
        request              = request,
        queryset             = site_search,
        template_name        = "dirty/search.html",
        template_object_name = "dirty",
        paginate_by          = 20,
        extra_context        = {"q" : q},
    )  


While this should be pretty self-explanatory, the process goes like this: q is taken from the GET request and if it is over three characters long, it is searched for in the dirtword column, through the description and also through the m2m relationship of tags__name. Yup, it is pretty nifty to be able to access relationship in this way (tags__name). You will notice that at the end of each search it says "__icontains" -- this simply does a fuzzy search for the word. Once the queryset is created (the filter) I've added a .distinct() on the end --this prevents multiple rows from being returned to the template. If there isn't a search, or it isn't long enough, a random list will be returned.
One thing I like to do is include the search as extra_context -- this allows you to say something like "you've searched for..." at the top of your search. I couldn't imagine implementing a search feature as being any easier.

Django Syndication with Colddirt

Published on

Creating feeds in Django is freaking simple. I'll start with an example of just updating a feed with the newest objects (for instace, newest blog posts). Similar to the forms.py way of handling our different forms, I've created a feeds.py to handle the feeds.

feeds.py



from django.contrib.syndication.feeds import Feed
from colddirt.dirty.models import Dirt

class LatestDirts(Feed):
    title = "Cold Dirt"
    link = "/"
    description = "When you have dirt, you've got dirt.  Right..."
    copyright = 'All Rights Unreserved'
    
    def items(self):
        return Dirt.objects.all()[:50]



What this will do is query our Dirt DB and return an obj. The fields here are pretty well documented in the Django docs, besides being pretty obvious.

urls.py


We need three things in our urls.py -- first, import our feeds from feeds.py:
from colddirt.feeds import LatestDirts


Next, we map the feed we want to a name urls.py can use:
feeds = {
    'newdirt': LatestDirts,
}


Finally we create which URL to use for the feeds:
    (r'^feeds/(?P.*)/$', 'django.contrib.syndication.views.feed', {'feed_dict': feeds}),


When we look at a request, the process goes like this: it comes in as /feeds/newdirt/, which then gets looked up and matches newdirt in the feeds_dict. Next, LatestDirts is looked at and evaluated, and returned. But how is it returned? One final thing to do is create a template for the feed to use, which is where we can tell exactly what is to be displayed to the user.

templates/feeds/newdirt_title.html


{{ obj.dirtword }}

templates/feeds/newdirt_description.html


{{ obj.description }}

The naming for the templates, as usual, is important. If you want to have that little orange RSS button near your url, add this to your template's head:


So, there you have it, a simple example of how to use Django's syndication framework. I'll follow this up with another slightly more complex tutorial.

Django Syndication with Colddirt II

Published on

Since I've already covered a really simple syndication example, I'll move onto something a little more complex. Let's say you want to offer syndication that is slightly more custom. The Django syndication docs give an example from Adrian's Chicagocrime.org syndication of beats. I had to ponder a minute to get "custom" syndication to work, so here's my example from start to finish.
First, as usual, feeds.py

feeds.py


class PerDirt(Feed):

    link = "/"
    copyright = 'Copyright (c) 2007, Blog Mozaic'
    
    def get_object(self, bits):
        from django.shortcuts import get_object_or_404
        if len(bits) != 1:
            raise ObjectDoesNotExist
        my_dirt = get_object_or_404(Dirt, slug__exact=bits[0])
        return my_dirt

    def title(self, obj):
        return obj.slug
    
    def description(self, obj):
        return obj.description
    
    def items(self, obj):
        from django.contrib.comments.models import FreeComment
        return FreeComment.objects.filter(object_id=obj.id).order_by('-submit_date')

You can see that this differs slightly from the simpler syndication example. I'll not a few things. But first, I need to show urls.py:

urls.py


feeds = {
    'mydirt': PerDirt,
}

urlpatterns = patterns('',
    (r'^feeds/(?P<url>.*)/$', 'django.contrib.syndication.views.feed', {'feed_dict': feeds}),
)




Let's pretend the dirt word (or if you were doing a blog, you could do this based on slug) is "nifty". So, the process is like this: a request comes in as /feeds/mydirt/nifty/ -- it is first looked up in the feed_dict (because of the mydirt part) and then sent to PerDict. Once in PerDict it hits the first def, get_object. One of the things that confused me at first is what the 'bits' part is. Simply put: it is the crap after you tell Django which feed to use. Similar to the beats example, I'm testing to make sure the length is only one -- so if the word doesn't exist or somebody just types in feeds/mydirt/nifty/yeehaaa/ -- they will get an error. Next the object is looked up, in this case the dirt word (in your case, maybe a blog entry).
The title and description are self-explanatory. The items are a query from the FreeComment database, ordered by date. What we need next is the correct templates.

templates/feeds/mydirt_title.html



Comment by {{ obj.person_name }}





Once again, the filename is important (mydirt_title). obj.person_name is the name from the comment.

templates/feeds/mydirt_description.html





{{ obj.comment }}
Posted by: {{ obj.person_name }}
Published: {{ obj.submit_date }}





If you are curious how to get that little orange icon next to your site's url, you do this:


That's it. Hopefully I've explained how to create somewhat custom syndication feeds, in case you needed another example.

Simple Ajax with Django

Published on Friday, June 1, 2007

So, the Django developers, in my opinion, are freaking smart. Instead of bundling Django with a particular library, they have added XML and JSON serialization; us humble users can choose whatever AJAX library we want. Prototype 1.5.1 has been pretty fun to work with, so I'll kick off this demo with a really simple example.
How simple? The intended goal is to have the total number of 'dirts' update without user intervention. Laaaammmeee. If you are a visual type of person, take a look on the Colddirt huh page. That number automatically increases without user intervention. And this is how.
The process (some pseudocode) will go like this:

check /dirt_count/ for an update

if update:

make number bigger

else:

check less frequently


Pretty simple, eh?

urls.py


    (r'^dirt_count/$', views.dirt_count),

As you can see, it just sends the request to the view.

views.py


def dirt_count(request):
    from django.http import HttpResponse
    countd = str(Dirt.objects.all().count())
    return HttpResponse(countd, mimetype='text/plain')


Pretty simple -- count the dirts. That makes sense.

dirty.js


new Ajax.PeriodicalUpdater('dirtcount', '/dirt_count/', {
  method: 'get',
  frequency: 3,
  decay: 2,
});


Yea, Prototype makes that real easy. Just make sure to have a element in your template somewhere with an id of 'dirtcount'.

templates/huh.html


0

Django Newforms Usage in Colddirt

Published on

I hear many complaints and questions about newforms, but I've personally found it rather easy and logical to use. There are numerous ways for you to use do forms in Django, and most likely the best way to see them all is to read the docs. On the Colddirt demo site, this is how I used newforms. I'll take the index page as an example.
I've accessed the newforms module like so:
from django import newforms as forms


The next thing to look at is the actual creation of the form. You can keep your forms in models.py, although there is a trend going to keep them in a forms.py file. That is they are for Colddirt.

forms.py



attrs_dict = { 'class': 'norm_required' }
tag_dict = { 'class': 'tag_required' }

class DirtyForm(forms.Form):
    description = forms.CharField(widget=forms.Textarea(attrs=textbox_dict), label='What\'s your cold dirt?')
    tag_list = forms.CharField(max_length=150, widget=forms.TextInput(attrs=tag_dict), label='Tags')


I'm keeping it simple for now. Some key things to note is the field type (CharField) and the widget type (Textarea/TextInput). You can guess what each means. Here's a gem for your tool chest: how do you apply CSS to the forms? That is what the 'attrs=' part is about -- that will put in a class for you to assign CSS to. Nifty. The label creates a 'label' element that you can access. Let's render the form and send it to the template.
To get a form displayed we need to generate the form, and send it to the template.

views.py


dirt_form = DirtyForm() 


Send it to the template.

views.py


    return list_detail.object_list(
           request,
           queryset = Dirt.objects.order_by('?')[:100],
           template_name = "dirty/dirt_list.html",
           template_object_name = "dirty",
           paginate_by = 10,
           extra_context = {'form': dirt_form.as_ul()}
        )


That's it, although we will revisit this index view shortly. One important thing to note is the .as_ul() appended to the form element. This tells the template to encapsulate the form elements as list elements (as opposed to say, a table). Now, let's display the form.

templates/dirt_list.html


{% if form %}
        
    {{ form }}
{% endif %}


The form thus appears because of the block, {{ form }}. You can see in the action type that it will post the data to the index page->view. Let's revisit the entire index view now.

views.py



def index(request):
    import re
    from django.template.defaultfilters import slugify
    dirt_form = DirtyForm()
    if request.method == 'POST':
        dirt_form = DirtyForm(request.POST)
        if dirt_form.is_valid():
            # I opted not to create an m2m relationship for several
            # reasons. Note: the latest_word is some random word.
            latest_word = Word.objects.filter(is_taken__exact=False).order_by('?')[:1]            
            latest_word[0].is_taken=True
            latest_word[0].save()
            new_dirt = Dirt(description=dirt_form.clean_data['description'],
                            dirtword=latest_word[0].dirtyword)
            new_dirt.save()
            # Credit for the tag parsing goes to Ubernostrum (James B)
            # from the Cab (great for learning!) project
            # I opted to not store tag_list in each entry
            # Splitting to get the new tag list is tricky, because people
            # will stick commas and other whitespace in the darndest places.
            new_tag_list = [t for t in re.split('[\s,]+', dirt_form.clean_data['tag_list']) if t]
            # Now add the tags
            for tag_name in new_tag_list:
                tag, created = Tagling.objects.get_or_create(name=slugify(tag_name), slug=slugify(tag_name))
                new_dirt.tags.add(tag)
            return HttpResponseRedirect(new_dirt.get_absolute_url())
    return list_detail.object_list(
           request,
           queryset = Dirt.objects.order_by('?')[:100],
           template_name = "dirty/dirt_list.html",
           template_object_name = "dirty",
           paginate_by = 10,
           extra_context = {'form': dirt_form.as_ul()}
        )


Let me pretend I am the form and have just been submitted to the view. First I'm tested if I'm a POST. Next, my data is dumped into the dirt_form variable. I'm then tested if I'm valid data or not (validation explanation next). Since I'm valid data, stuff happens. In the instance of Colddirt, a random word is taken from the Word database. The word is then updated as is_taken, and saved. Then the dirt is actually created. One thing to notice is how we access form data:
description=dirt_form.clean_data['description']


So, the new dirt (with description and new word) is saved. Next, let's deal with the tags. Credit goes to James for parsing the tag_list.
            new_tag_list = [t for t in re.split('[\s,]+', dirt_form.clean_data['tag_list']) if t]
            # Now add the tags
            for tag_name in new_tag_list:
                tag, created = Tagling.objects.get_or_create(name=slugify(tag_name), slug=slugify(tag_name))
                new_dirt.tags.add(tag)


You can see dirt_form.clean_data used again. Another neat trick is to use slugify to make sure your tags are lowercase and aren't all weirdo like. The user is then redirected to the absolute url of the dirt the just created.
So what about validation? Don't think I forgot this one. Validation (from what I have seen) is actually really easy. I'm going to first display the entire form.

forms.py


class DirtyForm(forms.Form):
    description = forms.CharField(widget=forms.Textarea(attrs=textbox_dict), label='What\'s your cold dirt?')
    tag_list = forms.CharField(max_length=150, widget=forms.TextInput(attrs=tag_dict), label='Tags')
 
    def clean_description(self):
        import re
        if self.clean_data.get('description'):
            value = self.clean_data['description']
            if len(value) > 20 and not re.search('[<>]', value):
                try: 
                    hasNoProfanities(value, None)
                    return value
                except:
                    raise forms.ValidationError(u'Extremily dirty words, racial slurs and random crap characters are not allowed in dirt.') 
            else:
                raise forms.ValidationError(u'A little more detail please. No HTML.')
               
    def clean_tag_list(self):
        if self.clean_data.get('tag_list'):
            value = self.clean_data['tag_list']
            try: 
                hasNoProfanities(value, None)
                return value
            except:
                raise forms.ValidationError(u'Extremily dirty words or racial slurs are not allowed!') 


There is a fair amount of normal validation that occurs in the is_valid process, but here is some extra validation I added. Inside the DirtyForm class (as you can see) simply add a test for if the data is 'clean' or not (I don't know how to beter phrase this -- "send the data to the cleaners"). I'm testing the description to make sure it is long enough, and to make sure it doesn't have <>'s in it (to prevent XSS and odd stuff). If it detects them, the lower error is displayed. I've also tied in the hasNoProfanities validation, which pulls the words from my settings file. Not that I care if people swear or not, I'm mainly using this to prevent racial slurs, which I do care about.
So, there you have it, one example of how newforms is used in a 'live' site. I hope this is helpful for somebody, I wish I could have seen more newforms examples when I started learning. If you are truly stumped on something, take a look inside the django source (/tests/regressiontests/forms/tests.py) for a lot of examples of every way you could use newforms.

Colddirt Information

Published on Tuesday, May 29, 2007


Note: Colddirt's source code is run from Django SVN, check out on May 10th-ish. If you are using a newer branch, some things have changed. i.e. clean_data has been renamed cleaned_data. Remember to check the BackwardsIncompatible page in the wiki.

Part 1: Simple AJAX with Django
Part 2: Django Newforms Usage
Part 3: Search with Django
Part 4: Django's Syndication Framework (simple)
Part 5: Django's Syndication Framework (little more complex)

Feel free to see the source code from Github.

Ubuntu Upgrade

Published on Saturday, May 26, 2007

Having been in I.T. for quite a few years, upgrading can sometimes be quite a hassle. Having switched to Linux for many years, the crazy upgrade madness of windows is gone. So, how easy is upgrading in Linux?





Yea, pretty darn easy, I must admit. And my system certainly isn't normal! It has been an upgrade from Debian Sarge(ish) -> Ubuntu LTS(ish) -> 6.10 -> 7.04... The only hiccup was at some point when the fonts were reinstalling (I totally ignored the "close all programs" warning:





Solved: NO PUBKEY

Published on Friday, May 25, 2007

I've received this error more than once, so I'm finally writing my notes how I solve it.

Error message:


W: GPG error: http://security.debian.org stable/updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A70DAF536070D3A1

This really is just your standard don't-have-the-gpg-keys error. So, get'em -- take the last eight digits from the long NO_PUBKEY string that is displayed on your computer. If you are using Debian 4.0, the above key is likely correct; if you are using Ubuntu or another version of Debian, it will be wrong. (The last eight digits are used as an identifier at the keyservers). Then:
gpg --keyserver subkeys.pgp.net --recv-keys 6070D3A1
gpg --export 6070D3A1 | apt-key add -


Repeat if necessary. All done, just do an apt-get update and no more warning!

Bare-metal Restore

Published on

As you can see by my previous post, my question to squeeze more req/sec from the server, I decided to try out Gentoo (again, last time was four years ago). Now, I like Gentoo, there is no doubt about that. However, I realized things took just too long to get set up. I guess that is the disadvantage of a source based package management tool. Back to Debian I go.

Two hours later everything was up and running -- and I guess I can't complain about a two hour bare-metal restore from one distro to another. And let me iterate, this isn't just a typical LAMP boxen. It's got:


  • apache/mod_php/ssl with ~10 domains

  • apache/mod_python/ssl with ~4 domains

  • lighttpd with ~5 domains (static files)

  • about 8 gigs of web data/images

  • svn repos + web_dav access

  • mysql restored

  • postfix(chroot)/dovecot/sasl + mysql auth

  • home dirs restored

  • chrooted users again



I'm sure I missed something on this list, I was typing pretty quick. Well, that's the update. I'm gonna go tinker with mod_cache some.



Postfix/Dovecot + MySQL

Published on Thursday, May 24, 2007

As you can see by another post, I decided to reinstall the server. This isn't really a problem, I have pretty good backups. I've installed apache and friends a bagillion times. However, Postfix(chroot)+Dovecot authenticating from MySQl, that doesn't install quite so smoothly.
Just for my future reference, and maybe helpful for somebody, someday. Clearly not a tutorial. The postfix chroot = /var/spool/postfix

cannot connect to saslauthd server: No such file or directory



First, get the saslauthd files into the postfix chroot. Edit /etc/conf.d/saslauthd (or /etc/default/saslauthd), and add this:
SASLAUTHD_OPTS="-m /var/spool/postfix/var/run/saslauthd"


Second, add it to the init script.
stop() {
        ebegin "Stopping saslauthd"
        start-stop-daemon --stop --quiet /
--pidfile /var/spool/postfix/var/run/saslauthd/saslauthd.pid
        eend $?
}


Third, maybe, change /etc/sasl2/smtpd.conf (or /etc/postfix/sasl/smtpd.conf) and add this:
saslauthd_path: /var/run/saslauthd/mux


Ok, that error should go away now.

Recipient address rejected: Domain not found;


(Host or domain name not found. Name service error for name=domain.com



These are actually the same type of error. Copy /etc/resolv.conf into the chroot.

fatal: unknown service: smtp/tcp



Copy /etc/services into the chroot.
I searched google for these answers, to a certain degree at least, but couldn't really find much. Then I remembered "crap, this is a chroot, it needs things" -- and fixed stuff. If you came here from google, and these super quick notes were helpful, feel free to leave a comment, or contact me directly if you have any questions.

The Gentoo test

Published on

I have a love-hate relationship with Linux. I love it because if there is a problem, I can actually tinker and find the problem and fix it. But I hate it because I like to tinker.

Recently I've been doing a fair amount of Django programming -- enjoying every minute of it. After completing several of my projects I decided to do some benchmarks, and the results are in! Generally I can server cached/semi-cached pages at about 200req/sec. 200req! Considering this is 5,000,000 or so requests a day, and a number I am never going to reach, I still began to wonder: why isn't it higher? I mean, serving a static html page is at like 1000+ req/sec, so why would a cached page be significantly different? I started exploring and noticed that Apache would spike the CPU. Ok, time to be thorough, and as I said, I like to tinker.

I tried lighttpd as a fastcgi to python -- not a significant different, basically the same. Next I tried several versions of python -- one from 2.4 and one from 2.5, one as a package and one from source -- same results. High cpu usage. Thinking it could be something related to my VPS (or some odd limit within Debian) I decided, ok, I'll reinstall.

I reinstalled and got things working pretty quickly. The only slight hiccup was postfix/dovecot, cause postfix insists on being in a jail (and my configs are all setup for that). Also, Chinese support in Apache isn't working. Regardless, I re-ran the benchmarks and the results were the same -- so, it isn't related to my previous install after all. Doh.

I'll evaluate Gentoo as a server setup for a little while, but I'm thinking I'll do a quick reinstall of Debian.


Generating a Self-Signed SSL Cert

Published on Friday, May 11, 2007

I have the need to generate an SSL cert (Apache2) about once every 3 months. And since I'm cheap, I don't ever actually *buy* one, I just self-sign it. And every time I forget the commands needed. So, here they are, for my reference only.
1) Generate Private Key

openssl genrsa -des3 -out server.key 1024

2) Generate a CSR

openssl req -new -key server.key -out server.csr

3) Remove passphrase

cp server.key server.key.org
openssl rsa -in server.key.org -out server.key

4) Generate Self-Signed Cert

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

Django SVN Update Goes Splat

Published on Wednesday, May 9, 2007

I'm writing this just in case somebody runs into this same issue. I'm about to go live with a website and figured it would be best to have the latest SVN snapshot checked out from Django. I updated, and noticed that my voting module didn't quite work as expected. I was getting the following error:
'module' object has no attribute 'GenericForeignKey'


I jumped into Trac and noticed that just yesterday some things were rearranged. In short, if you are using generic relations, you'll need to change two parts of your code. First, the generic relations field must be imported out of conttenttype.
from django.contrib.contenttypes import generic


And second, you'll need to change the 'location prefix' (for lack of a better description:
From:
generic_field = models.GenericRelation(SomeOtherModel)


To:
generic_field = generic.GenericRelation(SomeOtherModel)


All should be find from there on out. For more information, take a look at the reference wiki article.

PNG Transparency and IE

Published on Monday, May 7, 2007

I Hate IE!
I've vowed to not use transparent PNGs until almost everybody has switched to IE7, where they are actually supported (despite being supported by every other major browser). I've done the hacks, and have had good results. I like using PNGs, I'll admit it. Inkscape exports them directly, however one slight problem: transparency still exists. This isn't really a problem since I'm not layering images, or is it?
My initial assumption is that IE would simple pull the white background and everything would be dandy. Well, we all know what they say about assumptions.


A few options exist:

  • Convert them to GIFs

  • Try some sneaky PNG IE hack

  • Do a rewrite and send all IE6 traffic to download firefox. Err... Do a rewrite and send all IE6 traffic to download firefox

  • Open each in GIMP and add a white background

  • Use ImageMagick and convert the background to white.


We have a winner! The problem is, for the life of me, I couldn't figure out a simple convert command to do this. The quick bash script would suffice:
#/bin/bash
CONVERT=/usr/bin/convert
for image in *.png; do
 $CONVERT -background white $image $image
 echo "Finished converting: $image"
done


Note:This is gonna convert all PNGs.


So, no the transparent GIFs have a "white" base layer, IE renders them fine, normal browswers render the images fine, and I'm allowed a cup of coffee. I hope this helps somebody, if so, leave a note!

PSAD and Syslog-NG

Published on Wednesday, April 18, 2007

I really like using PSAD, both on my server and my laptop. You never know where the mean people are. I also seem to use syslog-ng quite often, meanwhile PSAD seems oriented to syslog. This is fine, and I'm pretty sure the install.pl for the source built will configure syslog-ng.conf automatically. However, I almost always tend to stick with packages if I can -- if they are even remotely close to the current version.
Anyways, if you need to get syslog-ng.conf configured for PSAD, this is what you need to do:
Add this code to the "# pipes" section, maybe stick to keeping it alphabetical.
destination psadpipe { pipe("/var/lib/psad/psadfifo"); };


Next, go down a little to the "# filters" section, add this:
filter f_kerninfo { facility(kern); };


And finally in the last section, add this:
log {
        source(s_all);
        filter(f_kerninfo);
        destination(psadpipe);
};


Restart syslog-ng, and you are good to go. Cheers to Michael Rash at Cipherdyne for his work on PSAD.

Simple Chrooted SSH

Published on Thursday, March 29, 2007

You might be asking: why would you want to chroot ssh? Why use ssh anyways? Here are the quick answers:

  • FTP usually sucks. Unless sent over SSL, all information is sent cleartext.

  • SSH usually rules. SSH sends all data over an encrypted channel -- the main drawback is: you can often browse around the system, and if permissions aren't set right, read things you shouldn't be able to.

  • Chroot'd SSH rocks. The solution to both the above problems
.


So, let me tell a quick story.
When I started uni in 2001 I was a nerd. Still a nerd, I guess. I was cramped in my apartment on campus with like 5 boxes, most of them crappy p100s running Linux or OpenBSD. Life was good.
I started a CS degree (shifted into Business with a focus on IT), and we were told to use the school's main servers to compile our programs. The other interesting thing is that all user accounts were visible when logged in via ssh -- but hey, that is just the nature of Linux. I knew this, but asked the head I.T. person "why don't you jail the connections?" He responded quickly telling me to go away.
Well, shortly after making the comment (although solutions existed at the time being), pam-chroot was released. This is right about the time students figured they could spam everybody in the school, some 25,000 emails, quickly and easily -- 'cause all the accounts were displayed. Sweet -- now we can chroot individual ssh connections.
This quick demo will be on Debian, we'll create a pretend user named "karl." (I'll assume you've already added the user before beginning these steps). Also, the jails will be in /var/chroot/{username}

First: Install libpam-chroot and makejail

session required pam_chroot.so

kelvin@server ~$ sudo apt-get install libpam-chroot makejail


Second: makejail config file



Put the following in /etc/makejail/create-user.py:
#Clean the jail

cleanJailFirst=1


preserve=["/html", "/home"]


chroot="/var/chroot/karl"


users=["root","karl"]

groups=["root","karl"]


packages=["coreutils"]



Edit: If you need to use SFTP also, try this config:


cleanJailFirst=1

preserve=["/html", "/home"]

chroot="/home/vhosts/karl"

forceCopy=["/usr/bin/scp", "/usr/lib/sftp-server", /

 "/usr/bin/find", "/dev/null", "/dev/zero"]

users=["root","karl"]

groups=["root","karl"]

packages=["coreutils"]


As you'll see, there is a "preserve" directive. This is so that when you "clean" the jail (if you need to refresh the files, for instance), you won't wipe out anything important. I created an /html so that the user can upload their web docs to that file.

Third: configure libpam_chroot


Add the following to /etc/pam.d/ssh:
# Set up chrootd ssh

session required pam_chroot.so


Forth: allow the actual user to be chrootd


Edit /etc/security/chroot.conf and add the following:
karl /var/chroot/karl


Fifth: create/chown the chroot'd dir


kelvin@server ~$ sudo mkdir -p /var/chroot/karl/home

kelvin@server ~$ sudo chown /var/chroot/karl/home


Now you should be able to log in, via the new username karl.

Layer Images Using ImageMagick

Published on Thursday, March 22, 2007

For one of my webapp projects I'm needing to layer two images. This isn't a problem on my laptop -- I just fire up GIMP, do some copy 'n pasting, and I'm done. However, since everything needs to be automated (scripted), and on a server -- well, you get the point.
The great ImageMagick toolkit comes to the rescue. This is highly documented elsewhere, so I'm going to be brief.

Take this:





And add it to this:





I first tried to use the following technique:
convert bg.jpg -gravity center world.png -composite test.png

This generated a pretty picture, what I wanted. What I didn't want was the fact that the picture was freaking 1.5 megs large, not to mention the resources were a little high:
real    0m7.405s
user    0m7.064s
sys     0m0.112s


Next, I tried to just use composite.
composite -gravity center world.png bg.png output.png

Same results, although the resource usage was just a tad lower. So, what was I doing wrong? I explored a little and realized I was slightly being a muppet. I was using a bng background that was 1.2 megs large (long story). I further changed the compose type to "atop," as that is what appeared to have the lowest resource usage. I modified things appropriately:
 composite -compose atop -gravity center world.png bg.jpg output.jpg


This also yielded an acceptable resource usage.

The result:



A Dying Laptop

Published on Tuesday, March 20, 2007



I have the pleasure of owning an old T23 laptop. To show you how old this puppy is, the current T series is at T60, and those have been out for over a year. This laptop was made in 2012, and I picked it up somewhat discounted late in 2003. It is now March 2007, and this puppy is still rock solid.

You heard me, it is almost six years old and still working fine -- that is testimony to how well this laptop was built. There are several small cracks around the case, but nothing you would notice by just walking by. This laptop has been to more countries than many people.

I had the first problem this weekend, and it isn't even related to the laptop. The hard drive, a 30G I put in at some point, started to crap out on me. Bad sectors were everywhere, so some of the programs were slightly unhappy (e.g. I couldn't boot into X).

I'm going to buy a new laptop soon, I promise, about the time my MBA goals are reached. Until then, I'll continue to be frugal, and deal with the bad sectors. Being a good IT nerd, everything is backed up to an external hard drive (and most stuff backed up remotely).

Luckily I'm using Linux -- so was able to runs fsck/smartmontools a few times in recovery mode, make the bad blocks happy, and continue as "normal." Phew, disaster averted.

One More Point Linux

Published on Thursday, March 15, 2007

It should come as a surprise that I enjoy using Linux. For the record, the first time I booted into Linux on my own was 1997, this was just before entering high school. So, while some of my tech friends played with NT, I was rumbling with the Penguin. Starting in 2000 I was using Linux as my main operating system, sometimes supplemented by OS X, and only using Windows when the gaming urge surfaced. In 2004 I mostly dropped playing any games, which resulted in dropping Windows -- and besides for work, I haven't used it since.

For me, I'll admit, there are three things that Linux still lacks:


  • Simplistic video conferencing support

  • Video editing support

  • Gaming



I know that all of these are supported, but, in my opinion, not particularly well. Well, I don't care about any of these enough to actually need windows, but it would be nice to see them improve.

So, I'm set. I'm 100% legal (don't steal a single piece of software). And don't have to be too afraid of virus'. What prompted me to write this little excerpt? A recent article at the Washington Post scared the beejeepers out of me, and makes me wish even more for Vista to either cure security problems, or everybody move over to Linux. The article details the aftermath a virus can cause, not on damaging one's computer, but on capturing information. The author further details his experience hunting down the data. This was one of the better articles I've read, and I thoroughly enjoyed the further details. If you want a little more motivation to move to Linux (or just tighten your machine), then I suggest you take a few moments to read the articles as well.

The Risk in Risk Mitigation

Published on

Back in the day the barrier to entry for the Internet was quite high. The technology used required a steep learning curve, the equipment extremely expensive, and sometimes even hard to acquire. Fast forward to 2007 and things have certainly changed. If you know any tech people you can likely get free hosting for a small website, and even more demanding websites can be hosted for not much. The cost of dedicated servers has dropped even more. And the final kicker: web services. I've started to think of some web services not as a service, but more like outsourcing requirements.

This very dependency adds risk for a multitude of reasons, and when your entire web application platform revolves around a third party, such as is the case with mashups, you incur great risk.


One of the nice things when requirements are outsourced is the fact that risk is mitigated. I'll use SmugMug as an example. In summary, they moved their storage to Amason's S3 network, which is something I will be utilizing as well. Amazon's S3 (and other web services) continue to drive down the barrier of entry -- now you don't even need to purchase hugely expensive servers for the sole purchase of storage! If you don't need to purchase them, you also don't need to manage them. Risk mitigated.

However, continuing the slight allusion from The Other Blog's article on mashups, I see a slight problem with the outsourcing of requirements. While the following thought isn't particularly innovative: mitigating risk and outsourcing requirements creates a dependency on the third-party. This very dependency adds risk for a multitude of reasons, and when your entire web application platform revolves around a third party, such as is the case with mashups, you incur great risk.

But, as is evident by the fact that I've had stitches nine different times, I'm still going to do some cool mashups anyways, so stay tuned.

I Hate GeoIP Advertising

Published on



Remember, I'm in a coffee shop? GeoIPized Girls = have to sit with back to wall.

GeoIP is the concept that allows you to map a user's IP to the country it came from (I hacked up my commenting system just so you can see an example, if you are a visual type of person). Advertisers have long since picked up on this as a way to make their ads "better," hopefully becoming more appealing to their clientele. Sometimes the adaptation doesn't quite work as expected. This morning I was searching for an eBook on Embedded Linux (it is really hard to find English tech books here) -- and just had to snap a few screenshots. The advertisers picked up my IP, from Taiwan, and "customized" the advertisement. Needless to say, I've seen a total of five white females here over the entire last year.

This reminded me just how much crap people know about me; since I'm in a coffeeshop, I don't really care, but still. So, why do I hate GeoIP advertising?


  • It reminds me how much crap people know about me

  • It reminds me that I should be using the Tor network

  • Instead of displaying contextual ads that I even might be a little inclined to click, it reminds me how much I hate advertising

  • Remember, I'm in a coffee shop? GeoIPized Girls = have to sit with back to wall.


Being curious, I decided to see if I could teleport the girls to other parts of the world. Enter Tor, Privoxy and FoxyProxy. I turned them all on, went back and remembered just how cool these tools really are.


Python, AST and SOAP

Published on Wednesday, March 7, 2007

For one of my projects I need to generate thumbnails for a page. And lots and lots and lots of them. Even though I can generate them via a python script and a very light "gtk browser", I would prefer to mitigate the server load. To do this I've decided to tap into the Alexa Thumbnail Service. They allow two methods: REST and SOAP. After several hours of testing things out, I've decided to toss in the towel and settle on REST. If you can spot the error with my SOAP setup, I owe you a beer.
I'm using the ZSI module for python.

1. wsdl2py


I pull in the needed classes by using wsdl2py.
wsdl2py -b http://ast.amazonaws.com/doc/2006-05-15/AlexaSiteThumbnail.wsdl


2. Look at the code generated.


See AlexaSiteThumbnail_types.py and AlexaSiteThumbnail_client.py.

3. Write python code to access AST over SOAP.




#!/usr/bin/env python
import sys
import datetime
import hmac
import sha
import base64
from AlexaSiteThumbnail_client import *

print 'Starting...'

AWS_ACCESS_KEY_ID = 'super-duper-access-key'
AWS_SECRET_ACCESS_KEY = 'super-secret-key'

print 'Generating signature...'

def generate_timestamp(dtime):
    return dtime.strftime("%Y-%m-%dT%H:%M:%SZ")

def generate_signature(operation, timestamp, secret_access_key):
    my_sha_hmac = hmac.new(secret_access_key, operation + timestamp, sha)
    my_b64_hmac_digest = base64.encodestring(my_sha_hmac.digest()).strip()
    return my_b64_hmac_digest

timestamp_datetime = datetime.datetime.utcnow()
timestamp_list = list(timestamp_datetime.timetuple())
timestamp_list[6] = 0
timestamp_tuple = tuple(timestamp_list)
timestamp_str = generate_timestamp(timestamp_datetime)

signature = generate_signature('Thumbnail', timestamp_str, AWS_SECRET_ACCESS_KEY)

print 'Initializing Locator...'

locator = AlexaSiteThumbnailLocator()
port = locator.getAlexaSiteThumbnailPort(tracefile=sys.stdout)

print 'Requesting thumbnails...'

request = ThumbnailRequestMsg()
request.Url = "alexa.com"
request.Signature = signature
request.Timestamp = timestamp_tuple
request.AWSAccessKeyId = AWS_ACCESS_KEY_ID
request.Request = [request.new_Request()]

resp = port.Thumbnail(request)




4. Run, and see error.


ZSI.EvaluateException: Got None for nillable(False), minOccurs(1) element 
(http://ast.amazonaws.com/doc/2006-05-15/,Url), 



 xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" 
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" 
xmlns:ZSI="http://www.zolera.com/schemas/ZSI/" 
xmlns:ns1="http://ast.amazonaws.com/doc/2006-05-15/" 
xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

[Element trace: /SOAP-ENV:Body/ns1:ThumbnailRequest]


5. Conclusion



I'm not entirely certain what I'm doing wrong. I've also written another version but actually with NPBinding connecting to the wsdl file. It seems to work much better, as it fully connects, and I get a 200, but it doesn't return the thumbnail location in the response, and I get a:
TypeError: Response is "text/plain", not "text/xml"


So, while I have things working fine with REST, I would like to get the SOAP calls working. One beer reward.

AWS in Python (REST)

Published on Saturday, March 3, 2007

As some of you may know, I have some projects cooked up. I don't expect to make a million bucks (wish me luck!), but a few extra bills in the pocket wouldn't hurt. Plus, I'm highly considering further education, which will set me back a few-thirty grand. That said, one of my projects will rely heavily on Amazon Web Services. Amazon has, for quite some time now, opened up their information via REST and SOAP. I've been trying (virtually the entire day) to get SOAP to work, but seem to get snagged on a few issues. Stay tuned.
However, in my quest to read every RTFM I stumbled upon a post regarding Python+REST to access Alexa Web Search. After staring at Python code, especially trying to grapple why SOAP isn't working, updating the outdated REST code was a 5 minute hack. So, if you are interested in using Alexa Web Search with Python via Rest, look below:


websearch.py



#!/usr/bin/python

"""
Test script to run a WebSearch query on AWS via the REST interface.  Written
 originally by Walter Korman (shaper@wgks.org), based on urlinfo.pl script from 
  AWIS-provided sample code, updated to the new API by  
Kelvin Nicholson (kelvin@kelvinism.com). Assumes Python 2.4 or greater.
"""

import base64
import datetime
import hmac
import sha
import sys
import urllib
import urllib2

AWS_ACCESS_KEY_ID = 'your-access-key'
AWS_SECRET_ACCESS_KEY = 'your-super-secret-key'

def get_websearch(searchterm):
    def generate_timestamp(dtime):
        return dtime.strftime("%Y-%m-%dT%H:%M:%SZ")
    
    def generate_signature(operation, timestamp, secret_access_key):
        my_sha_hmac = hmac.new(secret_access_key, operation + timestamp, sha)
        my_b64_hmac_digest = base64.encodestring(my_sha_hmac.digest()).strip()
        return my_b64_hmac_digest
    
    timestamp_datetime = datetime.datetime.utcnow()
    timestamp_list = list(timestamp_datetime.timetuple())
    timestamp_list[6] = 0
    timestamp_tuple = tuple(timestamp_list)
    timestamp = generate_timestamp(timestamp_datetime)
    
    signature = generate_signature('WebSearch', timestamp, AWS_SECRET_ACCESS_KEY)
    
    def generate_rest_url (access_key, secret_key, query):
        """Returns the AWS REST URL to run a web search query on the specified
        query string."""
    
        params = urllib.urlencode(
            { 'AWSAccessKeyId':access_key,
              'Timestamp':timestamp,
              'Signature':signature,
              'Action':'WebSearch',
              'ResponseGroup':'Results',
              'Query':searchterm, })
        return "http://websearch.amazonaws.com/?%s" % (params)
    
    # print "Querying '%s'..." % (query)
    url = generate_rest_url(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, searchterm)
    # print "url => %s" % (url)
    print urllib2.urlopen(url).read()



You run it like this:
>>> from websearch import get_websearch
>>> get_websearch('python')

Hamachi Rules

Published on Tuesday, February 27, 2007

I've been playing around more with Hamachi, and have decided that it officially rules. Since I'm a big Linux guy I don't have access to some features, but the program seems to be a gem. It is brainlessly easy to install (even when doing 20 things at once), and works quite well. Thanks to Ben and Sean for helping me test it out.

Ian the Apt

Published on Wednesday, February 14, 2007

You know you are too nerd like when your conversations are like this. Preface: I posted some packages I needed to upgrade into a Skype window (much better than a clipboard)....



[11:56:03] Kelvin Nicholson: sorry, needed to post that somewhere

[11:56:04] Ian FItzpatrick: i am not apt!

[11:56:15] ... you can't pass packages to me ;)

[11:56:34] Kelvin Nicholson: blah blah blah

[11:56:43] ... apt-get upgrade ian

[11:57:02] Ian FItzpatrick: apt-get error: unmet dependency, "beer 1.0-4 not found"

[11:57:14] Kelvin Nicholson: yea, that got a good laugh

Version 3.0 Part Two

Published on Monday, February 5, 2007

Well, I'm basically all done upgrading to Version 3.0, I deserve a cake or something. Here's the 411:

For the past few years I have been using Mambo, then Joomla, to manage the content on my site. It worked quite well, and was in PHP, so I could add or remove any code. Indeed, I've written a decent amount of PHP apps. In early 2004 I wrote a PHP platform to track adventures people had gone on, and networked people seeking to go on adventures with each other. I never marketed it, and mainly created it to learn PHP, but it was a CMS (Content Management System), and a little more. Late in 2004 I wrote another blog-esque platform for my second trip to Europe. It was pretty cool, I'll admit: Casey and I each had a blog, and people could leave us "dares" and/or messages -- and we could easily update our status. Overall, it worked great. You can also see the projects section of my site for some of the other things I've done in PHP.

Fast forward a few years, and here it is in early 2007. I've never really liked PHP all that much, but I couldn't put my thumb on it. Deciding to switch to something else, I picked up and read the book, Beginning Python, from Novice to Professional. If anybody is looking for a well written book, I would highly recommend this one. Anyways, with my goal to drop PHP in mind, I held the debate of Django and TurboGears. I went through the demos for each, and felt like I really played around with them. Ultimately it came down to 1) Django has obvious crazy cool caching, 2) Django has pretty darn good documentation, and a freaking online book, and 3) the "powered by" sites are quite impressive -- both the length of the list and the large amount of traffic some of these sites entertain.

So I went with Django. My friend in New Zealand, Ben Ford, has been ragging me for two months to get my ass in gear and learn it, saying I would love it. And he is right, the framework is simply beautiful. For the last week I've been reading through the documentation, going through the online book (both are incomplete, in my opinion, but compliment each other nicely). I think it is important to write your own code instead of just repeating examples, so my goal: transform my blog/site by using just Django.

So, while some of the kinks still need to be worked out, everything is no transfered over. I'll mention my experiences shortly, but overall: I'm very impressed.

Resize a VMWare Image of Windows XP

Published on Saturday, January 27, 2007

 

Over the years I have been shrinking the number of computers I own.  At one point my dorm was littered with old P100s, running whatever OS I felt like playing with at the time.  

VMWare comes to help.  In this recent oops, an image I made (for Windows XP), was slightly too small.  I didn't feel like reinstalling XP + cruft, so just resized the image.  This is the process:


1) Make Clone or just backup your VMWare image.

2) Note: if your disk is a Dynamic Disk, you won't be able use GParted.  There is a chance you can use Disk Management inside Computer Managemen inside XP.

3) Turn off VMWare image.

4) Grow the image.  



 vmware-vdiskmanager -x sizeGB yourimagename.vmdk 



5) Download the GParted LiveCD

5) Change the CD-ROM drive of your  VMWare image to boot from the ISO you just downloaded.

6) Boot VMWare image.  Make sure to press ESC right when it starts.

7) Follow the instructions for GParted. I had to select the Xvesa option, then Done.  Choose your language and keyboard and resolution.

8) GParted will come up.  First delete the partition (the empty one!), and make sure it says unallocated.  Then go up to Edit and hit Apply.  Then select the partition and say Resize.  Hit apply again.

9) Reboot image.  Windows XP will come up, and go through checking the disk.  It will reboot again, and you should then be able to log in.

Version 3.0

Published on Friday, January 26, 2007

To all my loyal (but most likely few, and mainly with the same last name as me):

I think I'm going to redo the structure of my site, yes, it is about time.  I've been using a CMS (Content Management System) to categorize everything, but it is time to ditch that and write something.  Time to stop being lazy.  Time to stop procrastinating.  Time to learn something new.

Let me make this clear: version 3.0 surely won't be based off PHP.

Darn You HiNet

Published on Wednesday, January 24, 2007

As you may know by now, I'm in Taiwan.  Live is pretty good here, especially the internet -- I'm sitting on a pretty decent 12M/1M connection.  But the company I have my internet through seems pretty laid back about network usage
-- which for me isn't good.  Their entire IP subnet appears banned from IRC, which means I have to be a little sneaky when I want to talk to my I.T. friends in New Zealand.  Worse, some websites even banned the entire subnet:

Your access to this site has been denied because of the large amount of abuse produced by the users from your country. The access will be enabled once we perform the investigation of the abuse issues. Thank you for your patience and understanding.

 

Luckily, Tor+Privoxy+FoxyProxy provides a quick way around any blocks that might be setup.  So, to the developers of these solutions, my kindest thanks.