On my Mac (Lion) I was getting an error when uploading images: "Upload a valid image. The file you uploaded was either not an image or a corrupted image."

I guess django uses PIL for verification of a valid image. The problem as it turns out was that PIL was installed without JPEG support and I was trying to upload a JPEG image.

The fix is very simple. You have to install libjpeg and then you have to reinstall PIL.

brew install jpeg
pip install pil --upgrade

Add this to your ~/.gitconfig to add these two super useful aliases. A guy at work shared these with me. The provide a nice way to view your history on the command line in a much more pretty format. Mainly they add some color and add visualization of branching and merging.

[alias]
        hist = log --pretty=format:'%Cred%h%Creset %C(bold blue)<%an>%Creset%C(yellow)%d%Creset %Cgreen(%cr)%Creset%n%w(80,8,8)%s' --graph
        histfull = log --pretty=format:'%Cred%h%Creset %C(bold blue)<%an>%Creset%C(yellow)%d%Creset %Cgreen(%cr)%Creset%n%w(80,8,8)%s%n' --graph --name-status

At my work we are now using Oracle 11g database. The production version of this thing is quite hefty so I wanted something lighter while developing on my laptop. Oracle provides a lightweight edition called Oracle XE. Unfortunately I had to install 10g (see compatibility notes below). Using 10g instead when the rest of the company is on 11g has only caused me one pain point so far and I was able to work around it. I'll explain more later. So, without further ado, here's what I learned when trying to get this sucker installed.

Compatability notes

I mentioned above that I had to use 10g since I'm running 32 bit. Here's some more specific notes about what version you have to use for various platforms.

• If you have 64 bit Ubuntu/Debian, then you have to follow the alien instructions below.
• If you have 64 bit RedHat, Fedora or CenOS, just skip the alien stuff and install the RPM directly.
• If you have a 32 bit linux OS (like me), install Oracle 10g XE instead. There are packages for lots of distributions in that version so nothing special is required.

Prerequisites

You need a JDK. For Ubuntu, I just installed openjdk-6.

 
sudo apt-get install openjdk-6-jdk
You also need ant installed
sudo apt-get install ant
 

Converting 11g 64 bit RPM into a .deb using Alien

Download the RPM from Oracle's website. The instructions below assume you have a copy of it. Note: I got a few weird errors while installing this, but it seems that they were benign.

 
sudo apt-get install alien libaio1 chkconfig
sudo alien -d --scripts oracle-xe-11.2.0-0.5.x86_64.rpm
 

Install the package

If you are installing an RPM, I haven't actually done that. I assume it would be easy though. If you're installing a deb (either converted by alien or a deb directly from Oracle in the case of 10g), then here's some more details.

 
sudo dpkg -i oracle-xe-11.2.0-0.5.x86_64.rpm
sudo /etc/init.d/oracle-xe configure
#if you don’t want oracle to start automatically on boot, run this:
sudo /sbin/chkconfig oracle-xe off
 

Here are the answers I gave to questions during installation.

 
Port for Oracle Application Express: 8085 (so it doesn’t conflict with Tomcat, JBoss, etc.)
Port for DB Listener: 1521 (default)
Password for SYS and SYSTEM: Oracle11 (because this is standard)
 

You should now add yourself to the dba group. When I was doing this I noticed that oracle wasn’t in this group either. Added that user too, but probably unnecessary. This will allow you to start/stop your XE database among other things.

Now you need to setup some env variables for your user and for your oracle user:

 
export ORACLE_SID=XE
alias vi='vim'
export ORACLE_BASE=/u01/app/oracle #(or just $HOME for oracle user)
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/xe
export ORACLE_TERM=xterm
export _EDITOR=vim
export NLS_LANG=american_america.utf8
export TNS_ADMIN=$ORACLE_HOME/network/admin
export ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$PATH
 

I found that the paths are a bit different if you’re installing 10g. Here’s what worked for me:

 
export ORACLE_SID=XE
alias vi='vim'
export ORACLE_BASE=/usr/lib/oracle/xe/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/server
export ORACLE_TERM=xterm
export _EDITOR=vim
export NLS_LANG=american_america.utf8
export TNS_ADMIN=$ORACLE_HOME/network/admin
export ORA_NLS33=$ORACLE_HOME/nls/data
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$PATH
 

Troubleshooting

In $TNS_ADMIN/tnsnames.ora, your computer’s hostname is used in the default XE config. But for me, my hostname was not resolvable. Either add it to /etc/hosts or get a DNS entry for your host...or just change this to localhost, which is what I did.

Eventually you’re going to start getting an error like this due to the fact that XE limits the number of concurrent connections too much:

org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (java.sql.SQLException: Listener refused the connection with the following error:
ORA-12519, TNS:no appropriate service handler found
The Connection descriptor used by the client was:
localhost:1521:XE
)

So let’s up this limit with the following commands:

 
sqlplus system/Oracle10
alter system set processes=150 scope=spfile;
quit;
sudo /etc/init.d/oracle-xe restart
 

I'm in the middle of Clean Code (Robert C. Martin) and I've been convinced that comments are generally bad. When I first started reading about the concept it seemed crazy. I've been taught for all my nerd-life that all code should be well documented. How could that suddenly be completely wrong?

As it turns out, both are right. Code should be well documented and comments are bad. You should document your code by use of excellently formatted code and descriptive function and variable names. Classes and functions should be relatively short. Functions should do one thing and classes should have only a single responsibility.

Okay, so having well formed and descriptive code is good, but why does that make comments bad? Because comments are not directly associated with logic. There is no guarantee that a comment is accurate. You have to hope that every programmer that made changes to the code has updated the comments accordingly. What's worse, comments give you a license to write bad code. If a block of code needs a hefty description, then it is probably convoluted, has poorly named variables, has too much logic in a single function, or something else along those lines. If you feel that a block of code really needs a comment to explain how it works, you should spend your time improving the code so that it is more clean and easy to read instead.

A genius can write software that is so complex that very few people can understand it. Is that admirable. That's debatable. Is it good for business? Absolutely not. Your goal should be to make it look like it was easy to write, because in order to do that, it must be easy to read and if it is easy to read, then it will be easy to maintain.

Clean Code is a fantastic book and I recommend you give it a read.

More and more I am loving the concept of deploying code as a package with dependencies, of which the Java WAR and EAR formats are prime examples. In working with perl, python/django and nodejs, I have often relied on globally installed libraries to satisfy dependencies. This has a few major drawbacks.

• If you are deploying multiple applications to the same server, you might be stuck with using the same version of a library to satisfy a common dependency.
• Versions of dependencies are not inherently tied to versions of your application.
• Preparing a server to run your application requires installation of many libraries, and is therefore more complex and takes more time.

I have implied two requirements to solve these problems. The first is to deploy dependencies with your application. The second is that the versions of your application must be associated with the versions of its dependencies, perhaps by including the libraries in your versioned code repository, or by including a record of the versions and having a build process that resolves the dependencies and includes them in the deployable package.

I am not necessarily suggesting that Java WARs and EARs are the only formats to do this well. Rather, I am learning from their example that deploying in such a manner is desirable. It seems to me that you should be able to achieve the same results with nodejs or python or * by simply following that deployment pattern.

Add a comment

Just for fun, I thought I'd implement a binary search of a dictionary recently. Really the goal was to implement a Boggle solver and the dictionary was a part of that. This dictionary needs to be able to do only two things:

1. Load a dictionary from file.
2. Given a string, report if it is a word or not and also if it is a prefix of another word. For example, glut is a word and it is also a prefix of gluten.

To solve a Boggle board, you need to search the Dictionary many times (at least you do the way I chose to solve it). So your search needs to be fast. A very simple and fast algorithm is a binary search. In Big O notation, binary search is O(log(2,n)). That is, as the size of the search set doubles, the number of computations increases by only one. This is very efficient. Let's look at the numbers a bit.

As you can see, we can search a dictionary of over a trillion words with only 40 comparisons. Why can we do this so efficiently? In a binary search we divide the search set in half each step by comparing our word against the middle word in the search set. For this to work this dictionary must be sorted. There is also one other subtle, but very important requirement here. The words in the dictionary must be randomly accessible. If we were tempted to use a Linked List to store our words (in order to improve insert efficiency), we would end up and something more like O(n) which is much worse.

package com.synchrosinteractive.boggle;
 
import java.util.ArrayList;
 
public class Dictionary {
  public static final byte WORD = 1 << 0;
  public static final byte PREFIX = 1 << 1;
  public static final byte NONE = 1 << 2;
 
  public static byte binarySearch(ArrayList<String> words, String str) {
    int min = 0;
    int max = words.size() - 1;
    int mid = 0;
    int cmpRes = 0;
    while (true) {
      mid = (min + max) / 2;
      cmpRes = str.compareTo(words.get(mid));
      if (cmpRes == 0) {
        byte result = WORD;
        if (words.size() - 1 > mid && words.get(mid + 1).startsWith(str)) {
          result |= PREFIX;
        }
        return result;
      } else if (min == max) {
        if (words.size() - 1 > mid && words.get(mid + 1).startsWith(str)) {
          return PREFIX;
        } else {
          return NONE;
        }
      } else if (cmpRes > 0) {
        min = mid + 1 > max ? mid : mid + 1;
      } else {
        max = mid - 1 < min ? mid : mid - 1;
      }
    }
  }
}

Let's take a look at this. The first thing we do is to define our min and max indexes, which to begin will include all words in the dictionary. Next we will begin a loop that will choose the mid index between min and max and compare the string to that middle word. If the search string is equal to that word, we are done. If it is greater than the middle word, then we know that the target word must be in the latter half of the search set (if it exists in the search set).

You may notice one deviation the code above makes from the typical binary search. When it finds the word, it is not quite done. In our case, a search string can be both a word and a prefix of one ore more other words. Therefore, once we have found that it is a word, we still need to verify if it is a prefix of another word. Due to the order nature of the dictionary we are searching, if the word is also a prefix of another word, it will be the very next word. So all we have to do is a quick check of the next word using the startsWith method of the String class.

Similarly, in the case where this is the last word (where min == max) and we have not yet found a match, we cannot just stop. We have to check if the search string is a prefix of the next word once again.

One thing to watch for in an algorithm like this is unnecessary use of recursion. We could pass the search set into the outer function over and over again recursively and this would not affect our Big O efficiency. However, it would eat up unnecessary memory. Sometimes recursion is the right way to go, but when there's a relatively simple non-recursive alternative, then avoid recursion. In this case, it was very natural to avoid recursion.

Lastly, you may notice that results are being bassed back using a bitmap. This allows me to pass back a lightweight, primitive value that can contain multiple results or flags. However, this means that to test the results you must use a bitmask. The static variables in this class make this fairly simple. All you have to do is perform a bitwise and against the result and the bitmask in question and check that the result is greater than zero. For example:

assert (result & Dictionary.WORD) > 0)

This will check that the search found the search string to be a word.

Using this algorithm against a dictionary of 213,560 words (medium size) on my Core 2 Duo laptop, I was able to perform about 3.6 million searches per second.

Work has been crazy busy lately and I needed a mental break. What's the best way to get a break from programming? Do more programming of course! So I decided to make Boggle in Java. I didn't make the UI. I just made the domain models to build random boggle boards and to find all possible solutions for a given board.

I'm pretty happy with the speed of the result. On my humble laptop, it can create and solve 1000 boards in a bit under one and a half seconds. If you're bored too, you can see the boggle source code. I used a binary search algorithm for dictionary searches and a recursive "dead-end" algorithm for the board traversal and solving (okay, I just made that up because I don't know what else to call it).

Time to run a single unit test in Grails that doesn't actually do anything except assertTrue Boolean.TRUE: 7.6 seconds

Time to run all unit and integration tests for a small project I'm working on in Node.js: 0.8 seconds

Time to run unit tests in pretty much anything, including Java: a lot faster than Grails

Enough said.

This evening I tried to connect to my MongoDB instance using the command line mongo tool and got the following error message: Error: couldn't connect to server 127.0.0.1} (anon):1137. It turns out that the server wasn't running due to an improper shutdown at some point in the past. Here's what you need to do to fix it.

Note: my instructions are specific to Ubuntu 10.04 (& probably other versions of Ubuntu).

Why Won't MongoDB Start?

One of the lame things about MongoDB is that it doesn't handle a crash on it's own very well. If the database crashes or otherwise doesn't shut down properly, then the next time you try to start it, you'll get a nice error saying that there is an old lockfile and that you probably need to remove it and run mongod --repair. However, on my laptop running Ubuntu 10.04, I ran into a few confusing things that I'd like to share, hopefully to save someone else out there some time.

Symptoms

• Try to connect (type mongo on the command line). You should get a connect failed error.
• Verify that MongoDB is not running. Run sudo status mongodb. It should report mongodb stop/waiting.
• Verify that you are unable to start MongoDB. Run sudo start mongodb. It will report mongodb start/running, process XXXX no matter what. But if you run sudo status mongodb again, you'll get stop/waiting.
• Check your logs to see that you in fact have the same problem as me. Your logs should be at /var/log/mongodb/mongodb.log. You should see an error like this:

  ************** 
  old lock file: /var/lib/mongodb/mongod.lock.  probably means unclean shutdown
  recommend removing file and running --repair
  see: http://dochub.mongodb.org/core/repair for more information
  *************

Instructions To Get MongoDB Up Again

• Manually remove the lockfile: sudo rm /var/lib/mongodb/mongod.lock
• Run the repair script: sudo -u mongodb mongod -f /etc/mongodb.conf --repair.
  • You must run this command as the mongodb user. If you run it as root, then root will own files in /var/lib/mongodb/ that are necessary to run the mongodb daemon and therefore when the daemon trys to run later as the mongodb user, it won't have permissions to start. In that case you'll get this error: Unable to create / open lock file for lockfilepath: /var/lib/mongodb/mongod.lock errno:13 Permission denied, terminating.
  • On Ubuntu, you must specify the configuration file /etc/mongodb.conf using the -f flag. Otherwise it will look for the data files in the wrong place and you will see the following error: dbpath (/data/db/) does not exist, terminating.
• Now you can start your MongoDB server with sudo start mongodb and verify it is running with sudo status mongodb and by trying to connect to it with mongo test.

Today I taught an Intro to Perl class for about 15 people. Their programming experience ranged from none to very little, perhaps with a few exceptions. The class was only 1/2 hour long. My goal was for each participant to leave the class with the following:

• Understanding of what Perl is and what it is good for
• Have Perl installed and able to make/run a "Hello, World" program
• Understand what CPAN is and be able to install and use a package, at least by copying and pasting the SYNOPSIS section

This presentation was intended to be accompanied by my explanations, but you may still find it useful. Enjoy!