Socket.IO seems to have some issues with a few of it's transports. The idea behind the transports is that the browser is supposed to attempt to each and failover to the next until it finds one that works. What I am observing is that there are a couple transports that actually cause the connection to fail, and no failover takes place. Unfortunately, one of these transports is WebSocket, which is most efficient transport in the list. So, here is the list on transports I use, with helpful notes, to make sure that everything will work on most major browsers:

transports: [
  //'websocket', //breaks chrome5, should be preferred :(
  'server-events', //i assume used in Opera, never seen it used
  //'flashsocket', //breaks android 2.1, chrome5
  'htmlfile', //preferred in IE8
  'xhr-multipart', //not supported on android 2.1, chrome5, preferred in FF3.6
  'xhr-polling' //preferred in chrome5, android 2.1, iPhone
]

Here's the setup that I did my tests against:

• Socket.IO version: 0.5.3
• OS: Ubuntu 10.04
• Browsers Tested:
  • IE8
  • Firefox 3.6
  • Chrome 5
  • Andoird 2.1
  • iPod Touch

I did some research last night about Socket.IO/Comet/cometD/Bayeaux/jetty/Kaazing. I learned some interesting things and found out that some of my assumptions were wrong.

• What I thought: Comet was a specific implementation of long polling and similar methods, comparable to Socket.IO on Node.js.
• Actual: Comet is an umbrella term for any type of server push technology. cometD is a collection of software that implements Comet, and uses the Bayeaux protocol to more formally define how things work.

• What I thought: Comet is the "old" way of doing things and WebSockets is the "new" way that is undeniably superior to Comet.
• Actual: WebSockets is so young that not only is it not well supported, but it's also not well defined and has major flaws. There is a good argument to be made that Comet (if well implemented, as is the case with jetty) is the more wise choice for now.

• What I thought: You have to choose Comet or WebSockets up front for your project.
• Actual: It appears that Comet 2 supports use of WebSockets as a transport mechanism, so you can switch transparently between WebSockets, long polling and jsonp long polling. This way, WebSockets can be used transparently to improve performance where it is supported. Socket.IO on Node.js also suppots use of WebSockets and other Comet technologies transparently.

• What I thought: If WebSockets is not supported and you have to fall back to long polling, you may have major performance problems.
• Actual: Jetty can handle up to 20,000 active client connections (10,000 without major tweaking). Granted, plenty of implementations will need more than that, but I don't think a pure WebSocket based solution would do much better. You're going to have to figure out a way to scale this to multiple servers anyway for true scalability. 10k clients per server seems pretty respectable. Furthermore, it sounds like cometD and jetty will allow use of WebSockets where possible, so if there is efficiency to be gained via WebSockets, you will still get it.

I feel that my biggest gap in understanding at the moment is Kaazing. I understand that it is more of a gateway that operates separately from the rest of your stack. I assume that it operates under the hood using typical Comet technologies, but which ones it chooses and how it balances them is something I don't know yet. And I also don't know how many connections it can handle and if there is built in support for clustering and what not. I have heard positive things about Kaazing so I think it might be a major contender that I still need to research.

Some more interesting tidbits

• Socket.IO supports WebSockets, Long Polling, Flash sockets and more, all transparently. It seems like it's lined up to compete pretty well with cometD/jetty. My only fear is that I don't really know how well the fallback mechanisms will perform, whereas with jetty I know that the Bayeaux protocol runinning on long polling performs amazingly well. Also, Socket.IO is generally much less mature than jetty...like a lot.
• cometD has non-browser clients available. You could create a desktop application that connects to the same service as your web application. What interests me more is to be able to use the Java client app to connect an Android app to your service. I'm not sure if this would work, but it seems like it would.

Links and notes I have gathered during my research so far

• http://dev.w3.org/html5/websockets/ - Formal definition of web sockets...so far...
• "Comet" is a common term for various methods of socket-like client-server communications. See http://en.wikipedia.org/wiki/Comet_(programming)
• CometD is a collection of implementations of "Comet" using the Bayeux Protocol (see http://cometd.org/documentation/cometd)
• Interesting article on Comet(Bayeaux) vs WebSockets -- http://www.infoq.com/news/2010/03/websockets-bayeux
• Jetty can scale to 20,000 client connections (http://blogs.webtide.com/gregw/2008/01/07/1199704334432.html)
• Socket.IO
  • http://en.wikipedia.org/wiki/Internet_socket
  • http://socket.io/
  • http://github.com/LearnBoost/Socket.IO-node
• Kaazing
  • http://www.kaazing.com/
  • http://www.websockets.org/
  • Not free, but there is a developer version limited to 25 concurrent connections
  • It is a gateway as opposed to being integrated with your app server
  • To see docs, download product, run gateway.start and navigate to http://localhost:8000/documentation/getting-started.html (I can't find docs online)
  • Chrome didn't work for me for the echo demo (Chrome v5, Kaazing v10.05)

node-apac is a client library for accessing the Amazon Product Advertising API, written in JavaScript for Node.js. Here is the set of commands to use to get going with node-apac and get the example script working. These commands are intended to be run on a fresh install of Ubuntu Server 10.04, so I will walk you through installing everything you need.

sudo apt-get update #helps to make sure next command is successful
sudo apt-get install git-core g++ libssl-dev curl
git clone http://github.com/ry/node.git
cd node
sudo ./configure
sudo make
sudo make install
curl http://npmjs.org/install.sh | sudo sh
sudo npm install apac@latest
wget http://github.com/dmcquay/node-apac/raw/master/examples/example-item-search.js
#now update awsId, awsSecret & assocId in example-item-search.js
node example-item-search.js

Need to compare Geo Location coordinates? Then you just might love GeoHash as much as I do. Many GeoLocation hashing algorithms facilitate scalable proximity searching. GeoHash is such an algorithm and is in the public domain. There are also many implementations of Geohash available. I am using davetroy's JavaScript implementation of GeoHash. The README of davetroy's project explains the algorithm quite well:

The Geohash algorithm was first described by Gustavo Niemeyer in February 2008. By interleaving latitude and longitude information in a bitwise fashion, a composite value is generated that provides a high resolution geographic point, and is well suited for storage or transmission as a character string.

Geohash also has the property that as the number of digits decreases (from the right), accuracy degrades. This property can be used to do bounding box searches, as points near to one another will share similar Geohash prefixes.

If you need help visualizing this, checkout the geohash demonstrator.

davetroy's JavaScript implementation of GeoHash is dead simple. There are only two public methods you need to know, as follows.

string encodeGeoHash(latitude, longitude)
object decodeGeoHash(geohash)

Using GeoHash for Proximity Searching

Let's look at how we might use GeoHash for simple, scalable proximity searching. Suppose we have the following locations and our current location as follows:

var locations = [
  { latitude: 47.8559705, longitude: -181.9415665 },
  { latitude: 47.8558705, longitude: -181.9416665 },
  { latitude: 47.8557705, longitude: -181.9417665 },
  { latitude: 47.8556705, longitude: -181.9418665 },
  { latitude: 47.8555705, longitude: -181.9419665 }
];
 
var currentLocation = { latitude: 47.8557715, longitude: -181.9417665 };

Now, let's efficiently find the two closest locations to our current location. First, I would start by computing the GeoHash for each location.

currentLocation.geoHash = encodeGeoHash(currentLocation.latitude, currentLocation.longitude);
for (var i = 0; i < locations.length; i++) {
  locations[i].geoHash = encodeGeoHash(locations[i].latitude, locations[i].longitude);
}

Now, we can find the nearest locations with the following function.

function getNearest(currentLocation, locations, maxNeighbors) {
  var matching = {},
      accuracy = 12,
      matchCount = 0;
  while (matchCount < maxNeighbors &amp;&amp; accuracy > 0) {
    var cmpHash = currentLocation.geoHash.substring(0,accuracy);
    for (var i = 0; i < locations.length; i++) {
      if (locations[i].geoHash in matching) continue; //don't re-check ones that have already matched
      if (locations[i].geoHash.substring(0,accuracy) === cmpHash) {
        matching[locations[i].geoHash] = locations[i];
        matchCount++;
        if (matchCount === maxNeighbors) break;
      }
    }
    accuracy--;
  }
  var tmp = [];
  for (var geoHash in matching) {
    tmp.push(matching[geoHash]);
  }
  return tmp;
}

This function takes three parameters. The first two are pretty self explanatory. The last, maxNeighbors, is how many matches we want to find. We start by checking the full geoHash for matches. We then truncate the geoHashes by one more character each iteration of the while loop, thereby looking for more broad matches each time. When we have enough matches, we break from the loop. We store results in a map with geoHash as the key so that we can keep track of which locations have already matched so that we can avoid duplicates. We convert this to an array before returning the results.

This code solved my needs. What is absolutely wonderful about this is not only that it is pretty simple, but that it is scalable. If you're into Hadoop, this logic would by very easy to use with MapReduce. However you want to set up your architecture, this algorithm sets you up well for scalability.

So, now that we've got our logic all worked out, let's find the nearest two locations.

var nearest = getNearest(currentLocation, locations, 2);

And the contents of nearest will be as follows:

[
  { geoHash: "b0800pbh040n", latitude: 47.8557705, longitude: -181.9417665 },
  { geoHash: "b0800pbh8hb5", latitude: 47.8558705, longitude: -181.9416665 }
]

One final note. If you are trying to use this in node.js (like me), you can install this using npm (npm install geohash@latest). Have fun!

Limitations

It should be noted that there are limitations to using geohash for proximity searches. From wikipedia:

One limitation of the Geohash algorithm is in attempting to utilize it to find points in proximity to each other based on a common prefix. Edge case locations close to each other but on opposite sides of the Equator or a meridian can result in Geohash codes with no common prefix[1].

Secondly a Geohash essentially defines a bounding box within which a location lies, therefore two locations may be spatially very close but have different geohashes. In order to be useful to proximity searches, the surrounding eight geohashes of a geohash must be calculated and the locations matching these pulled out, therefore complicating potential usage in proximity searches.

In the code above, I am not checking the eight surrounding geohashes for proximity searching, but instead relying only on size of the matching prefix. This approach is less accurate, but much more simple.

See Also

• Wikipedia article on Geohash
• Geographiclib -- A LGPL'ed C++ library for performing tons of geographic calculations. Looks pretty thorough, but I haven't looked at it yet.
• Natural Area Code -- Another way to hash geo positions
• Military Grid Reference System
• MSP GEOTRANS -- An application program which allows you to easily convert geographic coordinates among a wide variety of coordinate systems, map projections, and datums

I'm excited about web sockets and you should be too. The idea behind web sockets is to allow a client to maintain a connection with a server, the main benefit of which is allowing the server to send data to the client instead of the client having to poll the server for data. The result is real time event driven client/server communications with low overhead.

Long polling has long been the best option to accomplish this behavior. Check out chat.nodejs.org for an example of a chat application that uses long polling. What I'd like to point out here is that it was already possible for a server to send data to the client on-demand. So, what makes web sockets so fantastic? It seems to me that the main game change here is scalability. Long polling is essentially a hack to emulate a persistent socket connection. There's a lot more overhead involved in long polling. Web Sockets allows this to be done the natural way -- by actually keeping a socket open.

But wait! Web Sockets are only supported by a few browsers! That's right. You really can't make a website utilizing Web Sockets just yet because it's only supported by the latest versions of a few browsers. That's where web socket wrapper libraries come in handy. These libraries attempt to use Web Sockets and if they're not available, they are emulated using other methods such as Flash sockets or even long polling if necessary. I've been looking into a few of these lately and I will write about what I learn.

So far I have been focused on web sockets for Node.js because that is my latest pet technology. There appears to be two leading options. The first is Socket.IO and the second is node-websocket-server. From what I have been told and what I have experienced, Socket.IO is more simple and easy to use compared to node-websocket-server which is a bit more complicated, but also more scalable for your effort. They both appear to be very young which makes me a bit nervous. There are libraries like Kaazing which are much more complete and proven. This is expected since Node.js is very young itself.

This is all very fun and I'll post more as I learn more.

My wife and I recently got new smart phones. We are joined at the hip with AT&T so our phone choices were dictated by that. My wife ended up with the iPhone 4 and I chose the HTC Aria, which runs Android (version 2.1 at the moment). Since we have been excited about and learning our phones together, I feel like I've got a pretty good, unbiased view of why you might prefer one over the other.

Why you will love iPhone 4

• Fast & Responsive - Android devices are catching up, but no Android I have tried is quite as fast and responsive as an iPhone
• Excellent Hardware (screen, camera, etc) - The camera on the iPhone 4 takes much better pictures than the HTC Aria. However, there are plenty of other Androids out there with amazing hardware.
• FaceTime - I have not found any software comparable to iPhone 4's FaceTime for the Android, nor does my Aria have a camera on the front to make it possible. This is lame that it is only available through Apple though. You should be able to accomplish this with Skype. I have to think that in the not too distant future there will be Android devices available with a camera on the front and with Skype to use this camera. However, AT&T seems to like blocking out cool apps, so even if that does become available, AT&T might block me from using it unless I root my device.
• Easy to learn and use
• No backups necessary - On my Aria, I'm starting to get quite a few pictures, videos, purchased apps and purchased music. I need to backup all of this stuff. I also need to copy files the other way such as new music from my PC. I need some kind of two-way sync application and I haven't found a decent one just yet. With iPhone 4 (or any of Apple's products) you don't have to worry about this. iTunes takes care of it all for you. I have to admit that is pretty nice.
• Apps in App Store are higher quality on average - Sure the Android App Marketplace is catching up in terms of number of apps, but a lot of them are crap quality. I even found a "Hello World" app. Seriously? It seems to me that it is easier to get an app into the Android Marketplace than into Apple's. This has an upside in that Google doesn't try to control everything. But the downside is that there really is a lot more junk to sift through.

Why you will love HTC Aria (& other Android devices)

• Open - Not tied to Apple or Google or anybody - I hate the fact that I own a ton of music that can only be played on iTunes. Amazon has a nice app for this now.
• More flexible system components (e.g. keyboard) - Pretty much everything in the system can be replaced, such as the keyboard. Don't like the keyboard? No problem. Try another. Don't like the keyboard on your iPhone 4? Too bad, you're stuck with it.
• Stock keyboard has mic option - That's right, you can just click the little mic button and say what you're thinking instead of typing it. I have found it to be pretty darn accurate too.
• Smaller - This applies specifically to my Aria. Since I carry my phone around in my pocket a lot, this is pretty nice.
• Less fragile, case is not a must - The iPhone 4 is glass and very breakable. The Aria is lighter and has a mostly plastic exterior and seems less likely to break if you drop it. I'll still use a case with mine, but it's not nearly as crucial as with the iPhone 4.
• Develop apps on any platform - If you want to develop apps for the iPhone, you need a Mac.
• Easily replace or upgrade parts - Want more memory? Just buy a new Micro SD card (I just bought a 16 GB one from Amazon for $27) and put it in. Want more memory for your iPhone 4? Well, first of all and iPhone with more memory will cost you an arm an a leg. Second, it is not upgradeable. You're stuck with your choice.

My last post was about a library I wrote for node.js for connecting to Amazon's Product Advertising API (called node-apac). Once I had written the barebones version, I wanted to share it to others that might be looking for the same thing and so my search began to find out the best way to share open source software for node.js. I was looking for something akin to Perl's CPAN.

Kiwi

The first thing I came across was kiwi. Kiwi works almost identically to CPAN, which is what I was looking for. In the end, Kiwi turned out to not be the best option, so I'm not going to describe everything in detail. But here's the idea of how it works. There is a web-accessible repository of libraries that is accessible via a command line interface. Using the command line interface, you can both submit packages to the repository and install packages from the repository.

Node Package Manager (npm)

So, what's wrong with kiwi? Not that much really, except that it's not the "popular" one and, with package managers, that is crucial. I asked around in the #Node.js IRC channel and was informed that Node Package Manager (npm) is the latest and greatest and has the most momentum and community support. At least for the foreseeable future, npm is what you want to use. Conceptually there's not very much different from Kiwi. There is still an online repository that you access via a command line utility. There is one thing I found significant however. With Kiwi, you have to use kiwi in your code to require libraries. For example, to use the express.js web development framework, after installing via kiwi, you would do this in your code:

var kiwi = require('kiwi'),
    express = kiwi.require('express');

You can see here that you become dependent on kiwi by using it. npm is much better at staying out of the way. npm installs packages so that they can be found with a normal require() call. So, you can switch from using npm to installing packages by some other means without having to change your code.

Brief Tutorial on npm

What I'd like to do at this point is to give you a jump start with npm. After finishing this article you should be able to use npm to publish a library you have written or use npm to install a package you need. However, I should state first that node.js and npm are not well supported on platforms other than linux. I'm pretty sure support is planned, but for the time being, you might just find yourself up against a lot of frustration.

Getting npm

To get started, we have to work out some dependencies first. Basically all you need is node.js. You also need v8, but installing node.js will take care of that for you. So start by going to the node.js download page and follow the installation instructions there. Next, go to npm's github page and, once again, follow the installation instructions there.

Adding code to the repository

Adding a library of your own is very easy, I promise. All you have to do is a minor code change (a handful of lines), create a file called package.json (around 20 lines for a simple project), create an npm account and publish your code to npm. Let's take a closer look at each step using node-apac as an example.

In node-apac, there are two "classes", each in a separate file, that I'd like to make available to anyone that uses the package. However, using npm, as far as I know, you can only specify one file as the main file for exporting things. So, I had to make a third file that imports the other two and exports everything I want all at once. Here's what it looks like:

exports.OperationHelper =
    require('./OperationHelper').OperationHelper;
exports.RequestSignatureHelper =
    require('./RequestSignatureHelper').RequestSignatureHelper;

That's pretty simple, right? Then, when using the library, if someone wants to use the OperationalHelper, they would do something like this:

var opHlpr = require('apac').OperationalHelper;

Again, pretty simple stuff here. Now let's look at the package.json file. This is the file that explains all about your package. In this file you can give it a name, description, tell it where to find that "main" JS file where you export everything and a bunch of other fancy stuff like dependencies. Let's take a brief look at node-apac's package.json:

{ "name" : "apac"
, "description" : "Amazon Product Advertising API Client for Node"
, "version" : "0.0.1"
, "author" : "Dustin McQuay "
, "repository" :
  { "type" : "git"
  , "url" : "git://github.com/dmcquay/node-apac.git"
  }
, "bugs" : { "web" : "http://github.com/dmcquay/node-apac/issues" }
, "os" : [ "linux", "darwin", "freebsd" ]
, "directories" : { "lib" : "./lib/" }
, "main" : "./lib/apac"
, "engines" : { "node" : ">=0.1.97" }
, "dependencies" : { "libxmljs" : "*" },
, "licenses" :
  [ { "type" : "MIT"
    , "url" : "http://github.com/dmcquay/node-apac/raw/master/LICENSE"
    }   
  ]
}

This is a fairly minimal package.json file. You can read more about the format of this file at http://github.com/isaacs/npm/blob/master/doc/json.md. I'm going to skip over the obvious stuff. First let's look at version. This is an important field to npm. It is used for dependency resolution and must follow a specific format. You should read more at semver.org, but the minimal to understand for now is that it must be three integers separated by periods. The next interesting field is "main". This is where to find the main JS file that we wrote earlier. I have placed mine in a directory called lib as you can see. The name of the file is apac.js, but you leave off the ".js" here for some reason. This library is written for node, so I include that as in engine and it depends on libxmljs so I include that in the dependencies. The "*" means that any version of the library will do.

Everything is pretty well set up now so we're ready to create an npm account and publish our code. Creating an npm account is done on the command line using the "npm adduser" command. With npm, commands are issued in the "npm [subcommand]" format, and in this case, adduser is the subcommand. To learn more about the adduser subcommand, just type "npm help adduser" or similar for any other subcommand. To create our user, it will look something like this:

npm adduser <username> <password> <email>

Ya, this is a little bit lame. Your password is going to end up in your history file and another plain text config file and I'm guessing it is sent over the network in plain text each time you authenticate. So don't use your online banking password for this.

We're almost done! Now all you have to do is issue the "npm publish" command. You need to be in the root of your project, where the package.json file should also be located. Then just type "npm publish". Yay, you did it!

If you got lost anywhere, you might find it useful to take a look at the source for node-apac for a complete, working example.

Installing a Package

Now that we've added node-apac to the registry, we can now install it using the npm install command:

npm install node-apac

That doesn't work though! Ya, I know. It will for most packages, but if you make a new package like this that is a pre 1.0.0 release, it won't know which version of the package to install. The reason is that it is going to look for the latest version tagged with @stable, but there is none since we have not tagged any versions as stable yet. Furthermore, I don't think you can tag a pre 1.0.0 release as stable. All we have to do to get around this is to specify that we want to install the version tagged as @latest, which is always the most recent version. Let's try again.

npm install node-apac@latest

I'm writing a website using express.js which will need to connect to the Amazon Product Advertising API. This API is REST based with XML as the format of choice, so one would think that it is fairly simple to use. As I dug deeper it wasn't quite that simple. Amazon requires that the requests are signed in a very special way. It's not exactly rocket science to sign the request, but it is definitely not something you want to have to deal with every time you make a request. It it is a totally mundane task that is the same for every application, so I figured that there must be a library out there to do this for you.

After looking around a bit, I did not find any "official" client libraries. I did find a number of code examples that you could use to do the heavy lifting, but none that would work in JavaScript. When most of the world thinks of JavaScript, they think of client-side JavaScript. Creating the request signature requires access to your secret key with AWS which you must not share with anyone. Generating a request signature in client-side JavaScript would expose this secret key to anyone who is sophisticated enough to view source -- not good. So I can see why there wasn't much interest in creating a JavaScript libarary for this purpose.

However, my intentions were quite different. My JavaScript code will run server-side using Node.js. Therefore it is totally safe and I need said library. So, I went about creating a library called node-apac and have made it publicly available. The easiest way to get it is by using npm (Node Package Manager). If you already have npm installed, then all you have to do is type "npm install apac" (the library is called "apac" in the npm registry as opposed to node-apac per npm package name recommendations). Keep in mind that this library depends on node.js. If you try to use it in a browser you won't have any luck.

In this library, there are two classes. The main one you'll need is OperationHelper. You instantiate it only once, no matter how many API calls you make. You'll pass it a few things such as your AWS id and secret key for initialization. Then, for each API call you want to make, you'll use the execute() function, passing it operation-specific parameters. The second class provided is RequestSignatureHelper which is responsible for signing your requests. This is the class that's probably providing the most value and you can use it directly if you'd like. But, I recommend just using OperationHelper, which uses RequestSignatureHelper behind the scenes.

That's it. If you happen to be developing a node.js app that needs to access the Amazon Product Advertising API, I'm sure you'll enjoy. If you have any problems, please let me know by filing a issue.

There's about a million different ways to embed flash. It's not as simple as just embedding it like you would an image. You need to deal with browser compatibility problems, the fact that active content won't run automatically on IE without some help, how to degrade gracefully on devices that don't support flash (e.g. iPad), and more.

I found an article on A List Apart called Flash Embedding Cage Match by Bobby Van Der Sluis which explains the whole situation very well. If you want to really understand what's going on, read that article. Don't worry, it's a quick read. In the article, he doesn't really claim which option is best, but in my opinion, SWFObject either wins or is at least a tie. Note that the article is a tad old and SWFObject 2 has since been released.

So, if you just need to embed some flash quickly, just check out swfobject on Google Code. This library offers two means of embedding flash (static and dynamic), depending on your preferences. I suggest using the dynamic method because it offers superior degradation and is more simple. The downside is that your flash content will only show if flash and JavaScript are both supported. That is acceptable in most circumstances.

Here's how it works:

1. Create some static (non-flash) content to show in case flash is not available
2. Include swfobject.js
3. Call swfobject.embedSWF() with a few simple parameters

Here's some actual code:

 
 
 
 
 
 
<div id="my_flash_content">
    <p>Flash is not available, here's a cool image instead...</p>
    <img src="waycoolimage.jpg" />
</div>
 
 
 

Generally you can just do this manually (it's really easy). However, if you want to get a little deeper into what options are available for configuring flash, swfobject_examples.zip contains an application (in Adobe AIR and HTML formats) that guides you through all of the various configuration options. Read more about this on the SWFObject 2 beginner's guide.

Have you ever been asked a dumb question? Ever asked a dumb question? I asked just such a question at work a while back to Brendan Gibson (now at Netflix), who was working with me on the ODAT sites at Backcountry.com (e.g. Steep and Cheap). The reply was something like this: "Here's a good website that explains the answer: http://tinyurl.com/cd3zfp". Go ahead and follow that link. It directs you to a site called Let Me Google That For You (LMGTFY.com). When the site loads, it looks like the Google search page and it types the question in for you and then takes you to the real Google search results. Genious. So the next time you get an annoying question, don't just give them an answer. Give them a good laugh too. :)