Hacking at heat-maps and Google Maps API

I've been hacking away at a map visualization that I'm surprised hasn't appeared elsewhere already - a heatmap that interpolates between sample points. 

Every other heat map I've come across approximates interpolation by rendering circles with fading alpha at sample points and relies on the density of points to achieve a smooth result.

Below is an early screenshot of a canvas-based heatmap overlay I'm writing that is built from the Delaunay triangulation of the underlying data points (that's quite a mouthful!). 

I'm using a scanline-based Gouraud shading algorithm to render the triangles to the canvas myself having tried and failed to get WebGL, SVG and CSS to do the job for me (none of these can draw triangles with three different coloured vertices with any serious speed).

Prices for 2BR units across Sydney's inner west.

If there is interest, I will clean up and release the code once I have it working reasonably well and post it somewhere. For now, I'm rethinking my choice of Gouraud shading because with under-sampled data like I have here, it gives some very ugly, sharp lines.

I'm currently using it to display property values but hopefully others can think up some more good uses for it. 

GNU SDR and EZCap

Another nerdy post. I seem to be getting worse! :P

My ezcap USB dongle (RTL1832U/E4000) arrived from dealextreme yesterday and it took a whole hour of tweaking settings and wikipedia browsing to go from fuzzy crackle to working (and I think I understand the basics of it) software defined radio FM decoding. I started with someone elses grc file (although I can't remember who's) and tweaked it until I got to this setup.

In the top FFT you can see the station I'm tuned to in the middle and two stations on either side (they seem to be spaced every 800khz apart in Sydney). The bottom shows the filtered signal for the station I'm tuned to.

An amateur radio expert would probably laugh at me as I arrived at these settings through nothing but a bit of a rough direction and a lot of sheer luck but it seems to sound pretty good! Now I need to think of a legitimate use for this. :)

An experiment in Van Emde Boas layouts

Given an arbitrarily large sorted, balanced binary tree of values, the obvious way to search for elements is binary search. In undergraduate classes I learned that a binary search requires log(n) operations and is thus as efficient as you can get without throwing additional storage space at the problem. One thing my lecturer didn't dive into though was the effect of CPU cache on this otherwise simple little log(n) algorithm.

How to do dirt-cheap, cloud-based, encrypted backups (Part 2)

In my previous post I describing the method I used to store data online, I referred to what I was doing as a "backup" but it would have probably been more accurate to call it a near-real-time off-site mirror. In this post, I describe the pitfalls of my previous system and describe my much-improved latest technique.

How to do dirt-cheap, cloud-based, encrypted backups (Part 1)

I've been dabbling on code with the final aim of building a peer-to-peer distributed filesystem for a while now and I seem to keep hitting diversions along the way. In an internal monologue somewhat along the lines of "What would MacGyver do?", the idea for this concoction of open-source software and cloud services was born. In about 20 minutes or so of messing about (assuming you're comfortable with linux), I'll explain how you too can sleep sounder at night for $0.125/GB/month.

NSW Property price heatmap

I've been considering buying a property of late but without a strong grasp of the greater Sydney area's geography I found I couldn't really judge whether a price was appropriate or not for a given suburb. Seeing how much as I love data, I threw together Property Hot Spots . Its backed by couchdb and aside from the initial tile generation, is essentially static. It could do with a design cleanup and perhaps a canvas-based heatmap that isn't so ugly but it served its purpose so I figured I'd post it here in case others find it useful. The newest data displayed here is from Jan 2012. The oldest a few years back. I might add date of sale somehow in a future change if I can find an uncluttered way to display it. :)

Property Hot Spots

How to read Japanese books above you Kanji level

After moving back to Australia, my kanji has gone seriously downhill. Some friends from Japan recently brought me over the 3 books of Haruki Murakami's 1Q84 for me to study with and I quickly found I was utterly useless on my own without a dictionary.

Its a chicken-and-egg problem that I presume many other scholars of Japanese also have. Books with simple kanji are often targeted at younger audiences and can be painfully boring for adults. Books targeted at adults assume a kanji level much higher than I currently have. The end result is you get bored by either the content you're reading or the constant turning to your dictionary.

So my new Japanese study pipeline involves:

1. Scan a chapter at 300 DPI greyscale.
2. Run OCR over it.
3. Correct any errors. (I am seeing about a 95-98% accuracy rate so this is quick.)
4. Save the text as UTF-8 HTML.
5. Use Rikai-kun as necessary to get super-quick dictionary lookups when you need them.

This process is not as involved as it sounds. Its probably about 30-60 seconds per page on average amortized across all the conversion tasks. I would have spent MUCH longer than this fumbling with my dictionary if I'd tried to read it the traditional way. 

Also as a bonus, I am now building up a personal digital copy I can carry with me much more easily than the 3 hard cover books on the shelf!

So you want to get your ustream video off your iPhone?

UStream.tv is great for streaming events to distant relatives and such. My wife and I used their iPhone app to great success for our wedding last year. Unfortunately, for videos that we were able to record but not stream due to limited WiFi access, we have had no luck getting the incredibly slow uploads to actually complete. After a year and numerous attempts, I thought I'd find my own way to get these video's off the device without resorting to jailbreaking, etc.

Ustream sends a HTTP request to ask for the location of an FTP server to upload your videos to. We will pretend to be that server.

1. On an ubuntu box (call it 192.168.1.10):

  $ sudo apt-get install ftpd wireshark

2. Find or setup a linux router between you and the internet and run this on it:


  $ iptables -t nat -I PREROUTING -p tcp --dport 21 -j DNAT --to-destination 192.168.1.10
  $ iptables -t nat -I PREROUTING -p tcp -d 74.217.100.0/24 -j REDIRECT --to-port 21

The first line sends all FTP traffic to your local FTP server instead of wherever it was originally headed.
The second line redirects all ports sent to the ustream FTP server IP subset (at the time of writing at least. ping red37.ustream.tv) to port 21.
Together they make sure all the traffic ustream tries to send to itself gets sent to your ubuntu box instead.

3. Now you either a. mess with your FTP server to allow all usernames / passwords to work (edit the source code, mess with authentication modules, etc.) or b. do as I did and run tcpdump on your linux router:

  $ sudo tcpdump -i eth0 "src iphone_ip or dst iphone_ip" -s 2000  -w iphone.pcap

3a. Go to your iphone and try to upload a video. It will get stuck at Uploading 0%. Cancel the upload. Ctrl-C tcpdump and open the file on your ubuntu workstation with wireshark.

  $ wireshark iphone.pcap

3b. The username and password ustream uses will be in a binary blob in the response to a request sent to http://rgw.ustream.tv/gateway.php that you should have in your iphone.pcap log file.

3c. Run off and create the users as required:

  $ sudo useradd -m 1_12345_12345
  $ sudo passwd 1_12345_12345
  New password: .....

  New password (again): .....

4. Go to your phone, hit "Upload".

5. Profit! (Or simply savour your new-found ability to watch your precious videos wherever you like!)