How to do dirt-cheap, cloud-based, encrypted backups (Part 1)

I've been dabbling on code with the final aim of building a peer-to-peer distributed filesystem for a while now and I seem to keep hitting diversions along the way. In an internal monologue somewhat along the lines of "What would MacGyver do?", the idea for this concoction of open-source software and cloud services was born. In about 20 minutes or so of messing about (assuming you're comfortable with linux), I'll explain how you too can sleep sounder at night for $0.125/GB/month.

NSW Property price heatmap

I've been considering buying a property of late but without a strong grasp of the greater Sydney area's geography I found I couldn't really judge whether a price was appropriate or not for a given suburb. Seeing how much as I love data, I threw together Property Hot Spots . Its backed by couchdb and aside from the initial tile generation, is essentially static. It could do with a design cleanup and perhaps a canvas-based heatmap that isn't so ugly but it served its purpose so I figured I'd post it here in case others find it useful. The newest data displayed here is from Jan 2012. The oldest a few years back. I might add date of sale somehow in a future change if I can find an uncluttered way to display it. :)

Property Hot Spots

How to read Japanese books above you Kanji level

After moving back to Australia, my kanji has gone seriously downhill. Some friends from Japan recently brought me over the 3 books of Haruki Murakami's 1Q84 for me to study with and I quickly found I was utterly useless on my own without a dictionary.

Its a chicken-and-egg problem that I presume many other scholars of Japanese also have. Books with simple kanji are often targeted at younger audiences and can be painfully boring for adults. Books targeted at adults assume a kanji level much higher than I currently have. The end result is you get bored by either the content you're reading or the constant turning to your dictionary.

So my new Japanese study pipeline involves:

1. Scan a chapter at 300 DPI greyscale.
2. Run OCR over it.
3. Correct any errors. (I am seeing about a 95-98% accuracy rate so this is quick.)
4. Save the text as UTF-8 HTML.
5. Use Rikai-kun as necessary to get super-quick dictionary lookups when you need them.

This process is not as involved as it sounds. Its probably about 30-60 seconds per page on average amortized across all the conversion tasks. I would have spent MUCH longer than this fumbling with my dictionary if I'd tried to read it the traditional way. 

Also as a bonus, I am now building up a personal digital copy I can carry with me much more easily than the 3 hard cover books on the shelf!

So you want to get your ustream video off your iPhone?

UStream.tv is great for streaming events to distant relatives and such. My wife and I used their iPhone app to great success for our wedding last year. Unfortunately, for videos that we were able to record but not stream due to limited WiFi access, we have had no luck getting the incredibly slow uploads to actually complete. After a year and numerous attempts, I thought I'd find my own way to get these video's off the device without resorting to jailbreaking, etc.

Ustream sends a HTTP request to ask for the location of an FTP server to upload your videos to. We will pretend to be that server.

1. On an ubuntu box (call it 192.168.1.10):

  $ sudo apt-get install ftpd wireshark

2. Find or setup a linux router between you and the internet and run this on it:


  $ iptables -t nat -I PREROUTING -p tcp --dport 21 -j DNAT --to-destination 192.168.1.10
  $ iptables -t nat -I PREROUTING -p tcp -d 74.217.100.0/24 -j REDIRECT --to-port 21

The first line sends all FTP traffic to your local FTP server instead of wherever it was originally headed.
The second line redirects all ports sent to the ustream FTP server IP subset (at the time of writing at least. ping red37.ustream.tv) to port 21.
Together they make sure all the traffic ustream tries to send to itself gets sent to your ubuntu box instead.

3. Now you either a. mess with your FTP server to allow all usernames / passwords to work (edit the source code, mess with authentication modules, etc.) or b. do as I did and run tcpdump on your linux router:

  $ sudo tcpdump -i eth0 "src iphone_ip or dst iphone_ip" -s 2000  -w iphone.pcap

3a. Go to your iphone and try to upload a video. It will get stuck at Uploading 0%. Cancel the upload. Ctrl-C tcpdump and open the file on your ubuntu workstation with wireshark.

  $ wireshark iphone.pcap

3b. The username and password ustream uses will be in a binary blob in the response to a request sent to http://rgw.ustream.tv/gateway.php that you should have in your iphone.pcap log file.

3c. Run off and create the users as required:

  $ sudo useradd -m 1_12345_12345
  $ sudo passwd 1_12345_12345
  New password: .....

  New password (again): .....

4. Go to your phone, hit "Upload".

5. Profit! (Or simply savour your new-found ability to watch your precious videos wherever you like!)

epoll_threadpool

I set out a few months back to build a distributed filesystem with features similar to dropbox but with 100% encrypted storage that is shared amongst friends in a "dark-net" of sorts. In the process, I found myself wanting a fast, light-weight RPC system and, in turn, a fast, light-weight event queuing system. After fighting with libevent and msgpack-rpc, I eventually decided to write my own and epoll_threadpool was born.

The library is still in its infancy but I don't expect it to grow much (if at all) in size. Its Linux only (epoll-based), dependencies are light, speed should be reasonable and all tests are passing so I thought I'd throw it out to the world to see if anyone finds it useful.

Features:

• epoll-based
• runs a thread pool, executing events on the first available thread.
• easy-to-use IOBuffer class for streaming data.
• TCP client and server support.

Available on github as epoll_threadpool.

Qantas Rant

Qantas rant! I also object to Mr Joyce's pay hike but lets put things in real numbers with respect to union demands. Pilots already get paid (in my opinion) WAY more than they should thanks to strong unionism and after a 17% hike - way more than inflation - a second hike seems plain greedy.

A senior pilot's wage is around $500k according to this and presumably a senior co-pilot would be around $350k. According to this most pilots work 215hrs a month (75 hours of that flying). In hourly rates, that's $193/hr for pilots and $135 for co-pilots.

If a plane flies 80% of the time, there are 86.4 of the 108 qantas grounded planes in the air at any given time with a pilot and copilot each. That's an operational cost of $28339.20 an hour (24/7) or $248 million a year. A 2.5% pay increase for pilots will this cost the airline $6.2 million in flight time alone. That doesn't cover ground-based preparation and other duties.

There are no parties here that are NOT being greedy but the union is trying to tarnish the brand in order to push around management. Strike action is their biggest hammer and they've been swinging it around with way too little regard for the people they hit with it for way too long. If pilots are not happy with their conditions they should walk with their feet (out the door). If their salaries are average as they claim for senior pilots, they shouldn't have trouble finding work elsewhere.

As for the CEO's salary, keep separate issues separate. That's a topic for another rant.

Can OpenWRT save my Netgear DGN2000?

I made the mistake of buying a Netgear DGN2000 ADSL2+ modem when I first got ADSL. Not only does it not support IPv6, its WiFi range is pathetic, bridging between WiFi and ethernet seems to die after several days of operation, and under heavy load the device hard-crashes, requiring a reboot. I suspect the crashing might be due to poor thermal design and perhaps my specific device but clearly the software is not also without some blame. Given my experiences, I would NEVER recommend this device. But, now that I have one that I can't return, I'm going to document the process of installing OpenWRT to see if I can give this thing a second lease on life.