Tuesday, 11 July 2017

Ethereum Mining

All the kids are doing it. I've been doing it too for a while with 3 cards. And it's a total waste of money. Don't do it! Don't bother! Stick your money in something else, BUY Ethereum if you must!

The problem is that we're all fighting over the same piece of pie. The more miners, the smaller the share. And before you shout "You just want it all for yourself!" and pull out the pitchforks, let me explain why it's too late, and why I'll shortly be stopping.

Rewind back to late May 2017 when the price shot up to $200. I assume a larger contributing force behind this were investors in Japan and elsewhere in Asia realizing that bitcoin was technically less capable than it's newer cousin, Ethereum. Several Asian exchanges started supporting it. Japan introduced a law legitimizing crypto-currencies and the rest was greed feeding greed.

Of course, I joined the bandwagon. I got in around here and shortly after the price shot up to $400. At the peak, my little 3 card setup was making me almost $20 USD a day. It's around this point that I imagine everyone doing this pats themselves on the back, tells themselves how clever they are and does an entirely unrealistic projection of how much money they are going to make.

Since then, the value has dropped back to ~$220 USD but the miners haven't stopped. They've been piling on like crazy. News of GPU shortages have only piqued interest and fueled the fire further. What these punters are forgetting to factor in is not just the price correction, but the difficulty increase. I thought I was clever. I thought I had factored it in but my projections were off. It turns out the world is far sillier and greedier than I expected.

Ethereum, like all proof-of-work crypto-currencies, targets a certain "block rate". This is effectively the rate at which blocks of the transaction ledger get signed off on. To avoid any old person declaring their ledger is the ledger, miners spend a lot of time working on solving a complex computing problem. The one that solves it gets their name attached to the ledger and 5 ETH for their trouble. The network is designed so that this happened with a median time of 15 seconds.

To simply that even further: Every hour, roughly 1200 ETH at $225 each ($270kUSD), is up for grabs for miners.

It's easy to see why a gold rush was inevitable. But....

See that little red box on the difficulty evolution. Thats a bump of over 100 trillion in difficulty. At 33MH/s hashing rate (a single GTX1070 graphics card), that's equivalent to 30 million cards jumping on the mining bandwagon... overnight! Yes, there is some statistical error in there, but in any case it's a big increase!

Current returns are only a few dollars a day and electricity needs to be factored into that. Paying back your $500 graphics cards is going to take you an eternity if current trends continue.


What happened, I suspect, is that some early big investors bought ETH and then realized the markup they were paying and have decided to mine instead. Meanwhile, ICOs rode the hype, sucking money out of the system by driving up demand and then cashing out. There have also been exchange hacking and network capacity issues that have scared people off.

I have seen photos of shops that churn out hundreds of rigs with thousands of cards at inflated prices. These are making the news and gullible first timers are jumping in without understanding the ecosystem. The clever money at the moment is in selling the shovel to the gold miner I assume...

I imagine there are a lot of people out there that have invested in this with cards on back-order so there is a delay in adding capacity to the network but already we're dangerously cost to returns dipping below the cost of electricity.

For me, I was going to burn the electricity on heating anyway so I'll continue through the winter and then sell it all off. It was an interesting flash in the pan, but still just a flash.

Wednesday, 27 April 2016

DNS-over-HTTPS

I noticed by chance in early April that Google Public DNS (the 8.8.8.8/8.8.4.4 people) now offer DNS-over-HTTPS. I thought this would a nice little addition for privacy but it seems so new that nothing out there supports it! A few weekends later, I now have a little proxy daemon and for the past week I've been running it on my OpenWRT router without issue! It's not perfect but I've uploaded the code here if anyone is interested.

Cleanups, some security auditing and test coverage work needed but I feel it's working well enough to release it to others at this stage. Hope it's helpful to someone else!

Friday, 4 December 2015

Old Mac Mini runs old OpenSSH with broken cipher set

I couldn't find much about this on the interwebs but I happen to know an OpenSSH developer so I went straight to the experts here.

Basically, if you run OpenSSH on a mac mini, you shouldn't trust Apple to give you updates and you're implementation may well be full of holes. The latest OpenSSH version is 7.1 and my 2009 mac mini with all the updates was reporting
$ ssh -V
OpenSSH_5.5p1, OpenSSL 0.9.8n 24 Mar 2010
Wow... That's not good.

If you MUST run with this version and you want to be security conscious (and NSA paranoid) so you've restricted your allowed cipher list on your client machines, note that aes128-gcm is advertised by this broken Apple build but not actually supported by the binary. This will look like an immediate disconnection after connecting. You have this problem if your system.log contains:

$ tail -n 100 /var/log/system.log| grep fatal
Dec 4 11:16:09 macmini.lan sshd[27028]: fatal: matching cipher is not supported: aes128-gcm@openssh.com [preauth]

The quick fix (you should probably upgrade OpenSSH anyway, somehow..) is to add a cipher line to /etc/sshd_config as follows:
Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour

Hope that saves someone else some confusion. :D

Wednesday, 18 February 2015

Playing with the ESP8266

We live in amazing times. For $2.50 each I got some ESP8266 on AliExpress and flashed nodemcu on them with minimal fuss. I've now got kilobytes of flash spare on a WiFi device with 10 digital IO, 1 analog input and a standby current draw of 78 micro amps!

Thanks to the author of this for the instructions. Basically just put the FTDI adapter in 3.3v mode and get breadboarding with the magic pinouts described on the esptool github page and flashed the latest firmware binary blob. Reboot and I get a Lua prompt via minicom. Worked flawlessly.

Next up, DS18B20 temperature sensor, an I2C humidity sensor and some soldering. :)


Saturday, 10 January 2015

Reverse-engineering Efergy's internet-connected smart power meter


I just got myself one of these Efergy Home Hub Online things. I'm super impressed with the physical build quality. Sadly though, nothing's perfect. With this unit two fundamental things bothered me, both which are addressable to some degree.

1. Problematic DHCP?


I plugged it in and the network port came alive. Data LEDs blinked but the solid red light indicating the device was booting stayed on forever. I changed the Ethernet cable, port, lots of reboots, turned my 1Gbit ports down to 100Mbit... Nothing. It turned out whatever DHCP client they've implemented doesn't seem to work with a Fritz Box 7390. I've never seen this issue before. I returned the first unit thinking it was bad. When the second unit did the same thing I tried shoving the Ethernet cable into the Ethernet port of my desktop and fired up udhcp. To my surprise, that did the trick. Now I need to figure out some way to get this on my network without forwarding traffic through my desktop that isn't always on.. Urgh!

2. The all-your-data-are-belong-to-us cloudy thing.


Everything these days seems to want to provide an app. That alone isn't a problem, but in order to serve data to the app, your data generally lives in the cloud and this is where I reach for my tin-foil hat and begin the fun little process of reverse engineering the protocol this thing uses. :D

I don't think there is anything sinister going on here. I just want control of my data and would like my device to continue to work if (or rather when) the manufacturer decides to stop hosting it. The web interface this thing provides is pretty slick and there is both an iPhone and Android app but when it comes to my personal data, I'd rather hold it myself, thanks! So on that note...

The solution: Reverse Engineering!


The hardware I presume was actually produced by Efergy but it seems that the software for the "hub" device was provided by a company called Hildebrand based out of London and it's Hildebrand that actually receive the raw meter readings and store them on their servers. The Efergy website hosts the interface but the raw data goes to a sensornet.info domain that is registered to Hildebrand.

Boot Sequence


I have a box stamped with HK 1.1 Firmware AU on the bottom of it. I can't guarantee other regions behave the same way but here goes.
  1. DHCP request is broadcast to network. Device waits for a repsonse.
  2. DHCP response received. Device starts resolving using DHCP-provided DNS server the hostnames "uk.pool.ntp.org" and "ff.ee.dd.aabbcc.h2.sensornet.info" where "aabbccddeeff" is the MAC address of the device.
  3. Device requests from the resovled sensornet host IP the URL https://<ip>/get_key.html. Response is HTTP/200 of the form "TT|ALPHANUMERIC"
  4. Device requests from the same host IP the URL "https://<ip>/check_key.html?p=TT&ts=
    0000019D&h=<some hash>. HTTP/200 with an empty response is returned.
  5. Device starts posting periodically to https://<ip>/h2 with "Content-Type: application/eh-data" and content of the form: "123456|1|EFCT|P1,0.00.".
  6. Device periodically also sends requests to https://>ip>/h2 with "Content-Type: application/eh-ping" and an empty body, presumably just to let the service know it's alive.
Some other nice tidbits:
  • SSL is used but certificates are never checked so MITM is very easy to do to read the data.
  • The "ts" GET argument in check_key.html is not a timestamp. I've seen it go backwards and always seems close to zero. I suspect it's irrelevant as we can return "success" on any data. We don't really care about authentication here.
  • The data format seems to be "<Sensor ID> | 1 | <sensor type> | <INPUT>,<Reading>."
  • I suspect unit of measurement to be in milliamps but haven't yet confirmed this. The user will have to take voltage and power factor into account to work out kW and kW/h.

Update: A fake cloudy thing.


I had a Raspberry Pi that I wasn't doing much with so I turned it into my data logger by running my own DHCP, DNS and HTTPS servers, each pointing the device to the rPi instead of the hildebrand servers. I have a USB to Ethernet dongle to talk to the hub and the rPi ethernet adapter to talk to my LAN. Win! Source code is on github here.


Saturday, 20 September 2014

Graphing HDD health with smartctl

I proudly built myself a front door for my TV cabinet recently - the very same TV cabinet that houses my NAS. Two weeks later and two crashes of my NAS box (that coincidentally uses my drives for swap), I discover 2 of my 4 HDD's had started giving errors, one had completely died. Turns out this thing called "ventilation" is important after all! *shrugs*

Saturday, 15 June 2013

rPi + tvheadend + shepherd

My home TV setup involves a file server (FreeNAS), a Mac Mini running XBMC and a Raspberry Pi running TVHeadend.

In setting up the rPi side of this, I came across a lot of scattered instructions so I thought I'd bring them together. Nothing is particularly difficult but I suspect I'll be doing this again at some stage in the future so for the sake of myself and others...

Sunday, 19 May 2013

The state of 3D printing

The world of 3D printing has intrigued me for years. An industrial designer friend of mine has been tinkering with 3D printed prototypes and objects for a few years now and I've been surveying the state of things for a while. I particularly liked the look of theForm1 on kickstarter although the limited choice of material is a potential issue. Anyway, for whatever reason, after a few hours of trawling shapeways, I feel compelled to rehash my favourites:

8. I have trouble believing the claim that this actually works. The sheer number of components here and the tolerances they must have is incredible.
#5

7. An elegant chopstick holder that would look at home at the most sophisticated dinner table.
With hashi 1

6. A lens cap holder that attaches to your camera strap. Ingeniously simple and useful and it looks so professional it's hard to tell from the picture what the object actually is.
Description

5. A Galaxy S3 case + credit card holder + money clip + bottle opener. Seriously, I wish I had a Galaxy S3 right now.


4. A blast from my childhood past! Evil tentacle!


3. Serious jewellery. Sure, you have to take it to a professional jeweller to get it properly made but the fact the designs like this can be based on a 3D printed base is still very impressive.


2. You can actually get stuff printed in stainless steel now. Awesome!
Size example


1. This mug looks incredible. Printed in ceremic, the detail is amazing. The price is the only reason I've held back and I assume these will drop significantly with time.



Sunday, 24 March 2013

You CAN still get a pre-paid data SIM card in Japan

I read a lot of blogs stating you couldn't get a pre-paid SIM card in Japan. It's true that the big players (NTT Docomo, Softbank, AU) don't sell these anymore but turns out that there is still at least one way to do this.

A company called b-Mobile sells pre-paid SIM cards valid for 1 month (about $USD32) and 3 month (about $USD100) with 1GB of data each. Yes, I wonder if anyone goes with the 3 month option...

The cards don't support calls but they run on NTT Docomo's LTE, HSDPA, 3G networks and data coverage seems very good.

You can only get them from big retailers. In Osaka, that means BIC Camera in Namba or, in my case, Yodobashi Camera in Umeda. You also need to activate them in Japanese - with a Japanese mobile apparently. So it definitely helps if you speak Japanese, have a Japanese friend, or if you're super convincing, maybe you can persuade the sales person to do it for you. From what I could tell, the U300 SIM they used to offer in English, pre-activated 24 hours after mail order purchase is no longer on offer. :(

In my case, I've been running on a 1GB, 1 month MicroSIM for a week now and using SkypeIn for receiving incoming calls. Calls are very high latency (almost unusable) but it has served its purpose so far - if I get a call that isn't working well, I just fall back to contact via email.

Best of luck Japanese travellers!

Friday, 8 March 2013

The worlds ugliest webm streaming webserver?

I was trying to find an easy way to get low latency video from a webcam to a remote browser today. Requirements:

  1. Quick deployment
  2. Runs on Raspberry Pi
  3. Runs with out-of-the-box debs (see quick deployment)
My hacky solution was a Django python app that uses the StreamingHttpResponse class, gstreamer and a pipe. Disguisting, but works well. Sadly, latency is about 10 seconds to localhost so its not exactly live... 

import pygst
pygst.require("0.10")
import gst
def grab(request):
  """ Return a webm live stream from the first attached webcam. """
  class VideoStreamer():
    def __init__(self):
      self.pipein, self.pipeout = os.pipe()
      self.player = gst.parse_launch ("v4l2src ! video/x-raw-yuv,width=640,height=480,framerate=10/1 ! ffmpegcolorspace ! vp8enc max-latency=1 lag-in-frames=1 ! webmmux name='m' streamable=true ! fdsink fd=%d" % self.pipeout)
      self.player.set_state(gst.STATE_PLAYING)
    def start(self):
      fd = os.fdopen(self.pipein)
      try:
        while True:
          yield fd.read(4096)
      except Exception, e:
        print "Exception was ", e
    def __del__(self):
      self.player.set_state(gst.STATE_NULL)
      os.close(self.pipeout)
  return StreamingHttpResponse(VideoStreamer().start(), content_type="video/webm")

Requires pygst and django 1.5 (use pip).

Monday, 4 March 2013

Bricked Netgear Stora? Arduino to the rescue!

My latest attempt at getting offsite backups working for me in the most convenient manner possible involves scattering storage devices around at places I frequently visit such as relatives places, etc. I thought I'd re-purpose a Netgear Stora device I had lying about (P.S. Don't ever buy one of these if you value your privacy) by modding the firmware on it. Turns out, an arduino makes a great TTL serial adapter if you short the reset pin to GND. :) 

I'm also particularly proud of my ghetto MacGyver-like pin connectors. They were created with PVC tape rolled around the leg of a resistor and then cut into thirds and slipped over each pin to hold them in place. (Yes, I need to get myself some more electronics gear...)

Pin outs here in case anyone stumbles across this wanting to do something similar. :)

 

Saturday, 12 January 2013

DataMule - My first Android app. :)

I've been toying with various backup solutions for months now. I've tried tahoe-lafs. I tried my own hacky scripts and before that I even had the audacious idea of writing my own P2P backup software from scratch, starting at the low-level RPC layer.. My current approach is to use duplicity and throw the files on Google's cloud storage. The biggest problem with this latest approach is that full backups of my 100GB of data take about 12.5 days of maxing out my ADSL uplink. QoS eases some of the frustrations trying to use the internet while this is happening but doesn't completely remove them.

So DataMule is going to be a simple Android app that includes an SFTP/SSH client and configuration for pairs of WIFI SSIDs.

When you come into range of your "source" SSID, the SD card partition on your phone gets filled up with data copied via SFTP from your source server (until full). When you come into range of your "destination" SSID, the process is reversed and data is copied to the destination machine via SFTP. To complete the data migration, rsync is kicked off on the destination host with flag to delete after copying. The rsync will connect directly back to the source and copy (from the source) to the new file. This will just kick off a file hash verification and either delete the file (if its OK) or fix it if its not (with hopefully minimal bandwidth).

A HDD at work, a HDD at my parents place and an Android app and I'm all set. So far I've got a skeleton app working but its very very rough.

More updates to come!

Sunday, 16 September 2012

Hacking at heat-maps and Google Maps API

I've been hacking away at a map visualization that I'm surprised hasn't appeared elsewhere already - a heatmap that interpolates between sample points. 

Every other heat map I've come across approximates interpolation by rendering circles with fading alpha at sample points and relies on the density of points to achieve a smooth result.

Below is an early screenshot of a canvas-based heatmap overlay I'm writing that is built from the Delaunay triangulation of the underlying data points (that's quite a mouthful!). 

I'm using a scanline-based Gouraud shading algorithm to render the triangles to the canvas myself having tried and failed to get WebGL, SVG and CSS to do the job for me (none of these can draw triangles with three different coloured vertices with any serious speed).

Prices for 2BR units across Sydney's inner west.
If there is interest, I will clean up and release the code once I have it working reasonably well and post it somewhere. For now, I'm rethinking my choice of Gouraud shading because with under-sampled data like I have here, it gives some very ugly, sharp lines.

I'm currently using it to display property values but hopefully others can think up some more good uses for it. 

Friday, 25 May 2012

GNU SDR and EZCap

Another nerdy post. I seem to be getting worse! :P

My ezcap USB dongle (RTL1832U/E4000) arrived from dealextreme yesterday and it took a whole hour of tweaking settings and wikipedia browsing to go from fuzzy crackle to working (and I think I understand the basics of it) software defined radio FM decoding. I started with someone elses grc file (although I can't remember who's) and tweaked it until I got to this setup.



In the top FFT you can see the station I'm tuned to in the middle and two stations on either side (they seem to be spaced every 800khz apart in Sydney). The bottom shows the filtered signal for the station I'm tuned to.

An amateur radio expert would probably laugh at me as I arrived at these settings through nothing but a bit of a rough direction and a lot of sheer luck but it seems to sound pretty good! Now I need to think of a legitimate use for this. :)

Sunday, 29 April 2012

An experiment in Van Emde Boas layouts

Given an arbitrarily large sorted, balanced binary tree of values, the obvious way to search for elements is binary search. In undergraduate classes I learned that a binary search requires log(n) operations and is thus as efficient as you can get without throwing additional storage space at the problem. One thing my lecturer didn't dive into though was the effect of CPU cache on this otherwise simple little log(n) algorithm.

Sunday, 25 March 2012

How to do dirt-cheap, cloud-based, encrypted backups (Part 2)

In my previous post I describing the method I used to store data online, I referred to what I was doing as a "backup" but it would have probably been more accurate to call it a near-real-time off-site mirror. In this post, I describe the pitfalls of my previous system and describe my much-improved latest technique.

Tuesday, 20 March 2012

How to do dirt-cheap, cloud-based, encrypted backups (Part 1)

I've been dabbling on code with the final aim of building a peer-to-peer distributed filesystem for a while now and I seem to keep hitting diversions along the way. In an internal monologue somewhat along the lines of "What would MacGyver do?", the idea for this concoction of open-source software and cloud services was born. In about 20 minutes or so of messing about (assuming you're comfortable with linux), I'll explain how you too can sleep sounder at night for $0.125/GB/month.

Wednesday, 15 February 2012

NSW Property price heatmap

I've been considering buying a property of late but without a strong grasp of the greater Sydney area's geography I found I couldn't really judge whether a price was appropriate or not for a given suburb. Seeing how much as I love data, I threw together Property Hot Spots . Its backed by couchdb and aside from the initial tile generation, is essentially static. It could do with a design cleanup and perhaps a canvas-based heatmap that isn't so ugly but it served its purpose so I figured I'd post it here in case others find it useful. The newest data displayed here is from Jan 2012. The oldest a few years back. I might add date of sale somehow in a future change if I can find an uncluttered way to display it. :)

Friday, 10 February 2012

How to read Japanese books above you Kanji level

After moving back to Australia, my kanji has gone seriously downhill. Some friends from Japan recently brought me over the 3 books of Haruki Murakami's 1Q84 for me to study with and I quickly found I was utterly useless on my own without a dictionary.

Its a chicken-and-egg problem that I presume many other scholars of Japanese also have. Books with simple kanji are often targeted at younger audiences and can be painfully boring for adults. Books targeted at adults assume a kanji level much higher than I currently have. The end result is you get bored by either the content you're reading or the constant turning to your dictionary.

So my new Japanese study pipeline involves:

  1. Scan a chapter at 300 DPI greyscale.
  2. Run OCR over it.
  3. Correct any errors. (I am seeing about a 95-98% accuracy rate so this is quick.)
  4. Save the text as UTF-8 HTML.
  5. Use Rikai-kun as necessary to get super-quick dictionary lookups when you need them.
This process is not as involved as it sounds. Its probably about 30-60 seconds per page on average amortized across all the conversion tasks. I would have spent MUCH longer than this fumbling with my dictionary if I'd tried to read it the traditional way. 

Also as a bonus, I am now building up a personal digital copy I can carry with me much more easily than the 3 hard cover books on the shelf!

Saturday, 7 January 2012

So you want to get your ustream video off your iPhone?

UStream.tv is great for streaming events to distant relatives and such. My wife and I used their iPhone app to great success for our wedding last year. Unfortunately, for videos that we were able to record but not stream due to limited WiFi access, we have had no luck getting the incredibly slow uploads to actually complete. After a year and numerous attempts, I thought I'd find my own way to get these video's off the device without resorting to jailbreaking, etc.

Ustream sends a HTTP request to ask for the location of an FTP server to upload your videos to. We will pretend to be that server.

1. On an ubuntu box (call it 192.168.1.10):

  $ sudo apt-get install ftpd wireshark

2. Find or setup a linux router between you and the internet and run this on it:


  $ iptables -t nat -I PREROUTING -p tcp --dport 21 -j DNAT --to-destination 192.168.1.10
  $ iptables -t nat -I PREROUTING -p tcp -d 74.217.100.0/24 -j REDIRECT --to-port 21

The first line sends all FTP traffic to your local FTP server instead of wherever it was originally headed.
The second line redirects all ports sent to the ustream FTP server IP subset (at the time of writing at least. ping red37.ustream.tv) to port 21.
Together they make sure all the traffic ustream tries to send to itself gets sent to your ubuntu box instead.

3. Now you either a. mess with your FTP server to allow all usernames / passwords to work (edit the source code, mess with authentication modules, etc.) or b. do as I did and run tcpdump on your linux router:

  $ sudo tcpdump -i eth0 "src iphone_ip or dst iphone_ip" -s 2000  -w iphone.pcap

3a. Go to your iphone and try to upload a video. It will get stuck at Uploading 0%. Cancel the upload. Ctrl-C tcpdump and open the file on your ubuntu workstation with wireshark.

  $ wireshark iphone.pcap

3b. The username and password ustream uses will be in a binary blob in the response to a request sent to http://rgw.ustream.tv/gateway.php that you should have in your iphone.pcap log file.

3c. Run off and create the users as required:

  $ sudo useradd -m 1_12345_12345
  $ sudo passwd 1_12345_12345
  New password: .....

  New password (again): .....


4. Go to your phone, hit "Upload".

5. Profit! (Or simply savour your new-found ability to watch your precious videos wherever you like!)