Reverse-engineering Efergy's internet-connected smart power meter

I just got myself one of these Efergy Home Hub Online things. I'm super impressed with the physical build quality. Sadly though, nothing's perfect. With this unit two fundamental things bothered me, both which are addressable to some degree.

1. Problematic DHCP?

I plugged it in and the network port came alive. Data LEDs blinked but the solid red light indicating the device was booting stayed on forever. I changed the Ethernet cable, port, lots of reboots, turned my 1Gbit ports down to 100Mbit... Nothing. It turned out whatever DHCP client they've implemented doesn't seem to work with a Fritz Box 7390. I've never seen this issue before. I returned the first unit thinking it was bad. When the second unit did the same thing I tried shoving the Ethernet cable into the Ethernet port of my desktop and fired up udhcp. To my surprise, that did the trick. Now I need to figure out some way to get this on my network without forwarding traffic through my desktop that isn't always on.. Urgh!

2. The all-your-data-are-belong-to-us cloudy thing.

Everything these days seems to want to provide an app. That alone isn't a problem, but in order to serve data to the app, your data generally lives in the cloud and this is where I reach for my tin-foil hat and begin the fun little process of reverse engineering the protocol this thing uses. :D

I don't think there is anything sinister going on here. I just want control of my data and would like my device to continue to work if (or rather when) the manufacturer decides to stop hosting it. The web interface this thing provides is pretty slick and there is both an iPhone and Android app but when it comes to my personal data, I'd rather hold it myself, thanks! So on that note...

The solution: Reverse Engineering!

The hardware I presume was actually produced by Efergy but it seems that the software for the "hub" device was provided by a company called Hildebrand based out of London and it's Hildebrand that actually receive the raw meter readings and store them on their servers. The Efergy website hosts the interface but the raw data goes to a sensornet.info domain that is registered to Hildebrand.

Boot Sequence

I have a box stamped with HK 1.1 Firmware AU on the bottom of it. I can't guarantee other regions behave the same way but here goes.

1. DHCP request is broadcast to network. Device waits for a repsonse.
2. DHCP response received. Device starts resolving using DHCP-provided DNS server the hostnames "uk.pool.ntp.org" and "ff.ee.dd.aabbcc.h2.sensornet.info" where "aabbccddeeff" is the MAC address of the device.
3. Device requests from the resovled sensornet host IP the URL https://<ip>/get_key.html. Response is HTTP/200 of the form "TT|ALPHANUMERIC"
4. Device requests from the same host IP the URL "https://<ip>/check_key.html?p=TT&ts=
   0000019D&h=<some hash>. HTTP/200 with an empty response is returned.
5. Device starts posting periodically to https://<ip>/h2 with "Content-Type: application/eh-data" and content of the form: "123456|1|EFCT|P1,0.00.".
6. Device periodically also sends requests to https://>ip>/h2 with "Content-Type: application/eh-ping" and an empty body, presumably just to let the service know it's alive.

Some other nice tidbits:

• SSL is used but certificates are never checked so MITM is very easy to do to read the data.
• The "ts" GET argument in check_key.html is not a timestamp. I've seen it go backwards and always seems close to zero. I suspect it's irrelevant as we can return "success" on any data. We don't really care about authentication here.
• The data format seems to be "<Sensor ID> | 1 | <sensor type> | <INPUT>,<Reading>."
• I suspect unit of measurement to be in milliamps but haven't yet confirmed this. The user will have to take voltage and power factor into account to work out kW and kW/h.

Update: A fake cloudy thing.

I had a Raspberry Pi that I wasn't doing much with so I turned it into my data logger by running my own DHCP, DNS and HTTPS servers, each pointing the device to the rPi instead of the hildebrand servers. I have a USB to Ethernet dongle to talk to the hub and the rPi ethernet adapter to talk to my LAN. Win! Source code is on github here.

Graphing HDD health with smartctl

I proudly built myself a front door for my TV cabinet recently - the very same TV cabinet that houses my NAS. Two weeks later and two crashes of my NAS box (that coincidentally uses my drives for swap), I discover 2 of my 4 HDD's had started giving errors, one had completely died. Turns out this thing called "ventilation" is important after all! *shrugs*

rPi + tvheadend + shepherd

My home TV setup involves a file server (FreeNAS), a Mac Mini running XBMC and a Raspberry Pi running TVHeadend.

In setting up the rPi side of this, I came across a lot of scattered instructions so I thought I'd bring them together. Nothing is particularly difficult but I suspect I'll be doing this again at some stage in the future so for the sake of myself and others...

The state of 3D printing

The world of 3D printing has intrigued me for years. An industrial designer friend of mine has been tinkering with 3D printed prototypes and objects for a few years now and I've been surveying the state of things for a while. I particularly liked the look of theForm1 on kickstarter although the limited choice of material is a potential issue. Anyway, for whatever reason, after a few hours of trawling shapeways, I feel compelled to rehash my favourites:

8. I have trouble believing the claim that this actually works. The sheer number of components here and the tolerances they must have is incredible.


7. An elegant chopstick holder that would look at home at the most sophisticated dinner table.


6. A lens cap holder that attaches to your camera strap. Ingeniously simple and useful and it looks so professional it's hard to tell from the picture what the object actually is.


5. A Galaxy S3 case + credit card holder + money clip + bottle opener. Seriously, I wish I had a Galaxy S3 right now.


4. A blast from my childhood past! Evil tentacle!


3. Serious jewellery. Sure, you have to take it to a professional jeweller to get it properly made but the fact the designs like this can be based on a 3D printed base is still very impressive.


2. You can actually get stuff printed in stainless steel now. Awesome!



1. This mug looks incredible. Printed in ceremic, the detail is amazing. The price is the only reason I've held back and I assume these will drop significantly with time.

You CAN still get a pre-paid data SIM card in Japan

I read a lot of blogs stating you couldn't get a pre-paid SIM card in Japan. It's true that the big players (NTT Docomo, Softbank, AU) don't sell these anymore but turns out that there is still at least one way to do this.

A company called b-Mobile sells pre-paid SIM cards valid for 1 month (about $USD32) and 3 month (about $USD100) with 1GB of data each. Yes, I wonder if anyone goes with the 3 month option...

The cards don't support calls but they run on NTT Docomo's LTE, HSDPA, 3G networks and data coverage seems very good.

You can only get them from big retailers. In Osaka, that means BIC Camera in Namba or, in my case, Yodobashi Camera in Umeda. You also need to activate them in Japanese - with a Japanese mobile apparently. So it definitely helps if you speak Japanese, have a Japanese friend, or if you're super convincing, maybe you can persuade the sales person to do it for you. From what I could tell, the U300 SIM they used to offer in English, pre-activated 24 hours after mail order purchase is no longer on offer. :(

In my case, I've been running on a 1GB, 1 month MicroSIM for a week now and using SkypeIn for receiving incoming calls. Calls are very high latency (almost unusable) but it has served its purpose so far - if I get a call that isn't working well, I just fall back to contact via email.

Best of luck Japanese travellers!

The worlds ugliest webm streaming webserver?

I was trying to find an easy way to get low latency video from a webcam to a remote browser today. Requirements:

1. Quick deployment
2. Runs on Raspberry Pi
3. Runs with out-of-the-box debs (see quick deployment)

My hacky solution was a Django python app that uses the StreamingHttpResponse class, gstreamer and a pipe. Disguisting, but works well. Sadly, latency is about 10 seconds to localhost so its not exactly live... 

import pygst
pygst.require("0.10")
import gst

def grab(request):
  """ Return a webm live stream from the first attached webcam. """
  class VideoStreamer():
    def __init__(self):
      self.pipein, self.pipeout = os.pipe()
      self.player = gst.parse_launch ("v4l2src ! video/x-raw-yuv,width=640,height=480,framerate=10/1 ! ffmpegcolorspace ! vp8enc max-latency=1 lag-in-frames=1 ! webmmux name='m' streamable=true ! fdsink fd=%d" % self.pipeout)
      self.player.set_state(gst.STATE_PLAYING)
    def start(self):
      fd = os.fdopen(self.pipein)
      try:
        while True:
          yield fd.read(4096)
      except Exception, e:
        print "Exception was ", e
    def __del__(self):
      self.player.set_state(gst.STATE_NULL)
      os.close(self.pipeout)
  return StreamingHttpResponse(VideoStreamer().start(), content_type="video/webm")

Requires pygst and django 1.5 (use pip).

Bricked Netgear Stora? Arduino to the rescue!

My latest attempt at getting offsite backups working for me in the most convenient manner possible involves scattering storage devices around at places I frequently visit such as relatives places, etc. I thought I'd re-purpose a Netgear Stora device I had lying about (P.S. Don't ever buy one of these if you value your privacy) by modding the firmware on it. Turns out, an arduino makes a great TTL serial adapter if you short the reset pin to GND. :) 

I'm also particularly proud of my ghetto MacGyver-like pin connectors. They were created with PVC tape rolled around the leg of a resistor and then cut into thirds and slipped over each pin to hold them in place. (Yes, I need to get myself some more electronics gear...)

Pin outs here in case anyone stumbles across this wanting to do something similar. :)