Removing Mosaic's (Part 1)

I had an interesting chat with a friend last week where I mentioned how I thought video mosaic's provided very little security and should't be relied upon to protect the identity of anyone or anything. It was the now famous untwirling case that led to the capture of a paedophile in Thailand that originally got me thinking about this. Anyway, after an hour of chatting I was convinced I could decode at least a subset of mosaic use-cases so I set off on my quest to learn OpenCV. Long story short: Don't use a mosaic to protect someones privacy.

Suprisingly, it only took 5 hours of coding to get this result:

[caption id="attachment_100" align="aligncenter" width="604" caption="Screenshots of video taken with hand-held camera after 1 second and 20 second."][/caption]

As you can see, the two versions are chalk and cheese! The artifacts you see in the bottom are partially a result of tracking errors and partially a result of me slightly moving over the 20 second interval in which I took this. Now  before panicking too much, let me explain that this is a rather contrived situation here.

1. I am trying to stay still.
2. I am purposely moving the camera.
3. The background is static.
4. I am assuming each mosaic tile's colour is based on the centre-most pixel under the tile.

[caption id="attachment_101" align="alignright" width="150" caption="The colour of a mosaic tile is assumed to come from the centre-most pixel under the tile."][/caption]

I am simply tracking the motion of the camera and using this and my assumptions about the mosaic tile transform and relatively static image to reverse the transform.

The tracking process here is quite simple. The red dots scattered around the picture are features tracked over time. OpenCV provides an easy way to select such features using the cvGoodFeaturesToTrack function. In my case:

cvGoodFeaturesToTrack(grayscale, 0, 0, numCorners, &nc, 0.4, 5.0);

In my haste to test out this idea, I haven't tuned any of the parameters. I've just picked values that seemed sane with my limited knowledge of OpenCV and run with it.

Once I have a set of points, I use this to track the motion of the video frame over time. As features disappear for whatever reason from frame-to-frame, I search out and assign new ones. These features are combined to provide a common point of reference for every frame of the video.

Using this point of reference, I can determine location from which a mosaic tile's colour was drawn and use samples over multiple frames of video to form a higher resolution representation of the actual image.

In the current version, I don't support camera rotation. OpenCV does provides functions to help track motion in full 3D but I guess 5 hours wasn't enough time for me to absorb all the math but I've got a few other ideas to track moving objects behind mosaics so this experiment is far from over!