The Blog of Ian Mercer.

A wireless sensor network using Moteino boards

Cover Image for A wireless sensor network using Moteino boards

Over the past few months I've been experimenting with the Netduino and Arduino as ways to expand the inputs on my home automation system without running more wires off to various corners of the house. An Arduino with Ethernet can have a group of local sensors attached to it which it can process and then POST the sensor values to a REST API on the main home automation computer. The Arduino is, itself, quite remarkable both for the number of inputs it has and for its relatively low cost.

I should also mention the Netduino. I started out with a few of those, hoping that I could keep the whole system in C#. But, sadly my experience was not great as they would simply hang when performing web requests back to the home automation computer. Reluctantly I switched to Arduino and since then all my effort have been on the Arduino platform. Of course, I immediately hit every possible combination of buffer overrun, bad pointer arithmetic, ... and other problems you get from writing in a language with no safety checks, but my C++ skills gradually returned. I was delighted to find the `String` class but immediately abandoned it when I realized how critical memory usage was and how much everyone on the Arduino forums hated it.

The Arduino worked great, I have a few with Ethernet shields on them but switched quickly to the Arduino Ethernet which has Ethernet built right in. Sadly it does not have the low-profile ethernet jack that the Netduino has so you need to use extra long stacking headers or a second layer in order to avoid physical collisions.

One Arduino now handles all 1-Wire inputs from the boiler and radiant floor monitoring system, another handles the solar sensor and an external temperature sensor. They run quite reliably, posting updates at regular intervals.

One idea I developed for this was an API where after posting a new value the server responds with three values: Resolution, Minimum Interval and Maximum Interval. The Arduino then adjusts how it sends future updates to respect these new values. This allows the sensor configuration to be changed on the server without deploying new code to the devices. The Resolution and Minimum Interval values throttle the amount of data that is being sent (e.g. 0.5 degree resolution, no more than once a minute) whilst the Maximum Interval is used to make sure every Arduino is still alive even if the sensor isn't changing value. A fairly simple API but it works well.

public class SensorResponse 
{
	/// <summary>
	/// Sequence number - increments each time a value is added
	/// </summary> 
	public long sequence { get; set; }

	/// <summary>
	/// Minimum interval between calls (seconds) 
	/// </summary> 
	public double min { get; set; }

	/// <summary>
	/// Max interval between calls (seconds) 
	/// </summary>
	public double max { get; set; }

	/// <summary>
	/// Minimum change between calls 
	/// </summary> public
	double resolution { get; set; } 
}

Sensor values are stored using VariableWithHistory that I blogged about previously.

This all works well, but I wanted even higher reliability so I set about building a hardware watchdog shield using a 74HC4060 chip and instructions you can find online. This works as follows: within the main loop you need to trigger a given digital output once every few seconds, if you don't (because the code has hung) then the reset line is brought low automatically and the Arduino reboots. The only snag is that a larger sketch may take too long to upload, so simply disconnect the watchdog shield during development and connect it when it's time to deploy. I also decided that I wanted a PCB version of this watchdog board so I contracted someone on ODesk to design one for me and it's now off being manufactured. I added a prototyping area to the board and the necessary connector and resistor for a 1-Wire connection. I have a small initial run coming and if there's interest I may produce more for sale.

But then my Spark Core devices arrived. These have Wifi on board and an Arduino compatible programming language. The IDE however runs in the cloud and you have a web interface to program them. It took a full evening to get them to work correctly, most of the instructions I followed simply did not work and left the cores flashing one of their many color combinations that mean "not working". Eventually I connected using Putty and was able to enter my network SSID and passphrase and got them online. But the problems continued - failures to connect to the devices, somewhat annoying UI where the currently selected Core isn't shown on the main IDE page so it's easy to push to the wrong core. But the final nail in their coffin was a bug whereby long running HTTP requests simply hang the device (known issue, they are working on it). I haven't given up on them totally, but for now they are just a toy and aren't going to be playing any part in the home automation system.

And then I discovered what may be the best device yet for wireless home automation: The Moteino. This is an Arduino Uno clone but much smaller, with the option for an on-board 433Mhz radio. I ordered some with the RFM69, high powered option and the flash memory chip which brings the price up to $24.95. That's still a lot cheaper than an Arduino Ethernet or an Arduino Uno with an Ethernet shield. Unlike the WIFI option of the Spark, which has a fairly poor range when using the onboard antenna, this thing, with a small whip antenna (actually just a piece of wire about 6" long), can reach from one end of our house to the other.

After soldering on the antenna it took just a few minutes to download my first send/receive program and they were soon flashing LEDs happily to each other across the room in perfect sync.

I created a quick 'Sender' based on the samples provided. I post sensor data as JSON but create the JSON using sprintf because a full JSON library isn't realistic for these small devices. But now, instead of posting the data directly to my server it sends it out over the 433Mhz radio to be received by a gateway device that I built. One trick I used to allocate unique device numbers was to use the unique serial number in the Flash chip to create the radio ID number. Of course there's no guarantee that two won't collide but the odds are fairly good (for now) and if I do find a collision I'll adjust the XOR calculation to find one that doesn't create collisions with the devices I have.

nodeId = 27;
flash.readUniqueId(); 
for (byte i=0;i<8;i++) 
{
	nodeId \^= flash.UNIQUEID[i]; 
}

The gateway device consists of a RFM69HW module coupled to an Arduino Ethernet. I was going to order an adapter board that could handle the small spacing on the RFM69W module but couldn't wait for it so I ran solid core wires from each pad down to an Arduino prototyping shield and left the radio module floating above the board suspended by the wires. The only other tricky piece here was that the Ethernet and the radio need to share the Serial Peripheral Interface (SSI). I had to move the Slave Select (SS) from the RFM69W's normal connection pin (D10) to a different pin and set that new pin in code. There's also the need for a level-converter (not shown in this image).

	radio.setCS(9); // Change SS pin if necessary (conflicts with Ethernet)

Other than that the gateway code was very simple, it receives a message from a numbered device and posts the message to the home automation server along with the device number and the received signal strength.

	[HttpPost]
	[Route("api/moteino/{uid}/{rssi}")]
	public string MoteinoGateway(int uid, int rssi, MoteinoStatus status) ...

I plan to locate the Gateway device in the attic so it has excellent coverage of the whole house and beyond ...

Since the range on these is so good I may try adding some sending nodes to our cars. I might even put some extra gateway devices around the place so I can triangulate on signal strength, maybe even put one at the barn and one at work so the house knows exactly when I arrive at each location. And since the device will be mobile it can also begin to collect information as I drive about. Now the house can be aware of where each car is, how much time I spend driving each day, how long it took me to get to work and home, ... and many more, potentially interesting things. As always, I don't know exactly what I'm going to use all this data for but sometimes interesting scenarios just happen!

Related Stories

Cover Image for Time Series Data Compression

Time Series Data Compression

This new technique to compress the time series data collected by my home automation system seems to be working really well.

Ian Mercer
Ian Mercer
Cover Image for Home Automation

Home Automation

I've been working on home automation for over 15 years and I'm close to achieving my goal which is a house that understands where everyone is at all times, can predict where you are going next and can control lighting, heating and other systems without you having to do or say anything. That's a true "smart home".

Ian Mercer
Ian Mercer
Cover Image for Digital Twins are never identical

Digital Twins are never identical

Digital Twin are an online representation of a real world object, a copy of its properties in the digital world and a way to send updated and commands to it. In effect I've been making them for years but now they have a trendy name.

Ian Mercer
Ian Mercer
Cover Image for Home Automation Sensors

Home Automation Sensors

An overview of the many sensors I've experimented with for home automation including my favorite under-floor strain gauge, through all the usual PIR, beam and contact sensors to some more esoteric devices like an 8x8 thermal camera.

Ian Mercer
Ian Mercer
Cover Image for Why smarthomes are hard

Why smarthomes are hard

Why automated learning is hard for a smart home. The perils of over-fitting, under-fitting and how the general unpredictable nature of life makes it hard to build a system that learns your behavior.

Ian Mercer
Ian Mercer
Cover Image for Collinearity test for sensor data compression

Collinearity test for sensor data compression

One way to reduce the volume of sensor data is to remove redundant points. In a system with timestamped data recorded on an irregular interval we can achieve this by removing co-linear points.

Ian Mercer
Ian Mercer
Cover Image for Event blocks

Event blocks

Home automation systems need to respond to events in the real world. Sometimes it's an analog value, sometimes it's binary, rarely is it clean and not susceptible to problems. Let's discuss some of the ways to convert these inputs into actions.

Ian Mercer
Ian Mercer
Cover Image for Logistic function - convert values to probabilities

Logistic function - convert values to probabilities

Another super useful function for handling sensor data and converting to probabilities is the logistic function 1/(1+e^-x). Using this you can easily map values onto a 0.0-1.0 probability range.

Ian Mercer
Ian Mercer
Cover Image for ATAN curve for probabilities

ATAN curve for probabilities

In a home automation system we often want to convert a measurement into a probability. The ATAN curve is one of my favorite curves for this as it's easy to map overything onto a 0.0-1.0 range.

Ian Mercer
Ian Mercer
Cover Image for Home Construction Advice

Home Construction Advice

Several years ago we did a major remodel. I did all of the finish electrical myself and supervised all of the rough-in electrical. I also put in all of the electrical system and water in our barn. I have opinions ...

Ian Mercer
Ian Mercer
Cover Image for T-Mobile home internet

T-Mobile home internet

I'm testing a T-Mobile Home Internet device as a backup to XFinity and a way to offload half our monthly traffic to avoid the XFinity 1.2TB cap

Ian Mercer
Ian Mercer
Cover Image for Probabilistic Home Automation

Probabilistic Home Automation

A probabilistic approach to home automation models the probability that each room is occupied and how many people are in that room.

Ian Mercer
Ian Mercer
Cover Image for Multiple hypothesis tracking

Multiple hypothesis tracking

A statistical approach to understanding which rooms are occupied in a smart house

Ian Mercer
Ian Mercer
Cover Image for A state machine for lighting control

A state machine for lighting control

An if-this-then-that style rules machine is insufficient for lighting control. This state machine accomplishes 90% of the correct behavior for a light that is controlled automatically and manually in a home automation system.

Ian Mercer
Ian Mercer
Cover Image for Home Automation States

Home Automation States

Understanding the many different 'states' a house can have is critical to creating great home automation

Ian Mercer
Ian Mercer
Cover Image for Graphing gigabytes of home automation data with tableau

Graphing gigabytes of home automation data with tableau

Some interesting charts from the gigabytes of data my home automation system produces

Ian Mercer
Ian Mercer
Cover Image for iBeacons for Home Automation

iBeacons for Home Automation

My investigations into using iBeacons for home automation

Ian Mercer
Ian Mercer
Cover Image for iBeacon meetup in Seattle - January 2015

iBeacon meetup in Seattle - January 2015

My notes on the iBeacon meetup in Seattle held in January 2015

Ian Mercer
Ian Mercer
Cover Image for Home Automation Systems as a Graph

Home Automation Systems as a Graph

Using nodes and links to represent a home and all the devices in it

Ian Mercer
Ian Mercer
Cover Image for N-Gram Analysis of Sensor Events in Home Automation

N-Gram Analysis of Sensor Events in Home Automation

Using n-gram analysis to spot patterns in sensor activations

Ian Mercer
Ian Mercer
Cover Image for Xamarin Forms Application For Home Automation

Xamarin Forms Application For Home Automation

Building a Xamarin Forms application to control my home automation system

Ian Mercer
Ian Mercer
Cover Image for The Internet of Hubs (and things)

The Internet of Hubs (and things)

Maybe it should be called the Internet of Hubs instead

Ian Mercer
Ian Mercer
Cover Image for Showing home status with just a single RGB LED

Showing home status with just a single RGB LED

Multicolored LEDs can convey a lot of information in a small space

Ian Mercer
Ian Mercer

JSON Patch - a C# implementation

Ian Mercer
Ian Mercer
Cover Image for The home as a user interface

The home as a user interface

Ian Mercer
Ian Mercer

A RESTful API for sensor data

POSTing data to a home automation system from Arduino devices

Ian Mercer
Ian Mercer
Cover Image for The Internet of Boilers

The Internet of Boilers

An experiment to measure every aspect of an HVAC / boiler system

Ian Mercer
Ian Mercer

VariableWithHistory - making persistence invisible, making history visible

A novel approach to adding history to variables in a programming language

Ian Mercer
Ian Mercer
Cover Image for A Quantified House - My Talk to the Seattle Quantified Self Meetup

A Quantified House - My Talk to the Seattle Quantified Self Meetup

My talk to the Seattle Quantified Self meetup

Ian Mercer
Ian Mercer

Integrating an Android phone into my home automation system

Some new features for my home automation using an Android phone

Ian Mercer
Ian Mercer

Before there was the web there was BeebTel

Just thought I should mention that I built a web-like system before the web existed

Ian Mercer
Ian Mercer

My first programme [sic]

At the risk of looking seriously old, here's something found on a paper tape

Ian Mercer
Ian Mercer
Cover Image for The Internet of Dogs

The Internet of Dogs

Connecting our dog into the home automation

Ian Mercer
Ian Mercer
Cover Image for GreenGoose Review

GreenGoose Review

A review of the now defunct GreenGoose sensor system

Ian Mercer
Ian Mercer
Cover Image for Home power meters revisited

Home power meters revisited

Ian Mercer
Ian Mercer
Cover Image for Home Automation Calendar Integration

Home Automation Calendar Integration

Ian Mercer
Ian Mercer

Web site crawler and link checker (free)

Ian Mercer
Ian Mercer

Smart home energy savings - update for 2010

Ian Mercer
Ian Mercer
Cover Image for A smart power strip

A smart power strip

Ian Mercer
Ian Mercer
Cover Image for What does a Smart House do at Halloween?

What does a Smart House do at Halloween?

My favorite home automation features for Halloween

Ian Mercer
Ian Mercer
Cover Image for Home Automation Top Features

Home Automation Top Features

Ian Mercer
Ian Mercer
Cover Image for Weather Forecasting for Home Automation

Weather Forecasting for Home Automation

Ian Mercer
Ian Mercer
Cover Image for How can I tell if my house is smart?

How can I tell if my house is smart?

Ian Mercer
Ian Mercer

Home Automation Block Diagram

Ian Mercer
Ian Mercer

Elliott 803 - An Early Computer

Ian Mercer
Ian Mercer

World's Smartest House Demonstration

Ian Mercer
Ian Mercer

Looking forward to the new year and our new datacenter

Historical note about moving my servers into a datacenter

Ian Mercer
Ian Mercer

Future proof your home with a new conduit system?

Running conduit can be expensive but maybe you don't need one to every room

Ian Mercer
Ian Mercer

Second Drobo Update

At this point things were looking up for my Drobo

Ian Mercer
Ian Mercer

It's all about disk speed

Why disk speed is the most critical aspect for most modern PCs and servers

Ian Mercer
Ian Mercer

Comcast woes and a new monitoring utility

Monitoring a cable modem using its HTML management interface

Ian Mercer
Ian Mercer

Core duo desktop machine runs cool

Ian Mercer
Ian Mercer
Cover Image for New Home Automation Server

New Home Automation Server

Ian Mercer
Ian Mercer
Cover Image for World's Smartest House

World's Smartest House

Over 15 years of experimentation with home automation

Ian Mercer
Ian Mercer
Cover Image for Preparing for death

Preparing for death

A friend died last year, it wasn't unexpected. He left a lot for his friends to cleanup. Maybe these notes can help someone else prepare better.

Ian Mercer
Ian Mercer
Cover Image for World's Smartest House Videos

World's Smartest House Videos

A collection of videos about my smart home efforts

Ian Mercer
Ian Mercer