The Blog of Ian Mercer.

Filtering techniques

Even the more reliable sensors will glitch occasionally and some of the less reliable sensors will glitch all the time. Filtering sensor data to ensure that these bad values don't get through is essential. Here are some techniques:

Moving Average

This has reasonable results and is very simple to code. You can maintain a circular buffer of values and calculate an average each time you add a new value by averaging all the values in the buffer. In it's simplest for you can keep just one value and do something like:

   current = 0.9 * current + x * 0.1

But it's not great. The glitch values still move the average, and the output lags the input.

Median filter

Instead of taking the average of the buffer, take the median value. This means you need to copy and sort the buffer but that's fairly easy even in C. This approach also lags but it is great for rejecting the occasional stupid spike.

Kalman filter

The Kalman filter is a bit harder to understand but essentially it works a bit like a moving average but it also learns how much of the new value to bring in each time and how much of the old value to maintain. It can also operate on multidimensional data.

Recommendation

In addition to rejecting just plain stupid outliers (e.g. 212 degrees for an indoor temperature), I recommend a small median filter to reject outliers followed by a kalman filter for most situations.

Kalman Code (C)

/*
      Kalman filter
*/

struct Kalman
{
    double err_measure;
    double err_estimate;
    double q;
    double current_estimate;
    double last_estimate;
    double kalman_gain;
};

void kalman_initialize(struct Kalman *k)
{
    k->current_estimate = NAN;   // marker value used by time interval check
    k->last_estimate = NAN;      // marker value, so first real value overrides it
    k->err_measure = 10.0;
    k->err_estimate = 10.0;
    k->q = 0.15;   // was 0.25, moved down for more frequent updates
}

float kalman_update(struct Kalman *k, double mea)
{
    // First time through, use the measured value as the actual value
    if (isnan(k->last_estimate))
    {
        k->last_estimate = mea;
        k->current_estimate = mea;
        return mea;
    }

    k->kalman_gain = k->err_estimate / (k->err_estimate + k->err_measure);
    k->current_estimate = k->last_estimate + 
        k->kalman_gain * (mea - k->last_estimate);
    k->err_estimate = (1.0 - k->kalman_gain) * k->err_estimate + 
        fabs(k->last_estimate - k->current_estimate) * k->q;
    k->last_estimate = k->current_estimate;

    return k->current_estimate;
}

Related Stories

Bluetooth Tracking Project

My year long Bluetooth project that won the $20,000 HCI and Microsoft competition during lockdown has continued to grow and now reliably tracks how many people are in the house and outside and can locate any device down to room level.

Ian Mercer
Ian Mercer

Home Automation Sensors

An overview of the many sensors I've experimented with for home automation including my favorite under-floor strain gauge, through all the usual PIR, beam and contact sensors to some more esoteric devices like an 8x8 thermal camera.

Ian Mercer
Ian Mercer

3d Printed ESP32 Brick

ESP32 provides a great platform for sensors around the house but by the time you've added a USB power brick, cable and enclosure it's quite messy. I wanted a device that I could just plug in with no exposed wires and no mounting needed so I designed one in OpenSCAD.

Ian Mercer
Ian Mercer

Bluetooth Sensing for Home Automation

Bluetooth sensing for home automation is a great proxy for people counting as it can detect and locate each cellphone in the house. iBeacons attached to tools, cars and pets can provide a 'find my anything' feature too.

Ian Mercer
Ian Mercer

Gas sensors

Gas sensors come in many different flavors including CO2, VOC and particulate sensors.

Ian Mercer
Ian Mercer

Humidity Sensors (DHT11, DHT22, AM2320)

Humidity sensors are great for controlling extractor fans in bathrooms and other damp spaces.

Ian Mercer
Ian Mercer

Light sensors for Home Automation

Having at least one light sensor is critical for any home automation system that controls lightng. Lights need to be turned on when it's dark not at specific times of day, especially here in Seattle when it can be dark and cloudy at any time of day.

Ian Mercer
Ian Mercer

Microwave Doppler Sensors (RCWL-0516)

Microwave doppler sensors can be found in some alarm sensors but there are also available very cheaply as a separate component. They offer exceptional range but suffer from false triggers requiring a probailistic approach to people sensing.

Ian Mercer
Ian Mercer

Optical-beam sensors

Optical-beam sensors are reliable and can cover a long-distance such as across a garage or aisle-way. When they include multiple-beams they have good false-trigger rejection.

Ian Mercer
Ian Mercer

PIR Sensors for Home Automation

PIR sensors are cheap and easy to use but they suffer from slow response times and low repeat rates.

Ian Mercer
Ian Mercer

Strain-gauges

Strain-gauges are my top-rated sensor for home automation because they are invisible, reliable and can be tuned to detect people and ignore pets.

Ian Mercer
Ian Mercer

Temperature sensors for home automation

Temperature sensors I've experimented with for home automation.

Ian Mercer
Ian Mercer

The Grideye 8x8 camera sensor

Experiments with an 8x8 IR camera for privacy-preserving people detection using cameras.

Ian Mercer
Ian Mercer

CCTV Cameras as Home Automation Sensors

CCTV cameras are an option for detecting people but within the home there are privacy concerns that need to be addressed.

Ian Mercer
Ian Mercer

Pressure Sensors for Home Automation

Pressure sensors can detect HVAC system operation and could potentially detect clogged filters.

Ian Mercer
Ian Mercer