# Filtering techniques

Even the more reliable sensors will glitch occasionally and some of the less reliable sensors will glitch all the time. Filtering sensor data to ensure that these bad values don't get through is essential. Here are some techniques:

## Moving Average

This has reasonable results and is very simple to code. You can maintain a circular buffer of values and calculate an average each time you add a new value by averaging all the values in the buffer. In it's simplest for you can keep just one value and do something like:

``````   current = 0.9 * current + x * 0.1
``````

But it's not great. The glitch values still move the average, and the output lags the input.

## Median filter

Instead of taking the average of the buffer, take the median value. This means you need to copy and sort the buffer but that's fairly easy even in C. This approach also lags but it is great for rejecting the occasional stupid spike.

## Kalman filter

The Kalman filter is a bit harder to understand but essentially it works a bit like a moving average but it also learns how much of the new value to bring in each time and how much of the old value to maintain. It can also operate on multidimensional data.

## Recommendation

In addition to rejecting just plain stupid outliers (e.g. 212 degrees for an indoor temperature), I recommend a small median filter to reject outliers followed by a kalman filter for most situations.

## Kalman Code (C)

``````/*
Kalman filter
*/

struct Kalman
{
double err_measure;
double err_estimate;
double q;
double current_estimate;
double last_estimate;
double kalman_gain;
};

void kalman_initialize(struct Kalman *k)
{
k->current_estimate = NAN;   // marker value used by time interval check
k->last_estimate = NAN;      // marker value, so first real value overrides it
k->err_measure = 10.0;
k->err_estimate = 10.0;
k->q = 0.15;   // was 0.25, moved down for more frequent updates
}

float kalman_update(struct Kalman *k, double mea)
{
// First time through, use the measured value as the actual value
if (isnan(k->last_estimate))
{
k->last_estimate = mea;
k->current_estimate = mea;
return mea;
}

k->kalman_gain = k->err_estimate / (k->err_estimate + k->err_measure);
k->current_estimate = k->last_estimate +
k->kalman_gain * (mea - k->last_estimate);
k->err_estimate = (1.0 - k->kalman_gain) * k->err_estimate +
fabs(k->last_estimate - k->current_estimate) * k->q;
k->last_estimate = k->current_estimate;

return k->current_estimate;
}
``````

## Related Stories

### Time Series Data Compression

This new technique to compress the time series data collected by my home automation system seems to be working really well.

Ian Mercer

### Bluetooth Tracking Project

My year long Bluetooth project that won the \$20,000 HCI and Microsoft competition during lockdown has continued to grow and now reliably tracks how many people are in the house and outside and can locate any device down to room level.

Ian Mercer

### Home Automation Sensors

An overview of the many sensors I've experimented with for home automation including my favorite under-floor strain gauge, through all the usual PIR, beam and contact sensors to some more esoteric devices like an 8x8 thermal camera.

Ian Mercer

### Collinearity test for sensor data compression

One way to reduce the volume of sensor data is to remove redundant points. In a system with timestamped data recorded on an irregular interval we can achieve this by removing co-linear points.

Ian Mercer

### 3d Printed ESP32 Brick

ESP32 provides a great platform for sensors around the house but by the time you've added a USB power brick, cable and enclosure it's quite messy. I wanted a device that I could just plug in with no exposed wires and no mounting needed so I designed one in OpenSCAD.

Ian Mercer

### Bluetooth Sensing for Home Automation

Bluetooth sensing for home automation is a great proxy for people counting as it can detect and locate each cellphone in the house. iBeacons attached to tools, cars and pets can provide a 'find my anything' feature too.

Ian Mercer

### Gas sensors

Gas sensors come in many different flavors including CO2, VOC and particulate sensors.

Ian Mercer

### Humidity Sensors (DHT11, DHT22, AM2320)

Humidity sensors are great for controlling extractor fans in bathrooms and other damp spaces.

Ian Mercer

### Light sensors for Home Automation

Having at least one light sensor is critical for any home automation system that controls lightng. Lights need to be turned on when it's dark not at specific times of day, especially here in Seattle when it can be dark and cloudy at any time of day.

Ian Mercer

### Microwave Doppler Sensors (RCWL-0516)

Microwave doppler sensors can be found in some alarm sensors but there are also available very cheaply as a separate component. They offer exceptional range but suffer from false triggers requiring a probailistic approach to people sensing.

Ian Mercer

### Optical-beam sensors

Optical-beam sensors are reliable and can cover a long-distance such as across a garage or aisle-way. When they include multiple-beams they have good false-trigger rejection.

Ian Mercer

### PIR Sensors for Home Automation

PIR sensors are cheap and easy to use but they suffer from slow response times and low repeat rates.

Ian Mercer

### Strain-gauges

Strain-gauges are my top-rated sensor for home automation because they are invisible, reliable and can be tuned to detect people and ignore pets.

Ian Mercer

### Temperature sensors for home automation

Temperature sensors I've experimented with for home automation.

Ian Mercer

### Event blocks

Home automation systems need to respond to events in the real world. Sometimes it's an analog value, sometimes it's binary, rarely is it clean and not susceptible to problems. Let's discuss some of the ways to convert these inputs into actions.

Ian Mercer

### The Grideye 8x8 camera sensor

Experiments with an 8x8 IR camera for privacy-preserving people detection using cameras.

Ian Mercer

### CCTV Cameras as Home Automation Sensors

CCTV cameras are an option for detecting people but within the home there are privacy concerns that need to be addressed.

Ian Mercer

### Pressure Sensors for Home Automation

Pressure sensors can detect HVAC system operation and could potentially detect clogged filters.

Ian Mercer