The Blog of Ian Mercer.

VariableWithHistory - making persistence invisible, making history visible

In a typical .NET application variables have a short lifetime. When they go out of scope or the application ends their value is lost. Also, you cannot ask a variable what its value was 1 hour ago, or what its average, maximum or minimum value was yesterday.

Yet, such a variable would be extremely useful when writing a Home Automation System because you often need to make comparisons between a current value and some historical average, or between two ranges (e.g. was the kitchen more or less occupied than yesterday). Now, normally you wouldn't want to mix persistence up with the representation of a value in your code (see 'Separation of Concerns'), but in this case I decided that it was worth mixing the two concepts because the benefits of doing so were so great.

So I created a class called VariableWithHistory<T> which is the abstract base class for IntegerWithHistory, DoubleWithHistory, BoolWithHistory, StringWithHistory and a number of others.  The first property worth noting on these classes is the .Current property.  This always gives you the latest value that has been set.  Setting the .Current value stores both the value and the DateTime (Utc of course) at which the value became current.  A history of all past values is maintained in MongoDB up to some suitable limit per variable (each variable can have its own adjustable history size in bytes by using MongoDB's capped collections).  If the new value is the same as the old one no update is made, the implicit behavior being that the value changed and stayed there until it changes again, so if you want to know what the value is now it is the same as the last change recorded.

With this new variable type in place any object in the house can have any number of persistent fields on it (bool occupied, double temperature, string triggeredBy, ...).  Updating these values is as simple as assigning to their .Current property.  When the system loads, each value comes back with the value it had when the system was shut down.  To accomplish this every VariableWithHistory is given a unique id (based on the unique id of it's parent, e.g. a room).

So far so good, shut down, restart and the house doesn't need to query a device to know if it's on or off and all the long running Sequential Logic Blocks I use for rules (e.g. .Delay(days:2)) carry on running as if nothing happened.  This is particularly useful since I typically deploy a new version almost every day and some logic blocks have long delays built into them.

But besides providing simple recovery from a reboot, these persistent variables allow me to do some much more interesting things.

int CountTransitions(DateTimeRange range, T direction);

Counts how many transitions there have been to the value T in a given time range, e.g. how many times did the driveway alarm go 'true' this evening?

Dictionary<T, double> Fractional(DateTimeRange range);

Builds a histogram of all the values seen in the time range, e.g. 50% hot, 20% cold, 30% warm for a string variable that tracks temperature

DateTimeOffset LastChangedState

e.g. when was this sensor last triggered?

TimedValue<T> ValueAtTime(DateTimeOffset dt)

What was the value at a given time in the past, e.g. what was the temperature at the same time yesterday?

Each specific type of VariableWithHistory<T> may also have additional methods relevant to the type T.  For example, on DoubleWithHistory there is a method double Average(DateTimeOffset minValue, DateTimeOffset maxValue) which gets the average value over the specified time range.  On BoolWithHistory there is a method double PercentageTrue(DateTimeRange range) which you could use to find the average occupancy for a room yesterday.

My initial implementation waited for the database to write each update before allowing any queries but now I simply cache the Current value and assume that queries will probably get executed after updates and that the average temperature yesterday is close enough with or without the last 100ms of updates.  I did try to keep this class isolated from MongoDB but in the end the benefit of some of the atomic update capabilities in MongoDB made it easier to just take the dependency.

My previous implementation of this feature used my own in-memory database, MongoDB has slowed it down a bit but I've gained the ability to archive terabytes of sensor data which should prove useful for my next project which is to add some machine learning to the system.

Related Stories

Cover Image for Time Series Data Compression

Time Series Data Compression

This new technique to compress the time series data collected by my home automation system seems to be working really well.

Ian Mercer
Ian Mercer
Cover Image for Home Automation

Home Automation

I've been working on home automation for over 15 years and I'm close to achieving my goal which is a house that understands where everyone is at all times, can predict where you are going next and can control lighting, heating and other systems without you having to do or say anything. That's a true "smart home".

Ian Mercer
Ian Mercer
Cover Image for Home Automation Sensors

Home Automation Sensors

An overview of the many sensors I've experimented with for home automation including my favorite under-floor strain gauge, through all the usual PIR, beam and contact sensors to some more esoteric devices like an 8x8 thermal camera.

Ian Mercer
Ian Mercer
Cover Image for Collinearity test for sensor data compression

Collinearity test for sensor data compression

One way to reduce the volume of sensor data is to remove redundant points. In a system with timestamped data recorded on an irregular interval we can achieve this by removing co-linear points.

Ian Mercer
Ian Mercer
Cover Image for Event blocks

Event blocks

Home automation systems need to respond to events in the real world. Sometimes it's an analog value, sometimes it's binary, rarely is it clean and not susceptible to problems. Let's discuss some of the ways to convert these inputs into actions.

Ian Mercer
Ian Mercer
Cover Image for Logistic function - convert values to probabilities

Logistic function - convert values to probabilities

Another super useful function for handling sensor data and converting to probabilities is the logistic function 1/(1+e^-x). Using this you can easily map values onto a 0.0-1.0 probability range.

Ian Mercer
Ian Mercer
Cover Image for ATAN curve for probabilities

ATAN curve for probabilities

In a home automation system we often want to convert a measurement into a probability. The ATAN curve is one of my favorite curves for this as it's easy to map overything onto a 0.0-1.0 range.

Ian Mercer
Ian Mercer
Cover Image for My love/hate relationship with Stackoverflow

My love/hate relationship with Stackoverflow

Stackoverflow is a terrific source of information but can also be infuriating.

Ian Mercer
Ian Mercer
Cover Image for Probabilistic Home Automation

Probabilistic Home Automation

A probabilistic approach to home automation models the probability that each room is occupied and how many people are in that room.

Ian Mercer
Ian Mercer
Cover Image for Multiple hypothesis tracking

Multiple hypothesis tracking

A statistical approach to understanding which rooms are occupied in a smart house

Ian Mercer
Ian Mercer
Cover Image for A state machine for lighting control

A state machine for lighting control

An if-this-then-that style rules machine is insufficient for lighting control. This state machine accomplishes 90% of the correct behavior for a light that is controlled automatically and manually in a home automation system.

Ian Mercer
Ian Mercer
Cover Image for Home Automation States

Home Automation States

Understanding the many different 'states' a house can have is critical to creating great home automation

Ian Mercer
Ian Mercer
Cover Image for Graphing gigabytes of home automation data with tableau

Graphing gigabytes of home automation data with tableau

Some interesting charts from the gigabytes of data my home automation system produces

Ian Mercer
Ian Mercer
Cover Image for iBeacons for Home Automation

iBeacons for Home Automation

My investigations into using iBeacons for home automation

Ian Mercer
Ian Mercer
Cover Image for iBeacon meetup in Seattle - January 2015

iBeacon meetup in Seattle - January 2015

My notes on the iBeacon meetup in Seattle held in January 2015

Ian Mercer
Ian Mercer
Cover Image for Home Automation Systems as a Graph

Home Automation Systems as a Graph

Using nodes and links to represent a home and all the devices in it

Ian Mercer
Ian Mercer
Cover Image for N-Gram Analysis of Sensor Events in Home Automation

N-Gram Analysis of Sensor Events in Home Automation

Using n-gram analysis to spot patterns in sensor activations

Ian Mercer
Ian Mercer
Cover Image for Xamarin Forms Application For Home Automation

Xamarin Forms Application For Home Automation

Building a Xamarin Forms application to control my home automation system

Ian Mercer
Ian Mercer
Cover Image for The Internet of Hubs (and things)

The Internet of Hubs (and things)

Maybe it should be called the Internet of Hubs instead

Ian Mercer
Ian Mercer
Cover Image for Showing home status with just a single RGB LED

Showing home status with just a single RGB LED

Multicolored LEDs can convey a lot of information in a small space

Ian Mercer
Ian Mercer
Cover Image for A wireless sensor network using Moteino boards

A wireless sensor network using Moteino boards

The diminutive Arduino boards include a powerful transmitter/receiver

Ian Mercer
Ian Mercer

JSON Patch - a C# implementation

Ian Mercer
Ian Mercer
Cover Image for The home as a user interface

The home as a user interface

Ian Mercer
Ian Mercer

A RESTful API for sensor data

POSTing data to a home automation system from Arduino devices

Ian Mercer
Ian Mercer
Cover Image for The Internet of Boilers

The Internet of Boilers

An experiment to measure every aspect of an HVAC / boiler system

Ian Mercer
Ian Mercer

Websites should stop using passwords for login!

A slightly radical idea to eliminate passwords from many of the websites you use just occasionally

Ian Mercer
Ian Mercer

Dynamically building 'Or' Expressions in LINQ

How to create a LINQ expression that logically ORs together a set of predicates

Ian Mercer
Ian Mercer

Neo4j Meetup in Seattle - some observations

Some observations from a meetup in Seattle on graph databases and Neo4j

Ian Mercer
Ian Mercer
Cover Image for A Quantified House - My Talk to the Seattle Quantified Self Meetup

A Quantified House - My Talk to the Seattle Quantified Self Meetup

My talk to the Seattle Quantified Self meetup

Ian Mercer
Ian Mercer

Updated Release of the Abodit State Machine

A hierarchical state machine for .NET

Ian Mercer
Ian Mercer

Integrating an Android phone into my home automation system

Some new features for my home automation using an Android phone

Ian Mercer
Ian Mercer

My first programme [sic]

At the risk of looking seriously old, here's something found on a paper tape

Ian Mercer
Ian Mercer

Building a better .NET State Machine

A state machine for .NET that I've released on Nuget

Ian Mercer
Ian Mercer
Cover Image for The Internet of Dogs

The Internet of Dogs

Connecting our dog into the home automation

Ian Mercer
Ian Mercer
Cover Image for GreenGoose Review

GreenGoose Review

A review of the now defunct GreenGoose sensor system

Ian Mercer
Ian Mercer

A simple state machine in C#

State machines are useful in many contexts but especially for home automation

Ian Mercer
Ian Mercer

Convert a property getter to a setter

Ian Mercer
Ian Mercer
Cover Image for Home power meters revisited

Home power meters revisited

Ian Mercer
Ian Mercer
Cover Image for Home Automation Calendar Integration

Home Automation Calendar Integration

Ian Mercer
Ian Mercer

Smart home energy savings - update for 2010

Ian Mercer
Ian Mercer
Cover Image for A smart power strip

A smart power strip

Ian Mercer
Ian Mercer

MongoDB Map-Reduce - Hints and Tips

Ian Mercer
Ian Mercer
Cover Image for What does a Smart House do at Halloween?

What does a Smart House do at Halloween?

My favorite home automation features for Halloween

Ian Mercer
Ian Mercer
Cover Image for Home Automation Top Features

Home Automation Top Features

Ian Mercer
Ian Mercer
Cover Image for Weather Forecasting for Home Automation

Weather Forecasting for Home Automation

Ian Mercer
Ian Mercer

Lengthening short Urls in C#

Ian Mercer
Ian Mercer
Cover Image for How can I tell if my house is smart?

How can I tell if my house is smart?

Ian Mercer
Ian Mercer

Why don't you trust your build system?

Ian Mercer
Ian Mercer

ASP.NET MVC SEO - Solution Part 1

Ian Mercer
Ian Mercer

Home Automation Block Diagram

Ian Mercer
Ian Mercer

Elliott 803 - An Early Computer

Ian Mercer
Ian Mercer

Building sitemap.xml for SEO ASP.NET MVC

Ian Mercer
Ian Mercer

World's Smartest House Demonstration

Ian Mercer
Ian Mercer

Continuous Integration -> Continuous Deployment

What is "quality" in terms of a released software product or website?

Ian Mercer
Ian Mercer

Making a bootable Windows 7 USB Memory Stick

Here's how I made a bootable USB memory stick for Windows 7

Ian Mercer
Ian Mercer

Tip: getting the index in a foreeach statement

A tip on using LINQ's Select expression with an index

Ian Mercer
Ian Mercer

Future proof your home with a new conduit system?

Running conduit can be expensive but maybe you don't need one to every room

Ian Mercer
Ian Mercer

SQL Server - error: 18456, severity: 14, state: 38 - Incorrect Login

A rant about developers using the same message for different errors

Ian Mercer
Ian Mercer

WCF and the SYSTEM account

Namespace reservations and http.sys, my, oh my!

Ian Mercer
Ian Mercer

404 errors on IIS6 with ASP.NET 4 Beta 2

Ian Mercer
Ian Mercer

Mixed mode assembly errors after upgrade to .NET 4 Beta 2

Fixing this error was fairly simple

Ian Mercer
Ian Mercer

The EntityContainer name could not be determined

How to fix the exception "the entitycontainer" name could not be determined

Ian Mercer
Ian Mercer

Shortened URLs should be treated like a Codec ...

Expanding URLs would help users decide whether or not to click a link

Ian Mercer
Ian Mercer

Tagging File Systems

Isn't it time we stopped knowing which drive our file is on?

Ian Mercer
Ian Mercer

A great site for developing and testing regular expressions

Just a link to a site I found useful

Ian Mercer
Ian Mercer

Introducing Jigsaw menus

A novel UI for menus that combines a breadcrumb and a menu in one visual metaphor

Ian Mercer
Ian Mercer

Entity Framework in .NET 4

Ian Mercer
Ian Mercer

Fix for IE's overflow:hidden problem

Ian Mercer
Ian Mercer

A better Tail program for Windows

A comparison of tail programs for Windows

Ian Mercer
Ian Mercer

Measuring website browser performance

Found this great resource on website performance

Ian Mercer
Ian Mercer

Amazon Instance vs Dedicated Server comparison

Some benchmark performance for Amazon vs a dedicated server

Ian Mercer
Ian Mercer

System.Data.EntitySqlException

Hints for dealing with this exception

Ian Mercer
Ian Mercer

Agile Software Development is Like Sailing

You cannot tack too often when sailing or you get nowhere. Agile is a bit like that.

Ian Mercer
Ian Mercer

Exception Handling using Exception.Data

My latest article on CodeProject covers the lesser known Exception.Data property

Ian Mercer
Ian Mercer

Javascript error reporting

Sending client-side errors back to a server for analysis

Ian Mercer
Ian Mercer

AntiVirus Software is the Worst Software!

When your anti-virus software starts stealing your personal data, it's time to remove it!

Ian Mercer
Ian Mercer

ASP.NET Custom Validation

How to solve a problem encountered with custom validation in ASP.NET

Ian Mercer
Ian Mercer

Optimization Advice

Some advice on software optimization

Ian Mercer
Ian Mercer

Linq's missing link

LinqKit came in handy back in 2009

Ian Mercer
Ian Mercer

Google Chart API

Ian Mercer
Ian Mercer

Cache optimized scanning of pairwise combinations of values

Using space-filling curves to optimize caching

Ian Mercer
Ian Mercer

Threading and User Interfaces

A rant about how few software programs get threading right

Ian Mercer
Ian Mercer

Take out the trash!

Why Windows shutdown takes so long

Ian Mercer
Ian Mercer
Cover Image for New Home Automation Server

New Home Automation Server

Ian Mercer
Ian Mercer

Dell upgrades - a pricey way to go

Ian Mercer
Ian Mercer
Cover Image for World's Smartest House

World's Smartest House

Over 15 years of experimentation with home automation

Ian Mercer
Ian Mercer

Programming mostly C#

Ian's advice on programming

Ian Mercer
Ian Mercer
Cover Image for World's Smartest House Videos

World's Smartest House Videos

A collection of videos about my smart home efforts

Ian Mercer
Ian Mercer