VariableWithHistory - making persistence invisible, making history visible

In a typical .NET application variables have a short lifetime. When they go out of scope or the application ends their value is lost. Also, you cannot ask a variable what its value was 1 hour ago, or what its average, maximum or minimum value was yesterday.

Yet, such a variable would be extremely useful when writing a Home Automation System because you often need to make comparisons between a current value and some historical average, or between two ranges (e.g. was the kitchen more or less occupied than yesterday). Now, normally you wouldn’t want to mix persistence up with the representation of a value in your code (see ‘Separation of Concerns’), but in this case I decided that it was worth mixing the two concepts because the benefits of doing so were so great.

So I created a class called VariableWithHistory\ which is the abstract base class for IntegerWithHistory, DoubleWithHistory, BoolWithHistory, StringWithHistory and a number of others. The first property worth noting on these classes is the .Current property. This always gives you the latest value that has been set. Setting the .Current value stores both the value and the DateTime (Utc of course) at which the value became current. A history of all past values is maintained in MongoDB up to some suitable limit per variable (each variable can have its own adjustable history size in bytes by using MongoDB’s capped collections). If the new value is the same as the old one no update is made, the implicit behavior being that the value changed and stayed there until it changes again, so if you want to know what the value is now it is the same as the last change recorded.

With this new variable type in place any object in the house can have any number of persistent fields on it (bool occupied, double temperature, string triggeredBy, …). Updating these values is as simple as assigning to their .Current property. When the system loads, each value comes back with the value it had when the system was shut down. To accomplish this every VariableWithHistory is given a unique id (based on the unique id of it’s parent, e.g. a room).

So far so good, shut down, restart and the house doesn’t need to query a device to know if it’s on or off and all the long running Sequential Logic Blocks I use for rules (e.g. .Delay(days:2)) carry on running as if nothing happened. This is particularly useful since I typically deploy a new version almost every day and some logic blocks have long delays built into them.

But besides providing simple recovery from a reboot, these persistent variables allow me to do some much more interesting things.

int CountTransitions(DateTimeRange range, T direction);Counts how many transitions there have been to the value T in a given time range, e.g. how many times did the driveway alarm go ‘true’ this evening?

Dictionary\ Fractional(DateTimeRange range);Builds a histogram of all the values seen in the time range, e.g. 50% hot, 20% cold, 30% warm for a string variable that tracks temperature

DateTimeOffset LastChangedStatee.g. when was this sensor last triggered?

TimedValue\ ValueAtTime(DateTimeOffset dt)What was the value at a given time in the past, e.g. what was the temperature at the same time yesterday?

Each specific type of VariableWithHistory\ may also have additional methods relevant to the type T. For example, onDoubleWithHistorythere is a methoddouble Average(DateTimeOffset minValue, DateTimeOffset maxValue)which gets the average value over the specified time range. On BoolWithHistory there is a method double PercentageTrue(DateTimeRange range) which you could use to find the average occupancy for a room yesterday.

My initial implementation waited for the database to write each update before allowing any queries but now I simply cache the Current value and assume that queries will probably get executed after updates and that the average temperature yesterday is close enough with or without the last 100ms of updates. I did try to keep this class isolated from MongoDB but in the end the benefit of some of the atomic update capabilities in MongoDB made it easier to just take the dependency.

My previous implementation of this feature used my own in-memory database, MongoDB has slowed it down a bit but I’ve gained the ability to archive terabytes of sensor data which should prove useful for my next project which is to add some machine learning to the system.

Mon Feb 04 2013 06:00:24 GMT-0800 (Pacific Standard Time)

Next page: Dynamically building 'Or' Expressions in LINQ

Previous page: Neo4j Meetup in Seattle - some observations