Home Automation

A Quantified House – My Talk to the Seattle Quantified Self Meetup

I delivered the following presentation to a meetup of the Quantified Self group in Seattle tonight. The evening was a fascinating fusion of medicine, technology and personal improvement. My talk fell between a session on personal genome sequencing and another on measuring the effects of coffee on blood pressure.

Updated Release of the Abodit State Machine

I published a new version of the Abodit State Machine to Nuget this evening. You can find it here.

One breaking change in this version is that the state machine is now specified using three Type parameters instead of two:

public class OccupancyStateMachine : 
          StateMachine<OccupancyStateMachine, Event, BuildingArea>

The third type parameter, TContext, is a context object that can be passed in with every event occurrence or tick. This means that you don’t need to store any extraneous data in the state machine itself and can keep it as a pure representation of the state of the system.

In the example above I have an OccupancyStateMachine and the context is a BuildingArea. Each call to EventHappens now takes the event that happened and a BuildingArea object.

When you define your state machine you will need to include 4 parameters in each lambda expression.

Here, for example, is the current state machine for a BuildingArea in my home automation. It uses a hierarchy of states with two base states: Not Occupied and Occupied. It has timers for activity within a room or for occupancy within rooms that are contained by a floor. Note how it also exposes an IObservable<State> so that other objects can subscribe to state machine changes. I didn’t want to take the Rx dependency in the state machine class itself but you can see how easy it is to hook it up.

Of interest also is the way I represent occupancy as three distinct states, the extra one ‘Asleep’ represents a room that is not-occupied in the sense that there is no motion there now but there was at some point during the evening before.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Abodit.StateMachine;
using log4net;
using Abodit.Units;
using AboditUnits.Units;
using System.Reactive.Subjects;
using System.Reactive.Linq;

namespace Abodit
{
    /// <summary>
    /// An Occupancy State machine handles not occupied, occupied, asleep
    /// </summary>
    [Serializable]
    public class OccupancyStateMachine : StateMachine<OccupancyStateMachine, Event, BuildingArea>
    {
        private readonly Subject<State> watch = new Subject<State>();
        public IObservable<State> Watch { get { return watch.AsObservable(); } }

        public override void OnStateChanging(StateMachine<OccupancyStateMachine, Event, BuildingArea>.State newState, BuildingArea context)
        {
            watch.OnNext(newState);
        }

        public static readonly State Starting = AddState("Starting");

        public static readonly State NotOccupied = AddState("Not occupied",
                (m, e, s, c) => { 
                                m.CancelScheduledEvent(eTick);          // Stop the clock
                                m.IsTimerRunning = false;
                                m.IsRecentlyOccupied = false;
                                m.IsHeavilyOccupied = false;
                                m.After(new TimeSpan(hours:0, minutes:5, seconds:0), e5MinutesSinceOccupied);
                                m.After(new TimeSpan(hours:24, minutes:0, seconds:0), e24hoursSinceOccupied);
                                m.After(new TimeSpan(hours:48, minutes:0, seconds:0), e48hoursSinceOccupied);
                             },
                (m, e, s, c) => { });

        public static readonly State NotOccupiedIn5Minutes = AddState("Not occupied in over 5 minutes",
                (m, e, s, c) => { },
                (m, e, s, c) => { }, NotOccupied);

        public static readonly State NotOccupiedInOver24Hours = AddState("Not occupied in over 24 hours",
                (m, e, s, c) => { },
                (m, e, s, c) => { }, NotOccupiedIn5Minutes);

        public static readonly State NotOccupiedInOver48Hours = AddState("Not occupied in over 48 hours",
                (m, e, s, c) => { },
                (m, e, s, c) => { }, NotOccupiedInOver24Hours);

        public static readonly State NotOccupiedInOver1Week = AddState("Not occupied in over 1 week",
                (m, e, s, c) => { },
                (m, e, s, c) => { }, NotOccupiedInOver48Hours);

        public static readonly State Asleep = AddState("Asleep",
                (m, e, s, c) =>
                {
                    // Set a timer going for morning
                    var now = TimeProvider.Current.Now.LocalDateTime;
                    var morning = now.Hour < 8 ? now.AddHours(-now.Hour + 8) : now.AddHours(24 - now.Hour + 8);
                    m.At(morning.ToUniversalTime(), eMorning);
                },
                (m, e, s, c) => { },
                parent:NotOccupied);

        public static readonly State Occupied = AddState("Occupied",
                (m, e, s, c) =>
                {
                    m.IsRecentlyOccupied = true;
                    // Add a timer that runs while we are occupied
                    m.Every(new TimeSpan(hours:0, minutes:0, seconds:10), eTick);
                    // And set a timer going to mark 5 minutes since occupied
                    m.After(new TimeSpan(hours:0, minutes:5, seconds:0), e5MinutesAfterBecomingOccupied);
                    m.CancelScheduledEvent(e5MinutesSinceOccupied);
                    m.CancelScheduledEvent(e24hoursSinceOccupied);
                    m.CancelScheduledEvent(e48hoursSinceOccupied);
                },
                (m, e, s, c) => { });

        public static readonly State HeavilyOccupied = AddState("Heavily occupied",
                (m, e, s, c) => { },
                (m, e, s, c) => { },
                parent:Occupied);

        private static readonly Event eStart = new Event("Starts");
        private static readonly Event eUserActivity = new Event("User activity");
        private static readonly Event eTick = new Event("Tick");
        private static readonly Event eTimeout = new Event("Timeout");
        private static readonly Event eMorning = new Event("Morning");
        private static readonly Event e5MinutesAfterBecomingOccupied = new Event("5 minutes after becoming occupied");
        private static readonly Event e5MinutesSinceOccupied = new Event("5 minutes since occupied");
        private static readonly Event e24hoursSinceOccupied = new Event("24 hours since occupied");
        private static readonly Event e48hoursSinceOccupied = new Event("48 hours since occupied");

        private static readonly Event eAllChildrenNotOccupied = new Event("No child occupied");
        private static readonly Event eAtLeastOneChildOccupied = new Event("At least one child occupied");

        private double decliningActivity = 0.0;         // Up 1000 every UserInput, down x0.9 every n seconds
        private const int ActivityPerUserInput = 1000;
        private const double rateOfDecline = 0.92;

        public bool IsTimerRunning { get; set; }
        public bool IsRecentlyOccupied { get; set; }
        public bool IsHeavilyOccupied { get; set; }

        static OccupancyStateMachine()
        {
            // On startup we transition immediately to starting
            // but we want an event call to do this so we aren't doing any work
            // in the constructor, and so the initialization only happens when it's
            // a true 'cold start' not a 'warm start' from some database state
            Starting
                .When(eStart, (m, s, e, c) => { return NotOccupied; });

            // Note: This is a hierarchical state machine so NotOccupied includes Asleep
            NotOccupied
                .When(eAtLeastOneChildOccupied, (m, s, e, c) => 
                {
                    return Occupied;
                })
                .When(e5MinutesSinceOccupied, (m, s, e, c) =>
                {
                    // Could signal something??
                    return s;
                })
                .When(e24hoursSinceOccupied, (m, s, e, c) =>
                {
                    // Could signal something??
                    return s;
                })
                .When(e48hoursSinceOccupied, (m, s, e, c) =>
                {
                    // Could signal something??
                    return s;
                })
                .When(eUserActivity, (m, s, e, c) =>
                {
                    m.After(c.OccupancyTimeout, eTimeout);                // start a new timeout
                    m.IsTimerRunning = true;
                    return Occupied;
                });

            // Asleep is a substate of not occupied so no need for more logic on becoming occupied ...
            Asleep
                .When(eMorning, (m, s, e, c) =>
                {
                    // Eliminate Asleep if appropriate
                    return NotOccupied;
                });

            // Occupied includes recently occupied and heavily occupied ...
            Occupied
                .When(e5MinutesAfterBecomingOccupied, (m, s, e, c) => 
                {
                    m.IsRecentlyOccupied = false;
                    return s;
                })
                .When(eUserActivity, (m, s, e, c) =>
                {
                    // Accumulate activity ...
                    m.decliningActivity += ActivityPerUserInput;

                    m.CancelScheduledEvent(eTimeout);               // cancel the old timeout

                    m.After(c.OccupancyTimeout, eTimeout);                // start a new timeout
                    m.IsTimerRunning = true;

                    if (m.decliningActivity > 20 * ActivityPerUserInput)
                        return HeavilyOccupied;
                    else
                        return s;
                })
                .When(eAllChildrenNotOccupied, (m, s, e, c) =>
                    {
                        if (m.IsTimerRunning)
                        {
                            // If the timer is running ... wait until it runs out
                            return s;
                        }
                        else
                        {
                            DateTime nowLocal = TimeProvider.Current.Now.LocalDateTime;
                            if (nowLocal.Hour > 17)
                                return Asleep;
                            else
                                return NotOccupied;
                        }
                    })
                .When(eTick, (m, s, e, c) =>
                    {
                        m.decliningActivity *= rateOfDecline;
                        return s;
                    })
                .When(eTimeout, (m, s, e, c) =>
                    {
                        DateTime nowLocal = TimeProvider.Current.Now.LocalDateTime;
                        if (nowLocal.Hour > 17)
                            return Asleep;
                        else
                            return NotOccupied;
                    });

            HeavilyOccupied.When(eTick, (m, s, e, c) =>
            {
                // Same code as Occupied but this one will override if we are in HeavilyOccupied mode
                m.decliningActivity *= rateOfDecline;
                // Fall back to just occupied when ...
                if (m.decliningActivity < 0.2 * ActivityPerUserInput)
                    return Occupied;
                else
                    return s;

            });


        }

        public OccupancyStateMachine()
            : base(Starting)
        {
        }

        public OccupancyStateMachine(State initialState)
            : base(initialState)
        {
        }

        public override void Start()
        {
            this.EventHappens(eStart, null);
        }

        public void UserActivity(BuildingArea ba)
        {
            this.EventHappens(eUserActivity, ba);
        }

        public void AllChildrenNotOccupied(BuildingArea ba)
        {
            this.EventHappens(eAllChildrenNotOccupied, ba);
        }

        public void AtLeastOneChildOccupied(BuildingArea ba)
        {
            this.EventHappens(eAtLeastOneChildOccupied, ba);
        }
    }
}

Integrating an Android phone into my home automation system

My home automation system has long had the capability to communicate with a Panasonic PBX. From the PBX it gets a flow of information about every phone call in or out the house. It also has a couple of caller-ID-to-serial-port devices that give it an earlier notification of incoming calls with Caller ID information. Using these various inputs it builds a database of every caller and every call (time of day, duration, person).

If it sees a call from someone not already in the database it will ask (via chat) for you to enter their full name. It then updates its database replacing the caller ID name (which is often useless, especially for mobile numbers). You can also query it using the natural language interface to ask about any calls you might have missed, or to lookup a number by time of day or by a fragment of their name. You can even ask complex queries like “who called last year on a friday after 5pm” and it will construct an efficient SQL query to get the results.

It also synchronizes all these contact records with Google Contacts.

But until recently my mobile phone hasn’t been part of the home automation system. Yes, I can use it as an input device for Google Talk, and yes, the house still notices when it comes and goes because the house tracks every device that ever gets an IP address on the local network, but other than that it really doesn’t ‘understand’ much about my cell phone.


But that’s about to change. Recently I installed Tasker on my Android phone and using that app I can now set up a whole variety of triggers that can report back to the home automation system information such as phone calls made or received, GPS location, wifi-located position, phone unlocks, shakes and more.

So I’ve extended the web interface on the Home Automation web server to accept POSTs from Tasker with updates from my cell phone. These are placed into a PubSubHub implementation that uses SignalR to distribute messages to any connected clients. The home automation server is itself a client of this service (it publishes information about every device change in the house and the PubSubHub shares those updates with any connected Web client) so you get real-time updates for what’s happening in the house on the house’s web page. Extending that architecture to include messages from remote devices like the Android phone was easy and I plan to use it in the future for other remote devices, such as a Netduino with a collection of environment and HVAC sensors on it (more about that later).

As to precisely what I’ll do with this new capability I have a long list of features to implement now:

1) Logging all cell phone calls to the same database, automatically building my contacts list
2) Tracking how long it takes to get to work by each of the various ways I can go, correlating that with the traffic flow information and automatically figuring out which route I should take for future trips
3) Shake cellphone to change music in the house
4) Adjusting the heating at home based on how far away we are (and thus the soonest we could get back)
5) Finishing up my semantic, location-aware shopping list (knows which store you are in and what you need there and presents it in order by aisle)
6) Automatically delivering notifications by the best possible means (talking on the speakers at home, XMPP, or by email if I’m in a different time zone)
etc.

The Internet of Dogs

In my previous post about GreenGoose I described my initial experiences with this “Internet Of Things in a box” product. Recently I’ve been trying their API and have integrated it into my Home Automation System.

Click image to see it all.

The initial integration was easy, I used the new ASP.NET WebApi Core Libraries (from Nuget) together with Newtonsoft Json.Net. GreenGoose’s datetime format is somewhat quirky but hopefully they’ll move to a more standard one soon. They are, however, also about to switch to OAuth so it’s going to require some more work when that happens.

Aside from a few simple WebAPI calls and some Json parsing the rest was just a matter of connecting up the appropriate TimeSeries classes that I use to track values that vary over time, declaring a few graphs, and deciding what to log. With that in place I can now spin up a home automation ‘sensor’ corresponding to any GreenGoose sensor Id and my home automation system will add all of the relevant graphs and charts, triggers and more for that device.

What’s interesting is that a single sensor potentially serves a couple of different purposes. The dog collar sensor for example polls regularly back to the base station so it can potentially be used to sense both how much exercise the dog has had but also simply whether the dog is at home or not which could be really handy for anyone with a dog that’s learned to ignore the invisible fence! Each sensor can, through the TimeSeries objects also offer additional data and triggers that can be used elsewhere in the home, for example, an alert if the dog was walked less than half and hour each day.

GreenGoose Review

GreenGoose Hardware
One of the conclusions I’ve reached from my many years experimenting with home automation is that one of the missing ingredients is affordable, wireless sensors with long battery life. One of the best solutions I’ve found so far is to use CADDX alarm sensors because they are wireless, have exceptionally long battery life and can be connected to a PC using a reliable serial connection. Proponents of the Internet of Things talk about every device having an IP address and recently some wifi-connected sensors have appeared. I’m skeptical of that approach: it seems unlikely however that they will get the kind of battery life you need to be practical and you really don’t need a separate IP address for every single sensor.

What I believe we need is a hub device that connects to the internet and then really cheap wireless sensors that communicate back through it using a connection technology that requires much less power than wifi. So when GreenGoose recently announced their sensors I was one of the first to place a pre-order and this week I received one of their first device shipments. This blog post discusses some of my initial impressions.

First off I have to say I don’t like the egg shape or the color of the hub. Fortunately it’s in the wiring closet next to the router so I don’t need to see it.

GreenGoose Mounting on food containerThe sensors themselves are remarkably small considering they claim a battery life of 1.5 years and the fact that they include both a sensor, some intelligence and a wireless transmitter. Two of the sensors were stick-on devices which I stuck on the dog and cat food containers respectively even though one was labelled food and the other treats. The other two were shaped like dog tags. Here you can see one of the stick-on sensors mounted inside the lid of a dog food container.  You can view a teardown of the GreenGoose sensors and hub on this Flickr page.

Connecting the sensors up to my account took two attempts, three sensors were found on the first try and the remaining sensor on the second try. That’s apparently fairly normal. There were a few configuration problems with my account and the iPad software (actually iPhone software in 2x mode) and the web site is clearly still under development with daily bug fixes and improvements happening each time I try it. But, I have to say the folks at GreenGoose have been incredibly responsive to my every comment or bug report.

GreenGoose User Interface
So now I’m getting reports from the sensors on their web site and overall things seem to be working quite well. It’s certainly not 100% reliable yet in terms of capturing every event so I don’t think these are going to be the perfect home automation sensor that I’m still waiting and hoping for, but, in all fairness, the folks at GreenGoose aren’t selling these sensors as alarm panel sensors, but as more of a fun game tracking activity and reminding people to go walk the dog. I’ll update this blog when I have a better idea exactly how accurate they are at capturing activity.

The biggest gotcha however is that my original plan to use the sensors on different devices has been somewhat thwarted by the realization that each different sensor type is calibrated to detect a specific kind of motion. You cannot re-purpose them like you would a normal motion or contact sensor. So putting the dog collar sensor on my key chain isn’t producing the results I’d expected. We will have to wait for GreenGoose to come out with sensors to put on doors, cars, shoes, cellphones, iPads and whatever else we want to track. I’ve suggested to GreenGoose that they explain this better on their web site as many hobbyists will try to repurpose the sensors for applications other than the ones for which they were designed.

Another issue at the moment is that the web site forces you to assign the sensors to a specific pet and they only support one pet name. They say they’ll add multiple-pet support soon.

Next I plan to try their API and will report back on how that goes.

Bottom line: this is remarkable technology – a mems accelerometer, some computing power, a radio transmitter and a 1.5 year battery in a package the size of a stick of gum is quite a feat of engineering. Some of the software is a bit ‘immature’ but it’s improving daily. Overall this is a very affordable way to try out some ‘quantified self’ ideas for yourself, it might help you walk the dog more often (our dogs wouldn’t let you forget to do that, but maybe yours would!), and the support from GreenGoose has been great.

A traffic service that answers “which way should I go?”

Which way should I go?  Traffic

Most traffic reports (on the radio or in text message alerts) are fairly useless. Like weather reports they contain lots of irrelevant information that could be eliminated with just a bit of extra context. In fact, most of the information they deliver is completely irrelevant to you as an individual located in one spot and hoping to get to another spot. Furthermore they aren’t actionable – telling me the traffic is slow on SR-520 and on I-90 isn’t interesting unless you can tell me which is the best way to go given where I am now and where I want to be.

So this weekend I added a new feature to the home automation that uses the WSDOT’s excellent traffic feed API to calculate a traffic report just for me. Recently I’ve started driving from the north end of Bellevue to the south end of Sammamish during rush hour. There are two very different paths I can take: SR-520 or I-405 to I-90. If either route has a problem I should take the other. So now I get an XMPP (chat) message from 4PM to 6PM whenever the optimal path changes from one route to the other. It’s the absolute minimum information I need and it’s 100% actionable.

For the moment the calculation is fairly simple, I simply maintain a list of the FlowDataID values along each route and then calculate a total ‘slowness’ factor based on the sum of those segments. If one way is much better than the other it generates an alert. If it goes back to being roughly equal the alert is cleared.

Since the calculation is purely relative (route A vs route B) it’s also fairly immune to day-of-week / school-holidays and other factors that have a significant impact on traffic but no impact on the only actionable decision I need to consider.

One other interesting point from the graph is just how spiked the traffic is on SR-520 compared to I-90.

Programming a smart home with a fluent, domain-specific language

In response to a question I received recently, here is an example of the fluent extensions to C# that my home automation system uses to define how it should behave. In effect this is a domain specific language for modeling home automation behavior as a finite state machine. Note how you can use either a purely declarative sequence or you can use procedural code within a Do() block.

            Living.FloorSensor
                .Then(Entrance.HallBeam, 30)
                .Provided(Time.Bedtime)
                .ProvidedNot(Home.DinnerGuests)
                .PulseStretch(10 * 60)      // don't announce it too often - every 10 minutes
                .Do("Garage doors warning", () =>
            {
                if (Garage.GarageDoors.AreAnyOpen)
                {
                    FirstFloor.Kitchen.AllMediaZones.AnnounceString("Excuse me, I think you may want to close the garage doors.");
                }
            });

In many ways this is similar to the Reactive Framework from Microsoft except my work on this predates the availability of Reactive Framework and unlike the Reactive Framework my ‘sequential logic blocks’ include persistence logic so the entire state can be saved to disk (as it is every second). This is important because some transitions in the state machine might be several hours or even days long and you need to be able to restart the system and carry on exactly where you left off.

One key benefit of the declarative approach over the procedural approach is that the declarative approach can explain itself. So, the log entry for a light going on can show that the light was turned on because ‘it was dark’ and ‘we had visitors’ and ‘someone came into the room’. Compared to traditional home automation systems where you either have no logging at all or you have a log of what happened, this kind of logging is invaluable for figuring out what went wrong when the logic gets complicated. So in this example I should have moved the test for Garage.GarageDoors.AreAnyOpen out to a Provided clause which would allow it to be part of the reverse-chain logic explanation.

Partial results can be captured at any point in these logic chains and then reused in other chains because each partial result is itself a Sensor device that implements the full fluent interface.

Ultimately I plan to hook the logging for what happened back up to the NLP engine which will allow users to ask the home ‘Why did you put the driveway light on last night around 9pm?’ and ultimately I plan to allow the logic itself to be defined using natural language.

Home power meters revisited

In an earlier post I discussed the utility (or otherwise) of the 24 hour power consumption graph and questioned why Google and Microsoft were both investing in this approach to home energy efficiency. Since then both Google and Microsoft have stopped their efforts in this area.


Interestingly, in Europe I’m seeing more and more homes with devices like the one shown here that provide real-time power consumption information. One of the more interesting uses for devices like these is as a check that everything has been turned off when a homeowner is about to leave the house. A quick glance at the meter can reveal if a heater has been left on in a bedroom. Of course the main water heater has the largest impact on the reading but homeowners learn what numbers represent ‘normal’ and can see at a glance when something else has been left on. Clearly a true smart home that can turn devices off when they are no longer in use is still a better long-term solution for this scenario but it’s interesting to see how a fairly simple device can at least provide an indication that everything is off without a significant investment in replacing light switches and device controllers. What would be nicer however would be if the meter included some kind of machine learning so it could show at a glance if the home is in a minimal power state or not.

Home Automation Calendar Integration

Calendar Home Automation Integration One feature of my home automation system is Google Calendar integration. What this enables is two things: (i) the house can record what happened on the calendar so I can see at a glance what’s been going on back home and (ii) the ability to put events on the calendar for the house to do certain things.

For example, we recently bought some new baby chicks and they need to be kept warm at light. An infra-red lamp in the chicken house is connected to an X-10 controlled outlet (behind a GFCI, of course since this is outdoors). The device that controls that outlet is called “Chicken lights” but also has several shorter aliases (e.g. “Chickens”). Using the home’s natural language engine that outlet can be controlled remotely by the chat interface, by the email interface, by the voice interface or by the calendar interface.

In this case, a simple recurring event on the calendar to turn the light on every night for 9 hours is sufficient and I can set the recurring event to stop after a few weeks because by then the chicks will no longer need it.

This complete integration of Calendar and Natural Language Engine functionality certainly makes for a very easy way to control and monitor a smart home!

Net result … happy chicks:

Smart home tracks network devices by mac address

Smart home tracks network devices

One surprisingly useful component of my home automation system is its ability to track every device that ever connects to the router at home by mac address.

Every few minutes my smart home scans the local subnet looking for connected devices. It does this by pinging each IP address in the local address range and whenever it finds a device it gets its mac address and compares it to a list of known mac addresses. Once it’s found a device it pings it more frequently to check that it’s still connected to the network.

Whenever it finds a new device (for example, a friend visiting with a cell phone or laptop that connect to Wifi) it can ask for that device to be identified (using the chat interface) and can track its comings and goings from then on. If you happen to enter the house with a smart phone with Wifi turned on you have just become an unwitting part of my experiment and your phone’s mac address will be tracked from here on out.

Many consumer electronic devices now connect to the internet and each is thus tracked by the home automation system to determine how long it has been on for each day. This is used to calculate the instantaneous power consumption of the home along with the lights and other devices that the home automation can control or monitor.

Aside from estimating power consumption and tracking visitors to the house, one of the most useful aspects of this feature is that it’s great for finding lost devices. Did your child lose their iPod touch at home or is it at a friend’s house? A quick look at the log shows when it was last seen in the house confirming that it’s still inside somewhere. Ultimately it might be able to cross-reference which rooms were occupied with when the device was last seen and thus give you an idea which couch or bedroom to go search (future feature), but for now, at least you know you aren’t searching in vain. Ideally I’d have three access points with firmware that can track the signal strength of each device in the home so I can locate it more precisely but I haven’t found a router offering that capability yet. My laptop can track and report how far away from the router it is but that’s not as useful as being able to track individual devices from the router.

Another possible use for this is tracking screen time – how many hours did your kids spend watching TV or playing on the XBox when they were in fact meant to be working on their homework? etc. etc. Of course at some point this may all become a bit too Orwellian but as I’ve mentioned before, part of the experiment I’ve been conducting with my home automation is ‘technical’ but the larger portion is ‘social’ – which features are genuinely useful and which are just too much.