My News
Understanding Dates and Times in Natural Language
Dec 27th
One of the more challenging aspects of understanding natural language is dealing with date and time expressions. There are many different ways a user could refer to a specific date and time. They might say “Next Tuesday at 4pm”, they might give a specific date in any of several different forms, they might refer to other time ranges “First Tuesday in January 2012 at 4pm” etc.
Whilst my natural language engine can’t understand every possible date time expression (e.g. the second Wednesday after the first Friday in May 2010) it does handle a huge variety.
Clearly the .NET provided DateTime class is wholly inadequate to express the kinds of date/time expressions your users might enter. To deal with that I’ve created my own classes that represent items like a specific Time of day, a DateTimeRange, a DateTimeRangeCollection, …
TemporalSets are the most general result of parsing a datetime expression since they can represent any date time expression. Broadly they split into two categories: finite and infinite expressions. “Tuesday at 5pm” is an infinite time expression. “Tuesday at 5pm January 2012″ is a finite time expression. Sometimes you will want to accept an infinite expression and interpret it as a future or past finite occurrence. For that I have a MergePreferPast and MergePreferFuture method that operate on a TemporalSetCollection. The demonstration code on BitBucket shows this in action.
TemporalSets also have unique capabilities around both providing query expressions (for database searches) and generative expressions (for adding dates to a calendar).
If you’d like to try out the latest date / time expression parsing code in my Natural Language Engine you can visit the demo and try typing “define” followed by a date / time expression.
If you find any expressions that ought to work, please feel free to email or Tweet them to me.
Here’s a sample session:
define june 23rd 2010 Absolute:[DATETIMERANGE 6/23/2010 at 12:00 AM to 6/23/2010 at 11:59 PM] define January 19th Future:[Thursday 1/19/2012], [Saturday 1/19/2013], [Sunday 1/19/2014], [Monday 1/19/2015], [Tuesday 1/19/2016], [Thursday 1/19/2017], [Friday 1/19/2018], [Saturday 1/19/2019], [Sunday 1/19/2020], [Tuesday 1/19/2021] Past: [Wednesday 1/19/2011] ...
A traffic service that answers “which way should I go?”
Oct 17th

Most traffic reports (on the radio or in text message alerts) are fairly useless. Like weather reports they contain lots of irrelevant information that could be eliminated with just a bit of extra context. In fact, most of the information they deliver is completely irrelevant to you as an individual located in one spot and hoping to get to another spot. Furthermore they aren’t actionable – telling me the traffic is slow on SR-520 and on I-90 isn’t interesting unless you can tell me which is the best way to go given where I am now and where I want to be.
So this weekend I added a new feature to the home automation that uses the WSDOT’s excellent traffic feed API to calculate a traffic report just for me. Recently I’ve started driving from the north end of Bellevue to the south end of Sammamish during rush hour. There are two very different paths I can take: SR-520 or I-405 to I-90. If either route has a problem I should take the other. So now I get an XMPP (chat) message from 4PM to 6PM whenever the optimal path changes from one route to the other. It’s the absolute minimum information I need and it’s 100% actionable.
For the moment the calculation is fairly simple, I simply maintain a list of the FlowDataID values along each route and then calculate a total ‘slowness’ factor based on the sum of those segments. If one way is much better than the other it generates an alert. If it goes back to being roughly equal the alert is cleared.
Since the calculation is purely relative (route A vs route B) it’s also fairly immune to day-of-week / school-holidays and other factors that have a significant impact on traffic but no impact on the only actionable decision I need to consider.
One other interesting point from the graph is just how spiked the traffic is on SR-520 compared to I-90.
“Remember Everything” … a long-term project
Sep 15th
“Remember Everything” connects nearly all of my projects into one giant solution that, well, remembers everything and has a natural language interface over it.
As inputs it will take information from my home automation system, my whole-network storage crawler, Google calendar, email, Twitter, blog, web crawler, an address-monitoring browser add-on I plan to write, the weather and traffic feeds, and, of course my natural language engine.
All this data will be put into MongoDB and can then be queried. Relationships between entities will be created using a semantic-web triple store and reasoner.
Together these capabilities will allow queries like:-
* Copy all the photos I took last week onto c:\vacationPhotos
* Send img_0938.jpg to mum.
* Who called last Monday?
* Show pictures from last month taken on sunny days.
* What was happening two weeks ago when X called?
* Who called yesterday when I was in a meeting?
* What song was playing around 9pm last night?
* How long did I spend on the phone to my accountant last week?
* What web pages did I read last week about the Semantic Web?
* Send the web page I tweeted about last night to my Kindle.
* We need butter and olives.
* What do I need to buy from QFC? (a semantic shopping list concept, more on that later …)
In addition to the shopping lists concept (that’s already in my home automation system but lacks the semantic reasoning) the system will take any subject-verb-object phrase and remember it and then allow you to query it back later, e.g.
* My son read 20 pages tonight (making the weekly reading report easier)
* How many pages did he read this week?
* I took the red pill at 10AM
* I walked 2 miles this morning
* I ran 4 miles
* How much exercise did I do this week when it wasn’t raining? (summarizing values semantically and mathematically)
* The Audi was serviced this week (remembering schedules so you can check if an item is overdue)
* My BA frequent flyer number is #### (remembering numbers you need to look up often)
* I took the day off on friday (vacation reporting)
* I spent $12.95 on lunch (expense reporting)
…
Whenever you have anything you need to remember the system will be able to remember it, recall it, and where possible aggregate or summarize it using math and/or semantic reasoning (e.g. running subClassOf exercise, butter subClassOf dairy product, dairy products areSoldAt QFC, …).
By linking my natural language engine to a triple store I can even allow users to teach it new concepts:
By silently monitoring your email, Twitter stream, calendar, activity in the house, … it will be able to answer questions based on the context not just on the content in ways that we take for granted as humans but which are not possible for computers today.
Home network crawler – cataloging every file on the home LAN with C# and MongoDB
Aug 22nd

Map-Reduce in action: The glaciers in Greenland 'map' the canyon walls into streams of rocks called lateral moraine. As the glaciers merge these rocks are 'reduced' into streams in the middle called 'medial' moraine. (A photo I took over Greenland this summer.)
I’m not a huge fan or RAID arrays – they mostly mean there’s another component to go wrong (the controller card) and when they do go wrong you can lose all your data just as easily as if it were all on one drive. I prefer a multiple copy strategy, an “Amazon S3 for the home” if you like. The downside of this is that there are multiple copies of each file across the home network and as I have several generations of hard drives the mapping from primary to secondary to tertiary is complex and hard to manage! It’s also really hard to find a single file when there are so many places to look and it’s nigh on impossible to be sure that I have the necessary three copies of every important file in the right places at all times.
So this weekend I embarked on a small project to catalog every file, directory and storage volume on the entire home network including drives that are only sometimes connected. The software has been running all weekend and is close to cataloging everything. It’s found 5 million files so far representing over 6TB of data!
The architecture I chose for this software was an agent that runs on each PC to catalog all of the attached volumes. This client uploads all the directories and files that it finds to a MongoDB database running on the same Atom server as the main storage array. The poor little Atom server’s 4GB of RAM has been in constant use but the server has remained responsive, in part because it boots from an SSD drive.
Each volume, directory and file is represented by a document in MongoDB in a single collection. The agent calculates an MD5 hash for each file and extracts metadata from MP3, WMA and JPG files. It also stores all of the key file dates (created, updated, accessed) and references to parent directories, volume identifiers and the currently connected PC. It does not assume that a volume is always connected to the same computer – you can unplug an external drive from one and put it somewhere else and it will all work just fine.
I implemented a re-startable tree scan that uses a couple of DateTime stamps to be able to determine which directories need to be scanned during the current pass and which ones have already been scanned. Any agent can be killed at any time and restarted and it will carry on walking the directory tree right where it left off. It will even continue correctly in the case where you move a volume from one PC to another.
Each agent uses the Parallel Task library’s Parallel.ForEach to crawl each volume in parallel and to parse multiple files from each directory simultaneously.
By storing all of the file metadata in Mongo DB it’s easy to use Map-Reduce to calculate some interesting statistics for the files on the network.
For example, to create a summary of file sizes I can use a Map function:
function Map() {
if (this.Size && this._t == "FileInformation")
{
var size = this.Size;
if (size < 1024)
emit ("kb", {count:1, size:this.Size});
else if (size < 1024*1024)
emit ("mb", {count:1, size:this.Size});
else if (size < 1024*1024*1024)
emit ("gb", {count:1, size:this.Size});
else if (size < 1024*1024*1024*1024)
emit ("tb", {count:1, size:this.Size});
else
emit ("tb+", {count:1, size:this.Size});
}
}
and a reduce function:
function Reduce(key, arr_values) {
var count = 0;
var size = 0;
for(var i in arr_values)
{
count = count + arr_values[i].count;
size = size + arr_values[i].size;
}
return {count:count, size:size};
}
Map-Reduce operations like this take about 20 minutes to run (on the Atom server with just 4GB of RAM) whereas any query serviced by one of the indexes on the MongoDB collection is almost instantaneous.
I’ve been using the excellent MongoVue to run simple map-reduce scripts like this and to keep track of how quickly the database is growing.
Map-reduce can also be used to find duplicate files – by emitting the MD5 hash as the key and some information about the file as the value I can find every copy of every file across every computer on the home network.
Since I have the file name and metadata for every file on the home network I can also easily find any file using MongoDB’s regex matching feature against the path.
The Hard Parts
For starters you’ll need a library that can handle long file names. Then you’ll need to fix it to provide at least the functionality that FileInfo and DirectoryInfo give you in .NET.
Next you’ll need to learn about reparse-points and hard-links and you’ll need to skip over them because with them in place the file system is not a tree; it’s a cyclical graph in which a simple crawler will quickly get confused or stuck.
You’ll also want to store the NTFS file Id and the unique Volume ID for every file so you can track it when the file is moved or the removable drive is connected to a different computer.
So how well does it work?
This all seems to work really well. Nearly every volume has now been cataloged. It’s located about 5M files occupying over 6TB of space. The worst case offender for the number of copies of the same file is 100+. I’ve used the find feature in MongoDB to find a file I was missing and I’m better able to plan how to arrange directories and file generations across the various hard drives I have.
What’s next
Well, of course this needs to be connected to the home automation system and my Natural Language engine so you can ask “send a copy of IMG_0228 from last week to X” or “where are all the spreadsheets I created last year?” That will be fairly easy.
After that I hope to incorporate backup features into the agents too so they can automatically keep the required number of copies of each file according to its importance. I’d also like to set up a rotating set of external drives that go in the fire safe when not connected and when they are connected they get updated with the latest copies of all the important files.
I’d also like to be able to get the agents to move whole groups of directories around between drives as juggling the directory layout each time a new hard drive is added to the system is always a time consuming process.
Comments or Questions?
Does everyone else have a hard time managing multiple computers, hard drives, directories and multiple copies of files? What tools do you use to do this? Is there anything commercially available that I could have used instead? Would a tool like this be useful to you? Should I publish the code somewhere? Comments and questions are always welcome here or on twitter.
C# Natural Language Engine connected to Microsoft Dynamics CRM 2011 Online
Jun 5th
In an earlier post I discussed some ideas around a Semantic CRM.
Recently I’ve been doing some clean up work on my C# Natural Language Engine and decided to do a quick test connecting it to a real CRM. As you may know from reading my blog, this natural language engine is already heavily used in my home automation system to control lights, sprinklers, HVAC, music and more and to query caller ID logs and other information.
I recently refactored it to use the Autofac dependency injection framework and in the process realized just how close my NLP engine is to ASP.NET MVC 3 in its basic structure and philosophy! To use it you create Controller classes and put action methods in them. Those controller classes use Autofac to get all of the dependencies they may need (services like an email service, a repository, a user service, an HTML email formattting service, …) and then the methods in them represents a specific sentence parse using the various token types that the NLP engine supports. Unlike ASP.NET MVC3 there is no Route registration; the method itself represents the route (i.e. sentence structure) that it used to decide which method to call. Internally my NLP engine has its own code to match incoming words and phrases to tokens and then on to the action methods. In a sense the engine itself is one big dependency injection framework working against the action methods. I sometimes wish ASP.NET MVC 3 had the same route-registration-free approach to designing web applications (but also appreciate all the reasons why it doesn’t).
Another improvement I made recently to the NLP Engine was to develop a connector for the Twilio SMS service. This means that my home automation system can now accept SMS messages as well as all the other communication formats it supports: email, web chat, XMPP chat and direct URL commands. My Twilio connector to NLP supports message splitting and batching so it will buffer up outgoing messages to reach the limit of a single SMS and will send that. This lowers SMS charges and also allows responses that are longer than a single SMS message.
Using this new, improved version of my Natural Language Engine I decided to try connecting it to a CRM. I chose Microsoft Dynamics CRM 2011 and elected to use the strongly-typed, early-bound objects that you can generate for any instance of the CRM service. I added some simple sentences in an NLPRules project that allow you to tell it who you met, and to input some of their details. Unlike a traditional forms-based approach the user can decide what information to enter and what order to enter it in. The Natural Language Engine supports the concept of a conversation and can remember what you were discussing allowing a much more natural style of conversation that some simple rule-based engines and even allowing it to ask questions and get answers from the user.
Here’s a screenshot showing a sample conversation using Google Talk (XMPP/Jabber) and the resulting CRM record in Microsoft CRM 2011 Online. You could have the same conversation over SMS or email. Click to enlarge.
Based on my limited testing this looks like another promising area where a truly fluent, conversational-style natural language engine could play a significant role. Note how it understands email addresses, phone numbers and such like and in code these all become strongly typed objects. Where it really excels is in temporal expressions where it can understand things like “who called on a Saturday in May last year?” and can construct an efficient SQL query from that.
Issaquah Triathlon Photos
Jun 4th
I had the opportunity to photograph the Issaquah Triathlon today. Here are some pictures from the event:
From Issaquah Triathlon 2011, posted by Ian Mercer on 6/04/2011 (200 items)
- IMG_5463
- IMG_5458
- IMG_5437
- IMG_5433
- IMG_5420
- IMG_5407
- IMG_5406
- IMG_5352
- IMG_5348
- IMG_5347
- IMG_5331
- IMG_5305
- IMG_5266
- IMG_5186
- IMG_5160
- IMG_5137
- IMG_5134
- IMG_5123
- IMG_5495
- IMG_5491
- IMG_5488
Generated by Facebook Photo Fetcher
For more information about the Issaquah Triathlon please visit their Facebook page: http://www.facebook.com/IssaquahTri
SQL Random Order – some unexpected benefits
May 16th
In this StackOverflow answer I explained the technique used by my home automation system to generate a random ordering for songs during playback.
By using a column called ‘Shuffle’ containing random integer values it becomes easy to generate a repeatable random ordering of the records in a database table. The query to access the table is …
Random random = new Random(); int seed = random.Next(); result = result.OrderBy(s => (~(s.Shuffle & seed)) & (s.Shuffle | seed)); // ^ seed);
Which uses an XOR operation in the database and orders by the results by the result of that operation.
This approach has several advantages as I outlined in my Stackoverflow answer:
Efficient: SQL handles the ordering, no need to fetch the whole table
Repeatable: (good for testing) – can use the same random seed to generate the same random order
Works on most (all?) Entity Framework supported databases
In my home automation system I calculate a new random seed each day and maintain the same seed for the whole day. This means I can play a playlist (actually there are no playlists, only tags, but that’s another story) and reliably get back to the same point in it.
One advantage that I hadn’t considered when I first added this feature is that because the shuffle column is across all songs and because playlists are just tags on songs I can now ensure that you only hear each song once a day even if it occurs in multiple playlists! In fact, I can simply calculate the value of shuffle XOR today’s random seed and then request only songs that are greater than that value making it easy to continue any playlist from any point without having to remember any state other than a single value.
This is the approach used by my home automation system to randomize playlists. It picks a new seed each day giving a consistent order during the day (allowing easy pause / resume capabilities) but a fresh look at each playlist each new day.
Home Automation Calendar Integration
May 5th
One feature of my home automation system is Google Calendar integration. What this enables is two things: (i) the house can record what happened on the calendar so I can see at a glance what’s been going on back home and (ii) the ability to put events on the calendar for the house to do certain things.
For example, we recently bought some new baby chicks and they need to be kept warm at light. An infra-red lamp in the chicken house is connected to an X-10 controlled outlet (behind a GFCI, of course since this is outdoors). The device that controls that outlet is called “Chicken lights” but also has several shorter aliases (e.g. “Chickens”). Using the home’s natural language engine that outlet can be controlled remotely by the chat interface, by the email interface, by the voice interface or by the calendar interface.
In this case, a simple recurring event on the calendar to turn the light on every night for 9 hours is sufficient and I can set the recurring event to stop after a few weeks because by then the chicks will no longer need it.
This complete integration of Calendar and Natural Language Engine functionality certainly makes for a very easy way to control and monitor a smart home!
Net result … happy chicks:
Class-free persistence and multiple inheritance in C# with MongoDB
May 4th
Much as I appreciate Object Relational Mappers and the C# type system there’s a lot of work to do if you just want create and persist a few objects. MongoDB alleviates a lot of that work with its Bson serialization code that converts almost any object into a binary serialized object notation and provides easy round tripping with JSON.
But there’s no getting around the limitations of C# when it comes to multiple inheritance. You can use interfaces to get most of the benefits of multiple inheritance but implementing a tangled set of classes with multiple interfaces on them can lead to a lot of duplicate code.
What if there was a way to do multiple inheritance without every having to write a class? What if we could simply declare a few interfaces and then ask for an object that implements all of them and a way to persist it to disk and get it back? What if we could later take one of those objects and add another interface to it? “Crazy talk” I hear you say!
Well, maybe not so crazy … take a look at the open source project impromptu-interface and you’ll see some of what you’ll need to make this reality. It can take a .NET dynamic object and turn it into an object that implements a specific interface.
Combine that with a simple MongoDB document store and some cunning logic to link the two together and voila, we have persistent objects that can implement any interface dynamically and there’s absolutely no classes in sight anywhere!
Let’s take a look at it in use and then I’ll explain how it works. First, let’s define a few interfaces:
public interface ILegs
{
int Legs { get; set; }
}
public interface IMammal
{
double BodyTemperatureCelcius { get; set; }
}
// Interfaces can use multiple inheritance:
public interface IHuman: IMammal, ILegs
{
string Name { get; set; }
}
// We can have interfaces that apply to specific instances of a class: not all humans are carnivores
public interface ICarnivore
{
string Prey { get; set; }
}
Now let’s take a look at some code to create a few of these new dynamic documents and treat them as implementors of those interfaces. First we need a MongoDB connection:
MongoServer MongoServer = MongoServer.Create(ConnectionString);
MongoDatabase mongoDatabase = MongoServer.GetDatabase("Remember", credentials);
Next we grab a collection where we will persist our objects.
var sampleCollection = mongoDatabase.GetCollection<SimpleDocument>("Sample");
Now we can create some objects adding interfaces to them dynamically and we get to use those strongly typed interfaces to set properties on them.
var person1 = new SimpleDocument();
person1.AddLike<IHuman>().Name = "John";
person1.AddLike<ILegs>().Legs = 2;
person1.AddLike<ICarniovore>().Prey = "Cattle";
sampleCollection.Save(person1);
var monkey1 = new SimpleDocument();
monkey1.AddLike<IMammal>(); // mark as a mammal
monkey1.AddLike<ILegs>().Legs = 2;
monkey1.AddLike<ICarniovore>().Prey = "Bugs";
sampleCollection.Save(monkey1);
Yes, that’s it! That’s all we needed to do to create persisted objects that implement any collection of interfaces. Note how the IHuman is also an IMammal because our code will also support inheritance amongst interfaces. We can load them back in from MongoDB and get the strongly typed versions of them by using .AsLike
So next, let’s take a look at how we can query for objects that support a given interface and how we can get strongly typed objects back from MongoDB:
var query = Query.EQ("int", typeof(IHuman).Name);
var humans = sampleCollection.Find(query);
Console.WriteLine("Examine the raw documents");
foreach (var doc in humans)
{
Console.WriteLine(doc.ToJson());
}
Console.WriteLine("Use query results strongly typed");
foreach (IHuman human in humans.Select(m => m.AsLike<IHuman>()))
{
Console.WriteLine(human.Name);
}
Console.ReadKey();
So how does this ‘magic’ work? First we need a simple Document class. It can be any old object class, no special requirements. At the moment it does wrap these interface properties up in a document inside it called ‘prop’ making it just a little bit harder to query and index but still fairly easy.
/// <summary>
/// A very simple document object
/// </summary>
public class SimpleDocument : DynamicObject
{
public ObjectId Id { get; set; }
// All other properties are added dynamically and stored wrapped in another Document
[BsonElement("prop")]
protected BsonDocument properties = new BsonDocument();
/// <summary>
/// Interfaces that have been added to this object
/// </summary>
[BsonElement("int")]
protected HashSet<string> interfaces = new HashSet<string>();
/// <summary>
/// Add support for an interface to this document if it doesn't already have it
/// </summary>
public T AddLike<T>()
where T:class
{
interfaces.Add(typeof(T).Name);
foreach (var @interface in typeof(T).GetInterfaces())
interfaces.Add(@interface.Name);
return Impromptu.ActLike<T>(new Proxy(this.properties));
}
/// <summary>
/// Cast this object to an interface only if it has previously been created as one of that kind
/// </summary>
public T AsLike<T>()
where T : class
{
if (!this.interfaces.Contains(typeof(T).Name)) return null;
else return Impromptu.ActLike<T>(new Proxy(this.properties));
}
}
Then we need a simple proxy object to wrap up the properties as a dynamic object that we can feed to Impromptu:
public class Proxy : DynamicObject
{
public BsonDocument document { get; set; }
public Proxy(BsonDocument document)
{
this.document = document;
}
public override bool TryGetMember(GetMemberBinder binder, out object result)
{
BsonValue res = null;
this.document.TryGetValue(binder.Name, out res);
result = res.RawValue;
return true; // We always support a member even if we don't have it in the dictionary
}
/// <summary>
/// Set a property (e.g. person1.Name = "Smith")
/// </summary>
public override bool TrySetMember(SetMemberBinder binder, object value)
{
this.document.Add(binder.Name, BsonValue.Create(value));
return true;
}
}
And that’s it! There is no other code required. Multiple-inheritance and code-free persistent objects are now a reality! All you need to do is design some interfaces and objects spring magically to life and get persisted easily.
[NOTE: This is experimental code: it's a prototype of an idea that's been bugging me for some time as I look at how to meld Semantic Web classes which have multiple inheritance relationships with C# classes (that don't) and with MongoDB's document-centric storage format. Does everything really have to be stored in a triple-store or is there some hybrid where objects can be stored with their properties and triple-store statements can be reserved for more complex relationships? Can we get semantic web objects back as meaningful C# objects with strongly typed properties on them? It's an interesting challenge and this approach appears to have some merit as a way to solve it.]
Finally got the 1U Atom Server racked up
Apr 9th
The Atom server I added to my home network is finally installed in the rack and I’ve begun moving storage over to it and off the rather overloaded home automation server. The 1U immediately below it houses 4 SATA drives (mostly 2TB now) with USB/Firewire connections. Readers of my blog will recall my disdain for RAID as a “backup” technology (it’s for availabilitynot backup) so the storage scheme I used is original ->backup -> second backup so there are three copies of everything.
The Atom server is running 64-bit Windows Server and seems surprisingly fast. I plan to run MongoDB on it too.










































































































































































































