.NET

A simple state machine in C#

Within the Abodit Natural Language engine there is often a need to track the state of various elements of a conversation. For example, is the user logged in or not, have they verified their email address, what instructional text have we offered them so far, …

To make this easier I decided to add a simple state machine class to the Abodit utilities provided with my NLP Engine. There are, of course, a plethora of existing state machines on the web. Some of them are based on older .NET technology lacking use of generics and functional programming techniques. Others go overboard with fluent-style interfaces when a simple inheritance-based approach from an abstract base class would actually be simpler, less code, and more powerful. Most of them I didn’t discover until after I’d built this one. In any case, it’s always a good learning exercise to try to build something from scratch, so here goes …

First let’s take a look at the result. Here’s how you can define a state machine that derives from this new StateMachine class:

      public class LoginOutStatemachine : StateMachine<LoginOutStatemachine>
    {
        public static void ReportEnter(LoginOutStatemachine m, Event e, State state)
        {
            Console.WriteLine(m.User + " entered state " + state + " via " + e);
        }

        public static void ReportLeave(LoginOutStatemachine m, State state, Event e)
        {
            Console.WriteLine(m.User + " left state " + state + " via " + e);
        }

        public static State Initial = new State("Initial", ReportEnter, ReportLeave);
        public static State LoggedIn = new State("Logged In", ReportEnter, ReportLeave);
        public static State LoggedOut = new State("Logged Out", ReportEnter, ReportLeave);
        public static State Deleted = new State("Deleted", ReportEnter, ReportLeave);

        private static Event eLogsIn = new Event("Logs In");
        private static Event eLogsOut = new Event("Logs Out");
        private static Event eDeletesAccount = new Event("Account Deleted");

        static LoginOutStatemachine()
        {
            Initial
                    .When(eLogsIn, (m, s, e) => { Console.WriteLine("Logging in " + m.User); return LoggedIn; })
                    .When(eDeletesAccount, (m, s, e) => { Console.WriteLine("Deleting account " + m.User); return Deleted; });
            LoggedIn
                    .When(eLogsOut, (m, s, e) => { Console.WriteLine("Logging out " + m.User); return LoggedOut; })
                    .When(eDeletesAccount, (m, s, e) => { Console.WriteLine("Account deleted " + m.User); return Deleted; });
            LoggedOut
                    .When(eLogsIn, (m, s, e) => { Console.WriteLine("Logging in " + m.User); return LoggedIn; })
                    .When(eDeletesAccount, (m, s, e) => { Console.WriteLine("Account deleted " + m.User); return Deleted; });
        }

        public User User { get; private set; }

        public LoginOutStatemachine(State initial, User user)
            : base(initial)
        {
            this.User = user;
        }

        // Expose the events as public methods

        public void LogsIn()
        {
            this.EventHappens(eLogsIn);
        }

        public void LogsOut()
        {
            this.EventHappens(eLogsOut);
        }

        public void DeletesAccount()
        {
            this.EventHappens(eDeletesAccount);
        }
    }

As you can see, you define the states and the events for the state machine using static definitions. (Events trigger state changes and associated actions). Typically I’ll make the States public but the events private and instead provide method calls for each event that is allowed.

Each state can also have an Action that fires on entering the state and an action that fires on leaving the state and each action is provided with all of the parameters it might need (the state machine instance, the state it is going to or from, and the event that caused the transition to happen). In this case all of these entry and exit events are linked to the same method that simply reports what happened.

To define what happens when an given event is received by the state machine you create the static constructor as shown and then, using a fluent interface you define for each initial state, the transition to a new state by calling the When method passing it the event and the action to take when that event happens from the initial state specified. At the end of the method you must return the new state:

Initial
  .When(eLogsIn, (m, s, e) => { Console.WriteLine("Logging in " + m.User); return LoggedIn; })
  .When(eDeletesAccount, (m, s, e) => { Console.WriteLine("Deleting account " + m.User); return Deleted; });

The (m, s, e) parameters give you the state machine itself, the state you are coming from and the event that has been received. By passing your method all of these values I make it easy for you to access any properties of the state machine itself (e.g. a User object) and also allow you to write a single method that handles more than one event type or more than one initial state but which can still be parameterized by those values.

The other minor trick is that the StateMachine class is a generic in the state machine class itself. A small trick that allows access to `T` as the type of the inherited state machine class and thus to any additional properties you define there.

Note how your state machine class can have properties like `User` which allows the transition code to access any additional data it needs. You create an instance of the state machine for each user (all the heavy lifting is done in the static definition so the state machine remains a light-weight object).

In the case of the NLP engine you can pass an `IListener` in to the state machine constructor also so that you can `Say` messages back to the user. Since the state machine is such a light-weight object you can afford to create it for each message interaction with the user and the information you need to persist is just the current state (which I will soon make into a string lookup).

If you want to use the actual state machine in any of your own projects (gratis), here’s the current code:

    /// <summary>
    /// A state machine allows you to track state and to take actions when states change
    /// This state machine provides a fluent interface for defining states and transitions
    /// </summary>
    /// <remarks>
    /// Nasty generic of self so we can refer to the inheriting class in here
    /// </remarks>
    [Serializable]
    [DebuggerDisplay("Current State = {CurrentState.Name}")]
    public abstract class StateMachine<T> where T:StateMachine<T>
    {
        public State CurrentState { get; set; }

        public StateMachine(State initial)
        {
            this.CurrentState = initial;
        }

        /// <summary>
        /// An event has happened, transition to next state
        /// </summary>
        public void EventHappens(Event @event)
        {
            this.CurrentState = this.CurrentState.OnEvent((T)this, @event);
        }

        /// <summary>
        /// An event that causes the state machine to transition to a new state
        /// </summary>
        /// <remarks>
        /// Defined as a nested class so that this state machine's events can only be used with it
        /// </remarks>
        [DebuggerDisplay("Event = {Name}")]
        public class Event
        {
            public string Name { get; private set; }
            public Event(string name)
            {
                this.Name = name;
            }
            public override string ToString()
            {
                return "~" + this.Name + "~";
            }
        }

        /// <summary>
        /// A state that the state machine can be in
        /// </summary>
        /// <remarks>
        /// Defined as a nested class so that this state machine's states can only be used with it
        /// </remarks>
        [DebuggerDisplay("State = {Name}")]
        public class State
        {
            /// <summary>
            /// The Name of this state
            /// </summary>
            public string Name { get; private set; }

            public Action<T, State, Event> ExitAction { get; private set; }
            public Action<T, Event, State> EntryAction { get; private set; }

            private readonly IDictionary<Event, Func<T, State, Event, State>> transitions = new Dictionary<Event, Func<T, State, Event, State>>();

            /// <summary>
            /// Create a new State with a name and an optional entry and exit action
            /// </summary>
            public State(string name, Action<T, Event, State> entryAction = null, Action<T, State, Event> exitAction = null)
            {
                this.Name = name;
                this.EntryAction = entryAction;
                this.ExitAction = exitAction;
            }

            public State When(Event @event, Func<T, State, Event, State> action)
            {
                transitions.Add(@event, action);
                return this;
            }

            public State OnEvent(T parent, Event @event)
            {
                Func<T, State, Event, State> transition = null;
                if (transitions.TryGetValue(@event, out transition))
                {
                    State newState = transition(parent, this, @event);
                    if (newState != this)
                    {
                        // Entry and exit actions only fire when CHANGING state
                        if (this.ExitAction != null) this.ExitAction(parent, this, @event);
                        if (newState.EntryAction != null) newState.EntryAction(parent, @event, newState);
                    }
                    return newState;
                }
                else
                    return this;        // did not change state
            }

            public override string ToString()
            {
                return "*" + this.Name + "*";
            }
        }
    }
}

For further reading on State Machines I recommend this Wikipedia Article.

Programming a smart home with a fluent, domain-specific language

In response to a question I received recently, here is an example of the fluent extensions to C# that my home automation system uses to define how it should behave. In effect this is a domain specific language for modeling home automation behavior as a finite state machine. Note how you can use either a purely declarative sequence or you can use procedural code within a Do() block.

            Living.FloorSensor
                .Then(Entrance.HallBeam, 30)
                .Provided(Time.Bedtime)
                .ProvidedNot(Home.DinnerGuests)
                .PulseStretch(10 * 60)      // don't announce it too often - every 10 minutes
                .Do("Garage doors warning", () =>
            {
                if (Garage.GarageDoors.AreAnyOpen)
                {
                    FirstFloor.Kitchen.AllMediaZones.AnnounceString("Excuse me, I think you may want to close the garage doors.");
                }
            });

In many ways this is similar to the Reactive Framework from Microsoft except my work on this predates the availability of Reactive Framework and unlike the Reactive Framework my ‘sequential logic blocks’ include persistence logic so the entire state can be saved to disk (as it is every second). This is important because some transitions in the state machine might be several hours or even days long and you need to be able to restart the system and carry on exactly where you left off.

One key benefit of the declarative approach over the procedural approach is that the declarative approach can explain itself. So, the log entry for a light going on can show that the light was turned on because ‘it was dark’ and ‘we had visitors’ and ‘someone came into the room’. Compared to traditional home automation systems where you either have no logging at all or you have a log of what happened, this kind of logging is invaluable for figuring out what went wrong when the logic gets complicated. So in this example I should have moved the test for Garage.GarageDoors.AreAnyOpen out to a Provided clause which would allow it to be part of the reverse-chain logic explanation.

Partial results can be captured at any point in these logic chains and then reused in other chains because each partial result is itself a Sensor device that implements the full fluent interface.

Ultimately I plan to hook the logging for what happened back up to the NLP engine which will allow users to ask the home ‘Why did you put the driveway light on last night around 9pm?’ and ultimately I plan to allow the logic itself to be defined using natural language.

Convert a property getter to a setter

In the process of adding some strongly-typed extensions to the official MongoDB C# driver I needed some code to convert a property getter into a property setter. The following static method does that, turning e => e.Field into an Action method you can call to set the value of ‘Field’.

        /// <summary>
        /// Convert a lambda expression for a getter into a setter
        /// </summary>
        public static Action<T, U> GetSetter<T,U>(Expression<Func<T, U>> expression)
        {
            var memberExpression = (MemberExpression)expression.Body;
            var property = (PropertyInfo)memberExpression.Member;
            var setMethod = property.GetSetMethod();

            var parameterT = Expression.Parameter(typeof(T), "x");
            var parameterU = Expression.Parameter(typeof(U), "y");

            var newExpression =
                Expression.Lambda<Action<T, U>>(
                    Expression.Call(parameterT, setMethod, parameterU),
                    parameterT,
                    parameterU
                );

            return newExpression.Compile();
        }

What if … you could define and query an Ontology using natural language?

Tonight as an experiment I connected my NLP engine to my MongoDB triple-store and built a reasoner that does simple-entailment over just RDFS:subClassOf and OWL:disjointWith

Try it out at http://nlp.abodit.com/demo

Ignore the instructions on the right and try building a simple ontology,

Start by creating some root level class:

car is a class

Now expand it:

A BMW is a car
An Audi is a car
what is a BMW
BMW is a car
what is an Audi
Audi is a car
is an Audi a BMW
I don’t know, under the open world assumption it could be
an Audi is not a BMW
is an Audi a BMW
No

Simple but promising.

Dynamic persistence with MongoDB – look, no classes! Multiple inheritance in C#!

In an earlier post I explained a technique to create a class-free persistence layer using MongoDB. [Read that post first, then come back here.]

Since then I’ve refined the techniques involved and created a cleaner implementation that does away with the `.props` collection on each object. Now when you add an interface to an object you get exactly what you expected in the persisted data.

To use it you first need to register the serialization code somewhere in your startup code…

            BsonSerializer.RegisterSerializationProvider(new MongoDynamicSerializationProvider());

The Serialization provider is quite simple:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using MongoDB.Bson.Serialization;

namespace MongoData.Dynamic
{
    public class MongoDynamicSerializationProvider : IBsonSerializationProvider
    {

        public IBsonSerializer GetSerializer(Type type)
        {
            if (typeof(MongoDynamic).IsAssignableFrom(type))
                return MongoDynamicBsonSerializer.Instance;
            return null;
        }
    }
}

The serializer is a bit more involved. It uses an interface map to decide what type to return for each serialized object. This is critical because many different .NET types can map onto the same BSon serialized value and only by maintaining this map can we get back to the original type. It’s also
critical for handling nested object graphs containing different types.

using System;
using System.Collections.Concurrent;
using System.Dynamic;
using System.Linq;
using System.Linq.Expressions;
using System.Runtime.CompilerServices;
using Microsoft.CSharp.RuntimeBinder;
using MongoDB.Bson.IO;
using MongoDB.Bson.Serialization;
using MongoDB.Bson.Serialization.Serializers;
using MongoDB.Bson;
using MongoDB.Bson.Serialization.IdGenerators;
using System.Collections.Generic;
using ImpromptuInterface;

namespace MongoData.Dynamic
{
    public class MongoDynamicBsonSerializer : BsonBaseSerializer
    {
        private static MongoDynamicBsonSerializer instance = new MongoDynamicBsonSerializer();

        public static MongoDynamicBsonSerializer Instance
        {
            get { return instance; }
        }

        public override object Deserialize(BsonReader bsonReader, Type nominalType, IBsonSerializationOptions options)
        {
            var bsonType = bsonReader.CurrentBsonType;
            if (bsonType == BsonType.Null)
            {
                bsonReader.ReadNull();
                return null;
            }
            else if (bsonType == BsonType.Document)
            {
                var os = new ObjectSerializer();
                MongoDynamic md = new MongoDynamic();
                bsonReader.ReadStartDocument();

                Dictionary<string, Type> typeMap = null;

                // scan document first to find interfaces
                {
                    var bookMark = bsonReader.GetBookmark();
                    if (bsonReader.FindElement(MongoDynamic.InterfacesField))
                    {
                        md[MongoDynamic.InterfacesField] = BsonValue.ReadFrom(bsonReader).AsBsonArray.Select(x => x.AsString);
                        typeMap = md.GetTypeMap();
                    }
                    else
                    {
                        throw new FormatException("No interfaces defined for this dynamic object - can't deserialize it");
                    }
                    bsonReader.ReturnToBookmark(bookMark);
                }

                while (bsonReader.ReadBsonType() != BsonType.EndOfDocument)
                {
                    var name = bsonReader.ReadName();

                    if (name == "_id")
                    {
                        md[name] = BsonValue.ReadFrom(bsonReader).AsObjectId;
                    }
                    else if (name == MongoDynamic.InterfacesField)
                    {
                        // Read it and ignore it, we already have it
                        BsonValue.ReadFrom(bsonReader);
                    }
                    else
                    {
                        if (typeMap == null) throw new FormatException("No interfaces define for this dynamic object - can't deserialize");
                        // lookup the type for this element according to the interfaces
                        Type elementType;
                        if (typeMap.TryGetValue(name, out elementType))
                        {
                            var value = BsonSerializer.Deserialize(bsonReader, elementType);
                            md[name] = value;
                        }
                        else
                        {
                            // This is a value that is no longer in the interface, maybe a column you removed
                            // not really much we can do with it ... but we need to read it and move on
                            var value = BsonSerializer.Deserialize(bsonReader, typeof(object));
                            md[name] = value;

                            // As with all databases, removing elements from the schema is always going to cause problems ...
                        }
                    }
                }
                bsonReader.ReadEndDocument();
                return md;
            }
            else
            {
                var message = string.Format("Can't deserialize a {0} from BsonType {1}.", nominalType.FullName, bsonType);
                throw new FormatException(message);
            }
        } 

        public override bool GetDocumentId(object document, out object id, out Type idNominalType, out IIdGenerator idGenerator)
        {
            MongoDynamic x = (MongoDynamic)document;
            id = x._id;
            idNominalType = typeof(ObjectId);
            idGenerator = new ObjectIdGenerator();
            return true;
        }

        public override void SetDocumentId(object document, object id)
        {
            MongoDynamic x = (MongoDynamic)document;
            x._id = (ObjectId)id;
        }

        public override void Serialize(BsonWriter bsonWriter, Type nominalType, object value, IBsonSerializationOptions options)
        {
            if (value == null)
            {
                bsonWriter.WriteNull();
                return;
            }
            var metaObject = ((IDynamicMetaObjectProvider)value).GetMetaObject(Expression.Constant(value));
            var memberNames = metaObject.GetDynamicMemberNames().ToList();
            if (memberNames.Count == 0)
            {
                bsonWriter.WriteNull();
                return;
            }

            bsonWriter.WriteStartDocument();
            foreach (var memberName in memberNames)
            {
                // ToDo: handle all those _id Id id variants?
                bsonWriter.WriteName(memberName);

                object memberValue;
                if (memberName == "_id") memberValue = ((MongoDynamic)value)._id;
                else if (memberName == "int") memberValue = ((MongoDynamic)value).@int;
                else memberValue = Impromptu.InvokeGet(value, memberName);

                if (memberValue == null)
                    bsonWriter.WriteNull();
                else
                {
                    var memberType = memberValue.GetType();
                    var serializer = BsonSerializer.LookupSerializer(memberType);
                    serializer.Serialize(bsonWriter, memberType, memberValue, null);
                }
            }
            bsonWriter.WriteEndDocument();
        }
    }
}

And finally, the actual MongoDynamic class:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Dynamic;
using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using ImpromptuInterface;

namespace MongoData.Dynamic
{
    /// <summary>
    /// All MongoDynamic objects support this interface because every object needs an _id in MongoDB
    /// </summary>
    public interface IId
    {
        ObjectId _id { get; set; }
    }

    /// <summary>
    /// MongoDynamic is like an ExpandoObject that also understands document Ids and uses Improptu interface
    /// to act like any other collection of interfaces ...
    /// It can be serialized and deserialized from BSon and thus stored in a MongoDB database.
    /// </summary>
    /// <remarks>
    /// This simple class gives you the ability to define database objects using only .NET interfaces - no classes!
    /// Those objects can be dynamically extended to support any interface you want to add to them - polymorphism!
    /// When loaded back from the database the object will support all of the interfaces that were ever applied to it.
    /// Adding a new field is easy.  Removing one works too.
    /// All fields must be nullable since they may not be present on earlier instances of an object type.
    /// </remarks>
    public class MongoDynamic : DynamicObject, IId
    {
        [BsonId(Order=1)]
        public ObjectId _id { get; set; }

        // Dumb name for a property - which is why I chose it - very unlikely it will ever conflict with a real property name
        public const string InterfacesField = "int";

        /// <summary>
        /// Interfaces that have been added to this object
        /// </summary>
        /// <remarks>
        /// We always begin by supporting the _id interface
        /// Order is important, we need to see this field before we can deserialize any others
        /// </remarks>
        [BsonElement(InterfacesField, Order=2)]
        internal HashSet<string> @int = new HashSet<string>(){ typeof(IId).FullName };

        /// <summary>
        /// A text version of all interfaces - mostly for debugging purposes, stored in alphabetical order
        /// </summary>
        [BsonIgnore]
        public string InterfacesAsText
        {
            get { return string.Join(",", this.@int.OrderBy(i => i)); }
        }

        /// <summary>
        /// Add support for an interface to this document if it doesn't already have it
        /// </summary>
        public T AddLike<T>()
            where T : class
        {
            @int.Add(typeof(T).FullName);
            // And also act like any interfaces that interface implements (which will include ones they represent too)
            foreach (var @interface in typeof(T).GetInterfaces())
                @int.Add(@interface.FullName);
            return Impromptu.ActLike<T>(this, this.GetAllInterfaces());
        }

        /// <summary>
        /// Add support for multiple interfaces
        /// </summary>
        public T AddLike<T>(Type[] otherInterfaces)
            where T : class
        {
            var allInterfaces = otherInterfaces.Concat(new[] { typeof(T) });
            var allInterfacesAndDescendants = allInterfaces.Concat(allInterfaces.SelectMany(x => x.GetInterfaces()));
            foreach (var @interface in allInterfacesAndDescendants)
                @int.Add(@interface.FullName);
            return Impromptu.ActLike<T>(this, this.GetAllInterfaces());
        }

        /// <summary>
        /// Cast this object to an interface only if it has previously been created as one of that kind
        /// </summary>
        public T AsLike<T>()
            where T : class
        {
            if (!this.@int.Contains(typeof(T).FullName)) return null;
            else return Impromptu.ActLike<T>(this, this.GetAllInterfaces());
        }

        // A rather large cache of all interface types loaded into the App Domain
        private static List<Type> cacheOfTypes = null;

        // A cache of the interface types corresponding to a given 'key' of interface names
        private static Dictionary<string, Type[]> cacheOfInterfaces = new Dictionary<string, Type[]>();

        public Type[] GetAllInterfaces()
        {
            // We always behave like an object with an Id plus any other interfaces we have
            var key = string.Join(",", this.@int.OrderBy(i => i));
            if (!cacheOfInterfaces.ContainsKey(key))
            {
                if (cacheOfTypes == null)
                {
                    var assemblies = AppDomain.CurrentDomain.GetAssemblies();
                    cacheOfTypes = assemblies.SelectMany(ass => ass.GetTypes()).Where(t => t.IsInterface).ToList();
                }
                var interfaces = cacheOfTypes.Where(t => this.@int.Any(i => i == t.FullName));

                // Could trim the interfaces to remove any that are inherited from others ...
                cacheOfInterfaces.Add(key, interfaces.ToArray());
            }
            return cacheOfInterfaces[key];
        }

        /// <summary>
        /// Get a mapping from a field name to a type according to the interfaces on this object
        /// </summary>
        /// <returns></returns>
        public Dictionary<string, Type> GetTypeMap()
        {
            Dictionary<string, Type> typeMap = new Dictionary<string, Type>();
            var interfaces = this.GetAllInterfaces();
            foreach (var mi in interfaces.SelectMany(intf => intf.GetProperties()))
            {
                typeMap[mi.Name] = mi.PropertyType;
            }
            return typeMap;
        }

        /// <summary>
        /// Becomes a Proxy object that acts like it implements all of the interfaces listed as being supported by this Entity
        /// </summary>
        /// <remarks>
        /// Because the returned object supports ALL of the interfaces that have ever been added to this object
        /// you can cast it to any of them.  This enables a type of polymorphism.
        /// </remarks>
        public object ActLikeAllInterfacesPresent()
        {
            return Impromptu.DynamicActLike(this, this.GetAllInterfaces());
        }

        [BsonIgnore]
        // BsonIgnore because Bson serialization will happen on the dynamic interface this class exposes not on this dictionary
        private Dictionary<string, object> children = new Dictionary<string, object>();

        /// <summary>
        /// Fetch a property by name
        /// </summary>
        public override bool TryGetMember(GetMemberBinder binder, out object result)
        {
            if (binder.Name == "_id") { result = this._id; return true; }
            else if (binder.Name == InterfacesField) { result = this.@int; return true; }
            else
            {
               children.TryGetValue(binder.Name, out result);
               result = null;                         // we hope that it's nullable!  If not you have an issue
               return true;                           // when you do a database migration or query a nullable field it won't be in 'children'
            }
        }

        /// <summary>
        /// Set a property (e.g. person1.Name = "Smith")
        /// </summary>
        public override bool TrySetMember(SetMemberBinder binder, object value)
        {
            if (binder.Name == "_id") { this._id = (ObjectId)value; return true; }      // you shouldn't need to use this
            if (binder.Name == InterfacesField) throw new AccessViolationException("You cannot set the interfaces directly, use AddLike() instead");
            if (!this.GetTypeMap().ContainsKey(binder.Name)) throw new ArgumentException("Property '" + binder.Name + "' not found.  You need to call AddLike to specify the interfaces you want to support.");
            children[binder.Name] = value;
            return true;
        }

        public override IEnumerable<string> GetDynamicMemberNames()
        {
            return new[]{"_id", InterfacesField}.Concat(children.Keys);
        }

        /// <summary>
        /// An indexer for use by serialization code
        /// </summary>
        internal object this[string key]
        {
            get
            {
                if (key == "_id") return this._id;
                else if (key == InterfacesField) return this.@int;
                else return children[key];
            }

            set
            {
                if (key == "_id" && value is BsonObjectId) this._id = ((BsonObjectId)value).Value;
                else if (key == "_id") this._id = (ObjectId)value;
                else if (key == InterfacesField) this.@int = new HashSet<string>((IEnumerable<string>)value);
                else children[key] = value;
            }
        }
    }
}

You’ll need Impromptu interface (from Nuget) to build this. To use it, you write code like this to save to MongoDB:

            MongoDynamic entity = new MongoDynamic();
            var user = entity.AddLike<IUser>();         // *** Add the IUser fields to it ...
            user.Name = name;                           // Use it as if it were an IUser
            // save it to the database as normal

And to retrieve an object you create a query as normal and then query for MongoDynamic objects like so …

            var user = database.GetCollection<MongoDynamic>("***collectionName***").FindOne(query);
            if (user == null) return null;
            return user.AsLike<IUser>();

Typically you will want your query to reference the field called int (where all the interfaces are stored) so you can query for objects that support a specific type (if you do, you’ll want to add an index on that field). [NB the name was chosen to be one you were unlikely to ever use in .NET]

MongoDynamic objects are polymorphic – you can morph them to support any other interface at any time like so …

            user.AddLike<ISomeOtherInterface>();

Home network crawler – cataloging every file on the home LAN with C# and MongoDB

Map-Reduce in operation in Greenland

Map-Reduce in action: The glaciers in Greenland 'map' the canyon walls into streams of rocks called lateral moraine. As the glaciers merge these rocks are 'reduced' into streams in the middle called 'medial' moraine. (A photo I took over Greenland this summer.)

With the addition of two more 3TB drives to the home network it’s becoming impossible to track files and to remember where each one is and whether it’s a backup of some other disk or not. There are 8 computers on the home network and over 10TB of storage distributed between them. Much of the storage is concentrated on a single machine running Windows Server 2008. It’s a low-powered Atom server connected to a Sans Digital 1U Rackmount Sans Digital disk array running in JBOD mode (just a bunch of disks).

I’m not a huge fan or RAID arrays – they mostly mean there’s another component to go wrong (the controller card) and when they do go wrong you can lose all your data just as easily as if it were all on one drive. I prefer a multiple copy strategy, an “Amazon S3 for the home” if you like. The downside of this is that there are multiple copies of each file across the home network and as I have several generations of hard drives the mapping from primary to secondary to tertiary is complex and hard to manage! It’s also really hard to find a single file when there are so many places to look and it’s nigh on impossible to be sure that I have the necessary three copies of every important file in the right places at all times.

So this weekend I embarked on a small project to catalog every file, directory and storage volume on the entire home network including drives that are only sometimes connected. The software has been running all weekend and is close to cataloging everything. It’s found 5 million files so far representing over 6TB of data!

The architecture I chose for this software was an agent that runs on each PC to catalog all of the attached volumes. This client uploads all the directories and files that it finds to a MongoDB database running on the same Atom server as the main storage array. The poor little Atom server’s 4GB of RAM has been in constant use but the server has remained responsive, in part because it boots from an SSD drive.

Each volume, directory and file is represented by a document in MongoDB in a single collection. The agent calculates an MD5 hash for each file and extracts metadata from MP3, WMA and JPG files. It also stores all of the key file dates (created, updated, accessed) and references to parent directories, volume identifiers and the currently connected PC. It does not assume that a volume is always connected to the same computer – you can unplug an external drive from one and put it somewhere else and it will all work just fine.

I implemented a re-startable tree scan that uses a couple of DateTime stamps to be able to determine which directories need to be scanned during the current pass and which ones have already been scanned. Any agent can be killed at any time and restarted and it will carry on walking the directory tree right where it left off. It will even continue correctly in the case where you move a volume from one PC to another.

Each agent uses the Parallel Task library’s Parallel.ForEach to crawl each volume in parallel and to parse multiple files from each directory simultaneously.

By storing all of the file metadata in Mongo DB it’s easy to use Map-Reduce to calculate some interesting statistics for the files on the network.

For example, to create a summary of file sizes I can use a Map function:

function Map() {
	if (this.Size && this._t == "FileInformation")
	{
		var size = this.Size;

		if (size < 1024)
			emit ("kb", {count:1, size:this.Size});
		else if (size < 1024*1024)
			emit ("mb", {count:1, size:this.Size});
		else if (size < 1024*1024*1024)
			emit ("gb", {count:1, size:this.Size});
		else if (size < 1024*1024*1024*1024)
			emit ("tb", {count:1, size:this.Size});
		else
			emit ("tb+", {count:1, size:this.Size});
	}
}

and a reduce function:

function Reduce(key, arr_values) {

	var count = 0;
	var size = 0;

	for(var i in arr_values)
	{
		count = count + arr_values[i].count;
		size = size + arr_values[i].size;
	}

	return {count:count, size:size};
}

Map-Reduce operations like this take about 20 minutes to run (on the Atom server with just 4GB of RAM) whereas any query serviced by one of the indexes on the MongoDB collection is almost instantaneous.

I’ve been using the excellent MongoVue to run simple map-reduce scripts like this and to keep track of how quickly the database is growing.

Map-reduce can also be used to find duplicate files – by emitting the MD5 hash as the key and some information about the file as the value I can find every copy of every file across every computer on the home network.

Since I have the file name and metadata for every file on the home network I can also easily find any file using MongoDB’s regex matching feature against the path.

The Hard Parts

For starters you’ll need a library that can handle long file names. Then you’ll need to fix it to provide at least the functionality that FileInfo and DirectoryInfo give you in .NET.

Next you’ll need to learn about reparse-points and hard-links and you’ll need to skip over them because with them in place the file system is not a tree; it’s a cyclical graph in which a simple crawler will quickly get confused or stuck.

You’ll also want to store the NTFS file Id and the unique Volume ID for every file so you can track it when the file is moved or the removable drive is connected to a different computer.

So how well does it work?

This all seems to work really well. Nearly every volume has now been cataloged. It’s located about 5M files occupying over 6TB of space. The worst case offender for the number of copies of the same file is 100+. I’ve used the find feature in MongoDB to find a file I was missing and I’m better able to plan how to arrange directories and file generations across the various hard drives I have.

What’s next

Well, of course this needs to be connected to the home automation system and my Natural Language engine so you can ask “send a copy of IMG_0228 from last week to X” or “where are all the spreadsheets I created last year?” That will be fairly easy.

After that I hope to incorporate backup features into the agents too so they can automatically keep the required number of copies of each file according to its importance. I’d also like to set up a rotating set of external drives that go in the fire safe when not connected and when they are connected they get updated with the latest copies of all the important files.

I’d also like to be able to get the agents to move whole groups of directories around between drives as juggling the directory layout each time a new hard drive is added to the system is always a time consuming process.

Comments or Questions?

Does everyone else have a hard time managing multiple computers, hard drives, directories and multiple copies of files? What tools do you use to do this? Is there anything commercially available that I could have used instead? Would a tool like this be useful to you? Should I publish the code somewhere? Comments and questions are always welcome here or on twitter.

Stop writing rude software! Use LASTINPUTINFO instead.

Can you imagine what life would be like it people behaved like software programs do?

You’d be working away on something when someone would interrupt, steal your attention, and demand a response. You’d be interrupted in the middle of sentences all the time and while you were dealing with one interruption someone else could come up and interrupt you again.

You wouldn’t put up with people like that so why do you put up with software that behaves that way?

Windows itself is one of the worst offenders: the dreaded dialog that explains that updates have been installed and it wants to reboot, right this instant has caused me significant inconvenience in the past as it steals focus and then grabs the next return character and assumes I really did want to reboot right now, right in the middle of a blog post!

There really is no excuse for writing rude software. Windows includes an API called LASTINPUTINFO that can tell you if the user is busy typing or moving the mouse and you can delay your annoying toast pop-up, or worse that focus-stealing modal dialog until you think the user is ready for it. The C# code below shows how to use this API call to get a number of seconds since the last user input. Simply delay your notification or dialog until an appropriate time has passed (e.g. 5 seconds) and only then interrupt the user).

Background processing

Similarly if your background processing is hammering the disk drive you can make it more polite and throttle it back when the user is active on their computer. (You did, of course do all that background processing on a lower priority thread, didn’t you!)

One other area you might want to consider is using BITS to download files instead of hammering their internet connection to fetch files in the background.

The Code

So here’s the code you should use from today to make your software polite:

    public static class Input
    {
        [DllImport("User32.dll")]
        private static extern bool
                GetLastInputInfo(ref LASTINPUTINFO plii);

        private struct LASTINPUTINFO
        {
            public uint cbSize;
            public uint dwTime;
        }

        /// <summary>
        /// How many seconds since last user input
        /// </summary>
        public static double SecondsSinceLastInput()
        {
            LASTINPUTINFO lastInPut = new LASTINPUTINFO();
            lastInPut.cbSize = (uint)System.Runtime.InteropServices.Marshal.SizeOf(lastInPut);
            GetLastInputInfo(ref lastInPut);

            uint idle = (uint)Environment.TickCount - lastInPut.dwTime;
            return idle/1000.0;
        }

    }

C# Natural Language Engine connected to Microsoft Dynamics CRM 2011 Online

In an earlier post I discussed some ideas around a Semantic CRM.

Recently I’ve been doing some clean up work on my C# Natural Language Engine and decided to do a quick test connecting it to a real CRM. As you may know from reading my blog, this natural language engine is already heavily used in my home automation system to control lights, sprinklers, HVAC, music and more and to query caller ID logs and other information.

I recently refactored it to use the Autofac dependency injection framework and in the process realized just how close my NLP engine is to ASP.NET MVC 3 in its basic structure and philosophy! To use it you create Controller classes and put action methods in them. Those controller classes use Autofac to get all of the dependencies they may need (services like an email service, a repository, a user service, an HTML email formattting service, …) and then the methods in them represents a specific sentence parse using the various token types that the NLP engine supports. Unlike ASP.NET MVC3 there is no Route registration; the method itself represents the route (i.e. sentence structure) that it used to decide which method to call. Internally my NLP engine has its own code to match incoming words and phrases to tokens and then on to the action methods. In a sense the engine itself is one big dependency injection framework working against the action methods. I sometimes wish ASP.NET MVC 3 had the same route-registration-free approach to designing web applications (but also appreciate all the reasons why it doesn’t).

Another improvement I made recently to the NLP Engine was to develop a connector for the Twilio SMS service. This means that my home automation system can now accept SMS messages as well as all the other communication formats it supports: email, web chat, XMPP chat and direct URL commands. My Twilio connector to NLP supports message splitting and batching so it will buffer up outgoing messages to reach the limit of a single SMS and will send that. This lowers SMS charges and also allows responses that are longer than a single SMS message.

Using this new, improved version of my Natural Language Engine I decided to try connecting it to a CRM. I chose Microsoft Dynamics CRM 2011 and elected to use the strongly-typed, early-bound objects that you can generate for any instance of the CRM service. I added some simple sentences in an NLPRules project that allow you to tell it who you met, and to input some of their details. Unlike a traditional forms-based approach the user can decide what information to enter and what order to enter it in. The Natural Language Engine supports the concept of a conversation and can remember what you were discussing allowing a much more natural style of conversation that some simple rule-based engines and even allowing it to ask questions and get answers from the user.

Here’s a screenshot showing a sample conversation using Google Talk (XMPP/Jabber) and the resulting CRM record in Microsoft CRM 2011 Online. You could have the same conversation over SMS or email. Click to enlarge.

A natural language interface to CRM

Based on my limited testing this looks like another promising area where a truly fluent, conversational-style natural language engine could play a significant role. Note how it understands email addresses, phone numbers and such like and in code these all become strongly typed objects. Where it really excels is in temporal expressions where it can understand things like “who called on a Saturday in May last year?” and can construct an efficient SQL query from that.

Class-free persistence and multiple inheritance in C# with MongoDB

Much as I appreciate Object Relational Mappers and the C# type system there’s a lot of work to do if you just want create and persist a few objects. MongoDB alleviates a lot of that work with its Bson serialization code that converts almost any object into a binary serialized object notation and provides easy round tripping with JSON.

But there’s no getting around the limitations of C# when it comes to multiple inheritance. You can use interfaces to get most of the benefits of multiple inheritance but implementing a tangled set of classes with multiple interfaces on them can lead to a lot of duplicate code.

What if there was a way to do multiple inheritance without every having to write a class? What if we could simply declare a few interfaces and then ask for an object that implements all of them and a way to persist it to disk and get it back? What if we could later take one of those objects and add another interface to it? “Crazy talk” I hear you say!

Well, maybe not so crazy … take a look at the open source project impromptu-interface and you’ll see some of what you’ll need to make this reality. It can take a .NET dynamic object and turn it into an object that implements a specific interface.

Combine that with a simple MongoDB document store and some cunning logic to link the two together and voila, we have persistent objects that can implement any interface dynamically and there’s absolutely no classes in sight anywhere!

Let’s take a look at it in use and then I’ll explain how it works. First, let’s define a few interfaces:

    public interface ILegs
    {
        int Legs { get; set; }
    }

    public interface IMammal
    {
        double BodyTemperatureCelcius { get; set; }
    }

    // Interfaces can use multiple inheritance:

    public interface IHuman: IMammal, ILegs
    {
        string Name { get; set; }
    }

    // We can have interfaces that apply to specific instances of a class: not all humans are carnivores

    public interface ICarnivore
    {
        string Prey { get; set; }
    }

Now let’s take a look at some code to create a few of these new dynamic documents and treat them as implementors of those interfaces. First we need a MongoDB connection:

            MongoServer MongoServer = MongoServer.Create(ConnectionString);
            MongoDatabase mongoDatabase = MongoServer.GetDatabase("Remember", credentials);

Next we grab a collection where we will persist our objects.

            var sampleCollection = mongoDatabase.GetCollection<SimpleDocument>("Sample");

Now we can create some objects adding interfaces to them dynamically and we get to use those strongly typed interfaces to set properties on them.

            var person1 = new SimpleDocument();
            person1.AddLike<IHuman>().Name = "John";
            person1.AddLike<ILegs>().Legs = 2;
            person1.AddLike<ICarniovore>().Prey = "Cattle";
            sampleCollection.Save(person1);

            var monkey1 = new SimpleDocument();
            monkey1.AddLike<IMammal>();                 // mark as a mammal
            monkey1.AddLike<ILegs>().Legs = 2;
            monkey1.AddLike<ICarniovore>().Prey = "Bugs";
            sampleCollection.Save(monkey1);

Yes, that’s it! That’s all we needed to do to create persisted objects that implement any collection of interfaces. Note how the IHuman is also an IMammal because our code will also support inheritance amongst interfaces. We can load them back in from MongoDB and get the strongly typed versions of them by using .AsLike() which returns a value of type T or null if that object doesn’t implement the interface T. But that’s not all, we can even add new interfaces to them later allowing an object to change type over time! Now, of course, you could do a lot of this just with dynamic types but then you lose Intellisense and compile time checking.

So next, let’s take a look at how we can query for objects that support a given interface and how we can get strongly typed objects back from MongoDB:

            var query = Query.EQ("int", typeof(IHuman).Name);
            var humans = sampleCollection.Find(query);

            Console.WriteLine("Examine the raw documents");

            foreach (var doc in humans)
            {
                Console.WriteLine(doc.ToJson());
            }

            Console.WriteLine("Use query results strongly typed");

            foreach (IHuman human in humans.Select(m => m.AsLike<IHuman>()))
            {
                Console.WriteLine(human.Name);
            }

            Console.ReadKey();

So how does this ‘magic’ work? First we need a simple Document class. It can be any old object class, no special requirements. At the moment it does wrap these interface properties up in a document inside it called ‘prop’ making it just a little bit harder to query and index but still fairly easy.


        /// <summary>
        /// A very simple document object
        /// </summary>
        public class SimpleDocument : DynamicObject
        {
            public ObjectId Id { get; set; }

            // All other properties are added dynamically and stored wrapped in another Document
            [BsonElement("prop")]
            protected BsonDocument properties = new BsonDocument();

            /// <summary>
            /// Interfaces that have been added to this object
            /// </summary>
            [BsonElement("int")]
            protected HashSet<string> interfaces = new HashSet<string>();

            /// <summary>
            /// Add support for an interface to this document if it doesn't already have it
            /// </summary>
            public T AddLike<T>()
                where T:class
            {
                interfaces.Add(typeof(T).Name);
                foreach (var @interface in typeof(T).GetInterfaces())
                    interfaces.Add(@interface.Name);
                return Impromptu.ActLike<T>(new Proxy(this.properties));
            }

            /// <summary>
            /// Cast this object to an interface only if it has previously been created as one of that kind
            /// </summary>
            public T AsLike<T>()
                where T : class
            {
                if (!this.interfaces.Contains(typeof(T).Name)) return null;
                else return Impromptu.ActLike<T>(new Proxy(this.properties));
            }

        }

Then we need a simple proxy object to wrap up the properties as a dynamic object that we can feed to Impromptu:

        public class Proxy : DynamicObject
        {
            public BsonDocument document { get; set; }

            public Proxy(BsonDocument document)
            {
                this.document = document;
            }
            public override bool TryGetMember(GetMemberBinder binder, out object result)
            {
                BsonValue res = null;
                this.document.TryGetValue(binder.Name, out res);
                result = res.RawValue;
                return true;            // We always support a member even if we don't have it in the dictionary
            }

            /// <summary>
            /// Set a property (e.g. person1.Name = "Smith")
            /// </summary>
            public override bool TrySetMember(SetMemberBinder binder, object value)
            {
                this.document.Add(binder.Name, BsonValue.Create(value));
                return true;
            }
        }

And that’s it! There is no other code required. Multiple-inheritance and code-free persistent objects are now a reality! All you need to do is design some interfaces and objects spring magically to life and get persisted easily.

[NOTE: This is experimental code: it's a prototype of an idea that's been bugging me for some time as I look at how to meld Semantic Web classes which have multiple inheritance relationships with C# classes (that don't) and with MongoDB's document-centric storage format. Does everything really have to be stored in a triple-store or is there some hybrid where objects can be stored with their properties and triple-store statements can be reserved for more complex relationships? Can we get semantic web objects back as meaningful C# objects with strongly typed properties on them? It's an interesting challenge and this approach appears to have some merit as a way to solve it.]

Random names in C# using LINQ and a touch of functional programming

Today I needed some random names for testing. I wanted them to look like real names but to also look obviously made up so that nobody would ever treat them as a real record in the database. Below is a quick C# snippet to generate random names. It uses LINQ and a couple of anonymous functions to accomplish a task that would have taken many more lines in a traditional procedural style. Enumerable.Range() in particular is a really handy method for generating stuff.

Random r = new Random();
string alphabet = "abcdefghijklmnopqrstuvwyxzeeeiouea";
Func<char> randomLetter = () => alphabet[r.Next(alphabet.Length)];
Func<int, string> makeName =
  (length) => new string(Enumerable.Range(0, length)
     .Select(x => x==0 ? char.ToUpper(randomLetter()) : randomLetter())
     .ToArray());
string first = makeName(r.Next(5) + 5);
string last = makeName(r.Next(7) + 7);
string company = makeName(r.Next(7) + 7) + " Inc.";