By building a simple in-memory graph of all these relationships my engine can now use them to infer interface types on objects. To define an interface corresponding to a Wordnet synset like a mammal (wn30:synset-mammal-noun-1) you simply define an interface in the namespace ‘Noun’ having a name of ‘mammal1′. Through the type inheritance specified in the Wordnet file all mammals now inherit that interface automatically and you can write natural language rules that ask for a mammal and they will get any type of mammal defined in Wordnet.

For example:

Recently I’ve been doing some clean up work on my C# Natural Language Engine and decided to do a quick test connecting it to a real CRM. As you may know from reading my blog, this natural language engine is already heavily used in my home automation system to control lights, sprinklers, HVAC, music and more and to query caller ID logs and other information.

I recently refactored it to use the Autofac dependency injection framework and in the process realized just how close my NLP engine is to ASP.NET MVC 3 in its basic structure and philosophy! To use it you create Controller classes and put action methods in them. Those controller classes use Autofac to get all of the dependencies they may need (services like an email service, a repository, a user service, an HTML email formattting service, …) and then the methods in them represents a specific sentence parse using the various token types that the NLP engine supports. Unlike ASP.NET MVC3 there is no Route registration; the method itself represents the route (i.e. sentence structure) that it used to decide which method to call. Internally my NLP engine has its own code to match incoming words and phrases to tokens and then on to the action methods. In a sense the engine itself is one big dependency injection framework working against the action methods. I sometimes wish ASP.NET MVC 3 had the same route-registration-free approach to designing web applications (but also appreciate all the reasons why it doesn’t).

Another improvement I made recently to the NLP Engine was to develop a connector for the Twilio SMS service. This means that my home automation system can now accept SMS messages as well as all the other communication formats it supports: email, web chat, XMPP chat and direct URL commands. My Twilio connector to NLP supports message splitting and batching so it will buffer up outgoing messages to reach the limit of a single SMS and will send that. This lowers SMS charges and also allows responses that are longer than a single SMS message.

Using this new, improved version of my Natural Language Engine I decided to try connecting it to a CRM. I chose Microsoft Dynamics CRM 2011 and elected to use the strongly-typed, early-bound objects that you can generate for any instance of the CRM service. I added some simple sentences in an NLPRules project that allow you to tell it who you met, and to input some of their details. Unlike a traditional forms-based approach the user can decide what information to enter and what order to enter it in. The Natural Language Engine supports the concept of a conversation and can remember what you were discussing allowing a much more natural style of conversation that some simple rule-based engines and even allowing it to ask questions and get answers from the user.

Here’s a screenshot showing a sample conversation using Google Talk (XMPP/Jabber) and the resulting CRM record in Microsoft CRM 2011 Online. You could have the same conversation over SMS or email. Click to enlarge.

Based on my limited testing this looks like another promising area where a truly fluent, conversational-style natural language engine could play a significant role. Note how it understands email addresses, phone numbers and such like and in code these all become strongly typed objects. Where it really excels is in temporal expressions where it can understand things like “who called on a Saturday in May last year?” and can construct an efficient SQL query from that.

The store is currently hooked up to an Euler reasoner that can reason against it, lazily fetching just the necessary records from the SQL database that backs the Entity Model.

Here’s the EDMX showing how I modeled the Ontology Store:

Technologies like the Web Ontology Language (OWL) and RDF allow for the construction of complex ontologies that define what things are, and how they relate. Using these ontologies automated reasoning can be applied to generate new facts or to prove or disprove assertions.

This sounds like the ideal companion to the Natural Language Processing (NLP) Engine that I have already created for the my home automation system. With reasoning powers added to the natural language engine and the ability to augment the knowledge base by adding new assertions the whole system will be much more powerful. One day it might even be possible to create the entire home definition using a natural language text file and to query the system using rich natural language queries.

So, the first step is to find an existing ontology store and reasoning engine. A quick web search reveals that most are built in Java. There were a couple of links I came across later for .NET: http://razor.occams.info/code/semweb/ and http://www.intellidimension.com/products/semantics-server/. There’s also an interesting Q&A site at http://semanticoverflow.com which has lots of useful information on it.

But rather than starting with some existing library I really wanted to understand more deeply how an ontology store works and how a reasoning engine functions, so over the course of a couple of evenings I created my own. I now have a triple store and a simple reasoning engine. Here’s an actual conversation so you can see what it’s capable of so far and can perhaps get a glimpse at how powerful this concept could be:-

house is a class
contains is a property
contain is a property
contains is the same as contain
contain is the same as contains
contains is transitive
contain is transitive
first floor is a class
room is a class
kitchen is a room
first floor contains kitchen
house contains first floor
does house contain kitchen

House: Yes, house contain kitchen because [house contains first floor] -> [first floor contains kitchen]

As you can see my semantic store can already represent classes, relationships between classes, new relationships (‘contains’), relationships between relationships (‘same as’).  For such a small amount of code it’s quite surprising what this system can now handle in terms of knowledge representation and simple reasoning.

Next time I get some spare time I’ll hook it up to the actual home model so you can start to query that in much more powerful ways than before.

Stay tuned!

The natural language engine itself is a separate component that I hope one day to use in other applications.

Existing Natural Language Engines

• Have a large, STATIC dictionary data file
• Can parse complex sentence structure
• Hand back a tree of tokens (strings)
• Don’t handle conversations

C# NLP Engine

• Defines strongly-typed tokens in code
• Uses type inheritance to model ‘is a’
• Defines sentences in code
• Rules engine executes sentences
• Understands context (conversation history)

Sample conversation

Goals

• Make it easy to define tokens and sentences (not XML)
• Safe, compile-time checked definition of the syntax and grammar (not XML)
• Model real-world inheritance with C# class inheritance:
• ‘a labrador’ is ‘a dog’ is ‘an animal’ is ‘a thing’
• Handle ambiguity, e.g.

play something in the air tonight in the kitchen
remind me at 4pm to call john at 5pm

C# NLP Engine Structure

Tokens – Token Definition

• A hierarchy of Token-derived classes
• Uses inheritance, e.g. TokenOn is a TokenOnOff is a TokenState is a Token. This allows a single sentence rule to handle multiple cases, e.g. On and Off
• Derived from base Token class
• Simple tokens are a set of words, e.g. « is | are »
• Complex tokens have a parser, e.g. TokenDouble

A Simple Token Definition

public class TokenPersonalPronoun : TokenGenericNoun
{
   internal static string wordz { get { return "he,him,she,her,them"; } }
}

• Recognizes any of the words specified
• Can use inheritance (as in this example)

A Complex Token

public abstract class TokenNumber : Token
{
  public static IEnumerable<TokenResult> Initialize(string input)
  {
  …

• Initialize method parses input and returns one or more possible parses.

TokenNumber is a good example:

• Parses any numeric value and returns one or more of TokenInt, TokenLong, TokenIntOrdinal, TokenDouble, or TokenPercentage results.

The catch-all TokenPhrase

public class TokenPhrase : Token

TokenPhrase matches anything, especially anything in quote marks

   e.g. add a reminder "call Bruno at 4pm"

The sentence signature to recognize this could be

(…, TokenAdd, TokenReminder, TokenPhrase, TokenExactTime)

This would match the rule too …

add a reminder discuss 6pm conference call with Bruno at 4pm

TemporalTokens

A complete set of tokens and related classes for representing time

• Point in time, e.g. today at 5pm
• Approximate time, e.g. who called at 5pm today
• Finite sequence, e.g. every Thursday in May 2009
• Infinite sequence, e.g. every Thursday
• Ambiguous time with context, e.g. remind me on Tuesday (context means it is next Tuesday)
• Null time
• Unknowable/incomprehensible time

TemporalTokens (Cont.)

Code to merge any sequence of temporal tokens to the smallest canonical representation,

e.g.

the first thursday in may 2009

->
{TIMETHEFIRST the first} + {THURSDAY thursday} + {MAY in may} + {INT 2009 -> 2009}

->
[TEMPORALSETFINITESINGLEINTERVAL [Thursday 5/7/2009] ]

TemporalTokens (Cont.)

Finite TemporalClasses provide

All TemporalClasses provide

Existing Token Types

• Numbers (double, long, int, percentage, phone, temperature)
• File names, Directories
• URLs, Domain names
• Names, Companies, Addresses
• Rooms, Lights, Sensors, Sprinklers, …
• States (On, Off, Dim, Bright, Loud, Quiet, …)
• Units of Time, Weight, Distance
• Songs, albums, artists, genres, tags
• Temporal expressions
• Commands, verbs, nouns, pronouns, …

Rules – A simple rule

   /// <summary>
   /// Set a light to a given state
   /// </summary>
   private static void LightState(NLPState st, TokenLight tlight, TokenStateOnOff ts)
   {
      if (ts.IsTrueState == true) tlight.ForceOn(st.Actor);
      if (ts.IsTrueState == false) tlight.ForceOff(st.Actor);
      st.Say("I turned it " + ts.LowerCased);
   }

Any method matching this signature is a sentence rule:-  NLPState, Token*

Rule matching respects inheritance, and variable repeats … (NLPState st, TokenThing tt, TokenState tokenState, TokenTimeConstraint[] constraints)

Rules are discovered on startup using Reflection and an efficient parse graph is built allowing rapid detection and rejection of incoming sentences.

State – NLPState

• Every sentence method takes an NLPState first parameter
• State includes RememberedObject(s) allowing sentences to react to anything that happened earlier in a conversation
• Non-interactive uses can pass a dummy state
• State can be per-user or per-conversation for non-realtime conversations like email
  • Chat (e.g Jabber/Gtalk)
  • Web chat
  • Email
  • Calendar (do X at time Y)
  • Rich client application
  • Strongly-typed natural language engine
  • Compile time checking, inheritance, …
  • Define tokens and sentences (rules) in C#
  • Strongly-typed tokens: numbers, percentages, times, dates, file names, urls, people, business objects, …
  • Builds an efficient parse graph
  • Tracks conversation history

User Interface

Works with a variety of user interfaces

Summary

Future plans

Expanded corpus of knowledge

• Company names, locations, documents, …

Generate iCal/Gdata Recurrence

• From TimeExpressions