Here is an explanation of the natural language engine that powers my home automation system. It’s a strongly-typed natural language engine with tokens and sentences being defined in code. It currently understands sentences to control lights, heating, music, sprinklers, … You can ask it who called, you can tell it to play music in a particular room, … it tells you when a car comes down the drive, when the traffic is bad on I-90, when there’s fresh snow in the mountains, when it finds new podcasts from NPR, … and much more.

The natural language engine itself is a separate component that I hope one day to use in other applications.

Existing Natural Language Engines

  • Have a large, STATIC dictionary data file
  • Can parse complex sentence structure
  • Hand back a tree of tokens (strings)
  • Don’t handle conversations

C# NLP Engine

  • Defines strongly-typed tokens in code
  • Uses type inheritance to model ‘is a’
  • Defines sentences in code
  • Rules engine executes sentences
  • Understands context (conversation history)

Sample conversation

Goals

  • Make it easy to define tokens and sentences (not XML)
  • Safe, compile-time checked definition of the syntax and grammar (not XML)
  • Model real-world inheritance with C# class inheritance:
  • ‘a labrador’ is ‘a dog’ is ‘an animal’ is ‘a thing’
  • Handle ambiguity, e.g.
play something in the air tonight in the kitchen
remind me at 4pm to call john at 5pm

C# NLP Engine Structure

Tokens – Token Definition

  • A hierarchy of Token-derived classes
  • Uses inheritance, e.g. TokenOn is a TokenOnOff is a TokenState is a Token. This allows a single sentence rule to handle multiple cases, e.g. On and Off
  • Derived from base Token class
  • Simple tokens are a set of words, e.g. « is | are »
  • Complex tokens have a parser, e.g. TokenDouble

A Simple Token Definition

public class TokenPersonalPronoun : TokenGenericNoun
{
   internal static string wordz { get { return "he,him,she,her,them"; } }
}
  • Recognizes any of the words specified
  • Can use inheritance (as in this example)

A Complex Token

public abstract class TokenNumber : Token
{
  public static IEnumerable<TokenResult> Initialize(string input)
  {
  …
  • Initialize method parses input and returns one or more possible parses.

TokenNumber is a good example:

  • Parses any numeric value and returns one or more of TokenInt, TokenLong, TokenIntOrdinal, TokenDouble, or TokenPercentage results.

The catch-all TokenPhrase

public class TokenPhrase : Token

TokenPhrase matches anything, especially anything in quote marks

   e.g. add a reminder "call Bruno at 4pm"

The sentence signature to recognize this could be

(…, TokenAdd, TokenReminder, TokenPhrase, TokenExactTime)

This would match the rule too …

add a reminder discuss 6pm conference call with Bruno at 4pm

TemporalTokens

A complete set of tokens and related classes for representing time

  • Point in time, e.g. today at 5pm
  • Approximate time, e.g. who called at 5pm today
  • Finite sequence, e.g. every Thursday in May 2009
  • Infinite sequence, e.g. every Thursday
  • Ambiguous time with context, e.g. remind me on Tuesday (context means it is next Tuesday)
  • Null time
  • Unknowable/incomprehensible time

TemporalTokens (Cont.)

Code to merge any sequence of temporal tokens to the smallest canonical representation,

e.g.

the first thursday in may 2009

->
{TIMETHEFIRST the first} + {THURSDAY thursday} + {MAY in may} + {INT 2009 -> 2009}

->
[TEMPORALSETFINITESINGLEINTERVAL [Thursday 5/7/2009] ]

TemporalTokens (Cont.)

Finite TemporalClasses provide

  • A way to enumerate the DateTimeRanges they cover

    • All TemporalClasses provide

  • A LINQ expression generator and Entity-SQL expression generator allowing them to be used to query a database

    • Existing Token Types

    • Numbers (double, long, int, percentage, phone, temperature)
    • File names, Directories
    • URLs, Domain names
    • Names, Companies, Addresses
    • Rooms, Lights, Sensors, Sprinklers, …
    • States (On, Off, Dim, Bright, Loud, Quiet, …)
    • Units of Time, Weight, Distance
    • Songs, albums, artists, genres, tags
    • Temporal expressions
    • Commands, verbs, nouns, pronouns, …

    Rules – A simple rule

       /// &lt;summary&gt;
   /// Set a light to a given state
   /// &lt;/summary&gt;
   private static void LightState(NLPState st, TokenLight tlight, TokenStateOnOff ts)
   {
      if (ts.IsTrueState == true) tlight.ForceOn(st.Actor);
      if (ts.IsTrueState == false) tlight.ForceOff(st.Actor);
      st.Say("I turned it " + ts.LowerCased);
   }

    Any method matching this signature is a sentence rule:-  NLPState, Token*

    Rule matching respects inheritance, and variable repeats … (NLPState st, TokenThing tt, TokenState tokenState, TokenTimeConstraint[] constraints)

    Rules are discovered on startup using Reflection and an efficient parse graph is built allowing rapid detection and rejection of incoming sentences.

    State – NLPState

    • Every sentence method takes an NLPState first parameter
    • State includes RememberedObject(s) allowing sentences to react to anything that happened earlier in a conversation
    • Non-interactive uses can pass a dummy state
    • State can be per-user or per-conversation for non-realtime conversations like email
      • Chat (e.g Jabber/Gtalk)
      • Web chat
      • Email
      • Calendar (do X at time Y)
      • Rich client application
      • Strongly-typed natural language engine
      • Compile time checking, inheritance, …
      • Define tokens and sentences (rules) in C#
      • Strongly-typed tokens: numbers, percentages, times, dates, file names, urls, people, business objects, …
      • Builds an efficient parse graph
      • Tracks conversation history

      • User Interface

      Works with a variety of user interfaces

      Summary

      Future plans

      Expanded corpus of knowledge

    • Company names, locations, documents, …

      • Generate iCal/Gdata Recurrence

    • From TimeExpressions