
Natural language processing, understanding and generation has been a passion of mine for over a decade. I first started implementing it for my [home automation system](/2010/09/speaking-talking-home-automation/) but it grew into a major project of its own. At the time there was nothing that could do what I wanted, and there still isn't anything that has all the functionality my engine offers.

For starters I wanted to be able to build natural language understanding (NLU) features directly into my programs. In a typical system offered today to integrate NLU into your app the notion of recognition is totally separate from the `intents` that you implement in your application and often, all you have is a web API calling your code with some 'intent' data. By contrast, in my system, your code can interact with the natural language engine, updating the vocabulary on the fly, interpreting new tokens that you add to your code, adding new intents in one step along with the sentence structures that represent them, and, most importantly of all, delivering strongly-typed tokens that you can interact with directly including units of measure (distance, time, temperature, ...) and temporal expressions (Monday, last week, three weeks ago). You can also take the tokens passed to your rules and ask to conjugate them, so when you ask "Create a task ..." it can reply "I created a task ...". Strong integration with your own code (rather than a web api call) means you can do database lookups to add new words or phrases.

Along the way I realized that the engine I had built really didn't need spaces to go a good job and that I could extend it to allow 'twitter-speak' where peoplerunthewordstogether.

Integration with WordNet from Princeton University provides a rich vocabulary with synonyms so now you can define sentences using a meaning instead of a word and it will recognize all the words that could have that meaning. I compile all of WordNet into two Nuget packages: one with the more common words, one with the less common words. Compiling it into code means that my engine can use Visual Studio's Intellisense to show meanings when you hover over a word. This helps you find the right sense for a word when there are multiple possble meanings.

You can read more about my [Natural Language Engine for .NET here](http://nlp.abodit.com). You'll also find more blog posts about [#nlp here](/topics/nlp/).


I could not find a Natural Language Processing engine when I needed one for my home automation system so I developed my own. After 10 years of on and off development I now have a unique NLP engine for .NET that is easy to configure but incredibly powerful for precise command and control applications. It doesn't use a tokenizer so it doesn't care if you run words together.

Natural Language Processing


Current user interfaces are primarily limited to a single input device.
We type on a keyboard, we point and drag with a mouse or we speak to an
application that accepts only voice. Our computer doesn't know if we are
sat in front of it, let alone whether we are talking to someone else or
looking at something else. As a result computers appear stupid and rude.
They are oblivious to our frantic mouse movements on screen 2 while we
are looking at screen 1 wondering where the mouse pointer is hiding.
They pop up messages and make noises even though we are busy with
another task or in the middle of a conversation with someone else.

This will change. In the future computers will become more aware of
their surroundings and of their users. They will track your eyes, they
will listen constantly to what you are saying and what is going on in
the room around them and they will begin to appear to be smarter and a
whole lot less rude than they are today.

But I also think that UI will become multimodal. We will combine mouse
movement with speech, computers will ask questions and learn. For
example, I was just working on a repetitive edit on a block of code. In
the future computers will ask, using a spoken voice, whether we want
them to apply the same edit to the remainder of the lines in that block.
And because the interaction is multi-modal it does not interrupt the
flow of what you are doing. There is no pop-up message to distract you
visually, there is no dismiss button you need to click to make Clippy go
away. As humans we do fairly well at isolating different inputs (visual,
verbal, auditory, tactile, ...) and can ignore one while we focus on
another. Multi-modal UI will take advantage of that and will appear less
intrusive and far less annoying than previous attempts at assistive
technology.

But even without computerized assistants, multimodal UI will become
common. You will be able to select a block of text and say "make it
larger" without having to go find the right command in a dialog launched
from a menu. As we move from desktops to tablets this capability becomes
even more important because we have less screen real-estate and because
typing and mousing are harder.

Multimodal UI also benefits from the combinatorial, multiplicative
effect of different inputs. You only need one command "make it larger"
rather than make paragraph larger, make image larger, make space between
lines larger, make margin larger, ...

Multimodal UI also saves a lot of mouse movement and solves many of the
issues around selections (e.g. do they go away when you select a menu
item, or click on a different application).

Multimodal UI will also dramatically reduce how often you need to click:
merely pointing at something and saying the command will be sufficient.

I strongly believe that this will be a key part of future user
interfaces, but in the meantime, let me go dismiss this dialog asking if
I want to reboot to install updates...


The future of user interface design is multi-modal and context aware


One of the more challenging aspects of understanding natural language is
dealing with date and time expressions. There are many different ways a
user could refer to a specific date and time. They might say "Next
Tuesday at 4pm", they might give a specific date in any of several
different forms, they might refer to other time ranges "First Tuesday in
January 2012 at 4pm" etc.

Whilst my natural language engine can't understand every possible date
time expression (e.g. the second Wednesday after the first Friday in May
2010) it does handle a huge variety.

Clearly the .NET provided DateTime class is wholly inadequate to express
the kinds of date/time expressions your users might enter. To deal with
that I've created my own classes that represent items like a specific
**Time** of day, a **DateTimeRange**, a **DateTimeRangeCollection**, ...

**TemporalSets** are the most general result of parsing a datetime
expression since they can represent any date time expression. Broadly
they split into two categories: finite and infinite expressions.
"Tuesday at 5pm" is an infinite time expression. "Tuesday at 5pm January
2012" is a finite time expression. Sometimes you will want to accept an
infinite expression and interpret it as a future or past finite
occurrence. For that I have a MergePreferPast and MergePreferFuture
method that operate on a **TemporalSetCollection**. The demonstration
code on [BitBucket](http://nlp.abodit.com/home/download)shows this in
action.

**TemporalSets** also have unique capabilities around both providing
query expressions (for database searches) and generative expressions
(for adding dates to a calendar).

If you'd like to try out the latest date / time expression parsing code
in my [Natural Language Engine](http://nlp.abodit.com) you can visit the
[demo](http://nlp.abodit.com/home/demo) and try typing "define" followed
by a date / time expression.

If you find any expressions that ought to work, please feel free to
email or Tweet them to me.

Here's a sample session, you can try it out on [nlp.abodit.com](https://nlp.abodit.com/demo/datetime-input):

```markup
define june 23rd 2010 
Absolute:[DATETIMERANGE 6/23/2010 at 12:00 AM to 6/23/2010 at 11:59 PM]

define January 19th 
Future:[Thursday 1/19/2012], [Saturday 1/19/2013], [Sunday 1/19/2014], [Monday 1/19/2015], [Tuesday 1/19/2016], [Thursday
1/19/2017], [Friday 1/19/2018], [Saturday 1/19/2019], [Sunday 1/19/2020], [Tuesday 1/19/2021] Past: [Wednesday 1/19/2011] ...
```


Understanding Dates and Times in Natural Language


I've been working to get a release of my NLP Engine out the door but wanted
to boost the built in dictionary / thesaurus before release. So this
weekend I integrated Wordnet into the engine. Wordnet is available in
RDF as a series of triples. As well as all the word definitions grouped
into synonym sets (synsets) it also includes relationships like class
relationships '[hyponym](http://en.wikipedia.org/wiki/Hyponymy)' or
'isa' and part relationships
'[meronym](http://en.wikipedia.org/wiki/Meronym)' or
'[holonym](http://en.wikipedia.org/wiki/Holonymy)'.

By building a simple in-memory graph of all these relationships my
engine can now use them to infer interface types on objects. To define
an interface corresponding to a Wordnet synset like a mammal
(wn30:synset-mammal-noun-1) you simply define an interface in the
namespace 'Noun' having a name of 'mammal1'. Through the type
inheritance specified in the Wordnet file all mammals now inherit that
interface automatically and you can write natural language rules that
ask for a mammal and they will get any type of mammal defined in
Wordnet.


Integrating Wordnet with Natural Language Processing (NLP)


Here's a working example of the kind of semantic and mathematical summarization possible.

One of my long-term project ideas that is gradually coming together
piece-by-piece is a system I call "Remember Everything" (for want of a
better name).

"Remember Everything" connects nearly all of my projects into one giant
solution that, well, *remembers everything* and has a natural language
interface over it.

As inputs it will take information from my home automation system, my
[whole-network storage
crawler](/posts/2011-08-home-network-crawler-storage-file-mongodb/),
Google calendar, email, Twitter, blog, web crawler, an
address-monitoring browser add-on I plan to write, the weather and
traffic feeds, and, of course my [natural language
engine](http://nlp.abodit.com).

All this data will be put into MongoDB and can then be queried.
Relationships between entities will be created using a semantic-web
triple store and reasoner.

Together these capabilities will allow queries like:-

* Copy all the photos I took last week onto c:\\vacationPhotos
* Send img\_0938.jpg to mum.
* Who called last Monday?
* Show pictures from last month taken on sunny days.
* What was happening two weeks ago when X called?
* Who called yesterday when I was in a meeting?
* What song was playing around 9pm last night?
* How long did I spend on the phone to my accountant last week?
* What web pages did I read last week about the Semantic Web?
* Send the web page I tweeted about last night to my Kindle.
* We need butter and olives.
* What do I need to buy from QFC? (a semantic shopping list concept, more on that later ...)

In addition to the shopping lists concept (that's already in my home
automation system but lacks the semantic reasoning) the system will take
any subject-verb-object phrase and remember it and then allow you to
query it back later, e.g.

* My son read 20 pages tonight *(making the weekly reading report
easier)*
* How many pages did he read this week?
* I took the red pill at 10AM
* I walked 2 miles this morning
* I ran 4 miles
* How much exercise did I do this week when it wasn't raining? (summarizing values
semantically and mathematically)
* The Audi was serviced this week *(remembering schedules so you can check if an item is overdue)*
* My BA frequent flyer number is #### *(remembering numbers you need to look up often)*
* I took the day off on friday *(vacation reporting)*
* I spent \$12.95 on lunch *(expense reporting)* ...

Whenever you have anything you need to remember the system will be able
to remember it, recall it, and where possible aggregate or summarize it
using math and/or semantic reasoning (e.g. running subClassOf exercise,
butter subClassOf dairy product, dairy products areSoldAt QFC, ...).

By linking my natural language engine to a triple store I can even allow
users to teach it new concepts:

[![Using Natural Language to populate a triple store](/images/uploads/2011/09/TeachNewMeaning.png "Using Natural Language to populate a triple store")](/images/uploads/2011/09/TeachNewMeaning.png)

By silently monitoring your email, Twitter stream, calendar, activity in
the house, ... **it will be able to answer questions based on the
context not just on the content** in ways that we take for granted as
humans but which are not possible for computers today.


Remember Everything ... a long-term project


What if ... you could define and query an Ontology using natural language?
==========================================================================

Tonight as an experiment I connected my NLP engine to my MongoDB
triple-store and built a reasoner that does simple-entailment over just
RDFS:subClassOf and OWL:disjointWith

Try it out at http://nlp.abodit.com/demo/demo

Ignore the instructions on the right and try building a simple ontology,

Start by creating some root level class:

car is a class

Now expand it:

```markup
	a BMW is a car
	an Audi is a car
	what is a BMW 
	*BMW is a car*
	what is an Audi
	*Audi is a car*
	is an Audi a BMW
	*I don't know, under the open world assumption it could be*
	an Audi is not a BMW
	is an Audi a BMW *No*
```

Simple but promising.


What if ... you could define and query an Ontology using natural language?


In an [earlier
post](/2011/03/a-semantic-web-ontology-driven-approach-to-crm)
I discussed some ideas around a [Semantic
CRM](/2011/03/a-semantic-web-ontology-driven-approach-to-crm).

Recently I've been doing some clean up work on my C\# Natural Language
Engine and decided to do a quick test connecting it to a real CRM. As
you may know from reading my blog, this natural language engine is
already heavily used in my home automation system to control lights,
sprinklers, HVAC, music and more and to query caller ID logs and other
information.

I recently refactored it to use the Autofac dependency injection
framework and in the process realized just how close my NLP engine is to
ASP.NET MVC 3 in its basic structure and philosophy! To use it you
create Controller classes and put action methods in them. Those
controller classes use Autofac to get all of the dependencies they may
need (services like an email service, a repository, a user service, an
HTML email formattting service, ...) and then the methods in them
represents a specific sentence parse using the various token types that
the NLP engine supports. Unlike ASP.NET MVC3 there is no Route
registration; the method itself represents the route (i.e. sentence
structure) that it used to decide which method to call. Internally my
NLP engine has its own code to match incoming words and phrases to
tokens and then on to the action methods. In a sense the engine itself
is one big dependency injection framework working against the action
methods. I sometimes wish ASP.NET MVC 3 had the same
route-registration-free approach to designing web applications (but also
appreciate all the reasons why it doesn't).

Another improvement I made recently to the NLP Engine was to develop a
connector for the [Twilio](http://twilio.com) SMS service. This means
that my home automation system can now accept SMS messages as well as
all the other communication formats it supports: email, web chat, XMPP
chat and direct URL commands. My Twilio connector to NLP supports
message splitting and batching so it will buffer up outgoing messages to
reach the limit of a single SMS and will send that. This lowers SMS
charges and also allows responses that are longer than a single SMS
message.

Using this new, improved version of my Natural Language Engine I decided
to try connecting it to a CRM. I chose Microsoft Dynamics CRM 2011 and
elected to use the strongly-typed, early-bound objects that you can
generate for any instance of the CRM service. I added some simple
sentences in an NLPRules project that allow you to tell it who you met,
and to input some of their details. Unlike a traditional forms-based
approach the user can decide what information to enter and what order to
enter it in. The Natural Language Engine supports the concept of a
conversation and can remember what you were discussing allowing a much
more natural style of conversation that some simple rule-based engines
and even allowing it to ask questions and get answers from the user.

Here's a screenshot showing a sample conversation using Google Talk
(XMPP/Jabber) and the resulting CRM record in Microsoft CRM 2011 Online.
You could have the same conversation over SMS or email. Click to
enlarge. [A natural language interface to
CRM](/images/uploads/2011/06/NLPCRM-300x171.png#center "A natural language interface to CRM")

Based on my limited testing this looks like another promising area where
a truly fluent, conversational-style natural language engine could play
a significant role. Note how it understands email addresses, phone
numbers and such like and in code these all become strongly typed
objects. Where it really excels is in temporal expressions where it can
understand things like "who called on a Saturday in May last year?" and
can construct an efficient SQL query from that.


C# Natural Language Engine connected to Microsoft Dynamics CRM 2011 Online


I was inspired by a question on [semanticoverflow.com](http://semanticoverflow.com) which asked if there
was a language in which the concepts of the Semantic Web could be
expressed directly, i.e. you could write statements and perform
reasoning directly in the code without lots of parentheses, strings and
function calls.

Of course the big issue with putting the semantic web into .NET is the
lack of multiple inheritance. In the semantic web the class 'lion' can
inherit from the 'big cat' class and also from the 'carnivorous animals'
class and also from the 'furry creatures' class etc. In C\# you have to
pick one and implement the rest as interfaces. But, since C\# 4.0 we
have the dynamic type. Could that be used to simulate multiple
inheritance and to build objects that behave like their semantic web
counterparts?

The DynamicObject in C# allows us to perform late binding and
essentially to add methods and properties at runtime. Could I use that
so you can write a statement like "canine.subClassOf.mammal();" which
would be a complete Semantic Web statement like you might find in a
normal triple store but written in C\# without any 'mess' around it.
Could I use that same syntax to query the triple store to ask questions
like "if (lion.subClassOf.animal) ..." where a statement without a
method invocation would be a query against the triple store using a
reasoner capable of at least simple transitive closure? Could I also
create a syntax for properties so you could say `lion.Color("yellow")`
to set a property called Color on a lion?

Well, after one evening of experimenting I have found a way to do just
that. Without any other declarations you can write code like this:

```csharp
dynamic g = new Graph("graph");

// this line declares both a mammal an an animal
g.mammal.subClassOf.animal();

// we can add properties to a class g.mammal.Label("Mammal");
// add a subclass below that g.carnivore.subClassOf.mammal();
// create the cat family g.felidae.subClassOf.carnivore();
// define what the wild things are - a separate hierarchy of things
g.wild.subClassOf.domesticity();

// back to the cat family tree g.pantherinae.subClassOf.felidae();
// these one are all wild (multiple inheritance at work!)
g.pantherinae.subClassOf.wild();
g.lion.subClassOf.pantherinae();

// experiment with properties
// these are stored directly on the object not in the triple store 
g.lion.Color("Yellow");

// complete the family tree for this branch of the cat family
g.tiger.subClassOf.pantherinae(); 
g.jaguar.subClassOf.pantherinae();
g.leopard.subClassOf.pantherinae();
g.snowLeopard.subClassOf.leopard();
```
	
Behind the scenes dynamic objects are used to construct partial
statements and then full statements and those full statements are added
to the graph. Note that I'm not using full Uri's here because they
wouldn't work syntactically, but there's no reason each entity couldn't
be given a Uri property behind the scenes that is local to the graph
that's being used to contain it.

Querying works as expected: just write the semantic statement you want
to test. One slight catch is that I've made the query return an
enumeration of the proof steps used to prove it rather than just a
simple bool value. So use \`.Any()\` on it to see if there is any proof.

```csharp
// Note that we never said that cheetah is a mammal directly.
// We need to use inference to get the answer. 
// The result is an enumeration of all the ways to prove that 
// a cheetah is a mammal
var isCheetahAMammal = g.cheeta.subClassOf.mammal;

// we use .Any() just to see if there's a way to prove it 
Console.WriteLine("Cheetah is a wild cat : " + isCheetahAMammal.Any());
```

Behind the scenes the simple statement "g.cheeta.subClassOf.mammal" will
take each statement made and expand the subject and object using a
logical argument process known as simple entailement. The explanation it
might give for this query might be:

```markup
> because [cheeta.subClassOf.felinae], [felinae.subClassOf.felidae],
> [felidae.subClassOf.mammal]
```

As you can see, integrating Semantic Web concepts [almost] directly into
the programming language is a pretty powerful idea. We are still nowhere
close to the syntactic power of prolog or F\# but I was surprised how
far vanilla C\# could get with dynamic types and a fluent builder. I
hope to explore this further and to publish the code sometime. It may
well be "the world's smallest triple store and reasoner"!

This code will hopefully also allow folks wanting to experiment with
core semantic web concepts to do so without the 'overhead' of a
full-blown triple store, reasoner and lots of RDF and angle brackets!
When I first came to the Semantic Web I was amazed how much emphasis
there was on serialization formats (which are boring to most software
folks) and how little there was on language features and algorithms for
manipulating graphs (the interesting stuff). With this experiment I hope
to create code that focuses on the interesting bits.

The same concept could be applied to other in-memory graphs allowing a
fluent, dynamic way to represent graph structures in code. There's also
no reason it has to be limited to in-memory graphs, the code could
equally well store all statements in some external triple store.

The code for this experiment is available on bitbucket:
[https://bitbucket.org/ianmercer/semantic-fluent-dynamic-csharp](https://bitbucket.org/ianmercer/semantic-fluent-dynamic-csharp)


Extending C# to understand the language of the semantic web


Photo credit: [tompagenet](http://www.flickr.com/photos/73422480@N00/4951687101/ "tompagenet")

Suppose for a minute that your house could talk.  What would you want it to tell you?

That's a question I've been considering over the past few years because unlike 
most of you, my house can talk!  There are ceiling speakers in many rooms, the 
house knows which rooms are occupied, and it talks to you when it has information
it needs to communicate.

I've experimented with many different announcements over the years, some
of which turned out to be useless, some which just became annoying after
too many repetitions, and a few that withstood the test of time and
daily usage to become fixtures in my smart home.

Here's a list of some of the current announcements my house makes:-

​1. It announces who is calling.  "Talking Caller ID" if you will,
except this one has a database behind it and whenever it sees a number
it doesn't recognize it will ask you who it is so it knows how to speak
their name next time.  (The asking is done over instant messenger or
email, a separate topic I'll write about soon.)

​2. It announces that you have missed phone calls.  It does this when
the house has been unoccupied for a while and you walk back in.  It
doesn't do it right away because you are probably carrying stuff and
banging doors.  Instead, it waits a moment and then when it detects you
moving again in the house it announces all the calls you missed while
you were away.

​3. It announces the weather for the day in the morning when you get up
and head into the bathroom.

​4. It announces the weather for tomorrow if it's already afternoon time
today and you turn on any media zone.  But, of course, only if it hasn't
announced it for a while (repetition gets boring fast!).

​5. It can announce the artist and title of each song as it is playing.
 A simple chat command *'dj on'* enables this feature.  At the start of
every song it 'ducks' the music, announces the song, and then brings the
volume back up, just like a professional voice-over announcement.

​6. It tells you if the traffic is bad on the way into Bellevue this
morning, where 'bad' is defined as more than seven minutes above the
average for this day of the week.  Knowing that it's bad on Monday
morning isn't useful.  Knowing that it's much worse than normal is.

​7. It announces the time every 10 minutes in the bathroom in the
morning rush helping everyone stay on track.

​8. It announces when it's time to wake up using a schedule defined on
Google calendar.  This means you can adjust the alarm clock as easily as
you can change an appointment, no fiddly black on black buttons under an
alarm clock to push.  It also means that the alarm can handle weekdays
vs weekends and holidays.

​9. It announces if there has been fresh snow overnight at the local ski
resort: time to go skiing!  But it also warns you if the pass is closed
as it sometimes is in winter after a heavy snowfall overnight.

​10. It announces that it's garbage collection day if it thinks you
haven't taken the trash out.

​11. It warns you that the garage doors are open if you try to go
upstairs in the evening leaving them open.  That's a bit like HAL
controlling the pod bar doors but in reverse: it tells you to close
them!

​12. It complains that the fish are hungry if nobody has opened the door
to their tank for 24 hours.  Each 6 hours after that the announcements
become more urgent, culminating in a "mine mine mine" audio clip from
Finding Nemo.  (All our other pets do a fine job at indicating their
hunger but the fish lacked this ability until now).

​13. It says "good night" when you turn the lights off at the end of the
day :)

​14. You can also ask it to remind you of anything with a verbal
reminder, e.g. the command *'remind me in 15 minutes say "Turn oven on"
in study'* does exactly what it says.

​15. It tells us if anyone goes into our barn unexpectedly.

​16. It can also scare intruders when it detects unusual motion that
doesn't fit the normal pattern - nobody expects a house to talk to them
when they are burglarizing it!  Fortunately I've never had to test this
feature.

Crucially it does very few of these announcements when it detects that
we have visitors.  Party music should not be interrupted by home
automation announcements!  It also learned early on that any
announcement in a bedroom when the owners were asleep was a very, very
bad idea punishable by instant termination and recompilation!


If your house could talk to you, what would it say?


This weeks side-project was the creation of an ontology store using
Entity Framework 4. An ontology store stores axioms consisting of
Subject, Predicate, Object which are usually serialized as RDF, OWL, N3,
... Whereas there's lots of details about these serialization formats,
the actual mechanics of how to store and manipulate them was somewhat
harder to come by. Nevertheless, after much experimentation I came up
with an Entity Model that can store Quads (Subject, Predicate, Object
and Meta) or Quins (Subject, Predicate, Object, Meta, Graph). The
addition of Meta allows one Axiom to reference another. The addition of
Graph allows the store to be segmented making it easy to import some N3
or RDF into a graph, then flush that graph if it is no longer needed or
if a newer version becomes available.

The store is currently hooked up to an Euler reasoner that can reason
against it, lazily fetching just the necessary records from the SQL
database that backs the Entity Model.

Above is the EDMX showing how I modeled the Ontology Store:


An ontology triple (quad) store for RDF/OWL using Entity Framework 4


Recently I've been considering how the Semantic Web will impact home
automation.

Technologies like the [Web Ontology Language
(OWL)](http://en.wikipedia.org/wiki/Web_Ontology_Language) and RDF allow
for the construction of complex ontologies that define what things are,
and how they relate. Using these ontologies automated reasoning can be
applied to generate new facts or to prove or disprove assertions.

This sounds like the ideal companion to the Natural Language Processing
(NLP) Engine that I have already created for the my home automation
system. With reasoning powers added to the natural language engine and
the ability to augment the knowledge base by adding new assertions the
whole system will be much more powerful. One day it might even be
possible to create the entire home definition using a natural language
text file and to query the system using rich natural language queries.

So, the first step is to find an existing ontology store and reasoning
engine. A quick web search reveals that most are built in Java. There
were a couple of links I came across later for .NET:
[http://razor.occams.info/code/semweb/](http://razor.occams.info/code/semweb/)
and
[http://www.intellidimension.com/products/semantics-server/](http://www.intellidimension.com/products/semantics-server/).
There's also an interesting Q&A site at http://semanticoverflow.com
which has lots of useful information on it.

But rather than starting with some existing library I really wanted to
understand more deeply how an ontology store works and how a reasoning
engine functions, so over the course of a couple of evenings I created
my own. I now have a triple store and a simple reasoning engine. Here's
an *actual conversation* so you can see what it's capable of so far and
can perhaps get a glimpse at how powerful this concept could be:-

````text
house is a class
contains is a property
contains is transitive
first floor is a class 
room is a class 
kitchen is a room 
first floor contains kitchen 
house contains first floor
does house contain kitchen?
House: Yes, house contain kitchen because [house contains first floor]
 ... [first floor contains kitchen]
````

As you can see my semantic store can already represent classes,
relationships between classes, new relationships ('contains'),
relationships between relationships ('same as').  For such a small
amount of code it's quite surprising what this system can now handle in
terms of knowledge representation and simple reasoning.

Next time I get some spare time I'll hook it up to the actual home model
so you can start to query that in much more powerful ways than before.

Stay tuned!


Applying the Semantic Web to Home Automation


Please visit [nlp.abodit.com](https://nlp.abodit.com) for information about my natural language engine. The
information below is out of date.


Here is an explanation of the natural language engine that powers my
home automation system. It's a strongly-typed natural language engine
with tokens and sentences being defined in code. It currently
understands sentences to control lights, heating, music, sprinklers, ...
You can ask it who called, you can tell it to play music in a particular
room, ... it tells you when a car comes down the drive, when the traffic
is bad on I-90, when there's fresh snow in the mountains, when it finds
new podcasts from NPR, ... and much more.

The natural language engine itself is a separate component that I hope
one day to use in other applications.

Existing Natural Language Engines
---------------------------------

* Have a large, STATIC dictionary data file
* Can parse complex sentence structure
* Hand back a tree of tokens (strings)
* Don’t handle conversations

C# NLP Engine
--------------

* Defines strongly-typed tokens in code
* Uses type inheritance to model ‘is a’
* Defines sentences in code
* Rules engine executes sentences
* Understands context (conversation history)

Sample conversation
-------------------

[![Sample Conversation](/images/uploads/2010/02/SampleConversation.png "Sample Conversation with Natural Language Engine (NLP)")](/images/uploads/2010/02/SampleConversation.png)


Goals
-----

* Make it easy to define tokens and sentences (not XML)
* Safe, compile-time checked definition of the syntax and grammar (no XML)
* Model real-world inheritance with C\# class inheritance: ‘a labrador’ is ‘a dog’ is ‘an animal’ is ‘a thing’
* Handle ambiguity,

    e.g.

```csharp
    play something in the air tonight in the kitchen
    remind me at 4pm to call john at 5pm
```

C\# NLP Engine Structure
------------------------

[![](/images/uploads/2010/02/NLPStructure.png "Natural Language Engine Structure")](/images/uploads/2010/02/NLPStructure.png)

Tokens - Token Definition
-------------------------

* A hierarchy of Token-derived classes
* Uses inheritance, e.g. **TokenOn** is a **TokenOnOff** is a
    **TokenState** is a **Token**. This allows a single sentence rule to handle multiple cases, e.g. On and Off
* Derived from base Token class
* Simple tokens are a set of words, e.g. « is | are »
* Complex tokens have a parser, e.g. TokenDouble

A Simple Token Definition
----

````csharp
public class TokenPersonalPronoun : TokenGenericNoun 
{
    internal static string wordz { get { return "he,him,she,her,them"; } } 
}
````

* Recognizes any of the words specified
* Can use inheritance (as in this example)

A Complex Token
----

```csharp
public abstract class TokenNumber : Token 
{ 
    public static IEnumerable\<TokenResult\> Initialize(string input) { … 

-   Initialize method parses input and returns one or more possible
    parses.
````
**TokenNumber** is a good example:

-   Parses any numeric value and returns one or more of TokenInt, TokenLong, TokenIntOrdinal, TokenDouble, or TokenPercentage results.

The catch-all **TokenPhrase**
-----------------------------

```csharp
public class TokenPhrase : Token
```

TokenPhrase matches anything, especially anything in quote marks

  e.g. add a reminder "call Bruno at 4pm"

The sentence signature to recognize this could be

````csharp
    (…, TokenAdd, TokenReminder, TokenPhrase, TokenExactTime)
````

This would match the rule too …

````csharp
    add a reminder discuss 6pm conference call with Bruno at 4pm
````

TemporalTokens
----

A complete set of tokens and related classes for representing time

* Point in time, e.g. today at 5pm
* Approximate time, e.g. who called at 5pm today
* Finite sequence, e.g. every Thursday in May 2009
* Infinite sequence, e.g. every Thursday
* Ambiguous time with context, e.g. remind me on Tuesday (context
    means it is next Tuesday)
* Null time
* Unknowable/incomprehensible time

TemporalTokens (Cont.)
----

Code to merge any sequence of temporal tokens to the smallest canonical
representation,

e.g.

````csharp
the first thursday in may 2009
   -> {TIMETHEFIRST the first} + {THURSDAY thursday} + {MAY in may} + {INT 2009 -\> 2009}
    -> [TEMPORALSETFINITESINGLEINTERVAL [Thursday 5/7/2009] ]
````

TemporalTokens (Cont.)
----

Finite **TemporalClasses** provide

A way to enumerate the DateTimeRanges they cover

All **TemporalClasses**provide

A LINQ expression generator and Entity-SQL expression generator allowing
them to be used to query a database

Existing Token Types
----

* Numbers (double, long, int, percentage, phone, temperature)
* File names, Directories
* URLs, Domain names
* Names, Companies, Addresses
* Rooms, Lights, Sensors, Sprinklers, …
* States (On, Off, Dim, Bright, Loud, Quiet, …)
* Units of Time, Weight, Distance
* Songs, albums, artists, genres, tags
* Temporal expressions
* Commands, verbs, nouns, pronouns, …

Rules - A simple rule
----

````csharp
	/// <summary>
	/// Set a light to a given state 
	///</summary>
	private static void LightState(NLPState st, TokenLight tlight, TokenStateOnOff ts)
	{
		if (ts.IsTrueState == true)
			tlight.ForceOn(st.Actor); 
		if (ts.IsTrueState == false)
			tlight.ForceOff(st.Actor);
		st.Say("I turned it " + ts.LowerCased); 
	}
````

Any method matching this signature is a sentence rule:-  NLPState, Token\*

Rule matching respects inheritance, and variable repeats … (NLPState st,
TokenThing tt, TokenState tokenState, TokenTimeConstraint[] constraints)

Rules are discovered on startup using Reflection and an efficient parse
graph is built allowing rapid detection and rejection of incoming
sentences.

State - NLPState
----------------

* Every sentence method takes an NLPState first parameter
* State includes RememberedObject(s) allowing sentences to react to
    anything that happened earlier in a conversation
* Non-interactive uses can pass a dummy state
* State can be per-user or per-conversation for non-realtime
    conversations like email
    -   Chat (e.g Jabber/Gtalk)
    -   Web chat
    -   Email
    -   Calendar (do X at time Y)
    -   Rich client application

    -   Strongly-typed natural language engine
    -   Compile time checking, inheritance, …
    -   Define tokens and sentences (rules) in C\#
    -   Strongly-typed tokens: numbers, percentages, times, dates, file
        names, urls, people, business objects, …
    -   Builds an efficient parse graph
    -   Tracks conversation history
* Company names, locations, documents, …
* From TimeExpressions


A strongly-typed natural language engine (C# NLP)


One of the features of my home automation system is the ability to call
the house up and speak to it to ask it to do things. To enable this I'm
using Voxeo and their [excellent developer
program](http://www.voxeo.com/developers/evolution-ivr-developer.jsp)
which gives me an inbound phone number and the ability to host a
VoiceXML application with voice recognition and text to speech.

Using Linq-to-XML I can build the required VoiceXML hand it off to Voxeo
when they request it during an incoming call, e.g.

```csharp
 XDocument doc = new XDocument( new XDeclaration("1.0", "utf-8",  "yes"), 
    new XComment("Voice XML feed"), new XElement("vxml", 
    new XAttribute("version", "2.1"), 
    new XElement("form", new XAttribute("id", "MainMenu"), 
    new XElement("field", new XAttribute("name", "room"), 
    hereswhatIdid, 
    new XElement("grammar", new XAttribute("type", "text/gsl"), new XCData( "[" + @" [ " +  WholeHouseGrammar() + " ]" + "]")), 
    new XElement("noinput", "I did not hear anything. Please try again", 
    new XElement("reprompt")), new XElement("nomatch", "I did not recognize that command. Please try again", new XElement("reprompt"))), ...
```

The 'WholeHouseGrammar' is generated on the fly by examining the
hierarchy of objects in the house so it includes every room, light,
sprinkler, ...

At present the house only understands on/off commands but in the future
I hope to hook it up to the natural language engine so that the grammar
is instead constructed dynamically from every sentence that the house
understands today when you chat to it.

So currently you say things like 'Front right sprinklers on' when you
notice the lawn needs an extra cycle as you drive out the house.  In the
future you'll be able to ask it more complex questions like 'who called
last week on friday after 4pm?' which is a question it can already
handle using the chat interface.


Home Automation with Voxeo (VoxML/Voice XML)


Welcome to my blog. In here you'll find articles about my multi-decade-long pursuit of the ["world's smartest house"](/pages/home-automation) and other passions. My smart house technology controls heating, lighting, sprinklers, music and more. It talks over the speakers, it plays music by tags in any collection of rooms, and you can even chat to it on messenger using natural language.

Home Automation
----

Articles about [#homeautomation](/topics/home-automation/) describe a system that includes over 200 sensors within a house and nearby barn that together create an environment where you don't need to touch a light switch during the course of a typical day, where the heating operates in an optimum manner, and where music plays across multiple rooms controlled by a natural language interface.

You'll find blog posts on my [favorite features](/posts/2010-09-home-automation-top-features), a discussion on representing [the home as a graph](/posts/2014-11-home-automation-systems-graph) and much more.

You'll find posts about [#sensors](/topics/sensors/) including  [strain-gauges](/posts/sensors-strain-gauge/), PIR, optical beam, microwave-doppler and Bluetooth including the pros and cons of each technology and my experiences with them.

If you want to dive into the technical details around how the house became smart you'll find articles on many of the techniques I've tried including things like [#multiple-hypothesis-tracking](/posts/2018-02-2018-02-11-multiple-hypothesis-tracking/). So take a look at [#homeautomation](/topics/home-automation).

![Natural language meets home automation](/images/NlpScreenShot.png)

Natural Language Engine
----

You'll find a few articles here about the history behind my natural language engine for .NET, but head over to [nlp.abodit.com](https://nlp.abodit.com) for the full picture. My [NLP engine](/posts/nlp) is unique in the way it integrates into an existing .NET desktop or server application. Responding to sentences is as simple as writing a method with the permutations of tokens you want to recognize. It offers full intellisense support for *meanings* (Synsets) with dictionary-like definitions. It handles permutations and optional phrases, it handles grammar production rules and it includes a large set of existing tokens that can recognize numbers, formulas, units of measure and the widest set of temporal expressions possible. And by temporal expressions, I don't just mean picking a datetime from the input, I mean expressions that can represent infinite ranges (`every sunday in May`) and complex expressions like `last sunday after 5pm`. My natural language engine is also a probailistic parser able to return a sequence of possible parses in descending order of likelihood. It understands some of the ambiguity in the english language like the fact that 'and' sometimes means 'or'.

Travel and Photography
----

And last, but not least, I'm also documenting some of my favorite travels and photos, so check out the Photography and soon Travel topics below.

Technical Details
----

This blog was created using `next.js`, a `ReactJS` static web-site generator. I converted an existing sample blog to Typescript and added a recommendation engine to display relevant "More reading..." articles after each article that you read. This is all driven using YAML front matter on markdown posts, the entire navigation structure is derived from a 'priority' field, a 'date' field and a 'tags' field on each post.


Over 15 years of experimentation with home automation

The Blog of Ian Mercer.

Topic: nlp