Remember Everything ... a long-term project

9/16/2011, 3:26 PM

Here's a working example of the kind of semantic and mathematical summarization possible.

One of my long-term project ideas that is gradually coming together piece-by-piece is a system I call "Remember Everything" (for want of a better name).

"Remember Everything" connects nearly all of my projects into one giant solution that, well, remembers everything and has a natural language interface over it.

As inputs it will take information from my home automation system, my whole-network storage crawler, Google calendar, email, Twitter, blog, web crawler, an address-monitoring browser add-on I plan to write, the weather and traffic feeds, and, of course my natural language engine.

All this data will be put into MongoDB and can then be queried. Relationships between entities will be created using a semantic-web triple store and reasoner.

Together these capabilities will allow queries like:-

Copy all the photos I took last week onto c:\vacationPhotos
Send img_0938.jpg to mum.
Who called last Monday?
Show pictures from last month taken on sunny days.
What was happening two weeks ago when X called?
Who called yesterday when I was in a meeting?
What song was playing around 9pm last night?
How long did I spend on the phone to my accountant last week?
What web pages did I read last week about the Semantic Web?
Send the web page I tweeted about last night to my Kindle.
We need butter and olives.
What do I need to buy from QFC? (a semantic shopping list concept, more on that later ...)

In addition to the shopping lists concept (that's already in my home automation system but lacks the semantic reasoning) the system will take any subject-verb-object phrase and remember it and then allow you to query it back later, e.g.

My son read 20 pages tonight (making the weekly reading report easier)
How many pages did he read this week?
I took the red pill at 10AM
I walked 2 miles this morning
I ran 4 miles
How much exercise did I do this week when it wasn't raining? (summarizing values semantically and mathematically)
The Audi was serviced this week (remembering schedules so you can check if an item is overdue)
My BA frequent flyer number is #### (remembering numbers you need to look up often)
I took the day off on friday (vacation reporting)
I spent $12.95 on lunch (expense reporting) ...

Whenever you have anything you need to remember the system will be able to remember it, recall it, and where possible aggregate or summarize it using math and/or semantic reasoning (e.g. running subClassOf exercise, butter subClassOf dairy product, dairy products areSoldAt QFC, ...).

By linking my natural language engine to a triple store I can even allow users to teach it new concepts:

By silently monitoring your email, Twitter stream, calendar, activity in the house, ... it will be able to answer questions based on the context not just on the content in ways that we take for granted as humans but which are not possible for computers today.