The Blog of Ian Mercer.

The future of user interface design is multi-modal and context aware

Cover Image for The future of user interface design is multi-modal and context aware

Current user interfaces are primarily limited to a single input device. We type on a keyboard, we point and drag with a mouse or we speak to an application that accepts only voice. Our computer doesn't know if we are sat in front of it, let alone whether we are talking to someone else or looking at something else. As a result computers appear stupid and rude. They are oblivious to our frantic mouse movements on screen 2 while we are looking at screen 1 wondering where the mouse pointer is hiding. They pop up messages and make noises even though we are busy with another task or in the middle of a conversation with someone else.

This will change. In the future computers will become more aware of their surroundings and of their users. They will track your eyes, they will listen constantly to what you are saying and what is going on in the room around them and they will begin to appear to be smarter and a whole lot less rude than they are today.

But I also think that UI will become multimodal. We will combine mouse movement with speech, computers will ask questions and learn. For example, I was just working on a repetitive edit on a block of code. In the future computers will ask, using a spoken voice, whether we want them to apply the same edit to the remainder of the lines in that block. And because the interaction is multi-modal it does not interrupt the flow of what you are doing. There is no pop-up message to distract you visually, there is no dismiss button you need to click to make Clippy go away. As humans we do fairly well at isolating different inputs (visual, verbal, auditory, tactile, ...) and can ignore one while we focus on another. Multi-modal UI will take advantage of that and will appear less intrusive and far less annoying than previous attempts at assistive technology.

But even without computerized assistants, multimodal UI will become common. You will be able to select a block of text and say "make it larger" without having to go find the right command in a dialog launched from a menu. As we move from desktops to tablets this capability becomes even more important because we have less screen real-estate and because typing and mousing are harder.

Multimodal UI also benefits from the combinatorial, multiplicative effect of different inputs. You only need one command "make it larger" rather than make paragraph larger, make image larger, make space between lines larger, make margin larger, ...

Multimodal UI also saves a lot of mouse movement and solves many of the issues around selections (e.g. do they go away when you select a menu item, or click on a different application).

Multimodal UI will also dramatically reduce how often you need to click: merely pointing at something and saying the command will be sufficient.

I strongly believe that this will be a key part of future user interfaces, but in the meantime, let me go dismiss this dialog asking if I want to reboot to install updates...

Related Stories

Cover Image for Natural Language Processing

Natural Language Processing

I could not find a Natural Language Processing engine when I needed one for my home automation system so I developed my own. After 10 years of on and off development I now have a unique NLP engine for .NET that is easy to configure but incredibly powerful for precise command and control applications. It doesn't use a tokenizer so it doesn't care if you run words together.

Ian Mercer
Ian Mercer
Cover Image for Digital Twins are never identical

Digital Twins are never identical

Digital Twin are an online representation of a real world object, a copy of its properties in the digital world and a way to send updated and commands to it. In effect I've been making them for years but now they have a trendy name.

Ian Mercer
Ian Mercer
Cover Image for Why smarthomes are hard

Why smarthomes are hard

Why automated learning is hard for a smart home. The perils of over-fitting, under-fitting and how the general unpredictable nature of life makes it hard to build a system that learns your behavior.

Ian Mercer
Ian Mercer
Cover Image for Event blocks

Event blocks

Home automation systems need to respond to events in the real world. Sometimes it's an analog value, sometimes it's binary, rarely is it clean and not susceptible to problems. Let's discuss some of the ways to convert these inputs into actions.

Ian Mercer
Ian Mercer
Cover Image for Logistic function - convert values to probabilities

Logistic function - convert values to probabilities

Another super useful function for handling sensor data and converting to probabilities is the logistic function 1/(1+e^-x). Using this you can easily map values onto a 0.0-1.0 probability range.

Ian Mercer
Ian Mercer
Cover Image for ATAN curve for probabilities

ATAN curve for probabilities

In a home automation system we often want to convert a measurement into a probability. The ATAN curve is one of my favorite curves for this as it's easy to map overything onto a 0.0-1.0 range.

Ian Mercer
Ian Mercer
Cover Image for Home Construction Advice

Home Construction Advice

Several years ago we did a major remodel. I did all of the finish electrical myself and supervised all of the rough-in electrical. I also put in all of the electrical system and water in our barn. I have opinions ...

Ian Mercer
Ian Mercer
Cover Image for T-Mobile home internet

T-Mobile home internet

I'm testing a T-Mobile Home Internet device as a backup to XFinity and a way to offload half our monthly traffic to avoid the XFinity 1.2TB cap

Ian Mercer
Ian Mercer
Cover Image for iBeacon meetup in Seattle - January 2015

iBeacon meetup in Seattle - January 2015

My notes on the iBeacon meetup in Seattle held in January 2015

Ian Mercer
Ian Mercer
Cover Image for Home Automation Systems as a Graph

Home Automation Systems as a Graph

Using nodes and links to represent a home and all the devices in it

Ian Mercer
Ian Mercer
Cover Image for Showing home status with just a single RGB LED

Showing home status with just a single RGB LED

Multicolored LEDs can convey a lot of information in a small space

Ian Mercer
Ian Mercer
Cover Image for A wireless sensor network using Moteino boards

A wireless sensor network using Moteino boards

The diminutive Arduino boards include a powerful transmitter/receiver

Ian Mercer
Ian Mercer

JSON Patch - a C# implementation

Ian Mercer
Ian Mercer
Cover Image for The home as a user interface

The home as a user interface

Ian Mercer
Ian Mercer
Cover Image for A Quantified House - My Talk to the Seattle Quantified Self Meetup

A Quantified House - My Talk to the Seattle Quantified Self Meetup

My talk to the Seattle Quantified Self meetup

Ian Mercer
Ian Mercer

Integrating an Android phone into my home automation system

Some new features for my home automation using an Android phone

Ian Mercer
Ian Mercer

Before there was the web there was BeebTel

Just thought I should mention that I built a web-like system before the web existed

Ian Mercer
Ian Mercer

My first programme [sic]

At the risk of looking seriously old, here's something found on a paper tape

Ian Mercer
Ian Mercer
Cover Image for The Internet of Dogs

The Internet of Dogs

Connecting our dog into the home automation

Ian Mercer
Ian Mercer

Finally got the 1U Atom Server racked up

Ian Mercer
Ian Mercer

Timelapse video using the GoPro HD Hero

Ian Mercer
Ian Mercer

Web site crawler and link checker (free)

Ian Mercer
Ian Mercer

Smart home energy savings - update for 2010

Ian Mercer
Ian Mercer

I should have created Four Square ...

Ian Mercer
Ian Mercer
Cover Image for How can I tell if my house is smart?

How can I tell if my house is smart?

Ian Mercer
Ian Mercer

Asian Gadgets

Ian Mercer
Ian Mercer

A great video explaining the Semantic Web

Ian Mercer
Ian Mercer

Interesting Twitter Posts March 15th-

Ian Mercer
Ian Mercer

Twitter links for Week beginning March 8th

Ian Mercer
Ian Mercer

Elliott 803 - An Early Computer

Ian Mercer
Ian Mercer

Facebook, social gaming and points

Ian Mercer
Ian Mercer

Useful Twitter links Feb 8-Feb 15 2010

Ian Mercer
Ian Mercer

Looking forward to the new year and our new datacenter

Historical note about moving my servers into a datacenter

Ian Mercer
Ian Mercer

When will people learn to backup?

A rant about RAID

Ian Mercer
Ian Mercer

Future proof your home with a new conduit system?

Running conduit can be expensive but maybe you don't need one to every room

Ian Mercer
Ian Mercer

Balloon Boy was much ado about nothing - Twitter

Some of the more witty comments on Twitter about the Balloon Boy hoax

Ian Mercer
Ian Mercer

Shortened URLs should be treated like a Codec ...

Expanding URLs would help users decide whether or not to click a link

Ian Mercer
Ian Mercer

Tagging File Systems

Isn't it time we stopped knowing which drive our file is on?

Ian Mercer
Ian Mercer

WMPnetwk.exe started using 50% of my CPU

Uninstalling Windows Media Player - the end of an era

Ian Mercer
Ian Mercer

Introducing Jigsaw menus

A novel UI for menus that combines a breadcrumb and a menu in one visual metaphor

Ian Mercer
Ian Mercer

Amazon Instance vs Dedicated Server comparison

Some benchmark performance for Amazon vs a dedicated server

Ian Mercer
Ian Mercer

Agile Software Development is Like Sailing

You cannot tack too often when sailing or you get nowhere. Agile is a bit like that.

Ian Mercer
Ian Mercer

AntiVirus Software is the Worst Software!

When your anti-virus software starts stealing your personal data, it's time to remove it!

Ian Mercer
Ian Mercer

Second Drobo Update

At this point things were looking up for my Drobo

Ian Mercer
Ian Mercer

It's all about disk speed

Why disk speed is the most critical aspect for most modern PCs and servers

Ian Mercer
Ian Mercer

Comcast woes and a new monitoring utility

Monitoring a cable modem using its HTML management interface

Ian Mercer
Ian Mercer

Core duo desktop machine runs cool

Ian Mercer
Ian Mercer
Cover Image for Giving up on Internet Explorer

Giving up on Internet Explorer

Ian Mercer
Ian Mercer
Cover Image for New Home Automation Server

New Home Automation Server

Ian Mercer
Ian Mercer

Dell upgrades - a pricey way to go

Ian Mercer
Ian Mercer
Cover Image for World's Smartest House

World's Smartest House

Over 15 years of experimentation with home automation

Ian Mercer
Ian Mercer