The Blog of Ian Mercer.

The 'Learning Database' or 'Why do we need so many different databases?'

It's 2011 and database management and design is still a tedious job. Software does more amazing things every day but when it comes to databases it's back to 1970 in many ways.

My ideal database would borrow from RDBMS (like SQL Server), Document databases (like MongoDB), Graph Databases and Semantic Web Triple Stores; it would be the perfect hybrid of all of these and it would configure itself to be as efficient as possible answering queries.

Initially it might start storing data using a triple store format (since that's one of the simplest forms of database and yet is capable of some of the richest expression of facts), it might use some graph database techniques to improve performance on the triple store. As triples accumulate regarding a specific subject (or object) it would automatically cluster them and use MongoDB-like documents to store the data in a way that can be retrieved efficiently (like I do in my MongoDB triple store). As more data piles up and the important columns become apparent it would switch some of the data to a relational database format and add any necessary indexes. All this would happen totally automatically.

It would of course also include map-reduce, using whatever language you want to use, and it would shard and replicate itself across servers automatically to maximize performance or reliability or whatever you want it to maximize. Advanced semantic reasoning capabilities would also be included so you can ask complex SPARQL queries expressing logic that you simply cannot do with relational databases today.

At this point the DBAs in the audience are freaking out, partly because they don't have a job any more, but also because they want to ask all the usual questions "what about constraints?", "what about foreign keys?", and of course "what about stored procedures?" because they can't live without that Cobol-esque T-SQL syntax. Well, guess what DBAs, OWL is a much more powerful way of expressing all those constraints. Using OWL and other semantic web technologies you can build complex ontologies that define how the data is represented and the constraints on it. You can even create an Ontology for your rules, and an Ontology for your Ontology for you rules, .... and so on.

I think this is where databases will end up, and I hope that CJ Date (one of the fathers of modern databases) will still be around to see it!

Related Stories

Cover Image for Time Series Data Compression

Time Series Data Compression

This new technique to compress the time series data collected by my home automation system seems to be working really well.

Ian Mercer
Ian Mercer
Cover Image for Digital Twins are never identical

Digital Twins are never identical

Digital Twin are an online representation of a real world object, a copy of its properties in the digital world and a way to send updated and commands to it. In effect I've been making them for years but now they have a trendy name.

Ian Mercer
Ian Mercer
Cover Image for Why smarthomes are hard

Why smarthomes are hard

Why automated learning is hard for a smart home. The perils of over-fitting, under-fitting and how the general unpredictable nature of life makes it hard to build a system that learns your behavior.

Ian Mercer
Ian Mercer
Cover Image for Collinearity test for sensor data compression

Collinearity test for sensor data compression

One way to reduce the volume of sensor data is to remove redundant points. In a system with timestamped data recorded on an irregular interval we can achieve this by removing co-linear points.

Ian Mercer
Ian Mercer
Cover Image for Logistic function - convert values to probabilities

Logistic function - convert values to probabilities

Another super useful function for handling sensor data and converting to probabilities is the logistic function 1/(1+e^-x). Using this you can easily map values onto a 0.0-1.0 probability range.

Ian Mercer
Ian Mercer
Cover Image for ATAN curve for probabilities

ATAN curve for probabilities

In a home automation system we often want to convert a measurement into a probability. The ATAN curve is one of my favorite curves for this as it's easy to map overything onto a 0.0-1.0 range.

Ian Mercer
Ian Mercer
Cover Image for Home Construction Advice

Home Construction Advice

Several years ago we did a major remodel. I did all of the finish electrical myself and supervised all of the rough-in electrical. I also put in all of the electrical system and water in our barn. I have opinions ...

Ian Mercer
Ian Mercer
Cover Image for T-Mobile home internet

T-Mobile home internet

I'm testing a T-Mobile Home Internet device as a backup to XFinity and a way to offload half our monthly traffic to avoid the XFinity 1.2TB cap

Ian Mercer
Ian Mercer
Cover Image for Home Automation Systems as a Graph

Home Automation Systems as a Graph

Using nodes and links to represent a home and all the devices in it

Ian Mercer
Ian Mercer
Cover Image for Showing home status with just a single RGB LED

Showing home status with just a single RGB LED

Multicolored LEDs can convey a lot of information in a small space

Ian Mercer
Ian Mercer
Cover Image for A wireless sensor network using Moteino boards

A wireless sensor network using Moteino boards

The diminutive Arduino boards include a powerful transmitter/receiver

Ian Mercer
Ian Mercer

JSON Patch - a C# implementation

Ian Mercer
Ian Mercer
Cover Image for A Quantified House - My Talk to the Seattle Quantified Self Meetup

A Quantified House - My Talk to the Seattle Quantified Self Meetup

My talk to the Seattle Quantified Self meetup

Ian Mercer
Ian Mercer

Integrating an Android phone into my home automation system

Some new features for my home automation using an Android phone

Ian Mercer
Ian Mercer

Before there was the web there was BeebTel

Just thought I should mention that I built a web-like system before the web existed

Ian Mercer
Ian Mercer

My first programme [sic]

At the risk of looking seriously old, here's something found on a paper tape

Ian Mercer
Ian Mercer
Cover Image for The Internet of Dogs

The Internet of Dogs

Connecting our dog into the home automation

Ian Mercer
Ian Mercer

Web site crawler and link checker (free)

Ian Mercer
Ian Mercer

Smart home energy savings - update for 2010

Ian Mercer
Ian Mercer

A great video explaining the Semantic Web

Ian Mercer
Ian Mercer

Elliott 803 - An Early Computer

Ian Mercer
Ian Mercer

Looking forward to the new year and our new datacenter

Historical note about moving my servers into a datacenter

Ian Mercer
Ian Mercer

Second Drobo Update

At this point things were looking up for my Drobo

Ian Mercer
Ian Mercer

It's all about disk speed

Why disk speed is the most critical aspect for most modern PCs and servers

Ian Mercer
Ian Mercer

Comcast woes and a new monitoring utility

Monitoring a cable modem using its HTML management interface

Ian Mercer
Ian Mercer

Core duo desktop machine runs cool

Ian Mercer
Ian Mercer
Cover Image for New Home Automation Server

New Home Automation Server

Ian Mercer
Ian Mercer
Cover Image for Preparing for death

Preparing for death

A friend died last year, it wasn't unexpected. He left a lot for his friends to cleanup. Maybe these notes can help someone else prepare better.

Ian Mercer
Ian Mercer