MySQL Cluster is an in-memory database running across a cluster of machines that tries to be robust to failures of nodes.
I had been looking at it off and on for a while. My curiosity was peaked when I watched Stewart Smith's talk on Google Video, "A Googly MySQL Cluster Talk", and investigated a few questions I had after that talk.
Unsurprisingly, MySQL Cluster is not a giant, simple, virtual database that runs transparently over a cluster, nodes dropping in an out of service at will, with read-write replication and data migration all done automatically and robustly. MySQL Cluster is cool, but it is not quite that cool.
The core design behind MySQL Cluster seems to have started with one thought: Instead of storing tables in the local filesystem, what if we stored them in the memory of nearby machines on the network?
This is a really neat idea. I remember some fun research work a while back that tried to exploit the fact that it is an order of magnitude faster for a process to access data from RAM on a remote machine than it is to access data from local disk. Yes, disks are that slow.
Most of this research work focused on experiments with paging out to the free memory of other machines on the network instead of local disk, but the idea still seems similar to what the MySQL folks are doing with MySQL Cluster.
So, I am guessing MySQL folks started by deciding to experiment with a new type of in-memory storage engine, one that would say, this data isn't in this machine's memory, but in that machine's memory over there.
Then, it looks like they added a bunch of stuff on top to try to make this robust. Logging of transactions out to the local filesystem, replicas of the table fragments, and goodies like that. MySQL Cluster was born.
There's a lot of hard work left to be done though. According to their limitations page, new data nodes cannot be added without a restart of the entire cluster, so you cannot add new boxes in to increase your capacity with the system live.
Other serious issues appear to include performance issues with some queries, windows of opportunity for data loss, potential for stray locks, temporary transaction failures when nodes are dropped or rebooted, complicated configuration and maintenance of the cluster, and other problems.
Even so, it is impressive what the MySQL folks have built. It may not be a big, virtual database, but it is a big step in the right direction.
Update: There are more details on how MySQL handles failures and recovery in a 2005 VLDB paper, "Recovery Principles of MySQL Cluster 5.1" (PDF).