A current trend in progress at the hardware level, is the displacement of disk by memory - with memory becoming the default storage mechanism for data. On the surface it doesn’t sound like a big change. Hey. Stuff goes faster. It’s more reliable. Great. Nevertheless, it’s the most disruptive change that the IT world has ever had to accommodate. Here’s why:

Just about all the software that we regularly use assumes a disk-based environment. It assumes that data is stored on a spinning medium that has read heads hovering over a magnetic surface. All our software has been designed on that assumption, for over 4 decades.

The Dawn of 64 bit Computing

I realized that something fundamental was changing when I first ran into CEP/ESP products. These are products, like StreamBase and Apama, that processes data streams. They included a full event stream middleware solution from a vendor called Kabira and data fabric products like Gemstone and Avokia. New products of this ilk seem to appear regularly. They manage data in memory and they access data heaps that are bigger than 2 gigabytes, which is the limit of 32 bit addressing. They are fundamentally 64 bit memory-oriented products, which could (in theory) address a memory space as big as 4 exabytes.

But Memory is Expensive

Yes it is. But something funny happened to it on the way to the iPod Nano. The volume of memory required by consumer electronics started to grow dramatically. The twin forces of the iPod Nano and digital cameras drove market growth and this started pushing the price of flash memory down. Flash memory is expected to cost $9 per gigabyte by 2009. (DRAM will probably cost 2 to 3 times as much). We’ve now reached the point where flash disk has driven out spinning disk for volumes of 16 gigabytes and below. We have also witnessed the dawn of a new device: the flash-based subnotebook (of which Negraponte’s OLPC is one). This memory-for-disk replacement trend will continue until disk is completely defunct.

What does it mean?

Right now it doesn’t mean much as regards most of our applications and how they work. It’s a little too early for anything to be visible other than the appearance of new software that is memory oriented. (I came across yet another such product this morning, called QlikView). Ultimately though, it will make a big difference, because holding data in databases (relational or otherwise) that were physically designed for disk will make no sense. We will use quite different memory-based structures, if we access data direct from memory - and that in turn means that we will have to re-engineer all our applications.

Indeed we will need to re-engineer our operating environments. Operating systems will have to work in a different way - to provide direct address access to memory oriented storage structures. Grids will become more important because of the memory spaces they will provide access to. Specific products, particularly databases and transaction engines, will take on changed roles or disappear altogether (either is possible, because access speeds will get much faster). All-in-all, the nature of software will change.

It’s coming to a computer near you, sometime in the next 5-10 years.

  Subscribe to HaveMacWillBlog in a reader