CeBIT, a live impression.

Share this post:

Column door Frank van der Wal

CeBIT. For those who don’t know CeBIT, it’s a fair. A big one, perhaps the largest one in Europe, I don’t know. Big in the sense that the hotel rooms are so rare and therefore so expensive that the organisation rents  apartments from individuals to house all of the visitors and exhibitors. If you want to see all of the booths it will take multiple days. It’s the fair that even Angela Merkel is attending, well, opening even.

I had the pleasure to represent IBM with another 1.000 or so colleagues. IBM has more than 100  booths showcasing our products and services. From Softlayer to Watson, from Power Systems to Identity Management. The works. I would have liked to write something about every booth, but that would produce a blog too long.

Two things though I would like to highlight. The first one is that just next to the z Systems stand there was a pedestal of IBM Research. The scientists present were displaying and explaining the work they do on the microserver which I wrote about a couple of months back. It is part of the Square Kilometer Array, a virtual radio telescope to examine the earliest phases of our universe. As the name gives away, this telescope covers an area of a square kilometer and consists of many little antennas receiving cosmic radio waves. The scientists were displaying the microserver and casually telling me that they need those high performing yet low power consuming server to work on the 14 ExaByte of data (!!!!) each of the antenna’s produces  daily. So, 14 ExaByte is 14, followed by 18 0’s bytes that the antenna receives. That’s BIG DATA. The problem, is that the larger part of the data is crap. The microservers are doing algorithms that squeeze the 14 EB down to a mere 1 PetaByte (we are talking only 15 trailing 0’s here) and stores it for further use. Wow!

The next topic, and here is the bridge, is talk about the ‘other’ Big Data in relation to the good old mainframe. At CeBIT I was delighted to see many young colleagues working on z Systems. I’ve spoken to many of them and had interesting conversations with the Big Data crew.

Big Data in this context, is not so much about the 14 ExaByte of data produced daily, it is more about the combination of very structured data companies have, nicely tucked away in databases (like DB2 or Oracle) and the unstructured data that is produced by Social Media, PDF’s and office documents like spreadsheets. These two types of data couldn’t be further away from each other in terms of how to access, search and store them. To make sense of  structured data you need to ‘query’ the database. To look into a PDF you need other techniques.

The friends from Apache defined a way of storing and processing the unstructured data in a very efficient way, called Hadoop. The storage is done by slicing the large files into small pieces and spread it around many little servers with storage.
Next, you would like to process that data. In the structured world you transport that data out of the database to the processor, but if you have large amounts of unstructured data scattered over those servers, you bring compute power to the data. What Hadoop does is to breakdown the processing part into small parts that are send to the little servers. In this way the work is run in parallel. Hadoop takes care about collecting and assembling all the partial results from the nodes and present the complete answer (42 of course!).

Hadoop is an Open Source implementation and although very well maintained, for companies it implies a risk. IBM took this Hadoop and ruggedized it for enterprises, while maintaining the open standards at a 100%. We call it BigInsights and it runs also on z Systems.

Of course, processing -big- data is not enough. You have to do something with it, like analyzing, adding value etc. Especially with the combination of the two data types this becomes interesting and if the mainframe holds those two, well, the structure is there to do analytics and all kind of other data-enriching activities.

So, Big Data is not only for low-end cheap servers with low priced storage, it is also there for enterprise-class environments.

From CeBIT:

Tschüss!