The ultimate in Big Data: Processing everything in the entire universe since time began

Share this post:

Professor Michael Garrett is allowed to be a little nonchalant about the current interest in Big Data. As a leading radio astronomer, he scans outer space for radio waves that might indicate intelligent life on other planets. He gathers, processes and analyzes as much data as possible from the entire known universe, going back to the beginning of cosmic time. Now that’s Big Data.

“The Big Data revolution is great for us,” admits Professor Garrett, Director of the Netherlands Institute for Radio Astronomy (ASTRON) and a professor at Leiden University. “In radio astronomy, this has been the core challenge for decades: We’ve never been able to analyze our datasets fully, simply because they’ve always been so huge.”

A typical science project from ASTRON’s largest telescope can generate around 1,000 TB (1 petabyte). “And that’s at the limit of what can be done in terms of data processing. In the future, we’ll be able to generate much larger volumes than that – but we wouldn’t be able to process it with current-generation technology.”

IBM Faculty Award
In August, in his role as professor in radio astronomy, Professor Garrett was among 13 winners of $10,000 academic grants at the IBM Big Data and Analytics Faculty Awards 2013. He has earmarked his prize money to go towards ASTRON’s efforts in the Search for Extra-terrestrial Intelligence (SETI), where it will be used to develop algorithms and data-intensive analysis techniques that will also be applicable to other fields outside of astronomy. The goal is to study time-variable radio phenomena using a range of innovative SETI algorithms developed at Berkeley University.

Unsurprisingly, the constant need for the highest-level data processing has led to a symbiotic relationship between radio astronomers and technology companies. “We couldn’t do our work without access to the very best supercomputers. But equally, companies like IBM have always been eager to work with us because it allows them to test-drive their newest technology at the very frontiers of data processing requirements. A research-led company like IBM has always looked at it as an opportunity to ensure their technology and expertise were the best they could possibly be.”

ASTRON began working with IBM during the construction of its LOFAR telescope in 2006. The LOFAR – the largest connected radio telescope – is a distributed system of 48 stations across the Netherlands and five surrounding countries, pooling data from an effective 300,000 square meter radius of the sky. “Only IBM could provide the solution we needed for the amount of data we were going to generate. That’s how we became one of the first to test out the IBM Blue Gene supercomputer, helping IBM tweak its design to optimal performance.”

Through the ASTRON & IBM Center for Exascale Technology, the two organizations are currently carrying out fundamental research into technologies needed to develop the Square Kilometer Array (SKA), a €1 billion project to build the world’s largest and most sensitive radio telescope. Once it is completed in 2024, the SKA is expected to generate, in data terms, 100 times the current global internet traffic per day. “We’ve been working to design the hardware, electronics, high-speed communications and processing requirements with IBM long before the technology itself will ever hit the marketplace, and long before we’ve started constructing the telescope.”

Which leads to the obvious question: Is the truth out there? “The search for extra-terrestrial life hasn’t even started, so we just don’t know. Although people have been doing SETI experiments for 50 years, they have only been able to cover a tiny fraction of the sky.” New radio telescopes like LOFAR and SKA will allow scientists to look – at record frequency and time resolutions – at substantial segments of the sky simultaneously.

“LOFAR’s range is a thousand times better than anything we’ve had in the past, and the SKA is probably going to be a million times better. ”The SKA will be so sensitive that it would be able to detect an airport radar on a planet 50 light years away. “We have huge advantages with the technology now. This is really the true beginning of the search. Who knows what we might find?”