Three universities are making strides in an extensive project to build a computing infrastructure capable of processing the vast amounts of data that will come from the MeerKAT telescope.

The Inter-University Institute for Data Intensive Astronomy (IDIA) has developed a scientific research cloud to enable astronomers to carry out scientific research on the data generated by large science instruments like the MeerKAT telescope and other Square Kilometre Array (SKA) pathfinder telescopes.

“The SKA is addressing the fundamental questions about the universe in which we live. But these fantastic science goals come with a catch . . . and that is the tremendously huge stream of data. To do the investigations, we need to have the tools to work with that data,” IDIA director Professor Russ Taylor explained during an event at the Iziko Planetarium and Digital Dome, in Cape Town, this week.

He said the universities of Cape Town, the Western Cape and Pretoria had been building a research cloud to allow for this.

With the IDIA research cloud now in place, astronomers in IDIA partner universities and their scientific collaborators are able to process the data coming from the MeerKAT telescope with their scientific analysis tools running in the cloud.

So far, five MeerKAT large survey projects have requested to use IDIA to process their data.

They are MeerKAT International GigaHertz Tiered Extragalactic Exploration Survey (Mightee), which looks at where galaxies come from and how they evolve; MeerKAT HI Observations of Nearby Galactic Objects: Observing Southern Emitters, or Mhongoose, which examines what happens to hydrogen gas in galaxies; Looking at the Distant Universe with the MeerKAT Array, or Laduma, which investigates what happened to hydrogen gas in the early universe; the Fornax Survey, which looks into how galaxies interact; and ThunderKAT, which looks into some of the most powerful explosions in the universe.

Images screened at the Iziko Planetarium were built on the IDIA computing infrastructure using faint radio waves from a distant part of the universe captured by MeerKAT for the Mightee project. The images result from the illustrated journey of data from the MeerKAT telescope to IDIA, then through the scientific analysis pipelines built by scientists, finally turning the data into images that are used for research.

Taylor said IDIA was very proud that South African universities were spearheading the initiative. 

“We need to build the solutions here to work with data. IDIA is about innovation and providing access to data. It’s also about training a new generation of people to work with the data.

“The main users of SKA and MeerKAT will be researchers at universities . . . and South African universities have the opportunity to begin the journey in South Africa. We need researchers to play with the data and test new approaches.

IDIA democratises that process so that researchers can bring their ideas to the data and science.”

IDIA visualisation developer Angus Comrie said speed was essential.

“It's important to give researchers what they want quickly. Cloud computing enables us to give them visual feedback very quickly.”

Masters student Sibusiso Mdhluli said IDIA infrastructure had helped him immensely.

“Astronomers in the old days used to take pictures and assess individual images. Machine learning takes all those images and analyses them in a short space of time.” He said he was particularly interested in seeing how other disciplines such as medicine could benefit from the infrastructure.

IDIA is working with the South African National Research Network, as well as the South African Radio Astronomy Observatory, which has a science processing facility at the Centre for High-Performance Computing.