A Real Time Processing system for big data in astronomy: Applications to HERA

P. La Plante; P. K.G. Williams; M. Kolopanis; J. S. Dillon; A. P. Beardsley; N. S. Kern; M. Wilensky; Z. S. Ali; Z. Abdurashidova; J. E. Aguirre; P. Alexander; Y. Balfour; G. Bernardi; T. S. Billings; J. D. Bowman; R. F. Bradley; P. Bull; J. Burba; S. Carey; C. L. Carilli; C. Cheng; D. R. DeBoer; M. Dexter; E. de Lera Acedo; J. Ely; A. Ewall-Wice; N. Fagnoni; R. Fritz; S. R. Furlanetto; K. Gale-Sides; B. Glendenning; D. Gorthi; B. Greig; J. Grobbelaar; Z. Halday; B. J. Hazelton; J. N. Hewitt; J. Hickish; D. C. Jacobs; A. Julius; J. Kerrigan; P. Kittiwisit; S. A. Kohn; A. Lanman; T. Lekalake; D. Lewis; A. Liu; D. MacMahon; L. Malan; C. Malgas; M. Maree; Z. E. Martinot; E. Matsetela; A. Mesinger; M. Molewa; M. F. Morales; T. Mosiane; S. Murray; A. R. Neben; B. Nikolic; A. R. Parsons; R. Pascua; N. Patra; S. Pieterse; J. C. Pober; N. Razavi-Ghods; J. Ringuette; J. Robnett; K. Rosie; M. G. Santos; P. Sims; C. Smith; A. Syce; N. Thyagarajan; H. Zheng

doi:10.1016/j.ascom.2021.100489

A Real Time Processing system for big data in astronomy: Applications to HERA

P. La Plante, P. K.G. Williams, M. Kolopanis, J. S. Dillon, A. P. Beardsley, N. S. Kern, M. Wilensky, Z. S. Ali, Z. Abdurashidova, J. E. Aguirre, P. Alexander, Y. Balfour, G. Bernardi, T. S. Billings, J. D. Bowman, R. F. Bradley, P. Bull, J. Burba, S. Carey, C. L. CarilliC. Cheng, D. R. DeBoer, M. Dexter, E. de Lera Acedo, J. Ely, A. Ewall-Wice, N. Fagnoni, R. Fritz, S. R. Furlanetto, K. Gale-Sides, B. Glendenning, D. Gorthi, B. Greig, J. Grobbelaar, Z. Halday, B. J. Hazelton, J. N. Hewitt, J. Hickish, D. C. Jacobs, A. Julius, J. Kerrigan, P. Kittiwisit, S. A. Kohn, A. Lanman, T. Lekalake, D. Lewis, A. Liu, D. MacMahon, L. Malan, C. Malgas, M. Maree, Z. E. Martinot, E. Matsetela, A. Mesinger, M. Molewa, M. F. Morales, T. Mosiane, S. Murray, A. R. Neben, B. Nikolic, A. R. Parsons, R. Pascua, N. Patra, S. Pieterse, J. C. Pober, N. Razavi-Ghods, J. Ringuette, J. Robnett, K. Rosie, M. G. Santos, P. Sims, C. Smith, A. Syce, N. Thyagarajan, H. Zheng

Earth and Space Exploration, School of (SESE)

Research output: Contribution to journal › Article › peer-review

8 Scopus citations

Abstract

As current- and next-generation astronomical instruments come online, they will generate an unprecedented deluge of data. Analyzing these data in real time presents unique conceptual and computational challenges, and their long-term storage and archiving is scientifically essential for generating reliable, reproducible results. We present here the real-time processing (RTP) system for the Hydrogen Epoch of Reionization Array (HERA), a radio interferometer endeavoring to provide the first detection of the highly redshifted 21 cm signal from Cosmic Dawn and the Epoch of Reionization by an interferometer. The RTP system consists of analysis routines run on raw data shortly after they are acquired, such as calibration and detection of radio-frequency interference (RFI) events. RTP works closely with the Librarian, the HERA data storage and transfer manager which automatically ingests data and transfers copies to other clusters for post-processing analysis. Both the RTP system and the Librarian are public and open source software, which allows for them to be modified for use in other scientific collaborations. When fully constructed, HERA is projected to generate over 50 terabytes (TB) of data each night, and the RTP system enables the successful scientific analysis of these data.

Original language	English (US)
Article number	100489
Journal	Astronomy and Computing
Volume	36
DOIs	https://doi.org/10.1016/j.ascom.2021.100489
State	Published - Jul 2021

Keywords

Astronomy — Software
Data analysis — Physical sciences and engineering
Data analysis — Software
Development
Methods

ASJC Scopus subject areas

Astronomy and Astrophysics
Computer Science Applications
Space and Planetary Science

Access to Document

10.1016/j.ascom.2021.100489

Cite this

La Plante, P., Williams, P. K. G., Kolopanis, M., Dillon, J. S., Beardsley, A. P., Kern, N. S., Wilensky, M., Ali, Z. S., Abdurashidova, Z., Aguirre, J. E., Alexander, P., Balfour, Y., Bernardi, G., Billings, T. S., Bowman, J. D., Bradley, R. F., Bull, P., Burba, J., Carey, S., ... Zheng, H. (2021). A Real Time Processing system for big data in astronomy: Applications to HERA. Astronomy and Computing, 36, Article 100489. https://doi.org/10.1016/j.ascom.2021.100489

La Plante, P, Williams, PKG, Kolopanis, M, Dillon, JS, Beardsley, AP, Kern, NS, Wilensky, M, Ali, ZS, Abdurashidova, Z, Aguirre, JE, Alexander, P, Balfour, Y, Bernardi, G, Billings, TS, Bowman, JD, Bradley, RF, Bull, P, Burba, J, Carey, S, Carilli, CL, Cheng, C, DeBoer, DR, Dexter, M, de Lera Acedo, E, Ely, J, Ewall-Wice, A, Fagnoni, N, Fritz, R, Furlanetto, SR, Gale-Sides, K, Glendenning, B, Gorthi, D, Greig, B, Grobbelaar, J, Halday, Z, Hazelton, BJ, Hewitt, JN, Hickish, J, Jacobs, DC, Julius, A, Kerrigan, J, Kittiwisit, P, Kohn, SA, Lanman, A, Lekalake, T, Lewis, D, Liu, A, MacMahon, D, Malan, L, Malgas, C, Maree, M, Martinot, ZE, Matsetela, E, Mesinger, A, Molewa, M, Morales, MF, Mosiane, T, Murray, S, Neben, AR, Nikolic, B, Parsons, AR, Pascua, R, Patra, N, Pieterse, S, Pober, JC, Razavi-Ghods, N, Ringuette, J, Robnett, J, Rosie, K, Santos, MG, Sims, P, Smith, C, Syce, A, Thyagarajan, N & Zheng, H 2021, 'A Real Time Processing system for big data in astronomy: Applications to HERA', Astronomy and Computing, vol. 36, 100489. https://doi.org/10.1016/j.ascom.2021.100489

@article{c38cdafa9a3346b1aacfd04c599a74d6,

title = "A Real Time Processing system for big data in astronomy: Applications to HERA",

abstract = "As current- and next-generation astronomical instruments come online, they will generate an unprecedented deluge of data. Analyzing these data in real time presents unique conceptual and computational challenges, and their long-term storage and archiving is scientifically essential for generating reliable, reproducible results. We present here the real-time processing (RTP) system for the Hydrogen Epoch of Reionization Array (HERA), a radio interferometer endeavoring to provide the first detection of the highly redshifted 21 cm signal from Cosmic Dawn and the Epoch of Reionization by an interferometer. The RTP system consists of analysis routines run on raw data shortly after they are acquired, such as calibration and detection of radio-frequency interference (RFI) events. RTP works closely with the Librarian, the HERA data storage and transfer manager which automatically ingests data and transfers copies to other clusters for post-processing analysis. Both the RTP system and the Librarian are public and open source software, which allows for them to be modified for use in other scientific collaborations. When fully constructed, HERA is projected to generate over 50 terabytes (TB) of data each night, and the RTP system enables the successful scientific analysis of these data.",

keywords = "Astronomy — Software, Data analysis — Physical sciences and engineering, Data analysis — Software, Development, Methods",

author = "{La Plante}, P. and Williams, {P. K.G.} and M. Kolopanis and Dillon, {J. S.} and Beardsley, {A. P.} and Kern, {N. S.} and M. Wilensky and Ali, {Z. S.} and Z. Abdurashidova and Aguirre, {J. E.} and P. Alexander and Y. Balfour and G. Bernardi and Billings, {T. S.} and Bowman, {J. D.} and Bradley, {R. F.} and P. Bull and J. Burba and S. Carey and Carilli, {C. L.} and C. Cheng and DeBoer, {D. R.} and M. Dexter and {de Lera Acedo}, E. and J. Ely and A. Ewall-Wice and N. Fagnoni and R. Fritz and Furlanetto, {S. R.} and K. Gale-Sides and B. Glendenning and D. Gorthi and B. Greig and J. Grobbelaar and Z. Halday and Hazelton, {B. J.} and Hewitt, {J. N.} and J. Hickish and Jacobs, {D. C.} and A. Julius and J. Kerrigan and P. Kittiwisit and Kohn, {S. A.} and A. Lanman and T. Lekalake and D. Lewis and A. Liu and D. MacMahon and L. Malan and C. Malgas and M. Maree and Martinot, {Z. E.} and E. Matsetela and A. Mesinger and M. Molewa and Morales, {M. F.} and T. Mosiane and S. Murray and Neben, {A. R.} and B. Nikolic and Parsons, {A. R.} and R. Pascua and N. Patra and S. Pieterse and Pober, {J. C.} and N. Razavi-Ghods and J. Ringuette and J. Robnett and K. Rosie and Santos, {M. G.} and P. Sims and C. Smith and A. Syce and N. Thyagarajan and H. Zheng",

note = "Funding Information: This material is based upon work supported by the National Science Foundation, United States of America under Grant No. 1636646 , the Gordon and Betty Moore Foundation, and institutional support from the HERA collaboration partners, United States of America . HERA is hosted by the South African Radio Astronomy Observatory, which is a facility of the National Research Foundation, an agency of the Department of Science and Innovation. Parts of this research were supported by the Australian Research Council Centre of Excellence for All Sky Astrophysics in 3 Dimensions (ASTRO 3D) , through project number CE170100013. G. Bernardi acknowledges funding from the INAF PRIN-SKA 2017, Italy project 1.05.01.88.04 (FORECaST), support from the Ministero degli Affari Esteri della Coorperazione Internazionale – Direzione Generale per la Promozione del Sistema Paese Progetto di Grande Rilevanza, Italy ZA18GR02 and the National Research Foundation of South Africa (Grant No. 113121 ) as part of the ISARP RADIOSKY2020 Joint Research Scheme, from the Royal Society, UK and the Newton Fund, UK under grand NA150184 and from the National Research Foundation of South Africa (Grant No. 103424 ). P. Bull acknowledges funding for part of this research from the European Research Council (ERC) under the European Union{\textquoteright}s Horizon 2020 research and innovation program (Grant Agreement No. 948764 ), and from STFC, UK Grant ST/T000341/1 . J.S. Dillon gratefully acknowledges the support of the NSF AAPF, United States of America award #1701536 . N Kern acknowledges support from the MIT Pappalardo, UK fellowship. A. Liu acknowledges support from the New Frontiers in Research Fund Exporation grant program, Canada , the Canadian Institute for Advanced Research (CIFAR) Azrieli Global Scholars program , a Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant and a Discovery Launch Supplement , the Sloan Research Fellowship, and the William Dawson Scholarship at McGill, United States of America . Funding Information: This material is based upon work supported by the National Science Foundation, United States of America under Grant No. 1636646, the Gordon and Betty Moore Foundation, and institutional support from the HERA collaboration partners, United States of America. HERA is hosted by the South African Radio Astronomy Observatory, which is a facility of the National Research Foundation, an agency of the Department of Science and Innovation. Parts of this research were supported by the Australian Research Council Centre of Excellence for All Sky Astrophysics in 3 Dimensions (ASTRO 3D), through project number CE170100013. G. Bernardi acknowledges funding from the INAF PRIN-SKA 2017, Italy project 1.05.01.88.04 (FORECaST), support from the Ministero degli Affari Esteri della Coorperazione Internazionale ? Direzione Generale per la Promozione del Sistema Paese Progetto di Grande Rilevanza, ItalyZA18GR02 and the National Research Foundation of South Africa (Grant No. 113121) as part of the ISARP RADIOSKY2020 Joint Research Scheme, from the Royal Society, UK and the Newton Fund, UK under grand NA150184 and from the National Research Foundation of South Africa (Grant No. 103424). P. Bull acknowledges funding for part of this research from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (Grant Agreement No. 948764), and from STFC, UK Grant ST/T000341/1. J.S. Dillon gratefully acknowledges the support of the NSF AAPF, United States of America award #1701536. N Kern acknowledges support from the MIT Pappalardo, UK fellowship. A. Liu acknowledges support from the New Frontiers in Research Fund Exporation grant program, Canada, the Canadian Institute for Advanced Research (CIFAR) Azrieli Global Scholars program, a Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant and a Discovery Launch Supplement, the Sloan Research Fellowship, and the William Dawson Scholarship at McGill, United States of America. Publisher Copyright: {\textcopyright} 2021 The Author(s)",

year = "2021",

month = jul,

doi = "10.1016/j.ascom.2021.100489",

language = "English (US)",

volume = "36",

journal = "Astronomy and Computing",

issn = "2213-1337",

publisher = "Elsevier BV",

}

TY - JOUR

T1 - A Real Time Processing system for big data in astronomy

T2 - Applications to HERA

AU - La Plante, P.

AU - Williams, P. K.G.

AU - Kolopanis, M.

AU - Dillon, J. S.

AU - Beardsley, A. P.

AU - Kern, N. S.

AU - Wilensky, M.

AU - Ali, Z. S.

AU - Abdurashidova, Z.

AU - Aguirre, J. E.

AU - Alexander, P.

AU - Balfour, Y.

AU - Bernardi, G.

AU - Billings, T. S.

AU - Bowman, J. D.

AU - Bradley, R. F.

AU - Bull, P.

AU - Burba, J.

AU - Carey, S.

AU - Carilli, C. L.

AU - Cheng, C.

AU - DeBoer, D. R.

AU - Dexter, M.

AU - de Lera Acedo, E.

AU - Ely, J.

AU - Ewall-Wice, A.

AU - Fagnoni, N.

AU - Fritz, R.

AU - Furlanetto, S. R.

AU - Gale-Sides, K.

AU - Glendenning, B.

AU - Gorthi, D.

AU - Greig, B.

AU - Grobbelaar, J.

AU - Halday, Z.

AU - Hazelton, B. J.

AU - Hewitt, J. N.

AU - Hickish, J.

AU - Jacobs, D. C.

AU - Julius, A.

AU - Kerrigan, J.

AU - Kittiwisit, P.

AU - Kohn, S. A.

AU - Lanman, A.

AU - Lekalake, T.

AU - Lewis, D.

AU - Liu, A.

AU - MacMahon, D.

AU - Malan, L.

AU - Malgas, C.

AU - Maree, M.

AU - Martinot, Z. E.

AU - Matsetela, E.

AU - Mesinger, A.

AU - Molewa, M.

AU - Morales, M. F.

AU - Mosiane, T.

AU - Murray, S.

AU - Neben, A. R.

AU - Nikolic, B.

AU - Parsons, A. R.

AU - Pascua, R.

AU - Patra, N.

AU - Pieterse, S.

AU - Pober, J. C.

AU - Razavi-Ghods, N.

AU - Ringuette, J.

AU - Robnett, J.

AU - Rosie, K.

AU - Santos, M. G.

AU - Sims, P.

AU - Smith, C.

AU - Syce, A.

AU - Thyagarajan, N.

AU - Zheng, H.

N1 - Funding Information: This material is based upon work supported by the National Science Foundation, United States of America under Grant No. 1636646 , the Gordon and Betty Moore Foundation, and institutional support from the HERA collaboration partners, United States of America . HERA is hosted by the South African Radio Astronomy Observatory, which is a facility of the National Research Foundation, an agency of the Department of Science and Innovation. Parts of this research were supported by the Australian Research Council Centre of Excellence for All Sky Astrophysics in 3 Dimensions (ASTRO 3D) , through project number CE170100013. G. Bernardi acknowledges funding from the INAF PRIN-SKA 2017, Italy project 1.05.01.88.04 (FORECaST), support from the Ministero degli Affari Esteri della Coorperazione Internazionale – Direzione Generale per la Promozione del Sistema Paese Progetto di Grande Rilevanza, Italy ZA18GR02 and the National Research Foundation of South Africa (Grant No. 113121 ) as part of the ISARP RADIOSKY2020 Joint Research Scheme, from the Royal Society, UK and the Newton Fund, UK under grand NA150184 and from the National Research Foundation of South Africa (Grant No. 103424 ). P. Bull acknowledges funding for part of this research from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (Grant Agreement No. 948764 ), and from STFC, UK Grant ST/T000341/1 . J.S. Dillon gratefully acknowledges the support of the NSF AAPF, United States of America award #1701536 . N Kern acknowledges support from the MIT Pappalardo, UK fellowship. A. Liu acknowledges support from the New Frontiers in Research Fund Exporation grant program, Canada , the Canadian Institute for Advanced Research (CIFAR) Azrieli Global Scholars program , a Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant and a Discovery Launch Supplement , the Sloan Research Fellowship, and the William Dawson Scholarship at McGill, United States of America . Funding Information: This material is based upon work supported by the National Science Foundation, United States of America under Grant No. 1636646, the Gordon and Betty Moore Foundation, and institutional support from the HERA collaboration partners, United States of America. HERA is hosted by the South African Radio Astronomy Observatory, which is a facility of the National Research Foundation, an agency of the Department of Science and Innovation. Parts of this research were supported by the Australian Research Council Centre of Excellence for All Sky Astrophysics in 3 Dimensions (ASTRO 3D), through project number CE170100013. G. Bernardi acknowledges funding from the INAF PRIN-SKA 2017, Italy project 1.05.01.88.04 (FORECaST), support from the Ministero degli Affari Esteri della Coorperazione Internazionale ? Direzione Generale per la Promozione del Sistema Paese Progetto di Grande Rilevanza, ItalyZA18GR02 and the National Research Foundation of South Africa (Grant No. 113121) as part of the ISARP RADIOSKY2020 Joint Research Scheme, from the Royal Society, UK and the Newton Fund, UK under grand NA150184 and from the National Research Foundation of South Africa (Grant No. 103424). P. Bull acknowledges funding for part of this research from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (Grant Agreement No. 948764), and from STFC, UK Grant ST/T000341/1. J.S. Dillon gratefully acknowledges the support of the NSF AAPF, United States of America award #1701536. N Kern acknowledges support from the MIT Pappalardo, UK fellowship. A. Liu acknowledges support from the New Frontiers in Research Fund Exporation grant program, Canada, the Canadian Institute for Advanced Research (CIFAR) Azrieli Global Scholars program, a Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant and a Discovery Launch Supplement, the Sloan Research Fellowship, and the William Dawson Scholarship at McGill, United States of America. Publisher Copyright: © 2021 The Author(s)

PY - 2021/7

Y1 - 2021/7

N2 - As current- and next-generation astronomical instruments come online, they will generate an unprecedented deluge of data. Analyzing these data in real time presents unique conceptual and computational challenges, and their long-term storage and archiving is scientifically essential for generating reliable, reproducible results. We present here the real-time processing (RTP) system for the Hydrogen Epoch of Reionization Array (HERA), a radio interferometer endeavoring to provide the first detection of the highly redshifted 21 cm signal from Cosmic Dawn and the Epoch of Reionization by an interferometer. The RTP system consists of analysis routines run on raw data shortly after they are acquired, such as calibration and detection of radio-frequency interference (RFI) events. RTP works closely with the Librarian, the HERA data storage and transfer manager which automatically ingests data and transfers copies to other clusters for post-processing analysis. Both the RTP system and the Librarian are public and open source software, which allows for them to be modified for use in other scientific collaborations. When fully constructed, HERA is projected to generate over 50 terabytes (TB) of data each night, and the RTP system enables the successful scientific analysis of these data.

AB - As current- and next-generation astronomical instruments come online, they will generate an unprecedented deluge of data. Analyzing these data in real time presents unique conceptual and computational challenges, and their long-term storage and archiving is scientifically essential for generating reliable, reproducible results. We present here the real-time processing (RTP) system for the Hydrogen Epoch of Reionization Array (HERA), a radio interferometer endeavoring to provide the first detection of the highly redshifted 21 cm signal from Cosmic Dawn and the Epoch of Reionization by an interferometer. The RTP system consists of analysis routines run on raw data shortly after they are acquired, such as calibration and detection of radio-frequency interference (RFI) events. RTP works closely with the Librarian, the HERA data storage and transfer manager which automatically ingests data and transfers copies to other clusters for post-processing analysis. Both the RTP system and the Librarian are public and open source software, which allows for them to be modified for use in other scientific collaborations. When fully constructed, HERA is projected to generate over 50 terabytes (TB) of data each night, and the RTP system enables the successful scientific analysis of these data.

KW - Astronomy — Software

KW - Data analysis — Physical sciences and engineering

KW - Data analysis — Software

KW - Development

KW - Methods

UR - http://www.scopus.com/inward/record.url?scp=85112499911&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85112499911&partnerID=8YFLogxK

U2 - 10.1016/j.ascom.2021.100489

DO - 10.1016/j.ascom.2021.100489

M3 - Article

SN - 2213-1337

VL - 36

JO - Astronomy and Computing

JF - Astronomy and Computing

M1 - 100489

ER -

A Real Time Processing system for big data in astronomy: Applications to HERA

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this