Big Data and Official Statistics

Big data is a component of the Fourth Industrial Revolution. The deep penetration of digital technology has turned data into an essential component of the production process. Data are automatically generated by machines during the course of operation and during interactions with humans. This paper describes the concept and composition of big data. Most of the big data are unstructured and include text, audio-video files, images, emails, log files, etc. Statisticians are more interested in structured data presented in a pre-defined database model. Big data offer new sources and opportunities that cannot be discounted. However, the use of big data requires proper assessment in terms of quality dimensions such as accuracy, comparability and methodological soundness. Against the backdrop of arguments regarding big data, some users view big data as a replacement of official statistics. Such a conclusion is premature for at least two reasons: first, only a small part of big data can be used for decision-making. Second, theory and practice prove that a small sample based on scientific methods can yield much more reliable and accurate estimates than the results obtained from the processing of large amounts of unstructured data. The paper assesses the possibility of using big data for Sustainable Development Goals (SDG) monitoring, which is a nationally owned process, and NSOs are accountable for the SDG data they report. If the data are derived from a big data source, irrespective of the level of technical sophistication used in data transformation, the reliability of such data might be questioned by the national institutions. The paper concludes that the reliability of data obtained from big data sources hinges on the quality of tools and methods applied to data transformation. Statisticians can play an important role in alerting society, decision-making bodies of the government and businesses about the reliability of information derived from the different sources.

1. Venetskii I.G., Venetskaya V.I. Basic Mathematical and Statistical Concepts and Formulas in Economic Analysis. Moscow: Statistics Publ.; 1974. 279 p. (In Russ.)

2. Cochran W. Sampling Techniques. New York: John Wiley & Sons; 1999. 428 p.

3. Kaplan R.M., Chambers D.A., Glasgow R.E. Big Data and Large Sample Size: A Cautionary Note on the Potential for Bias. CTS Journal. 2014;7(4):342-346. Available from: doi: https://doi.org/10.1111/cts.12178.

4. National Quality Assurance Frameworks. Report of the Secretary-General. 2012. Available from: https://unstats.un.org/unsd/statcom/doc12/2012-13-NQAF-R.pdf. (In Russ.)

5. Upadhyaya S., Kynclova P. Big Data - Its Relevance and Impact on Industrial Statistics. Vienna: UNIDO; 2017.

6. Cox D.R., Kartsonaki C., Keogh R.H. Big Data: Some Statistical Issues. Statistics & Probability Letters. 2018;(136):111-115. Available from: doi: https://doi.org/10.1016/j.spl.2018.02.015.

7. Calleja R., Rogerson A. Financing Challenges for Developing Statistical Systems: A review of Financing Options. PARIS21 Discussion Paper, No. 14. Paris; 2019. Available from: https://paris21.org/sites/default/files/2019-01/Financing%20challenges%20for%20developing%20statistical%20systems%20%28DP14%29.pdf.

8. UN Statistical Commission. Cape Town Global Action Plan for Sustainable Development Data. Adopted by the UN Statistical Commission at its 48th Session, March 2017. Available from: https://unstats.un.org/sdgs/hlg/Cape_Town_Global_Action_Plan_for_Sustainable_Development_Data.pdf.

9. Sachs J. et al. Sustainable Development Report 2019. New York: Bertelsmann Stiftung and Sustainable Development Solutions Network (SDSN), 2019. Available from: https://s3.amazonaws.com/sustainabledevelopment.report/2019/2019_sustainable_development_report.pdf.