Preface

How to use this book

The book focuses not only on the implementation of the Quality Indicators (QI) plugin in the Statistical Working System (SWS) for each disseminated domain, but it also discusses the data structure, workflow, data aggregates and the indicators themselves. Moreover, the book covers the Shiny app that has been developed to visualize the quality indicators and underlying data, draw on-the-fly charts and generate automatic pdf reports based on the selection parameters (dataset, area, element, item and year) chosen by the user.

The quality indicators hereby implemented have as reference the following documents: the Statistical Standard Series - Quality Indicators for External Users document produced by the Office of the Chief Statistician (OCS) and endorsed by the Interdepartmental Working Group (IDWG) on Statistics and the Quality Indicators - FAO ESS review for questionnaire-based FAOSTAT domains, which contains the ready-to-use quality indicators, based on the previous document.

The chapters are organized as follows:

Chapter Description
Ch 1 Provides the framework and entire overview.
Ch 2 Describes the quality indicators to be calculated in the SWS.
Ch ?? Provides the whole documentation of the module faoswsQualityIndicators.
Ch 4 Provides the whole documentation of the Shiny App of the Quality Indicators.

Why to use the SWS

The Statistical Working System (SWS) is a corporate system that supports data collection, processing and validation, metadata management, and quality assessment of FAO’s main statistical domains in line with the Generic Statistical Business Process Model. It has a dual nature – both technological and statistical – and for this reason, is developed by two different teams (the IT Division [CSI] and the Statistics Division [ESS]). The SWS provides an end-to-end solution for inputting and storing raw data, performing automated statistical processes (e.g. imputation, validation, etc.), and producing data that are ready for dissemination. Moreover, the reasons why you should use the SWS are:

  • The SWS improves data quality by centralizing and standardizing the maintenance of data, classification systems, methodologies, and standards.
  • The SWS is cost-efficient in terms of statistical development and functionalities:
    • Technical units can adopt methods and even pieces of codes developed for other units: documentation and scripts are open;
    • Users have direct access to some common datasets, thus removing duplication of efforts;
    • New functionalities reflect the requirements of all users and are available to all users, with economies of scale on IT development.
  • Users have full control of the quality of the data they are producing:
    • Increased use of advanced and semi-automated statistical methods reduce the need for manual interventions and ensure that results are reproducible;
    • Establishment and use of a ‘basic’ set of quality and performance indicators for processes and outputs based on the FAO Statistical Quality Assurance Framework (SQAF);
    • Assessment and publication of quality indicators on a regular basis.
  • The SWS ensures corporate data consistency through the adoption of statistical standards. In particular, the SWS
    • Facilitates the adoption of corporate standard classifications, metadata and code lists through seamless integration with the corporate Reference Data Management system;
    • Reduces the fragmentation and duplication of sources (e.g. population data), thus reducing inconsistencies across technical units;
    • Provides flexible tools for a wide range of data collection processes including web data scraping and questionnaire management, thus reducing the duplication of data collection efforts.
  • The SWS preserves institutional knowledge by recording the history of data revisions, automating methodology and calculations, and providing end users with unambiguous instructions to follow.

Go to the Statistical Working System website (restricted access).

Migration actors

Any migration into the SWS requires the interaction between at least three actors:

  1. The technical units responsible for the data dissemination of the several statistical processes related to Agriculture Production and Trade, Fertilizers, Pesticides, Land, Forestry Production and Trade, Producer Prices, Fisheries and AQUASTAT.

  2. The ESS - Methodological Innovation Team (ESS - SWS) responsible for the implementation and documentation of the required statistical processes. From the ESS - SWS team the focal points are:

  • Bruno Caetano Vidigal Statistician (ESS)
  • Carola Fabi Senior Statistician (ESS)
  1. The CSI - SWS the primary backend and frontend maintainer of the SWS and responsible for the implementation and documentation of non-statistical processes (IT infrastructure). The CSI - SWS team have as interlocutors:
  • Matteo Terrinoni Lead Front End Developer (CSI)
  • Enrico Anello Full Stack Developer (CSI)
  • Rohi Fadlun Full Stack Java Developer (CSI)
  • John Rowell Information Technology Officer (CSI)