Introduction

FAO Fisheries and Aquaculture Department is migrating its statistical processes into the Corporate Statistical Working System (SWS). This online book documents the significant steps that have been taken so far concerning this migration. The document describes the main institutional actors involved in the migration as well as the FIAS - SWS resources (code lists, datasets, data tables) that have been created to support the SWS plugins (R modules) to meet technical unit requirements. Furthermore, the SWS plugins and Shiny applications are presented in the form of chapters providing a detailed description of their workflows as well as results.

IMPORTANT: Use the Chrome browser to have a correct visualization of HTML outputs in this online document.

Migration actors

Any migration into the SWS requires the interaction between at least three actors:

The technical unit interested in automating either some or all its analytical processes in the SWS. The technical unit can be treated as client demanding services from the counterparts responsible for the implementation of its data and statistical assets in the SWS. Therefore, a successful FIAS - SWS migration depends on the coordination between the technical division and other parts. In the FIAS - SWS migration framework the technical unit is called FIAS and is represented by:

Stefania Vannuccini Senior Fishery Officer, FIAS
Adrienne Egger Fishery Officer, FIAS
Barbara Senfter Statistical clerk, FIAS
Thomas Berger Statistician, FIAS

The ESS - Methodological Innovation Team (ESS - SWS) responsible for the implementation and documentation of the required statistical processes. From the ESS - SWS team the focal points for FIAS - SWS process migration are:

Charlotte Taglioni Statistician (ESSD)
Carola Fabi Statistician (ESSD)

The CIO - SWS the primary backend and frontend maintainer of the SWS and responsible for the implementation and documentation of non-statistical processes (IT infrastructure). The CIO - SWS team have as interlocutors:

Enrico Anello Full Stack Developer (CIO)
John Rowell Information Technology Officer (CIO)
Matteo Terrinoni Lead Front End developer (CIO)

New FIAS methodology

The book focuses on SWS statistical processes and only recalls old methodology if useful to understand the new one. However, FIAS documents describing the old methodology can be found in the shared folder of the SWS.

Data Collection

The fisheries commodities production covers quantities of preserved and processed fishery and aquaculture commodities produced from domestic production as well as from imported raw materials.

Data include:

the quantities of preserved and processed fishery commodities domestically from nominal catches of fish, crustaceans, molluscs, other aquatic invertebrates and aquatic animals taken for commercial, industrial and subsistence purposes, by all types of fishing units operating in freshwater and marine areas; aquaculture production; imported raw material.
the output of preserved or processed fishery and aquaculture commodities produced on board of factoryships and by fishers and fish farmers’ families as domestic activities, e.g. drying, salting, smoking, etc.
the preserved and processed fishery commodities produced on board of domestic fish factoryships and fishing craft from the reporting country or area, even when landed directly in foreign ports.

Data do not include:

Fishery commodities produced on board of foreign fish factoryships or fishing craft and landed directly in domestic ports as already preserved or processed (i.e. no processing is done after landing).
“Fish, live”, “Fish fresh or chilled unprocessed”, “Live crustaceans”, “Live molluscs”, “Fresh unpeeled crustaceans” and “Fresh unshucked molluscs”, as these items cannot be considered to be either preserved or processed products.
Products derived from aquatic mammals, frogs, crocodiles, aquatic plants including seaweed, corals, sponges, pearls and fish leather.

The annual period used is the calendar year (1 January - 31 December) with the exception of few countries or areas which use a split-year in their reporting procedures. Exception list:

Bangladesh: Year ending 30 June
India: Year beginning April
Islamic Republic of Iran: Year beginning 20-23 March
Myanmar¹: Year beginning April
Nepal: Year ending 30 June
Pakistan: Year ending 30 June
Saudi Arabia: Islamic lunar (Hegira)

Data on the quantities of fishery commodities produced and on imports and exports, are expressed in tonnes and refer to the net weight of the commodities, so excluding the weight of the container and of any liquid added for preservation or flavor. The production element code used for the data is the \(5510\).

Countries are classified according to the UN M49.

Commodities data are classified according to the FAO’s International Standard Statistical Classification of Fishery Commodities (ISSCFC). ISSCFC has been developed by FAO for the collection of fishery and aquaculture commodities statistics. It covers products derived from fish, crustaceans, molluscs and other aquatic animals, plants and residues. This classification is based on the structure of the United Nations Standard International Trade Classification (SITC), with additional codes. In the ISSCFC, fisheries and aquaculture commodities are classified according to the species and to the degree of processing undergone. The latest version of the FAO ISSCFC classification is available.

Three classification master files are maintained. Currently they are in Access:

YBKlang
HS Masterfile
EUCN Masterfile

YBKlang is the main Masterfile.

The flags used for raw data are:

( , -) for official data
(M, -) for missing data for closed series
(O, -) for missing data not reported by the country and not to be imputed

All flags follow the SDMX standards.

Data also come with other information that will be listed in the processed_prod_national_detail_raw data table description.

FIAS processed production in SWS: proposed workflow

Figure . represents the essential points of the overall workflow for processed production data.

Figure .: Workflow for processed production data.

The first part of data collection and data entry is entirely performed by FIAS unit until data reach the first SWS datatable (questionnaire datatable); the second part of data imputation and validation uses two R-plugins, along with manual intervetion, to avoid unwanted overwriting in the raw datatable and an ‘ad-hoc’ Shiny application described in paragraph ??; the third part of data aggregation consists of an R-plugin to be run on the SWS.

SWS resources

SWS resources are R modules, data tables, data sets, and code lists. Data tables are typically used as auxiliary data to help R modules to achieve their goals, often a long-format four-to-six dimensional data set. The statistical domains in SWS, through code/reference lists, define the dimensions of the datasets. Therefore, datasets are primarily used to store code list - referenced values as either input and output in the SWS.

Because of the semi-standardized code list the questionnaire use to collect them, fishery processed production data are initially stored in data tables and migrated to a dataset only at the end of the process. The only dataset involved in the FIAS processed production is the Commodity dataset which has the dimensions listed in the description below.

Code lists

Code lists, also called reference lists in SWS parlance, are the dimensions making up the data sets that are designed by the user to store analytical results from SWS modules. Some dimensions are statistical-domain-specific and are defined by the technical unit to reflect its needs regarding data collection, processing, and dissemination while meeting FAO standards. Each data set dimension has a set of codes and their associated descriptions. Thus, code lists serve to the purpose of standardization, visualization, and metadata by associating standardized codes to standardized names in the SWS data set outputs. A typical SWS compliant data set has, therefore, the following dimensions/reference lists:

Geographic area. Representing a spatial scale the information is measured at. For example, countries, territories, regional aggregates, regional special groups aggregates, global aggregates. In SWS, the geographic area dimension used by FIAS data sets is named geographicAreaM49_fi. The dimension is specific for FIAS unit as it was initially decided and confirmed in a meeting held on April 10, 2019.
Items. Those one wants to take a measurement from. For example, commodities, commodity groups, land use types, species, etc. The FIAS - SWS framework item code/reference for commodities is named measuredItemISSCFC.
Elements. Often representing a measurement that can be taken across different items. For example, area, production, share. In SWS, the element dimension/code list used by FIAS dataset is named measuredElement.
Time (the time unit the data is displayed for: year, months, etc). In SWS, the time dimension used by FIAS data sets is named timePointYears.
Flag (A standardized label indicating origin and/or nature of a number in the data set, e.g. (Official number)). In SWS, the flag dimension used by FIAS data sets is named flagObservationStatus. Please check the OCS statistical standards and the flags document to understand the flagObservationStatus rational and obtain the description of flags.
Method (A standardized label indicating method utilized to obtain a number in the data set. In SWS, the method dimension used by FIAS data sets is named flagMethod. Please check the OCS statistical standards and the flags document to understand the flagMethod rational and obtain the description of flags. –>

Data tables and datasets

Data tables are mainly used to store information helping R modules to output analytical results. Information in data tables can be of a number of types. For example, conversion factors, arithmetic formulas, mapping between flags, mapping between international classifications, etc. Exceptionally, data are stored in data tables because of the initial codelist used by the FIAS unit to collect data. In SWS hierarchy, all data tables reside in a given statistical domain. In the FIAS - SWS migration framework, the data tables are in Fisheries Commodities domain. Below is a list of current available and filled data tables in the Fisheries Commodities domain:

Comm: quest processed_prod_national_detail (SWS identifier: processed_prod_national_detail_quest): where questionnaire data are initially uploaded by the FIAS unit into the SWS.
Comm: raw processed_prod_national_detail (SWS identifier: processed_prod_national_detail_raw): containing all the older time series plus the reviewed questionnaire data.
Comm: compare processed_prod_national_detail (SWS identifier: processed_prod_national_detail_compare): working datatable where new or conflictign data between the questionnaire and the raw datatable are reviewed by the user.
Comm: imputed processed_prod_national_detail (SWS identifier: processed_prod_national_detail_imputed): where raw (data from questionnaires) and estimated data are stored before being aggregated into the Commodity dataset.
Comm: ISSCFC Mapping - Export approach (SWS identifier: isscfc_mapping_export_approach): mapping table used to apply an imputation method approach based on export data.
Comm: ISSCFC Mapping - Primary Prod Approach (SWS identifier: isscfc_mapping_prod_approach): mapping table used to apply an imputation method approach based on primary production data.
ASFIS - ICS - ISSCAAP map (SWS identifier: map_asfis): mapping connecting the Asfis alphacodes to ISSCAAP groups.
SUA item mapping table (SWS identifier: fishery_item_mapping): mapping connecting the ISSCFC codes to ISSCAAP groups.

Datasets in the SWS are interchangeably used as module inputs/outputs and are composed by dimensions (aka reference/code lists). Because SWS datasets contain code lists following international standards as dimensions, there is little space for variation of dimension names across SWS datasets for the sake of standardization. In the FIAS - SWS framework for processed production, the dataset is:

Commodities (total) (SWS identifier: commodities_total): containing all Fishery commodities from production and trade data.

Only exports↩

Migration of Fishery processed production Statistical Processes into the SWS