1 Data

There are three main data sources for trade data involved in the Fishreries Trade module: UNSD, Eurostat, and TDM. The raw data are harvested by the Statistical Working System Team for both UNSD (Comtrade) and Eurostat by downloading through their API’s respectively, while the data from TDM database is provided by the FIAS. For less than ten countries, we get some information from other sources: FTR, national websites, etc.

The frequency of harvesting the trade data needs to be agreed with ESS. There is an effort of the ESS and SWST to provide the user of the trade data a tool that enables users to harvest the data whenever they want. This tool and the process to harvest trade data is currently under development and aim to harvest only data from Comtrade and Eurostat. The other source should be define by FIAS unit.

Currently, Comtrade and Eurostat data are stored in different data tables under the domain trade-input-data. The partner of data tables name that store the Comtrade data is “ct_tariffline_unlogged_YYYY”, where YYYY means the respective year, while for the Eurostat is “ce_combinednomenclature_unlogged_YYYY”. The TDM data is stored at data table “fishtrade_tdm_YYYY” under domain Fisheries Commodities. The other sources have the same standard name “fishtrade_other_sources_YYYY” under domain Fisheries Commodities. Altought, there are different data source, the data structure should have the same shape and columns.

In Table 1.1 is shown the chapters that must be downloaded to cover all commodities for the Fisheries Trade Module. The chapters were defined and validated by the technical unit.

Table 1.1: Chapters considered for Fisheries Trade module
Code Description
03 Fish and crustaceans, molluscs and other aquatic invertebrates
05 Animal originated products; not elsewhere specified or included
12 Oil seeds and oleaginous fruits; miscellaneous grains, seeds and fruit, industrial or medicinal plants; straw and fodder
13 Lac; gums, resins and other vegetable saps and extracts
15 Animal or vegetable fats and oils and their cleavage products; prepared animal fats; animal or vegetable waxes
16 Meat, fish or crustaceans, molluscs or other aquatic invertebrates; preparations thereof
21 Miscellaneous edible preparations
23 Food industries, residues and wastes thereof; prepared animal fodder

After filtering these chapters, some codes must be excluded, which are those codes that start by 030760 - Snails, Other Than Sea Snails, Live, Fresh, Chilled, Frozen, Dried, Salted Or In Brine or 160558 - Prepared Or Preserved Snails, Other Than Sea Snails. These codes are stored at data table fishtrade_param_hs_exclude under domain Fisheries Commodities.

In Table 1.2, we go through the chapter by selecting the commodities which the code starts with the code shown in the table below. In this case, the filter is more specific than the previous one.

Table 1.2: Codes considered for Fisheries Trade module
Code Description
03 Fish and crustaceans, molluscs and other aquatic invertebrates
0508 Coral and similar materials, unworked or simply prepared, shells of molluscs, crustaceans or echinoderms and cuttle-bone, not cut to shape powder and waste thereof
0509 Sponges, natural; of animal origin
051191 Animal products; of fish or crustaceans, molluscs or other aquatic invertebrates; dead animals of chapter 03, unfit for human consumption
051199 Animal products; n.e.c. in chapter 5
121220 Seaweeds and other algae; of a kind used primarily for human consumption, fresh, chilled, frozen or dried, whether or not ground
130231 Mucilages and thickeners; agar-agar, whether or not modified, derived from vegetable products
1504 Fats and oils and their fractions of fish or marine mammals; whether or not refined, but not chemically modified
1604 Prepared or preserved fish; caviar and caviar substitutes prepared from fish eggs
1605 Crustaceans, molluscs and other aquatic invertebrates, prepared or preserved
210390 Sauces and preparations therefor; mixed condiments and mixed seasonings
230120 Flours, meals and pellets; of fish or of crustaceans, molluscs or other aquatic invertebrates
230990 Dog or cat food; (not put up for retail sale), used in animal feeding

1.1 UNSD - COMTRADE

The tariff line data from UNSD contains multiple rows with identical combination of reporter/partner/commodity/flow/year/qunit (unit of quantity). Those are transactions registered separately, thus rows containinig non-missing values and quantities can be aggregated. Missing variables of the same type are also aggregated if they are all missing, as they will produce a missing aggregated value for missing disaggregated values while correctly summing the remaining variables.

  • Commodities at Tariff Line level, i.e., at the national code level. The code is composed by at least six digits up to twelve, where the first six digits follow the Harmonized System, and the other digits are country-specific.
The Harmonized System is an international nomenclature for the classification of products. It allows participating countries to classify traded goods on a common basis for customs purposes. At the international level, the Harmonized System (HS) for classifying goods is a six-digit code system.
— Harmonized System

  • UNSD Tariff line data reports area code with Tariff line M49 standard (which are different for official M49). The area code is converted in FAO country code using a specific conversion table (unsdpartnersblocks under domain trade-reference-files) provided by Team ENV.

  • Countries that are not supposed to exist in the year for which the module runs are removed from the data (e.g., Serbia did not exist as a single country in 2004).

  • European countries (as reporters) already in Eurostat data are removed.

  • Area codes not mapping to any FAO country code are separately saved and removed from further analyses. All countries mapping to code 252 (which correponds to undefined areas) are mapped to the 896 M49 code (“Other nei”).

The flow codes of re-Import (code 4) are recoded into Import (code 1) and codes of re-Export (code 3) are kept. This procedure is applied following UNSD standards:

Exports of a country can be distinguished as exports of domestic goods and exports of foreign goods. The second class is generally referred to as re-exports. The exports shown in our database contain both the exports of domestic and foreign goods. Re-exports are exports of foreign goods in the same state as previously imported; they are to be included in the country exports. It is recommended that they be recorded separately for analytical purposes. This may require the use of supplementary sources of information in order to determine the origin of re-exports, i.e., to determine that the goods in question are indeed re-exports rather than the export of goods that have acquired domestic origin through processing. Re-imports are goods imported in the same state as previously exported. They are included in the country imports. It is recommended that they be recorded separately for analytical purposes. This may require the use of supplementary sources of information in order to determine the origin of re-imports, i.e., to determine that the goods in question are indeed re-imports rather than the import of goods that have acquired foreign origin through processing. There are several reasons why an exported good might return to the country of origin. The exported good might be defective, the importer might have defaulted on payments or cancelled the order, the authorities might have imposed an import barrier, or demand or prices in the country of origin might have made it worthwhile to bring the good back.

1.2 Eurostat

  • Only numeric codes of reporters and partners are kept (letters are not allowed; basically this removes the “EU” total).

  • Only numeric CN8 codes (hs) are kept (letters are not allowed).

  • Only stat_regime equal to 4 is kept.

In this system [“Statistical regime 4” or “Total trade”], the recorded aggregates include all goods entering or leaving the economic territory of a country with the exception of simple transit trade. In particular, all goods received into customs warehouses are recorded as imports, regardless of whether they subsequently go into free circulation in the Member State of receipt. Similarly, outgoing goods from customs warehouses are included in the general trade aggregates, at the time they leave the Member State.

  • Eurostat classifies areas in their geonomenclature. These codes are converted in FAO country codes using a specific conversion table, stored in the SWS, provided by Team B/C. Area codes not mapping to any FAO country code is reported and the records for these area codes are removed. All countries mapping to code 252 (which correpond undefined areas) are mapped to the 896 M49 code (“Other nei”).

  • Commodity codes are reported in CN8 codes (Combined Nomenclature 8 digits). The codes are converted in FCL (FAO Commodity List) codes.

  • Values are converted from EUR to USD using the table, stored in the SWS, with avarage EUR/USD exchange rate for each year provided by Team B/C.

1.3 TDM and other sources

The TDM and other sources are processed using the same function that processing the data from Comtrade. Some small adaption should be done to fit well the data from these different data sources.

1.4 Data Builder

The Data Builder is a module developed to build the data by choosing the country, year, and data source for each flow (import, export). For example, the user can select the year 2019, Brazil, and TDM for both sources. Given this selection, the data builder will get the raw data from the specific data sources and then process it. The process is composed basically by, renaming columns, filtering chapter, converting M49 codes, and recode variables. After to process the raw data the Data Builder store them in the SWS as the data table fishtrade_built_data under domain “Fisheries Commodities”.

In Figure 1.1, we show the data build workflow. The user must select the country, year, and data source, and then the module will save the data built into SWS for being used for the next processes. Once that the user have build and validated the data, the next step is to perform a standardization/mapping. The details are divided between UNSD Tariff line and Eurostat due to the nature of the differences among the two datasets.

Data builder data flow.

Figure 1.1: Data builder data flow.

1.4.1 Functions

  • get_tradedata: It is a wrapper function that includes all functions for several data sources. It works like a hub.

In Figure 1.2, we show the schema to build the data, as well as the datatables and domains involved in the process.

Data builder: functions schema

Figure 1.2: Data builder: functions schema