Chapter 2 The Quality Indicators
This chapter describes the QI to be calculated in the SWS. These indicators have been proposed by OCS and endorsed by the IDWG. Nevertheless, for a better description and information regarding the indicators below, please read the following document.
The indicators can be split into eight dimensions:
- Relevance
- Accuracy
- Reliability
- Timeliness
- Punctuality
- Coherence
- Comparability
- Accessibility
We will focus on the description and implementation of two dimensions: Accuracy and Reliability. Although all quality dimensions are very important and should be taken into account in order to measure the quality of a statistical output, accuracy and reliability are the only ones that can be measured using the disseminated data. The remain ones are feasible to measure once the technical units provide us with information regarding their calendars/deadlines.
2.1 Accuracy
According to the Statistical Standard Series, accuracy is “the closeness of an estimate to the true value of what is measured. In this case, we are not interested in the accuracy of the incoming country data, but rather in assessing solely FAO’s contribution to the overall accuracy of the final statistical outputs calculated and disseminated by FAO (usually regional or global aggregates).”
Below you find the indicators implemented in the SWS.
2.1.1 Questionnaire reporting rate
\[\begin{equation} QRR = \frac{Q_R}{Q} \times 100 \end{equation}\]
Where \(Q_R\) is the number of returned questionnaires completely or partially filled with valid information (a questionnaire partially filled whose information is not used for calculation of the final statistical outputs because not relevant or because being of poor quality should be counted as a non-response) and \(Q\) is the number of dispatched questionnaires.
2.1.2 Percentage of missing data points
\[\begin{equation} PM = \frac{n_{Missing}}{n_{Relevant}} \times 100 \end{equation}\]
\(100-PM\) returns the percentage of available data points.
2.1.3 Percentage of official data points
\[\begin{equation} RO = \frac{n_{Official}}{n_{Available}} \times 100 \end{equation}\]
2.1.4 Percentage of imputed data points
\[\begin{equation} RI = \frac{n_{Imputed}}{n_{Available}} \times 100 \end{equation}\]
2.1.5 Percentage of mirror data points
\[\begin{equation} RM = \frac{n_{Mirror}}{n_{Available}} \times 100 \end{equation}\]
2.1.6 Contribution of imputed values to the final aggregate
\[\begin{equation} CIVT = \frac{\sum_{j=1}^{n_I} y_{Ij} + \sum_{j=1}^{n_F} y_{Fj}}{\sum_{j=1}^{n_{available}} y_{.j}} \times 100 \end{equation}\]
In practice, the denominator, \(\hat t_y = \sum_{j=1}^{n_{available}} y_{.j}\) is the estimated total of the variable Y (at the “World” or regional level) that is disseminated externally and the numerator is the sum of the values that are obtained through imputation (flags “I” and “F”).
2.1.7 Contribution of official values to the final aggregate
\[\begin{equation} COVT = \frac{\sum_{j=1}^{n_{official}} y_{official}}{\sum_{j=1}^{n_{available}} y_{.j}} \times 100 \end{equation}\]
2.1.8 Contribution of questionnaire values to the final aggregate
\[\begin{equation} CQVT = \frac{\sum_{j=1}^{n_{quest}} y_{quest}}{\sum_{j=1}^{n_{available}} y_{.j}} \times 100 \end{equation}\]
2.1.9 Contribution of mirror values to the final aggregate
\[\begin{equation} CMVT = \frac{\sum_{j=1}^{n_{mirror}} y_{mirror}}{\sum_{j=1}^{n_{available}} y_{.j}} \times 100 \end{equation}\]
2.2 Reliability
As stated by the Statistical Standard Series, reliability “indicates how close the initial estimates are to the subsequent or final estimates.”.
The revision indicators are presented hereinafter.
2.2.1 Mean Revision
The Mean Revision is an average of the differences between the latest and the previous disseminated value over all terms of the time series:
\[\begin{equation} MR = \frac{1}{(t_m - t_0 +1)} \sum_{t=t_0}^{t_m}(X_{Lt} - X_{Pt}) \end{equation}\]
\(X_{Lt}\): latest available estimate of the aggregate (regional or “World” level) for the variable of interest at time t;
\(X_{Pt}\): previous available estimate of the aggregate (regional or “World” level) for the variable of interest at time t;
\(t_0\): starting year in the time series: the starting year should not be a year before a break in the time series;
\(t_m\): last year in the time series.
2.2.2 Mean Absolute Revision
\[\begin{equation} MAR = \frac{1}{(t_m - t_0 +1)} \sum_{t=t_0}^{t_m}|X_{Lt} - X_{Pt}| \end{equation}\]
2.2.3 Relative Mean Absolute Revision
\[\begin{equation} RMAR_L = \frac{\sum_{t=t_0}^{t_m}|X_{Lt} - X_{Pt}|}{\sum_{t=t_0}^{t_m}X_{Lt}} \end{equation}\]