Accuracy and Reliability

This section discusses how accurately and reliably these statistics portray reality.

Data quality

The quality of information collected from each farm is very high as data are based on fully reconciled farm accounts. Data for each farm is also validated against a comprehensive set of quality assurance checks.

The potential for processing errors is regarded as low risk. This is due to much of the collection being based on reconciled accounts, the extensive use of cross-checking validation routines and that the vast majority of farms have previous records in the survey which are used to identify inaccuracies in returns.

Results are examined alongside previous outputs and related evidence from alternative sources in order to ensure that the data and methods being used are reliable.

Potential outlying results that may have an impact on the analysis are also identified. Such outliers, when identified, may be excluded from specific analysis to ensure that the results are representative of the population being described.

Although the quality of information for each farm business in the survey is considered to be high, relatively low sample sizes do mean that the results are subject to a degree of uncertainty in terms of representing overall national averages by farm type. Weighting is applied to account for representation as best as possible, more information is available in Farm Business Income (FBI) methodology

In some cases, accounts may not be finalised until after the deadline for submission of data. In such cases estimated records are updated and the published figures are revised in the following year. In this sense, the first release of data for a particular year may be regarded as provisional.

Sampling error and bias

A total of 397 farms were in the 2023-24 sample. This accounts for 4% of the total relevant agricultural holdings in Scotland. As the survey does not cover the entire population, the FBI estimates are susceptible to sampling error.

As the data collected through the FBS is of a highly sensitive nature, the refusal rate of farms approached to participate is high. Non-responders (farms refusing to participate) may have different characteristics to responders (farms willing to cooperate), which could lead to biased results. Currently there has been no assessment of non-response bias in the FBS for Scotland.

Sampling effects

The sampling strategy of the FBS is based on a stratified simple random sample and is effectively designed as a panel survey with little change in the membership of the sample between years. The amount of sample change varies year on year.

In most years, sample turnover is small. In some years, a larger turnover or turnover of farms with specific characteristics will mean that changes in average farm income are more strongly influenced by sample variation.

In 2023-24 there was a larger than normal turnover (higher than seen in the previous 3 years) in the samples of livestock and dairy farms. A larger turnover does not necessarily have noticable impacts on results. However, in 2023-24 the proportion of smaller grazing livestock farms in the overall sample has increased slightly compared to the previous year. This has contributed to some stronger decreases in average values compared to the previous year.

The survey has actively recruited an increased number of smaller farms in order to improve proportional representation of the sample against the wider population. This ensures that individual farms do not end up representing unrealistically large proportions of the whole population when weighting is applied and ensures that results are as accurate and reliable as possible.

As farm income varies each year for all farms, it is difficult to determine the size of the impact of sample turnover on reported estimates. The changes in the average reported figures as a result of sample change are also likely to be smaller than these might be for a random sample drawn from the population each year.

Where changes are strongly attributed to changes in the sample representation, these are reported on alongside the official statistics.

For example, in 2023-24 the official statistics publication notes that:

For LFA cattle and sheep farms, the fall in income was mostly driven by a decrease in payment schemes output of 13% (£9,300). In 2023-24 the fixed rate of the Basic Support Payment Scheme did not increase with inflation. This resulted in a real terms decrease in support payments for many farms. Changes to farms included in the sample has also contributed to the observed fall in payment scheme outputs. On average, for LFA cattle and sheep and LFA sheep farm types, farms entering the survey were receiving lower payments from the Basic Payment Scheme than those exiting.