Agricultural tenancy data collection for Scotland: findings report

Usability assessment

The two question sets presented in the previous section were designed, reviewed, and tested by stakeholders through interviews, workshops, and a survey. The results of our usability assessment for each question set are presented in this section, combining qualitative insights from interviews and workshops with quantitative survey results.

Visual heatmap

The heatmap below summarises the relative performance of each set across key usability considerations:

The scoring criteria were based on the following scale:

• Green: Performs very well; meets objectives with minimal risk, effort, or barriers. Strong likelihood of positive outcomes.
• Amber: Performs moderately; meets objectives with some effort, uncertainty, or minor barriers. Partial risk of issues or limitations.
• Red: Performs poorly; fails to meet objectives, with high effort, uncertainty, or significant barriers. High risk of issues or failure.

Overview of usability by question set

To complement the visual assessment, the following section presents qualitative commentaries on the underlying factors influencing usability. These provide further context on the strengths, limitations, and practical implications of Set A (JAC) and Set B (SAF), as well as opportunities for improvements or adaptations. The recommendations presented cover both short-term refinements and more substantial, longer-term considerations for the Scottish Government.

Usability assessment: Set A (JAC)

Set A (JAC) was endorsed by some stakeholders for its operational feasibility and sample coverage, with the reintroduction of tenancy questions through the JAC viewed as the most practical approach. From a respondent perspective, building on the original questions, rather than introducing major changes, was seen as a way to minimise confusion. Maintaining direct, simple, and logical questions, consistent with the discontinued agricultural tenancy questions, was regarded as the most user-friendly approach. While these stakeholders acknowledged the risks associated with previous data quality concerns, most agreed that the targeted refinements made to terminology, clarity, and in-form guidance would improve comprehension.

However, Set A (JAC) also raised several concerns. The most significant drawback, mentioned by nearly all stakeholders, was the lack of motivation to complete the census return. This was seen to directly undermine response reliability, as respondents face no penalties or follow-up for incomplete or inaccurate returns. As discussed in the Context section, this was also considered a key factor contributing to the 2021 JAC’s data quality concerns.

Moreover, some stakeholders viewed the data granularity of the JAC as less useful than that of the SAF, since it collected data at the holding level rather than the land parcel level. However, others noted that Set A (JAC) could still enable stronger linkages between tenancy information and broader farm data – such as livestock or crop records – through other JAC items, providing policymakers with a more comprehensive view of the agricultural sector. Given these differing perspectives, data granularity may not be a decisive factor when choosing between the two question sets.

As a result of reportedly increasing administrative pressures on farmers and land managers, optional forms like the JAC were described as often deprioritised and perceived to involve higher completion effort. Stakeholders also highlighted that the JAC is issued during particularly busy farming periods and shortly after SAF submissions, further increasing respondent burden and reducing the likelihood of timely and accurate completion.

Finally, both Set A (JAC) and Set B (SAF) were considered moderately effective in their ability to identify unfamiliar tenancy types, particularly when paired with the guidance in Appendix A, due to their use of direct questions. While this was not viewed as a major factor behind the JAC’s previous data quality issues, stakeholders noted that, without clear incentives or consequences to complete the form, respondents who were uncertain about their tenancy details would be less likely to check the accuracy of their responses compared with Set B (SAF).

Recommendations for improved usability of Set A (JAC)

Recommendations to enhance the usability of Set A (JAC), informed by insights collected through interviews, workshops, and survey responses, are outlined below. They are organised according to key usability considerations, prioritised based on their relative importance as indicated by the frequency with which stakeholders mentioned them. It should be noted that these recommendations have not been incorporated into the redesigned question sets presented earlier, as they extend beyond the scope of simply redesigning the questions included in the 2021 JAC.

• To reduce completion effort, several stakeholders strongly recommended removing the requirement to report land to the nearest 0.01 hectare. While this level of precision is beneficial for data linkage and consistency with Scottish Government datasets, it was considered highly technical and burdensome for respondents. Stakeholders also noted that the margin of error at such precision can be substantial, risking spurious precision. Possible alternatives included: 1) introducing range categories (e.g., 0-50, 50-100 hectares) while making reporting to the nearest 0.01 hectare optional for respondents who know this information; or 2) reducing the level of granularity, for example, reporting to the nearest hectare.
• To improve data granularity, many stakeholders recommended itemising tenancy types, capturing how many of each tenancy type a respondent holds rather than the total number of hectares for each tenancy type (which may include combining more than one tenancy of the same type). This was described as a “critical variable” for monitoring the sector and assessing the impact of legislative reforms, such as the separation of housing from agricultural tenancies. Collecting this information would provide a valuable basis for understanding trends, movements, and decision-making within the sector.
• To further reduce completion effort, some stakeholders suggested removing questions that require reporting sum totals. Collecting this information manually adds effort for respondents and increases the risk of inaccuracies, whereas the totals could instead be calculated automatically during data analysis.
• To improve response reliability and response motivation, stakeholders recommended that the Scottish Government host or promote drop-in sessions to support completion of the JAC. These sessions could be held at agricultural events, facilitated by trusted sector organisations, or at local area offices. They could also be used to communicate the purpose, benefits, and importance of completing the JAC.
• To further enhance respondent motivation, stakeholders also suggested automatically sending paper forms to respondents who have not yet submitted electronic responses within a predetermined timeframe. Paper forms were considered more effective at capturing attention and provided an accessible alternative for respondents with lower digital literacy.

Usability assessment: Set B (SAF)

Set B (SAF) was viewed as the most effective option for improving response reliability. This was primarily attributed to the SAF’s direct link with financial subsidies, which would strongly motivate respondents to complete the tenancy questions accurately and in full. Stakeholders noted that, compared with the JAC, the inclusion of these questions in the SAF would almost “guarantee” completion.

As noted above, both Set A (JAC) and Set B (SAF) were considered moderately effective in their ability to identify unfamiliar tenancy types. However, because of its financial link, the SAF was seen as more likely to prompt respondents who were uncertain about their tenancy details to verify their tenancy records, rather than provide unreliable or incomplete responses.

The completion effort associated with the SAF was also generally perceived to be lower than that of the JAC, despite the SAF’s more detailed data requirements. This was largely because many farmers and land managers use land agents to complete the form, and because key documentation, such as tenancy records and land parcel information, is already accessed during the SAF process and therefore appears less burdensome.

From a data user perspective, the SAF was preferred by many stakeholders for its greater data granularity, as it collects data at the land parcel level rather than at the holding level (as in the JAC). This level of precision was considered particularly valuable to analysts and policymakers, enabling a more detailed understanding of the sector and facilitating efficient linkage with other Scottish Government datasets.

However, the most significant drawback of Set B (SAF) for stakeholders was its sample coverage, particularly the potential exclusion of some agricultural holdings currently captured by the JAC. A preliminary, high-level analysis drawing on JAC, LPIS, and SAF holdings data was conducted for this project, and it estimated that between 13,000 and 18,000 holdings could potentially be missed if tenancy data collection were moved from the JAC to the SAF. This is because the number of holdings associated with businesses submitting a SAF is estimated at approximately 32,000, compared with a total JAC population of 45,000 to 50,000 (exact figures to be confirmed with the SAF and JAC teams).

Although the difference in count appears substantial, this initial analysis estimated that, in area terms, the overlap between the populations is 90%. Moreover, these figures refer to all agricultural holdings rather than holdings held by tenant farmers only. This could indicate that most omissions may relate to smaller holdings that are not part of payment schemes. As outlined in the recommendations below, further analysis should be conducted by the Scottish Government to confirm these figures.

Finally, some practical considerations related to operational feasibility were raised by a small number of stakeholders. These included concerns about whether adding tenancy questions might overcomplicate the SAF for both respondents and analysts, though this was significantly outweighed by stakeholders who supported the SAF for its lower completion efforts. Others raised concerns about potential technical or design constraints within the current SAF system that could affect how or when the questions are integrated. Moreover, from a respondent perspective, the SAF submission period was also noted as a particularly busy time, coinciding with key agricultural activities (such as lambing and calving) and with land agents managing multiple submissions simultaneously, which could pose risks to both response rates and data accuracy.

Recommendations for improved usability of Set B (SAF)

Recommendations for enhancing the usability of Set B (SAF), informed by insights collected through interviews, workshops, and survey responses, are summarised below. These are divided between general and usability considerations, and ordered by how frequently they were mentioned by stakeholders.

• To enhance sample coverage, it is first recommended that the Scottish Government undertake a comprehensive assessment to determine the number of holdings potentially omitted by the SAF compared with the JAC, in order to establish a more reliable estimate. If this assessment confirms a substantial gap, some stakeholders suggested introducing an additional data collection channel, such as a standalone survey, to capture equivalent data for any agricultural holdings not covered by the SAF. Any such questions would need to align with the granularity of the SAF (i.e., data collected at the land parcel level) to ensure compatibility for data linkage across both vehicles.
• More broadly, stakeholders emphasised that, given the financial implications of the SAF, any new tenancy questions must be clearly presented and effectively communicated to avoid unnecessary confusion for respondents or causing delays to subsidy payments. While the detailed design of this approach is outside the scope of this research, it is noted for consideration that clarity is required on whether the new questions have any direct link to payment eligibility and whether completion of these questions is mandatory or optional.