Agricultural tenancy data collection for Scotland: methodology report

Methodology

This section outlines the methodological process undertaken to develop potential routes for collecting agricultural tenancy data. Our research approach was designed to produce practical, effective, and user-friendly question sets. It involved three phases:

1. Phase 1: Discovery. To develop an initial understanding of the factors influencing historic agricultural survey approaches and identify best practice examples, an exploratory literature review was conducted along with a high-level review of agricultural surveys from other comparable countries. In parallel, the potential issues that contributed to data quality concerns for agricultural tenancy questions in the historic JAC were discussed with sector experts through ten scoping interviews. The outcomes of these activities were consolidated into a diagnostic Issue Map that defined overarching concerns and challenges to inform the next phase.

2. Phase 2: Question Co-design. This phase began with the development of a long list of tenancy questions, reflecting the initial issues and best practice identified in the discovery phase. The long list was then reviewed and tested with sector experts through three workshops (and one written submission), gathering feedback on several usability considerations. To review the legal accuracy of our materials and assess the feasibility of the proposed Single Application Form (SAF) question set, two additional in-depth interviews were conducted with a legal expert and a Scottish Government representative. Feedback was integrated into the Issue Map and the long list of tenancy questions to produce refined versions for testing in the subsequent phase.

3. Phase 3: Question Testing. This final phase involved testing the revised long list of questions with farmers (both tenant farmers and owner occupiers) and land managers through five in-depth interviews and 15 survey responses. The purpose of this phase was to assess the clarity, usability, and accuracy of the question sets in a real-world context, and to arrive at a suitable short list of tenancy questions to recommend to the Scottish Government for potential inclusion.

These three phases are described in detail below, outlining the research activities undertaken and the decision-making processes that informed the development of the final question sets presented in the accompanying report.

Phase 1: Discovery

The Discovery phase focused on identifying common concerns and challenges surrounding the collection of agricultural tenancy data. Insights from this stage informed the development of best practice principles to improve data quality and ensure our draft tenancy questions developed in Phase 2 were clear, usable, and fit for purpose.

Literature review

First, an exploratory literature review was undertaken. This involved an examination of documents, reports, and technical notes related to the JAC and SAF, as well as tenancy data collection and the agricultural sector more broadly. This included:

• Publications relevant to the Scottish agricultural tenancy sector from key organisations, such as the Scottish Government, the James Hutton Institute, and the Central Association of Agricultural Valuers (CAAV).
• Publicly available guidance for the JAC and the SAF.
• Historic JAC outputs and trends, published by the Scottish Government.
• Information and trends for crofting in Scotland.
• Research papers on improving the quality and policy relevance of agricultural data.
• Studies and papers on best practice for the design and delivery of agricultural data.

A high-level keyword search was conducted across academic databases (e.g., ScienceDirect) and selected government and industry websites (e.g., the Scottish Government, the James Hutton Institute) to identify relevant academic and grey literature (e.g., government reports, policy documents, published guidance).

In total, 20 papers published between 2000 and 2025, from both the United Kingdom and international contexts (e.g., the United States, South Africa), were identified as relevant and reviewed in full. This approach ensured coverage of the most up-to-date and comparative literature. The complete list of documents reviewed is presented in Appendix F.

The literature review gathered insights on the barriers and enablers to high-quality tenancy data collection. To complement this, agricultural tenancy data collection practices were examined in comparator countries, including Wales, England, the United States, Canada, and New Zealand. This international review explored how other jurisdictions collect agricultural tenancy data, address data quality challenges, and manage associated risks. Examples of best practice approaches identified included providing clear respondent instructions, using plain language, and offering sufficient context regarding the survey’s purpose and confidentiality. However, due to the limited published literature available and the contextual specificity of agricultural tenancy legislation in Scotland, the applicability of international insights was limited.

Scoping interviews

In parallel, discussions on agricultural tenancy data were held with sector experts through ten online scoping interviews. These discussions explored historical and current factors influencing the quality of tenancy data, the value of collecting such data, and potential approaches for improving data collection (including from international comparators). Emerging findings from the literature review were tested during these interviews, and participants were also asked to provide additional recommendations for relevant sources to supplement the desk-based research.

Diagnostic Issue Map

Following the completion of research activities, findings from the literature review and scoping interviews were consolidated into a diagnostic Issue Map. The map categorised the initial issues identified according to six distinct categories: 1) Issue ranking, 2) Issue identified, 3) Causal category, 4) Primary impact type, 5) Effect on data quality, and 6) Perceived impact. This initial Issue Map provided the foundation for the subsequent stages of the research.

N.B. The version presented in Appendix A has been updated from the original produced at the start of the project. It incorporates refinements made through later phases to reflect stakeholder feedback and the evolving understanding of potential data quality issues. A full written summary of the issues is provided in the accompanying Findings Report.

Key output from Phase 1: Initial version of the diagnostic Issue Map (final version can be found in Appendix A).

Phase 2: Question co-design

Phase 2 focused on translating the insights from the Discovery phase into practical and testable sets of agricultural tenancy questions. This involved close collaboration with our delivery partner at the James Hutton Institute and sector experts to review, refine, and validate the developing question sets before proceeding to farmer testing in the next phase.

Long list of agricultural tenancy questions

The draft long list of tenancy questions was developed using insights generated during the Discovery phase. Each of the three question sets was designed to address specific issues identified in the diagnostic Issue Map, and included:

1. Question Set A: June Agricultural Census (Targeted refinements) – This set retained the overall structure of the JAC 2021 tenancy questions but introduced targeted refinements to terminology, clarity, and in-text guidance based on initial stakeholder feedback. It was developed on the assumption that the primary drivers of previous data quality concerns were respondents’ reluctance or limited motivation to share tenancy information, rather than flaws in the structure or presentation of the questions themselves.

2. Question Set B: Single Application Form (Alternative vehicle) – This set explored the feasibility of introducing agricultural tenancy questions within the SAF, leveraging the greater importance respondents place on this form due to its direct link to financial subsidies. It assumed that data quality issues may have stemmed from lower respondent motivation when completing the JAC compared with the SAF.

3. Question Set C: June Agricultural Census (Branching logic) – This set adapted the original JAC questions into a branching logic format. Respondents were first asked to report any tenancy type(s) they were familiar with. If they indicated holding tenancy types they were unfamiliar with, they were presented with a series of indirect branching questions (e.g., “What was the original start date of the tenancy?”) designed to infer the relevant tenancy type(s). This approach was based on the assumption that data quality concerns may reflect difficulties farmers and land managers face due to legislative complexity.

To accompany the question sets, we also prepared supplementary guidance (Appendix E) informed by initial stakeholder feedback. The guidance was designed to help respondents accurately report their tenancy arrangements.

It is important to clarify that this research focused on how the Scottish Government could enhance the collection of tenancy data and improve overall data quality. Making changes to the data collected through the original census questions, such as adding, modifying, or removing variables, is ultimately a longer-term consideration for the Scottish Government, to be taken forward in line with evolving data needs and policy priorities. Therefore, the revised question sets retain the original items from the discontinued JAC 2021 agricultural tenancy questions, while the accompanying Findings Report outlines stakeholder recommendations for additions, modifications, or removals for the Scottish Government to consider.

An overview of the three question sets is presented in the table below, including a summary of their key features, rationales, and main trade-offs. Both the question sets and accompanying guidance were reviewed by our delivery partner at the James Hutton Institute before presenting these to sector experts to ensure accuracy, relevance, and methodological rigour.

Question sets

Engagement with sector experts

After developing the draft long list of tenancy questions, we conducted three online workshops with sector experts (n=13) to test the accuracy, usability, and feasibility of each question set. Discussions also explored the issues identified in the Issue Map, assessing whether these aligned with participants’ practical experiences and identifying any gaps.

To make the best use of participants’ expertise, the workshops were structured around three stakeholder groups:

1. Legal experts (i.e., professional bodies and advisory organisations with expertise in agricultural tenancy matters).

2. Representative industry bodies (i.e., organisations covering landownership, farming, and tenant interests).

3. Scottish Government representatives with expertise in agricultural tenancy matters. Participants were identified through a mapping exercise informed by the Scottish Government, our delivery partner at the James Hutton Institute, experts engaged during the scoping phase, and referrals from contacted participants.

Each workshop lasted 90 minutes and followed a semi-structured discussion guide, ensuring that core research questions were addressed while allowing flexibility for unanticipated insights. We also accepted written submissions from stakeholders who were unable to attend but wished to contribute; one written submission was received and analysed alongside the workshop data.

In parallel, we carried out two additional in-depth interviews with targeted stakeholders to examine specific elements of the question sets in greater detail. One interview was held with a legal expert to review the accuracy of the branching logic and accompanying guidance (see Appendix E), and another with a Scottish Government representative to assess the feasibility of introducing agricultural tenancy questions into the SAF.

Long list refinement

All data collected during this phase was analysed thematically and categorised according to key insights, areas of consensus, and recommended revisions. This enabled us to identify which elements of the draft question sets were working well, where stakeholders raised concerns, and where further refinement was required. These findings were then used to refine both the long list of tenancy questions and the diagnostic Issue Map, ensuring that each reflected stakeholder expertise and the practical realities of agricultural tenancy data collection in Scotland.

It should be noted that at this stage, stakeholders regarded Set A: JAC (Targeted refinements) and Set B: SAF (Alternative vehicle) as the most user-friendly options. In comparison, Set C: JAC (Branching logic) was the least preferred, with stakeholders raising concerns that it was overly burdensome and risked coming across as patronising. It was also considered to carry the greatest risk of misclassifying tenancy types because the branching relies on inferring classifications rather than reporting them directly – a particular challenge given the complexity of Scottish tenancy legislation. Many also rejected the premise that lack of clarity among farmers and landowners about tenancy type was the main cause of data quality issues that led to the removal of the previous JAC tenancy questions, rendering the rationale for Set C less convincing. However, despite this, Set C was retained for the final phases of farmer testing to validate these assumptions.

Key output from Phase 2: Refined long list of agricultural tenancy questions and updated Issue Map.

Phase 3: Question testing

The final phase involved testing the refined question sets with farmers and land managers through a structured programme of activities to assess clarity, usability, and accuracy in a real-world context. The aim of this phase was to arrive at a short list of tenancy questions, which could be recommended to the Scottish Government for potential inclusion.

A summary of the testing undertaken for each question set is provided below:

It should be noted that Set B could not be directly tested through the survey. As described in the previous phase, this set proposed introducing agricultural tenancy questions into the SAF rather than reinstating them in the JAC, in order to leverage the SAF’s direct link with sector support payments. Although the question design is largely similar to Set A, with minor adaptations to reflect the SAF’s structure and level of data collected, the critical feature of Set B, in comparison, is the SAF vehicle itself. For this reason, it was not possible to test the usability or effectiveness of the set within this research.

However, farmers and land managers were still invited to provide feedback on the usability of Set B through interviews and indirect survey questions (e.g., ‘Would embedding agricultural tenancy questions in the SAF make it easier for you to provide accurate information?’). Where feedback on terminology or language in Set A was relevant, corresponding refinements were also applied to Set B.

Interviews with farmers and land managers

One-to-one interviews (n=5) were conducted with tenant farmers and land managers first to validate the issues identified in the earlier phases, gather feedback on the three question sets, and undertake cognitive testing to understand how respondents interpreted and answered the three-part survey draft developed for wider testing (see below).

Each interview lasted approximately 30 minutes and followed a semi-structured guide, allowing space for unanticipated insights relevant to the research. Findings were analysed thematically and used to refine each question set, update the Issue Map, and adjust the survey draft before progressing to the next phase of testing.

Survey with farmers and land managers

Two surveys were conducted to test the usability of question sets A and C with a broader pool of farmers (both tenant farmers and owner occupiers) and land managers. The aim was to determine which question design format provided the clearest, most accurate, and most user-friendly method for reporting agricultural tenancy information.

The two surveys were developed to gather comparative insights on Set A and Set C, and comprised four sections:

1. Section 1: Question testing: Respondents completed either Set A or Set C, depending on the link they were provided, and were asked to answer as accurately as possible.

2. Section 2: User experience: This section captured respondents’ feedback on the question set they completed, including general impressions, comprehension, and perceived accuracy.

3. Section 3: Alternative approaches: Respondents were presented with the two alternative question sets to provide indirect feedback and allow high-level comparison, including which set they considered easiest to complete and most user-friendly.

4. Section 4: Demographics: Optional demographic questions collected information about respondents and their agricultural holdings. This data was used solely to monitor the diversity of responses.

Both questionnaires were distributed by Alma Economics, the Scottish Government, and the James Hutton Institute, and were also shared directly with seven additional organisations with reach to farmers and land managers across Scotland. The survey links were evenly assigned across these organisations to attempt to achieve balanced response rates for the two designs. The surveys were open to farmers and land managers holding an agricultural tenancy in Scotland and remained live for a three-week period, from 7th October 2025 to 21st October 2025.

A total of 15 responses were received: ten respondents completed the Set A questionnaire, and five completed Set C. A breakdown of respondent demographics is presented below:

To analyse the survey results, we produced descriptive statistics for responses to Section 2 (User experience) to assess the performance of each set and reviewed comparative responses from Section 3 (Alternative approaches). Findings for both surveys are provided in Appendices B and C.

There are two key limitations to consider when interpreting the survey findings:

1. Low survey uptake: Despite wide distribution, overall participation was limited, with 15 survey responses received in total. Consequently, results are presented as raw counts rather than percentages (see Appendices B and C) and should be interpreted with caution. There is also potential selection bias as respondents may be already familiar with, and comfortable reporting, their tenancy information.

2. Limited testing of branching logic in Set C: Set C was designed to guide respondents who were unsure of their tenancy type(s) through indirect branching questions. The first part of the survey included direct questions to report tenancy agreements (identical to Set A), with a follow-up question asking respondents to indicate whether they would like to report any additional tenancy types but are uncertain about their classification. If respondents answered ‘yes,’ the branching logic would be triggered. However, all participants who completed this questionnaire indicated that they already knew their tenancy types, so the branching questions were not triggered. Because the branching functionality in Set C was not activated during testing, its performance could not be fully assessed. Given this, responses for Section 2 (User experience) in Set C were not deemed suitable for direct comparison with Set A.

Despite their limitations, the survey findings provided valuable insights from farmers and land owners that complemented the in-depth interviews and strengthened the overall evidence base. To ensure the robustness of our conclusions, these findings were not used as a standalone basis for decision-making but triangulated with evidence from other research activities to validate and test emerging insights. All resulting recommendations are presented transparently in the accompanying Findings Report, with any remaining gaps and areas requiring further investigation clearly highlighted.

Usability assessment

To arrive at a short list of tenancy question sets, insights from the interviews, survey, and engagement in previous phases were synthesised collectively to assess the overall suitability of each option. Each question set was evaluated against seven usability criteria, covering factors such as completion effort, response reliability, and data granularity. A visual heatmap was developed to summarise the relative performance of each set and illustrate the trade-offs involved in selecting one approach over another. The full visual heatmap is included in Appendix D.

The usability assessment indicated that Set A and Set B offered the strongest overall performance. Both sets received the greatest positive feedback from sector experts, farmers, and land managers across most criteria. Importantly, each addressed the key issues stakeholders believed were likely to have contributed to data quality concerns in the previously discontinued JAC tenancy questions.

While some survey respondents viewed Set C as user-friendly in principle, with around half in one survey rating it more favourably than Set A when presented with its rationale (see questions 13, 14, and 15 in Appendix B), this was not widely supported by evidence from other research phases. As noted earlier, many stakeholders felt that Set C was too burdensome given the length and complexity of the branching structure, risked coming across as patronising, and introduced too much risk by inferring tenancy types.

Critically, we also found no evidence that most farmers or land managers were unable to justify their tenancy type without assistance. In the branching logic version of the survey, respondents were first given the option to report their tenancy types directly, and none progressed to the branching stage, indicating that they already knew and were comfortable selecting their tenancy type. This aligned with workshop and interview insights, where stakeholders reported that, despite the complexity of tenancy legislation, most farmers and land managers were aware of their tenancy arrangements or could readily access the relevant documents if sufficiently motivated. Given that Set C was premised on the hypothesis that data quality issues stemmed from uncertainty about tenancy type, this lack of evidence weakened the rationale for the approach.

Taken together, these findings indicate that the branching logic approach was the least effective of the three question sets, as it introduced unnecessary complexity by attempting to infer tenancy types while offering limited benefit in addressing the suspected causes of past data quality challenges. Consequently, Set C was considered the least favourable option and excluded from the short list. However, stakeholders noted that the decision tree adapted from the branching logic could still serve as useful supporting guidance Set A and Set B for any respondents who may be unsure of their tenancy type. It is included in Appendix E and in the accompanying report.

Key output for Phase 3: Short list of two agricultural tenancy question sets, presented in the accompanying Findings Report.