Scottish Mentoring and Leadership Programme: impact and value for money evaluation

Glossary of key terms

Average Treatment Effect: The Average Treatment Effect is a concept used in research to understand how a specific treatment, action, or change impacts a group on average. Essentially, it is asking, 'if everyone in a group received a certain treatment, how would the average person in the group be affected? It measures the difference between what happened and what would have happened if there were no treatment.

Bias: Bias in data analysis occurs when the sample used is not representative of the population intended to be analysed. Selection bias typically arises when the selection process excludes certain types of participants or when certain groups are overrepresented or underrepresented in the sample, leading to an imbalance that affects the validity of the conclusions.

Care Experienced: A term used to describe when a person has lived in care at any time in their pre-adult life. It covers lots of different types of care such as foster care, kinship care (looked after by a family member or friend who is not the birth parent) and living in a residential home.

Confidence Intervals: Confidence intervals are ranges that are used to estimate where a true score or measurement lies, based on sample data. They provide a way to account for uncertainty by indicating a range within which the true value is likely to be found. In everyday terms, they offer a margin of error for estimations, such as providing the likelihood of a difference between two groups falling between two values.

Evaluation Methodology: This term refers to the systematic process used to assess the design, implementation, and outcomes of a programme or intervention. It involves selecting appropriate research designs, data collection methods, and analysis techniques to determine a programme's effectiveness, efficiency, and impact. Effective evaluation methodology provides evidence-based conclusions that can guide decisions on the future of the programme.

Exclusions: Exclusions refer to periods where pupils are temporarily or permanently removed from school. These can occur due to disciplinary reasons or policies that require separation from the school environment. The number of days a pupil is excluded can be an indicator of behavioural issues or challenges within the school setting. The SMLP programme aims to improve pupil engagement, reduce barriers to education, and reduce exclusion rates.

Intention-to-Treat Effect: The Intention-to-Treat Effect refers to a principle in the analysis of randomised controlled trials (RCTs) and observational studies. It involves including all participants in the groups they were originally assigned to (i.e. treatment or control), regardless of whether they completed the intervention as intended or not. This approach helps to preserve the benefits of randomisation by maintaining comparable groups and reducing selection bias. In the context of the SMLP evaluation, the Intention-to-Treat analysis compares outcomes of all eligible participants in the programme, whether they participated fully, partially or not at all, with those of a similar control group in non-participating settings. This method ensures that the results reflect the real-world effectiveness of the programme, taking into account potential dropouts and non-participation, thus providing a pragmatic estimate of the programmes overall impact.

Matched Control Sample: A matched control sample refers to a group of individuals selected from a population that shares similar characteristics with the treatment group under study, except for the intervention being evaluated. In research, particularly in observational studies or quasi-experimental designs, this method aids in mitigating confounding variables by ensuring that the compared groups are as similar as possible at baseline. This matching process often involves statistical techniques like propensity score matching (PSM) (explained below), which calculates the likelihood of participants being in the treatment group based on observed variables. By doing so, any differences in outcomes between the treatment and control groups can be more confidently attributed to the intervention itself, rather than to pre-existing differences.

Positive Post-School Destinations: Positive post-school destinations refer to the favourable outcomes that pupils achieve after they leave school. These include entering Higher or Further Education, securing a job, enrolling in training or apprenticeships, or engaging in personal development activities. The term emphasises achievements that positively influence young people's futures.

Propensity Score Matching (PSM): Propensity score matching is a statistical method used to create comparable groups for comparison when a true randomised experiment is not possible, such as when studying the effects of a programme that people choose to join. It works by calculating a ˜propensity score (the probability of being in the treatment group) for individuals based on their characteristics. Then, for each person in the treatment group, a statistically similar person from the control group is found using these scores. This ˜matches individuals and creates two groups that are more alike in their starting characteristics, reducing bias and allowing for a fairer comparison of outcomes.

Pupil Census: The Scottish Pupil Census is an annual data collection exercise by the Scottish Government that gathers statistics on pupils in all publicly funded schools in Scotland, including details on their demographics and other characteristics. This information, collected from local authorities and primarily sourced from school management systems, provides data for statistical analysis, policy development, monitoring educational performance, and resource allocation.

Regression: Regression analysis is a statistical method used to identify and quantify the relationship between a specific outcome and one or more factors that might drive it. Ordinary Least Squares (OLS) Regression does this by finding a ˜line of best fit through data, where the total distance from all the points to the line (either above or below it) is as small as possible. This shows what factors have the biggest impact and what factors can be ignored.

Scottish Index of Multiple Deprivation (SIMD): The Scottish Index of Multiple Deprivation (SIMD) is a tool developed by the Scottish Government for identifying areas with multiple deprivation across Scotland. It ranks small geographic areas (data zones) based on their relative deprivation levels across seven domains: Income, Employment, Health, Education, Access to Services, Crime, and Housing. SIMD combines these measures into a single relative index.

Scottish Credit and Qualifications Framework (SCQF) Levels: SCQF Levels are a system of twelve levels that indicate the complexity and difficulty of qualifications in Scotland, ranging from Level-1 (simplest) to Level-12 (most complex, like a doctoral degree). Level-5 qualifications tend to be taken in S4, Level-6 qualifications (Highers) in S5, and Level-7 (Advanced Highers) qualifications in S6.

Significance Levels: A significance level represents the degree of certainty that a finding is not due to random chance. For example, a 95% significance level indicates a high degree of confidence that the observed results reflect a real effect rather than a statistical fluke. Common thresholds include the 90%, 95%, and 99% levels, where higher percentages represent a stricter standard of proof and a lower risk of an accidental finding.

Staggered Treatment Design: This is an analytical approach used in evaluations of programmes or interventions that are rolled out sequentially over time, rather than implemented simultaneously across all groups. In staggered treatment designs, different groups (e.g. schools or regions) begin receiving the intervention at different times. This approach allows for the comparison between groups that have already received the intervention and those that have not yet received it, while also accounting for external factors that might change over time.

Staying-on Rates: The proportion of pupils who stay at school beyond the legal school leaving age into S5 and S6.

Waves: A ˜wave typically refers to a distinct period of data collection. In the context of this report, a wave relates to separate data for each academic year.