Early learning and childcare expansion programme: evaluation strategy

2. Overview of the evaluation strategy

2.1. Development of the evaluation strategy

2.1.1. Development of the evaluation framework for the expansion programme has taken place alongside policy development and implementation. This strategy further develops the evaluation framework set out in the Evaluability Assessment undertaken by NHS Health Scotland^[21] and progress reporting,^[22] including setting out our approach to the evaluation of child and parent outcomes, the measurement of family wellbeing and to assessing the longer-term economic benefits.^[23]

2.1.2. The strategy was being finalised in early 2020, however, publication was put on hold due to competing demands within Scottish Government in response to the Covid-19 pandemic. Research and analysis for the evaluation has continued during this period. The strategy has subsequently been reviewed and updated to reflect the impact of the pandemic, the necessary delay to the statutory implementation of the ELC expansion on evaluation plans and the current policy context.

2.1.3. Evidence requirements for the strategy will be kept under review. We will continue to consider, in an ongoing way, if our evidence sources are robust and comprehensive enough.

2.2. Governance and stakeholder engagement

2.2.1. In developing the evaluation approach, it was recognised that to be successful the strategy should:

• be possible to implement within reasonable timescales so learning can feed into policy implementation and development
• be proportionate, minimise duplication and costs, and utilise existing data as far as possible
• have a shared vision, understanding and ownership by all stakeholders
• be flexible and able to adapt to policy changes as the work progresses

2.2.2. In line with these principles, the evaluation methodology makes use of existing sources of information and reporting processes, along with commissioning of specific and targeted research. We are aware of the burden of information requests made of providers and local authorities.

2.2.3. It is also vital that stakeholders with an interest in the ELC expansion programme have information about the overall strategy, contribute to the evaluation process and participate in the research being undertaken. Key stakeholders were involved in the initial Evaluability Assessment. Scottish Government has worked very closely with PHS to develop and take forward the evaluation approach. The Scottish Government set up and continues to chair a Evaluation Working Group to oversee the evaluation approach (see Annex A for membership of the group). We are grateful to all those who have contributed time and expertise.

2.2.4. In addition, the work outlined here will feed into the Tackling Child Poverty Evaluation Strategy.^[24] Expanded ELC is one of the shortlisted policy actions included in the Child Poverty Evaluation Strategy, as it is anticipated that it will contribute to meeting the child poverty targets through influencing two of the three drivers of child poverty (income from employment and costs of living).

2.3. Aims and scope of the strategy

2.3.1. The overarching aims of this evaluation strategy are twofold: firstly, to produce robust evidence to assess the contribution of the ELC expansion to 1140 hours to outcomes for children, parents and families and to assess the longer term economic costs and benefits; and secondly, to set out the data that will be drawn on to provide ongoing evidence at a national level to continue to support realising the benefits of the expansion.

2.3.2. The aims and core questions for the ELC expansion evaluation have been developed based on the Theory of Change for the expansion programme set out in the Evaluability Assessment. The full Theory of Change is in Annex B. This Theory of Change is based on the principles of Getting it Right For Every Child (GIRFEC) – the Scottish Government's national approach to supporting children, young people and their families. As noted above, Figure 1 provides a simplified logic model for the expansion programme, setting out the framework for the evaluation questions.

2.3.3. As underlined in Section 1.3, the logic model outlines that, to achieve its high-level outcomes, the ELC expansion programme needs to realise intermediate outcomes of maximising take up of expanded ELC, especially among families who will benefit most. Further, to achieve its outcomes, expanded ELC needs to be high quality, sufficiently flexible and accessible and to increase access to affordable ELC. Finally, expanded ELC services need to support parental confidence and capacity (e.g. to engage with their child's learning, enhance the home learning environment and promote their own wellbeing and confidence). These intermediate outcomes rely on appropriate funding and governance being provided at national and local authority level (inputs), which has allowed sufficient capacity – e.g. staff, buildings – to be created across Scotland to provide 1140 hours (outputs).

2.3.4. The aims of the evaluation are, therefore, to:

a) Understand changes in the ELC sector following the expansion, including any unintended consequences:

• changes in the ELC workforce and capacity to support high quality ELC
• changes in the composition and capacity of the ELC sector

b) Evaluate the intermediate outcomes of the ELC expansion:

• delivering high quality, flexible, accessible and affordable ELC for all three and four year olds and eligible two year olds
• maximising take up of expanded ELC amongst eligible two year olds and other groups less likely to access their entitlement
• supporting parental confidence and capacity

c) Assess the contribution of the policy change to outcomes for children, parents and families:

• improving children's development, particularly among those from the most disadvantaged backgrounds, and narrowing the poverty-related outcomes gap
• increasing parents' opportunities to take-up or sustain employment, training, or study, in particular parents in disadvantaged circumstances
• improving family wellbeing, particularly for families in disadvantaged circumstances

d) Assess the economic costs and benefits of the expansion

2.3.5. The impacts on different groups of children and parents will be considered in relation to these aims e.g. differences in outcomes in relation to levels of area deprivation, urban or rural areas, parents in different income groups, children with different ages, children with additional support needs (ASN).

2.3.6. The evaluation approach set out here is primarily an outcomes evaluation. It does not consider the efficacy of the ELC expansion against other policy options or the counterfactual of "do nothing". As recognised above, ELC is only one contributing factor to the desired high-level outcomes, and it will not be possible to say the expansion has 'caused' specific high-level outcomes. The use of a before and after evaluation study design with different cohorts of children (see Section 5 for more detail on the evaluation methodology), alongside a theory of change where links between the expansion and the intermediate outcomes can more easily be made, will, however, allow an assessment of whether and how expanded ELC contributes to improved outcomes.

2.3.7. An evaluation of the pilot implementation of automatic entitlement to funded ELC for children born in August to December who defer their school start is being undertaken separately. It focuses on 1) providing information about implementation of the pilots in order to inform the full roll-out of the policy, including implications for advertising and communications, admissions processes, overall capacity and parental choice in the pilot local authorities; and 2) exploring the impact of the pilots on parental awareness and perceptions of the entitlement.

2.3.8. It is also important to note from the outset that isolating the specific contribution and 'added value' of the ELC expansion on outcomes for children and families within a complex and fluid policy context is difficult. The development of new childcare policies as well as other Scottish Government policies, strategies and funding streams are expected to contribute to outcomes, most notably: the Scottish Attainment Challenge, Pupil Equity Funding and the expansion of Free School Meal provision; the Promise; the Whole Family Wellbeing Fund; the Fairer Scotland Action Plan and Child Poverty Delivery Plans; Fair Work, including Fair Start Scotland and the Parental Employability Support Fund; Social Security, including Best Start Grants, Best Start Foods and the Scottish Child Payment; and the Inclusive Growth agenda.

2.3.9. UK-wide policies will also impact on outcomes for children and families in Scotland, including the ongoing impacts of UK Government welfare reform and EU exit. Further, the wider economic outlook is uncertain, with significant increases in the cost of living and ongoing impacts of the Covid-19 pandemic being felt by the sector and families, with many of these impact being felt most acutely by the most disadvantaged (discussed further in the next section).

2.4. Considering the impact of the Covid-19 pandemic on the evaluation outcomes and data collection

The childcare sector

2.4.1. The ELC sector in Scotland has experienced an extremely challenging period since the onset of the Covid-19 pandemic. This has included closure periods, new and regularly updated public health guidance that has had an impact on the operation of some services, and the delay to the statutory roll-out of 1140 hours. Staff have experienced significant challenges in their personal and professional lives, and have supported some children and families through traumatic experiences. Feedback from providers has suggested that changes to parental employment and working patterns have led to reductions in demand that have made it challenging for some services to continue, particularly where public funding accounts for a minority of income or the service does not deliver funded ELC hours.^[25]

2.4.2. The Scottish Government has worked closely with the sector, including through the ELC Covid Reference Group and the Childcare Sector Recovery Group (which later merged with the Childcare Sector Working Group), to understand the ongoing impacts that the pandemic has had on settings and staff. These Groups played a critical role in shaping the development of the Covid guidance for the sector, the overall approach to engagement and communications with the workforce and the financial support made available to providers during the pandemic. Discussions on the impacts of Covid, with a particular focus on case numbers in settings and overall trends, have taken place with public health experts through the sector "Covid Impacts Group".

2.4.3. A survey of private and third sector daycare of children services and childminding services was undertaken in June 2020 to understand the potential impacts of the reopening guidance on settings – in particular the expected changes in staffing, costs, capacity and private income generation.^[26] Other evidence drawn on to help understand the impact of the pandemic on childcare providers and staff includes reporting by the Care Inspectorate on the number of settings open/closed and trends in overall numbers of registered services as well as surveys undertaken by representative bodies of their members (including qualitative information provided at meetings of the Childcare Sector Recovery Group).

2.4.4. A Financial Sustainability Health Check of the sector was published in August 2021. A key source of evidence was a detailed survey of private and third sector daycare of children services and childminders undertaken during April and May 2021 to provide a comprehensive assessment of the current financial sustainability of the sector, including capturing the impacts of the pandemic on services.^[27] The report set out how Scottish Government will work with the sector and delivery partners to progress a series of actions to enable recovery and to support the long-term sustainability of the sector.

Young children and their families

2.4.5. The pandemic has had a profound impact on the lives of families in Scotland.^[28] While the long term effects of the pandemic on families and the economy are still not known, the growing evidence from a range of research studies suggests many young children and their families have been negatively affected, especially those who were already disadvantaged.^[29] For some families with young children the pandemic has had an impact on parental employment and household income. Increases in stress and reductions in parental wellbeing have been found, especially for mothers. Since the start of the pandemic a range of different support and funds have been provided by Scottish Government to families, with many targeted at low income families.^[30]

2.4.6. During 2020 and 2021, protective public health measures related to Covid-19 curtailed many interactions young children would regularly have in ELC settings and wider social contexts that provide learning experiences. Studies have suggested negative short to medium term impacts on children's sleep, mood, behaviour and development – especially social and emotional, and speech and language development – among a large minority of young children.^[31] There is also evidence, however, to suggest some beneficial impacts for some children and families, for example by creating opportunities for other meaningful interactions at home with family.^[32] Some more recent studies have suggested that wellbeing improved with the reopening of ELC settings and the gradual removal of protective measures during 2021. We would expect that ELC can play an important role in mitigating some of the negative impacts of the pandemic for children and families. However, it is likely that there will be medium and long-term impacts of the public health measures on the outcomes the ELC expansion seeks to influence.

2.4.7. Recent data from PHS, based on Child Health Reviews from the Child Health Surveillance Programme, indicates that there has been a rise in developmental concerns noted at Child Health Reviews at the 13-15 month and 27-30 month points.^[33] Similar trends have been found in early primary school. The most recent Achievement of Curriculum for Excellence (CfE) Level statistics report shows reductions in the proportions of primary school pupils achieving the expected CfE levels in literacy and numeracy over the Covid-19 pandemic period.^[34] The percentage of pupils achieving the expected CfE level in 2020/21 is lower than in 2018/19 for all stages and all organisers, including for the Early Level in the first year of primary school.

Implications for the evaluation

2.4.8. The likely ongoing impacts for children and families of the pandemic and mitigating public health measures have implications both for what the expansion of funded ELC can be expected to achieve – for example, if a higher proportion of children are experiencing developmental difficulties when they begin funded ELC due to the pandemic – and for the conclusions this evaluation will be able to draw. As underlined previously, isolating the contribution of the ELC expansion to outcomes for children and families within a complex policy context is difficult, and particularly so during a period of societal and economic disruption.

2.4.9. We will continue to review data and evidence to assess the ongoing impact of the pandemic on children and families, and on staff and for the sector. Relevant indicators from the National Performance Framework and Child Poverty Measurement Framework will also be used to help track the ongoing implications for child wellbeing and development and parental employment. Key indicators include:

• the percentage of children with a concern at their 27-30 month review,^[35] which will allow us to track changes in social and physical development of children aged between two and three years (before starting or at the beginning of funded ELC)
• child poverty and material deprivation^[36]
• labour market outcomes for households with children, and particularly low income households with children, such as unemployment and underemployment, and average hourly earnings and hours worked^[37]

2.4.10. We will use this and other data to help us contextualise and interpret the findings of the evaluation. We will also consider whether any further contextual data is required on how the pandemic has affected children and families involved in the evaluation.

2.4.11. Covid-19 protective measures have also had a significant impact on what is possible in terms of research methodology and data collection, with both practical and ethical consequences. Face-to-face data collection methods were suspended in March 2020 and only began being phased in for Scottish Government research during spring 2022. Other considerations have included the appropriateness of contacting participants at a specific point in time and how developments as the pandemic unfolded might influence research findings. This has had implications for some proposals originally planned as part of this strategy, including in terms of amendments to proposed methodology and timings; and data availability, comparability and quality. These are set out in more detail in the following sections. We will keep these considerations under review as the effects of Covid-19 continue to be felt.