Scottish Study of Early Learning and Childcare: final report

Appendix A – Methodology

Data collection

Timetable

Data were collected in six phases, between October 2018 and June 2024. Table A1 records the timing and focus of each phase.

Table A1: Data collection timetable

Phase	Timing	Cohort	Cohort description
Phase 1	October – December 2018	Eligible 2s at age two	Children aged between two years and two years six months receiving up to 600 hours per year of funded ELC
Phase 2	May – June 2019	ELC Leavers	Children aged between four years three months and five years six months eligible to start school in August 2019 receiving up to 600 hours per year of funded ELC
Phase 3	October – December 2019	Eligible 2s at age three	Children who participated at Phase 1, now aged between three years and three years six months, followed up after one year of up to 600 hours of funded ELC
Phase 3	October – December 2019	Comparator 3s	Children aged between three years and three years six months receiving up to 600 hours per year of funded ELC
Phase 4	October – December 2023	Eligible 2s at age two	Children aged between two years and two years six months receiving up to 1140 hours per year of funded ELC
Phase 5	May – June 2024	ELC Leavers	Children aged between four years three months and five years six months eligible to start school in August 2024 receiving up to 1140 hours per year of funded ELC
Phase 6	October – December 2024	Eligible 2s at age three	Children who participated at Phase 4, now aged between three years and three years six months, followed up after one year of up to 1140 hours of funded ELC
Phase 6	October – December 2024	Comparator 3s	Children aged between three years and three years six months receiving up to 1140 hours per year of funded ELC

Data collection activities

A number of different data collection activities were completed at each phase. These included:

• age-specific paper questionnaires completed by keyworkers
• paper questionnaires completed by one of the child’s parents/carers
• an online questionnaire for setting managers, and
• observations conducted by professionals seconded from the Care Inspectorate using either the Infant/Toddler Environment Rating Scale (ITERS-3) or the Early Childhood Environment Rating Scale (ECERS-3).

A summary of what was carried out at each phase is noted in Table A2.

Table A2: Data collection activities at each phase

Activity	Phase 1	Phase 2	Phase 3	Phase 4	Phase 5	Phase 6
Keyworker questionnaire	X	X	X	X	X	X
24 month	X	-	-	X	-	-
27 month	X	-	-	X	-	-
30 month	X	-	-	X	-	-
36 month	-	-	X	-	-	X
42 month	-	-	X	-	-	X
54 month	-	X	-	-	X	-
Parent questionnaire	X	X	X	X	X	X
Setting heads questionnaire	-	-	-	X	X	-
ITERS-3 setting observations	X	-	-	X	-	-
ECERS-3 setting observations	-	X	-	-	X	-

At each phase, settings included in the sample were invited to an information session, which ran through instructions for carrying out the surveys. They were also provided with a pack of written instructions, and materials to use to administer the surveys. These included information letters for parents, opt-out forms, instructions on how to select children for participation, and questionnaires. Participation by both the ELC setting and the parent were voluntary, and settings were asked to allow time for parents to decide whether they were happy for their child to be included before the keyworker questionnaire was completed.

Settings were asked to administer keyworker questionnaires specific to the age of the child, as these included measures of developmental milestones that would be misinterpreted if the wrong questionnaire was used.

Both the keyworker and the parent questionnaires were intended to be as similar as possible pre- and post-expansion, to allow for a direct comparison between those receiving up to 600 hours of funded ELC pre-expansion and 1140 hours post-expansion. However, a number of new questions were added to the parent questionnaire, to collect additional information post-expansion, and a small number of questions were amended to improve them or to update them in line with Scottish Government guidance on harmonised survey questions.^[56]

The setting managers’ questionnaire, which invited the setting head or manager to reflect on the effects of the expansion of funded ELC on their setting, was only included post-expansion, as it was not relevant before then. It was included at Phase 4 and Phase 5, but not Phase 6, as most of the settings at Phase 6 had already had an opportunity to participate and there was little or no benefit in them repeating the questionnaire. Settings which had not previously been involved in the SSELC but were involved at Phase 6 were ones to which Eligible 2s had moved (see below on ‘sampling’). Responses from the settings where the child had spent their year of funded ELC for eligible two-year-olds were considered more relevant to the study, so these new settings were not issued with the setting managers’ questionnaire.

The observations using either ITERS-3 (for settings catering for children under the age of three) or ECERS-3 (for settings catering for children aged three to five) were intended to provide an observational measure of the quality of ELC provided to children by the setting. They were included for a number of reasons: they centre on the experience of the child in the setting; allow for the effect of setting quality on child outcomes to be controlled for; and are relatively easy to administer given that only one three-hour observation is required. Further details on ITERS-3 and ECERS-3 are provided below under ‘Data collection instruments and derived variables’.

Further details of the data collection activities at each phase can be found in the SSELC individual phase reports, on the Early Learning and Childcare Expansion Evaluation part of the Scottish Government website.

Questionnaire contents

The contents of the keyworker, parent and setting heads questionnaires are set out in Tables A3, A4 and A5.

Table A3: Keyworker questionnaire contents

Questionnaire items	Phase 1	Phase 2	Phase 3	Phase 4	Phase 5	Phase 6
Child’s sex	No	Yes	Yes	Yes	Yes	Yes
Date of Birth	Yes	Yes	Yes	Yes	Yes	Yes
Postcode	Yes	Yes	Yes	Yes	Yes	Yes
Total hours registered to attend per week*	Yes	Yes	Yes	Yes	Yes	Yes
Funded hours per week*	Yes	Yes	Yes	Yes	Yes	Yes
Usual hours attended per week*	Yes	Yes	Yes	Yes	Yes	Yes
Month / year started	No	No	No	Yes	Yes	Yes
Long-term health condition	No	Yes	Yes	Yes	Yes	Yes
Additional support needs	No	No	No	Yes	Yes	Yes
Ages and Stages Questionnaire (ASQ) 24 month	Yes	N/A	N/A	Yes	N/A	N/A
ASQ 27 month	Yes	N/A	N/A	Yes	N/A	N/A
ASQ 30 month	Yes	N/A	N/A	Yes	N/A	N/A
ASQ 36 month	N/A	N/A	Yes	N/A	N/A	Yes
ASQ 42 month	N/A	N/A	Yes	N/A	N/A	Yes
ASQ 54 month	N/A	Yes	N/A	N/A	Yes	N/A
ASQ 60 month	N/A	Yes	N/A	N/A	Yes	N/A
Strengths and Difficulties Questionnaire (SDQ) Age 2-4	Yes	N/A	Yes	Yes	N/A	Yes
SDQ Age 4-15	N/A	Yes	N/A	N/A	Yes	N/A

* Questions on hours registered, hours attended and funded hours were not asked consistently across the phases and were not well answered at the earlier phases.

Table A4: Parent questionnaire contents

Questionnaire items	Phase 1	Phase 2	Phase 3	Phase 4	Phase 5	Phase 6
Child’s sex / date of birth / postcode	Yes	Yes	Yes	Yes	Yes	Yes
Hours and days at ELC setting	No	No	No	Yes	Yes	Yes
Full costs met by government	Yes	Yes	Yes	Yes	Yes	Yes
Hours paid for at ELC setting*	Yes	Yes	Yes	Yes	Yes	Yes
Month / year started	No	No	No	Yes	Yes	Yes
Travel time to ELC setting	Yes	Yes	Yes	Yes	Yes	Yes
Parental participation in activities at ELC setting*	Yes	Yes	Yes	Yes	Yes	Yes
Meals / snacks*	No	No	No	Yes	Yes	Yes
Other ELC / childcare (type, funding, hours, days, start date)*	Yes	Yes	Yes	Yes	Yes	Yes
Childcare before age of 2 / 3*	No	Yes	Yes	Yes	Yes	Yes
Feelings around support with childcare	Yes	Yes	Yes	Yes	Yes	Yes
Advantages / disadvantages of having child in ELC*	Yes	Yes	Yes	Yes	Yes	Yes
Respondent economic activity / work hours	Yes	Yes	Yes	Yes	Yes	Yes
Partner economic activity / work hours	No	No	No	Yes	Yes	Yes
Changes in work / education	No	No	No	Yes	Yes	Yes
Working pattern / location	No	No	No	Yes	Yes	Yes
Would work more if could afford good childcare	Yes	Yes	Yes	No	No	No
Lack of childcare reason for not working	Yes	Yes	Yes	Yes	Yes	Yes
Things done by respondent because child in ELC	Yes	Yes	Yes	Yes	Yes	Yes
Household relationships*	Yes	Yes	Yes	Yes	Yes	Yes
Whether main carer	No	No	No	Yes	Yes	Yes
Ethnicity*	Yes	Yes	Yes	Yes	Yes	Yes
Language spoken at home	Yes	Yes	Yes	Yes	Yes	Yes
Respondent educational qualifications*	Yes	Yes	Yes	Yes	Yes	Yes
Partner educational qualifications	No	No	No	Yes	Yes	Yes
Household income*	Yes	Yes	Yes	Yes	Yes	Yes
Child’s health in general	Yes	Yes	Yes	Yes	Yes	Yes
Concerns about child’s development	Yes	Yes	Yes	Yes	Yes	Yes
Support for speech and language development	No	No	No	Yes	Yes	Yes
Child’s longstanding health condition*	Yes	Yes	Yes	Yes	Yes	Yes
Child’s sleep	Yes	Yes	Yes	Yes	Yes	Yes
Whether child ever breastfed	Yes	Yes	Yes	Yes	Yes	Yes
Home learning environment	Yes	Yes	Yes	Yes	Yes	Yes
Parent-child warmth scale	Yes	Yes	Yes	Yes	Yes	Yes
Parent health in general	Yes	Yes	Yes	Yes	Yes	Yes
Respondent longstanding health condition*	Yes	Yes	Yes	Yes	Yes	Yes
Other household member health condition	Yes	Yes	Yes	Yes	Yes	Yes
Short Warwick-Edinburgh Mental Wellbeing Scale	Yes	Yes	Yes	Yes	Yes	Yes
Life satisfaction	Yes	Yes	Yes	Yes	Yes	Yes
Confusion, Hubbub and Order Scale	No	Yes	Yes	Yes	Yes	Yes
Coping as a parent	Yes	Yes	Yes	Yes	Yes	Yes

* Questions marked with an asterisk were not asked in the same way at all phases.

Table A5: Setting managers’ questionnaire contents

Questionnaire items	Phase 4	Phase 5
Support offered to families of eligible two-year-olds	Yes	N/A
Support offered to families of four- / five-year-olds	N/A	Yes
Support offered to families of all children	Yes	Yes
Effect of ELC expansion on support offered to families	Yes	Yes
Effect of cost-of-living crisis on support offered to families	Yes	Yes
Food provided to children receiving funded provision	Yes	Yes
Consultation with parents on food provision	Yes	Yes
Ease / challenges of providing food	Yes	Yes
Effect of expansion on quality of food provision	Yes	Yes
Effect of costs crisis on quality of food provision	Yes	Yes
Proportion of families taking up free meals / reasons not taken	No	Yes
Adaptations to meet requirements of expansion	Yes	Yes
Challenges in meeting requirements of expansion	Yes	Yes

Data collection instruments and derived variables

Ages and Stages Questionnaire (ASQ)

The Ages and Stages Questionnaire was administered as part of the keyworker observations.

Both the ASQ and SDQ questionnaires are widely used by Health Visitors across Scotland as part of their health reviews of pre-school children – the Child Health Programme.^[57] These questionnaires were also selected for inclusion in the Child Health Programme following an extensive review by academics and practitioners.^[58]

The ASQ is a structured assessment of a range of developmental domains to identify children at increased risk of developmental difficulties.^[59] The instrument includes 30 items split into five different domains: communication, gross motor, fine motor, problem solving and personal-social. The respondent indicates whether or not the child can complete the action or provide the response required by answering ‘yes’, ‘sometimes’ or ‘no’. Each domain produces a summary score which can be used to indicate whether the child's development is on schedule, needs monitoring or requires further assessment. Whilst it is designed to be completed by parents, it was deemed suitable for completion by the child's keyworker at their ELC setting because it is informed by observation of the child.

Strengths and Difficulties Questionnaire (SDQ)

The Strengths and Difficulties Questionnaire was administered as part of the keyworker observations.

The SDQ is a commonly used behavioural screening questionnaire designed for use with children aged between four and 16.^[60] The questionnaire includes 25 questions about a child's behaviour to which the respondent can answer ‘not true’, ‘somewhat true’ or ‘certainly true’. Responses can be combined to form five different measures of the child's development, namely emotional symptoms (e.g. excessive worrying), conduct problems (e.g. often fighting with other children), hyperactivity/inattention (for example, constantly fidgeting), peer relationship problems (e.g. not having close friends), and prosocial behaviour (e.g. being kind to others). In addition, the first four measures can be combined into a ‘total difficulties’ scale. Higher scores imply greater evidence of difficulties on each of the scales, with the exception of the prosocial behaviour scale where the reverse is true. In this report, recommended banded versions of the scales have been used to create the following categories: ‘close to average’, ‘slightly raised’, ‘high’ and ‘very high’, with ‘very high’ indicating multiple difficulties identified.^[61]

Short-form Warwick-Edinburgh Mental Wellbeing Scale (SWEMWBS)

The short-form of the Warwick-Edinburgh Mental Wellbeing Scale is a commonly used series of questions intended to measure mental wellbeing. Respondents are presented with seven statements about positive feelings and thoughts, and asked how frequently they have experienced them over the previous two weeks.

The scale is created by summing the frequency of responses for all seven items, with a score of 1 for ‘none of the time’ and a score of 5 for ‘all of the time’. This provides a score of between 7 and 35. In order to make the data comparable with other surveys, the scores have to be ‘Rasch-transformed’.

Cut offs to determine ‘high’ and ‘low’ wellbeing can be identified from population norms (taken from the 2011 Health Survey for England). High wellbeing is defined as the top 15% of scores, in the range from 27.5 to 35.0, and low wellbeing as the bottom 15%, in the range from 7.0 to 19.5.

Details of the scoring are available from the WEMWBS website.

Home learning environment scale

Four items were used to create the home learning environment scale. These each related to the number of days on which certain activities between the child and parent were conducted. The scale was created by summing the number of days on which each of the four activities had occurred in the last week.

Cut offs to determine quartiles for the Eligible 2s and Comparator 3s were created using data from Phase 1 of the study. This allows for comparison between phases. The bottom quartile included scores in the range 0 to 15, the second in the range 16 to 20, the third in the range 21 to 24, and the top in the range 25 to 28

The distribution of responses was slightly different at Phase 2, so alternative cut offs were created for the ELC Leavers. The bottom quartile included scores in the range 0 to 16, the second in the range 17 to 21, the third in the range 22 to 24, and the top in the range 25 to 28.

Parent-child warmth scale

The parent-child warmth scale used in the SSELC comprises the seven items that form the warmth dimension of the short form of the Mothers’ Object Relations Scale (MORS-SF). Details of the scale are available from the MORS website.

Each item was scored from 1 to 5, with 1 being ‘None of the time’ and 5 ‘All of the time’. Summing the seven items created a scale from 7 to 35. Three groups were created of roughly equal size, based on Phase 1 responses, to allow for comparison between the phases. Because of the skewed nature of the data, the bottom tertile (a score of 7 to 31) should not be interpreted as ‘low warmth’, as many of the respondents in this group would have answered ‘often’ or ‘all of the time’ to each of the statements. The middle tertile was for a score of 32 to 34 and the top tertile a score of 35.

Confusion, Hubbub and Order Scale

The ‘Confusion, Hubbub and Order Scale’ used in SSELC comprises four items taken from the original scale developed for the Louisville Twin Study in the 1980s.^[62] It has since been validated and used widely in family studies. The four items used in SSELC have been used to form a scale in other Scottish studies, including Growing Up in Scotland.^[63] They are intended to measure calmness and order within the household. Respondents were asked how much they agreed or disagreed with each of the statements:

It’s really disorganised in our home

You can’t really hear yourself think in our home

The atmosphere in our home is calm

First thing in the day, we have a regular routine at home

To form the scale, the first two of the above statements were scored 1 to 5, with 1 being ‘strongly disagree’ and 5 being ‘strongly agree’; the other two items were scored 1 to 5 in the reverse order, with 1 being ‘strongly agree’ and 5 ‘strongly disagree’. The sum of the four items created a scale from 4 to 20, with lower values indicating calmer households and higher values a greater degree of disorganisation / lack of routine. These questions were not asked at Phase 1, so tertiles were based on Phase 2 of the study, with the lowest tertile being a score of 4 to 6, the middle tertile, 7 or 8, and the highest tertile 9 to 20. The distribution of responses was similar for Phase 3, hence the same cut offs were used for Eligible 2s and Comparator 3s as well as for ELC Leavers.

Long-term health conditions / disability

Information about long-term health conditions was collected in both the parent and keyworker questionnaires.

From Phases 2 to 6, the keyworker questionnaire asked:

• “Does this child have any long-term health conditions that may affect their development?” (Yes/No)

Pre-expansion the keyworker was encouraged to complete the whole questionnaire, whereas post-expansion it was suggested that administrative staff might be able to complete some of the information. It is possible that this change has led to a lack of comparability across the phases.

At all phases, the parent questionnaire asked:

• ‘Does your child have any physical or mental health conditions or illnesses lasting, or expected to last for 12 months or more?’ (Yes/No); and
• ‘Does this condition / do these conditions or illness(es) have an effect on your child at play or on any other normal activity for a child her / his age?’ (Yes, a lot/Yes, a little/No)

Similar questions were asked of the respondent themselves, and from Phases 4 to 6, of the respondent’s partner.

The questions asked in the parent questionnaire are as recommended in Scottish Government guidance on data collection on disability.^[64] Having a ‘disability’ is defined as answering ‘Yes’ to the first question and ‘Yes, a lot’ or ‘Yes, a little’ to the second.

The question from the keyworker questionnaire was used when looking at ASQ and SDQ outcomes for the ELC Leavers. The questions from the parent questionnaire were used in all the regression analyses, as well as when looking at parent outcomes.

Equivalised household income

The parent questionnaires asked about the total income of the household from all sources before tax, including all earnings, benefits, tax credits and interest from savings. For comparability purposes, it is common practice to convert this to an ‘equivalised household income’, as it is recognised that a household comprising one adult and one child needs less income to live on than a household comprising two adults and two children.

Equivalised household income adjusts household income according to the typical income requirements for the number of people in the household. The modified OECD adjustment has been used in this case, whereby household income is divided by a household size factor, which is the sum of 0.67 for the first adult in the household, 0.33 for each subsequent adult or child aged 14 or above, and 0.20 for each child aged 13 or below.

Cut points for the equivalised income deciles have been taken from a national survey of people in households in Scotland, the Scottish Health Survey. This survey was chosen as it is the best source of health and health behaviour data in Scotland, and allows for comparison of findings from SSELC with those from the whole population, to set the SSELC findings in context. Different years of the Health Survey were used for each phase of SSELC: 2017 for Phase 1, 2018 for Phases 2 and 3, 2022 for Phase, and 2023 for Phases 5 and 6. These were the latest available Health Survey data at the time of preparing SSELC data for each of the individual phase reports.

For Phases 5 and 6, the lowest equivalised income decile includes, for example, families of one adult and one child under 14 with an income of below £11,745, and families of two adults and two children under 14 with an income of below £18,900 per year. More detail is provided in the individual phase reports.

Infant / Toddler Environment Rating Scale (ITERS-3)

To gather information on the characteristics of ELC settings, inspectors from the Care Inspectorate (acting as observers independent of their regulatory roles) conducted observations of around 150 settings at both Phase 1 (pre-expansion) and Phase 4 (post-expansion) using the most recent version of the Infant / Toddler Environment Rating Scale (ITERS-3). The ITERS-3 was developed in the United States by the Environment Rating Scale Institute along with the Early Childhood Environment Rating Scale (ECERS). Both scales are widely used in English speaking countries. ITERS-3 is designed for use in settings where most children are under 36 months and as such, it was deemed suitable for use with the eligible two-year-olds involved in Phase 1 and Phase 4 of the SSELC. Although many settings observed did not have a specific two-year-old room, using ITERS-3 allowed for age-appropriate criteria to be observed.

It is important to note that these tools are not the only method of assessing setting quality in Scotland. Indeed, the Care Inspectorate ratings provide a broader measure of the quality of practice and policy within settings that have also been found to be related to children's outcomes in Scotland.^[65]

As with the Care Inspectorate inspection methodology, the setting observations focused on outcomes. However, the methodology differed in that the ITERS-3 tool was used to observe for three hours, with no consultation with setting staff and no professional dialogue or explicit feedback provided. This was because the observations were intended to be a snapshot to inform the study and control for variation in child outcome data, rather than serving as an assessment of an individual setting's quality. During the ITERS-3 observations, observers looked at the six domains specifically for two-year-olds. In contrast, during a formal inspection, Care Inspectorate inspectors consider a range of areas that impact on experiences for all children attending the setting, not just those in specific age groups. The key areas covered during a formal inspection are likely to include some or all of the domain areas but can also cover other aspects of the provision to evaluate the overall quality of the setting.

The ITERS-3 scale comprises 33 items across six different subscales: space and furnishings; personal care routines; language and books; activities; interaction, and programme structure.

• Space and furnishings include observation of indoor space; furnishings for care, play, and learning; room arrangement; and display for children.
• Personal care routines include observation of meals and snacks; toileting; health practices; and safety practices.
• Language and books include observation of talking with children; encouraging vocabulary development; responding to children’s communication; encouraging children to communicate; staff use of books with children; and encouraging children’s use of books.
• Activities includes observation of fine motor; art; music and movement; blocks; dramatic play; nature and science; maths and number; appropriate use of technology; promoting acceptance of diversity; and gross motor.
• Interaction includes observation of supervision of gross motor play; supervision of non-gross motor play and learning; peer interaction; staff-child interaction; providing physical warmth and touch; and guiding children’s behaviour.
• Programme structure includes observation of schedule and transitions; free play; and group play activities.

In line with ITERS-3 guidance, each subscale is scored from 1 to 7. These scores are calculated by averaging the score for each item within the subscale. Each of the 33 items are also scored from 1 to 7. These scores are calculated using the indicators contained within each individual item. Indicators are grouped under scores of 1 (inadequate), 3 (minimal), 5 (good), and 7 (excellent), with each indicator providing an example of what should be observed relevant to each score. Indicators themselves are scored as yes or no depending on whether the indicator has been observed. In some cases, observers are able to record indicators or items as not applicable; these are then excluded when calculating item or subscale scores. A score of 1 is given if any indicator grouped under 1 is scored yes. For an item to score a 7, each indicator grouped under 7 must be scored yes.

Early Childhood Environment Rating Scale (ECERS)

Observations using the Early Childhood Environment Rating Scale (ECERS-3) were carried out at both Phase 2 (pre-expansion) and Phase 5 (post-expansion) of the SSELC in the same manner as for ITERS (see above). ECERS is designed for use in settings where most children are aged between three and five and as such, it was deemed suitable for use with the four- and five-year-olds involved in Phases 2 and 5 of the SSELC.

The ECERS scale comprises 35 items across six different subscales: space and furnishings; personal care routines; language and literacy; learning activities; interaction, and programme structure.

• Space and furnishings includes observation of: indoor space; furnishings for care, play, and learning; room arrangement; space for play; space for privacy; display for children; and play equipment.
• Personal care routines includes observation of: meals and snacks; toileting; health practices; and safety practices.
• Language and literacy includes observation of: encouraging children to expand vocabulary and use language; encouraging children to communicate; staff use of books with children; and encouraging children's use of books and familiarity with print.
• Learning activities includes observation of: fine motor; art; music and movement; blocks; dramatic play; nature and science; maths materials, understanding of written numbers and the use of maths in daily events; appropriate use of technology; and promoting acceptance of diversity.
• Interaction includes observation of: supervision of gross motor play; individualised teaching and learning; peer interaction; staff-child interaction; and discipline.
• Programme structure includes observation of: transitions and waiting times; free play; and group play activities.

Scores were calculated for each subscale in the same way as for ITERS-3.

Sampling

The general principle for sampling children was the same across the six phases of data collection, although there were some differences in the way this was carried out.

A sample of children of the appropriate age for the phase (see section on data collection) receiving funded ELC was drawn via a two stage process. First, a stratified sample of ELC settings was drawn, using an extract of the ELC Census collection of settings providing funded ELC to children of the relevant age as a sampling frame. For larger settings, a second stage was involved. If there were more than 10 eligible children at the setting, a sample of 10 children was drawn by setting staff from those eligible. Where there were 10 or fewer eligible children all were invited to take part.

Selection of settings

A systematic sample of ELC settings was drawn, stratified by local authority. Settings within each local authority were ordered by SIMD quintile and setting size before selection. To give all eligible children an equal chance of being selected, all settings were also given equal selection probabilities, except for those more than 10 eligible children. These were given a proportionately higher probability of selection. Settings that were opted out by their local authority lead before the sampling were excluded.

Phase 1 and Phase 4

When Phase 1 of the SSELC took place, some local authorities chose not to take part, or were not eligible to take part because they had already rolled out the expansion of funded ELC to eligible two-year-olds. Within those authorities that did agree to participate, some settings were not eligible for the same reason. As a result, the number of settings across Scotland eligible and willing to participate was limited. Hence, rather than a random selection, all were invited to take part. Consequently, findings from Phase 1 were treated as a census of eligible two-year-olds receiving 600 hours of funded ELC in participating local authorities. While this does not allow us to properly generalise the findings to the whole of Scotland, the similarities between this group of children and families due to their eligibility at age two appear to outweigh any differences due to their uneven spread across the country. Findings can therefore be treated as broadly comparable to Phase 4, particularly where regression analysis has been used to control for differences in the samples.

Phase 2 and Phase 5

At Phases 2 and 5, settings in the most deprived areas were deliberately oversampled, so that half the children sampled attended settings in the 20% most deprived areas. This was done in order to facilitate analysis of the poverty-related outcomes gap, as it may be expected that children attending settings in deprived areas were more likely to live in deprived areas and to come from low-income households. In reality, while half of the achieved sample attended settings in deprived areas, the proportion living in such areas was only around 30%, and the proportion from households in the bottom equivalised income quintile (compared with Scottish population norms for all household types) was around 20%.

Phase 3 and Phase 6

There were two distinct elements to the Phase 3 and Phase 6 samples. First, there was the follow-up of Eligible 2s after a year of funded ELC. This was done primarily by contacting the settings they had attended at age two. At Phase 6, for those who could not be traced by this route, further methods of tracing were used, contacting local authority leads and parents, using contact details explicitly collected for this purpose.

The second element was the sample of Comparator 3s. Rather than using the latest available list of settings from the Care Inspectorate database, the samples from Phases 2 and 5 were used instead. This was done for a number of reasons. First, it alleviated the need for further settings observations, as we already had observations using the ECERS-3 tool for these settings. Second, the settings already had knowledge of the study from the previous phase, and so it was hoped they would be more likely to engage with it. Third, the sample was expected to be unbiased, as nearly all settings catering for ELC Leavers also catered for three-year-olds, and vice-versa. Oversampling in deprived areas was not a requirement of Phases 3 and 6, so a random sample of one quarter of the settings in deprived areas from the Phase 2 and Phase 5 samples was selected, along with all settings from non-deprived areas, with the exclusion of any which had explicitly opted out.

Achieved sample sizes

Table A6 shows achieved sample sizes for keyworker and parent questionnaires at each of the phases of data collection after the removal of any invalid questionnaires.

Table A6: Achieved sample sizes

Phase / cohort	No. children	No. keyworker questionnaires	No. parent questionnaires
Phase 1 Eligible 2s at age two	586	574	428
Phase 2 ELC Leavers	1910	1846	1382
Phase 3 Eligible 2s at age three	391	376 (372 with keyworker questionnaire at both Phases 1 and 3)	269 (228 with parent questionnaire at both Phases 1 and 3)
Phase 3 Comparator 3s	851	811	565
Phase 4 Eligible 2s at age two	500	486	341
Phase 5 ELC Leavers	2292	2177	1648
Phase 6 Eligible 2s at age three	289	285 (278 with keyworker questionnaire at both Phases 4 and 6)	164 (137 with parent questionnaire at both Phases 4 and 6)
Phase 6 Comparator 3s	898	851	516

Further details on sampling, including estimated response rates, can be found in the individual Phase reports.

Children’s ages at completion of questionnaires

The keyworker questionnaires recorded age-dependent developmental milestones. Table A7 shows the average ages of participating children for each of the questionnaires. Differences in average ages pre- and post-expansion are all within what may be expected given sampling variation.

Table A7: Average ages of children in each cohort / for each questionnaire

Cohort / Questionnaire	Baseline data	Post-expansion data
Eligible 2s at age two	28.2 months	28.4 months
24-month questionnaire	25.7 months	25.3 months
27-month questionnaire	27.6 months	27.5 months
30-month questionnaire	29.6 months	29.7 months
Eligible 2s at age three	39.7 months	40.8 months
36 month questionnaire	38.2 months	39.2 months
42 month questionnaire	40.6 months	41.3 months
Comparator 3s	40.1 months	40.5 months
36 month questionnaire	38.2 months	38.6 months
42 month questionnaire	40.9 months	41.3 months
ELC Leavers	58.4 months	57.9 months
54-month questionnaire	54.7 months	54.2 months
60-month questionnaire	60.3 months	60.6 months

Weighting

Weights are commonly applied to survey data so that the achieved sample better represents the population it was drawn from. Groups that are under-represented in the achieved sample are given higher weights than those that are over-represented, with the aim of weighted data matching the population distribution by key characteristics. Survey estimates produced using the weighted data should then be closer to estimates that would have been gained from the whole population of interest.

Separate weights were created at each Phase for responses to the keyworker questionnaire and for responses to the parent questionnaire. The same basic weighting approach was applied at all Phases, except for Phase 1. As the sample was not random at Phase 1, no weights were created. This was carried forward for the Eligible 2s at age three at Phase 3, who were the same children one year later.

The basic weighting approach consisted of two elements: selection weighting and non-response modelling. The first stage adjusted for differential probability of selection (for settings and children) resulting from the sample design. The second stage adjusted for differences in the profiles of sampled and responding settings, using logistic regression modelling. Calibration weighting, which adjusts the profile of the weights to match estimates of the population, could not be used due to the absence of detailed population estimates for each group of children.

When using data from both the keyworker and parent questionnaires, parent weights were used in the analysis. When using data for the Eligible 2s from both Phases 4 and 6, Phase 6 weights were used. At Phase 6, separate weights were created for the Eligible 2s at age three and the Comparator 3s.

Further details of the methods used to produce each set of weights are provided in the individual Phase reports.

Data analysis

Datasets

The data for use in this report come from all six phases of the SSELC. Pseudonymised datasets were created from the responses to the questionnaires to allow comparisons between children receiving 600 hours of funded ELC (baseline data) and those receiving 1140 hours (post-expansion data) as shown in Table A8.

Table A8: Datasets used for comparisons of pre-expansion and post-expansion data

Cohort data to be compared	Baseline data	Post-expansion data
Eligible 2s at age two	Phase 1	Phase 4
Eligible 2s at age three	Phase 3	Phase 6
Change for Eligible 2s following one year of funded ELC	Difference between Phase 1 and Phase 3	Difference between Phase 4 and Phase 6
Comparator 3s	Phase 3	Phase 6
Difference between Eligible 2s at age three and Comparator 3s	Phase 3	Phase 6
ELC Leavers	Phase 2	Phase 5

Missing data

There are several reasons why the data are not complete for all cases:

• Some parents did not complete a parent questionnaire, although a keyworker questionnaire was filled in for their child (or vice-versa).
• Some parents chose not to answer certain questions, such as household income.
• Some parents or keyworkers did not know the answer to certain questions.
• An error was made in the 27 month keyworker questionnaire that was originally sent out at Phase 1, resulting in two questions being missed from the ASQ problem solving domain. A second version was sent out, but keyworkers were not asked to provide answers to the missing questions if the questionnaire had already been completed. This error does not appear to have significantly affected the data. Instructions for scoring the ASQ allow for up to two missing answers on each domain, and results from the affected questionnaires were in line with those from the unaffected ones.
• Some questions were not asked at all phases.

There are multiple ways to handle missing data, depending on the amount of information available.

• Where the information can be identified from another questionnaire, that information has been used. This was the case for geographic data derived from the postcode, which was collected from both parents and keyworkers, and the sex of the child, and for weekly totals of hours attended at the ELC setting.
• Where an answer can be estimated from other answers, that has been done. This is the case, for example, in calculating ASQ scores. If the keyworker has left a question unanswered, then it will be assigned an average score of the other answers in that domain for that child, providing at least 4 of the 6 questions were answered, as per the ASQ-3 User Guide. This allows us to calculate domain scores without losing too many cases.

Data missing because a questionnaire was not completed have not been imputed. After the completion of the imputation as outlined above, the following conditions were applied to cases which still had missing data:

• Where data are missing for an outcome variable, the case has been excluded from the analysis of that outcome. For all the outcome variables included in the supplementary table, there does not appear to be more than 5% missing data at any phase.
• Where data are missing from a variable used as a cross-break, it appears in the ‘total’ column of the table, but not in any other column.
• Where data were missing from any of the variables feeding into the significance testing models for the ELC Leavers (see below on ‘significance testing’), they were imputed as outlined in Table A9. Figures for the Comparator 3s were very similar. Imputation was not required for the Eligible 2s. By imputing missing data in this way, the significance tests have been based on all cases shown in the tables.

Table A9: Imputation of variables in significance testing models (ELC Leavers)

Variable	Weight	Proportion of cases with missing data	Category into which imputed
Child sex	keyworker	0.2%	Modal (Boys)
Child long-term condition (keyworker data)	keyworker	2.9%	Modal (Does not have a long-term condition)
Single / Couple household	keyworker	31.4%	Separate category (effectively ‘No keyworker questionnaire’)
Area deprivation (home address)	keyworker	3.2%	Area deprivation of setting address
Respondent sex	parent	0.7%	Modal (Female)
Respondent age	parent	2.3%	Median (35 to 39)
Respondent limiting long-term condition	parent	2.0%	Modal (Does not have a long-term condition)
Single / Couple household	parent	3.1%	Modal (Couple)
Area deprivation (home address)	parent	1.1%	Area deprivation of setting address
Equivalised household income	parent	9.9%	Separate category

For the regression modelling of key outcomes (see below), missing data on predictor variables were imputed in a similar manner, into the median, modal, or a separate category, as appropriate, to ensure all cases with outcome data and parent questionnaire data could be included in the models.

Analysis

All analysis was carried out using the complex samples package in SPSS v29. This means that the clustering of children within settings was taken into account when determining significance, without the need to run multi-level models.

With the exception of the Eligible 2s pre-expansion (Phases 1 and 3), and the Eligible 2s regression models, all analysis was conducted using weighted data (see section on (‘Weighting’). The use of weighted data ensures the analysis is generalisable to children receiving funded ELC across Scotland. While this cannot be done for the Eligible 2s, the longitudinal nature of the sample means that we can still examine the effects of a year of funded ELC on this particular group of children.

Significance testing

Testing for significant differences between phases for a categorical outcome variable

Significance tests for categorical variables were carried out using a t-test within a complex-samples logistic-regression model. This has three particular advantages over other tests. First it allows us to take into account the clustering and stratification of the sample. Next, it requires the outcome variable to be binary, which forces the selection of a category of interest. Thus, we can compare, for example, the proportion of children on schedule for the ASQ communication domain, pre- and post-expansion. A chi-square or similar test, which considers all cells in the table, may or may not identify a significant difference between phases, due to differences in categories of lesser interest. Finally, the use of logistic regression allows us to control for other variables.

As a first step, the model for the outcome variable included just one factor, the phase of the data collection. Where this was significant, a second step was carried out, introducing additional factors, to determine whether the phase of data collection remained significantly associated with the outcome, when controlling for these other factors.

The factors to be included in the models for significance testing were determined in three ways: they had a degree of policy relevance and were known to be associated with child developmental outcomes / parental outcomes; there were complete data for most cases; and they had been highlighted in previous analysis of the SSELC data.

For child outcomes, which required complete data for the keyworker questionnaire, the agreed set of factors was child sex, presence of a long-term condition, single-parent / two-parent households and Scottish Index of Multiple Deprivation (SIMD), along with phase. Too many factors, particularly if they were not entirely independent of each other, would confuse the models. This would occur if more than one variable were taken from the parent questionnaire. A significant proportion of keyworker questionnaires had no associated parent questionnaire, so variables taken from the parent questionnaire could not be considered independent of each other as they would have missing data for the same 30% of cases (see section on ‘missing data’ for how these have been treated).

All adult outcomes come from the parent questionnaire, so there was not the same consideration around combining data from the two questionnaires. The requirement for factors to be independent of each other remained. The agreed list of factors used in the models to test significance for adult outcomes was sex of respondent, age of respondent, presence of long-term condition in respondent, single or couple household, equivalised income and SIMD, plus phase.

Testing for significant differences between phases for a mean value

Testing for significant differences between phases for a mean value was done as above, but using linear regression instead of logistic regression. In such cases, the outcome variable was not reduced to a binary, but treated as continuous. Only a small number of variables are presented as means in the supplementary tables: scores on the SDQ scales, parental wellbeing scale (SWEMWBS), parental life satisfaction, the parent-child warmth scale, and hours worked per week.

Testing for significant differences between phases for subgroups

For some outcomes, particularly for the ELC Leavers, break variables are included in the tables, with the intention of presenting change between the phases for different subgroups.

In these cases, the interest is in whether the level of change between phases for one category of the break variable is significantly different from the others e.g. boys and girls. To test this, models were constructed, similar to those described above using either logistic or linear regression, but with an additional term for the interaction between phase and sex.^[66] If this interaction term were significant in the model, then we could say that the change was different for boys compared with girls.

As a first step, the model for the outcome included terms for phase, the break variable specified, and the interaction between phase and the break variable. Similar to the method described above, a second step to add in the additional control variables was conducted if a significant difference for the interaction term was seen at the first step.

Testing the significance of the difference between phases of a change for the Eligible 2s after one year of ELC

To test whether the change for the Eligible 2s after a year of funded ELC was significantly different pre- and post-expansion, logistic or linear regression was again used. The models for outcome measures at age three included two factors: the lagged outcome (i.e. the same outcome measure but at age two) and the phase of data collection (Phase 3 or Phase 6 – i.e. pre- or post-expansion). No other factors were included in the model, as the measure at age two would be associated with factors such as sex, income and number of parents. Where phase was significant in the model, this means that the outcome was significantly different at each phase when controlling for the outcome at age two. In other words, the change after a year of ELC is greater (or less) post-expansion.

Modelling key outcomes

Eight key outcomes were identified in the introduction to this report, four for the ELC Leavers, and four for the Eligible 2s. These were explored through thirteen regression models, listed in Table A10.

Table A10: Regression models

Key outcome	Type of model	Base
4+ ASQ domains on schedule	Logistic	ELC Leavers
4+ ASQ domains on schedule	Logistic	ELC Leavers receiving only funded ELC
4+ ASQ domains on schedule	Logistic	ELC leavers receiving around 22-30 hours a week of ELC
4+ ASQ domains on schedule	Logistic	ELC leavers in lowest income quintile
ASQ communication domain on schedule	Logistic	ELC Leavers
SDQ total difficulties score close to average	Logistic	ELC Leavers
Whether in work, study or training	Logistic	Mothers of ELC Leavers
Whether in work, study or training	Logistic	Mothers of ELC Leavers receiving only funded ELC
Whether in work, study or training	Logistic	Mothers of ELC Leavers in lowest income quintile
4+ ASQ domains on schedule	Autoregressive logistic	Eligible 2s at age three
ASQ communication domain on schedule	Autoregressive logistic	Eligible 2s at age three
SDQ total difficulties score close to average	Autoregressive logistic	Eligible 2s at age three
Mental wellbeing (Rasch-transformed SWEMWBS)	Autoregressive linear	Parents of Eligible 2s at age three (where same respondent as at age two)

For the ELC Leavers, the model of four or more ASQ domains on schedule was followed up with three similar models for specific subgroups. These are intended to help clarify whether the increase in hours had an impact on the outcome. By looking only at those receiving funded ELC, there is a more direct comparison between those receiving 600 hours pre-expansion, and those receiving 1140 hours post-expansion. By looking only at those receiving around 22-30 hours of ELC (i.e. the number of funded hours provided post-expansion) we were attempting to look at what would have happened between 2019 and 2024 if funded hours had been at post-expansion levels for the whole period. This is useful to help contextualise the other findings. Models for the low-income group allowed us to see whether the impact of the expansion is different for this group than for the whole group. In a similar manner, the model of whether mothers of ELC Leavers were in employment, study or training was followed up with models for two subgroups.

Findings from these models in relation to the expansion of funded ELC have been given primacy over findings from the significance tests described above. This is because their more careful construction better controls other factors associated with the outcome, so we can say with more certainty whether the phase of data collection is associated with the outcome, keeping other things constant. The models also identify the key drivers of the outcome. Thus, they are able to indicate what it is about a child or a household or the ELC setting that is best able to predict a child’s development or a mother’s likelihood of being in work, study or training.

Types of model

Linear regression is a commonly used method for exploring the relationship between one or more predictor variables and a continuous outcome variable. It uses the ordinary least squares estimation method to produce an equation of the form:

X = β₀ + β₁X₁ + β₂X₂ + … + β_nX_n

where the outcome variable, X, can take any value and all the predictor variables, X₁ to X_n, are either binary or continuous. This means that for a variable X_i, holding all other predictor variables constant, the outcome variable, X, is predicted to be higher by a value of β_i (the coefficient) when X_i = 1 than when X_i = 0. For continuous predictor variables, the outcome variable is predicted to be higher by a value of β_i for each unit increase of the predictor variable. A coefficient of greater than 0 means that when the predictor variable takes the value 1, the outcome variable is predicted to be higher than it would be if the predictor took the value 0. This difference is considered statistically significant when the significance value < 0.05.

Logistic regression is a method for exploring the relationship between one or more predictor variables and a binary outcome variable. It uses the maximum likelihood estimation method to produce an equation of the form:

log[p(X) / (1-p(X))] = β₀ + β₁X₁ + β₂X₂ + … + β_nX_n

where the outcome variable, X, and all the predictor variables, X₁ to X_n, are binary. This means that for a variable X_i, holding all other predictor variables constant, the odds [p(X) / (1-p(X))] of the outcome variable, X, taking the value 1 when X_i = 1 are exp(β_i) times the odds of the outcome variable taking the value 1 when X_i = 0. Exp(β_i) is known as the odds ratio. An odds ratio of greater than 1 means that when the predictor variable takes the value 1, the likelihood of the outcome taking the value 1 is higher than it would be if the predictor took the value 0. This difference is considered statistically significant when the significance value < 0.05.

Autoregressive models account for the effect of the outcome at a previous time point, known as a lagged outcome. For the Eligible 2s, this means we have included the outcome at age two as a predictor for the same outcome at age three, along with the phase of data collection and any other factors identified. In this type of model, it is expected that the lagged outcome dominates, as the outcome at age two is likely to be a very strong predictor of the outcome at age three. Many fixed effects (variables which do not change over time, or change in a fixed way, such as sex and age) will not appear in the model, as they are accounted for within the lagged outcome – although if there is a strong association with the outcome, it may still be evident.

Examples of how to read the models and interpret the reported statistics are included on the notes page within the regression models supplementary tables.

Model building

In order to build the models, two approaches were taken. First, a base model was built using factors identified in previous research, including previous SSELC reports. Secondly, a manual stepwise approach was used to add and take away variables from this model until a stable model was achieved. The phase of the study (i.e. pre- or post-expansion) was always included in the model. All models for the ELC Leavers were constructed using weighted data, while models for the Eligible 2s at age three were constructed using unweighted data. For the child models, only those with both parent and keyworker questionnaires were included for the ELC Leavers. For the Eligible 2s, parent questionnaires at age three were required, as well as keyworker questionnaires at both ages. For the parent models, only those with parent questionnaires were included for the ELC Leavers, and those with parent questionnaires at both ages for the Eligible 2s.

The final models for the ELC Leavers, as reported, include only variables significantly associated with the outcome, plus the phase of data collection. For the Eligible 2s, they additionally included the lagged outcome (although, as expected, this showed a significant association in all cases).

Regression models require the predictor variables to be independent of each other. All the models were tested for multicollinearity to assess this requirement. Where two or more predictor variables were too closely correlated, only one of the variables has been included in the model (e.g. we would not include being in work in the same model as being in full-time work). Variables strongly associated with Phase (e.g. hours attended at the setting) have been excluded from all models.

Data management

Missing values on any of the predictor variables were recoded into the modal or median category, as appropriate, unless there were more than 30 missing cases, when they were coded as a separate category (e.g. income). See above for further details of missing data.

All scales and continuous predictor variables were recoded into ordinal variables in the most meaningful way possible. This is because it was not possible to determine suitable transformations to create linear associations with the outcome variables. The only exception to this was in the autoregressive model for parental wellbeing, where the lagged outcome of SWEMWBS at age two was allowed because of its linear association with the outcome of SWEMWBS at age three.

Predictor variables with multiple categories were collapsed into ones with fewer categories, either where categories had too few cases, or where collapsing the variable made the model easier to follow.

Variables included in the models

A large number of variables were considered for input into the models. These are listed in the notes page of the regression models supplementary tables. They were selected on the basis that they could reasonably be associated with the outcome variable, and the direction of association is most likely to be from the input to the output variable, although it is recognised that many associations are circular to some extent.

Variables considered include factors related to the health of the child, other forms of childcare used, employment, income, education, household composition (including having a younger child in the household), developmental risk factors, the home environment, parental health and wellbeing, characteristics of the setting (e.g. size, provider type, urban-rural indicator), the quality of ELC provision as measured by ITERS-3, ECERS-3, or the official Care Inspectorate Key Questions. All of these, to different extents, are known to be associated with child developmental outcomes.