Scottish Study of Early Learning and Childcare: Phase 6

Appendix B – Methodology

Aims

Phase 6 of the SSELC was designed to provide post-expansion data on several specific child and parent outcomes as well as information about socio-economic characteristics, family and household circumstances, characteristics of childcare use and a range of additional circumstances, experiences and behaviours known to be associated with child outcomes. The aim was to follow up children who had taken part at Phase 4 of the study at age two, to gather data on progress following one year of funded childcare, and to gather data on a nationally representative sample of children of the same age – three-years-old.

Sampling

Two samples of children aged between three years and three years six months with a funded ELC place in Scotland were recruited for Phase 6.

Eligible 2s

The first sample, Eligible 2s, followed up children with parent or keyworker responses in Phase 4. At Phase 4 of the study, data was collected about 500 children aged between 2 years and 2 years 6 months who were eligible for and receiving funded ELC provision and their parents. Participants were recruited via ELC settings in 30 local authority areas. See the Phase 4 report for further details. Phase 4 Fieldwork was conducted between October and December 2023. A total of 341 questionnaires were received from parents / carers and 486 from keyworkers in 152 different settings. Attempts were made to follow up all of these children. Of the 500 children who took part at Phase 4, 415 were believed to be attending the same setting or another setting which took part at Phase 4 (155 separate settings); 75 were traced to new settings (59 settings) and 10 could not be traced.

Comparator 3s

The second, Comparator 3s, was a new sample of children selected from settings that were sampled in Phase 5. For the Comparator 3s sample, first, a stratified sample of ELC settings was drawn from settings that were selected (but did not necessarily respond to) Phase 5. For larger settings, a second stage was involved. If there were more than 10 eligible children at the setting, a sample of 10 children was drawn by setting staff from those eligible. The majority of settings had no more than 10 eligible children. In these cases, all eligible children at the setting were invited to take part.

A stratified random sample of 312 ELC settings was drawn, consisting of a main sample of 260 settings and a reserve sample of 52 settings. The sample was stratified by deprivation groups (ELC settings in the most deprived 20% of areas, based on SIMD, vs the rest). In Phase 5, settings in the most deprived quintile were oversampled to maximise the ability to analyse data in relation to the poverty-related outcomes gap. In Phase 6, all settings in SIMD quintiles 2 to 5 were selected from the Phase 5 sample, as well as 61 settings from the most deprived quintile.

Before selection, the Phase 5 sample was ordered by setting size, local authority, and deprivation score. Settings that had closed since the Phase 5 sample was drawn were excluded.

Weighting

Weights are commonly applied to survey data so that the achieved sample better represents the population it was drawn from. Groups that are under-represented in the achieved sample are given higher weights than those that are over-represented, with the aim of weighted data matching the population distribution by key characteristics. Survey estimates produced using the weighted data should then be closer to estimates that would have been gained from the whole population of interest.

As Phase 6 included multiple questionnaires, four sets of weights have been produced. These are for analysis of: Eligible 2s keyworker responses, Eligible 2s’ parent responses, Comparator 3s keyworker responses, and Comparator 3s’ parent responses. Different weighting approaches were used for Comparator 3s and Eligible 2s. The same weighting approach was used for both sets of Comparator 3s weights, consistent with the weighting of Phases 2, 3, and 5 of the project. The Eligible 2s weights were based on the Phase 4 weights for each responding case, with minor adjustments to bring the profiles of parent and keyworker responses from Phase 6 closer to their Phase 4 profiles.

A small number of survey responses (n = 9) could not be matched with either the Comparator 3s or Eligible 2s samples. These cases were therefore dropped and not given a weight.

Eligible 2s

As Eligibles 2s followed up children with responses in Phase 4, Eligible 2s weights were based on the Phase 4 weights for each sample. The weighting method was consistent with the general approach in Phase 3, which followed up children with responses in Phase 1. However, Phase 1 responses were not weighted unlike Phase 4 responses. In Phase 3, responses were checked for consistency with the Phase 1 response profile. Weights were not needed to adjust the Phase 3 profile to match Phase 1. In Phase 6, Eligible 2s were weighted by their Phase 4 weights and compared with the weighted profile of Phase 4 responses, to determine whether additional adjustment was needed to the Phase 4 weights for these cases.

Eligible 2s Keyworker Weights

Eligible 2s keyworker weights were created for 285 keyworker responses. Firstly, Phase 4 keyworker weights were matched onto the responses. 6 responses that did not have Phase 4 keyworker weights were assigned their Phase 4 parent weights. The weighted profiles of Phase 4 keyworker responses and Phase 6 Eligible 2s keyworker responses were compared, to identify any significant differences. Bias was identified in SIMD quintiles, therefore the weights were rescaled to bring the Phase 6 profile in line with Phase 4. This rescaling upweights the 1^st and 2^nd SIMD quintiles (the 40% most deprived areas) and downweights the 3^rd to 5^th SIMD quintiles in the Eligible 2s keyworker responding sample. To finalise the weights, responses from a small number of sets of twins each had their weight divided by 2. This equated to treating the twins as one child, for consistency with their treatment in Phase 4. The design effect of the final keyworker weights is 1.43 and the efficiency 70%.

Eligible 2s’ Parent Weights

Eligible 2s’ parent weights were created for 164 parent responses. Firstly, Phase 4 parent weights were matched onto the responses. 27 responses that did not have Phase 4 parent weights were assigned their Phase 4 keyworker weights. The weighted profiles of Phase 4 parent responses and Phase 6 Eligible 2s’ parent responses were compared, to identify any significant differences. Bias was identified in setting size band; therefore, the weights were rescaled to bring the Phase 6 profile in line with Phase 4. This rescaling upweights settings with fewer than 10 Eligible 2s and downweights settings with 10 or more.

To finalise the weights, two responses from a set of twins each had their weight divided by 2. This equated to treating the twins as one child, for consistency with their treatment in Phase 4. The design effect of the final parent weights is 1.55 and the efficiency 65%.

Comparator 3s

The basic weighting approach for Comparator 3s consisted of two elements: selection weighting and non-response modelling. The first stage adjusted for differential probability of selection (for settings and children) resulting from the sample design. The second stage adjusted for differences in the profiles of sampled and responding settings, using logistic regression modelling. Calibration weighting, which adjusts the profile of the weights to match estimates of the population, could not be used due to the absence of detailed population estimates for eligible three-year-olds.

Further details of the methods used to produce each set of weights are provided in the subsections below.

Comparator 3s Keyworker Weights

Comparator 3s keyworker weights were created for 851 keyworker responses. Up to 10 keyworker responses from each setting were allowed. First, selection weights for the settings were created from the inverse selection probability of each setting during sampling. As Phase 6 Comparator 3s were selected from the Phase 5 sample, selection probabilities from Phases 5 and 6 were multiplied together to calculate the overall probability.

Second, a setting-level logistic regression model was run, weighted by the selection weight. The outcome for this model was any keyworker responses from the setting and the covariates included were SIMD quintiles, setting type (LA or private/voluntary/non-profit), and size band. Non-response weights were calculated for the 129 settings with Comparator 3s keyworker responses as the reciprocal of the propensity to respond estimated from this model. Third, the setting-level non-response weights were combined with the setting selection weights and matched onto the 851 keyworker responses.

As the final step, child selection weights were calculated to adjust for children’s differential probability of selection between settings. These were calculated from the inverse of number of children selected per setting (if recorded on the response sheet) or number of children sampled (if not available from the response sheet) divided by estimated eligible children at the setting. The setting-level weights were combined with the child selection weights and checked for outliers. The top weights were trimmed at the 98^th percentile to improve efficiency. The design effect of the final Comparator 3s keyworker weights is 1.73 and their efficiency 58%.

Comparator 3s’ Parent Weights

Comparator 3s’ parent weights were created for 516 parent responses. Up to 10 parent responses from each setting were allowed. First, selection weights for the settings were created from the inverse selection probability of each setting during sampling. As Phase 6 Comparator 3s were selected from the Phase 5 sample, selection probabilities from Phases 5 and 6 were multiplied together to calculate the overall probability.

Second, a setting-level logistic regression model was run, weighted by the selection weight. The outcome for this model was any parent responses from the setting and the covariates included were SIMD quintiles, setting type (LA or private/voluntary/non-profit), and size band. Non-response weights were calculated for the 141 settings with parent responses as the reciprocal of the propensity to respond estimated from this model.

At the third step, setting-level non-response weights were combined with the setting selection weights and matched onto the 516 parent responses. Finally, child selection weights were calculated to adjust for children’s differential probability of selection between settings. These were calculated from the inverse of number of children selected per setting (if recorded on the response sheet) or number of children sampled (if not available from the response sheet) divided by estimated eligible children at the setting. The setting-level weights were combined with the child selection weights and checked for outliers. Top weights for four settings, approximately the 97.5^th percentile, were trimmed to improve efficiency. The design effect of the final parent weights is 1.82 and the efficiency 55%.

Analysis and reporting

The data analysis was performed using SPSS version 29. All analyses employed weighted data, except when describing the characteristics of the participating sample or the settings. Weighting was applied to ensure that the achieved sample more accurately represents the population from which it was drawn. Various weights were used depending on the variables included in the analysis (see above).

A test for statistical significance enables us to assess the confidence with which we can assert that two percentages being compared are genuinely different in the population, given the degree of uncertainty we are willing to accept in our sample. To compare Eligible 2s (or their parents) at age three with the same children at age two, significance testing was conducted in SPSS using paired samples t-tests and paired samples proportions tests. In both instances, a two-sided significance value was used. For comparisons between Eligible 2s at age three and Comparator 3s, or for comparing subgroups within these groups, significance testing was performed using logistic or linear regression.