Scottish Study of Early Learning and Childcare: Phase 5 Report

Appendix C: Details of data collection instruments

Ages and Stages Questionnaire (ASQ)

The Ages and Stages Questionnaire was administered as part of the keyworker observations.

Both the ASQ and SDQ questionnaires are widely used by Health Visitors across Scotland as part of their health reviews of pre-school children – the Child Health Programme.^[31] These questionnaires were also selected for inclusion in the Child Health Programme following an extensive review by academics and practitioners.^[32]

The ASQ is a structured assessment of a range of developmental domains to identify children at increased risk of developmental difficulties.^[33] The instrument includes 30 items split into five different domains: communication, gross motor, fine motor, problem solving and personal-social. The respondent indicates whether or not the child can complete the action or provide the response required by answering "yes", "sometimes" or "no". Each domain produces a summary score which can be used to indicate whether the child's development is on schedule, needs monitoring or requires further assessment. Whilst it is designed to be completed by parents, it was deemed suitable for completion by the child's keyworker at their ELC setting because it is informed by observation of the child.

Strengths and Difficulties Questionnaire (SDQ)

The Strengths and Difficulties Questionnaire was administered as part of the keyworker observations.

The SDQ is a commonly used behavioural screening questionnaire designed for use with children aged between four and 16.^[34] The questionnaire includes 25 questions about a child's behaviour to which the respondent can answer "not true", "somewhat true" or "certainly true". Responses can be combined to form five different measures of the child's development, namely emotional symptoms (e.g. excessive worrying), conduct problems (e.g. often fighting with other children), hyperactivity/inattention (for example, constantly fidgeting), peer relationship problems (e.g. not having close friends), and prosocial behaviour (e.g. being kind to others). In addition, the first four measures can be combined into a "total difficulties" scale. Higher scores imply greater evidence of difficulties on each of the scales, with the exception of the prosocial behaviour scale where the reverse is true. In this report, recommended banded versions of the scales have been used to create the following categories: "close to average", "slightly raised", "high" and "very high", with "very high" indicating multiple difficulties identified.^[35]

Shortform Warwick-Edinburgh Mental Wellbeing Scale (SWEMWBS)

The short form of the Warwick-Edinburgh Mental Wellbeing Scale is a commonly used series of questions intended to measure wellbeing. Respondents are presented with seven statements about positive feelings and thoughts, and asked how frequently they have experienced them over the previous two weeks.

The scale is created by summing the frequency of responses for all seven items, with a score of 1 for "none of the time" and a score of 5 for "all of the time". This provides a score of between 7 and 35. In order to make the data comparable with other surveys, the scores have to be "Rasch-transformed".

Cut offs to determine "high" and "low" wellbeing can be identified from population norms. High wellbeing is defined as the top 15% of scores, in the range from 27.5 to 35.0, and low wellbeing as the bottom 15%, in the range from 7.0 to 19.5.

Details of the scoring are available from the WEMWBS website.

Home learning environment scale

Four items were used to create the home learning environment scale. These each related to the number of days on which certain activities between the child and parent were conducted. The scale was created by summing the number of days on which each of the four activities had occurred in the last week.

Cut offs to determine quartiles were created using data from Phase 2 of the study. This allows for comparison between phases. The bottom quartile included scores in the range 0 to 16, the second in the range 17 to 21, the third in the range 22 to 24, and the top in the range 25 to 28.

Parent-child warmth scale

The parent-child warmth scale used in the SSELC comprises the seven items that form the warmth dimension of the short form of the Mothers' Object Relations Scale (MORS-SF). Details of the scale are available from the MORS website.

Each item was scored from 1 to 5, with 1 being "None of the time" and 5 "All of the time". Summing the seven items created a scale from 7 to 35. Three groups were created of roughly equal size, based on Phase 1 responses. Because of the skewed nature of the data, the bottom tertile (a score of 7 to 31) should not be interpreted as "low warmth", as many of the respondents in this group would have answered "often" or "all of the time" to each of the statements. The middle tertile was for a score of 32 to 34 and the top tertile a score of 35.

Confusion, Hubbub and Order Scale

The 'Parental confidence and capacity and home environment' section of the chapter on Parent Outcomes describes the 'Confusion, Hubbub and Order Scale' which was included to measure calmness and order within the household. The questionnaire included four statements, and respondents were asked how much they agreed or disagreed with each:

• It's really disorganised in our home
• You can't really hear yourself think in our home
• The atmosphere in our home is calm
• First thing in the day, we have a regular routine at home

To form the scale, the first two of the above statements were scored 1 to 5, with 1 being "strongly disagree" and 5 being "strongly agree"; the other two items were scored 1 to 5 in the reverse order, with 1 being "strongly agree" and 5 "strongly disagree". The sum of the four items created a scale from 4 to 20, with lower values indicating calmer households and higher values a greater degree of disorganisation / lack of routine. Tertiles were based on Phase 2 of the study, where the sample was representative of households with a four- or five-year-old child receiving funded ELC.

Early Childhood Environment Rating Scale (ECERS)

To gather information on the characteristics of ELC settings, inspectors from the Care Inspectorate (acting as observers independent of their regulatory roles) conducted observations of 150 settings using the most recent version of the Early Childhood Environment Rating Scale (ECERS-3). The ECERS was developed in the United States by the Environment Rating Scale Institute along with the Infant/Toddler Environment Rating Scale (ITERS-3). Both scales are widely used in English speaking countries. In the United Kingdom, ECERS has been used in both the Effective Provision of Pre-School Education (EPPE) study and in the more recent Study of Early Education and Development (SEED) in England.^[36] In Scotland, ECERS was used as long ago as 1994^[37] and has seen many applications since that time.

Both environment rating scales have a positive international reputation as a way of assessing the quality of provision in a snap-shot observation and as a tool which gives researchers access to the everyday experiences of children in their ELC settings. The scales have high reliability at indicator and item level when used by trained observers.^[38] Validity is also high in terms of their relationship to other ways of assessing quality and to measures of children's outcomes. Further, in conjunction with academics and the Care Inspectorate, some minor amendments were made to ensure that the ECERS was reflective of the aspects of quality that are expected in Scotland (e.g. that rainfall should not prevent outdoor play).

ECERS was used for a number of reasons: it centres on the experience of the child in the setting; it allows for the effect of setting quality on child outcomes to be controlled for; and it is relatively easy to administer given that only one three-hour observation is required. The tool can also be used to determine if particular setting characteristics contribute to differential outcomes in children. Furthermore, ECERS is designed for use in settings where most children are aged between three and five and as such, it was deemed suitable for use with the four- and five-year-olds involved in Phase 5 of the SSELC.

It is important to note that these tools are not the only method of assessing setting quality in Scotland. Indeed, the Care Inspectorate ratings provide a broader measure of the quality of practice and policy within settings that have also been found to be related to children's outcomes in Scotland.^[39]

As with the Care Inspectorate inspection methodology, the setting observations focused on outcomes. However, the methodology differed in that the ECERS tool was used to observe for three hours, with no consultation with setting staff and no professional dialogue or explicit feedback provided. This was because the observations were intended to be a snapshot to inform the study and control for variation in child outcome data, rather than serving as an assessment of an individual setting's quality. During the ECERS observations, observers looked at the six domains specifically for four- and five-year-olds. In contrast, during a formal inspection, Care Inspectorate inspectors consider a range of areas that impact on experiences for all children attending the setting, not just those in specific age groups. The key areas covered during a formal inspection are likely to include some or all of the domain areas but can also cover other aspects of the provision to evaluate the overall quality of the setting.

The ECERS scale comprises 35 items across six different subscales: space and furnishings; personal care routines; language and literacy; learning activities; interaction, and programme structure.

• Space and furnishings includes observation of: indoor space; furnishings for care, play, and learning; room arrangement; space for play; space for privacy; display for children; and play equipment.
• Personal care routines includes observation of: meals and snacks; toileting; health practices; and safety practices.
• Language and literacy includes observation of: encouraging children to expand vocabulary and use language; encouraging children to communicate; staff use of books with children; and encouraging children's use of books and familiarity with print.
• Learning activities includes observation of: fine motor; art; music and movement; blocks; dramatic play; nature and science; maths materials, understanding of written numbers and the use of maths in daily events; appropriate use of technology; and promoting acceptance of diversity.
• Interaction includes observation of: supervision of gross motor play; individualised teaching and learning; peer interaction; staff-child interaction; and discipline.
• Programme structure includes observation of: transitions and waiting times; free play; and group play activities.

In line with ECERS guidance, each subscale is scored from 1 to 7. These scores are calculated by averaging the score for each item within the subscale. Each of the 35 items are also scored from 1 to 7. These scores are calculated using the indicators contained within each individual item. Indicators are grouped under scores of 1 (inadequate), 3 (minimal), 5 (good), and 7 (excellent), with each indicator providing an example of what should be observed relevant to each score. Indicators themselves are scored as yes or no depending on whether the indicator has been observed. In some cases, observers are able to record indicators or items as not applicable; these are then excluded when calculating item or subscale scores. A score of 1 is given if any indicator grouped under 1 is scored yes. For an item to score a 7, each indicator grouped under 7 must be scored yes.