Family Nurse Partnership evaluation: methods and process

2. Methods

2.1. Study design 

Following the Medical Research Council’s (MRC) guidance on natural experiments, the FNP Programme in Scotland will be evaluated using a case-cohort design (8). We will generate a linked anonymised research database to compare routinely available health, education, and social care data between FNP Clients and Controls. 

2.1.1. Natural experiments

Natural experiments encompass a range of observational study designs used to evaluate the impact of population health and policy interventions, usually when an RCT is not possible (8,9). 

The key feature of natural experiments is that researchers do not have the ability to assign participants to ‘treatment’ and ‘control’ groups. Rather, divergences in law, policy or practice, across time and space, offer the opportunities for evaluation. These designs involve characterising periods of time or populations which have and have not been exposed to an intervention or policy. The validity of natural experiments depends on the ability that an observational design can create comparable ‘treatment’ and ‘control’ groups.  A challenge with non-randomised exposures is that individuals/groups are often selected on the basis of need or risk factors associated with the outcome.  As with other observational or quasi‐experimental designs, natural experiments will never unequivocally determine causation because the researcher cannot randomly allocate populations exposed. However, these designs provide a useful inferential tool to evaluate interventions and policies which are not or cannot be randomised, for example existing services.

2.1.1. Data linkage in Scotland

Electronic Data Research and Innovation Service (eDRIS) provides a single point of contact to assist researchers in study design, approvals and data access for projects using routinely collected data in Scotland. The role of eDRIS is to advise on the application submitted to the Public Benefit and Privacy Panel (PBPP), liaise with the trusted third party indexing team and the various data controllers on behalf of the study. eDRIS also oversee all data transfer and linkage to create the study cohort within the the safe haven.

National Services Scotland (NSS) Safe Haven is a secure environment in which data are linked and stored. Access is provided either remotely or via a secure access point. Both access methods allow trusted and authorised researchers to analyse anonymised individual level data while maintaining confidentiality. Remote access to the safe haven is via Citrix using an accredited organisation’s secure desktop / laptop.

2.2. Setting

FNP was implemented in January 2010 in one Health Board (HB) (NHS Lothian) to test the feasibility and acceptability of the programme within a local context. By the end of 2015, a total of 20 FNP teams/cohorts existed within 10 of the 14 NHS HBs in Scotland, delivering FNP at a local level. The participating HBs are shown in Table 1.

2.3. Study population

The study population will be all women who were eligible for the FNP Programme from 1st January 2009 to 31st March 2016 and their first-born child(ren). Cases will be defined as FNP Clients; all women (and first-borns) enrolled into FNP in the ten participating Scottish HBs since its initiation from 1^st January 2010 to 31^st March 2016. The Control population will be women eligible for enrolment in the FNP programme during a period when FNP recruitment was not offered in the same FNP catchment area:

i. in the 12 months prior to initiation of FNP recruitment [Pre-FNP] (starting 1^st January 2009)

ii. in the 12 months post FNP recruitment [Post-FNP] (ending 31^st March 2016)

iii. between periods of FNP recruitment (i.e. when recruitment was temporarily suspended due to caseload capacity being reached) [Interval].

Figure 1 shows the Cases and all potential Controls. Detail on the process of identification of Cases and Controls follows below.

Table 1: Participating Scottish NHS Health Boards with enrollment dates

Health Board	Geographical area	Team/Cohort	Date started enrolling	Date finished enrolling	Interval dates1
Ayrshire and Arran	East, North and South Ayrshire	A / 1	04/02/2013	03/02/2014	04/02/2014 to 09/03/2014
		Part-time nurse recruiting 17 Clients	10/03/2014	04/09/2014	05/09/2014 to 15/10/2015
		A / 2	16/10/2015	Ongoing recruitment2	N/A
Borders	Whole board	A / 1	01/08/2015	Ongoing recruitment2	N/A
Fife	Whole board, with specific percentages taken from each area	A / 1	01/08/2012	31/07/2013	01/08/2013 to 16/03/2014
	Whole board	B / 1	17/03/2014	31/03/2015	N/A (new notifications picked up by Team A)
	Whole board	A / 2	01/04/2015 (notifications picked up after Team B stopped recruiting)	May 2016	May 2016 to present and 2-3 week gap in March 2016
Forth Valley	Stirling, Clackmannanshire and Falkirk	A / 1	28/04/2014	21/05/2015	N/A
Grampian	Aberdeen City, Aberdeenshire and Moray	A / 1	18/05/2015	Ongoing recruitment2	N/A
Greater Glasgow and Clyde	Glasgow City, West Dunbartonshire, East Dunbartonshire	A /1	22/10/2012	01/11/2013	02/11/2013 to 18/10/2015
		A / 2	19/10/2015	Ongoing recruitment2	N/A
	East Renfrewshire, Inverclyde, Renfrewshire	B / 1	01/08/2014	28/01/2016	29/01/2016 to present
Highland	South and mid areas of Highland Council (Inner Moray Firth)	A/ 1	04/02/2013	26/05/2014	27/05/2014 to 31/12/2015
		A / 2	01/01/2016	Ongoing recruitment2	N/A
Lanarkshire	South Lanarkshire	A / 1	08/07/2013	15/10/2014	N/A
	North Lanarkshire	B / 1	08/07/2013	15/10/2014	N/A
Lothian	Edinburgh city	Pilot cohort	25/01/2010	31/10/2010	01/11/2010 to 31/08/2012
	Edinburgh city	A / 1 (not including pilot cohort)	01/09/20123 25/09/20124	24/09/2013	None (Overlap with Team B)
	West Lothian	A / 1 (not including pilot cohort)	01/03/2013	24/09/2013	None (Overlap with Team B)
	Edinburgh and West Lothian	B / 1	01/08/2013	31/07/2014	None (new notifications picked up by Team C)
	Mid Lothian	B / 1	01/04/2014	31/07/2014	None (new notifications picked up by Team C)
	Edinburgh, West and Midlothian	C / 1	01/08/2014	31/07/2015	None (overlap with other recruiting teams)
	Edinburgh, West and Midlothian	A/B/C Combined teams with rolling recruitment	01/04/2015	Ongoing recruitment2	N/A
Tayside	Whole board	A / 1	01/07/2011	30/09/2011	01/10/2011 to 18/10/2012
		A / 2	19/10/2012	20/11/2013	21/11/2013 to 31/12/2013
		A / 3	01/01/2014	Ongoing recruitment2	N/A
		A / 4	01/01/2016	Ongoing recruitment2	N/A

1. Interval dates are periods in between FNP recruitment when recruitment is temporarily suspended due to caseload capacity being reached

2. at 31/03/2016

3. Clients aged 16 and under only

4. For all eligible Clients

Figure 1: Identification of the study cohort  

2.3.1. Identifying Cases 

Cases will be defined as FNP Clients (and first-borns) enrolled into FNP in the ten participating Scottish HBs since its initiation from 1^st January 2010 to 31^st March 2016. This end date will give birth outcomes on women  (and their first-borns) around the end of October 2016 (depending on gestation at recruitment). FNP Clients will be identified from the FNP Scottish Information System (FNP SIS), a national database based on FNP data forms and which is accessible to family nurse (FN) teams through a secure web-based portal. The FNP National Unit (FNP NU) within NHS Education for Scotland (NES) will provide Community Health Index (CHI) numbers and other identifiers (name, date of birth (DOB), postcode) for all FNP Clients to the electronic Data Research and Innovation Service (eDRIS) team, who will link to the SMR02 Maternity Inpatient and Day Case dataset and a flag will be added to identify them as FNP Clients. Table 2 shows the FNP eligibility criteria applicable to the period under observation.

Table 2: FNP eligibility criteria 

Inclusion criteria
Women must be: living in an FNP-recruiting NHS Health Board area a first-time mother-to-be (women are eligible if a previous pregnancy resulted in a miscarriage, stillbirth or termination) aged 19 years or younger at time of last menstrual period (LMP) enrolled into FNP no later than 28 weeks.
Exclusion criteria
Women with an intention: to relinquish the baby to move outside FNP area

2.3.2. Identifying Controls

(a) Applying eligibility criteria

The identification of the remaining eligible population (potential Controls) using SMR02 will also be carried out by the eDRIS team. The SMR02 contains fields that map well to the FNP eligibility criteria (Table 3). The only field we have to use as a proxy to these eligibility criteria is gestation at time of antenatal booking instead of at time of enrolment into FNP. The process of identifying Controls will also exclude any pregnant women not registered on the SMR02 (i.e. women delivering at home or in non-NHS hospitals, around 1% of all births)(10). Ineligibility criteria (intention to relinquish the baby and / or moving outside FNP area) are applied to women during the FNP recruitment process but cannot be applied in the selection of Controls as they are not assessed in the SMR02 (or any other dataset). 

Table 3: Eligibility criteria applied to the SMR02 dataset fields to identify eligible Control women 

Inclusion criteria	Criteria applied to SMR02 dataset
Living in an FNP-recruiting NHS Health Board area	Postcode of mother at antenatal booking will be mapped to each FNP recruiting area1
A first-time mother-to-be (women are eligible if a previous pregnancy resulted in a miscarriage, stillbirth or termination)	A flag will be derived by eDRIS to confirm that the mother is a first time mother, examining any birth previous to the antenatal booking date.
Aged 19 years or younger at time of last menstrual period (LMP)	Estimated age (years) at LMP: Derived from maternal date of birth and LMP date
Enrolled into FNP no later than 28 weeks	Estimated gestation (weeks) at booking: Derived from date of booking and LMP date
Exclusion criteria
Mother-to-be will relinquish baby at birth	Not measurable at recruitment – minimal risk to numbers
Moving outside of the FNP catchment area before programme end	Not measurable at recruitment – minimal risk

¹ Either Health Board level or smaller geographical area such as Community Health Partnership level or where a recruitment area was defined by travel time

(b) Applying recruitment area criteria

Controls will be women who were eligible for enrolment in the FNP programme (using their antenatal booking date as a proxy) during a period when FNP recruitment was not offered in the same FNP catchment area. To further identify the potential Controls, a FNP recruitment table was provided to eDRIS (Table 1). This included the start and end dates of recruitment including any intervals when cohorts ceased recruitment, the geographical coverage (either entire Health Boards/ Community Health Partnerships level or exact postcodes for areas defined by travel time) for each FNP team. This allowed eDRIS to identify and flag potential Controls from the same FNP areas of recruitment, and categorise them as such. It also allowed them to categorise Controls further into those with an antenatal booking date:

i. in the 12 months prior to initiation of FNP recruitment [Pre-FNP];

ii. in the 12 months post FNP recruitment [Post-FNP];

iii. between periods of FNP recruitment (i.e. when recruitment is temporarily suspended due to caseload capacity being reached) [Interval].

A second potential Control cohort exists consisting of women eligible for enrolment into the programme during a period of active FNP recruitment and who: 

iv. were approached for FNP but not enrolled; 

v. were not approached (e.g. insufficient capacity in team to offer to all eligible women; near end of recruitment period and caseloads nearly full).

Following advice from the Public Benefit and Privacy Panel (PBPP), it was not possible to access these potential Controls. However, it is important to note that these groups may have different demographic, social or personal characteristics from those that enrolled in the programme and may differ non-randomly from the Controls in (i) to (iii) for some variables. It would be beneficial to the evaluation to understand the characteristics of Controls (iv) and (v) above and so aggregate statistics will be provided by eDRIS. 

2.3.3. Identifying children

A mother-child link is available within the SMR02 enabling a flag for any children born after the enrolment date (FNP Clients)/antenatal booking date (Controls) to be derived, typically between 15 to 35 weeks. Once the FNP Clients and Controls and their first-born children are identified, the eDRIS team will send the Community Health Index (CHI) number and a “FNP Client / Control” and a “mother/child” flag to National Records of Scotland (NRS) Indexing team. 

2.4. Data Access Approvals

2.4.1. Public Benefit and Privacy Panel (PBPP)

The PBPP is a governance structure of NHS Scotland with a remit to carry out information governance scrutiny of requests for linkage of and access to individual level health data on behalf of NHS Scotland. The full scope of the panel can be found here http://www.informationgovernance.scot.nhs.uk/pbpphsc/home/about-the-panel/. 

The PBPP operates on a two-tier structure utilising the proportionate governance method:

• Tier 1: Assesses the more technical, security and legalistic aspects of requests for data
• Tier 2: Considers the wider privacy issues relating to the use of health and social care data

An application was made to PBPP to access individual level data from national datasets held by NHS NSS, FNP data from individual NHS Scotland health boards and linking these to education and social care datasets provided by the Scottish Government Education Analytical Services (EAS).  Approval was sought to access individual level data from health boards via a Data Sharing Agreement (DSA) and from EAS via a data access application.

2.4.2. Education Analytical Services (EAS)

The EAS Division plays a key and leading role in the collection, analysis and publication of analytical evidence across the School Education and Young People sector. The EAS Data Access Panel is formed by the Head of the EAS Division - the ‘Information Asset Owner’ of data held by the EAS Division, the Divisional Data Access Officer, and other senior officials holding relevant datasets.  The purpose of the panel is to assess the objectives and data protection implications of data request applications as well as the security arrangements for the data requested. The application is assessed using the privacy risk matrix template to determine if the application is a simple or complex request.

An application was made to access individual level data held by EAS such as school and social care datasets and to link to the health datasets made available via eDRIS.

2.4.3. Requirements

In addition to the applications submitted to PBPP and EAS, the following governance and contractual aspects were considered and actioned as appropriate. These requirements were either outlined as required by the data access panels or were subsequently conditions of approval once considered by the panel.

Ethical approval: Advice about ethical requirements was sought from South East Scotland Research Ethics Service.Their recommendation was that under the terms of the governance arrangements for research ethics committees ethical approval was not required. This is because the project is an evaluation limited to using data obtained as part of usual care.

A Privacy Impact Assessment (PIA) was required by PBPP to identify and assess risk where data are being processed

A new information leaflet to be handed to all new FNP Clients about how their data will be stored and used was requested by PBPP and developed and issued by Scottish Government (SG) / FNP (2).

PBPP also requested assurances around the methodologies in the form of independent peer review. The study has an independent study steering committee (SSC) and a letter from the chair of this committee was submitted to respond to this request. The SSC provides on-going scientific scrutiny (i.e. consistent with this request).

All researchers are required to be approved researchers, defined by eDRIS as having completed appropriate information governance training (e.g. ‘MRC safe researcher’); have appropriate approvals in place; have read the NHS Confidentiality Code of Practice; affiliated with an Approved Organisation; and have read and signed the eDRIS User Agreement.

Secure all required data sharing and processing agreements between all data controllers and data processors (See Table 4).

Table 4: Required agreements for this data linkage study

Data	Data Controller	Share with	Processed by	Agreements required
FNP data	Health Board (x10) (Table 1)	Scottish Government		10 x Data Sharing Agreements (DSA)
			eDRIS | NHS Services Scotland (NSS)	10 x Memorandum of Understanding (MOU)
			National Records of Scotland (NRS)	10 x Data Processing Agreements (DPA)
NHS Scotland Health data	NHS Services Scotland (NSS)	Scottish Government	eDRIS | NHS Services Scotland (NSS)	Data Sharing Agreement (DSA)
Education Analytical Service	Scottish Government	Scottish Government	National Records of Scotland (NRS)	Data Processing Agreement (DPA)
Study Data	Scottish Government	Scottish Government	Cardiff University (CU)	Data Processing Agreement (DPA)

2.5. Datasets

Table 5 lists the datasets requested for use for this study alongside the Data Controller and the relevant panel to approve data access.

Table 5: Requested datasets

Dataset	Data Controller	PBPP	EAS
Scottish Morbidity Record (SMR) 00 – Outpatient Attendance	NHS NSS	√
SMR01 – General/Acute Inpatient and Day Case	NHS NSS	√
SMR02 – Maternity Inpatient and Day Case	NHS NSS	√
SMR04 - Mental Health Inpatient and Day Case	NHS NSS	√
Community prescribing and dispensing	NHS NSS	√
National Record for Scotland (NRS): deaths	NHS NSS	√
Community Health Index (CHI): demographics	NHS NSS	√
Unscheduled Care: Accident and Emergency (A&E)	NHS NSS	√
Child Health Systems Programme Pre-School (CHSP –PS): Health Visitor first visit; 6-8 week review; 27-30 month review; unscheduled review	NHS NSS	√
Child Health Systems Programme School (CHSP-S): Primary 1 - screening and assessment	NHS NSS	√
FNP Scottish Information System	Local Health Boards via FNP SIS	√
School/Pupil Census	SG		√
Attendance, Absence and Exclusions	SG		√
School Leavers (Summer and Christmas)	SG		√
Skills Development Scotland: destinations	SG		√
Children and Young People: Looked after children	SG		√
Children and Young People: Child protection register	SG		√
Scottish Credit and Qualifications Framework	SG		√
Achievement of Curriculum for Excellence levels collections	SG		√

SG - Scottish Government; NHS NSS - NHS National Services Scotland

PBPP – Public Benefit and Privacy Panel; EAS - Education Analytical Services

2.6. Data flow 

2.6.1. The Population Spine 

The Population Spine contains the personal identifiers of all individuals in Scotland who have been in contact with NHS Scotland. It is an existing set of records that covers most of the population, and has been linked to a high standard. The pupil census (data collected from publicly funded schools and their pupils on a set date each year) is matched to the spine to create an anonymised “read-through” index key which NRS and EAS hold at a person-level. When the cohort is identified on the spine the read-through keys will be sent from the indexing team to EAS to identify the cohort.

The pupil census is matched to the spine using only DOB, sex and postcode. This means that where there is more than one spine record with the same credentials the precision of these matches (in around 2-3% of cases, and mainly resulting from multiple births) will be around 50%.

The data linkage process for this project is shown in Figure 2, based on the Scottish Informatics Linkage Collaboration (SILC) procedure.

1 CHI numbers (plus other identifiers) for all FNP Clients are provided by FNP to the eDRIS team, who link to the SMR02 dataset and add a flag to identify them as FNP Clients. The identification of the remaining eligible population (potential controls) using SMR02 is also carried out by the eDRIS team.  

2 The controls are further restricted by FNP team recruitment areas and flagged accordingly (Pre-FNP, Post-FNP, Interval) and are given an anonymous ID and made available via the safe haven for the research team.

3 Once the final set of FNP Clients and controls are identified, eDRIS will identify the first-borns for each mother and send CHI number and a “case/control” and a “mother/child” flag to National Records of Scotland (NRS) Indexing team. 

4A NRS have a population spine which contains CHI number and where relevant, an Education identifer for each individual in Scotland. Using CHI number, NRS will identify all individuals on their spine. NRS will then generate  a different unique index along with CHI number and the mother/child flag to each of the health data providers.

4B Similarly, NRS will return the education identifier from their spine and a different unique index for each education / social work dataset will be sent to EAS. NRS will also construct a master control file which links together the different index numbers a person has in each dataset. This control file is sent to the safe haven. 

5 Each data provider will extract the required data from their data set (using CHI / education identifier). The index provided by NRS will be attached to the extract, CHI / education identifiers will be dropped and the files sent to the safe haven. 

6 An automated script called the linkage agent is then run on all files in the safe haven to replace the different index numbers in each dataset with a master index number that is common across all datasets.

7 A project specific account is created in the safe haven and approved data users at Cardiff University can access via a secure remote gateway via their own computer (11). The master index number allows the person analysing the data to see and link all the records belonging to an individual across all datasets without the need for access to personal identifiers. Data cannot be removed or transferred from the safe haven until it has been disclosure checked by eDRIS.

2.6.2. The pseudonymised dataset 

The data used for analysis will not contain any identifiable fields (e.g. participant names and CHI numbers). This was agreed in the application to PBPP. Disclosive geographical areas such as postcodes at antenatal booking were only used by eDRIS for linking FNP Clients to SMR02 and to identify Controls. Health board, Community Health Partnership area, an anonymous FNP team ID, and Scottish Index of Multiple Deprivation (SIMD) quintile were the only geographical data required for analysis. All de-identification (such as changing date of birth to week and year of birth, postcode to deprivation quintile) and deriving variables from identifiable data (e.g. time to event instead of dates of events) will be carried out by eDRIS prior to the data being available to researchers (11). 

Figure 2: Data flows