Child Poverty Action Group - advice, information and training services: evaluation

3. Secondary data – analysis and findings

This section reports on the key findings from the evaluation of the secondary data provided by CPAG in Scotland up to 2018/19.

CPAG in Scotland provided a range of data and information on their second tier advice, information and training services for review in the evaluation. These included:

• Quarterly grant funding progress reports detailing the number of cases supported by the Advice Line, number of participants attending training courses and events per topic, number of handbooks/factsheets/e-bulletins produced and distributed per topic, availability of the Appeals Toolkit, number of times E-learning courses were accessed as well as the number of people enrolled and the number of participants attending the annual welfare rights conference
• Annual conference evaluation data containing findings from evaluation forms completed by attendees of the annual Welfare Rights Conferences from 2014-2018
• Advice Line Database data containing details for each inquiry made by frontline workers/organisations/individuals to the Advice Line including the date, level of inquiry, subject and organisation/type of organisation making the enquiry
• Advice Line use survey data containing findings from the surveys launched annually, assessing the Advice Line Service users' views and ratings of the Advice Line
• Evaluation forms for E-learning courses containing feedback for the courses from participants as well as suggestions for improvements

3.1 Advice Line data

In total, the data provided by CPAG covered 16,413 enquiries, from 2014/15 to 2018/19. This section will consider the available data on the Advice Line, before briefly reviewing available feedback on training and data about use of online resources.

Overall there has been a 42% increase in the use of CPAG's advice helpline in the last five years (see Figure 2).

This increase in call volume in the last five years has been largely driven by the effect of increased enquiries about Universal Credit (UC), which has increased by a far greater extent than other types of enquiries (see Figures 3 and 4).

Figure 4. Percentage Increase/Decrease in enquiries to the Advice Line by enquiry type, 2014/15-2018/19

Enquiry Type	Percentage change
Benefits for Students	96%
DLA/AA	-11%
ESA	38%
HB	-2%
IS	-41%
JSA	-60%
PC	-23%
PIP	74%
UC	5922%
WTC/CTC	-12%
Total	71%

Universal Credit-related enquiries have increased from 32 in 2014/15 to 1,927 in 2018/19, a percentage increase of 5922%, far exceeding any other enquiry type and comprising 40% of all calls made to the advice line.

The most frequent users of the Advice Line between 2014/15 to 2018/19 were from Citizens Advice Bureaux (see Figures 5 and 6). Citizens Advice Bureaus (CABs) made more than 30% of the total number of enquiries to the Advice Line every year. From 2017/18 to 2018/19 the Advice Line saw an increased usage by organisations defining themselves as "Other voluntary sector" while, at the same time, organisations defining themselves as "Support providers" appear to have decreased by a roughly similar amount. The reason for this is unknown, however, it is likely that this is a data anomaly and "Support providers" may have been included in "Other voluntary sector" in this time period. "Local Authorities", "Housing Associations" and the remaining organisation types (Other/Not defined) appear to have made almost the same quantity of enquiries to the Advice Line over the period 2014/15 – 2018/19.

The number of enquiries made via the Advice Line did see an increase among all organisation types over the reference period, with the exception of those identified as 'Support providers' which, as mentioned above, could be an anomaly.

The majority of organisations make one enquiry to the Advice Line (58% averaged between 2014/15 and 2018/19), with a small number (8%) making 10 or more.

Call volume varies a little per quarter, but overall represents similar ranges across the year when averaged between 2014/15-2018/19 (see Figure 8).

From 2014/15 – 2018/19 most of the enquiries were made via the phone, however, the number of enquiries made by email has been steadily increasing (see Figure 9). There was a sharp decrease in the number of enquiries made by phone between 2014/15 and 2015/16 and, since then, a gradual increase at an approximately similar rate as the increase seen in the number of email enquiries.

3.2 Advice Line survey user data

CPAG in Scotland runs an annual survey to see the overall opinion of people with regards to its Advice Line. Survey results were consistent across the years. The findings, averaged from the data collected between 2014/15 and 2018/19 is presented below:

3.2.1 Advice Line

• 100 % of respondents were satisfied with the Advice Line's overall level of service
• An average of 20% (49 out of the 249 respondents between 2014/15 and 2018/19) reported that they had trouble contacting the Advice Line. Over the years the percentage has fluctuated but seen an overall increase from 10% of respondents in 2014 to 25% of respondents in 2019. Text responses across all years reported similar results, that the advice line is sometimes busy, even if they call multiple times
• 100% of respondents reported that the staff explained the advice clearly
• 100% of respondents reported that staff were knowledgeable
• 100% of respondents reported that staff were approachable
• 100% of respondents reported that the Advice Line helps them feel more confident that their advice is up-to-date and accurate
• 100% of respondents reported that the Advice Line helps them resolve issues for clients more quickly
• 100% of respondents reported that the Advice Line helps them achieve better outcomes for clients
• 100% of respondents reported that they feel better equipped to deal with similar cases
• 97% of respondents reported that the Advice Line helps them prepare information materials for clients
• 100% of respondents reported that the CPAG Advice Line is the best place to get second-tier benefits advice
• 96% of respondents reported that they shared the information they got from the Advice Line with colleagues

3.2.2 Outcomes for clients after using the Advice Line

• 38% of respondents reported that their client received general advice and information about eligibility for benefits
• 12% of respondents reported that their client was helped to make an application for benefits
• 16% of respondents reported they achieved financial gain for the client
• 9% of respondents reported that they achieved reconsideration/the appeal was resolved in the client's favour
• 31% of respondents reported that they referred the client on to another agency for more specialist advice

The overall findings from the surveys conducted by CPAG in Scotland suggest that most users are overwhelmingly supportive with regards to the quality of service provided. The only reported negative was increased difficulty in accessing the advice line.

3.3 E-learning

From 2014/15 to 2017/18 E-Learning courses were available on the following topics:

• Universal Credit
• Surviving the Appeals Process
• Scottish Welfare Fund
• Benefits Overview
• Making the most of the Handbook
• Planning and Paying for Funerals
• Supporting Hard-up Households

In this same time period E-Learning courses were accessed 72,419 times. 93% of respondents rated the E-Learning courses as either good or excellent.

Between 2014/15 and 2018/19 a total of 806 courses and events were held in-house by CPAG in Scotland. With the exception of 2016/17, where there was a small decrease, the number of training courses held per year has remained fairly consistent (see Figure 10., below).

Demand for training courses has been varied, depending on the subject, and the changes in the courses available in a given year make it challenging to generalise about overall trends. However, since its introduction in 2017/18, the training course on 'Universal Credit' has been the most widely attended. This can be seen in Figure 11. below.

Figure 11. No. and proportion of participants attending courses by category, 2014/15-2018/2019

Year	Topics	No. of participants	No. of courses	No. of events	Total courses & events for the year
2014-2015	Families/early years	334	11	7	178
	Students	226	7	7
	Children in care/disabled children	268	6	6
	Minority ethnic	333	16	2
	Welfare reform	1160	43	16
	Other	1046	51	6
2015-2016	Families/early years	432	13	9	161
	Students	345	15	5
	Children in care/disabled children	181	7	4
	Minority ethnic	310	7	9
	Welfare reform	743	26	10
	Appeals representation	78	4	0
	Other	957	47	5
2016-2017	Families/early years	284	12	4	141
	Students	182	7	3
	Children in care/disabled children	108	4	4
	Minority ethnic	224	5	3
	Welfare reform	1037	22	22
	Appeals representation	50	4	0
	Other	712	43	8
2017-2018	Families/early years	429	16	6	171
	Students	289	12	5
	Children in care	88	3	1
	Disability and carers	541	16	7
	Minority ethnic	115	3	4
	Universal credit	987	50	2
	Appeals representation	122	7	0
	Other	601	28	11
2018-2019	Families/early years	359	14	9	155
	Students	262	12	4
	Children in care	198	4	3
	Disability and carers	261	8	5
	Minority ethnic	228	4	4
	Universal credit	1002	48	6
	Appeals representation	68	5	0
	Other	530	21	8

With regards to the factsheets downloaded there has been an overall increase over this period of time. It is unclear exactly why there has been an increase, it is possible that this is due to increased use of computers and other internet-enabled devices over this period. This can be seen in Figure 12. below.

The use of e-bulletins was fairly consistent from 2014/15 to 2017/18 however there was a sharp decline in 2018/19 as a consequence of the introduction of General Data Protection Regulations (GDPR) which saw the mailing contact list reduced significantly. See Figure 13. below.

The number of handbook page views, over the five year period, initially declined and recently increased. It is unclear why this is the case. The data can be seen in Figure 14. below.

The number of appeal toolkit downloads increased in between 2015-2017, and have declined since. It is unclear why this is the case however it is possible that users are finding the appeal toolkit of less use or less necessary. This could be explored further by CPAG in Scotland. The data can be seen in Figure 15. below.

3.4 Annual Welfare Rights Conference

From 2014 – 2018 there were 1138 attendees at CPAG in Scotland's annual Welfare Rights Conferences. Figure 16. Below, shows the distribution for each year.

The table below shows the percentages of respondents to the post-conference evaluation survey that felt that conference was poor/fair or good/excellent.

Figure 17. Rating of annual Welfare Rights Conference, 2014-18

How would you rate the conference overall
Year	Poor/Fair	Good/Excellent
2014	8%	92%
2015	2%	98%
2016	2%	98%
2017	4%	96%
2018	2%	98%

Overall, between 2014 – 2018 the vast majority (96%) of respondents rated the annual Welfare Rights Conferences as either good or excellent.

3.5 Summary of Findings from Secondary Data Analysis

The principal findings from the analysis of the secondary data show that there has been an increase in the call volume to CPAG in Scotland's advice helpline, largely driven by enquiries about Universal Credit. This coincided with an increase in the number of people attending training courses on Universal Credit which, although only introduced in 2017/18, has been the most widely attended training in the years since. The most frequent users of the advice helpline have been Citizens Advice Bureaux (CABs). Most organisations accessing the advice line make just one enquiry. Call volume appears to stay steady across the year suggesting that demand remains consistent.

Users of the advice line are overwhelmingly supportive of it with the only reported negative being that there has been increasing difficulty, year by year, in accessing it. This could be due to demand increasing at a rate that outpaces resource allocation. Opinions of the E-learning courses on offer and the welfare rights conferences have also been particularly positive however there were concerns raised about the cost and limited geographical locations of the conferences.

There has been an increase in the overall number of factsheets downloaded, however, this may coincide with greater use of computers and electronic devices amongst frontline advisers since 2014/15. The use of e-bulletins remained fairly constant until the introduction of GDPR regulations, which resulted in a sharp decline. The number of Appeals Toolkit downloads increased between 2014/2015 and 2016/2017, but have declined since. It is unclear why this is the case, however, it is possible that users are finding the Appeals Toolkit less useful. The number of handbook page views saw a steady decrease from 2014/15 to 2016/17 and then an increase back to relatively similar numbers as 2014/15 without any clear cause.