AI performance evaluation and system impact: FOI release

Information requested

Please can Social Security Scotland provide:

Your request for information 1: Any internal evaluation reports or assessments of the AI Document Understanding and Client Advisor Knowledge Assist systems, including performance metrics like accuracy rates, staff time saved or cost savings, and impacts on service delivery or claimant outcomes.

Your request for information 2: Any targets or benchmarks used to assess the success of these systems during trials or in production.

Your request for information 3: Documents or emails discussing changes made to the system after deployment based on performance issues.

Please provide this information from the date of the system's pilot to the present.

Response

AI Document Understanding System

Response to your request for information 1

We have provided the Intelligent Automation pilot assessment report and the AI transparency report document relating to the AI document understanding system. Information outwith the scope of your request has been redacted.

Response to your request for information 2

While our aim is to provide information whenever possible, in this instance Social Security Scotland does not have the information you have requested. This is because no formal targets or benchmarks were used to assess the success of the AI document understanding system specifically.

This is a formal notice under section 17(1) of FOISA that Social Security Scotland does not have the information you have requested.

However, the AI document understanding tool is part of a wider intelligent automation project as explained in the attached document number 4.

The AI document understanding tool is a digital reader that uses optical character recognition (OCR) technology. We use the OCR technology to scrape information from PDF application forms which helps reduce processing times.

We have provided the Intelligent Automation pilot assessment report and the AI transparency report registration document. Both specify some of the potential benefits this technology may deliver for Social Security Scotland.

An exemption under section 38(1)(b) of FOISA (personal information) applies to a small amount of the information requested because it is personal data of a third party, e.g. names/contact details of individuals, and disclosing it would contravene the data protection principles in Article 5(1) of the General Data Protection Regulation and in section 34(1) of the Data Protection Act 2018. This exemption is not subject to the ‘public interest test’, so we are not required to consider if the public interest in disclosing the information outweighs the public interest in applying the exemption.

Response to your request for information 3

While our aim is to provide information whenever possible, in this instance Social Security Scotland does not have the information you have requested. This is because no changes were made to the system after deployment based on performance issues.

This is a formal notice under section 17(1) of FOISA that Social Security Scotland does not have the information you have requested.

Client Advisor Knowledge Assist System

Response to your request for information 1 - 3

While our aim is to provide information whenever possible, in this instance Social Security Scotland does not have some of the information you have requested.

This is a formal notice under section 17(1) of FOISA that Social Security Scotland does not have the information you have requested.

However, we have included the Client Advisor Knowledge Assist System proof of concept report as this may be considered an internal evaluation report

An exemption under section 38(1)(b) of FOISA (personal information) applies to a small amount of the information requested because it is personal data of a third party, e.g. names/contact details of individuals, and disclosing it would contravene the data protection principles in Article 5(1) of the General Data Protection Regulation and in section 34(1) of the Data Protection Act 2018. This exemption is not subject to the ‘public interest test’, so we are not required to consider if the public interest in disclosing the information outweighs the public interest in applying the exemption. The Client Advisor Knowledge Assist System was a proof of concept that was not taken further. Therefore, questions 2 and 3 cannot be answered as they are not applicable in this instance.

About FOI

The Scottish Government is committed to publishing all information released in response to Freedom of Information requests. View all FOI responses at https://www.gov.scot/foi-responses.

FOI 202500478412 - Information released - Documents

File type

PDF

File size

1.4 MB