We are testing a new beta website for gov.scot go to new site

Archive

This is an archived section of the Scottish Government website. External links, forms and search may not work on archived pages and content/contact details are likely to be out of date.

Heading

This page relates to the 2007 version of the National Performance Framework. Information about the current version of the NPF is available on the Scotland Performs Home Page.

Methodology

10 GUIDING PRINCIPLES AND METHODOLOGY

FOR ASSESSING AND REPORTING PROGRESS ON SCOTLAND PERFORMS

What is this web page about?

The Scotland Performs website measures and reports progress on Government in Scotland's priorities, set out in the National Performance Framework. This webpage sets out the 10 guiding principles for assessing and reporting progress on Scotland Performs. It also explains, through a series of Frequently Asked Questions, how these principles are applied in practice to the analytical and technical considerations required to enable progress to be measured and reported.

SCOTLAND PERFORMS - 10 GUIDING PRINCIPLES
  • Openness and transparency
  • Accountability and responsibility
  • Objectivity
  • Independent assessment
  • Dynamic site: real data, real time
  • Accessibility 24/7
  • Simplicity and clarity
  • Credibility to Parliament and the wider public
  • Shared responsibility for outcomes-based performance (with our partners)
  • Sharpening focus - driving improvement
ANALYTICAL AND TECHNICAL ISSUES - FREQUENTLY ASKED QUESTIONS

Category - Reporting Progress

What progress is reported on Scotland Performs?

How does Scotland Performs report on performance?

How often is Scotland Performs actually updated?

Category - Independent Assessment

How independent is the assessment of progress in Scotland Performs?

Are Scottish Government Ministers involved in the assessment of progress in Scotland Performs?

Category - Robustness of measurement issues

How do you ensure the robustness of the measurement issues?

How is the direction of travel performance arrow determined?

Category - Technical issues

How is the decision on the evaluation of the performance arrow made?

What time period do the performance arrows refer?

What are baselines?

Category - Further Information

Where can I find more information?

Category - Reporting Progress

What progress is reported on Scotland Performs?

Progress against the 45 National Indicators and 7 Purpose Targets set out in the National Performance Framework is tracked and reported through Scotland Performs. The website reports on how we are doing at a national level.

How does Scotland Performs report on performance?

It does this by

¨ providing a graph that shows the long term trend in the figures for each indicator and purpose target (where appropriate). Additional graphs are used in some cases to help the interpretability of more complex measures;

¨ adding narratives to the 'Current status' and 'How are we performing' sections on the relevant indicator or target pages which contain further analysis on the recent change in performance, the longer trend and the change since the baseline. They might also include - to some extent - assessment of delivery, where appropriate;

¨ assigning a direction of travel performance arrow, to each National Indicator and Purpose Target, which aim to evaluate the direction of change between the two most recent comparable data points. Four types of symbols are used:

- Performance Improving

- Performance Maintaining

- Performance Worsening

- Data being collected (for the ones where there are no two comparable data points available).

How often is Scotland Performs actually updated?

Scotland Performs is up-dated whenever relevant new statistical information is available to indicate our progress against the associated purpose target or indicator. It may not change every day, but it changes as and when new information becomes available to update the purpose targets and indicators. See Calendar of Updates on Scotland Performs for details of when updates are due.

For the vast majority of Indicators and Purpose Targets, up-dates are made as and when new information becomes available and in some cases updates are made within a few days - the soonest technically possible.

Category - Independent Assessment

How independent is the assessment of progress in Scotland Performs?

Performance is evaluated by a group of chief professional officials and senior analysts - Scotland Performs Technical Assessment Group - which provides an objective and impartial forum to consider methodological and technical issues relating to Scotland Performs.

The group has the final decision on the following:

  • measures and their methodology used for each Purpose Target and National Indicator;
  • performance arrows and their methodology;
  • narratives ('How are we performing' section and 'Current status' text);
  • presentation of data in the graph and the attached spreadsheet;
  • content of the Technical Notes.

The Group's decision is informed by analysis coming from Scottish Government Analysts in the relevant policy area.

Are Scottish Government Ministers involved in the assessment of progress in Scotland Performs?

No. To ensure that assessments are made on an objective and professional basis, the Scotland Performs Technical Assessment Group's decisions are kept wholly independent of Ministers and although Ministers are advised a few days in advance of impending changes, up-dates are never subject to Ministerial approval.

Category - Robustness of measurement issues

How do you ensure the robustness of the measurement issues?

Technical Notes, which can be accessed from Scotland Performs, set out the detail of how each of the 7 Purpose Targets and the 45 National Indicators will be measured. The two main aims of these Notes are to provide sufficient public accountability on the robustness of the measurement issues, and to ensure a shared understanding of the details of the indicators and targets among all of the delivery partners. They provide fuller details on the specific methodology and on the sources of data used.

How is the direction of travel performance arrow determined?

The criteria by which assessments are made about the direction of travel arrows for each Purpose Target and National Indicator is available on the Scotland Performs website.

Category - technical issues

How is the decision on the evaluation of the performance arrow made?

Each time a new figure for a National Indicator or Purpose Target becomes available Scotland Performs Technical Assessment Group compares it to the previous data point and calculates the recent change to be able to assign a performance arrow to it. The arrow decision is based on the specific arrow methodology used for that measure.

Each arrow methodology is decided by Scotland Performs Technical Assessment Group in advance of the update and published on the website, which adds further guarantees to an objective and impartial decision making. (Exceptions are only where an arrow is attached to a measure for the first time or in the rare case of amending the arrow methodology during the update process for a well-established technical reason.)

The vast majority of the arrow methodologies consists of symmetrical thresholds for the level of change that is 'expected' for a 'Performance Improving' or a 'Performance Worsening' arrow - if the recent change in the figures is less than the value of this threshold, a 'Performance Maintaining' arrow is assigned.

In general the following aspects are considered when selecting the threshold for a measure:

  • variance in the data;
  • medium to long-term trend;
  • the change required by the target;
  • the scale or the impact of the change;
  • how close the figure is to the maximum/ minimum value (if appropriate) - i.e. how much scope there is for changes in the figure.

This means that the thresholds are NOT necessarily about the level of change that would be considered statistically significant. This is partly because there are more factors to be taken into account when judging such complex matters - as mentioned above - and partly because a number of National Indicators and Purpose Targets are based on sources for which variance estimates are not available, as they come from administrative data.

What time period do the performance arrows refer?

The time period to which each performance arrow refers has been made clear in the narratives on the relevant page. These time periods vary due to individual differences between the measures in the following aspects:

  • data availability;
  • update frequencies;
  • methods for collecting, processing and quality assuring data.

Therefore in a few cases there are relatively big time lags between present and the period that the latest data point refers to, although every effort is made to reduce these as far as possible.

However, the evaluation of the recent change is based on the most up-to-date figures available at any given point of time - given the selected update frequency for each measure.

What are baselines?

A baseline data point for each National Indicator and Purpose Target has been selected. Baselines are used for comparison over a number of years to show the overall change in performance. The narrative in the 'Current Status' text usually contains summary information on change since the baseline.

Category - Further Information

Where can I find more information?

Technical Notes for the detailed methodology, sources and definitions used for the individual National Indicators and Purpose Targets

Performance at a glance on Scotland Performs for the most recent data and performance

Scottish Neighbourhood Statistics website for related local level data (where appropriate)

High Level Summary Statistics pages for further background analysis in certain topics of Scotland Performs

Scotland Performs team in the Office of the Chief Statistician if you have any queries or comments