TEC programme data review and evaluation: summary report

Methodology

The study was carried out in seven stages between September 2017 and March 2018. This paper is a brief overview of the findings with a focus on recommendations and evaluation options. The full report provides more detail on the approach taken and description of how the synthesis was carried out.

The research stages were as follows:

Stage 1: Develop logic models. A series of logic models were developed using primary and secondary research. The purpose was to systematically identify outcomes that could be used to assess the strength of the evidence base in each area of work.

Stage 2: Evidence classification. A total of 53 evaluation reports were received and catalogued according to evaluation type.

Stage 3: Data Review. Quantitative data was extracted and key variables were inputted into an Excel spreadsheet. Qualitative data was analysed using NVivo.

Stage 4: Comparing outcomes with data. Matrices were constructed to indicate which elements of the logic model were supported by data, the type of data (qualitative/quantitative/mixed) and providing an assessment of its quality.

Stage 5: Economic data review. Ten evaluation studies containing economic data were systematically analysed and data was extracted on key variables.

Stage 6: Implementation research. This comprised of two workshops with frontline and programme staff to understand barriers and enablers. This was supported by a secondary literature review on implementation science in TEC and a review of primary data gathered via the existing evaluations.

Stage 7: Measurement framework. Finally, a measurement framework and options for future evaluation were developed. This was informed by the previous evaluation activities and some additional literature searches.