Fireworks legislation and impacts: international evidence review

Desk-based review of evidence on the impact of fireworks, in the context of international legislation and regulations.


Appendix B: Methods

To conduct this review, a systematic process of search and assessment was followed, involving four broad stages:

1. Evidence search

2. Application of inclusion and exclusion criteria for assessing relevance

3. Quality assessment of studies

4. Synthesis of the body of evidence

The details of stages 1-3 are described below. The synthesis of evidence is presented in Section 4.

6.2.1. Search

In the first instance, the search for studies was carried out by the Scottish Government Library Service using KandE. KandE is an online search engine which covers a range of high quality databases, which are detailed below.

Table 3: List of databases searched

Search Engines

Academic Search Ultimate (asn)
AGRIS (edsagr)
Australian Research Data Commons (edsard)
BioOne Complete (edsbio)
Bloomsbury Collections (edsblc)
British Standards Online (edsbsi)
Business Source Index (bsx)
Cochrane Database of Systematic Reviews (edschh)
Credo Reference (edscrc)
Credo Reference: Academic Core (edscra)
Criminal Justice Abstracts with Full Text (i3h)
DigitalNZ (edsdnz)
Emerald Insight (edsemr)
ERIC (eric)
FT.com (edsfit)
GreenFILE (8gh)
Military & Government Collection (mth)
New Palgrave Dictionary of Economics Online (edsdeo)
Oxfam Policy & Practice (edsoxf)
Oxford Bibliographies (edsobb)
Oxford Dictionary of National Biography (edsodb)
Oxford Reference (edsoro)
Oxford's Who's Who & Who Was Who (edsoww)
Political Science Complete (poh)
Public Information Online (edspio)
RePEc (edsrep)
SAGE Knowledge (edsskl)
SAGE Research Methods (edsrem)
ScienceDirect (edselp)
Sociology Source Ultimate (sxi)

Journals

Directory of Open Access Journals (edsdoj)
JSTOR Journals (edsjsr)

Books

Books at JSTOR (edsjbk)
eBook Collection (EBSCOhost) (nlebk)

Library Services

Biodiversity Heritage Library (edsbhl)
British Library Document Supply Centre Inside Serials & Conference Proceedings (edsbl)
British Library EThOS (edsble)
Canadian Electronic Library (edscel)
E-LIS (Eprints in Library & Information Science) (edseli)
Library, Information Science & Technology Abstracts (lxh)

Archives

Archive of European Integration (edsupe)

This search was informed by a range of key words and phrases, including 'fireworks' combined with:

  • Sale
  • Licencing
  • Impact
  • Use / misuse
  • Anti-social behaviour
  • Crime
  • Noise
  • Air quality / environmental
  • Animal welfare
  • Regulations / Legislation

To ensure the evidence identified was up-to-date and relevant, the specified time coverage for the search was from 2009-2019. The geographical coverage included the UK, Europe, North America, Canada, Australia and New Zealand. These countries were selected because they are comparable to Scotland in terms of culture, attitudes to health and safety and air quality.

A series of broader searches were then conducted using Google and Google Scholar, as a sweep of studies that may not have been found in the initial search. In addition, a snowballing technique was employed whereby the references of studies were reviewed for additional evidence.

6.2.2. Inclusion and exclusion criteria

Using the initial search results, the relevance of the studies was assessed. The table below provides a summary of the inclusion and exclusion criteria applied to the selection of the studies.

Table 4: Inclusion and exclusion criteria

Inclusion criteria

Exclusion criteria

Study design

Primary empirical research (qualitative or quantitative), evaluation or secondary reviews

Primarily theoretical or conceptual in nature, lacking empirical evidence or explanation of methodology

Language

Written or available in English

Not written or available in English

Country

UK, Europe, North America, Australia and New Zealand, International

Asia[4], Africa, South America

Publication date

From 2009 to 2019

Pre-2009[5]

Publication format

Journal articles, peer-reviewed materials, working papers, evaluation, government reports, discussion papers, books and book chapters, other academic research

Student paper, dissertation, conference paper, news articles without clear indication of source

Aim of study

Studies exploring key issues around fireworks, including misuse/ASB, injury, noise, pollution, impact on vulnerable groups, animal welfare

Studies exploring other issues relating to fireworks, e.g. technical elements

Applying these criteria led to an evidence base comprising a wide range of sources, including academic journal articles, government reports, surveys, case studies, laboratory experiments, evaluations, evidence reviews, interviews and books.

25 of these studies were based in the UK; 14 in the US; 14 in European countries including Northern Ireland, Hungary, Italy, Malta, Netherlands, Norway, Poland, Portugal and Switzerland; 6 in Australia and New Zealand; 4 internationally and 3 from Asian countries including China, India and Japan. For one study identified, the country of origin was unclear.

6.2.3. Quality assessment

Each of the studies identified was then quality assessed. This involved identifying the key characteristics of the studies and their limitations, which are summarised in Appendix A.

The body of evidence identified in this report consists of 67 studies, many of which used high quality methods. In particular, 28 used quantitative methods such as surveys or analysis of injuries data, including 5 which used nationally representative data. There were also 10 studies based on case studies and/or qualitative methods such as interviews, providing a more in-depth insight into fireworks use.

However, as well as the limitations highlighted in Appendix A, the evidence base suffered from other shortcomings. In particular, there was a distinct lack of evidence based in Scotland or even the wider UK, and it is unclear how findings will apply to the Scottish context. For example, environmental impacts are found to be influenced by a range of factors that vary from one country to the next and so findings from other countries may not apply to Scotland. Further, there was a lack of literature on several themes present in the consultation and omnibus survey, including underage sales and anti-social behaviour.

Contact

Email: Socialresearch@gov.scot

Back to top