Cancer strategy - monitoring and evaluation: participatory systems mapping

Caveats and limitations

There are some general aspects of the system map which it is useful to be aware of. For example, there is a factor in the systems map “Patients living longer” which has a tendency to moderate any positive feedback loops by putting negative pressure on space and resource constraints. This has quite a strong impact in the system map, but does not take into consideration the timescales of different changes. However it does highlight that participants held the view that the current space and resource is not well-equipped for what the future might bring.

While a breadth of perspectives were sought when building the system map, the fact remains that they still represent only a snapshot of the system being studied. The system map is true and correct in the sense that it captures the perspectives of the group of participants at the point in time at which they were shared. The analysis considers only healthcare professionals’ views of what impacts patient experiences, not patients themselves. Opinions and perspectives are subject to change and nuance which cannot be captured using these methods.

The system map is a tool for understanding potential interactions in a complex system and stimulating discussions. It is not a predictive model.

The system map is open to interpretation. Every effort has been made to capture within the map why participants thought a certain causal relationship should exist, however this was not necessarily feasible in all cases. As such users may disagree with some casual relationships, or may see some alternate casual relationships that would change the dynamics of the system map.

All of the positive and negative links are equally weighted. This means that there is no consideration for the magnitude of the casual relationship. If a factor is increased/decreased the visualisation tool propagates this through the system, if there are multiple paths this change can be magnified or minimised. However, as there are no magnitudes represented in the map, the change visualised is highly unlikely to be representative of impact in reality, and is more of a measure of the connectedness. Say there were two paths of the same length from factor A to factor B, one is a chain of positive links, while the other is a chain of negative links. As the paths are the same length, and all magnitudes are assumed equal, increasing factor A would have no impact on factor B as the positive and negative chains of influence would cancel each other out. However it could be that in reality the negative causal relationships have a lower impact, and so an increase in factor A could in fact increase factor B. This is partly why this tool is purely for visualisation and discussion prompting purposes, and not a predictive model.

The aim was to capture the elements of the system that most directly influence the experiences and journeys of people impacted by cancer in Scotland. This means that some of the factors and links are indicative of wider NHS Scotland issues/pressures e.g. IT system; whilst others are more specific to the cancer pathways e.g. diagnostic capacity. The scope of a factor is therefore important to consider when interpreting the map and the possible impact of changes. As an example, a new and efficient IT system for something cancer specific (e.g. diagnosis) may have some locally positive changes. Increasing this factor in the system map assumes that there are improvements to the IT system across NHS Scotland, not just in a small pocket of the system, as such some of the impacts suggested by the visualisation tool would not be correct if only a component of the IT system was improved.

While the map is representing the overall cancer healthcare system, it could also be interpreted on a cancer-specific level. It is assumed that a lot of the factors and relationships would be present across different cancer pathways, although perhaps with different emphasis and impact. Some domain knowledge would be required to make judgements as to which factors and relationships do not apply if projecting this system map down to a specific cancer pathway.