Скачать или смотреть CECAN Webinar: Case-Based Modelling and Scenario Simulation for Ex-Post Evaluation

CECAN Webinar: Case-Based Modelling and Scenario Simulation for Ex-Post Evaluation

Presenters: Dr Corey Schimpf, Dr Brian Castellani and Dr Pete Barbrook-Johnson

Webinar Overview:
This seminar introduces case-based scenario simulation (CBSS) to evaluators. CBSS is well aligned with existing evaluation approaches, most directly realist evaluation which emphasizes ‘what works for whom in which circumstances’. CBSS’s approach enables users to explore in a simplified simulation environment the possibility of driving different clusters of outcomes in a desired evaluation direction, as well as explore counterfactuals. CBSS starts with the notion that interventions or policies may themselves not always be complex in nature, but that such interventions always happen in complex systems and that policy evaluators and analysts are in need of accessible tools or methods for studying how interventions unfold in these systems. In light of this challenge, we introduce a new approach that integrates two key methodologies for modeling and studying complexity: case-based methods and scenario simulation. The integrated approach of CBSS enables the identification and modeling of underlying groups of cases within a study and simulating hypothetical or ‘what-if’ scenarios to further explore how the system under study may respond to additional or alternative interventions. CBSS achieves this by modelling unique case types, their trajectories and the heterogeneous effects interventions may have on them. This work is grounded in a pluralist view of methodology, meaning that CBSS is not meant to replace other evaluation methods but rather to complement, support and extend them.

More concretely, for case-based modelling CBSS uses k-means and the self-organizing map, a type of neural network, together to uncover underlying case groups within the data so that the unique profiles of case groups and their dynamics or responses can be modelled. The scenario simulation part of CBSS maps the cases to an n x m grid representing different configurational possibilities for the case groups and allows the analyst to make targeted interventions on case groups, representing ‘what if’ or counterfactual scenarios, to explore if this improves, worsens, or alters case groups trajectories. We briefly present COMPLEX-IT, which is a lightweight open-source web application that supports the use of CBSS regardless of an analyst’s or researcher’s experience with these methods. In this presentation we discuss CBSS in relation to evaluation purpose (learning or accountability) and evaluation design (cross-sectional or longitudinal). From these dimensions we highlight four evaluation situations as combinations of each dimension and identify four documented evaluation cases that map to each situation. We discuss how CBSS could complement and extend evaluations in each situation, noting the ways in which CBSS can help attend to differences within cases, alternative scenarios, and attention to complex dynamics operative in the systems under study. We conclude with a discussion of the possibilities CBSS opens up for different types of evaluation as well as the commonalities across these situations and the other methodological implications it has for evaluation research.