Structured Analytic Methods
Now that we have plenty of time for reflection I think the weakest part of Carbon was the lack of exposition. Very few participants articulated strong reasons for their forecasts and we didn’t have many alternative hypotheses / models to consider. This made it difficult to weight whatever evidence we were able to track down.
One of the most interesting papers I’ve read recently is Stephen J. Coulthart’s ‘An Evidence-Based Evaluation of 12 Core Structured Analytic Techniques’. He derives three evidence-based principles from the publsihed data:
1. Two heads are not always better than one (face-to-face collaboration reduces idea-generation and creativity).
2. Weight and update (an important determinant of analytical accuracy is the proper weighting of evidence).
3. Careful conflict (conflict-inducing techniques, such as devil’s advocacy, can be constructive in intelligence analysis but only if carefully implemented).
Coulthart found that devil’s advocacy was “the technique with the most credible evidence base and highest efficacy”. This was also the thing that was most lacking on Carbon. Hopefully the next iteration of the HFC will (a) do more to encourage participation and (b) do more to facilitate polite debate about alternative hypotheses and different ways of modelling the IFPs.
Paper: https://www.tandfonline.com/doi/abs/10.1080/08850607.2016.1230706
One of the most interesting papers I’ve read recently is Stephen J. Coulthart’s ‘An Evidence-Based Evaluation of 12 Core Structured Analytic Techniques’. He derives three evidence-based principles from the publsihed data:
1. Two heads are not always better than one (face-to-face collaboration reduces idea-generation and creativity).
2. Weight and update (an important determinant of analytical accuracy is the proper weighting of evidence).
3. Careful conflict (conflict-inducing techniques, such as devil’s advocacy, can be constructive in intelligence analysis but only if carefully implemented).
Coulthart found that devil’s advocacy was “the technique with the most credible evidence base and highest efficacy”. This was also the thing that was most lacking on Carbon. Hopefully the next iteration of the HFC will (a) do more to encourage participation and (b) do more to facilitate polite debate about alternative hypotheses and different ways of modelling the IFPs.
Paper: https://www.tandfonline.com/doi/abs/10.1080/08850607.2016.1230706
