The Hunt Lab has been developing a comprehensive training package on advanced analytic review for intelligence analysts. Analytic review is the process by which members of the organisation assess the analytic quality of intelligence products – draft or final – and provide effective feedback to authors.
As part of the CREATE project, the Hunt Lab developed a new method (the “Reasoning Stress Test”, or RST) for assessing the quality of analytic reasoning in intelligence reports. This method is currently being validated as part of a study comparing it to existing best practice outcomes (you can read more about, and participate in, this study here).
The RST method takes a new approach to evaluating analytic reasoning in intelligence by focusing on problems, or “flaws”, in a text’s reasoning, and evaluating their impact on the text’s overall strength. This is in place of the currently accepted approach of judging the extent to which a text exemplifies certain features (such as “logical argumentation”) overall. The new approach is based on the idea that an argument is only ever as good as its most serious flaw.
In this project, the Hunt Lab is collaborating with domain experts to tailor and adapt the method for the workplace. The current, first phase of this project, supported in part by the Defence Science Institute (DSI), focuses on creating a first version of resources to support the method, including a manual, taxonomy of flaws, and all relevant training materials.
The training is a three-day, in-person training course covering both theoretical and practice-based learning of the RST method, the conceptual framework required for in-depth understanding of advanced analytic reasoning, and methods for effective review and feedback of an RST-based product evaluation.
More broadly, the project aims to operationalise the method based on further research and increased collaboration with the Australian intelligence community, adapt the method fully to the work of intelligence analysts and practitioners, and make these resources available to organisations across the Australian intelligence community. The medium and long term intended impact of the project is improved performance in assessing and improving the quality of intelligence reports, leading to better intelligence and ultimately to better decision making.
You can read more about the Hunt Lab’s tailored training offering on advanced analytic rigour here, or find out about other Hunt Lab activities by following the links below: