The Hunt Lab has partnered with the Defence Science Institute (DSI) and other domain experts from the Australian Intelligence Community to develop an Advanced Analytic Rigour training package based on the Reasoning Stress Test (RST) method.
Analytic Rigour
The course is designed for current and aspiring leaders who want to guide their teams on how to best enhance the rigour of their products and the logical justifications for their judgements. Analytic rigour is essential for well-reasoned, trustworthy and high-impact analytic products. But evaluating rigour is often neglected and can be very challenging if you don’t have an effective approach. In this course we teach a method of reviewing rigour called the Reasoning Stress Test, which was developed with funding from the US intelligence community.
Stress-Test Your Reasoning
The RST is a method for efficiently evaluating the quality of reasoning in analysis. It helps reviewers identify critical flaws in analysis and provide precise and practical feedback with specific improvements.
The method consists of 4 steps:
Identify the intended structure of the reasoning used to justify assessments, assisted by a detailed list of the main type of reasoning schemes (e.g., inference to best explanation)
Locate and describe potential reasoning flaws using a Taxonomy of reasoning flaws arranged into categories and akin to an expert’s mental model
Determine how the identified flaws impact the assessment, enabling the prioritisation of flaws that should be fixed first
Using the information developed in Steps 1–3, explain how and where the reasoning goes wrong and how to adjust it to enable it to succeed.
The Hunt Lab is currently completing research to validate this method against other existing approaches to reasoning evaluation.
The RST approach scaffolds the peer-review process without constraining it. Raters draw upon elements of the method to suit the particular circumstances. In collaboration with practitioners, the method is designed to be:
- Flexible
- Practical
- Precise
- Respectful of analysts’ expertise
Train in Advanced Analytic Rigour
The Advanced Analytic Rigour training can be run either online and in-person.
The sessions are interactive, engaging and practice-based. Participants develop their reasoning skills and familiarity with the RST method through extensive practice and discussion on both real-world argumentation and analysis and carefully constructed examples that progressively build an advanced understanding of and proficiency with the required concepts.
Training Outline
-
Weeks 1–3: Fundamentals of good reasoning
- The parts of arguments and fundamental concepts
- Argument structure: surface-level and underlying reasoning
- Identifying reasoning in natural language
- Analysing complex arguments
- Important argument stratagems
- Weeks 4–5: Reasoning flaws
- Understanding reasoning flaws
- Identifying reasoning flaws in argumentation
- Taxonomy of reasoning flaws
- Weeks 6–7: Flaw Impact
- Impact and argument structure
- Understanding the impact of flaws on an overall argument
- Assessing flaw impact
- Weeks 8–9: Fixing Flaws
- Determining appropriate approach to fixing flaws
- Strategies for fixing flaws and recommending changes
- Weeks 10–11: Providing Feedback
- Turning an RST evaluation into concrete feedback
- Giving effective feedback and recommendations
- Week 12: Consolidation
Developed by Experts
Ashley BarnettResearch Associate, Hunt Lab
Ashley Barnett is a PhD candidate at the University of Melbourne writing on the evaluation of reasoning in intelligence analysis, with a focus on the Reasoning Stress Test which he formulated for measuring the quality of reasoning in reports. He has previously researched how to teach argumentation to intelligence analysts, which involved developing and delivering training courses to professional analysts, and has developed questions for a critical thinking skills test for the intelligence community. All these areas of expertise have gone into designing the materials and challenging questions that form part of this training.
Tamar PrimoratzResearch Associate, Hunt Lab
Tamar Primoratz has a Masters in Publishing and Communications and is currently completing a Masters in Philosophy from the University of Melbourne. During her work at the Hunt Lab she has overseen the evaluation of the reasoning process, based on the ODNI’s rating scale, for the several of the Hunt Lab’s collective reasoning studies, and has developed validation studies on quality of reasoning metrics, and training materials for intelligence analysts.
Dr Tim van GelderDirector, Hunt Lab
Dr Tim van Gelder is the Director of the Hunt Laboratory for Intelligence Research in the School of BioSciences, University of Melbourne. He is an applied epistemologist with a background in philosophy and cognitive science. His primary areas of expertise are reasoning and collective intelligence. He has published widely, including in Behavioral and Brain Sciences, Cognitive Science, Journal of Philosophy and Journal of Public Deliberation. Prior to joining the University he had extensive experience assisting large organisations with analytic capability development, and expert elicitation.
What People are Saying
Ashley Barnett's other courses
Contact
Contact us to learn more or to book Hunt Lab experts to run this course at your organisation. We are also open to discussing collaborating on developing a tailored course to suit your organisation’s needs.