Advanced Analytic Rigour Training

The Hunt Lab has partnered with the Defence Science Institute (DSI) and other domain experts from the Australian Intelligence Community to develop an Advanced Analytic Rigour training package based on the Reasoning Stress Test (RST) method.

Analytic Rigour

The course is designed for current and aspiring leaders who want to guide their teams on how to best enhance the rigour of their products and the logical justifications for their judgements. Analytic rigour is essential for well-reasoned, trustworthy and high-impact analytic products. But evaluating rigour is often neglected and can be very challenging if you don’t have an effective approach. In this course we teach a method of reviewing rigour called the Reasoning Stress Test, which was developed with funding from the US intelligence community.

Stress-Test Your Reasoning

The RST is a method for efficiently evaluating the quality of reasoning in analysis. It helps reviewers identify critical flaws in analysis and provide precise and practical feedback with specific improvements.

The method consists of 4 steps:

Step 1: Review Reasoning
Identify the intended structure of the reasoning used to justify assessments, assisted by a detailed list of the main type of reasoning schemes (e.g., inference to best explanation)
Step 2: Identify Flaws
Locate and describe potential reasoning flaws using a Taxonomy of reasoning flaws arranged into categories and akin to an expert’s mental model
Step 3: Evaluate Impact
Determine how the identified flaws impact the assessment, enabling the prioritisation of flaws that should be fixed first
Step 4: Fix Flaws
Using the information developed in Steps 1–3, explain how and where the reasoning goes wrong and how to adjust it to enable it to succeed.

The Hunt Lab is currently completing research to validate this method against other existing approaches to reasoning evaluation.

The RST approach scaffolds the peer-review process without constraining it. Raters draw upon elements of the method to suit the particular circumstances. In collaboration with practitioners, the method is designed to be:

  • Flexible
  • Practical
  • Precise
  • Respectful of analysts’ expertise

Train in Advanced Analytic Rigour

The Advanced Analytic Rigour training can be run either online and in-person.

The sessions are interactive, engaging and practice-based. Participants develop their reasoning skills and familiarity with the RST method through extensive practice and discussion on both real-world argumentation and analysis and carefully constructed examples that progressively build an advanced understanding of and proficiency with the required concepts.

Training Outline

    Weeks 1–3: Fundamentals of good reasoning

  • The parts of arguments and fundamental concepts
  • Argument structure: surface-level and underlying reasoning
  • Identifying reasoning in natural language
  • Analysing complex arguments
  • Important argument stratagems
    Weeks 4–5: Reasoning flaws

  • Understanding reasoning flaws
  • Identifying reasoning flaws in argumentation
  • Taxonomy of reasoning flaws
    Weeks 6–7: Flaw Impact

  • Impact and argument structure
  • Understanding the impact of flaws on an overall argument
  • Assessing flaw impact
    Weeks 8–9: Fixing Flaws

  • Determining appropriate approach to fixing flaws
  • Strategies for fixing flaws and recommending changes
    Weeks 10–11: Providing Feedback

  • Turning an RST evaluation into concrete feedback
  • Giving effective feedback and recommendations
    Week 12: Consolidation

Developed by Experts

Ashley Barnett
Research Associate, Hunt Lab
Ashley Barnett is a PhD candidate at the University of Melbourne writing on the evaluation of reasoning in intelligence analysis, with a focus on the Reasoning Stress Test which he formulated for measuring the quality of reasoning in reports. He has previously researched how to teach argumentation to intelligence analysts, which involved developing and delivering training courses to professional analysts, and has developed questions for a critical thinking skills test for the intelligence community. All these areas of expertise have gone into designing the materials and challenging questions that form part of this training.
Tamar Primoratz
Research Associate, Hunt Lab
Tamar Primoratz has a Masters in Publishing and Communications and is currently completing a Masters in Philosophy from the University of Melbourne. During her work at the Hunt Lab she has overseen the evaluation of the reasoning process, based on the ODNI’s rating scale, for the several of the Hunt Lab’s collective reasoning studies, and has developed validation studies on quality of reasoning metrics, and training materials for intelligence analysts.
Dr Tim van Gelder
Director, Hunt Lab
Dr Tim van Gelder is the Director of the Hunt Laboratory for Intelligence Research in the School of BioSciences, University of Melbourne. He is an applied epistemologist with a background in philosophy and cognitive science. His primary areas of expertise are reasoning and collective intelligence. He has published widely, including in Behavioral and Brain Sciences, Cognitive Science, Journal of Philosophy and Journal of Public Deliberation. Prior to joining the University he had extensive experience assisting large organisations with analytic capability development, and expert elicitation.

What People are Saying

“The course had just the right amount of everything: complexity, homework, new things I had not heard about before but enough to also hang on to things that were familiar, the right amount of variety in subject matter – examples, exercises, the combination between theory and practice.”
“The training helped me look at topics with a different lens.”
“As a collaborative tool it’s very powerful.”
“I can’t begin to tell you how much I’m loving the course, and how grateful I am to be included. It’s giving me so much to think through, and rethink.”

Contact

Contact us to learn more or to book Hunt Lab experts to run this course at your organisation. We are also open to discussing collaborating on developing a tailored course to suit your organisation’s needs.