Analytic Leadership: Evaluating Analytic Rigour

The Hunt Lab has partnered with the Defence Science Institute and experts from the Australian Intelligence Community to develop training on analytic rigour

Analytic Rigour

The course is designed for current and aspiring leaders who want to guide their teams on how to best enhance the rigour of their products and the logical justifications for their judgements. Analytic rigour is essential for well-reasoned, trustworthy and high-impact analytic products. But evaluating rigour is often neglected and can be very challenging if you don’t have an effective approach. In this course we teach a method of reviewing rigour called the Reasoning Stress Test (RST), which was developed with funding from the US intelligence community.

Stress-Test Your Reasoning

The RST is a method for efficiently evaluating the quality of reasoning in analysis. It helps reviewers identify critical flaws in analysis and provide precise and practical feedback with specific improvements.

The method consists of 4 steps:

Step 1: Review Reasoning
Identify the intended structure of the reasoning used to justify assessments, assisted by a detailed list of the main type of reasoning schemes (e.g., inference to best explanation)
Step 2: Identify Flaws
Locate and describe potential reasoning flaws using a Taxonomy of reasoning flaws arranged into categories and akin to an expert’s mental model
Step 3: Evaluate Impact
Determine how the identified flaws impact the assessment, enabling the prioritisation of flaws that should be fixed first
Step 4: Fix Flaws
Using the information developed in Steps 1–3, explain how and where the reasoning goes wrong and how to adjust it to enable it to succeed.

The RST approach scaffolds the peer-review process without constraining it. Reviewers draw upon elements of the method to suit the particular circumstances. In collaboration with practitioners, the method is designed to be:

  • Flexible
  • Practical
  • Precise
  • Respectful of analysts’ expertise

Train in Advanced Analytic Rigour

The Advanced Analytic Rigour training can be run either online and in-person.

The sessions are interactive, engaging and practice-based. Participants develop their reasoning skills and familiarity with the RST method through extensive practice and discussion on both real-world argumentation and analysis and carefully constructed examples that progressively build an advanced understanding of and proficiency with the required concepts.

Training Outline

    Part 1: Fundamentals of good reasoning

  • The latest research on evaluating and improving reasoning
  • How to identify the intended reasoning structure and unstated assumptions in a text
  • Common types of reasoning stratagems all analysts should know
    Part 2: Fixing reasoning flaws

  • How to identify reasoning flaws
  • How to assess the impact of reasoning flaws
  • How to efficiently modify assessments to remove gaps and fix errors
    Part 3: Effective feedback

  • Reasoning as a collaborative process
  • The pitfalls of providing feedback and the art of clear and effective feedback
  • How to provide useful and practical advice

Developed by Experts

Ashley Barnett
Research Associate, Hunt Lab
Ashley Barnett is a philosopher at the Hunt Laboratory for Intelligence Research. He has over 10 years’ experience developing and delivering critical thinking training for intelligence agencies around the world. As part of a research project for the Intelligence Advanced Research Projects Activity (IARPA), he formulated a method of reasoning evaluation called the Reasoning Stress Test. In collaboration with the Australian Intelligence Community, the Reasoning Stress Test has been developed into a training program for intelligence managers that is run multiple times a year. His current research focuses on deception and the nature of reasoning in intelligence analysis, especially how analysts compare alternative explanations.
Tamar Primoratz
Research Associate, Hunt Lab
Tamar Primoratz has a Masters in Publishing and Communications and is currently completing a Masters in Philosophy from the University of Melbourne. During her work at the Hunt Lab she has overseen the evaluation of the reasoning process, based on the ODNI’s rating scale, for the several of the Hunt Lab’s collective reasoning studies, and has developed validation studies on quality of reasoning metrics, and training materials for intelligence analysts.
Dr Tim van Gelder
Director, Hunt Lab
Dr Tim van Gelder is the Director of the Hunt Laboratory for Intelligence Research in the School of BioSciences, University of Melbourne. He is an applied epistemologist with a background in philosophy and cognitive science. His primary areas of expertise are reasoning and collective intelligence. He has published widely, including in Behavioral and Brain Sciences, Cognitive Science, Journal of Philosophy and Journal of Public Deliberation. Prior to joining the University he had extensive experience assisting large organisations with analytic capability development, and expert elicitation.

What People are Saying

“This course provides an excellent framework for those working with intelligence on a daily basis, and those responsible for peer reviewing or moderating their work, to ensure that what they produce on a day-to-day basis is valid, appropriately supported and can stand up to close scrutiny. The course is easily digestible and well run despite being on a dense and sometimes daunting topic. I have absolute confidence that my work and the work of my peers will benefit from my having attended.”
“Revelatory. Helped to completely reshape my way of thinking around analysis and feedback. We’re all told generally in the office to assess our biases etc which is all well and good but having the knowledge to understand how we’re analysing, identify potential issues in our analysis and how to communicate/fix these issues is incredibly beneficial.”
“A great course for any practitioners wanting to improve their own analysis and also learn how to provide more effective feedback to other analysts. The course is well-paced and provides opportunities for consolidation before progressing through the material. I would recommend this course to anyone who wants to ensure best practice in analytical rigour.”
“A great course for any practitioners wanting to improve their own analysis and also learn how to provide more effective feedback to other analysts. The course is well-paced and provides opportunities for consolidation before progressing through the material. I would recommend this course to anyone who wants to ensure best practice in analytical rigour.”
“This course is challenging, and so worthwhile. I have been able to apply it directly to my current role and will continuously draw on the skills I’ve learned in future. Can’t recommend enough.”

Contact

Contact us to learn more or to book Hunt Lab experts to run this course at your organisation. We are also open to discussing collaborating on developing a tailored course to suit your organisation’s needs.