Five common pitfalls to avoid as an intelligence analyst
With more companies starting to produce their own intelligence reporting, we have identified five common pitfalls that intelligence analysts often encounter and suggestions on how to avoid them.
Common pitfalls include burying conclusions, failing to answer the original requirement, and succumbing to cognitive biases.
Companies considering how to develop an intelligence capability should note that AI is a useful tool for intelligence analysts but is not a substitute for human expertise.
Intelligence, specifically open source intelligence (OSINT), is increasingly being used in the private sector. While many people associate intelligence solely with governments and militaries, the reality is that intelligence has been employed in the business world for decades.
This article emerged from our research at Tyburn St. Raphael into the use of intelligence in the civil legal sector. Intelligence provides legal professionals with actionable insights, helping them determine the best strategies for their cases. Yet, junior intelligence analysts in law firms often have no prior experience with intelligence and no access to a community of experts to nurture their potential.
As the next generation of talent comes into contact with intelligence in more sectors, it is important to help those who have started producing intelligence reports for the first time to avoid some of the common pitfalls. While government agencies and militaries have well-established training programmes for new personnel, many businesses experimenting with intelligence for the first time will not have such processes.
This article identifies four common pitfalls, and one emerging pitfall, that intelligence analysts often fall into when producing reports. While these pitfalls apply to intelligence analysis and report writing more broadly, they are especially prevalent in sectors where intelligence is a comparatively nascent practice. This article will help analysts working in the legal sector avoid re-inventing the wheel; taking advantage of lessons learned in other fields where intelligence analysis is more established.
Pitfall One - the report’s findings are hidden
The key findings of an intelligence report should be immediately obvious to the reader, just as this article started with a summary of its key points. These findings represent the analyst’s critical conclusions and should be conveyed clearly and immediately.
For newer analysts it might seem logical to place the key findings at the end - allowing the consumer to understand their significance after reading how they were reached. However, this approach is impractical for most intelligence customers, who simply do not have the time to wade through 10 pages of detailed analysis.
A novice intelligence analyst may mistakenly believe that it is their methodology or subject matter expertise that they are judged on. In reality, the customer is interested in how the key findings can help them achieve their goals, not the work involved in producing the report.
Ideally, the whole report would be conveyed in 400 words, without adjectives. Use bullet points for clarity and brevity. More detailed explanations belong to the findings section. If your key findings exceed a page, then you should consider a re-write.
Pitfall two - the report does not answer the requirement
An intelligence report must answer the requirement. However, our research shows that intelligence reporting in the legal sector often fails to do so. Instead, reports answer something close to the request and often provide irrelevant information. In response to a request for intelligence on a person of interest, for example, a report might instead provide a screenshot of a target’s private social media account without any contextualisation of its significance. What can the consumer expect to do with that information?
Instead, the report’s findings should clearly answer the request and contextualise their significance to the case. It should answer the ‘so what?’ question. For instance, if the request is to uncover whether a target has undisclosed legal issues and dubious political ties, a report focusing on their social media activity, while potentially interesting, is not relevant.
In such a scenario, the report should instead provide intelligence about the target’s legal history and a network analysis of their political connections. It should then contextualise the risk/significance these findings pose for the case, enabling the consumer to make an informed decision. The purpose of intelligence reports is to provide actionable insight: an insight is only actionable if it is relevant. It can be tempting to include extra information to demonstrate your diligence, yet they must be relevant. If a consumer asks ‘so what?’ after reading your report, you have fallen into this common pitfall.
Pitfall three - the report was rushed or came too late
Producing this article took time; it required research, writing, and several adjustments. The same can be said for intelligence reports. Intelligence analysts are expected to produce reports quickly. Theoretically, the quicker intelligence is provided the more actionable the insights become. However, a common trap that new intelligence analysts fall into is allowing their consumers to pressure them into rushing a report.
Rushed reports are generally of lower quality. Collecting intelligence requires many steps, especially in the open source domain, where there is simply too much information to sift through. Likewise, analytical procedures require time to transform raw data into intelligence and contextualise its significance. Rushed reports miss key insights, and our research into the legal sector indicates that they reduce their value as a guide to dispute strategy.
On the other hand, it can be tempting for analysts to search for more and better information. This is a common pitfall across many sectors. In the digital age, there is always a possibility of finding a crucial piece of information somewhere. Yet, the time it takes to search for this information, which may not exist, detracts from the intelligence’s ability to be actionable. In these cases, the situation may have already changed, rendering the intelligence inactionable. A recommended practice is to concentrate instead on exploiting the information you already have, enhancing your analytical skills.
Intelligence is an industry where ambiguity is guaranteed. Analysts do not have the luxury of waiting for certainty as by definition the intelligence will no longer be useful. A good rule of thumb is to send the report when you feel like you have 80% of the information, helping you find the balance between gathering sufficient information and avoiding paralysis by analysis. If you feel that your report is rushed or you are still searching for the golden nugget, you have probably fallen into this pitfall.
Pitfall four - the report falls prey to cognitive biases
Analysis and contextualisation are key processes that transform information into intelligence. Nevertheless, intelligence analysts are humans - they are subject to common analytical challenges, such as cognitive biases, and these can undermine an intelligence report. Despite the work that has been done within intelligence, newer analysts regularly fall into these pitfalls.
A common pitfall is analysts’ tendency to be drawn towards information that supports their original hypothesis, rather than considering information that counters it. Junior intelligence analysts, conscious of their position in the workplace hierarchy, often omit information that contradicts their hypothesis, mistakenly believing that doing so bolsters their credibility with senior readers. In reality, it does the opposite, as it undermines the objectivity and accuracy of reports, leading to flawed decision-making. This pitfall is particularly notable in the legal sector where intelligence is overshadowed by the prestige of legal professionals, unlike government and military contexts where intelligence is a better established practice.
A good way to counter this pitfall is to use any of the range of structured analytical techniques, which will help you decipher incomplete information, generate hypotheses, and manage uncertainty.
Another common analytical trap we have noted in the legal sector is mirror imaging - also known as the Wykehamist Fallacy - a bias where analysts assume that others will make decisions in the same way they would. This is particularly evident in fraud cases, where a target’s actions often appear so bizarre that it is easy to call them ‘stupid’ and rule them out. This can result in underestimations in motivations or capabilities, leading to flawed assessments in the report. If you are avoiding information that challenges your hypothesis or side in a legal dispute, or reporting that a target’s actions are ‘stupid’, you have likely fallen into the pitfall.
The new pitfall - rushing into AI reliance
Reliance on artificial intelligence (AI) has become a growing pitfall across many sectors, with students often using it to write essays — much as AI could have written this article (but didn’t). AI will have a significant impact on intelligence broadly as a tradecraft, notably by automating collection processes and analysing large datasets.
One area it will also affect is report writing. Large language models (LLMs) and natural language generation (NLG) systems are capable of producing reports based on unprocessed data. These AIs can create intelligence reports quicker and more cheaply than traditional methods. However, just as there are issues with relying on AIs to produce essays, there are issues for report writing. AIs currently lack contextual awareness, the assessments they provide are often not contextualised or specific enough to the case. This affects the quality of reports.
Using AI to generate reports also raises concerns about the report’s regulatory and legal compliance. Intelligence, by nature, handles sensitive information, raising some serious privacy concerns about using AI to produce reports. Specifically, there are issues with AIs compliance with the General Data Protection Regulation, which governs what data may be collected and how it is stored. The new Data (Use and Access) Bill likewise proposes new regulations to AI in terms of handling sensitive data, indicating the challenges of using AI to write reports.
AI is certainly a useful tool which can be used to strengthen some processes in intelligence. While it can be tempting to rely on AI to write reports, it introduces serious legal considerations and numerous unknown variables. There will be a temptation for companies to use AI to rapidly generate an ‘intelligence’ capability. However, doing so runs the risk of substituting cost and production efficiency for human expertise and experience. This is a pitfall that will become ever more common. A better approach would be seeking guidance as to how these technologies can support your intelligence capabilities.
Conclusion
An intelligence analyst is judged on the quality of their reports. Our research indicates that higher-quality reports are more likely to be acted upon by consumers. Enhancing the quality of the reports is an easy way to maximise the value of intelligence and develop the practitioner’s skills.
This article provides some quick tips for analysts to avoid common pitfalls that can damage their reports and reputation. However, as intelligence is increasingly integrated into the private sector, organisations with little experience of intelligence will need to determine how to more thoroughly provide the next generation of analysts with the tools to produce quality intelligence and handle new pitfalls.
Tyburn St Raphael is a security boutique. Our experts come exclusively from UK government, military, and academic backgrounds and have decades of experience in delivering intelligence led investigations and solutions. We can provide intelligence capability or support the development of internal capabilities for our clients. If you have any questions, or wish to get in touch regarding further information, please contact info@tyburn-str.com.
