Few government policy analysts are trained in assessing evidence in a structured way and then assigning confidence ratings when providing formal advice to decision makers. This lack of training can be particularly problematic when certain language (“likely,” “probable,” “believe,” etc) is used in an unstructured way because these words can lead decision makers to conclude that recommended actions have a heft of evidence and assessment that, in fact, may not be present in the assessment.
Put simply: we don’t train people to clinically assess the evidence provided to them in a rigorous and structured way, and upon which they make analyses. This has the effect of potentially driving certain decisions that otherwise might not be made.
The government analysts who do have this training tend to come from the intelligence community, which has spend decades (if not centuries) attempting to divine how reliable or confident assessments are because the sources of their data are often partial or questionable.
I have to wonder just what can be done to address this kind of training gap. It doesn’t make sense to send all policy analysts to an intelligence training camp because the needs are not the same. But there should be some kind of training that’s widely and commonly available.
Robert Lee, who works in private practice these days but was formerly in intelligence, set out some high-level framings for how private threat intelligence companies might delineate between different confidence ratings in a blog he posted a few years ago. His categories (and descriptions) were:
Low Confidence: A hypothesis that is supported with available information. The information is likely single sourced and there are known collection/information gaps. However, this is a good assessment that is supported. It may not be finished intelligence though and may not be appropriate to be the only factor in making a decision.
Moderate Confidence: A hypothesis that is supported with multiple pieces of available information and collection gaps are significantly reduced. The information may still be single sourced but there’s multiple pieces of data or information supporting this hypothesis. We have accounted for the collection/information gaps even if we haven’t been able to address all of them.
High Confidence: A hypothesis is supported by a predominant amount of the available data and information, it is supported through multiple sources, and the risk of collection gaps are all but eliminated. High confidence assessments are almost never single sourced. There will likely always be a collection gap even if we do not know what it is but we have accounted for everything possible and reduced the risk of that collection gap; i.e. even if we cannot get collection/information in a certain area it’s all but certain to not change the outcome of the assessment.
While this kind of categorization helps to clarify intelligence products I’m less certain how effective it is when it comes to more general policy advice. In these situations assessments of likely behaviours may be predicated on ‘softer’ sources of data such as a policy actor’s past behaviours. The result is that predictions may sometimes be based less on specific and novel data points and, instead, on a broader psychographic or historical understanding of how an actor is likely to behave in certain situations and conditions.
Lee, also, provided the estimation probability that was developed in the early 1980s for CIA assessments. And I think that I like the Kent Word approach more if only because it provides a broader kind of language around “why” a given assessment is more or less accurate.
While I understand and appreciate that threat intelligence companies are often working with specific datapoints and this is what can lead to analytic determinations, most policy work is much softer than this and consequently doesn’t (to me) clearly align with the more robust efforts to achieve confidence ratings that we see today. Nevertheless, some kind of more robust approach to providing recommendations to decision makers is needed so that executives have a strong sense of an analyst’s confidence in any recommendation, and especially when there may be competing policy options at play. While intuition drives a considerable amount of policy work at least a little more formalized structure and analysis would almost certainly benefit public policy decision making processes.
Excited Pixels 