On Conveying Doubt

----
On Conveying Doubt
// Talos Blog

This post was authored by Matt Olney.

Typically, Talos has the luxury of time when conducting research. We can carefully draft a report that clearly lays out the evidence and leads the reader to a clear understanding of our well supported findings. A great deal of time is spent ensuring that the correct words and logical paths are used so that we are both absolutely clear and absolutely correct.  Frequently, the goal is to inform and educate readers about specific threats or techniques.

There are times, however, when we are documenting our research in something very close to real-time. The recent WannaCry and Nyetya events are excellent examples of this. Our goal changes here, as does our process. Here we are racing the clock to get accurate, impactful, and actionable information to help customers react even while new information is coming in.

In these situations, and in certain other kinds of investigations, it is necessary for us to talk about something when we aren't 100% certain we are correct.  I'll provide two examples from our Nyetya blog posts:

Example 1:

"Given the circumstances of this attack, Talos assesses with high confidence that the intent of the actor behind Nyetya was destructive in nature and not economically motivated."

This is our response to customers who were asking "If I pay will I get my data back?".   There were a number of indications that made us think that this was unlikely, but we couldn't necessarily prove that there was no way it could occur at the time we published.  We weren't certain, but it was important to share our analysis quickly because customers needed information in order to make time-sensitive decisions, so we did so with a clear statement that there was room for error.

Example 2:

"This is a significant loss in operational capability, and the Threat Intelligence and Interdiction team assesses with moderate confidence that it is unlikely that they would have expended this capability without confidence that they now have or can easily obtain similar capability in target networks of highest priority to the threat actor."

Here we are speaking about an actor's thought process.  Obviously we aren't in a position to authoritatively speak about what is going through an actor's head.  But we can look at a broad set of circumstances, analyze them in the light of our past observations and experiences, and then try to understand what underlying meaning they might have.  Based on what we saw, we thought it important to express that the actor may have additional capability it had not shown, so again, we spoke in plain language that gave the reader information they could evaluate.

Speaking with doubt doesn't mean guessing.  At Talos it means applying experience and knowledge to a set of information that is incomplete and trying to extract actionable intelligence from that information.  When we document our findings externally, we are under an obligation to be crystal clear if we are engaging in some form of speculation in order to develop a thoughtful assessment based on strong indicators.  This doesn't make the information less valuable, but it does allow the reader to correctly weigh the information when prioritizing their own response.  As we move ahead, when Talos communicates doubt, we will do so using the following as guidance:

Phrase	Estimated % Confidence
Low Confidence / Possible / Unlikely	<35%
Moderate Confidence / Probable / Likely	35% - 69%
High Confidence / Highly Probable / Highly Likely	>70%

Our primary mission is to place into our reader's hands the information they need to defend their systems and their networks.  We can't always wait until we are 100% certain of findings, particularly while we are in the midst of an incident.  By utilizing this language, we can share findings earlier and give customers the ability to evaluate our information and apply it to their defenses if necessary.

----

Read in my feedly

Sent from my iPhone