The Hacker Mind: MITRE ATT&CK Evaluations

MITRE ATT&CK catalogs the known tactics, techniques, and procedures of past advanced persistent threats, providing a roadmap for any red or blue team.

In this episode, Frank Duff, Director of ATT&CK Evaluations for MITRE Engenuity, talks about how both red and blue teams can directly benefit from ATT&CK, and how organizations -- and even some security vendors -- are now evaluating their solutions against it.

Vamosi:  Formed in 1958, MITRE, which is an acronym for MIT Research Establishment, manages federally funded research and development centers (FFRDCs) supporting several U.S. government agencies. And as a corporation, MITRE is sponsored by US federal agencies such as FAA, IRS, Department of Defense, Department of Homeland Security, Centers for Medicare and Medicaid and NIST. Perhaps more relevant to security, MITRE maintains the Common Vulnerabilities and Exposures (CVE) system and the Common Weakness Enumeration (CWE) project. Yeah, great but what does it do?  

Duff:  What MITRE does is provide usually high level understanding to our government customers on problems that they face, and that can range anything from work we do with the FAA to work we do with the NCC OE so the National Cybersecurity Center of Excellence, which is operated by NIST, a variety of programs in between. 

Vamosi:That's Frank Duff, Director of ATT&CK valuations at MITRE ingenuity

Duff:   MITRE itself is working in the public good, and MITRE ingenuity who I specifically work under these days, focused on delivering that impact to the broader public so engaging the private sector versus our traditional government space, which is more of miters traditional bailiwick

Vamosi: MITRE works with federally funded r&d centers and public private partnerships, it works across government to tackle the challenges to safety, stability, and well being in our nation. In a moment we'll hear about a cool new framework that MITRE developed it attempts to map potential threats by looking at past events and catalogs, all known tactics and techniques used by criminal hackers. It's a framework, a tabletop adversary emulation, that provides the good guys on red teams and blue teams with some guidance to harden their defenses against future ATT&CKs.

Welcome to The Hacker Mind, an original podcast from ForAllSecure. It's about challenging our expectations about the people who hack for a living. I'm Robert Vamosi, and in this episode I'm going to talk about miters ATT&CK framework and how red and blue teams can benefit from it, and how organizations, even security vendors are now testing their solutions against it.

We'll get back to MITRE Engenuity in a moment. First, let's look at the MITRE project called ATT&CK spelled ATT&CK. A few years ago, MITRE came up with Att&ck. It's a framework that describes tactics and techniques commonly used by criminal hackers and state adversaries, and it is proving to be useful when considering threat models. It is based on real world observations and can be used by red teams to emulate an ATT&CK, and see whether your organization or its security products can withstand a persistent ATT&CK. In fact, that's how it came about

Duff:  ATT&CK itself is just a project within the MITRE Corporation, right, so it was originally spun up as a research and development effort that was being done about eight years ago now, time gets foggy but but a long time ago and in a prior life right like I was sitting there as a blue Teamer. We had red teams that were coming at us every other month to assess our, our advances as, as the defenders, and we needed a common lexicon, to be able to describe what the red team was doing to a way that people like me as the blue team could understand.

Vamosi: Frank has been at MITRE for many years, and he was there when ATT&CK started.

Duff:  So I was I was not the guy that came up with the idea from ATT&CK from any points, I was I was on the defender side, right, like I came down and at the time, we were looking at how to improve the visibility within our networks right so everybody was focused on antivirus at the time, and keeping the bad guys out and and I think the median dwell time which is like the the time it took for an adversary to be detected was somewhere around 210 days if I recall, on average, right, that that's, that's a substantial part of a year. And so think about what a bad guy could do in your network for half a year plus, right, that's, that's not a good feeling and so we start up the research project to try to say okay well what can we do to defend against the adversary once right. And so we started building out our own sensors, because this was before the the time to EDR, those before the time of Sysmon or anything like that, we were like, alright, let's build out our own sensor that can capture process information and and command line arguments and let's figure out how we can advance defenses and. And so I was on that team, creating a bunch of those analytics using sensors we had homebrewed and. And then the rest is history we've, we've continued to evolve ATT&CK and evolve our work program and I went from a defender to a guy that manages a bunch of red teamers to do the evaluations. So it's it's a fun time 

Vamosi: ATT&CK started as a workshop exercise to document common tactics, techniques and procedures, T TPS that advanced persistent threats used against Windows Enterprise environments, advanced persistent threats are just as they seem. They're the long game operations where something as small as a single phishing email could escalate into millions of IDs being exfiltrated. So ATT&CK began as a couple of red teamers sitting around a table discussing how best they could present their findings to the people who actually make the decisions.

Duff:  And so, out emerged this Excel spreadsheet of different behaviors that the red team was performing, which would allow us to focus rather on hashes, or specific malware, it allows us to focus on the higher level behaviors to improve our defenses. And what we found was as we were trying to report up our value of doing these Red Team Blue exercises which we loved because they were super fun, but we had to sell the management of why they're doing these every other month. 

Vamosi: What ATT&CK does is provide sophisticated look into what others might assume to be pretty simple fact. You got breached. But how, what were the warning signs and what were the tools they could have used to prevent that from happening again.

Duff: much more in depth thing where you're trying to exercise your full scope of defenses right because again right when the ATT&CK was created, we went with this, this presumed breach model which is, you can put up all the walls you want, but sooner or later an adversary is going to get in, and then what, how do you get them out faster. How do you defend them, how do you detect them. And so, so that's where really red teaming and variations of it which include adversary emulation come into play.

Vamosi: So internally MITRE started developing the spreadsheet. This framework, then iterating on it until it was ready to release to the world,

Duff:  we found that ATT&CK actually proved a good scorecard for our continued growth right it's something we could use as a reference model to show this is what we tested against, and this is how we did and then this next one, These are the things that we focused on improving and then the next one. These are the other things we focused on improving and true, so we got to show that iterative growth. And so we thought it was really useful to communicate between leadership and the offenders and the defenders, that we thought you know what let's publicly release this thing and so we released it. It started off as a media wiki, and so it very much had a Wikipedia kind of feel to it describing just all the things that adversaries could do based on public reporting. We didn't, providing any new intelligence we were just providing at a different level of understanding.

Vamosi: ATT&CK is currently structured with 14 tactics across the top, indicating how an adversary would ATT&CK a system. The tactics are reconnaissance, resource development initial access, execution, persistence privilege escalation defensive Asian credential access discovery lateral movement collection, command and control exfiltration and impact these pretty much map the activities of most major ATT&CKs that we've seen. Maybe not all the tactics are used, but certainly not fewer,

Duff:  they're, they're always changing the number of them to be fair, we recently went through a movement that took what we called pre ATT&CK and merged it in to some degree the ATT&CK lifecycle. So, the ATT&CK framework Ed's itself, the enterprise ATT&CK had this portion that was all the things before you get on the network that adversaries do, let's bring that over impact was a extension last year which focuses on the other side of thing, what do they do, as, as kind of that end goal, do they wipe systems, do they encrypt it. In the case of ransomware. But ATT&CK itself has tactics at the high level, which, which provides an overarching goal that an adversary was trying to achieve

Vamosi: these tactics track against the behaviors we've seen from adversaries. Now beneath each of these tactics are between six and 36 techniques, each, which includes specific actions such as brute force network sniffing and audio capture. In total there are more than 200 of these techniques,

Duff:  started off as this small little Wikipedia site and grew and grew and grew from, I think it was around 40 to initially to 70 to 272 now there's all these sub techniques that are associated with it. Now it's just this huge matrix of all the things that that adversaries can do to operate within the network, right, so the things that how they can get initial access, how they can move across the network, how they can dump credentials or exfiltrate data or deliver impact to achieve their goals. So that's kind of the the Genesis and then the 10,000 foot view of the ATT&CK framework. And then there's the techniques underneath it, which are the how they can achieve that goal, it's more of the specifics for how they would like to accomplish one case it could be a credential access is a tactic and dumping credentials could be a technique, under that right into specifically, how to use the tool to extract them from L SATs. So, so yeah so so those are the kind of high level constructs there.

Vamosi: What's really cool is that you can start to map specific a PTS against this framework. In other words, if I want to see what techniques, say, a PT 29 or carbon AK uses within each of the 14 tactics, I can highlight those. Pretty cool,

Duff: Right so there's the ATT&CK Navigator Project, which, which we've also released that allows you to, to select the cells that would be relevant to you. So if you think of ATT&CK as more of the periodic table, which of the elements make up a specific compound for an adversary. And so what, what you can do is say APT 29 for instance is known to use these 60 different techniques to achieve their goals. And so when it comes to adversary emulation which is much more my, my focus area these days. we can build out on appropriate emulation of an adversary to understand how defenses would perform against a specific adversary of interest.

Vamosi:So you can look at an adversary color in the map, then see if your organization is protected against the techniques used by that adversary.

Duff: And similarly you can do heat maps to show defensive coverage so saying tool x, I have deployed in my environment, it provides me this, this broad capability, so I'm going to color in these cells as things that I know I'm defended against or think I'm defended against knowing is always hard but think I'm defended against or not defended against at all, or these are the things that I want to invest in. So so that that concept of what a lot of people refer to as like the heat map or the stoplight chart, there's different variations of pay on the use case, it resonates with a lot of people and again that was something we used in those early days to describe what we were doing and how we were improving, it's like our adversary X uses these things. That was the scale for which we were trying to test our environment, right, that that was what, what our criteria was. And then these are the things that we did well to defend it and these are the things we didn't do well. So it just provides a very referenceable thing.

Vamosi: In a previous episode, I talked about the concepts of red and blue teams, red teams ATT&CK blue Team's defense. And then there are the purple teams that do both. This is important for large organizations, so they don't have to wait until they are ATT&CKed, to see how they will bear. They can do exercises and shore up the weak spots.

Duff: Yeah, so it is a possible use that gets a little tricky adversaries continue to evolve and attribution is a delicate subject to put it, put it lightly and by far from where my expertise is. But I know that that right it's one thing to have an idea of what a adversary or a group of adversaries or a type of adversary would do, and generally know that they might do some of these other things, but trying to fingerprint an adversary too much just based on techniques is probably a little bit risky, but it does allow you to hone in on specific elements that that really you should focus your defensive event. Defensive investments with right and allows you to understand what you can defend what you can't defend

Vamosi: ATT&CK also allows you to move to the next level to get ahead of the adversary ATT & CK based on past events. So if an adversary is currently doing X, there's a reasonable chance at the move on to do why

Duff: next. If you see that credential dumping is happening. It makes perfect sense for you to, no matter what the adversary is for trying to figure out where those adversaries are, or where those credentials are getting used. Like, what machines, if I saw credential dumping happen on box X. What other machines did that off, connect to, and I should be monitoring those boxes extra specifically because right if you don't credentials oftentimes you move into lateral movement after that, using those same credentials. So there are some tricks that can be played. But more than anything, it allows you to just have the, these are the things I should be looking for in an environment or can be looking for. These are the things I know I've got analytics for and here are some things that maybe I need to kind of look in the weeds for look in the raw data for other than the analytics,

Vamosi: that's the blue team. What about the red or offensive team, how can they use ATT&CK.

Duff:  And that's exactly where, where my program has started right so ATT&CK itself as I referenced earlier is huge, right, it's one of those things that, that's really tough especially for organizations to get started with just because how do you prioritize which techniques to worry about. And so we're adversary emulation comes into play, and allows you to pick an adversary that would be relevant to your organization, maybe it's an AP three maybe it's in AP point nine maybe it's fin seven or carbon or any of these other groups but pick some adversary that you think would target you or potentially could target you

Vamosi: abt three carbon act, what these organizations have started to map out common behaviors and adversaries and have named them for convenience, it's shorthand but it's necessary that we create a common language as a reference point for these ATT&CKs.

Duff: That gives you a subset of techniques to start with, right, you think that this type of an adversary could go after you. You better be able to defend against that right if you, if you can't defend against what's happened in the past which is what ATT&CK is, you can't worry about the future right you have to start somewhere and so ATT&CK an adversary emulation lets you ground that evaluation in something that is known versus on hypothetical. And then you can you can continually run that same adversary over and over again and see that incremental improvement, you could do different adversaries, one of the things we do under the ATT&CK evaluations program which I run is in the first round we've chosen 83.

Vamosi: According to MITRE ATT three is a threat group that some researchers have attributed to China's Ministry of State Security. This group is responsible for campaigns known as Operation clandestine Fox operation clandestine Wolf and operation double tap as of June 2015 The group appears to have shifted from targeting primarily us victims to primarily political organizations in Hong Kong, a PT three lies on harvesting credentials, issuing on keyboard commands, versus Windows API, and using programs already trusted by the operating system, so called living off the land. Similarly, they are known to do elaborate scripting techniques, leverage exploits after access or use anti-EDR capabilities such as rootkits and boot kits.

Duff: There's a lot of information available on 83 It's a good starting point to understand what capabilities can do because they leveraged a lot of living off the land specifically by utilizing command shell, so running things right out in CMD so it's a lot of process data. But then what, what we wanted to do was then the next round to compare that against something that would be a higher bar for us to go after

Vamosi: they chose APT 29. According to MITRE APT 29 is a threat group that has been attributed to the Russian government. It has operated since 2008. This group reportedly compromised the Democratic National Committee, starting in the summer of 2015, APT 29 is distinguished by a commitment to stealth and sophisticated implementations of techniques via an arsenal of custom malware, APT 29 typically accomplishes its goals via custom compiled binaries and alternative execution methods, such as PowerShell and WEMI, APT 29 has been known to employ various operational cadences, such as smash and grab versus slow and deliberate. Depending on the perceived intelligence value and or infection method of its victims,

Duff: and so APD 29 is a much more sophisticated actor. They do a lot with scripting technologies which are harder to defend and detect and leverage those protections because it's PowerShell. And it's one of those that emerged, it was at the time one of those emerging techniques is still the most popular. So, you can choose some adversaries, even if they don't necessarily target your your organization maybe you want to do it as something of baselining environment versus leveling up your defending capabilities but in all cases adversary emulation lets you really have that, that known contact right you need to have that threat informed defense, not just the hypothetical, and then I think before ATT&CK a lot of people were jousting windmills right they they were thinking that they were going after the threat we're really, maybe they were sometimes maybe they weren't other times, but now we know

Vamosi: all of this starts to sound like pentesting.

Duff: Yeah, so I mean pentesting by definition has a little bit of a difference, right, pen tests are more usually centered around where, how to exploit vulnerabilities to get in. So right, you do your vulnerability assessment, you know where your vulnerabilities are, pen tests, go and exploit those vulnerabilities. Red Teaming start has a similar vein, but it's much more focused on once they're in. And then that's where the adversary emulation comes in it's once, once the pen test had penetrated the network, what can they do, and that's that's where it's the adversary emulation component to try to find all that host exploit post breach behavior.

Vamosi: I mentioned earlier that you can map ATT&CK against an adversary, but can you map it against an industry. Say you're in oil and gas,

Duff: anytime you try to generalize to a certain point right it's that breaking point of overgeneralization. I do think that there are specific things you can look at for instance when, When you're using ATT&CK to understand which adversaries would ATT&CK, your organization, right so if you know that, or that adversary, A, B and C are the ones most prevalent in your sector, like look at which techniques they have in common, right, and those are the ones who should have done first. So So there are definitely some things you can do that are more sector specific or vertical specific, but at the end of the day I think that that there's a certain amount of trying to make sure that you've got the broad coverage because again adversaries are going to change to different things and as soon as you defend the one, they'll find a different way, because they're persistent right they want to get in, they want their either IP or they want their, to be able to extort you for your ransoms they can get money or any of these other things.

Vamosi: Earlier I mentioned that Frank worked for a part of MITRE. What is MITRE Engenuity.

Duff: So MITRE itself is still the same old Corporation, it's always been right operating those centers to advanced capabilities for our government sponsors MITRE ingenuity is just delivering that same public interest. However, instead of working with our government. What we do is we'll work with. In my case, for instance vendors right we bring in industry, industry pays us to do evaluations.

Vamosi: This is more of a testing and training service of MITRE

Duff: they pay MITRE to perform it, so right we, we need to be able to execute these in some way. And so we just that's how that's how we do it, it's a, a single fee for all participants. So there's no favoritism going on or anything like that. And it's purely to pay for the evaluation the processing of results and release of results,

Vamosi: really this is not too different from a red and blue team activity, except in this case the blue team is the vendor and its particular product.

Duff: Since ATT&CKs been around, right, it was always focused on trying to advance our defensive capabilities, and a common extension from that is a, they start with a security stack, but that's great but now I want to improve on it, you know if I need this new sensor. So then you drop it down, so we've been looking at evaluating products at some level, since, pretty much the inception of ATT&CK. We quickly got into this, this place where we're programmed that I was running which was another research project that was trying to figure out how to take ATT&CK and transition this to the broader community both our traditional sponsors and not. And so what we did then was start using this methodology to evaluate products specifically so one single product deployment arrange rather than adversary against understanding how they would do. And then we would provide those results back to our government sponsors or customers. Well, that was great, except them all the fortune 500 for finding out that we're doing these things vendors are finding out that we're doing these things and next thing you know there's this clamoring that everybody wants wants us to be able to provide the service, more broadly,

Vamosi: MITRE is ideally focused on the government agencies, and in this case government procurement that said enterprises have shown increasing interest in ATT&CK and are looking for MITRE for solution guidance as well.

Duff: MITRE decided that we were going to try to do it at something very atypical for MITRE right historically again we are working with our governments answers and that's where all our funding comes from. And in this case it's like, alright well to scale this we have to figure out a different way to do it, and that's where vendors now pay us to perform these evaluations and and we execute them and we write, we went from seven in the initial launch to 21 the second round to now 29 In this third round so it's one of those things that's become kind of a more norm of industry versus kind of unique thing we're doing.

Vamosi: So again, red team and blue team activities,

Duff: so it's it's a long process to get an A, an evaluation ready. There's a lot of research that goes into it. and at a very high level what we do is we take all the open source reporting that's existed for specific adversaries that we've picked and and we go and process that we develop a plan of ATT&CK that would try to be inspired by that adversary, we're not we're not down to the level of trying to make sure that our tool sets are one to one matches what we're interested in is replicating the behaviors. And so we'll take their plan, and we'll sequence it in a way that we feel would be an appropriate representation of what they would do. We form the behaviors as we think they would, we use similar tooling that we think that they would. So we'd find this whole plan, and that is literally something that we copy and paste during execution when we get there, but it's a very structured plan for the setup portion right every vendor gets access to a common Cyber Range, we hosted currently in Microsoft Azure for enterprise evaluation, a Cyber

Vamosi: Range is a platform that provides hands on security practice to teams of professionals, kind of like a shooting range, where marks person would go to practice cyber ranges where an info professional can learn and improve their security skills.

Duff: It's not supposed to be realistic. It's supposed to be. Here's a few Windows boxes, here's a Linux box here's a Windows Server, deploy your solutions, and configure your solutions and the vendor does that themselves, so they kind of have to decide what they want evaluated and how they want it evaluated, which does make it a little apples to oranges when you're considering the results because some vendors will turn sensitivity up to 11 some of them will kind of do more like direct out of the box kind of things. It just depends and some of them will take, like, let's, let's try to collect this one sensor here that sometimes we collect sometimes we don't right it just, it's kind of like picking your poison. So, the vendor deploys their solution. The evaluation itself is built to be collaborative. And so one of the terms that the community uses a lot these days or it's increasing usage is purple teaming. And so that's the mindset we have and so what that means. So we have that plan that I talked about, which is a copy and paste and we execute that to a tee. Then what we do is we actually are sitting there with the vendor in the room or on the phone and in this past year due to COVID, but we sit there with the vendor and we say this is what we did. Show us what you got, and then they show us all their alerts all their telemetry that would show that

Vamosi: I'm like a review that I would do for ZDNet or CNET, where I have no idea what's possible or not just whether or not the solution passes my particular tests MITRE actually worked with the vendor, of course correcting as they go through the evaluation exercise,

Duff: we might say you know what that one is actually something else or that's just, that's just noise, but we're actually looking for is this specific API getting touched Do you have anything that would show that, and so then they go refocus and say okay this is what we have

Vamosi: this starts to blur the line crossing now from the Red team to help the blue team. This is clearly a purple team activity, where you're trying to do both offensive and defensive activities at the end of the day though, the goal is to strengthen the defense right,

Duff: and so we walk through the entire scenario from beginning to end with them, explaining what we do, they can ask any questions they have of us, we are completely open book and this is what we did this is where we did this is how we did it. And and we just collect that as we go and we are taking a bunch of screenshots and all that stuff processing that information as we go.

Vamosi: Let's look at one scenario, The carpet X scenario begins with a legitimate user executing a malicious payload delivered via spear phishing ATT&CKs targeting financial service institutions. Following the initial compromised carbon ACK expand your access to other hosts through privilege escalation credential access and lateral movement with the goal of compromising money processing services, automated teller machines, and financial accounts as carbon compromises, potentially valuable targets they establish persistence, so that they can learn the financial organization's internal procedures and technologies. Using this information, carbon ACH transfers funds to bank accounts under their control, completing their mission.

Duff: In this last round we did one for fin seven and one for carbon Act, which both leverage the carbon act malware so it's a little confusing, but they are believed to be distinct groups so we executed one one day one the other day, and did the same thing both where we execute and collect all the data, then there's a protections extension that's optional so that's now let's turn on protections and see what we could have stopped. And so for that one we execute it similar ways, but has some some some key differences and how we do it just to make the test functional, but But in all cases what we're trying to do is focus on ATT&CK, not about the malware but about the behaviors, what are you doing to detect or prevent the specific behaviors we're looking at. And then there's this big processing chain that requires us to do a bunch of data processing which is boring, and then we release the results at the end to everyone. And you get all the marketing buzz around it.

Vamosi: So by releasing the results MITRE ingenuity leaves it up to the vendors to make sense of how they fared.

Duff: Yeah, so So I think that that 's two different poles right. I think, largely speaking, the industry really appreciates that ATT&CK violations exist and is providing that data, to, to support or refute claims that that, that the vendors make. I do think that there's a lot of fatigue, around everybody saying that they want because of XY and Z, right, we got 100% coverage over here we got 100% coverage over here we're the best because the block the most or detect the most or where the quickest or fastest right and there's all different ways of cutting and I'm not, not saying that those are bad things for for vendors to say, but it's usually skewed towards their one perspective, and then as an end user, I'm just seeing everybody say that they won without the, the underlying context of what the results actually mean. So then I have to go in and look at I being the user would have to go back to the ATT&CK eval site and actually see like art, who's blowing smoke who's saying good things, but again like I don't I don't fault the vendors for saying that they did well, they shouldn't feel like they did well because we're working with them to ensure that they can improve and do well and do the best that they can. And at the end of the day, each one of these solutions has a different focus, has a different selling feature and has something that they view as a key part of their strategy. And, and so they can each have their own amount of win, if you will. and that can be a true statement, but for you as a consumer, you have to think about which one is right for you, and that's where it gets a little bit more challenging

Vamosi: MITRE ingenuity releases a raw dump of the same tests used for each vendor. Recently they provided data from 29 companies, and how they would defend against carbon AK and fin seven, but they represent results from a sterile environment. In other words, actual mileage may vary.

Duff: Just saying, here's some slices and dices of that at a high level so you can dive in and see if it makes sense because at the end of the day, there's so much end user context that needs to be applied, does this solution make sense for you. It doesn't make sense for your context, your users that are in it, your analysts that are going to be using the tools, those things are things that we can evaluate and we don't evaluate what we do as we say, these are what these kiddos can do. And then here's some screenshots so you can see what it looks like when so you would know what it would look and feel like if he put it in your environment.

Vamosi: The ATT&CK evaluations published do not include critical rankings MITRE and keeping the stay objective leaves that for the individual vendors to discuss among their peers,

Duff: right, and that goes back to kind of the origins of MITRE and the program itself, right when we started MITRE didn't want to get in the business of providing our opinion of how a product performs all we want to do is lay out the facts and that people make determinations on themselves right we can collect data in a very rigorous way release that data, and then other organizations can use other people can use it. It is very great, that, that so many vendors jumped in initially right off the bat, recognizing that we were going to public release everything because that's been a mantra, since since the get go, we've just fallen listed all our results are going to be publicly available, that's that's miters mission in the public good, We have to do that. And we we encountered very little pushback for that, surprisingly enough, I mean everybody wants the results to look the best. But we, that's why we also design the evaluations how we did to include that, right, we, we provide the red team for these evaluations the vendors, the ones providing the blue team they're showing us what they can do, they're the ones configuring this,

Vamosi: so this isn't to create some sort of good housekeeping seal of approval is

Duff: it, so we have no desire to do that that seals approval right I mean from an overarching construct right so so for organizations like Splunk that leverage it right, they are not part of the evaluation process or at least haven't been to date. But I do think it's absolutely great that they're picking up ATT&CK, they're leveraging ATT&CK they're describing their detections and ATT&CK, and that goes for all these other companies that are using right, we are very focused on our evaluations on the detection space. But I know deception technologies, I know that reach and ATT&CK simulation tools use it, I know that that threat intel platforms use it, there's the number of people that use ATT&CK in this field is much greater than the people that don't at this point. And I think that that is, is absolutely great. Of course when a lot of people use it and, and if I was a fortune 500 I'd be saying what's you, what are you going to do to improve my ATT&CK coverage, you get a lot of oftentimes marketing hyperbole, that goes into there, and where there's marketing hyperbole. That's where there's usually a need for evaluations to take place just so again there's some source where people can go in fact check right there there's the data available that you can bounce back to say, alright, this vendor said that they had 100% coverage. Did they, well let's dive into it and see what that 100% coverage actually means.

Vamosi: With its 14 tactics and more than 200 techniques ATT&CK seems pretty complicated. It's not once you get the hang of it, to help people get used to its structure and to derive some value MITRE ingenuity has created the MITRE ATT&CK defender. It provides ATT&CK training and certification from ATT&CKs own subject matter experts.

Duff: There's a lot of good open resources around ATT&CK, ATT&CK has been a grassroots effort right it was developed by us not because we thought that it was going to be this huge thing. We just found it useful. And so we released it. And similarly, the industry as a whole has quite a lot of resources and releases things recently for engineers and it has this other program called MITRE ATT&CK defender, and that is a train open training program as well as certification. And so what we do is we release the training free and open. And then we have on the back end, a certification you can pay for to get certified in prints and stock assessments, so you can understand how you can evaluate your organization based on on ATT&CK.

Vamosi:  Currently there are two certifications, the ATT&CK Cyber Threat Intelligence CTI certification is for practitioners interested in certifying a mastery in the application of ATT&CK, to improve existing threat intelligence and experiential assessments, validated learners ability to map to ATT&CK from both finished reporting and raw data, perform CTI analysis using ATT&CK map data, make defensive recommendations based on research, and more. The second certification ATT&CK security operation center SOC certification validates defenders proficiency at using ATT&CK to perform rapid low overhead SOC assessments. The certification confirms defenders abilities to align modern security operations with ATT&CK or threat informed defense specific topics include analysis of sock tools and resources, interview and discussion capability ATT&CK with personnel and building recommendations based on the results.