Updated Results from the MITRE ATT&CK Endpoint Detection and Response Evaluation

by , | Mar 8, 2019

MITRE ATT&CK Endpoint Detection and Response

Back in December 2018, MITRE released the first round of its evaluations on EDR tools, including Carbon Black, CounterTack, Crowdstrike, Endgame, RSA, Sentinal One, and Windows Defender.  Specifically, MITRE tested the APT3 threat group (https://attack.mitre.org/groups/G0022/) against the products and rated how well they performed.

Above: APT3 Tactics highlighted in green


Recently MITRE published the first phase of its “Rolling Admissions” program, which added vendors FireEye and Cybereason.  Last time around (http://securityriskadvisors.com/blog/a-closer-look-at-mitre-attck-evaluation-data/), SRA scraped all the test result data from the MITRE results, and published it in a more head-to-head view, so that you could see how each vendor did against one another.

We recently updated our dataset (stored here: https://github.com/SecurityRiskAdvisors/mitreevalsdb) and have re-run some of our favorite queries to see how the new additions faired against the first wave of competitors.  What did we find?  Excellent performance from FireEye, and mid-pack performance from Cybereason.  In any case, this is a high level summary and detailed results should be examined if you’re seriously considering any of these products.  We tend to give the most credit to those orgs that went into the first round of this test blindly, and it seems that the ‘rolling admissions’ participants have a leg up in that they are taking an open-book test now.  That being said, Crowdstrike continued its dominance in this test, even while being from the first wave of participants.  Details below:

Query: select vendor, count(vendor) as total_detections from edr WHERE General = ‘yes’ or Specific = ‘yes’ group by vendor ORDER BY total_detections DESC;

If you want to recreate these results yourself, visit our github page here https://github.com/SecurityRiskAdvisors/mitreevalsdb to download mitreevals.db, then load that sqlite database into a DBMS, such as the web based system here: http://inloop.github.io/sqlite-viewer/

For more information, view the data yourself here! https://attackevals.mitre.org/evaluations.html



  • Evan specializes in network penetration testing, web application security testing, open source intelligence gathering, and security testing process automation. He has experience in a variety of industries including retail, insurance, financial services, and healthcare.

  • Mike is nationally recognized as a leader in the field of cybersecurity, has spoken at conferences including HITRUST, H-ISAC, RSS, and has contributed to national standards for health care and public health sector cybersecurity frameworks. Mike focuses on security architecture, cloud security, data protection, purple teams and electronic medical record protection programs. Mike joined Security Risk Advisors in 2018 was formerly the first CISO at University of Rochester Medical Center, has served as CTO in two roles, and as adjunct faculty at Rochester Institute of Technology.