Updated Results from the MITRE ATT&CK Endpoint Detection and Response Evaluation

by | Mar 8, 2019

MITRE ATT&CK Endpoint Detection and Response

Back in December 2018, MITRE released the first round of its evaluations on EDR tools, including Carbon Black, CounterTack, Crowdstrike, Endgame, RSA, Sentinal One, and Windows Defender.  Specifically, MITRE tested the APT3 threat group (https://attack.mitre.org/groups/G0022/) against the products and rated how well they performed.

Above: APT3 Tactics highlighted in green


Recently MITRE published the first phase of its “Rolling Admissions” program, which added vendors FireEye and Cybereason.  Last time around (http://securityriskadvisors.com/blog/a-closer-look-at-mitre-attck-evaluation-data/), SRA scraped all the test result data from the MITRE results, and published it in a more head-to-head view, so that you could see how each vendor did against one another.

We recently updated our dataset (stored here: https://github.com/SecurityRiskAdvisors/mitreevalsdb) and have re-run some of our favorite queries to see how the new additions faired against the first wave of competitors.  What did we find?  Excellent performance from FireEye, and mid-pack performance from Cybereason.  In any case, this is a high level summary and detailed results should be examined if you’re seriously considering any of these products.  We tend to give the most credit to those orgs that went into the first round of this test blindly, and it seems that the ‘rolling admissions’ participants have a leg up in that they are taking an open-book test now.  That being said, Crowdstrike continued its dominance in this test, even while being from the first wave of participants.  Details below:

Query: select vendor, count(vendor) as total_detections from edr WHERE General = ‘yes’ or Specific = ‘yes’ group by vendor ORDER BY total_detections DESC;

If you want to recreate these results yourself, visit our github page here https://github.com/SecurityRiskAdvisors/mitreevalsdb to download mitreevals.db, then load that sqlite database into a DBMS, such as the web based system here: http://inloop.github.io/sqlite-viewer/

For more information, view the data yourself here! https://attackevals.mitre.org/evaluations.html


Evan Perotti
Sr. Scientist | Archive

Evan specializes in technical security assessments including network penetration tests, purple teams, red teams, and cloud security. He has experience in a variety of industries including telecommunications, financial services, pharmaceuticals, and healthcare.

Evan maintains the internal SRA standards and methodologies for purple team projects.

Evan is a member of SRA’s internal Research and Innovation team where he works to research novel approaches to security problems as well as develop security tools and resources.