• All Blogs
SOC Incident Response

My 5 best SOC incident response hacks

Shai Gabay | Mar 30, 2017

In my former position as a bank SOC manager, I was burned more than once by breaches. During that tenure, I found that it often only took a simple change in approach to make the difference. I learned the hard way, but you don’t have to. In this post, I’d like to share some of the concepts that really helped me upgrade my incident response capabilities. It’s a well-known stat (in security circles, anyway) that attacks to the financial services sector can take up to 98 days to be discovered and remediated. Add 99 days to that and you’ve got the average amount time it takes to discover and remediate attacks in the Retail industry, a full 197 days.

What’s worse than the elongated dwell times is that for many of these breaches, the SIEM had generated alerts and/or indicators for the attacks, but busy SOCs are so inundated with alerts that they did not receive the attention needed in time. Had the incident response team responded to the alerts earlier, their organizations would have been spared a significant amount of damage.

The big question is this: What can you do differently to ensure that your organization doesn’t become another victim in the headlines?

Let’s start by understanding the main challenges:

Alert Fatigue

Security is deployed in layers. To cover all those layers, an in-depth and wide-scope approach is the best defense, but that means that security teams must employ a wide array of tools and security components. Each of these tools and components generates thousands of logs, all of which create thousands of alerts. This overabundance of alerts leads to what is referred to as Alert Fatigue. Many organizations deal with alert fatigue by tuning the alert threshold so that they only generate the amount of alerts they are capable of processing. That means that a lot of alerts that might actually be significant are being ignored, increasing the probability of being breached without noticing it.

Skill Shortage

There is a massive shortage of skilled security analysts and experts expect that by 2020 there will be over 1.5 million security positions left unfilled. There are new graduates entering the field but many companies are wary of hiring newbies who don’t yet have the skills needed to defend their most precious assets. And as attacks become more advanced, a wide variety of skills are needed to mitigate them. Even if organizations could find those highly skilled analysts, it doesn’t mean that they would be prepared to deal with every attack scenario that the organization may face.

Moving from High Value to High-Profit Targets

Attacker’s motivation is shifting from “high-value targets” to “highly profitable targets” because simply put, they want money. This means they are looking to attack mission-critical business process. Last year’s SWIFT Terminal breach offered a glimpse into what it means to hit “highly profitable targets” when hackers pilfered $81 million straight out of the Central Bank of Bangladesh via malware that circumvented their SWIFT Terminal messaging system.

Non-Alignment of Business and Security

Perhaps the most pressing challenge is that most organizations are not really aligning business with security. To address this new kind of advanced threat, both facets, security and the business as a whole, must work together to implement the right controls and visibility across the business data and applications.

Keeping up with Alerts

Back to those 197 days, it takes for the typical retail business to discover and remediate an attack; this means that we need the ability to search historical data for IOCs. For this type of forensic research to be possible, we must store data for several months if not longer and have powerful tools that make it easy to investigate massive amounts of historical data. A Big Data platform is now a critical part of creating effective incident response. Every additional second that an attacker has inside your networks allows them to do greater damage. Addressing these threats as soon as they occur is critical to defeating them.

Here is how we recommend addressing threats in a methodical, real-time manner, that will allow your team to keep up with high threat alerts so that nothing falls through the cracks:

Data Collection

  • All data should be canonized to ensure it’s structured and normalized. This is critical when collecting data from a variety of sensors and security components since every one of them can use different fields and meanings for each data type.
  • There is always a lingering question what data to collect and for how far back to save it. From my experience, there is no one answer. Each organization has to assess which systems and data are most critical to the business. I personally suggest saving at least 30 days of metadata on critical transactions, perhaps more. Furthermore, when an incident does occur, your team may know where to start the investigation, but there is really no way to know where it will lead.
  • All logs should be indexed and your team should be indexing and measuring your own data as well. Measuring your own data will help you get a better understanding of what is causing bottlenecks and how your team is performing with regards to efficiency and your KPIs.


  • When it comes to creating and fine-tuning SIEM rules, using big data security analytics can help your team understand and fine-tune the rules and thresholds to the right level to find the bottlenecks. This should become a routine task.
  • When teams receive an alert, they typically get only the metadata which is simply the tip of the iceberg. In order to be able to decide if it is a real breach or false positive, your team should be able to explore the raw data to understand the bigger picture and context. Doing so can help you simplify large amounts of the investigation and make sure your different tiers of analysts are investing their efforts in a coordinated and balanced manner.
  • Use big data analytics enhancement tools to help find abnormal occurrences and trends. It’s all about the use case so if you define clear use cases, everything can be done simply, even defining the right visualization to make sure the human eye can catch it.


  • Invest mainly in planning and designing the incident response process to automate and collect all necessary data with clear visibility into the context, business impact, and timeline.
  • Everything should be logged and measured. This the only way to make sure your incident response process is efficient and mature for hunting, creating rules and the investigation process.
  • Segregate the different analyst tiers to maximize team efficiency and importantly, to prevent skilled-staff burnout.
  • Collaborate and digest threat intelligence into the incident response process.

Business-Driven Security

While it’s possible to create an efficient and mature incident response process using big data, it’s not enough. Without business alignment, you’ll continue to focus time and resources in the wrong places. The solution is to create business-driven security, a risk-based approach to IR. Security and business are typically in constant conflict but to really protect the organization, collaboration is key when it comes to IR planning and incident handling.

Start by understanding where and what your critical business processes are. Then for each of them, work side by side with the business information security officer (BISO) to implement controls, enhance visibility and understand and map the attack vectors and create mitigation plans.

Based on your analysis, you can create a risk-based approach to the incident response process by:

  1. Risk Impact – Prioritizing based on the risk-impact instead of “first in, first served”. This means mapping assets and users that are crucial to the critical business process, implementing it as part of all playbooks and prioritizing alert handling based on the impact with a focus on what matters most.
  2. Mitigation and Response Tools – Defining and implementing mitigation and response tools as part of playbooks. It’s one thing to decide on business risk mitigation but it’s another thing entirely to have all the necessary tools to respond.
  3. Big Data Collection and Analysis – Defining the relevant data sources, collecting them in the Big Data platform and defining and creating specific dashboards for each business process. The idea is to provide the real context and understanding of what’s going on in the business process that may be compromised.
  4. 24/7 Monitoring of Mission Critical Processes – Conducting ‘round-the-clock’ monitoring for mission-critical business processes such as ATMs. In some cases, this can produce a big boost to the business’s bottom line.
  5. Ongoing Training and Simulation – Engaging analysts in routine training exercise via tabletops, wargames, technical exercise and system health checks.

It doesn’t matter how good and efficient your incident response process is, if you don’t engage the business interests, focus is lost and the goals of security and business collide. Security leaders must find a new way of dealing with the existing challenges, one that creates an effective IR process while taking business priorities into account.

Shai Gabay is Chief Innovation Officer at Cyberbit. Shai has over ten years experience in information security leadership, most recently serving as SOC Manager at one of Israel’s leading commercial banks.