The build or buy decision for SOCs has long been a challenge for enterprises. On one hand, most companies want the control and customization that comes with building an internal SOC but don’t want to miss out on the threat intelligence and incident response features that most external providers have become masters of. Therefore, we have seen the dramatic rise of the “hybrid SOC” or a SOC that is staffed with a mix of internal and external people(MDR/MSSPs).
In this blog, we will:
1. Describe what organizations use hybrid SOCs and why
2. Learn how to leverage the strengths of both MDR providers and SOCs to create a powerful hybrid security model.
3. Explore the impact of automation and AI in a hybrid SOC environment.
4. Discover specific examples of how a Hybrid SOC has worked effectively for a Top 5 Pharma company.
Who uses hybrid SOCs?
The degree to which an organization insources is generally correlated with that organization’s size, with Fortune 100s being more likely to insource than a 500-employee SMB. However, this is a trend and not a rule. We serve customers with 1,000 or less employees who have 24x7 SOCs that are staffed 100% internally. For hybrid SOCs, we tend to see their implementation as a sign of cyber maturity. We can look at people, process, and technology as lenses for why someone would want to outsource vs insource.
For people, typical functions that are outsourced are:
· Detection engineering
· Pen testing
· Threat hunting
· Tier 1 + Tier 2
· 24x7 monitoring + response automation
For process, we all know that MDR typically is a little r –recommendations and early response. Meanwhile, the Big R of Response is often handled and coordinated internally.
For technology, the decision centers around who ‘owns’ tooling from the observability & detection layer to ticket management to response capabilities.
This blog will zoom in on hybrid SOCs that have:
1. Detection tools ‘owned’ internally
2. External MDR providers who escalate identified threats to Internal SecOps team members
Why Choose the Hybrid Model – Advantages and Disadvantages
Let's analyze the challenges and advantages of hybrid SOCs across the dimensions of Scale, Speed, Context, Coverage, Consistency, and Communication:
Scale: Manual Effort to Investigate
Challenge: The volume of security alerts can overwhelm even the most sophisticated SOC. Hybrid SOCs must effectively distribute the workload between the internal team and the MDR to avoid excessive manual effort. If the division of labor isn't clear, or if automation isn't effectively utilized, investigations can become bogged down, leading to delays in response. This is especially true when dealing with complex or sophisticated attacks that require deep analysis.
Advantage: A well-designed hybrid SOC can leverage automation and the MDR's resources to handle the initial triage and investigation of alerts. This reduces the manual burden on the internal team, allowing them to focus on high-priority incidents and strategic initiatives. The MDR's scale and dedicated resources can also handle large-scale investigations more efficiently.
Speed: Hackers Moving Faster
Challenge: Attackers are constantly evolving their tactics and automating their attacks, making speed a critical factor in security operations. A hybrid SOC must ensure rapid detection, response, and containment of threats. Delays caused by communication breakdowns, unclear escalation paths, or inefficient processes can give attackers the upper hand.
Advantage: By combining the 24/7 monitoring capabilities of an MDR with the internal team's knowledge of the organization's environment, a hybrid SOC can significantly improve response times. The MSSP/MDR's expertise in incident response and threat intelligence can also accelerate the investigation and containment process. Clear SLAs and well-defined escalation procedures are crucial for maintaining speed.
Context: What Do We Know About People and Access to Systems? How Are We Actively Documenting and Sharing in a Dynamic Method?
Challenge: Understanding the context of security events is crucial for effective investigation and response. This includes knowing who has access to which systems, what their typical behavior is, and what the business impact of a potential compromise might be. Maintaining this context in a dynamic and shared manner between internal and external teams can be difficult. Static documentation quickly becomes outdated.
Advantage: A hybrid SOC can leverage the internal team's deep understanding of the organization's environment, users, and business processes to provide valuable context to the MDR. Implementing a shared knowledge base, collaborative platforms (or tool like Salem), and real-time communication channels can facilitate dynamic information sharing.
Coverage: You Want to Understand Who Has Your Back
Challenge: In a hybrid SOC, it's essential to have a clear understanding of roles, responsibilities, and escalation paths. Ambiguity about who is responsible for what can lead to confusion and delays during critical situations. Trust and clear communication are crucial for ensuring that everyone is on the same page.
Advantage: A well-defined RACI (Responsible, Accountable, Consulted, Informed) matrix and clear SLAs with the MDR can establish clear lines of responsibility. Furthermore 24/7 monitoring of alerts with a hybrid model helps alleviate coverage concerns.
Consistency: Getting Alerts at Various Times Throughout the Day/Week and Not Enough Information Pulled Together on the Alerts
Challenge: Inconsistent alert handling and a lack of correlation between alerts can lead to missed threats and inefficient investigations. If the MDR and the internal team use different tools and processes, it can be difficult to achieve consistent alert handling and information sharing.
Advantage: Standardizing alert handling procedures and establishing clear communication protocols can ensure consistent and efficient alert management. The MDR's expertise in threat analysis and correlation can also improve the accuracy and relevance of alerts.
Communication: Where to Share Oral History, Knowledge, Successes
Challenge: Effective communication is essential for the success of a hybrid SOC. This includes not only formal communication channels but also informal knowledge sharing and the documentation of lessons learned. Building a culture of open communication and collaboration between internal and external teams can be challenging.
Advantage: Establishing regular meetings, shared communication platforms (e.g., Slack, Microsoft Teams), and a centralized knowledge base can facilitate effective communication and knowledge sharing. Encouraging open dialogue, documenting incident response procedures, and sharing lessons learned can help to build a strong and collaborative team. Celebrating successes and recognizing contributions can also foster a positive and productive working relationship.
Despite the complexities of integration, collaboration, and potential cultural challenges, organization choose a hybrid SOC model because the advantages can often significantly outweigh the disadvantages, especially when implemented thoughtfully. Organizations often choose a hybrid SOC model to achieve a balance between cost, capability, and control. It allows them to leverage external expertise and resources to enhance their security posture while retaining internal knowledge and control over critical security functions. By carefully addressing the potential challenges through clear communication, well-defined roles, and strong partnerships, organizations can maximize the benefits of a hybrid SOC and achieve a more robust and cost-effective security program.
F100 Pharma: MDR Optimization Use Case
In this section, we want to specifically highlight our client, an H-ISAC member, and their journey to optimize their hybrid SOC using Salem as a middleman between their internal operations and their MDR.
Background: Our client is a Fortune 100 Pharma company with a typical cybersecurity setup. Prior to implementing Salem, they sent alerts directly from their XDR to an MDR, and the threat notifications from the MDR come back across the fence.
Objective: This customer wanted to significantly reduce MTTR and increase their consistency in off-hour performance
Before instituting Salem, this customer had an ineffective relationship with their MDR. In order to affect change, our client had to backchannel with their MDR account management to try to get anything done. However, part of the problem was that the MDR had their contracted objectives (including MTTR) and other contracted SLAs so trying to influence outcomes better than contracted results was near impossible. Therefore, they started looking for other ways they can create influence outside of the existing system.
This is where Salem came in. The client contracted Salem to route all the same alerts that previously went directly to their MDR to Salem. They also created a list of priority alerts that they warranted immediate attention from their MDR. Now, Salem identifies a priority threat (which happens once or twice a day) and sends that notification to the MDR separately via a slack channel. MDR analysts then acknowledge receipt and close out with a conclusion.
This new process allowed the client to circumvent account management and get their priority alerts looked at immediately from the MDR.
Results: Reduction in MTTR by 5x.
Shared Value: MDR’s sub 2 min response to Salem alerts improves confidence and response times. Also, the client teaches Salem about important business context (through interacting with Salem’s interface) which is then exposed to the MDR, helping them support their investigation.
What did we observe?
This use case gave us three interesting insights into MDR challenges and how to influence successful outcomes.
1. Influencing priority
Observation: Client created a mechanism (a Slack channel) to directly communicate with the MDR analysts, which meant they could circumvent account management. This allowed for MDR analysts to acknowledge receipt in under 2 minutes, which resulted in an actual MTTR reduction for all alerts from 1 hour to 10 minutes.
What are the underlying mechanisms at play here?
1. An MDR has to be very flexible with how they accept data because they have multiple customers with different tech stacks. These differences in customer profiles can create delays in transiting data from where it starts to where it needs to go. For instance, we have seen it take 20 minutes for an EDR cybersecurity tool to signal to an XDR. This can add a considerable amount of time to when an alert is created to when it becomes available.
2. MDRs rely heavily on automation to be efficient in the aggregate but it can be inefficient for the individual alert.
3. There can be queuing issues. MDRs are responsible for multiple customers and have other priority schemes. It’s a shared service and first come is first served.
Conclusion: The hardest part of any MDR service is consistently and with speed, identifying all the right events, 24/7, 365 days a year. Any positive influence you can exert will improve outcomes for both parties.
2. Context still matters
Observation: Our client works periodically with Salem to train the system on business context and priorities. Salem enriches alerts and leverages context to better identify priority alerts to send to the MDR. The result is an increase in client confidence that the alerts they most care about are being reacted to.
What are the underlying mechanisms at play here?
1. Situational awareness: It’s impractical for MDR analysts to maintain situational awareness for all customers.
2. Not every alert is equally well supported by an MDR. Low and medium alerts require significant business insights to deduce the 1 in 100alert that matters and is therefore hard to investigate from an outsider perspective.
Consider this: Many alerts, particularly in the identity and cloud space, are adjudicated by understanding the real application and person behind the system name and account. In order for an MDR to be effective, they have to have access to information. So, finding ways to add context into the system purposefully will have a tremendous impact on the analysts ability to decide if something is a threat or routine business activity.
3. The human element
Observation: Before Salem, the client noticed off-hour MDR performance metrics to be noticeably worse. Salem sends threa notifications as events transpire, including Sunday afternoons. By instituting Salem, the client reduced occurrences of the Monday morning surprise.
What are the underlying mechanisms at play here?
1. The cost of being wrong: Our biased mind points to an image of a less skilled analyst not making the right choice. However, our observation was an analyst less willing to wake someone up for fear that they would be wrong.
2. Signs of life: MDRs don’t make a habit of signaling when their investigation starts. However, when a chat is started for every event that Salem surfaces, analysts are forced to act, giving them the push to escalate based on pre-set priorities.
Consider this: If a conversation has already started about an event, regardless of what started it, then the requirement of the analyst is to both acknowledge the investigation has begun and acknowledge the outcome and resolution. Therefore, the client now gets a signal that they were not privy to before – that someone is looking into something important. That is what created improved dynamic between analysts and the client. This interaction takes pressure off of the analyst making the initial notification during off-hours.
Ideas for optimizing your Hybrid SOC
1. Start chatting
Establish a direct line of communication with your MDR’s analysts, identify the alerts you would never want to miss, no matter what, and build automation to push those alerts into chat where you and the MDR’s analysts can both see them
2. Contextualize your data
When you read alerts, you see the meaning behind the data. Dedicate time every couple of weeks to review recent events that were meaningful to your organization. Then, find ways to tag the objects in those alerts with what you know about them. If you can automate this, even better!