Data Quality: The Unsung Hero of Effective Security Operations

As organizations increasingly rely on AI for enhancing Security Operations Center (SOC) workflows—such as automating threat triage and generating incident reports—one unsettling question arises: Is the effectiveness of SOC more contingent on the large language model (LLM) employed, or does it rest heavily on the quality of the security data being processed?

Recent academic discussions present a divided perspective. Research published in Frontiers in Artificial Intelligence highlights substantial variability in the accuracy and clarity of outputs among prominent models like Claude 3.5 Sonnet and ChatGPT 4o. In contrast, findings from the Information Systems journal stress that high-quality training and testing data are vital for trustworthy AI applications, focusing on dimensions such as accuracy, completeness, and consistency.

Recent investigations favor the argument for superior data quality, establishing it as a pivotal element governing the successes in security operations. Research shows that high-fidelity network evidence can enhance security performance by 2-4 times across key investigative metrics, prompting CISOs to reevaluate their operational strategies.

Understanding the Impact of Data Quality

The implications of these findings extend well beyond theoretical discussions. For CISOs, prioritizing high-quality data manifests in tangible operational benefits:

• Concrete telemetry reduces mean time to respond (MTTR), enhancing responsiveness to threats.
• Precise data control leads to better cost management, particularly regarding token expenditures.
• Demonstrable returns on security investments affirm the effectiveness of the security team, strengthening justifications for resource allocation.

With this evidence, security professionals can influence SOC architecture decisions, effectively lower analyst turnover rates driven by alert fatigue, and present measurable security effectiveness to executive teams and boards.

Evaluating Data Quality in Security Operations

The Provably Better Data research initiative set out to quantify the true contributors to AI effectiveness within enterprise security. The project employed a rigorous test framework to evaluate model performance against two operational benchmarks:

• A Capture the Flag (CTF) scenario featuring a 44-question investigation based on a Volt Typhoon attack campaign.
• An incident response analysis task utilizing data from a Salt Typhoon dataset.

The tests manipulated data quality as the sole variable, processing four distinct sources of network telemetry under identical experimental conditions:

• Corelight enriched logs
• Open source nDPI firewall logs
• Snort 3 intrusion detection system alerts
• NetFlow connection telemetry

Each dataset underwent multiple iterations in compliance with an Open Cybersecurity Schema Framework (OCSF) normalized schema. The generated responses from LLMs, including Anthropic Claude Opus 4.6 and Google Gemini Pro 3.1 Preview, were compared across identical prompts generated in controlled runs.

The Limits of AI: Evidence Boundaries

Despite their advanced reasoning capabilities, frontier language models’ quality of conclusions hinges on the evidence available. If data lacks detailed protocol-level context, AI cannot derive insights from what wasn't gathered. The investigation must rely on the context provided by data sources. For instance, a CTF question exploring the NetBIOS name for a given IP address yielded different results based on data context:

• Corelight logs: yielded the correct response, FINANCE01, by extracting information from the server_nb_computer_name field in NTLM logs.
• Firewall logs: recognized NTLM activity but fell short in interpreting the details, failing to provide the correct answer.

These gaps in data can hinder operational effectiveness. While firewall logs offer some visibility, a lack of comprehensive protocol data leaves investigations incomplete. The presence of sufficient evidence enables AI to provide actionable insights, reducing the burden on human analysts.

The Case for High-Fidelity Data

Measurements from the project clearly demonstrate the correlation between data quality and security outcomes. Participation in the CTF benchmark revealed stark differences in accuracy based on source data:

• Corelight logs achieved a 95.2% accuracy rate.
• Firewall logs reflected a 58.3% accuracy rate.
• Snort 3 alerts delivered a 39.4% accuracy rate.
• NetFlow logs finished with a mere 25.8% accuracy rate.

This disparity indicates that access to richer telemetry directly enhances model performance, enabling higher accuracy in responses to investigative queries. Corelight logs empowered each LLM to address all 44 questions, while NetFlow records limited responses to just 15.

The incident response assessments echoed these findings, with significant variance in evidence coverage:

• Corelight logs provided a 90.3% evidence coverage rate.
• Firewall logs reached 61.3%.
• NetFlow records captured 30.9%.
• Snort 3 alerts stood at only 21.2%.

For critical Tier 1 investigation requirements, Corelight logs enabled models to address 91.7% of necessary questions, compared to only 18.3% from NetFlow logs and 10% from Snort 3 alerts. The result: High-quality data delivered a fivefold increase in critical incident visibility over basic flow records.

Investigation timelines also reflected significant improvement with enhanced data quality. When equipped with Corelight logs, LLMs completed investigations in an average of 14.7 minutes, while NetFlow logs extended this time to 27.0 minutes and firewall logs to 26.3 minutes. The finding: Data of lower quality nearly doubled investigation times as models faced repeated retries.

Strategies for Security Leaders

As enterprises utilize AI to expedite threat detection and incident management, the automation process does not occur instinctively; it hinges on complete, structured, and protocol-aware data.

The results indicate that improving models or complex prompt engineering won't resolve foundational data inadequacies. SOC leaders planning for future operational investments should thus prioritize data quality over model choice to enhance outcomes.

For a thorough examination of the methodologies and findings, security leaders can access the complete experiments outlined in the Provably Better Data white paper available on Corelight’s platform.

Corelight Network Detection and Response: Improving security outcomes by 2-4x necessitates access to high-fidelity network evidence as a prerequisite for effective AI-driven security. Discover more here.