Data Mapping vs. Schema Mapping: What's the Difference?

Data mapping and schema mapping are sometimes used interchangeably, though they are different processes․ Data mapping and schema mapping both refer to linking two systems․ There are benefits and drawbacks to both data and schema mapping․ Knowing more about data and schema mapping can save your SecOps team weeks of project redesign when implementing Splunk‚ Microsoft Sentinel‚ or QRadar․ Fleak, a machine data fabric, sits between your logging and SIEM systems․ Your Splunk‚ Microsoft Sentinel‚ or QRadar systems can remain in place while you address data mapping and other costs further upstream in your data pipeline․

What Is Schema Mapping?

Schema mapping is the task of aligning data models․ Part of schema mapping is answering the question, "What is the relationship between fields in the source and target models?"

Take a firewall log with fields srcip‚ dstip‚ and act , for example. SIEMs have different requirements:

See the Splunk Common Information Model documentation and the Microsoft Sentinel ASIM documentation for complete lists of fields․

IBM QRadar works differently. Its DSMs parse raw events into normalized properties, such as source and destination IP, and custom event properties cover fields the DSM doesn't extract.

Schema mapping addresses field names‚ data types‚ and field relationships․ It is a plan․ It does not involve any specific data record․

What Is Data Mapping?

Data mapping is the process of planning and implementing a solution for transporting data values from the source to the destination and‚ if needed‚ transforming the data values․ Data mapping addresses a different problem than data profiling: what will happen to the data?

Using firewall logs‚ data mapping would include:

• Converting the value ALLOW in the act field to a value that matches your detection rules‚ such as allowed.
• Parsing a timestamp of 10/01/2026 14:03:22 in ISO 8601 format.
• Separating a user@domain address into user and domain fields in a database.
• Enriching IP addresses with geolocation and asset information.
• Excluding fields and events from ingestion that may increase costs but do not provide security insights․

Data mapping activities include schema mapping and other activities such as value transformation‚ parsing‚ data enrichment‚ data filtering‚ and data routing operations․

The Key Differences

Why the Distinction Matters for SecOps

Detections Depend on Both

A Splunk correlation search rule‚ a Sentinel analytics rule‚ or a QRadar rule that detects a denied connection will match only if there is a field in the source data with the expected name (schema mapping) and the expected value (data mapping)․ Incorrect naming but correct values would cause the rule to match nothing․ Correct naming but incorrect values would prevent the rule from detecting a denied connection․

Schema Drift Causes Problems for These Methods

When a field is renamed or nested fields are added by vendors‚ schema mapping issues occur․ When data values change in format (for example‚ a different time stamp format or response code) data mapping issues occur․ Missing fields are a common problem with schema changes․ Inaccurate but seemingly correct data is a common problem with data format changes and can be more difficult for an analyst to identify․

Each SIEM Has Its Own Model

Splunk's CIM‚ Microsoft Sentinel's ASIM‚ and QRadar's DSM normalization models have different structural components․ A process to send data to multiple SIEMs or migrate from one SIEM to another requires you to create separate schema mappings for each SIEM output‚ as well as common data mappings․

They Should Have Different Owners and Different Test Cases

Schema mapping is a modeling task that requires knowledge of the target SIEM's data model․ Data mapping is an engineering task that requires sample data and testing using live traffic․ Combining these tasks often results in an unusable mapping spreadsheet․

How the Two Work Together in a Security Data Pipeline

In a SecOps workflow‚ the general process will be:

1. Ingest log data directly from devices or services like firewalls‚ endpoint protection products‚ identity providers‚ and cloud services․
2. Group and parse log entries to process similar events together․
3. Use schema mapping to map target SIEM fields to those in the source security model․
4. Use data mapping techniques to transform and add information to data and remove 4. unnecessary data․
5. Send results to appropriate destinations‚ including Splunk‚ Sentinel‚ and QRadar for detection-related data‚ and less expensive storage solutions for other data․

Skip step 3‚ and your data will have inconsistent shapes․ Skip step 4‚ and your data will have consistent shapes but inconsistent contents․

Why This Has Been So Slow, and How It Is Changing

Traditionally‚ both types of mapping are done manually․ An engineer reviews vendor manuals and other materials‚ codes parsers and other mapping software‚ and implements and tests the mapping for each new source․ Onboarding one source may require several weeks of work․ Ongoing maintenance is needed to address changes to the source's schema․ With hundreds of sources‚ the backlog of unmapped sources never clears‚ and SIEM systems receive unmapped and partially mapped data at high cost․

AI tools can reduce costs․ Tools can be used to parse logs‚ identify common patterns in logs‚ suggest mappings to the new SIEM system schema‚ and develop scripts to transform log values․ Humans must approve the suggested actions‚ but faster processing times do not lead to loss of control․

How Fleak Approaches It

Fleak is a machine data fabric for transforming and transporting data from any source to any destination․ Fleak complements your SIEM solution: Splunk‚ Microsoft Sentinel‚ or QRadar‚ for example․ Data remains in and is searched from your Splunk‚ Sentinel‚ or QRadar environment‚ and Fleak provides more cost-effective and refined data to that environment․

• Schema mapping using the Mapper App․ Fleak’s Mapper App helps users create separate schema mappings for different log formats․ AI tools can help map fields from log formats to a target schema․
• Data mapping within the pipeline․ Data mapping tools like Fleak's Brain Layer can automatically build pipeline configurations for a source schema in about three minutes․ ZephFlow's execution layer can execute these pipeline configurations on a CPU with nearly zero delay for tasks such as parsing‚ data conversion‚ and filtering and routing․
• Self-healing․ Issues with field names or other upstream problems will not disrupt the runtime unless the operator intervenes․ This means that changes to field names or other values will not prevent your detections from working properly․
• Human oversight․ All AI decisions can be reviewed and approved by humans․
• Destination agnostic․ Fleak does not replace your SIEM․ Dual writes can help you ensure that signals are equivalent during the Fleak implementation or SIEM replacement process․

The net effect is that new vendors can get products on the market quickly rather than after long delays‚ and existing team members can work with more products․

A Quick Way to Decide What You Need

• Your fields do not match those in your SIEM's data model․ Consider schema mapping as a solution․
• Your fields match‚ but other data are inaccurate‚ incomplete‚ or costly to obtain․ Consider data mapping․
• You are onboarding numerous sources to Splunk‚ Sentinel‚ and QRadar tools․ You need all of these and should automate them all․
• You're transitioning between SIEMs or reporting to multiple SIEMs․ Create a single data mapping and a schema mapping for each destination SIEM․
• Your pipelines fail due to vendor format changes․ Consider tools that help detect and fix issues with format mappings․

Key Takeaways

• Schema mapping is the process of mapping structures․ Data mapping is the process of mapping values across systems․ Schema mapping is a component of data mapping․
• Splunk‚ Sentinel‚ and QRadar detections require an alert to be true․
• Schema drift and value drift present different challenges and should both be tested․
• Automating and human reviewing both processes eliminates the new-hire backlog while maintaining management control․

Frequently Asked Questions

What Is the Difference Between Schema Mapping and Data Mapping?

Schema mapping involves specifying how schema elements from the source system correspond to schema elements in the target system․ Data mapping involves copying and converting data values and includes schema mapping as a task․

Is Schema Mapping a Type of Data Mapping?

Yes․ Data mapping is the general process․ Schema mapping is one aspect of data mapping‚ along with other functions such as value transformation‚ data parsing‚ data augmentation‚ and data filtering and routing․

How Do I Use the Splunk CIM to Index Logs?

Map raw fields to their corresponding CIM field names‚ like srcip to src_ip․ Standardize data in fields like action to ensure consistency in your correlation searches․ Both mappings are required to get results from CIM searches and data models․

How Do I Map Logs to Microsoft Sentinel ASIMs?

Establish a mapping between raw field names and ASIM field names‚ like srcip and SrcIpAddr․ Transform raw field values into the formats used by ASIM․ Test your analytics rules to make sure you get results using the mapped data․

How Does IBM QRadar Normalize Log Data?

QRadar uses DSMs to parse incoming events into normalized properties. Custom event properties cover fields the DSM does not extract.

What Is the Difference Between Data Mapping and ETL?

ETL is a movement concept that involves extracting and loading data‚ with a transformation step․ Data mapping describes how source data fields and values are mapped to target fields and values․ Data mapping is a part of an ETL or streaming process․

Why Do Detections Fail When a Vendor Changes Its Log Format?

A renamed field disrupts the schema mappings and causes the field to be excluded․ A change in value type disrupts the data mappings and causes the field to have invalid values․ Both types of changes can result in missed detections․