Schema mapping bridges your data between its source and its destination. It renames and reshapes the fields and values in log and event data standardizing it for ingestion at the destination, in most cases, a SIEM. Without it, detections and dashboards can't interpret the data properly.
This scenario might sound familiar. A team connects a new firewall. Events start arriving in the SIEM, but no detection rules match. The data isn’t the problem, the format is. One logging tool uses the source address field name; another uses source IP; a third uses a different field name. Every tool is looking for the same value under a different name, so none of them find it.
Schema mapping fixes this. It unifies incoming log and event data (field names, types and values) into the format each destination needs.
What Is Schema Mapping Used For?
Renaming fields is only the start. A working mapping gets three things right for every field: the name, the data type and the value format.
But there’s more to the puzzle:
- The destination expects the port as a number, but the source sends it as text.
- The destination and source use different terms to describe their events. For example, the source logs a dropped connection as "deny" when the destination expects "blocked".
- The source writes timestamps without a year or time zone. The destination needs both.
Schema mapping ensures that all of these differences are covered before arriving at the destination.
Why Would I Want to Map One Log Source to Multiple Versions of a Schema?
Every security and observability platform organises event data its own way:
- Splunk uses the Common Information Model (CIM).
- Microsoft Sentinel uses the Advanced Security Information Model (ASIM).
- Google SecOps uses the Unified Data Model (UDM).
- IBM QRadar normalises events into its own event model.
- Observability tools increasingly follow OpenTelemetry semantic conventions.
These models aren't interchangeable. If you’re sending your logs to multiple destinations, each one needs its own mapping.
If this is not addressed, schema drift occurs.
What Is the Result of Mapping an Event?
Let’s consider this scenario: a firewall event with a source field. The source value may be an IP address or a hostname. An effective mapping checks which one it is and routes it to the appropriate IP address field. Events are reassembled correctly for each destination and the original data is saved for review.
Now extend this scenario to timestamps, actions, network zones, device details and any context your team adds. Cloud audit logs can be even harder because the fields in each event change depending on what was logged.
Where Do Schema Mappings Silently Go Wrong?
The biggest failures don’t prevent pipeline processing, they go undetected until the data makes it to its destination. A change could include renaming a field in the data, resulting in an empty destination field for every event. A SaaS provider could add a new type of audit event which lands in the data lake but is unusable because it can’t be queried.
Because data keeps flowing, these problems usually show up only when a dashboard looks wrong or an alert doesn't fire. A running pipeline doesn't mean the data is right. You need additional controls to assess the mapped data.
Where Should Schema Mapping Take Place in Your Pipeline?
Some organizations map at the source, others at query time. Fleak helps teams map in the pipeline.
|
Where you map |
Upside |
Downside |
|---|---|---|
|
At the source (agents, devices) |
Data leaves already formatted |
Every change means updating the whole fleet |
|
At query time |
Raw data stays untouched |
Every destination repeats the work, and every query pays for it |
|
In the pipeline |
One place to make changes |
Needs a pipeline layer you trust |
Mapping in the pipeline helps teams manage events flowing from sources to destinations by providing a single point for data processing and delivery tasks. When a source is added or changes, you fix the mapping once and every destination gets correctly formatted data. You can also choose to drop certain data before it reaches downstream storage, which cuts cost.
What Is Good Schema Mapping?
This is what good schema mapping looks like:
- You can compare any event before and after processing and see every change.
- You can audit parsing, normalisation and enrichment independently.
- The original event is kept, so analysts can check it and correct errors.
- Mappings are tested on real data, including edge cases and malformed events.
- A person reviews each mapping before it reaches production.
AI can help, but we recommend that a human has the final say before it goes live.
Where Fleak Fits
Fleak processes machine data before it reaches your data destination, such as SIEMs or data lakes. Fleak can be used alongside other systems to improve the quality of data sent to them. We enable users to create pipelines fast while having control over mappings and other configurations before sending the data to other systems.
Our aim is to provide each tool useful information and ensure that the process by which the tools are provided with this information is transparent to those carrying out the process.
Frequently Asked Questions
What Is Schema Mapping in Cybersecurity?
It's the practice of mapping security event attributes and schema in the original data format to an output schema supported by a SIEM, data lake, or other observability tool to enable use of the original data for searches and other security detection activities.
What Is the Difference Between Data Mapping and Normalization?
Normalization involves converting values in a schema to a common set of values (converting "deny" and "blocked" to a common single value). Schema mapping includes all aspects of converting from one schema to another, including changing field names and types and normalizing values, for one or more target schemas.
What Is ASIM?
Microsoft Sentinel's Advanced Security Information Model (ASIM) is a standard data model that Microsoft Sentinel applies to incorporated data sources to enable shared queries and detections across sources.
Where Should Schema Mapping Take Place: Source, Query, or Pipeline?
Mapping at the source or device level involves making fleet-wide updates each time. Mapping at query time saves on repeat effort but requires more work at query time for each destination. Mapping in an intermediate pipeline allows you to do most of the work at once and then provide outputs in different formats for different destinations when changes are made at the source.
How Can You Test Schema Mappings?
Test the pipeline with sample data events, including invalid or unexpected formats, and check the field names and values before and after processing, in addition to confirming that the pipeline ran successfully.
Try It on a Source You Already Own
Bring a log data sample and documentation about the destination format. We will help you map the data using your sample events and demonstrate the transformation process to help you understand how the data will look at the destination.