Log IQ

Log IQ allows identifying the format of unstructured log data, transforming it into structured columns within ingested events in JSON format. This helps in easy and optimized query, search, debug and visualize the data.

How Log IQ works

Log IQ requires specific HTTP headers when ingesting data to properly identify and parse log formats:

Required Headers

• X-P-Log-Source - Mandatory - Identifies the log format name (e.g., syslog, nginx_access, zookeeper)
• X-P-Extract-Log - Required for unstructured data - Specifies which field in the incoming JSON contains the raw log text (typically log)

Processing Logic

For structured data:

• Only X-P-Log-Source is required
• Parseable assumes the data is already in a structured format
• The specified format is used for validation and additional processing

For unstructured data:

• Both X-P-Log-Source and X-P-Extract-Log are required
• Parseable extracts the raw log text from the field specified in X-P-Extract-Log
• The system applies regex patterns based on the format specified in X-P-Log-Source
• If the content matches the format, it's parsed into structured fields
• If the content doesn't match the format, the original value is retained in the specified field

Outcome

• After successful format detection, a p_format field is added to the log event containing the log source name
• The dataset info is updated with an array of detected log sources
• Parseable UI (Prism) automatically displays filters on the p_format field
• If the log format is not detected, p_format_verified=false is added to the event
• Data is always ingested, regardless of format detection success

Note: Even if your unstructured data doesn't match any of the supported formats listed below, you must still specify both headers. Choose the format that most closely aligns with your log structure.

Example: Processing a Syslog Entry

Let's walk through a practical example of how Log IQ processes a syslog entry:

1. Original log sent by an agent (e.g., FluentBit):

{
    "log": "2025-07-11T14:57:33.000111+05:30 node01 exporter[9012]: [2025/07/11 14:57:33] [error] [output:http:http.8] Failed to push metrics to endpoint /metrics"
}

2. HTTP headers used when sending to Parseable:

X-P-Log-Source: syslog_log
X-P-Extract-Log: log

3. Parseable's processed output:

{
  "body": "[2025/07/11 14:57:33] [error] [output:http:http.8] Failed to push metrics to endpoint /metrics",
  "log": "2025-07-11T14:57:33.000111+05:30 node01 exporter[9012]: [2025/07/11 14:57:33] [error] [output:http:http.8] Failed to push metrics to endpoint /metrics",
  "log_hostname": "node01",
  "log_pid": "9012",
  "log_procname": "exporter",
  "log_syslog_tag": "exporter[9012]",
  "p_format": "syslog_log",
  "p_format_verified": "true",
  "p_src_ip": "127.0.0.1",
  "p_timestamp": "2025-07-11T09:20:23.019",
  "p_user_agent": "PostmanRuntime/7.44.1",
  "timestamp": "2025-07-11T09:27:33"
}

In this example:

1. The agent (like FluentBit) collects the log and places it in the log field
2. Parseable receives this with the appropriate headers
3. The system identifies it as a syslog format and extracts structured fields:
   • log_hostname: The host that generated the log ("node01")
   • log_pid: The process ID ("9012")
   • log_procname: The process name ("exporter")
   • log_syslog_tag: The syslog tag ("exporter[9012]")
   • body: The actual message content
4. Parseable adds its metadata fields:
   • p_format: The detected format ("syslog_log")
   • p_format_verified: Confirmation that the format was successfully detected
   • Other p_ prefixed fields with request metadata

This structured data is now ready for efficient querying and analysis.

Supported Formats

Parseable Log IQ supports a wide range of log formats. You can specify these formats using the X-P-Log-Source header when ingesting logs. The currently supported formats include:

Each format has specific patterns and fields that are extracted. When a log matches one of these formats, Parseable automatically extracts the structured fields and makes them available for querying and analysis.

Extracted Fields by Format

Below are the fields extracted for each supported log format:

This is not an exhaustive list of all fields for all formats. Each format has specific patterns and may extract additional fields based on the log content. When using Log IQ, you can explore the extracted fields in the Parseable UI or through SQL queries.