Skip to content

Latest commit

 

History

History
150 lines (95 loc) · 3.89 KB

ESQuery.md

File metadata and controls

150 lines (95 loc) · 3.89 KB
Raw

ESQuery

ESQuery is the query parser bundled with 411 for querying Elasticsearch. Each ESQuery query consists of one or more "commands", separated by the pipe (|) character. During execution, each command is translated into an Elasticsearch DSL query and sent to the server. Any results that are returned get passed off to the next command in the chain.

Structure

Each query is structured as follows:

    Options* SearchCommand ('|' [AggCommand | JoinCommand])* ('|' [TransactionCommand])?

In English:

  • A series of 0 or more options
  • A SearchCommand
  • A series of 0 or more AggCommands or JoinCommands
  • An optional TransactionCommand

with each command separated from the next by a pipe (|) character.

Syntax

Options

Options that can be enabled/disabled within each query. Each option is prefixed by a $.

allow_leading_wildcard

  • Allows * or ? as the first character. Makes for slow queries!
  • Type: bool
  • Example: $allow_leading_wildcard:true

sort

  • A list of fields to sort on.
  • Type: List
  • Example: $sort:[time:ASC, type:DESC, duration:DESC]

map

  • A mapping of fields and what to rename them to.
  • Type: Map
  • Example: $map:[request_ip:ip, geoip_location:loc]

Unused

These options are managed by 411.

  • date_field: The name of the field used as the timestamp.
  • to: The latest date to pull results from. (Unix timestamp)
  • from: The earliest date to pull results from. (Unix timestamp)
  • size: The maximum number of results to return.
  • flatten: Flatten any nested structures into a flat key-value map.
  • fields: The list of fields to return.

Commands

Search

A standard Lucene QueryParser query. Almost all of the features are available in ESQuery. The ones that aren't primarily relate to scoring, and are listed here:

  • Fuzzy searches
  • Proximity searches
  • Boosting a term

Must always be the first command.

  • Syntax: QUERY_PARSER_QUERY
  • Example: level:99 type:normal
Lists

Inserts an array of values into a query. These arrays can be defined in 411 via the Lists functionality. ESQuery runs a terms filter on the values that are provided.

  • Syntax: KEY:@LIST_NAME
  • Example: src_ip:@bad_ip_list

Join

Extracts the values from the previous command and makes them available to the following query.

  • Syntax: 'join' 'source:'SOURCE_FIELD 'target:'DEST_FIELD QUERY_PARSER_QUERY
  • Example: type:tcp | join source:src_ip target:dst_ip flags:0

Transaction

Combines documents with matching values for a given field. Must always be the final command (if used).

  • Syntax: 'trans' 'field:'FIELD_NAME
  • Example: trans field:request_uuid

Aggregation

Standard Elasticsearch aggregations. Multiple aggregations can be chained to nest them. ESQuery will return the bucketed data in a table. Any parameters that an aggregation takes can be passed in after setting the field.

Supported aggs: terms, sig_terms, card, max, avg, sum

  • Syntax: 'agg:'AGG_TYPE 'field:'FIELD_NAME (AGG_OPT':'AGG_OPT_VAL)*
  • Example: agg:terms field:user_id min_doc_count:50

Examples

Match all documents.

*

Get a count of requests to abc.com bucketed by ip_addr.

host:abc.com | agg:terms field:ip_addr

Find all requests to abc.com sorted by ip_addr and date.

$sort:[ip_addr:ASC, date:ASC] host:abc.com

Find all users with the same email as bob.

user:bob | join source:email target:email

Find and group all log lines associated with requests that came from 10.0.0.5.

ip_addr:10.0.0.5 | trans field:request_uuid

Find any documents that have a value from @include but not a value from @exclude.

tag:@include -tag:@exclude

Find the most common useragent.

* | agg:terms field:user_agent size:1