Kibana template(.json) is right?

[demantos]

I use IIS log for ELK and modify some config of logstash and kibana template.

• w3c-extended-iis.json
  -> just modified "index" name same as "index" name of logstash config file
• w3c-extended-iis.conf
  -> at filter, csv columns field modified to my iis log
  -> at output, "index" name is modified
  • original : index => "logstash-%{[type]}-%{+YYYY.MM.dd}"
  • modified : index => "iis_log_merge"

I configure all log is indexed one indice.

Result is good, but some panel data is broken like below.

I use filed name like below

REQUEST URI : cs_uri_stem
URI QUERY : cs_uri_query
USER-AGENT : cs_user_agent
COUNTRY : geoip.country_name

Default template use field name like cs_uri_stem.raw but when i use default template, result is always "Missing field".
So, I modified field name like above.

Please advice to me.

Regards

[cvandeplas]

Hi there,

What you are experiencing is the effect of tokenization.
If you try to index "Hello world" then Elasticsearch will explode that based on pre-defined tokens (spaces, dots, comma,..) , so he will store "hello" and "world" separately.

When we want to build Top X tables we however do not want that, not in the domain of forensics.

To modify this behavior you need to define "elasticsearch templates" to say that for this and this, he should not use tokenization, or he should at least give access to the original full field.
As changing templates is a more complex operation logstash does this magic for us.

When logstash connects to elasticsearch and creates a new index it will configure the index with a (default) template adding a ".raw" to each field.
However this template is only applied to indexes that start with "logstash-".

As your index name is "iis_log_merge" it this does not receive the .raw fields.

There are 2 options for you:
1/ (the difficult one) modify the template logstash gives and apply it to your index

2/ (the easy one) remove your index, and re-index your data with an index called "logstash-iis_log_merge" for example. All your fields will receive the magical .raw for free.
Also don't forget to also restore the original Kibana dashboard so that it also contains the .raw in the fields.

Kind regards
Christophe