syslog input drops messages with different ISO 8601 time formats

[Trolldemorted]

Logstash is unable to handle syslog input from systems with different time formats going into the same ES index.

• Version: 7.8.1
• Operating System: Official docker containers

version: "3"
services:
  elasticsearch:
    image: elasticsearch:7.8.1
    environment:
    - discovery.type=single-node
  kibana:
    image: kibana:7.8.1
    ports:
    - "5601:5601"
  logstash:
    image: logstash:7.8.1
    volumes:
    - ./config-dir:/config-dir
    command: logstash --path.settings=/config-dir
    ports:
    - "5044:5044"

• Config File (if you have sensitive info, please remove it):

input {
  syslog {
    port => 5044
  }
}
output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "rsyslog-%{+YYYY.MM.dd}"
  }
}

Steps to Reproduce:

• send a log message with a full timestamp: <167>2020-07-30T20:00:59.090Z Esxi01 Vpxa: verbose vpxa[C6BCB70] [Originator@6876 sub=VpxaHalCnxHostagent opID=WFU-1b1ac72d] Completed WaitForUpdatesDone callback
• send a log message with a short timestamp: <86>Jul 30 22:14:56 ubnt sudo: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/vyatta/sbin/ubnt_vtysh -c show ip route summary json

The first message causes logstash to mark the timestamp property as date, but logstash fails to supply a date when handling the second message:

elasticsearch_1  | "stacktrace": ["org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [timestamp] of type [date] in document with id '1RnDoXMB-RYw4kgdDFU1'. Preview of field's value: 'Jul 30 22:14:56'",
elasticsearch_1  | "at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:316) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:488) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:618) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:427) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:112) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:267) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:795) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:772) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:744) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:267) [elasticsearch-7.8.1.jar:7.8.1]",   
elasticsearch_1  | "at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:157) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:202) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:114) [elasticsearch-7.8.1.jar:7.8.1]",  
elasticsearch_1  | "at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:81) [elasticsearch-7.8.1.jar:7.8.1]",   
elasticsearch_1  | "at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:895) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:109) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:374) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:297) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$24(IndexShard.java:2802) [elasticsearch-7.8.1.jar:7.8.1]",        
elasticsearch_1  | "at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:113) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:285) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:237) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2776) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:836) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:293) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:256) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) [x-pack-security-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) [x-pack-security-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.transport.TransportService$8.doRun(TransportService.java:801) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
elasticsearch_1  | "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
elasticsearch_1  | "at java.lang.Thread.run(Thread.java:832) [?:?]",
elasticsearch_1  | "Caused by: java.lang.IllegalArgumentException: failed to parse date field [Jul 30 22:14:56] with format [strict_date_optional_time||epoch_millis]",     
elasticsearch_1  | "at org.elasticsearch.common.time.JavaDateFormatter.parse(JavaDateFormatter.java:169) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parse(DateFieldMapper.java:387) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.mapper.DateFieldMapper.parseCreateField(DateFieldMapper.java:628) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:294) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "... 40 more",
elasticsearch_1  | "Caused by: java.time.format.DateTimeParseException: Failed to parse with all enclosed parsers",
elasticsearch_1  | "at org.elasticsearch.common.time.JavaDateFormatter.doParse(JavaDateFormatter.java:196) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.common.time.JavaDateFormatter.parse(JavaDateFormatter.java:167) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parse(DateFieldMapper.java:387) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.mapper.DateFieldMapper.parseCreateField(DateFieldMapper.java:628) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:294) ~[elasticsearch-7.8.1.jar:7.8.1]",
elasticsearch_1  | "... 40 more"] }

logstash_1 | [WARN ] 2020-07-30 22:06:21.253 [[beats]>worker3] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"rsyslog-2020.07.30", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x7fb19247>], :response=>{"index"=>{"_index"=>"rsyslog-2020.07.30", "_type"=>"_doc", "_id"=>"1RnDoXMB-RYw4kgdDFU1", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [timestamp] of type [date] in document with id '1RnDoXMB-RYw4kgdDFU1'. Preview of field's value: 'Jul 30 22:14:56'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [Jul 30 22:14:56] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"Failed to parse with all enclosed parsers"}}}}}}