syslog input dies periodically

[tolleiv]

Hi,

my logstash server (1.5.2) keeps dying (reproducible) after ~1 hour runtime with this error:

  IOError: Connection reset by peer
           each at org/jruby/RubyIO.java:3542
   tcp_receiver at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-1.0.0/lib/logstash/inputs/syslog.rb:175
   tcp_listener at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-1.0.0/lib/logstash/inputs/syslog.rb:161
  UDP listener died {:exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-1.0.0/lib/logstash/inputs/udp.rb:79:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-1.0.0/lib/logstash/inputs/udp.rb:49:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:176:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:170:in `start_input'"], :level=>:warn}
  syslog listener died {:protocol=>:udp, :address=>"0.0.0.0:514", :exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-1.0.0/lib/logstash/inputs/syslog.rb:138:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-1.0.0/lib/logstash/inputs/syslog.rb:117:in `server'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-1.0.0/lib/logstash/inputs/syslog.rb:97:in `run'"], :level=>:warn}

using the following filter configuration:

  input {
        syslog {
              type => "syslog"
              port => 514
        }
  }

My server was installed from the official most recent package on Ubuntu 14.04. In addition to that the setcap cap_net_bind_service=+epi $(readlink -f /usr/bin/java) capabilities have been configured for java/logstash.

---

The workaround which seems to run stable for me atm. is to not use the syslog input but the tcp and udp inputs:

input {
    tcp {
        port => 514
        type => "syslog"
    }
    udp {
        port => 514
        type => "syslog"
    }
}

Still the syslog input should also run stable and it shouldn't crash the entire server.

I'd be happy to help and provide additional information.

Cheers

[tolleiv]

Just to confirm - the workaround is now running stable for more than one week. It would still be really nice to have the actual input working too.

[Wurzelmann]

I've got the exact same problem, the workaround is running stable, but I would want to know why it keeps crashing...

[stith]

I ran into this as well. After investigating this for a while I noticed that the logstash process has TONS of connections in the CLOSE_WAIT status, so the issue is that this plugin isn't closing its TCP connections properly. Something in this method will need to be fixed, though I don't know enough ruby to identify why the code already there doesn't work!

[ebuildy]

Hello, looks like it's not only related to syslog plugin, here my error:

The error reported is:
 initialize: name or service not known
�[33mUDP listener died {:exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>["/var/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-1.0.0/lib/logstash/inputs/udp.rb:79:in `udp_listener'", "/var/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-1.0.0/lib/logstash/inputs/udp.rb:49:in `run'", "/var/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/pipeline.rb:177:in `inputworker'", "/var/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/pipeline.rb:171:in `start_input'"], :level=>:warn}�[0m

�[31mA plugin had an unrecoverable error. Will restart this plugin.

Here a piece of my config:

    syslog
    {
        port => 25827
        type => "http_access"
    }

    udp
    {
        port => 25826
        buffer_size => 1452
        codec => collectd { }
        type => "collectd"
    }

Cheers,

[daniilyar]

This affects us, too ..

[xhostAB]

Here too

[machzqcq]

Tried the workaround above by mentioning type and port, but get this error now.

{:timestamp=>"2015-11-15T14:44:58.760000+0000", :message=>"The error reported is: \n Permission denied - bind(2)"}

Workaround that worked

Used Filebeat - https://github.com/elastic/filebeat to output to logstash on port 5044
Then tail the interceptor* log file and could see the stream. Of course add other outputs in logstash conf as needed (I had only the interceptor* log file to test it out)

 ### Logstash as output
  logstash:
    # Uncomment out this option if you want to output to Logstash. The default is false.
    enabled: true
    # The Logstash hosts
    hosts: ["localhost:5044"]

Then read the output of filebeat as input to logstash as follows

input {
beats {
        port => 5044
}
}

output {
   stdout { codec => rubydebug }
 file {
  path => "/var/log/logstash/interceptor-%{+YYYY-MM-dd}.log"
}
}

[tolleiv]

@machzqcq thanks for sharing, but how does filebeat help if you try to retrieve syslogs via TCP from your Cisco switch?

[houht]

but 5140 is OK：

input {

udp {
type => "haproxy"
port => "5140"
}
}