Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add RFC5424 format support for syslog input #23954

Merged
merged 12 commits into from
Apr 6, 2021
Merged

Add RFC5424 format support for syslog input #23954

merged 12 commits into from
Apr 6, 2021

Conversation

wph95
Copy link
Contributor

@wph95 wph95 commented Feb 10, 2021

What does this PR do?

Reopen from #20246
Resolves #6872
Add RFC5424 format support for syslog input

Why is it important?

Syslog input only support rfc3164, btw rfc3164 is obsoleted by rfc5424

More and more software is using rfc5424 instead of rfc3164.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Syslog
    • inputs config support RFC5424
    • detect format automatically
  • Ragel
    • HEADER
      • PRI
      • VERSION
      • TIMESTAMP
      • HOSTNAME
      • APP-NAME
      • PROCID
      • MSGID
    • STRUCTURED-DATA
      • SD-ELEMENT
      • SD-ID
      • SD-PARAM
      • Change Control
    • MSG
  • TEST
    • all examples in RFC5424 document
    • Some special boundary case tests

How to test this PR locally

Related issues

closes # #6872

wph95 added 11 commits July 27, 2020 23:20
@wph95
- add PRI schema
d8d403d 
- add VERSION schema
@wph95
- make test work
f1d2dc3 
- clean code
- change parser.go to parser/parser_rfc5424
@wph95
- Add TIMESTAMP format
eef6387
@wph95
- Add "AppName" format
e395b7b 
- Add "ProcID" format
- Add "MsgID" format
- Add "HostName" format
- improve test
@wph95
add STRUCTURED_DATA support
2ccf82e
@wph95
add MESSAGE support
6c61d51
@wph95
- syslog input config support rfc5424
147fdad 
- add auto detect format
@wph95
add param value escape support
2a86265
@wph95
- clean up the code
1357609 
- add more test
@wph95
Merge branch 'master' into add-syslog-support-rfc-5424
85efe88 
# Conflicts:
#	filebeat/input/syslog/rfc3164_parser.go
@wph95
update mod
7b26515
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 10, 2021
@andresrc andresrc added the Team:Elastic-Agent Label for the Agent team label Feb 10, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/agent (Team:Agent)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 10, 2021
@faec
Copy link
Contributor

faec commented Feb 16, 2021

jenkins, run the tests please

@kvch
Copy link
Contributor

kvch commented Feb 16, 2021

jenkins run tests

@faec
Copy link
Contributor

faec commented Feb 16, 2021

Thanks for the contribution, this looks great! Can you merge with the latest branch and re-push? It looks like your base used an older testing configuration that doesn't work with our current setup.

@wph95
Copy link
Contributor Author

wph95 commented Feb 17, 2021

Thanks for the contribution, this looks great! Can you merge with the latest branch and re-push? It looks like your base used an older testing configuration that doesn't work with our current setup.

np, will rebase to lastest soon :) tks your review and reply

@elasticmachine
Copy link
Collaborator

elasticmachine commented Feb 17, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: faec commented: jenkins run tests

  • Start Time: 2021-02-22T13:40:33.896+0000

  • Duration: 112 min 33 sec

  • Commit: c23abef

Test stats 🧪

Test Results
Failed 0
Passed 46914
Skipped 4944
Total 51858

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 46914
Skipped 4944
Total 51858

@faec
Copy link
Contributor

faec commented Feb 22, 2021

jenkins run tests

@faec faec self-requested a review February 22, 2021 15:20
andrewkroh
andrewkroh reviewed Feb 22, 2021
View reviewed changes
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you considered putting this parsing logic into a processor such users can parse both rfc3164 and rc5424 independent of the input type used? Like parsing syslog messages read from a file or taken from a kafka topic.

@wph95
Copy link
Contributor Author

wph95 commented Feb 23, 2021

Have you considered putting this parsing logic into a processor such users can parse both rfc3164 and rc5424 independent of the input type used? Like parsing syslog messages read from a file or taken from a kafka topic.

make sense!
I can move the logic to the processer in the next pr.
Personally, I don't really recommend doing it in this pr, which has enough work and spend a long time

@faec
Copy link
Contributor

faec commented Mar 2, 2021

Hi, just a followup to say that while I haven't had time to do a full line-by-line review yet, I think this looks good and we hope to merge it. Thank you for all the work that went into this, and I will follow up with more specific feedback soon :-)

@jamiehynds jamiehynds mentioned this pull request Mar 24, 2021
Closed
@faec faec mentioned this pull request Apr 6, 2021
Closed
7 tasks
faec
faec approved these changes Apr 6, 2021
View reviewed changes
Copy link
Contributor

@faec faec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you again, this PR was lovely to read :-)

@faec faec merged commit 3953756 into elastic:master Apr 6, 2021
v1v added a commit to v1v/beats that referenced this pull request Apr 7, 2021
@v1v
Merge remote-tracking branch 'upstream/master' into feature/mergify
c1238b8 
* upstream/master: (91 commits)
  [Filebeat] Change okta.target to nested field (elastic#24636)
  Add RFC5424 format support for syslog input  (elastic#23954)
  Fix links to Beats product pages (elastic#24821)
  [DOCS] Fix 'make setup' instructions for a new beat (elastic#24944)
  Remove duplicate decode_xml entry (elastic#24941)
  [libbeat] Add wineventlog schema to decode_xml processor (elastic#24726)
  [Elastic Agent] Add check for URL set when cert and cert key. (elastic#24904)
  feat: stage execution cache (elastic#24780)
  Fix error in Journalbeat commands (elastic#24880)
  Add baseline ECS 1.9.0 upgrade (elastic#24909)
  [Elastic Agent] Cloud container legacy apm files. (elastic#24896)
  [Elastic Agent]: Reduce allowed socket path length (elastic#24914)
  Add ability to destroy indices with wildcards in testing (elastic#24915)
  Add status subcommand to report status of running daemon. (elastic#24856)
  Fix types of fields GetHits and Ops in Metricbeat module for Couchbase (elastic#23287)
  Add support for Filestream input in elastic agent. (elastic#24820)
  Implement k8s secrets provider for Agent (elastic#24789)
  Sort processor list in docs (elastic#24874)
  Add support for SCRAM authentication in kafka metricbeat module (elastic#24810)
  Properly update offset in case of unparasable line (elastic#22685)
  ...
@wph95 wph95 deleted the add-syslog-support-rfc-5424 branch April 8, 2021 06:33
@faec faec mentioned this pull request Apr 13, 2021
Merged
6 tasks
@jamiehynds jamiehynds mentioned this pull request Apr 14, 2021
Closed
11 tasks
@faec faec mentioned this pull request Apr 15, 2021
Closed
27 tasks
@faec faec added the v7.13.0 label Apr 15, 2021
faec pushed a commit to faec/beats that referenced this pull request Apr 15, 2021
@wph95 @faec
Add RFC5424 format support for syslog input (elastic#23954)
c43cfa2 
* - add PRI schema
- add VERSION schema

* - make test work
- clean code
- change parser.go to parser/parser_rfc5424

* - Add TIMESTAMP format

* - Add "AppName" format
- Add "ProcID" format
- Add "MsgID" format
- Add "HostName" format
- improve test

* add STRUCTURED_DATA support

* add MESSAGE support

* - syslog input config support rfc5424
- add auto detect format

* add param value escape support

* - clean up the code
- add more test

* update mod

(cherry picked from commit 3953756)
@faec faec added the backport-v7.13.0 Automated backport with mergify label Apr 15, 2021
mergify bot pushed a commit that referenced this pull request Apr 15, 2021
@wph95
Add RFC5424 format support for syslog input (#23954)
77f83da 
* - add PRI schema
- add VERSION schema

* - make test work
- clean code
- change parser.go to parser/parser_rfc5424

* - Add TIMESTAMP format

* - Add "AppName" format
- Add "ProcID" format
- Add "MsgID" format
- Add "HostName" format
- improve test

* add STRUCTURED_DATA support

* add MESSAGE support

* - syslog input config support rfc5424
- add auto detect format

* add param value escape support

* - clean up the code
- add more test

* update mod

(cherry picked from commit 3953756)
@mergify mergify bot mentioned this pull request Apr 15, 2021
Merged
faec pushed a commit that referenced this pull request Apr 19, 2021
@mergify
Add RFC5424 format support for syslog input (#23954) (#25119)
6f6c8f5
@adriansr adriansr mentioned this pull request Jun 23, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewkroh andrewkroh andrewkroh left review comments

@faec faec faec approved these changes

Assignees

@faec faec

Labels
backport-v7.13.0 Automated backport with mergify Team:Elastic-Agent Label for the Agent team v7.13.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Syslog input to support RFC5424
6 participants
@wph95 @elasticmachine @faec @kvch @andrewkroh @andresrc