Development Build

Commits

• af153a2: Add missing SSL config to kafka admin clients (#710) (ppcad) #710

logprep-14.1.0

Logprep helm chart

logprep-14.0.1

Logprep helm chart

v14.0.0

Breaking

• remove AutoRuleCorpusTester
• removes the option to use synchronous bulk or parallel_bulk operation in favor of parallel_bulk in opensearch_output
• reimplement error handling by introducing the option to configure an error output
  • if no error output is configured, failed event will be dropped

Features

• adds health check endpoint to metrics on path /health
• changes helm chart to use new readiness check
• adds healthcheck_timeout option to all components to tweak the timeout of healthchecks
• adds desired_cluster_status option to opensearch output to signal healthy cluster status
• initially run health checks on setup for every configured component
• make imagePullPolicy configurable for helm chart deployments
• it is now possible to use Lucene compliant Filter Expressions
• make terminationGracePeriodSeconds configurable in helm chart values
• adds ability to configure error output
• adds option default_op_type to opensearch_output connector to set the default operation for indexing documents (default: index)
• adds option max_chunk_bytes to opensearch_output connector to set the maximum size of the request in bytes (default: 100MB)
• adds option error_backlog_size to logprep configuration to configure the queue size of the error queue
• the opensearch default index is now only used for processed events, errors will be written to the error output, if configured

Improvements

• remove AutoRuleCorpusTester
• adds support for rust extension development
• adds prebuilt wheels for architectures x86_64 on manylinux and musllinux based linux platforms to releases
• add manual how to use local images with minikube example setup to documentation
• move Configuration to top level of documentation
• add CONTRIBUTING file
• sets the default for flush_timeout and send_timeout in kafka_output connector to 0 seconds
• changed python base image for logprep to bitnami/python in cause of better CVE governance

Bugfix

• ensure logprep.abc.Component.Config is immutable and can be applied multiple times
• remove lost callback reassign behavior from kafka_input connector
• remove manual commit option from kafka_input connector
• pin mysql-connector-python to >=9.1.0 to accommodate for CVE-2024-21272 and update MySQLConnector to work with the new version

Details

• Handle UnicodeDecodeError in kafka input by @ppcad in #663
• make configuration immutable by @ekneg54 in #661
• Remove autorulecorpustester by @ekneg54 in #665
• add rust toolchain by @ekneg54 in #662
• add readiness checks by @ekneg54 in #652
• Fix release pipeline broken for arm images by @ekneg54 in #670
• Fix automatic docs generation by @ekneg54 in #671
• Add CONTRIBUTING file by @ekneg54 in #664
• add minor improvements by @ekneg54 in #676
• Fix exporter restart by @ekneg54 in #677
• Ensure kafka output flushes queue on shutdown by @ekneg54 in #679
• change base image to bitnami/python by @ekneg54 in #680
• Remove unneeded dependencies by @P4sca1 in #682
• Remove broken kafka implementations by @ekneg54 in #678
• Fix docker image and avoid implicit setuptools runtime dependency by @P4sca1 in #683
• test container image before push by @ekneg54 in #684
• make terminationGracePeriodSeconds configurable. by @ekneg54 in #686
• Scan images using Trivy by @P4sca1 in #685
• Dev autoruletester refactoring by @Malutthias in #594
• 629 no support for the mysql connector python library in version 900 by @ppcad in #688
• Add Lucene compliant regex filter expression by @djkhl in #675
• add abiltity to configure error output by @ekneg54 in #668
• prepare release 14 by @ekneg54 in #689

New Contributors

• @P4sca1 made their first contribution in #682
• @Malutthias made their first contribution in #594

Full Changelog: v13.1.2...v14.0.0

logprep-14.0.0

Logprep helm chart

logprep-13.4.0

Logprep helm chart

logprep-13.3.0

Logprep helm chart

v13.1.2

Bugfix

• fixes a bug not increasing but decreasing timeout throttle factor of ThrottlingQueue
• handle DecodeError and unexpected Exceptions on requests in http_input separately
• fixes unbound local error in http input connector

Details

• Fix throtteling queue increases progressively by @ekneg54 in #658
• fix http input bug by @djkhl in #657
• prepare release 13.1.2 by @ekneg54 in #659

Full Changelog: v13.1.1...v13.1.2

v13.1.1

Improvements

• adds ability to bypass the processing of events if there is no pipeline. This is useful for pure connector deployments.
• adds experimental feature to bypass the rule tree by setting LOGPREP_BYPASS_RULE_TREE environment variable

Bugfix

• fixes a bug in the http_output used by the http generator, where the timeout parameter does only set the read_timeout not the write_timeout
• fixes a bug in the http_input not handling decode errors

Details

• Fix http output timeout by @ekneg54 in #651
• bypass process_event if there is no pipeline by @ekneg54 in #656
• option to bypass rule tree by @ekneg54 in #655
• Fix http input decode error by @ekneg54 in #654

Full Changelog: v13.1.0...v13.1.1

13.1.0

Features

• pre_detector now normalizes timestamps with configurable parameters timestamp_field, source_format, source_timezone and target_timezone
• pre_detector now writes tags in failure cases
• ProcessingWarnings now can write tags to the event
• add timeout parameter to logprep http generator to set the timeout in seconds for requests
• add primitive rate limiting to http_input connector

Improvements

• switch to uvloop as default loop for the used threaded http uvicorn server
• switch to httptools as default http implementation for the used threaded http uvicorn server

Bugfix

• remove redundant chart features for mounting secrets

Details

• generalize chart version by @ekneg54 in #641
• remove redundant charts features by @ekneg54 in #647
• normalization of timestamp in pre detector by @djkhl in #646
• dev-revise-http-connector by @ekneg54 in #645

Full Changelog: v13.0.1...v13.1.0