CHANGELOG.next.asciidoc

Beats version HEAD

Check the HEAD diff

Breaking changes

Affecting all Beats - Update Go version to 1.19.10 35751 - Fix status reporting to Elastic-Agent when output configuration is invalid running under Elastic-Agent 35719

Auditbeat

Filebeat

Heartbeat

Metricbeat

Osquerybeat

Packetbeat

Winlogbeat

• Add "event.category" and "event.type" to Sysmon module for EventIDs 8, 9, 19, 20, 27, 28, 255 35193

Functionbeat

Elastic Logging Plugin

Bugfixes

Affecting all Beats - Support for multiline zookeeper logs 2496 - Allow clock_nanosleep in the default seccomp profiles for amd64 and 386. Newer versions of glibc (e.g. 2.31) require it. 33792 - Disable lockfile when running under elastic-agent. 33988 - Fix lockfile logic, retry locking 34194 - Add checks to ensure reloading of units if the configuration actually changed. 34346 - Fix namespacing on self-monitoring 32336 - Fix race condition when stopping runners 32433 - Fix concurrent map writes when system/process code called from reporter code 32491 - Log errors from the Elastic Agent V2 client errors channel. Avoids blocking when error occurs communicating with the Elastic Agent. 34392 - Only log publish event messages in trace log level under elastic-agent. 34391 - Fix issue where updating a single Elastic Agent configuration unit results in other units being turned off. 34504 - Fix dropped events when monitor a beat under the agent and send its Host info log entry. 34599 - Fix namespacing on self-monitoring 32336 - Fix race condition when stopping runners 32433 - Fix concurrent map writes when system/process code called from reporter code 32491 - Fix panics when a processor is closed twice 34647 - Update elastic-agent-system-metrics to v0.4.6 to allow builds on mips platforms. 34674 - The Elasticsearch output now splits large requests instead of dropping them when it receives a StatusRequestEntityTooLarge error. 34911 - Fix Beats started by agent do not respect the allow_older_versions: true configuration flag 34227 34964 - Fix performance issues when we have a lot of inputs starting and stopping by allowing to disable global processors under fleet. 35000 35031 - In cases where the matcher detects a non-string type in a match statement, report the error as a debug statement, and not a warning statement. 35119 - 'add_cloud_metadata' processor - add cloud.region field for GCE cloud provider - 'add_cloud_metadata' processor - update azure metadata api version to get missing cloud.account.id field - Make sure k8s watchers are closed when closing k8s meta processor. 35630 - Upgraded apache arrow library used in x-pack/libbeat/reader/parquet from v11 to v12.0.1 in order to fix cross-compilation issues 35640 - Fix panic when MaxRetryInterval is specified, but RetryInterval is not 35820 - Do not print context cancelled error message when running under agent 36006 - Fix recovering from invalid output configuration when running under Elastic-Agent 36016 - Improve StreamBuf append to improve performance when reading long lines from files. 35928 - Eliminate cloning of event in deepUpdate 35945 - Fix ndjson parser to store JSON fields correctly under target 29395 - Support build of projects outside of beats directory 36126

Auditbeat

Filebeat

• [Auditbeat System Package] Added support for Apple Silicon chips. 34433

• [Azure blob storage] Changed logger field name from container to container_name so that it does not clash with the ecs field name container. 34403

• [GCS] Added support for more mime types & introduced offset tracking via cursor state. Also added support for automatic splitting at root level, if root level element is an array. 34155

• [httpsjon] Improved error handling during pagination with chaining & split processor 34127

• [Azure blob storage] Added support for more mime types & introduced offset tracking via cursor state. 33981

• Fix EOF on single line not producing any event. 30436 33568

• Fix handling of error in states in direct aws-s3 listing input 33513 33722

• Fix httpjson input page number initialization and documentation. 33400

• Add handling of AAA operations for Cisco ASA module. 32257 32789

• Fix gc.log always shipped even if gc fileset is disabled 30995

• Fix handling of empty array in httpjson input. 32001

• Fix reporting of filebeat.events.active in log events such that the current value is always reported instead of the difference from the last value. 33597

• Fix splitting array of strings/arrays in httpjson input 30345 33609

• Fix Google workspace pagination and document ID generation. 33666

• Fix PANW handling of messages with event.original already set. 33829 33830

• Rename identity as identity_name when the value is a string in Azure Platform Logs. 33654

• Fix 'requires pointer' error while getting cursor metadata. 33956

• Fix input cancellation handling when HTTP client does not support contexts. 33962 33968

• Update mito CEL extension library to v0.0.0-20221207004749-2f0f2875e464 33974

• Fix CEL result deserialisation when evaluation fails. 33992 33996

• Fix handling of non-200/non-429 status codes. 33999 34002

• [azure-eventhub input] Switch the run EPH run mode to non-blocking 34075

• [google_workspace] Fix pagination and cursor value update. 34274

• Fix handling of quoted values in auditd module. 22587 34069

• Fixing system tests not returning expected content encoding for azure blob storage input. 34412

• [Azure Logs] Fix authentication_processing_details parsing in sign-in logs. 34330 34478

• Prevent Elasticsearch from spewing log warnings about redundant wildcard when setting up ingest pipelines. 34249 34550

• Gracefully handle Windows event channel not found errors in winlog input. 30201 34605

• Fix the issue of cometd input worker getting closed in case of a network connection issue and an EOF error. 34326 34327

• Fix for httpjson first_response object throwing false positive errors by making it a flag based object 34747 34748

• Fix errors and panics due to re-used processors 34761

• Add missing Basic Authentication support to CEL input 34609 34689

• [Gcs Input] - Added missing locks for safe concurrency 34914

• Fix the ignore_inactive option being ignored in Filebeat’s filestream input 34770

• Fix TestMultiEventForEOFRetryHandlerInput unit test of CometD input 34903

• Add input instance id to request trace filename for httpjson and cel inputs 35024

• Fix panic in TCP and UDP inputs on Linux when collecting socket metrics from OS. 35064

• Correctly collect TCP and UDP metrics for unspecified address values. 35111

• Fix base for UDP and TCP queue metrics and UDP drops metric. 35123

• Sanitize filenames for request tracer in httpjson input. 35143

• decode_cef processor: Fix ECS output by making observer.ip into an array of strings instead of string. 35140 35149

• Fix handling of MySQL audit logs with strict JSON parser. 35158 35160

• Sanitize filenames for request tracer in cel input. 35154

• Fix accidental error overwrite in defer statement in entityanalytics Azure AD input. 35153 35169

• Fixing the grok expression outputs of log files 35221

• Fixes "Can only start an input when all related states are finished" error when running under Elastic-Agent 35250 33653

• Move repeated Windows event channel not found errors in winlog input to debug level. 35314 35317

• Fix crash when processing forwarded logs missing a message. 34705 34865

• Fix crash when loading azurewebstorage cursor with no partially processed data. 35433

• Add support in s3 input for JSON with array of objects. 35475

• RFC5424 syslog timestamps with offset 'Z' will be treated as UTC rather than using the default timezone. 35360

• Fix syslog message parsing for fortinet.firewall to take into account quoted values. 35522

• [system] sync system/auth dataset with system integration 1.29.0. 35581

• [GCS Input] - Fixed an issue where bucket_timeout was being applied to the entire bucket poll interval and not individual bucket object read operations. Fixed a map write concurrency issue arising from data races when using a high number of workers. Fixed the flaky tests that were present in the GCS test suit. 35605

• Fix filestream false positive log error "filestream input with ID 'xyz' already exists" 31767

• Fix error message formatting from filestream input. 35658

• Fix error when trying to use include_message parser 35440

• Fix handling of IPv6 unspecified addresses in TCP input. 35064 35637

• Fixed a minor code error in the GCS input scheduler where a config value was being used directly instead of the source struct. 35729

• Improve error reporting and fix IPv6 handling of TCP and UDP metric collection. 35772

• Fix CEL input JSON marshalling of nested objects. 35763 35774

• Fix metric collection in GCPPubSub input. 35773

• Fix end point deregistration in http_endpoint input. 35899 35903

• Fix duplicate ID panic in filestream metrics. 35964 35972

• Improve error reporting and fix IPv6 handling of TCP and UDP metric collection. 35996

• Fix handling of NUL-terminated log lines in Fortinet Firewall module. 36026 36027

• Make redact field configuration recommended in CEL input and log warning if missing. 36008

• Fix handling of region name configuration in awss3 input 36034

• Fix panic when sqs input metrics getter is invoked 36101 36077

• Make CEL input’s now global variable static for evaluation lifetime. 36107

• Update mito CEL extension library to v1.5.0. 36146

Heartbeat

• Fix panics when parsing dereferencing invalid parsed url. 34702

• Fix broken zip URL monitors. NOTE: Zip URL Monitors will be removed in version 8.7 and replaced with project monitors. 33723

• Fix integration hashing to prevent reloading all when updated. 34697

• Fix release of job limit semaphore when context is cancelled. 34697

• Fix bug where states.duration_ms was incorrect type. 33563

• Fix handling of long UDP messages in UDP input. 33836 33837

• Fix browser monitor summary reporting as up when monitor is down. 33374 33819

• Fix beat capabilities on Docker image. 33584

• Fix serialization of state duration to avoid scientific notation. 34280

• Enable nodejs engine strict validation when bundling synthetics. 34470 with the ecs field name container. 34403 automatic splitting at root level, if root level element is an array. 34155

• Fix broken mapping for state.ends field. 34891

• Fix issue using projects in airgapped environments by disabling npm audit. 34936

• Fix broken state ID location naming. 35336

• Fix project monitor temp directories permission to include group access. 35398

• Fix output pipeline exit on run_once. 35376

• Fix formatting issue with socket trace timeout. 35434

• Update gval version. 35636

• Fix serialization of processors when running diagnostics. 35698

• Filter dev flags for ui monitors inside synthetics_args. 35788

• Fix temp dir running out of space with project monitors. 35843

• Fixing the grok expression outputs of log files 35221

Heartbeat

Heartbeat

Heartbeat

Heartbeat

Auditbeat

Filebeat

Auditbeat

Filebeat

• Sanitize filenames for request tracer in cel input. 35154

Heartbeat

Metricbeat

• in module/windows/perfmon, changed collection method of the second counter value required to create a displayable value 32305

• Fix and improve AWS metric period calculation to avoid zero-length intervals 32724

• Add missing cluster metadata to k8s module metricsets 32979 33032

• Add GCP CloudSQL region filter 32943

• Fix logstash cgroup mappings 33131

• Remove unused elasticsearch.node_stats.indices.bulk.avg_time.bytes mapping 33263

• Fix kafka dashboard field names 33555

• Add tags to events based on parsed identifier. 33472

• Support Oracle-specific connection strings in SQL module 32089 32293

• Remove deprecated metrics from controller manager, scheduler and proxy 34161

• Fix metrics split through different events and metadata not matching for aws cloudwatch. 34483

• Fix metadata enricher with correct container ids for pods with multiple containers in container metricset. Align kubernetes.container.id and container.id fields for state_container metricset. 34516

• Make generic SQL GA 34637

• Collect missing remote_cluster in elasticsearch ccr metricset 34957

• Add context with timeout in AWS API calls 35425

• Fix no error logs displayed in CloudWatch EC2, RDS and SQS metadata 34985 35035

• Remove Beta warning from IIS application_pool metricset 35480

• Improve documentation for ActiveMQ module 35113 35558

• Fix EC2 host.cpu.usage 35717

• Resolve statsd module’s prematurely halting of metrics parsing upon encountering an invalid packet. 35075

Osquerybeat

• Adds the elastic_file_analysis table to the Osquery extension for macOS builds. 35056

Packetbeat

• Fix double channel close panic when reloading. 35324

• Fix BPF filter setting not being applied to sniffers. 35363 35484

• Fix handling of Npcap installation options from Fleet. 35541

Winlogbeat

• Fix handling of event data with keys containing dots. 34345 34549

• Gracefully handle channel not found errors. 30201 34605

• Clarify query term limits warning and remove link to missing Microsoft doc page. 34715

• Improve documentation for event_logs.name configuration. 34931

• Move repeated channel not found errors to debug level. 35314 35317

• Fix panic due to misrepresented buffer use. 35437

• Prevent panic on closing iterators on empty channels in experimental API. 33966 35423

• Allow program termination when attempting to open an absent channel. 35474

Functionbeat

Functionbeat

Elastic Logging Plugin

Added

Affecting all Beats

• Added append Processor which will append concrete values or values from a field to target. 29934 33364

• Allow users to enable features via configuration, starting with the FQDN reporting feature. 1070 34456

• Add Hetzner Cloud as a provider for add_cloud_metadata. 35456

• Reload Beat when TLS certificates or key files are modified. 34408 34416

• Upgrade version of elastic-agent-autodiscover to v0.6.1 for improved memory consumption on k8s. 35483

• Added orchestrator.cluster.id and orchestrator.cluster.name fields to the add_cloud_metadata processor, AWS cloud provider. 35182

• Lowercase reported hostnames per Elastic Common Schema (ECS) guidelines for the host.name field. Upgraded github.com/elastic/go-sysinfo to 1.11.0. 35652

Auditbeat

Filebeat

• add documentation for decode_xml_wineventlog processor field mappings. 32456

• httpjson input: Add request tracing logger. 32402 32412

• Add cloudflare R2 to provider list in AWS S3 input. 32620

• Add support for single string containing multiple relation-types in getRFC5988Link. 32811

• Added separation of transform context object inside httpjson. Introduced new clause .parent_last_response.* 33499

• Cloud Foundry input uses server-side filtering when retrieving logs. 33456

• Add parse_aws_vpc_flow_log processor. 33656

• Update aws.vpcflow dataset in AWS module have a configurable log format and to produce ECS 8.x fields. 33699

• Modified aws-s3 input to reduce mutex contention when multiple SQS message are being processed concurrently. 33658

• Disable "event normalization" processing for the aws-s3 input to reduce allocations. 33673

• Add Common Expression Language input. 31233

• Add support for http+unix and http+npipe schemes in httpjson input. 33571 33610

• Add support for http+unix and http+npipe schemes in cel input. 33571 33712

• Add decode_duration, move_fields processors. 31301

• Add backup to bucket and delete functionality for the aws-s3 input. 30696 33559

• Add metrics for UDP packet processing. 33870

• Convert UDP input to v2 input. 33930

• Improve collection of risk information from Okta debug data. 33677 34030

• Adding filename details from zip to response for httpjson 33952 34044

• Allow user configuration of keep-alive behaviour for HTTPJSON and CEL inputs. 33951 34014

• Add support for polling system UDP stats for UDP input metrics. 34070

• Add support for recognizing the log level in Elasticsearch JVM logs 34159

• Add new Entity Analytics input with Azure Active Directory support. 34305

• Added metric sqs_lag_time for aws-s3 input. 34306

• Add metrics for TCP packet processing. 34333

• Add metrics for unix socket packet processing. 34335

• Add beta take over mode for filestream for simple migration from log inputs 34292

• Add pagination support for Salesforce module. 34057 34065

• Allow users to redact sensitive data from CEL input debug logs. 34302

• Added support for HTTP destination override to Google Cloud Storage input. 34413

• Added metric sqs_messages_waiting_gauge for aws-s3 input. 34488

• Add support for new Rabbitmq timestamp format for logs 34211

• Allow user configuration of timezone offset in Cisco ASA and FTD modules. 34436

• Allow user configuration of timezone offset in Checkpoint module. 34472

• Add support for Okta debug attributes, risk_reasons, risk_behaviors and factor. 33677 34508

• Fill okta.request.ip_chain.* as a flattened object in Okta module. 34621

• Fixed GCS log format issues. 34659

• Add nginx.ingress_controller.upstream.ip to related.ip 34645 34672

• Include NAT and firewall IPs in related.ip in Fortinet Firewall module. 34640 34673

• Add Basic Authentication support on constructed requests to CEL input 34609 34689

• Add string manipulation extensions to CEL input 34610 34689

• Add unix socket log parsing for nginx ingress_controller 34732

• Added metric sqs_worker_utilization for aws-s3 input. 34793

• Improve CEL input documentation 34831

• Add metrics documentation for CEL and AWS CloudWatch inputs. 34887 34889

• Register MIME handlers for CSV types in CEL input. 34934

• Add MySQL authentication message parsing and related.ip and related.user fields 34810

• Mention mito CEL tool in CEL input docs. 34959

• Add nginx ingress_controller parsing if one of upstreams fails to return response 34787

• Allow neflow v9 and ipfix templates to be shared between source addresses. 35036

• Add support for collecting IPv6 metrics. 35123

• Added support for decoding apache parquet files in awss3 input. 34662 35578

• Add oracle authentication messages parsing 35127

• Add sanitization capabilities to azure-eventhub input 34874

• Add support for CRC validation in Filebeat’s HTTP endpoint input. 35204

• Add support for CRC validation in Zoom module. 35604

• Add execution budget to CEL input. 35409

• Add XML decoding support to HTTPJSON. 34438 35235

• Add delegated account support when using Google ADC in httpjson input. 35507

• Add metrics for filestream input. 35529

• Add support for collecting httpjson metrics. 35392

• Add XML decoding support to CEL. 34438 35372

• Mark CEL input as GA. 35559

• Add metrics for gcp-pubsub input. 35614

• [GCS] Added scheduler debug logs and improved the context passing mechanism by removing them from struct params and passing them as function arguments. 35674

• Allow non-AWS endpoints for awss3 input. 35496 35520

• Under elastic-agent the input metrics will now be included in agent diagnostics dumps. 35798

• Add Okta input package for entity analytics. 35611

• Expose harvester metrics from filestream input 35835 33771

• Add device support for Azure AD entity analytics. 35807

• Improve CEL input performance. 35915

• Adding filename details from zip to response for httpjson 33952 34044

• Added support for min/max template functions in httpjson input. 36094 36036

• Add clean_session configuration setting for MQTT input. 16204

• Add fingerprint mode for the filestream scanner and new file identity based on it 34419 35734

• Add file system metadata to events ingested via filestream 35801 36065

• Allow parsing bytes in and bytes out as long integer in CEF processor. 36100 36108

• Add support for registered owners and users to AzureAD entity analytics provider. 36092

Auditbeat - Migration of system/package module storage from gob encoding to flatbuffer encoding in bolt db. 34817

Libbeat - Added support for apache parquet file reader. 34662 35183

Heartbeat - Users can now configure max scheduler job limits per monitor type via env var. 34307 - Added status to monitor run log report. - Removed beta label for browser monitors. 35424.

Metricbeat

• Add per-thread metrics to system_summary 33614

• Add GCP CloudSQL metadata 33066

• Add GCP Redis metadata 33701

• Remove GCP Compute metadata cache 33655

• Add support for multiple regions in GCP 32964

• Add GCP Redis regions support 33728

• Add namespace metadata to all namespaced kubernetes resources. 33763

• Changed cloudwatch module to call ListMetrics API only once per region, instead of per AWS namespace 34055

• Add beta ingest_pipeline metricset to Elasticsearch module for ingest pipeline monitoring 34012

• Handle duplicated TYPE line for prometheus metrics 18813 33865

• Add GCP Carbon Footprint metricbeat data 34820

• Add event loop utilization metric to Kibana module 35020

• Support collecting metrics from both the monitoring account and linked accounts from AWS CloudWatch. 35540

• Add new parameter include_linked_accounts to enable/disable metrics collection from multiple linked AWS Accounts 35648

• Migrate Azure Billing, Monitor, and Storage metricsets to the newer SDK. 33585

• Add support for float64 values parsing for statsd metrics of counter type. 35099

• Add kubernetes.deployment.status.* fields for Kubernetes module 35999

Osquerybeat

Packetbeat

• Added packetbeat.interfaces.fanout_group to allow a Packetbeat sniffer to join an AF_PACKET fanout group. 35451 35453

• Add AF_PACKET metrics. 35428 35489

• Under elastic-agent the input metrics will now be included in agent diagnostics dumps. 35798

• Add support for multiple regions in GCP 32964

Packetbeat

Winlogbeat

Functionbeat

Winlogbeat

• Set host.os.type and host.os.family to "windows" if not already set. 35435

• Handle empty DNS answer data in QueryResults for the Sysmon Pipeline 35207

• Under elastic-agent the input metrics will now be included in agent diagnostics dumps. 35798

Elastic Log Driver Elastic Logging Plugin

Deprecated

Auditbeat

Filebeat

Heartbeat

Metricbeat

Osquerybeat

Packetbeat

Winlogbeat

Functionbeat

Elastic Logging Plugin

Known Issues