diff --git a/Content/Guides/syslog-ng-guide-admin/destination-s3-amazon.htm b/Content/Guides/syslog-ng-guide-admin/destination-s3-amazon.htm new file mode 100644 index 000000000..7eb125b27 --- /dev/null +++ b/Content/Guides/syslog-ng-guide-admin/destination-s3-amazon.htm @@ -0,0 +1,439 @@ + + + + + + +

s3: Amazon S3

+

Available in version 4.4 and later versions.

+

The s3() destination sends log messages to the Amazon Simple Storage Service (Amazon S3) object storage service. Log messages can be sent over TCP, or encrypted with TLS.

+
+
Prerequisites:
+ +

To use the s3() driver, the scl.conf file must be included in the configuration:

+ 
@include "scl.conf"
+
+
+
Declaration:
s3(
+    url("http://localhost:9000")
+    bucket("syslog-ng")
+    access-key("my-access-key")
+    secret-key("my-secret-key")
+    object-key("${HOST}/my-logs")
+    template("${MESSAGE}\n")
+);
+
+
+
Creating objects
+

Objects can be created based on the following strategies:

+ +

All of these strategies can be used separately, or together.

+
+
+
Upload options
+

uploads objects using the multipart upload API. composes chunks locally. When a chunk reaches the size set in chunk-size() (by default 5 MiB), the chunk is uploaded. When an object is finished, the multipart upload is completed and S3 merges the chunks.

+

The upload can be manipulated by the chunk-size(), upload-threads(), and the max-pending-uploads() options.

+
+
+
Options
+

The following options are specific to the s3() destination.

+ +
+
access-key()
+ + + + + + + + + + + + + +
Type:string
Default:N/A
+

Description: The ACCESS_KEY (used with secret-key())of the service account used to access the S3 bucket.

+
+
+
bucket()
+ + + + + + + + + + + + + +
Type:string
Default:
+

Description: The name of the S3 bucket, for example, my-bucket.

+
+
+
canned-acl()
+ + + + + + + + + + + + + +
Type:string
Default:empty
+

Description: The ACL assigned to the object, if specified, for example, bucket-owner-read.

+

The following values are available:

+ +

If an invalid value is defined, the default value is used.

+
+
+
chunk-size()
+ + + + + + + + + + + + + +
Type:string
Default:5MiB
+

Description: The size of log messages that writes to the S3 object in a batch. If compression is enabled, the chunk-size() refers to the compressed size.

+
+
+
compression()
+ + + + + + + + + + + + + +
Type:boolean
Default:no
+

Description: The setting compression(yes) enables gzip compression, and implicitly adds a .gz suffix to the key of the created object. The level of the compression can be set using the compresslevel() option (0-9).

+
+
+
compresslevel()
+ + + + + + + + + + + + + +
Type:integer (0-9)
Default:9
+

Description: This option only has an effect if the compression() option is set to yes. The level of the compression can be set using the compresslevel() option (0-9).

+
+
+
flush-grace-period()
+ + + + + + + + + + + + + +
Type:integer [minutes]
Default:60
+

Description: After the grace period expires and no new messages are routed to the destination, flushes the contents of the buffer to the S3 object even if the volume of the messages in the buffer is lower than chunk-size().

+
+
+
log-fifo-size()
+ + + + + + + + + + + + + +
Type:number
Default:Use global setting.
+

Description: The number of messages that the output queue can store.

+
+
+
max-object-size()
+ + + + + + + + + + + + + +
Type:string
Default:5120GiB
+

Description: The maximal size of the S3 object. If an object reaches this size, appends an index ("-1", “-2”, etc.) to the end of the object key and starts a new object after rotation.

+
+
+
max-pending-uploads()
+ + + + + + + + + + + + + +
Type:integer
Default:32
+

Description: The max-pending-uploads() and upload-threads() options configure the upload of the chunks. Uploading happens in multiple threads to minimize network overhead.

+ +
+
+
object-key()
+ + + + + + + + + + + + + +
Type:template
Default:N/A
+

Description: The object key (or key name), that uniquely identifies the object in an Amazon S3 bucket.

+
+
+
object-key-timestamp()
+ + + + + + + + + + + + + +
Type:template
Default:
+

Description: The object-key-timestamp() option can be used to set a datetime-related template, which is appended to the end of the object.

+
+
Example:
"${R_MONTH_ABBREV}${R_DAY}"
+
+
+

When a log message arrives with a newer timestamp template resolution, the previous timestamped object gets completed and a new one is initiated with the new timestamp. If an older message arrives, it does not reopen the old object, but starts a new object with the key having an index appended to the old object.

+
+
+
persist-name()
+ + + + + + + + + + + + + +
Type:string
Default:N/A
+

Description: If the following error is displayed during the startup, set the persist-name() option of the duplicate drivers.

+ 
Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.
+

This error occurs when identical drivers in multiple sources are used, for example,if two file sources are configured to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").

+
+
+
region()
+ + + + + + + + + + + + + +
Type:string
Default:
+

Description: The regional endpoint where the bucket is stored, for example, us-east-1.

+
+
+
secret-key()
+ + + + + + + + + + + + + +
Type:string
Default:N/A
+

Description: The SECRET_KEY of the service account (complementing the acces-key()) used to access the S3 bucket.

+
+
+
storage-class()
+ + + + + + + + + + + + + +
Type:string
Default:STANDARD
+

Description: This option defines the storage class of the object.

+

The following values are available:

+ +

If an invalid value is defined, the default value is used.

+
+
+
upload-threads()
+ + + + + + + + + + + + + +
Type:integer
Default:8
+

Description: This option defines the number of threads that are used to upload data to S3 from a certain destination

+
+
+
template()
+ + + + + + + + + + + + + +
Type:template or template function
Default:${MESSAGE}\n
+

Description: This option defines the message as written to the Amazon S3 object. Templates and template functions can be used to format this message.

+
+
+
url()
+ + + + + + + + + + + + + +
Type:string
Default:N/A
+

Description: The URL of the S3 bucket, for example, https://my-bucket.s3.us-west-2.amazonaws.com.

+
+
+ + \ No newline at end of file diff --git a/Project/TOCs/syslog-ng-ose-guide-admin.fltoc b/Project/TOCs/syslog-ng-ose-guide-admin.fltoc index 258860ba8..96baaaa35 100644 --- a/Project/TOCs/syslog-ng-ose-guide-admin.fltoc +++ b/Project/TOCs/syslog-ng-ose-guide-admin.fltoc @@ -767,6 +767,10 @@ Link="/Content/Guides/syslog-ng-guide-admin/reference-destination-riemann.htm" Title="[%=System.LinkedTitle%]"> + +