Skip to content
This repository has been archived by the owner on May 6, 2024. It is now read-only.

Tpye feature related sections edited. #108

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

Tpye feature related sections edited. #108

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions Content/Guides/syslog-ng-guide-admin/configuring-destinations-mongodb.htm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,19 @@
<?xml version="1.0" encoding="utf-8"?>
<html xmlns:MadCap="http://www.madcapsoftware.com/Schemas/MadCap.xsd">
<head>
<meta name="description" content="" />
</head>
<body name="configuring-destinations-mongodb">
<h1 name="configuring-destinations-mongodb">mongodb(): Storing messages in a MongoDB database</h1>
<MadCap:keyword term="destination drivers:[&lt;span class=&quot;Code&quot; oldrole=&quot;parameter&quot;&gt;mongodb()&lt;/span&gt;, ' driver']">
</MadCap:keyword>
<p>The <span class="Code">mongodb()</span> driver sends messages to a <a href="https://www.mongodb.com/">MongoDB</a> database. MongoDB is a schema-free, document-oriented database. For the list of available optional parameters, see <MadCap:xref href="reference-destination-mongodb.htm"><span style="color: #04aada;" class="mcFormatColor">mongodb() destination options</span></MadCap:xref>.</p>
<p>The <span class="Code">mongodb()</span> driver sends messages to a <a href="https://www.mongodb.com/">MongoDB</a> database. MongoDB is a schema-free, document-oriented database. For the list of available optional parameters, see <MadCap:xref href="reference-destination-mongodb.htm#reference-destination-mongodb"><span style="color: #04aada;" class="mcFormatColor">mongodb() destination options</span></MadCap:xref>.</p>
<MadCap:snippetBlock src="../shared/wnt/note-server-mode-only.flsnp">
</MadCap:snippetBlock>
<div>
<h6>Declaration</h6><pre>mongodb(parameters);</pre>
</div>
<p>The <span class="Code">mongodb()</span> driver does not support creating indexes, as that can be a very complex operation in MongoDB. If needed, the administrator of the MongoDB database must ensure that indexes are created on the collections.</p>
<p>The <span class="Code">mongodb()</span> driver does not add the <span class="Code">_id</span> field to the message: the MongoDB server will do that automatically, if none is present. If you want to override this field from <MadCap:variable name="General.abbrev"></MadCap:variable>, use the <b class="Code">key()</b> parameter of the <span class="Code">value-pairs()</span> option.</p>
<p>The <span class="Code">mongodb()</span> driver does not add the <span class="Code">_id</span> field to the message: the MongoDB server will do that automatically, if none is present. If you want to override this field from <MadCap:variable name="General.abbrev"></MadCap:variable>, use the <span class="Code">key()</span> parameter of the <span class="Code">value-pairs()</span> option.</p>
<p>The <MadCap:variable name="General.abbrev"></MadCap:variable> <span class="Code">mongodb()</span> driver is compatible with MongoDB server version 1.4 and newer.</p>
<MadCap:snippetBlock src="../shared/wnt/note-typehinting.htm">
</MadCap:snippetBlock>
Expand All @@ -30,7 +31,7 @@ <h6 name="example-destination-mongodb"><a name="example-destination-mongodb"></a
)
);
};</pre>
<p>The following example shows the same setup using the deprecated libmongo-client syntax (as used in <MadCap:variable name="General.abbrev"></MadCap:variable> version <MadCap:conditionaltext MadCap:conditions="General.OSE">3.7</MadCap:conditionaltext>), and is equivalent with the previous example.</p><pre>destination d_mongodb {
<p>The following example shows the same setup using the deprecated libmongo-client syntax, and is equivalent with the previous example.</p><pre>destination d_mongodb {
mongodb(
servers("localhost:27017")
database("syslog")
Expand Down
229 changes: 118 additions & 111 deletions Content/Guides/syslog-ng-guide-admin/date-parser-options.htm
View file
Open in desktop

Large diffs are not rendered by default.

29 changes: 26 additions & 3 deletions Content/Guides/syslog-ng-guide-admin/filters-comparing.htm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<html xmlns:MadCap="http://www.madcapsoftware.com/Schemas/MadCap.xsd">
<head>
<meta name="description" content="" />
</head>
<body name="filters-comparing">
<h1 name="filters-comparing">Comparing macro values in filters</h1>
Expand All @@ -27,8 +26,32 @@ <h1 name="filters-comparing">Comparing macro values in filters</h1>
</MadCap:keyword>
<MadCap:keyword term="filters:['comparing values']">
</MadCap:keyword>
<p>Starting with <MadCap:variable name="General.abbrev"></MadCap:variable> version <MadCap:conditionaltext MadCap:conditions="General.OSE">3.2</MadCap:conditionaltext><MadCap:conditionaltext MadCap:conditions="General.PE">4 F1</MadCap:conditionaltext>, it is also possible to compare macro values and templates as numerical and string values. String comparison is alphabetical: it determines if a string is alphabetically greater or equal to another string. Use the following syntax to compare macro values or templates. For details on macros and templates, see <MadCap:xref href="customizing-message-format.htm"><span style="color: #04aada;" class="mcFormatColor">Customize message format using macros and templates</span></MadCap:xref>.</p><pre>filter &lt;filter-id&gt;
<p>Starting with <MadCap:variable name="General.abbrev"></MadCap:variable> version <MadCap:conditionaltext MadCap:conditions="General.OSE">3.2</MadCap:conditionaltext><MadCap:conditionaltext MadCap:conditions="General.PE">4 F1</MadCap:conditionaltext>, it is also possible to compare macro values and templates as numerical and string values. String comparison is alphabetical: it determines if a string is alphabetically greater or equal to another string. Use the following syntax to compare macro values or templates. For details on macros and templates, see <MadCap:xref href="customizing-message-format.htm"><span style="color: #04aada;" class="mcFormatColor">Customizing message format using macros and templates</span></MadCap:xref>.</p><pre>filter &lt;filter-id&gt;
{"&lt;macro-or-template&gt;" operator "&lt;value-or-macro-or-template&gt;"};</pre>
<h6 name="string-and-numerical-comparison"><a name="string-and-numerical-comparison"></a>String and numerical comparison</h6>
<p><MadCap:variable name="General.product" /> versions prior to 4.0 used separate operators for string comparisons (for example, <span class="Code">eq</span>). In version 4.0 and onwards, mathematical symbols can be used as operators (==, !=, >=), and <MadCap:variable name="General.product" /> automatically determines how to compare the arguments based on their type.</p>
<ul>
<li><p>If both sides of the comparison are strings, the comparison is string.</p></li>
<li><p>If at least one of the argument is numeric, the comparison is numeric.</p></li>
<li><p>Numbers not enclosed by quotation marks, are recognized as numeric.</p></li>
<li><p>It is possible to explicitly type-cast an argument as numeric.</p></li>
</ul>
<p><b>Examples:</b></p>
<ul>
<li><p><pre>if ("${.apache.httpversion}" == 1.0)</pre></p>
<p>The right side of the argument is 1.0 (a floating point literal), so the comparison is numeric.</p>
</li>
<li><p><pre>if (double("${.apache.httpversion}") == "1.0")</pre></p>
<p>The left side of the argument is explicitly type cast into double, the right side is a string (note the use of quotation marks), so the comparison is numeric.</p>
</li>
<li><p><pre>if ("${.apache.request}" == "/wp-admin/login.php")</pre></p>
<p>The left side of the argument is not type-cast, so it remains a string. The right side of the argument is also a string, so the comparison is string.</p>
</li>
</ul>
<div class="Note">
<p class="Hyphenation"><span class="AllNoteStyles">NOTE: </span>The old string operators can still be used for the sake of backwards compatibility, but it is advised and easier to use the numeric operators for strings as well.</p>
</div>
<p></p>
<div class="Example">
<h6 name="example-comparison"><a name="example-comparison"></a>Example: Comparing macro values in filters</h6>
<p>The following expression selects log messages containing a PID (that is, <span class="Code">${PID}</span> macro is not empty):</p><pre>filter f_pid {"${PID}" !=""};</pre>
Expand All @@ -44,7 +67,7 @@ <h6 name="example-comparison"><a name="example-comparison"></a>Example: Comparin
<p>The <span class="Code">$</span> character must be used before macros.</p>
</li>
<li>
<p>Using comparator operators can be equivalent to using filter functions, but is somewhat slower. For example, using <span class="Code">"${HOST}" eq "myhost"</span> is equivalent to using <span class="Code">host("myhost" type(string))</span>.</p>
<p>Using comparator operators can be equivalent to using filter functions, but is somewhat slower. For example, using <b class="Code">"${HOST}" eq "myhost"</b> is equivalent to using <b class="Code">host("myhost" type(string))</b>.</p>
</li>
<li>
<p>You can use any macro in the expression, including user-defined macros from parsers and results of pattern database classifications.</p>
Expand Down
41 changes: 15 additions & 26 deletions Content/Guides/syslog-ng-guide-admin/json-parser.htm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
<html xmlns:MadCap="http://www.madcapsoftware.com/Schemas/MadCap.xsd">
<head>
<link href="../../Resources/TableStyles/NoteTable_Yellow_DoNotEdit.css" rel="stylesheet" MadCap:stylesheetType="table" />
<meta name="description" content="" />
</head>
<body name="json-parser">
<h1 name="json-parser">JSON parser</h1>
Expand All @@ -15,7 +14,7 @@ <h1 name="json-parser">JSON parser</h1>
<MadCap:keyword term="parsers:['json-parser']">
</MadCap:keyword>
<p>JavaScript Object Notation (JSON) is a text-based open standard designed for human-readable data interchange. It is used primarily to transmit data between a server and web application, serving as an alternative to XML. It is described in <a href="https://tools.ietf.org/html/rfc4627">RFC 4627</a>. The <MadCap:variable name="General.abbrev"></MadCap:variable> application can separate parts of incoming JSON-encoded log messages to name-value pairs. For details on using value-pairs in <MadCap:variable name="General.abbrev"></MadCap:variable> see <MadCap:xref href="concepts-value-pairs.htm"><span style="color: #04aada;" class="mcFormatColor">Structuring macros, metadata, and other value-pairs</span></MadCap:xref>.</p>
<p>You can refer to the separated parts of the JSON message using the key of the JSON object as a macro. For example, if the JSON contains <span class="Code">{"KEY1":"value1","KEY2":"value2"}</span>, you can refer to the values as <b class="Code">${KEY1}</b> and <b class="Code" >${KEY2}</b>. If the JSON content is structured, <MadCap:variable name="General.abbrev"></MadCap:variable> converts it to dot-notation-format. For example, to access the value of the following structure <span class="Code">{"KEY1": {"KEY2": "VALUE"}}</span>, use the <b class="Code">${KEY1.KEY2}</b> macro.</p>
<p>You can refer to the separated parts of the JSON message using the key of the JSON object as a macro. For example, if the JSON contains <span class="Code">{"KEY1":"value1","KEY2":"value2"}</span>, you can refer to the values as <b class="Code">${KEY1}</b> and <b class="Code">${KEY2}</b>. If the JSON content is structured, <MadCap:variable name="General.abbrev"></MadCap:variable> converts it to dot-notation-format. For example, to access the value of the following structure <span class="Code">{"KEY1": {"KEY2": "VALUE"}}</span>, use the <b class="Code">${KEY1.KEY2}</b> macro.</p>
<table cellspacing="0" class="TableStyle-NoteTable_Yellow_DoNotEdit" style="width: 100%;mc-table-style: url('../../Resources/TableStyles/NoteTable_Yellow_DoNotEdit.css');">
<col class="TableStyle-NoteTable_Yellow_DoNotEdit-Column-Column1" style="width: 0.3in;">
</col>
Expand All @@ -29,21 +28,21 @@ <h1 name="json-parser">JSON parser</h1>
</p>
</td>
<td class="TableStyle-NoteTable_Yellow_DoNotEdit-BodyA-Column2-Body1"><span class="Yellow">Caution: </span>
<p>If the names of keys in the JSON content are the same as the names of <MadCap:variable name="General.abbrev"></MadCap:variable> soft macros, the value from the JSON content will overwrite the value of the macro. For example, the <span class="Code">{"PROGRAM":"value1","MESSAGE":"value2"}</span> JSON content will overwrite the <span class="Code">${PROGRAM}</span> and <span class="Code" >${MESSAGE}</span> macros. To avoid overwriting such macros, use the <b class="Code">prefix()</b> option.</p>
<p>Hard macros cannot be modified, so they will not be overwritten. For details on the macro types, see <MadCap:xref href="macros-hard-vs-soft.htm"><span style="color: #04aada;" class="mcFormatColor">Hard vs. soft macros</span></MadCap:xref>.</p>
<p>If the names of keys in the JSON content are the same as the names of <MadCap:variable name="General.abbrev"></MadCap:variable> soft macros, the value from the JSON content will overwrite the value of the macro. For example, the <b class="Code">{"PROGRAM":"value1","MESSAGE":"value2"}</b> JSON content will overwrite the <span class="Code">${PROGRAM}</span> and <span class="Code">${MESSAGE}</span> macros. To avoid overwriting such macros, use the <span class="Code">prefix()</span> option.</p>
<p>Hard macros cannot be modified, so they will not be overwritten. For details on the macro types, see <MadCap:xref href="macros-hard-vs-soft.htm#macros-hard-vs-soft"><span style="color: #04aada;" class="mcFormatColor">Hard vs. soft macros</span></MadCap:xref>.</p>
</td>
</tr>
</tbody>
</table>
<div class="Note">
<p class="Hyphenation"><span class="AllNoteStyles">NOTE:</span>

The JSON parser currently supports only integer, double and string values when interpreting JSON structures. As syslog-ng does not handle different data types internally, the JSON parser converts all JSON data to string values. In case of boolean types, the value is converted to 'TRUE' or 'FALSE' as their string representation.</p>
<p>The JSON parser discards messages if it cannot parse them as JSON messages, so it acts as a JSON-filter as well.</p>
When using the <span class="Code">json-parser()</span>, <MadCap:variable name="General.abbrev"></MadCap:variable> converts all elements of the JSON object to name-value pairs. Any type information carried by the incoming JSON object is preserved, and automatically propagated to other <MadCap:variable name="General.abbrev"></MadCap:variable> components (for example, a destination) if the component supports types. Elements without a type are handled as strings. JSON lists (arrays) are converted to lists, and can be manipulated using the <MadCap:xref href="reference-template-functions.htm#template-function-list"><span style="color: #04aada;" class="mcFormatColor">$(list-*)</span></MadCap:xref> template functions.</p>
<p>Prior to version 4.0, <MadCap:variable name="General.abbrev"></MadCap:variable> handled all data as string.</p>
<p>The JSON parser discards messages if they cannot be parsed as JSON messages, therefore acting as a JSON-filter as well.</p>
</div>
<p>To create a JSON parser, define a parser that has the <span class="Code">json-parser()</span> option. Defining the prefix and the marker are optional. By default, the parser will process the <span class="Code">${MESSAGE}</span> part of the log message. To process other parts of a log message with the JSON parser, use the <b class="Code">template()</b> option. You can also define the parser inline in the log path.</p>
<p>To create a JSON parser, define a parser that has the <span class="Code">json-parser()</span> option. Defining the prefix and the marker are optional. By default, the parser will process the <span class="Code">${MESSAGE}</span> part of the log message. To process other parts of a log message with the JSON parser, use the <span class="Code">template()</span> option. You can also define the parser inline in the log path.</p>
<div>
<h6>Declaration:</h6><pre>parser parser_name {
<h6>Declaration</h6><pre>parser parser_name {
json-parser(
marker()
prefix()
Expand All @@ -52,18 +51,13 @@ <h6>Declaration:</h6><pre>parser parser_name {
</div>
<div class="Example">
<h6>Example: Using a JSON parser</h6>
<p>In the following example, the source is a JSON encoded log message. The syslog parser is disabled, so that <MadCap:variable name="General.abbrev"></MadCap:variable> does not parse the message: <span class="Code">flags(no-parse)</span>. The json-parser inserts "<span class="Code">.json.</span>" prefix before all extracted name-value pairs. The destination is a file that uses the <span class="Code">format-json</span> template function. Every name-value pair that begins with a dot ("<span class="Code">.</span>") character will be written to the file (<span class="Code">dot-nv-pairs</span>). The log line connects the source, the destination and the parser.</p><pre>source s_json {
network(
port(21514
flags(no-parse)
);
<p>In the following example, the source is a JSON encoded log message. The syslog parser is disabled, so that <MadCap:variable name="General.abbrev"></MadCap:variable> does not parse the message: <b class="Code">flags(no-parse)</b>. The json-parser inserts "<span class="Code">.json.</span>" prefix before all extracted name-value pairs. The destination is a file, that uses the <span class="Code">format-json</span> template function. Every name-value pair that begins with a dot ("<span class="Code">.</span>") character will be written to the file (<span class="Code">dot-nv-pairs</span>). The log line connects the source, the destination and the parser.</p><pre>source s_json {
network(port(21514) flags(no-parse));
};

destination d_json {
file(
"/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n")
);
file("/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n"));
};

parser p_json {
Expand All @@ -76,17 +70,12 @@ <h6>Example: Using a JSON parser</h6>
destination(d_json);
};</pre>
<p>You can also define the parser inline in the log path.</p><pre>source s_json {
network(
port(21514)
flags(no-parse)
);
network(port(21514) flags(no-parse));
};

destination d_json {
file(
"/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n")
);
file("/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n"));
};

log {
Expand Down
Loading