transform: luaxform transform script #12141

jlucovsky · 2024-11-21T15:34:56Z

Continuation of #12136

Add the luaxform transform that uses a Lua script to receive and transform a sticky buffer.

Lua transforms receive arguments

(always): length of input buffer and input buffer
(always): number of additional arguments (may be 0)
(when present): argument array (bounded by argument count)

Arguments are passed to the Lua script as they appear in the rule:

 luaxform:./lua/lua-transform.lua, bytes 4, offset 13, hash sha256;

When the transform is invoked, 3 additional arguments are passed (arg count will be 3)

bytes 4
offset 13
hash sha256

The Lua script is responsible for parsing and using the arguments and returning a tuple: output-buffer, output-buffer-byte-count

I used this Lua script:

function init (args)
  -- Optional, no return value
end

function transform(input_len, input, argc, args)
    SCLogNotice("Input length: " .. input_len .. " arg-count: " .. argc)
    for i = 1, argc do
        print("Argument " .. i .. ":", args[i])
    end
    return string.upper(input), input_len
end

return 0

Example rule using the Lua script:

alert http any any -> any any (msg:"Lua Xform example"; flow:established;  file.data; luaxform:./lua/lua-transform.lua; content: "abc"; sid: 1;)

Link to ticket: https://redmine.openinfosecfoundation.org/issues/2290

Describe changes:

Luaxform transform changes

Updates

Address formatting issue

Provide values to any of the below to override the defaults.

To use an LibHTP, Suricata-Verify or Suricata-Update pull request,
link to the pull request in the respective _BRANCH variable.
Leave unused overrides blank or remove.

SV_REPO=
SV_BRANCH=OISF/suricata-verify#2090
SU_REPO=
SU_BRANCH=
LIBHTP_REPO=
LIBHTP_BRANCH=

This commit makes the detection engine thread context available for transforms to use. The Lua transform requires this value. Issue: 2290

Issue: 2290 This commit extends the hash table logic with an alternate free function that provides the detection engine context. Users that wish to use the next functionality must use the HashListTableInitWithCtx function when initializing the hash table. Using this interface will result in the hash table "free with context" function (new) being used instead.

Issue: 2290 Defer freeing the keyword hash table until the engine context has been freed. This eliminates a double-free from occurring. For the unittests ONLY, clear the keyword_hash to prevent a double free attempt.

Issue: 2290 This commit adds the source files for the new transform -- luaxform.

Issue: 2290

git grep -A 1 -w InspectionBufferSetup shows numbers cases of the pattern: - InspectionBufferSetup - InspectionBufferApplyTransforms Refactor the implementations of those functions into InspectionBufferSetupAndApplyTransforms to reduce function call count. Issuer: 2290

codecov · 2024-11-21T16:08:43Z

Codecov Report

Attention: Patch coverage is 83.78378% with 72 lines in your changes missing coverage. Please review.

Project coverage is 49.81%. Comparing base (13841c0) to head (4eaffd9).
Report is 28 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #12141      +/-   ##
==========================================
+ Coverage   49.77%   49.81%   +0.03%     
==========================================
  Files         909      912       +3     
  Lines      257884   258098     +214     
==========================================
+ Hits       128367   128574     +207     
- Misses     129517   129524       +7

Flag	Coverage Δ
fuzzcorpus	`60.87% <46.52%> (-0.07%)`	⬇️
livemode	`19.44% <13.19%> (+<0.01%)`	⬆️
pcap	`44.15% <30.32%> (-0.25%)`	⬇️
suricata-verify	`62.71% <83.79%> (+0.02%)`	⬆️
unittests	`8.97% <0.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

suricata-qa · 2024-11-21T19:50:03Z

ERROR:

ERROR: QA failed on SURI_TLPR1_alerts_cmp.

Pipeline 23484

catenacyber · 2024-11-28T14:25:37Z

src/detect-engine-helper.c

@@ -57,7 +57,7 @@ InspectionBuffer *DetectHelperGetData(struct DetectEngineThreadCtx_ *det_ctx,
            return NULL;

        InspectionBufferSetup(det_ctx, list_id, buffer, b, b_len);
-        InspectionBufferApplyTransforms(buffer, transforms);
+        InspectionBufferApplyTransforms(det_ctx, buffer, transforms);


Would you not want to generalize usage of DetectHelperGetData so that changing the prototype of InspectionBufferApplyTransforms affects less files ?

That's a good suggestion, but it will affect many source modules.

Can we consider that an optimization to follow this PR?

This PR affects many source modules :-p

It would affect less and be more focused if this code factorization happened first ;-)

catenacyber · 2024-11-28T14:26:57Z

src/detect-transform-luaxform.c

+
+void DetectTransformLuaxformRegister(void)
+{
+    sigmatch_table[DETECT_TRANSFORM_LUAXFORM].name = "luaxform";


Could it be rust ?

Yes, but I would like to move forward with this PR and then have a "rust conversion" issue (optimization?) to make those changes.

src/detect-transform-luaxform.h

catenacyber · 2024-11-28T14:29:23Z

src/detect-dce-stub-data.c

@@ -79,8 +79,8 @@ static InspectionBuffer *GetSMBData(DetectEngineThreadCtx *det_ctx,
            return NULL;
        SCLogDebug("have data!");

-        InspectionBufferSetup(det_ctx, list_id, buffer, data, data_len);
-        InspectionBufferApplyTransforms(det_ctx, buffer, transforms);
+        InspectionBufferSetupAndApplyTransforms(


Even better to use DetectHelperGetData ;-)

jlucovsky · 2024-11-30T14:47:57Z

Continued in #12187

jlucovsky added 7 commits November 21, 2024 10:08

transforms: Make ctx available

5d87ef1

This commit makes the detection engine thread context available for transforms to use. The Lua transform requires this value. Issue: 2290

detect/engine: Prevent double-free of keyword hash

a843a76

Issue: 2290 Defer freeing the keyword hash table until the engine context has been freed. This eliminates a double-free from occurring. For the unittests ONLY, clear the keyword_hash to prevent a double free attempt.

transform: Add luaxform files

f0d92b5

Issue: 2290 This commit adds the source files for the new transform -- luaxform.

doc: Document luaxform transform

aa061ac

Issue: 2290

clang/format: Formatting fixups

4eaffd9

jlucovsky requested review from jasonish, jufajardini and victorjulien as code owners November 21, 2024 15:34

jlucovsky mentioned this pull request Nov 21, 2024

transform: luaxform transform script #12136

Closed

catenacyber reviewed Nov 28, 2024

View reviewed changes

src/detect-transform-luaxform.h Show resolved Hide resolved

catenacyber reviewed Nov 28, 2024

View reviewed changes

jlucovsky mentioned this pull request Nov 30, 2024

transform: luaxform transform script #12187

Open

jlucovsky closed this Nov 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

transform: luaxform transform script #12141

transform: luaxform transform script #12141

jlucovsky commented Nov 21, 2024

codecov bot commented Nov 21, 2024 •

edited

Loading

suricata-qa commented Nov 21, 2024

catenacyber Nov 28, 2024

jlucovsky Nov 30, 2024

catenacyber Dec 1, 2024

catenacyber Nov 28, 2024

jlucovsky Nov 30, 2024

catenacyber Nov 28, 2024

jlucovsky commented Nov 30, 2024

transform: luaxform transform script #12141

transform: luaxform transform script #12141

Conversation

jlucovsky commented Nov 21, 2024

Provide values to any of the below to override the defaults.

codecov bot commented Nov 21, 2024 • edited Loading

Codecov Report

suricata-qa commented Nov 21, 2024

catenacyber Nov 28, 2024

Choose a reason for hiding this comment

jlucovsky Nov 30, 2024

Choose a reason for hiding this comment

catenacyber Dec 1, 2024

Choose a reason for hiding this comment

catenacyber Nov 28, 2024

Choose a reason for hiding this comment

jlucovsky Nov 30, 2024

Choose a reason for hiding this comment

catenacyber Nov 28, 2024

Choose a reason for hiding this comment

jlucovsky commented Nov 30, 2024

codecov bot commented Nov 21, 2024 •

edited

Loading