[TT-11912]: Added Analytics expiry period to OAS (#6825)

### **User description**
<details open>
<summary><a href="https://tyktech.atlassian.net/browse/TT-11912"
title="TT-11912" target="_blank">TT-11912</a></summary>
  <br />
  <table>
    <tr>
      <th>Summary</th>
      <td>[OAS] Analytics expiry period</td>
    </tr>
    <tr>
      <th>Type</th>
      <td>
<img alt="Story"
src="https://tyktech.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10315?size=medium"
/>
        Story
      </td>
    </tr>
    <tr>
      <th>Status</th>
      <td>In Dev</td>
    </tr>
    <tr>
      <th>Points</th>
      <td>N/A</td>
    </tr>
    <tr>
      <th>Labels</th>
      <td>-</td>
    </tr>
  </table>
</details>
<!--
  do not remove this marker as it will break jira-lint's functionality.
  added_by_jira_lint
-->

---

<!-- Provide a general summary of your changes in the Title above -->

## Description
Added retention period to the analytics config in the OAS definition
[TT-11912](https://tyktech.atlassian.net/browse/TT-11912)

<!-- Describe your changes in detail -->

## Related Issue

<!-- This project only accepts pull requests related to open issues. -->
<!-- If suggesting a new feature or change, please discuss it in an
issue first. -->
<!-- If fixing a bug, there should be an issue describing it with steps
to reproduce. -->
<!-- OSS: Please link to the issue here. Tyk: please create/link the
JIRA ticket. -->

## Motivation and Context

<!-- Why is this change required? What problem does it solve? -->

## How This Has Been Tested

<!-- Please describe in detail how you tested your changes -->
<!-- Include details of your testing environment, and the tests -->
<!-- you ran to see how your change affects other areas of the code,
etc. -->
<!-- This information is helpful for reviewers and QA. -->

## Screenshots (if appropriate)

## Types of changes

<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Refactoring or add test (improvements in base code or adds test
coverage to functionality)

## Checklist

<!-- Go over all the following points, and put an `x` in all the boxes
that apply -->
<!-- If there are no documentation updates required, mark the item as
checked. -->
<!-- Raise up any additional concerns not covered by the checklist. -->

- [ ] I ensured that the documentation is up to date
- [ ] I explained why this PR updates go.mod in detail with reasoning
why it's required
- [ ] I would like a code coverage CI quality gate exception and have
explained why


[TT-11912]:
https://tyktech.atlassian.net/browse/TT-11912?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


___

### **PR Type**
Enhancement, Tests


___

### **Description**
- Added `DisableExpireAnalytics` field to API definition for analytics
expiration control.

- Introduced `RetentionPeriod` struct for detailed analytics retention
configuration.

- Updated middleware logic to handle retention period extraction and
filling.

- Added unit tests for retention period functionality in traffic logs.


___



### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>api_definitions.go</strong><dd><code>Introduced
`DisableExpireAnalytics` field in API definition</code></dd></summary>
<hr>

apidef/api_definitions.go

<li>Added <code>DisableExpireAnalytics</code> field to API
definition.<br> <li> Enhanced analytics expiration functionality with a
new field.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6825/files#diff-9961ccc89a48d32db5b47ba3006315ef52f6e5007fb4b09f8c5d6d299c669d67">+1/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
  <td>
    <details>
<summary><strong>migration.go</strong><dd><code>Updated migration logic
for analytics expiration</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

apidef/migration.go

<li>Set <code>DisableExpireAnalytics</code> to true in
<code>SetDisabledFlags</code>.<br> <li> Ensured analytics expiration is
disabled in migration logic.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6825/files#diff-e1d9b55a26f9d6225d56d6f0161959217308e5ad4d6934e7d7df4595d9c2a130">+1/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
  <td>
    <details>
<summary><strong>middleware.go</strong><dd><code>Added `RetentionPeriod`
struct and related logic</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

apidef/oas/middleware.go

<li>Added <code>RetentionPeriod</code> struct for analytics retention
configuration.<br> <li> Updated traffic logs logic to handle retention
period.<br> <li> Implemented <code>Fill</code> and
<code>ExtractTo</code> methods for <code>RetentionPeriod</code>.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6825/files#diff-992ec7c28d25fd54f6491d295389757705cd114bc869a35cba50d42e548cdc6e">+61/-0</a>&nbsp;
&nbsp; </td>

</tr>
</table></td></tr><tr><td><strong>Tests</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>middleware_test.go</strong><dd><code>Added tests for
analytics retention period</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
</dd></summary>
<hr>

apidef/oas/middleware_test.go

<li>Added unit tests for <code>RetentionPeriod</code> functionality.<br>
<li> Verified retention period extraction and filling logic.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6825/files#diff-0af31cb29ae298a6ac3e402b283ab364a6fd793fd04f253ef7c4983234c17bef">+22/-0</a>&nbsp;
&nbsp; </td>

</tr>
</table></td></tr></tr></tbody></table>

___

> 💡 **PR-Agent usage**: Comment `/help "your question"` on any pull
request to receive relevant information

---------

Co-authored-by: Tit Petric <[email protected]>

apidef/api_definitions.go

@@ -750,6 +750,7 @@ type APIDefinition struct {
 	IPAccessControlDisabled              bool                   `mapstructure:"ip_access_control_disabled" bson:"ip_access_control_disabled" json:"ip_access_control_disabled"`
 	DontSetQuotasOnCreate                bool                   `mapstructure:"dont_set_quota_on_create" bson:"dont_set_quota_on_create" json:"dont_set_quota_on_create"`
 	ExpireAnalyticsAfter                 int64                  `mapstructure:"expire_analytics_after" bson:"expire_analytics_after" json:"expire_analytics_after"` // must have an expireAt TTL index set (http://docs.mongodb.org/manual/tutorial/expire-data/)
+	DisableExpireAnalytics               bool                   `mapstructure:"disable_expire_analytics" bson:"disable_expire_analytics" json:"disable_expire_analytics"`
 	ResponseProcessors                   []ResponseProcessor    `bson:"response_processors" json:"response_processors"`
 	CORS                                 CORSConfig             `bson:"CORS" json:"CORS"`
 	Domain                               string                 `bson:"domain" json:"domain"`

apidef/migration.go

@@ -438,6 +438,7 @@ func (a *APIDefinition) SetDisabledFlags() {
 	a.ConfigDataDisabled = true
 	a.Proxy.ServiceDiscovery.CacheDisabled = true
 	a.UptimeTests.Config.ServiceDiscovery.CacheDisabled = true
+	a.DisableExpireAnalytics = true
 
 	for i := 0; i < len(a.CustomMiddleware.Pre); i++ {
 		a.CustomMiddleware.Pre[i].Disabled = true

apidef/migration_test.go

@@ -642,6 +642,7 @@ func TestSetDisabledFlags(t *testing.T) {
 		},
 	}
 	expectedAPIDef := APIDefinition{
+		DisableExpireAnalytics: true,
 		CustomMiddleware: MiddlewareSection{
 			AuthCheck: MiddlewareDefinition{
 				Disabled: true,

apidef/oas/linter_test.go

@@ -89,6 +89,7 @@ func TestXTykGateway_Lint(t *testing.T) {
 			BasicAuth: nil,
 			OAuth:     nil,
 		}
+		settings.Middleware.Global.TrafficLogs.RetentionPeriod.Value = ReadableDuration(time.Minute * 10)
 	}
 
 	// Encode data to json

apidef/oas/middleware.go

@@ -5,6 +5,7 @@ import (
 	"net/http"
 	"sort"
 	"strings"
+	"time"
 
 	"github.com/TykTechnologies/tyk/apidef"
 )
@@ -1580,18 +1581,73 @@ type TrafficLogs struct {
 	// TagHeaders is a string array of HTTP headers that can be extracted
 	// and transformed into analytics tags (statistics aggregated by tag, per hour).
 	TagHeaders []string `bson:"tagHeaders" json:"tagHeaders,omitempty"`
+	// RetentionPeriod holds the configuration for the analytics retention, it contains configuration
+	// for how long you would like analytics data to last for.
+	RetentionPeriod *RetentionPeriod `bson:"retentionPeriod" json:"retentionPeriod,omitempty"`
 }
 
 // Fill fills *TrafficLogs from apidef.APIDefinition.
 func (t *TrafficLogs) Fill(api apidef.APIDefinition) {
 	t.Enabled = !api.DoNotTrack
 	t.TagHeaders = api.TagHeaders
+
+	if t.RetentionPeriod == nil {
+		t.RetentionPeriod = &RetentionPeriod{}
+	}
+	t.RetentionPeriod.Fill(api)
+	if ShouldOmit(t.RetentionPeriod) {
+		t.RetentionPeriod = nil
+	}
 }
 
 // ExtractTo extracts *TrafficLogs into *apidef.APIDefinition.
 func (t *TrafficLogs) ExtractTo(api *apidef.APIDefinition) {
 	api.DoNotTrack = !t.Enabled
 	api.TagHeaders = t.TagHeaders
+
+	if t.RetentionPeriod == nil {
+		t.RetentionPeriod = &RetentionPeriod{}
+		defer func() {
+			t.RetentionPeriod = nil
+		}()
+	}
+	t.RetentionPeriod.ExtractTo(api)
+}
+
+// RetentionPeriod holds the configuration for analytics retention.
+type RetentionPeriod struct {
+	// Enabled enables log analytics retention for the API
+	//
+	// Tyk classic API definition: `!disable_expire_analytics`.
+	Enabled bool `bson:"enabled" json:"enabled"`
+	// Value is the interval to keep the analytics record for
+	// The value of Value is a string that specifies the interval in a compact form,
+	// where hours, minutes and seconds are denoted by 'h', 'm' and 's' respectively.
+	// Multiple units can be combined to represent the duration.
+	//
+	// Examples of valid shorthand notations:
+	// - "1h"   : one hour
+	// - "20m"  : twenty minutes
+	// - "30s"  : thirty seconds
+	// - "1m29s": one minute and twenty-nine seconds
+	// - "1h30m" : one hour and thirty minutes
+	//
+	// An empty value is interpreted as "0s"
+	//
+	// Tyk classic API definition: `expire_analytics_after`.
+	Value ReadableDuration `bson:"value" json:"value"`
+}
+
+// Fill fills *RetentionPeriod from apidef.APIDefinition
+func (r *RetentionPeriod) Fill(api apidef.APIDefinition) {
+	r.Enabled = !api.DisableExpireAnalytics
+	r.Value = ReadableDuration(time.Duration(api.ExpireAnalyticsAfter) * time.Second)
+}
+
+// ExtractTo extracts *RetentionPeriod into apidef.APIDefinition
+func (r *RetentionPeriod) ExtractTo(api *apidef.APIDefinition) {
+	api.DisableExpireAnalytics = !r.Enabled
+	api.ExpireAnalyticsAfter = int64(r.Value.Seconds())
 }
 
 // GlobalRequestSizeLimit holds configuration about the global limits for request sizes.

apidef/oas/middleware_test.go

@@ -3,6 +3,7 @@ package oas
 import (
 	"encoding/json"
 	"testing"
+	"time"
 
 	"github.com/stretchr/testify/assert"
 
@@ -219,6 +220,46 @@ func TestTrafficLogs(t *testing.T) {
 		trafficLogs.ExtractTo(&convertedAPI)
 		assert.Empty(t, convertedAPI.TagHeaders)
 	})
+
+	t.Run("retention header enabled", func(t *testing.T) {
+		trafficLogs := TrafficLogs{
+			Enabled: true,
+			RetentionPeriod: &RetentionPeriod{
+				Enabled: true,
+				Value:   ReadableDuration(time.Minute * 2),
+			},
+		}
+
+		var convertedAPI apidef.APIDefinition
+		var resultTrafficLogs TrafficLogs
+		convertedAPI.SetDisabledFlags()
+		trafficLogs.ExtractTo(&convertedAPI)
+
+		assert.Equal(t, trafficLogs.RetentionPeriod.Enabled, !convertedAPI.DisableExpireAnalytics)
+		assert.Equal(t, int64(120), convertedAPI.ExpireAnalyticsAfter)
+
+		resultTrafficLogs.Fill(convertedAPI)
+		assert.Equal(t, trafficLogs, resultTrafficLogs)
+	})
+
+	t.Run("retention header disabled", func(t *testing.T) {
+		trafficLogs := TrafficLogs{
+			Enabled:         true,
+			RetentionPeriod: nil,
+		}
+
+		var convertedAPI apidef.APIDefinition
+		var resultTrafficLogs TrafficLogs
+
+		convertedAPI.SetDisabledFlags()
+		trafficLogs.ExtractTo(&convertedAPI)
+
+		assert.Equal(t, true, convertedAPI.DisableExpireAnalytics)
+		assert.Equal(t, int64(0), convertedAPI.ExpireAnalyticsAfter)
+
+		resultTrafficLogs.Fill(convertedAPI)
+		assert.Nil(t, resultTrafficLogs.RetentionPeriod)
+	})
 }
 
 func TestPluginConfig(t *testing.T) {

apidef/oas/oas_test.go

@@ -174,6 +174,7 @@ func TestOAS_ExtractTo_ResetAPIDefinition(t *testing.T) {
 	a.DisableRateLimit = false
 	a.DoNotTrack = false
 	a.IPAccessControlDisabled = false
+	a.DisableExpireAnalytics = false
 
 	// deprecated fields
 	a.Auth = apidef.AuthConfig{}
@@ -266,7 +267,6 @@ func TestOAS_ExtractTo_ResetAPIDefinition(t *testing.T) {
 		"APIDefinition.EnableIpWhiteListing",
 		"APIDefinition.EnableIpBlacklisting",
 		"APIDefinition.DontSetQuotasOnCreate",
-		"APIDefinition.ExpireAnalyticsAfter",
 		"APIDefinition.ResponseProcessors[0].Name",
 		"APIDefinition.ResponseProcessors[0].Options",
 		"APIDefinition.GraphQL.Enabled",
@@ -948,6 +948,9 @@ func TestMigrateAndFillOAS_DropEmpties(t *testing.T) {
 			Global: &Global{
 				TrafficLogs: &TrafficLogs{
 					Enabled: true,
+					RetentionPeriod: &RetentionPeriod{
+						Enabled: true,
+					},
 				},
 			},
 		}, baseAPI.OAS.GetTykExtension().Middleware)

apidef/oas/schema/x-tyk-api-gateway.json

@@ -1997,6 +1997,22 @@
           "items": {
             "type": "string"
           }
+        },
+        "retentionPeriod": {
+          "type": "object",
+          "properties": {
+            "enabled": {
+              "type": "boolean"
+            },
+            "value": {
+              "type": "string",
+              "pattern": "^(\\d+h)?(\\d+m)?(\\d+s)?$"
+            }
+          },
+          "required": [
+            "enabled",
+            "value"
+          ]
         }
       },
       "required": [

apidef/schema.json

@@ -19,6 +19,9 @@
     "expire_analytics_after": {
       "type": "number"
     },
+    "disable_expire_analytics": {
+      "type": "boolean"
+    },
     "id": {
       "type": "string"
     },

gateway/handler_error.go

@@ -293,7 +293,9 @@ func (e *ErrorHandler) HandleError(w http.ResponseWriter, r *http.Request, errMs
 			if orgExpireDataTime > 0 {
 				expiresAfter = orgExpireDataTime
 			}
-
+		}
+		if e.Spec.DisableExpireAnalytics {
+			expiresAfter = 0
 		}
 		record.SetExpiry(expiresAfter)
 

gateway/handler_success.go

@@ -305,6 +305,10 @@ func (s *SuccessHandler) RecordHit(r *http.Request, timing analytics.Latency, co
 			}
 		}
 
+		if s.Spec.DisableExpireAnalytics {
+			expiresAfter = 0
+		}
+
 		record.SetExpiry(expiresAfter)
 
 		if s.Spec.GlobalConfig.AnalyticsConfig.NormaliseUrls.Enabled {

gateway/proxy_muxer.go

@@ -287,7 +287,11 @@ func (gw *Gateway) flushNetworkAnalytics(ctx context.Context) {
 					APIID:        spec.APIID,
 					OrgID:        spec.OrgID,
 				}
-				record.SetExpiry(spec.ExpireAnalyticsAfter)
+				if spec.DisableExpireAnalytics {
+					record.SetExpiry(0)
+				} else {
+					record.SetExpiry(spec.ExpireAnalyticsAfter)
+				}
 				_ = gw.Analytics.RecordHit(&record)
 			}
 			gw.apisMu.RUnlock()