Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a threshold for expected zero values in the SPM script #5753

Merged
merged 3 commits into from
Jul 17, 2024
Merged

Add a threshold for expected zero values in the SPM script #5753

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
09a2c78
Add a threshold for expected zero values in the SPM script
FlamingSaint Jul 16, 2024
08c1cc3
Add better logs
FlamingSaint Jul 16, 2024
6a49726
Add suggested changes
FlamingSaint Jul 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions scripts/spm-integration-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,13 +61,18 @@ validate_service_metrics() {
# Store the values in an array
mapfile -t metric_points < <(echo "$response" | jq -r '.metrics[0].metricPoints[].gaugeValue.doubleValue')
echo "Metric datapoints found for service '$service': " "${metric_points[@]}"
# Check that all values are non-zero
# Check that atleast some values are non-zero after the threshold
local non_zero_count=0
local threshold=3
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
local non_zero_count=0
local threshold=3
local non_zero_count=0
local expected_non_zero_count=3
local zero_count=0
local expected_max_zero_count=3

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but tbh, I would consider deleting zero_count logic altogether. If zero values can be produced due to race conditions in the start-up, maybe we should only be looking for 3+ non-zero values and not worry about zeros.

If we still want to check for errors with a threshold, then it would make sense to only allow zeros at the beginning on the time series, once we start getting data I don't see a strong reason why zeros should show up again

Copy link
Member Author

@FlamingSaint FlamingSaint Jul 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea makes sense, I think we can just wait for 3 non-zero values. Maybe timeout if we still haven't gotten 3 values ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already have the timeout logic handled. This function just checks if "in the current state" the data satisfies the requirements. And that requirement could be 0{0-3} >0{3+}, i.e. no more than 3 leading zeroes followed by at least 3 non-zeros.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood. I will make the changes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@FlamingSaint I merged this for now to prevent spurious failures affecting other PRs, but let's make the change we discussed as a next improvement.

for value in "${metric_points[@]}"; do
if [[ $(echo "$value > 0.0" | bc) == "1" ]]; then
non_zero_count=$((non_zero_count + 1))
else
echo "❌ ERROR: Zero values not expected"
threshold=$((threshold - 1))
fi

if [[ $threshold -eq 0 ]]; then
echo "❌ ERROR: Zero values crossing threshold limit not expected (Threshold limit - '$threshold')"
return 1
fi
done
Expand Down
Loading