Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle null bot name correctly when trying to find region in monitoring #4592

Merged
merged 3 commits into from
Jan 8, 2025

Conversation

vitorguidi
Copy link
Collaborator

This fixes the following error in app engine crons:

Failed to flush metrics: expected string or bytes-like object, got 'NoneType'
Traceback (most recent call last):
  File "/layers/google.python.runtime/python/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/layers/google.python.runtime/python/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/layers/google.python.runtime/python/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/srv/clusterfuzz/_internal/metrics/monitor.py", line 171, in _flush_loop
    self._flush_function()
  File "/srv/clusterfuzz/_internal/metrics/monitor.py", line 134, in _flush_metrics
    logs.error(f'Failed to flush metrics: {e}')
LogError: Failed to flush metrics: expected string or bytes-like object, got 'NoneType'
Traceback (most recent call last):
  File "/srv/clusterfuzz/_internal/metrics/monitor.py", line 118, in _flush_metrics
    metric.monitoring_v3_time_series(series, labels, start_time, end_time,
  File "/srv/clusterfuzz/_internal/metrics/monitor.py", line 343, in monitoring_v3_time_series
    self.monitoring_v3_metric(time_series.metric, labels)
  File "/srv/clusterfuzz/_internal/metrics/monitor.py", line 321, in monitoring_v3_metric
    metric.labels['region'] = _get_region(bot_name)
                              ^^^^^^^^^^^^^^^^^^^^^
  File "/srv/clusterfuzz/_internal/metrics/monitor.py", line 637, in _get_region
    if re.match(pattern['pattern'], bot_name):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.runtime/python/lib/python3.11/re/__init__.py", line 166, in match
    return _compile(pattern, flags).match(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'NoneType'

@@ -625,6 +624,9 @@ def metrics_store():

def _get_region(bot_name):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not going to work for batch jobs.
THere are two ways to resolve this:

  1. Include the region in the batch job names.
  2. use some kind of GCE metadata server or something similaar to determine the region.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's already broken for batch jobs btw, so LGTMing

Copy link
Collaborator

@jonathanmetzman jonathanmetzman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@@ -625,6 +624,9 @@ def metrics_store():

def _get_region(bot_name):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's already broken for batch jobs btw, so LGTMing

@vitorguidi vitorguidi merged commit 738e024 into master Jan 8, 2025
7 checks passed
@vitorguidi vitorguidi deleted the chore/fix-gae-metrics-region branch January 8, 2025 18:55
jonathanmetzman pushed a commit that referenced this pull request Jan 8, 2025
…ng (#4592)

This fixes the following error in app engine crons:

```
Failed to flush metrics: expected string or bytes-like object, got 'NoneType'
Traceback (most recent call last):
  File "/layers/google.python.runtime/python/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/layers/google.python.runtime/python/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/layers/google.python.runtime/python/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/srv/clusterfuzz/_internal/metrics/monitor.py", line 171, in _flush_loop
    self._flush_function()
  File "/srv/clusterfuzz/_internal/metrics/monitor.py", line 134, in _flush_metrics
    logs.error(f'Failed to flush metrics: {e}')
LogError: Failed to flush metrics: expected string or bytes-like object, got 'NoneType'
Traceback (most recent call last):
  File "/srv/clusterfuzz/_internal/metrics/monitor.py", line 118, in _flush_metrics
    metric.monitoring_v3_time_series(series, labels, start_time, end_time,
  File "/srv/clusterfuzz/_internal/metrics/monitor.py", line 343, in monitoring_v3_time_series
    self.monitoring_v3_metric(time_series.metric, labels)
  File "/srv/clusterfuzz/_internal/metrics/monitor.py", line 321, in monitoring_v3_metric
    metric.labels['region'] = _get_region(bot_name)
                              ^^^^^^^^^^^^^^^^^^^^^
  File "/srv/clusterfuzz/_internal/metrics/monitor.py", line 637, in _get_region
    if re.match(pattern['pattern'], bot_name):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.runtime/python/lib/python3.11/re/__init__.py", line 166, in match
    return _compile(pattern, flags).match(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'NoneType'
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants