Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ISSUE] Not seeing metrics of all namespaces #255

Open
WaterKnight1998 opened this issue Nov 6, 2020 · 5 comments
Open

[ISSUE] Not seeing metrics of all namespaces #255

WaterKnight1998 opened this issue Nov 6, 2020 · 5 comments

Comments

@WaterKnight1998
Copy link

WaterKnight1998 commented Nov 6, 2020 •
edited
Loading

Hi, @jkohen @Pamir

I am adding this side car container to this deployment.

In GCP Monitoring I have selected k8s_container as Resource type and external/prometheus/go_memstats_alloc_bytes as Metric.

In pod_name I see the pods from the namespace where Prometheus is deployed. However, I cannot see the metrics for other namespaces although they are displayed from the Prometheus web interface.

Is it possible that metrics from other namespaces are not getting exported into stack driver? How can I get it works?
@jkohen
Copy link
Contributor

jkohen commented Nov 10, 2020

Hi sorry to hear about the trouble. I'm no longer involved on this project, someone from the Google Cloud team should pick this up soon.

@igorpeshansky
Copy link
Member

@WaterKnight1998 are there any errors in the container logs for the sidecar? What is the exact configuration of the sidecar?

@WaterKnight1998
Copy link
Author

WaterKnight1998 commented Nov 10, 2020 •
edited
Loading

@WaterKnight1998 are there any errors in the container logs for the sidecar? What is the exact configuration of the sidecar?

I am using it with kube-prometheus-stack. @igorpeshansky

Configuration:

       - name: sidecar
         image: gcr.io/stackdriver-prometheus/stackdriver-prometheus-sidecar:0.8.0
         imagePullPolicy: Always
         args:
           - '--stackdriver.project-id=k8s-project-example-waterknight'
           - '--prometheus.wal-directory=/data/wal'
           - '--stackdriver.kubernetes.location=europe-west6-a'
           - '--stackdriver.kubernetes.cluster-name=-k8s-cluster1'
         ports:
         - name: sidecar
           containerPort: 9091
         volumeMounts:
         - name: prometheus-prometheus-stack-kube-prom-prometheus-db
           mountPath: /data

Logs:

level=info ts=2020-11-10T17:29:52.956Z caller=main.go:293 msg="Starting Stackdriver Prometheus sidecar" version="(version=0.8.0, branch=master, revision=6540ca8d032a413513940f329ca2aaf14c267a53)"
level=info ts=2020-11-10T17:29:52.957Z caller=main.go:294 build_context="(go=go1.12, user=kbuilder@kokoro-gcp-ubuntu-prod-724580205, date=20200731-04:21:42)"
level=info ts=2020-11-10T17:29:52.957Z caller=main.go:295 host_details="(Linux 5.3.0-1036-gke #38-Ubuntu SMP Sun Sep 6 05:59:13 UTC 2020 x86_64 prometheus-prometheus-stack-kube-prom-prometheus-0 (none))"
level=info ts=2020-11-10T17:29:52.957Z caller=main.go:296 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2020-11-10T17:29:52.967Z caller=main.go:598 msg="Web server started"
level=info ts=2020-11-10T17:29:52.969Z caller=main.go:579 msg="Stackdriver client started"
level=info ts=2020-11-10T17:30:55.975Z caller=manager.go:153 component="Prometheus reader" msg="Starting Prometheus reader..."
level=info ts=2020-11-10T17:30:56.036Z caller=manager.go:215 component="Prometheus reader" msg="reached first record after start offset" start_offset=0 skipped_records=0
level=warn ts=2020-11-10T17:30:56.527Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:30:56.766Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:30:57.085Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:30:57.239Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[1-51,53,55-115]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[0]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[52,54]"
level=warn ts=2020-11-10T17:30:57.412Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:31:01.364Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[184].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[184]; Field timeSeries[185].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[185]; Field timeSeries[186].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[186]; Field timeSeries[187].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[187]; Field timeSeries[188].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[188]; Field timeSeries[189].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[189]; Field timeSeries[190].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[190]; Field timeSeries[191].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[191]; Field timeSeries[192].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[192]"
level=warn ts=2020-11-10T17:31:01.613Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:31:01.853Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[51].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[51]"
level=warn ts=2020-11-10T17:31:01.985Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:31:02.221Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[16-199]"
level=warn ts=2020-11-10T17:31:02.342Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-29,31-77,79,81-144]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[30]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[78,80]"
level=warn ts=2020-11-10T17:31:02.516Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:31:06.313Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[100].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[100]; Field timeSeries[101].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[101]; Field timeSeries[102].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[102]; Field timeSeries[103].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[103]; Field timeSeries[104].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[104]; Field timeSeries[72].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[72]; Field timeSeries[73].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[73]; Field timeSeries[74].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[74]; Field timeSeries[75].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[75]; Field timeSeries[76].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[76]; Field timeSeries[77].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[77]; Field timeSeries[78].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[78]; Field timeSeries[79].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[79]; Field timeSeries[80].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[80]; Field timeSeries[81].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[81]; Field timeSeries[82].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[82]; Field timeSeries[83].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[83]; Field timeSeries[84].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[84]; Field timeSeries[85].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[85]; Field timeSeries[86].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[86]; Field timeSeries[87].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[87]; Field timeSeries[88].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[88]; Field timeSeries[89].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[89]; Field timeSeries[90].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[90]; Field timeSeries[91].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[91]; Field timeSeries[92].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[92]; Field timeSeries[93].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[93]; Field timeSeries[94].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[94]; Field timeSeries[95].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[95]; Field timeSeries[96].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[96]; Field timeSeries[97].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[97]; Field timeSeries[98].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[98]; Field timeSeries[99].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[99]"
level=warn ts=2020-11-10T17:31:06.484Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:31:09.435Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[163].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[163]; Field timeSeries[164].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[164]; Field timeSeries[165].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[165]"
level=warn ts=2020-11-10T17:31:09.609Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:31:13.181Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:31:13.242Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-144,146-194,196,198,199]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[145]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[195,197]"
level=warn ts=2020-11-10T17:31:13.705Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:31:36.020Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:31:36.301Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:31:39.497Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[58].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[58]; Field timeSeries[59].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[59]; Field timeSeries[60].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[60]"
level=warn ts=2020-11-10T17:31:39.612Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:31:43.120Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[27-199]"
level=warn ts=2020-11-10T17:31:43.589Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-41,43-80,82,84-155]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[42]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[81,83]"
level=warn ts=2020-11-10T17:31:43.758Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:32:06.170Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:32:06.308Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:32:09.507Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[91].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[91]; Field timeSeries[92].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[92]; Field timeSeries[93].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[93]"
level=warn ts=2020-11-10T17:32:09.717Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:32:13.227Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:32:13.635Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-72,74-116,118,120-188]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[73]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[117,119]"
level=warn ts=2020-11-10T17:32:13.875Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:32:36.201Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:32:36.319Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:32:39.516Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[91].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[91]; Field timeSeries[92].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[92]; Field timeSeries[93].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[93]"
level=warn ts=2020-11-10T17:32:39.689Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:32:43.200Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:32:43.605Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-72,74-116,118,120-188]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[73]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[117,119]"
level=warn ts=2020-11-10T17:32:43.782Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:33:06.144Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:33:06.624Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:33:09.347Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[58].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[58]; Field timeSeries[59].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[59]; Field timeSeries[60].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[60]"
level=warn ts=2020-11-10T17:33:09.483Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:33:12.993Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[27-199]"
level=warn ts=2020-11-10T17:33:13.618Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-39,41-88,90,92-155]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[40]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[89,91]"
level=warn ts=2020-11-10T17:33:13.829Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:33:36.221Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:33:36.337Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:33:39.537Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[91].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[91]; Field timeSeries[92].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[92]; Field timeSeries[93].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[93]"
level=warn ts=2020-11-10T17:33:39.712Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:33:43.219Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:33:43.631Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-72,74-114,116-126,128-188]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[73]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[115,127]"
level=warn ts=2020-11-10T17:33:43.804Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:34:06.201Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:34:06.333Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:34:09.680Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[91].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[91]; Field timeSeries[92].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[92]; Field timeSeries[93].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[93]"
level=warn ts=2020-11-10T17:34:09.856Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:34:13.294Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:34:13.678Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-73,75-118,120,122-188]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[74]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[119,121]"
level=warn ts=2020-11-10T17:34:13.856Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:34:32.964Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[117].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[117]; Field timeSeries[118].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[118]; Field timeSeries[119].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[119]; Field timeSeries[120].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[120]; Field timeSeries[121].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[121]; Field timeSeries[122].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[122]; Field timeSeries[123].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[123]; Field timeSeries[124].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[124]; Field timeSeries[125].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[125]; Field timeSeries[126].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[126]; Field timeSeries[127].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[127]; Field timeSeries[128].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[128]; Field timeSeries[129].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[129]; Field timeSeries[130].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[130]; Field timeSeries[131].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[131]; Field timeSeries[132].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[132]; Field timeSeries[133].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[133]; Field timeSeries[134].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[134]; Field timeSeries[135].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[135]; Field timeSeries[136].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[136]; Field timeSeries[137].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[137]; Field timeSeries[138].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[138]; Field timeSeries[139].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[139]; Field timeSeries[140].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[140]; Field timeSeries[141].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[141]; Field timeSeries[142].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[142]; Field timeSeries[143].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[143]; Field timeSeries[144].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[144]; Field timeSeries[145].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[145]; Field timeSeries[146].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[146]; Field timeSeries[147].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[147]; Field timeSeries[148].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[148]; Field timeSeries[149].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[149]"
level=warn ts=2020-11-10T17:34:36.345Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:34:39.551Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[58].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[58]; Field timeSeries[59].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[59]; Field timeSeries[60].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[60]"
level=warn ts=2020-11-10T17:34:39.689Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:34:43.202Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[27-199]"
level=warn ts=2020-11-10T17:34:43.579Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-40,42-91,93,95-155]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[41]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[92,94]"
level=warn ts=2020-11-10T17:34:43.747Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:35:06.243Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:35:06.355Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:35:09.580Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[91].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[91]; Field timeSeries[92].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[92]; Field timeSeries[93].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[93]"
level=warn ts=2020-11-10T17:35:09.788Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:35:13.279Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:35:13.605Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-72,74-113,115-125,127-188]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[73]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[114,126]"
level=warn ts=2020-11-10T17:35:13.784Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:35:36.069Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:35:36.591Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:35:39.310Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[58].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[58]; Field timeSeries[59].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[59]; Field timeSeries[60].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[60]"
level=warn ts=2020-11-10T17:35:39.428Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:35:42.934Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[27-199]"
level=warn ts=2020-11-10T17:35:43.574Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-39,41-91,93,95-155]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[40]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[92,94]"
level=warn ts=2020-11-10T17:35:43.749Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:36:06.286Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:36:06.640Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:36:09.340Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[58].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[58]; Field timeSeries[59].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[59]; Field timeSeries[60].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[60]"
level=warn ts=2020-11-10T17:36:09.494Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:36:12.986Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[27-199]"
level=warn ts=2020-11-10T17:36:13.631Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-42,44-89,91,93-155]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[43]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[90,92]"
level=warn ts=2020-11-10T17:36:13.840Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:36:35.961Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:36:36.366Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:36:39.586Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[58].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[58]; Field timeSeries[59].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[59]; Field timeSeries[60].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[60]"
level=warn ts=2020-11-10T17:36:39.702Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:36:43.211Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[27-199]"
level=warn ts=2020-11-10T17:36:43.596Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-39,41-80,82-92,94-155]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[40]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[81,93]"
level=warn ts=2020-11-10T17:36:43.793Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:37:06.227Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:37:06.340Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:37:09.542Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[91].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[91]; Field timeSeries[92].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[92]; Field timeSeries[93].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[93]"
level=warn ts=2020-11-10T17:37:09.775Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:37:13.054Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:37:13.594Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-75,77-124,126,128-188]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[76]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[125,127]"
level=warn ts=2020-11-10T17:37:13.768Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:37:36.164Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:37:36.283Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:37:39.572Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[91].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[91]; Field timeSeries[92].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[92]; Field timeSeries[93].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[93]"
level=warn ts=2020-11-10T17:37:39.761Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:37:43.209Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:37:43.578Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-75,77-122,124,126-188]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[76]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[123,125]"
level=warn ts=2020-11-10T17:37:43.794Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:38:06.548Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:38:06.669Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:38:09.312Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[91].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[91]; Field timeSeries[92].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[92]; Field timeSeries[93].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[93]"
level=warn ts=2020-11-10T17:38:09.506Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:38:13.050Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:38:13.607Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-72,74-124,126,128-188]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[73]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[125,127]"
level=warn ts=2020-11-10T17:38:13.837Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:38:36.054Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:38:36.295Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:38:39.513Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[58].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[58]; Field timeSeries[59].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[59]; Field timeSeries[60].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[60]"
level=warn ts=2020-11-10T17:38:39.628Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:38:43.137Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[27-199]"
level=warn ts=2020-11-10T17:38:43.604Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-41,43-80,82-92,94-155]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[42]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[81,93]"
level=warn ts=2020-11-10T17:38:43.773Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:39:06.195Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:39:06.345Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:39:09.540Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[91].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[91]; Field timeSeries[92].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[92]; Field timeSeries[93].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[93]"
level=warn ts=2020-11-10T17:39:09.752Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:39:13.276Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:39:13.645Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-72,74-119,121,123-188]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[73]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[120,122]"
level=warn ts=2020-11-10T17:39:13.819Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:39:36.296Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[0].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[0]; Field timeSeries[10].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[10]; Field timeSeries[11].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[11]; Field timeSeries[12].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[12]; Field timeSeries[13].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[13]; Field timeSeries[14].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_vmstat_pgmajfault:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[14]; Field timeSeries[15].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[15]; Field timeSeries[16].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[16]; Field timeSeries[17].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[17]; Field timeSeries[18].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[18]; Field timeSeries[19].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[19]; Field timeSeries[1].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[1]; Field timeSeries[20].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance_device:node_disk_io_time_weighted_seconds:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[20]; Field timeSeries[21].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[21]; Field timeSeries[22].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[22]; Field timeSeries[23].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[23]; Field timeSeries[24].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[24]; Field timeSeries[25].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[25]; Field timeSeries[26].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_bytes_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[26]; Field timeSeries[27].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[27]; Field timeSeries[28].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[28]; Field timeSeries[29].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_receive_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[29]; Field timeSeries[2].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_num_cpu:sum\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[2]; Field timeSeries[30].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[30]; Field timeSeries[31].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[31]; Field timeSeries[32].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_network_transmit_drop_excluding_lo:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[32]; Field timeSeries[3].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[3]; Field timeSeries[4].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[4]; Field timeSeries[5].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu_utilisation:rate1m\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[5]; Field timeSeries[6].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[6]; Field timeSeries[7].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[7]; Field timeSeries[8].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_load1_per_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[8]; Field timeSeries[9].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_memory_utilisation:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[9]"
level=warn ts=2020-11-10T17:39:36.433Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:39:39.295Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[91].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[91]; Field timeSeries[92].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[92]; Field timeSeries[93].metric.type had an invalid value of \"external.googleapis.com/prometheus/instance:node_cpu:ratio\": The metric type must be a URL-formatted string with a domain and non-empty path.: timeSeries[93]"
level=warn ts=2020-11-10T17:39:39.482Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:39:43.294Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"
level=warn ts=2020-11-10T17:39:43.582Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-72,74-117,119,121-188]; The new labels would cause the metric external.googleapis.com/prometheus/kube_replicaset_labels to have over 10 labels.: timeSeries[73]; The new labels would cause the metric external.googleapis.com/prometheus/kube_secret_labels to have over 10 labels.: timeSeries[118,120]"
level=warn ts=2020-11-10T17:39:43.780Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = Internal desc = One or more TimeSeries could not be written: Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.: timeSeries[0-199]"

@clguo-tw
Copy link

same problem

@Vincoll
Copy link

Vincoll commented Jan 13, 2021

Same problem,
Metrics from kube-state-metrics that concerns others Namespaces are not present in Cloud Monitoring.
eg : kube_pod_created


KO :  kube_pod_created{container="kube-state-metrics",endpoint="http",instance="10.104.1.13:8080",job="kube-state-metrics",namespace="OtherNamespace",pod="xxx-75bb55f4f6-xgj2s",service="prom-operator-kube-state-metrics"}


OK : kube_pod_created{container="kube-state-metrics",endpoint="http",instance="10.104.1.13:8080",job="kube-state-metrics",namespace="SameNamespace",pod="prom-operator-kube-prometh-operator-5cff8d48f5-xjmqv",service="prom-operator-kube-state-metrics"}

I just have one unrelated error log :
One or more TimeSeries could not be written: The new labels would cause the metric external.googleapis.com/prometheus/kube_deployment_labels to have over 10 labels

Metrics from others sources are OK.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Vincoll @jkohen @igorpeshansky @WaterKnight1998 @clguo-tw