Skip to content

Latest commit

 

History

History
76 lines (55 loc) · 2.87 KB

known-issues.md

File metadata and controls

76 lines (55 loc) · 2.87 KB
copyright lastupdated
years
2018
2018-11-16

{:shortdesc: .shortdesc} {:new_window: target="_blank"} {:codeblock: .codeblock} {:pre: .pre} {:screen: .screen} {:tip: .tip}

Known issues (limitations)

{: #known-issues}

Instability due to high cpu usage by the Firehose Prometheus exporter

{: #instabilityCPU-issue}

CFEE environments include a Prometheus Firehose exporter for collecting Cloud Foundry metrics on http and https start and stop requests . A CFEE environment with a high number of http requests may cause the Prometheus firehose exporter to use a large amount of CPU. This can lead to some instability in the environment since the firehouse exporter pod runs on one of the Cloud Foundry control plane worker nodes.

You can determine if the firehose exporter pod is using high CPU, and potentially causing instability in the CFEE environment, by one of the following methods:

  1. Check the overall CPU usage of the CF control plane via the CFEE Overview page for that CFEE environment. See an illustrative example of high-cpu usage below: High CPU in Overview page

  2. Check the CPU usage in the Resource Usage > Applications page and see if there is an unusually high number of http requests. See an illustrative example of high-cpu usage below: High CPU in Reource Usage page

  3. Check the POD Grafana console (launched from the Monitoring page) that shows CPU usage for the Firehouse pod. See an illustrative example of high-cpu usage below: High CPU in Grafana console

If the firehouse exporter is using betwen 0.5 and 0.7 (50-70%) CPU, and the overall Control plane CPU is high, you may need to turn off the http request metric collection via the following steps:

  1. Create the config.yaml:

    kubectl -n cf-monitoring get deployment firehose-exporter -f yaml > config.yaml

    {: pre}

  2. Edit the config.yaml and add the following line in spec.template.spec.containers.args array:

    - --filter.events=ContainerMetric,CounterEvent,ValueMetric

    {: pre}

Example

Modified config.yam:

  ...
  template:
  ...
    spec:
      containers:
      - args:
        ...
        - --doppler.metric-expiration=10m
        - --metrics.environment=IBM-Cloud-Foundry-Enterprise-Environment
        - --filter.events=ContainerMetric,CounterEvent,ValueMetric
        image: boshprometheus/firehose-exporter

The following command would reconfigure and restart the worker node pod:

kubectl -n cf-monitoring apply -f config.yaml

{: pre}

For more information on the Firehose Prometheus exporter see Cloud Foundry Firehose Prometheus exporter{: new_window} External link icon.