Skip to content

Latest commit

 

History

History
270 lines (212 loc) · 11.6 KB

gitlab.md

File metadata and controls

270 lines (212 loc) · 11.6 KB
Raw

gitlab

Monitor Type: gitlab (Source)

Accepts Endpoints: Yes

Multiple Instances Allowed: Yes

Overview

GitLab is an open-source web-based git repository manager developed by GitLab Inc. GitLab has built-in features for creating wiki pages, issue-tracking and CI/CD pipelines. GitLab is bundled with Prometheus exporters that can be configured to export performance metrics of itself and that of the bundled software that GitLab depends on. These exporters publish Prometheus metrics at endpoints are scraped by this monitor.

REQUIREMENTS AND DEPENDENCIES

Software Version
GitLab 9.3+

INSTALLATION

CONFIGURATION

GitLab Configuration

Follow the instructions here to configure the GitLab's Prometheus exporters to expose metric endpoint targets. For GitLab Runner monitoring configuration go here.

Note that configuring GitLab by editing /etc/gitlab/gitlab.rb should be accompanied by running the command gitlab-ctl reconfigure in order for the changes to take effect.

Also, configuring Nginx by editing the file /var/opt/gitlab/nginx/conf/nginx-status.conf, for instance, should be accompanied by running command gitlab-ctl restart. Note that changes to the configuration file /var/opt/gitlab/nginx/conf/nginx-status.conf in particular are erased by subsequent runs of command gitlab-ctl reconfigure because gitlab-ctl reconfigure restores the original configuration file.

Below is a list of some of the Prometheus endpoint targets with links to their respective configuration pages. Note that target gitlab_monitor metrics are just targets gitlab_monitor_database, gitlab_monitor_process and gitlab_monitor_sidekiq metrics combined.

Agent Monitor Type Gitlab Doc Standard Port Standard Path
gitlab Gitlab doc 9168 /metrics
gitlab-gitaly Gitlab doc 9236 /metrics
gitlab-sidekiq Gitlab doc 8082 /metrics
gitlab-unicorn Gitlab doc 8080 /-/metrics
gitlab-workhorse Gitlab doc 9229 /metrics
prometheus/nginx-vts Gitlab doc 8060 /metrics
prometheus/node Gitlab doc 9100 /metrics
promteheus/postgres Gitlab doc 9187 /metrics
prometheus/prometheus Gitlab doc 9090 /metrics
prometheus/redis Gitlab doc 9121 /metrics
gitlab-runner Gitlab doc 9252 /metrics

GitLab Prometheus exporters, Nginx and GitLab Runner must be configured to listen to IP address(es) that include the IP address of the host or docker container of the SignalFx Smart Agent. For example, the configuration below in /etc/gitlab/gitlab.rb configures the GitLab Postgres Prometheus exporter to allow network connections on port 9187 from any IP address.

postgres_exporter['listen_address'] = '0.0.0.0:9187'

The above configuration can also be written as:

postgres_exporter['listen_address'] = ':9187'

Below is part of file /var/opt/gitlab/nginx/conf/nginx-status.conf showing the location /metrics block for metric related configuration. This file configures Nginx. The statement allow 172.17.0.0/16; allows network connection in the 172.17.0.0/16 IP range. The assumption is that the IP address associated with the SignalFx Smart Agent is in that IP range.

server {
    ...
    location /metrics {
    ...
    allow 172.17.0.0/16;
    deny all;
    }
}

Below is part of the global section of /etc/gitlab-runner/config.toml. This file configures GitLab Runner. The statement below configures GitLab Runner's Prometheus metrics HTTP server to allows network connection on port 9252 from any IP address.

listen_address = "0.0.0.0:9252"
...

Sample Config

To monitor everything we support in Gitlab, use the following configuration in the Smart Agent config:

monitors:
 - type: gitlab-unicorn
   host: localhost
   port: 8080

 - type: gitlab
   host: localhost
   port: 9168

 - type: gitlab-runner
   host: localhost
   port: 9252

 - type: gitlab-workhorse
   host: localhost
   port: 9229

 - type: gitlab-sidekiq
   host: localhost
   port: 8082

 - type: gitlab-gitaly
   host: localhost
   port: 9236

 - type: prometheus/postgres
   host: localhost
   port: 9187

 - type: prometheus/nginx-vts
   host: localhost
   port: 8060

You can, of course, use auto-discovery by specifying a discoveryRule instead of host and port.

Configuration

To activate this monitor in the Smart Agent, add the following to your agent config:

monitors:  # All monitor config goes under this key
 - type: gitlab
   ...  # Additional config

For a list of monitor options that are common to all monitors, see Common Configuration.

Config option Required Type Description
host yes string Host of the exporter
port yes integer Port of the exporter
username no string Basic Auth username to use on each request, if any.
password no string Basic Auth password to use on each request, if any.
useHTTPS no bool If true, the agent will connect to the exporter using HTTPS instead of plain HTTP. (default: false)
skipVerify no bool If useHTTPS is true and this option is also true, the exporter's TLS cert will not be verified. (default: false)
metricPath no string Path to the metrics endpoint on the exporter server, usually /metrics (the default). (default: /metrics)
sendAllMetrics no bool Send all the metrics that come out of the Prometheus exporter without any filtering. This option has no effect when using the prometheus exporter monitor directly since there is no built-in filtering, only when embedding it in other monitors. (default: false)

Metrics

These are the metrics available for this monitor. Metrics that are categorized as container/host (default) are in bold and italics in the list below.

  • ci_stale_builds (gauge)
  • gitlab_cache_misses_total (cumulative)
  • gitlab_database_rows (gauge)
  • gitlab_projects_with_jid (gauge)
  • gitlab_projects_without_jid (gauge)
  • gitlab_stuck_import_jobs_worker_runs_total (cumulative)
  • http_request_duration_seconds (cumulative)
  • http_request_duration_seconds_bucket (cumulative)
  • http_request_duration_seconds_count (cumulative)
  • pg_stat_table_idx_tup_fetch (gauge)
  • pg_stat_table_n_dead_tup (gauge)
  • pg_stat_table_n_tup_del (gauge)
  • pg_stat_table_n_tup_hot_upd (gauge)
  • pg_stat_table_n_tup_ins (gauge)
  • pg_stat_table_n_tup_upd (gauge)
  • pg_stat_table_seq_scan (gauge)
  • pg_stat_table_seq_tup_read (gauge)
  • process_age_seconds (gauge)
  • process_count (gauge)
  • process_memory_bytes (gauge)
  • ruby_file_descriptors (gauge)
  • ruby_gc_duration_seconds_total (cumulative)
  • ruby_gc_stat_count (gauge)
  • ruby_gc_stat_heap_allocatable_pages (gauge)
  • ruby_gc_stat_heap_allocated_pages (gauge)
  • ruby_gc_stat_heap_available_slots (gauge)
  • ruby_gc_stat_heap_eden_pages (gauge)
  • ruby_gc_stat_heap_final_slots (gauge)
  • ruby_gc_stat_heap_free_slots (gauge)
  • ruby_gc_stat_heap_live_slots (gauge)
  • ruby_gc_stat_heap_marked_slots (gauge)
  • ruby_gc_stat_heap_sorted_length (gauge)
  • ruby_gc_stat_heap_tomb_pages (gauge)
  • ruby_gc_stat_major_gc_count (gauge)
  • ruby_gc_stat_malloc_increase_bytes (gauge)
  • ruby_gc_stat_malloc_increase_bytes_limit (gauge)
  • ruby_gc_stat_minor_gc_count (gauge)
  • ruby_gc_stat_old_objects (gauge)
  • ruby_gc_stat_old_objects_limit (gauge)
  • ruby_gc_stat_oldmalloc_increase_bytes (gauge)
  • ruby_gc_stat_oldmalloc_increase_bytes_limit (gauge)
  • ruby_gc_stat_remembered_wb_unprotected_objects (gauge)
  • ruby_gc_stat_remembered_wb_unprotected_objects_limit (gauge)
  • ruby_gc_stat_total_allocated_objects (gauge)
  • ruby_gc_stat_total_allocated_pages (gauge)
  • ruby_gc_stat_total_freed_objects (gauge)
  • ruby_gc_stat_total_freed_pages (gauge)
  • ruby_memory_bytes (gauge)
  • ruby_sampler_duration_seconds_total (cumulative)
  • sidekiq_dead_jobs_total (gauge)
  • sidekiq_queue_latency (gauge)
  • sidekiq_queue_size (gauge)

Non-default metrics (version 4.7.0+)

The following information applies to the agent version 4.7.0+ that has enableBuiltInFiltering: true set on the top level of the agent config.

To emit metrics that are not default, you can add those metrics in the generic monitor-level extraMetrics config option. Metrics that are derived from specific configuration options that do not appear in the above list of metrics do not need to be added to extraMetrics.

To see a list of metrics that will be emitted you can run agent-status monitors after configuring this monitor in a running agent instance.

Legacy non-default metrics (version < 4.7.0)

The following information only applies to agent version older than 4.7.0. If you have a newer agent and have set enableBuiltInFiltering: true at the top level of your agent config, see the section above. See upgrade instructions in Old-style whitelist filtering.

If you have a reference to the whitelist.json in your agent's top-level metricsToExclude config option, and you want to emit metrics that are not in that whitelist, then you need to add an item to the top-level metricsToInclude config option to override that whitelist (see Inclusion filtering. Or you can just copy the whitelist.json, modify it, and reference that in metricsToExclude.