ObservIT

This is an aggregator for data collection that populates data in InfluxDB and uses Grafana which integrates natively with Infludb where you can draw your graphs.
We draw our standard graphs for each of the metrics we support but please feel free to personalize your own graphics.

Please Consult our wiki if you wish to contribute with new metrics https://github.com/ajcpereira/observIT.wiki.git

How it Works

The aggregator is a scheduler that collects information based in a YAML file (/opt/fjcollector/collector/conf/config.yaml), each schedule runs on it's own thread and it uses python, all the supported metrics may be found here:
https://github.com/ajcpereira/fjcollector/blob/main/functions/README.md

This is built on a virtual appliance which can be found here:
https://drive.proton.me/urls/6WE4036KX8#dHQ35JnT8Obz

In the future a script will be made available for create a fixed IP, atm the virtual appliance is based in DHCP.

You can make OS updates for security reasons using the following: "cd /home/fjcollector/fjcollector; make update_os"

Licensing

Regarding the 2 mention applications be aware of their licenses if you wish to use in your organization. We install both but they are not part of the code of the ObservIT, so you should check your policies or maybe you already have them in your organization and you just need to change the configfile to use your own:

Components	Licensing
Ubuntu	https://ubuntu.com/legal/intellectual-property-policy
make	https://www.gnu.org/licenses/gpl-3.0.html
curl	https://ubuntu.com/legal/open-source-licences
git	https://git-scm.com/about/free-and-open-source
docker engine	https://github.com/moby/moby/blob/master/LICENSE
Python	https://opensource.org/license/python-2-0
Grafana	https://grafana.com/licensing/
Influxdb	https://github.com/influxdata/influxdb/blob/master/LICENSE

Technical Section

Requirements

You need a virtualization environment, our virtual appliance, for a basic configuation up to 80 metrics needs the following requirements:

vCPU	Mem	Disk
4	8GB	30GB+15GB

*The 15GB disk can grow to accomodate your retention period, you will need the root access at this moment until we give you a script to do so with fjcollector user.

The only user available to you will be fjcollector and by default user and password are equal, please change this password as soon as you login.

Pre-Setup

Every metric that needs to make ssh, make sure you generate a private key and in the destination you use the public key in the authorized keys, then keep the private keys under:
/opt/fjcollector/collector/keys If different keys are used they may be specified in the config.yaml

Under the user fjcollector, pls do:

ssh-keygen -t rsa -b 2048

you can accept defaults with no passphrase. The files will be created under fjcollector/.ssh/id_rsa for the private key and fjcollector/.ssh/id_rsa.pub for the public key.

The private key should be specified in the config.yaml file, because it's running under a container, we recommend to put under /opt/fjcollector/collector/keys.

The public key must be added to the user used for the login (we assume fjcollector) so, /home/fjcollector/.ssh/authorized_keys owned by fjcollector and with 600 permissions, and the .ssh directory should be 700 with owner fjcollector.

If you are using a CS8000 the user must be created using:

ecs-add-user --user=fjcollector --role=csobserve

Keep in mind that some metrics require sudo permissions, set them accordingly

We recommend to you have a look in the functions Readme:
https://github.com/ajcpereira/fjcollector/blob/main/functions/README.md

We recommend you use relative paths:
host_keys: keys/id_rsa
logfile: logs/fjcollector.log

edit /opt/fjcollector/collector/config/config.yaml

config.yaml

Configuration manual can be found in https://github.com/ajcpereira/fjcollector/blob/main/config/README.md

systems:
  - name: MYCS0100
    resources_types: eternus_cs8000
    config:
      parameters:
        user: fjcollector
        host_keys: keys/id_rsa
        poll: 1
      metrics:
        - name: fs_io
        - name: fs
        - name: cpu
        - name: mem
        - name: drives
      ips:
        - ip:  192.168.18.46
          alias: linux14
        - ip:  192.168.18.46
          alias: linux02
global_parameters:
  repository: influxdb
  repository_port: 8086
  repository_protocol: tcp
  repository_api_key: AUTOGENERATED_DURING_SETUP
  loglevel: DEBUG
  logfile: logs/fjcollector.log
  auto_fungraph: yes
  grafana_api_key: AUTOGENERATED_DURING_SETUP
  grafana_server: grafana

Metrics

Consult the README inside the folder functions:
https://github.com/ajcpereira/fjcollector/blob/main/functions/README.md

Metrics Retention

By default the bucket that will retain the information in Influxdb is of 2 years, manual change must be done if you wish to have a different retention. At the moment this must be done with user admin in the future we expect to give this through scripting or other way.

Architecture