Extraction and polling (#72)

* xml and textfsm support * requests and scrpli_netconf support * fill poller main creds via js * bugfixes * js select error handling * serialization fixes * serializer parameters * dynamic rendering of serializer template fields * bugfixes * mkarr, subform validation * http poller fixes * introduce sync_in_migration * subform api validation * tests * docs * mknum * tests for jq
amyasnikov · Mar 6, 2024 · 63c8171 · 63c8171
1 parent 19e333d
commit 63c8171
Show file tree

Hide file tree

Showing 47 changed files with 1,103 additions and 228 deletions.
diff --git a/README.md b/README.md
@@ -25,21 +25,21 @@ Validity is the [NetBox](https://netbox.dev) plugin to write "auto tests" for yo
 3. Write compliance test as a Python expression, e.g.<br/>
 `device.config["ntp-servers"] == ["1.2.3.4", "5.6.7.8"]`<br/>
 or<br/>
-`not device.state.show_stp['enabled']`
+`'10.0.0.0/8' in {entry['prefix'] for entry in device.state.show_route}`
 
 4. Apply created test to specific devices and get the results per device (passed or failed).
 
 
 ## Why?
 Validity helps you to concentrate on what really matters - defining the criteria of healthy and valid network and following these criteria.
 
-Validity completely separates compliance test code from all other things like data collection, serialization and storage. This one encourages you to write short, clean and understandable compliance tests together with the mandatory description.
+Validity completely separates compliance test code from all the other things like data collection, parsing and storage. It encourages you to write short, clean and understandable compliance tests together with the mandatory description.
 
 
 ## Key Features
-* Truly vendor-agnostic. You can easily integrate any vendor config format using [TTP](https://github.com/dmulyalin/ttp)
+* Truly vendor-agnostic. You can easily integrate any vendor config format using [TTP](https://github.com/dmulyalin/ttp) or a bunch of other [serialization options](https://validity.readthedocs.io/en/latest/entities/serializers/)
 * Writing compliance tests using Python expressions and [JQ](https://stedolan.github.io/jq/manual/)
-* Direct polling of the devices via SSH or Telnet. More than 100 different platforms are available through [netmiko](https://github.com/ktbyers/netmiko) library.
+* Gathering configuration or state info directly from the devices via **SSH**, **Telnet**, **Netconf** or **REST API**.
 * Flexible selector system to apply the tests only to a specific subset of devices
 * Concept of **dynamic pairs**. With dynamic pair you can compare 2 different devices between each other (e.g. compare the configuration of 2 MC-LAG members).
 * **Test result explanation**. When some test fails, you can get the **explanation** of the calculation process step by step. It helps to identify the cause of the failure.

diff --git a/docs/entities/commands.md b/docs/entities/commands.md
@@ -18,13 +18,20 @@ Label value is used to access serialized command output in Compliance Test.
 E.g. test expression `device.state.sh_version` implies there is a Command with label `sh_version`.
 
 #### Type
-Type of the command. It defines other parameters that must be filled for this command. Right now only **CLI** type is available, but more types will appear in next releases.
+Type of the command. It defines other parameters that must be filled for this command. Command of one specific type can be bound only to the Poller with the matching Connection Type.
+
+| Command Type | Matching Poller Type |
+|--------------|----------------------|
+| CLI          | netmiko              |
+| NETCONF      | scrapli_netconf      |
+| JSON_API     | requests             |
+
 
 #### Retrieves configuration
 Defines either this command is supposed to retrieve device configuration or no. For each poller there can be **at most one** command which retrieves configuration.
 
 !!! note
-    Serialized state for command which retrieves configuration is always available through "config" key. Let's suppose we have a command with label `show_run` which has `retrieves_config=True`, then inside Compliance Test the serialized output of this command will be available through both `device.state.show_run` and `device.state.config`.
+    Serialized state for command which retrieves configuration is always available through "config" key. Let's suppose we have a command with label `show_run` which has `retrieves_config=True`, then inside Compliance Test the serialized output of this command will be available through both `device.state.show_run` and `device.config`.
 
 #### Serializer
 This field defines [Serializer](serializers.md) for Command output.
@@ -34,4 +41,42 @@ This block contains type-specific parameters.
 
 ### Type:CLI
 #### CLI Command
-This field contains text string which is going to be sent to device when polling occurs. 
+This field must contain text string which is going to be sent to device when polling occurs.
+
+### Type:NETCONF
+#### RPC
+This field must contain an XML RPC which is going to be sent to device via Netconf.
+
+Example:
+
+```
+<get-config>
+  <source>
+    <running/>
+  </source>
+</get-config>
+```
+
+### TYPE: JSON API
+This option supports both REST API and various JSON-based APIs which do not follow REST
+
+#### Method
+HTTP method used for polling. `Get` by default.
+
+#### URL Path
+Path part of the URL. Will be appended (via Jinja2 expression) to hostname part defined in Poller credentials
+Example: `/rest/ip/address/`
+
+#### Body
+
+Request body is optional. It may be useful for various JSON-based APIs which do not follow REST and may use POST or other queries for information retrieving.
+You can use Jinja2 expressions as values in body dictionary. Available context variables are `device` and `command`.
+Example:
+```json
+{
+  "data": {
+    "commamnd": "get-config",
+    "device": "{{ device.name }}"
+  }
+}
+```
diff --git a/docs/entities/pollers.md b/docs/entities/pollers.md
@@ -11,27 +11,45 @@ Name of the Poller. Must be unique.
 Set of [Commands](commands.md) which are going to be sent to devices.
 
 #### Connection Type
-This field defines the polling backend which will be used for this Poller.
+This field defines the library used to interact with devices (polling backend). At the moment there are 3 options available:
+
+* [netmiko](https://github.com/ktbyers/netmiko) for polling via SSH or Telnet
+* [scrapli_netconf](https://github.com/scrapli/scrapli_netconf) for polling via Netconf
+* [requests](https://github.com/psf/requests) for polling via REST or JSON API
 
 #### Public credentials, Private credentials
 
-These two fields must contain any credentials which will be passed to polling backend on its instantiation
-All the values of private credentials will be encrypted after submitting.
+These two fields must contain any credentials which will be passed to polling backend.
+All the values of private credentials will be encrypted after submitting. These values are stored encrypted in the DB, decryption occurs only to pass the value to Polling backend.
 
-!!! info
-    Let's consider an example to better understand how it works.
-    Let's suppose we have a Poller with:
-    * connection type: `netmiko`
-    * public credentials: `{"device_type": "cisco_ios", "username": "admin"}`
-    * private credentials: `{"password": "admin123"}`
-    When polling occurs, public and private credentials are merged (device primary IP will also be added there) and passed to **netmiko.ConnectHandler**
-    So, it means that in case of public/private credentials for **netmiko** you can define any keyword arguments [ConnectHandler](https://github.com/ktbyers/netmiko#getting-started-1) is ready to accept.
+!!! warning
+    DJANGO_SECRET_KEY is used as an encryption key. Consider it in case of data migrations.
 
 
-Private credentials are stored encrypted in the DB, decryption occurs only to pass the value to Polling backend.
+## Credentials and polling backend
 
-!!! warning
-    DJANGO_SECRET_KEY is used as an encryption key. Consider it in case of data migrations.
+Let's consider an example to better understand how credentials are passed to selected Conenction Type.
+Let's suppose we have a Poller with:
+* connection type: `netmiko`
+* public credentials: `{"device_type": "cisco_ios", "username": "admin"}`
+* private credentials: `{"password": "admin123"}`
+When polling occurs, public and private credentials are merged (device primary IP will also be added there) and passed to **netmiko.ConnectHandler**
+So, it means that in case of public/private credentials for **netmiko** you can define any keyword arguments [ConnectHandler](https://github.com/ktbyers/netmiko#getting-started-1) is ready to accept.
+
+
+The table below points out the entities which accept merged credentials from poller:
+
+| Connection Type | Entity that accepts credentials      |
+|-----------------|--------------------------------------|
+| netmiko         | netmiko.ConnectHandler               |
+| scrapli_netconf | scrapli_netconf.driver.NetconfDriver |
+| requests        | requests.request                     |
+
+For **requests** case there is some extra logic here:
+1. `url` credential accepts Jinja2 expression, `device` and `command` are available as context variables. Default URL value:<br/>
+`https://{{device.primary_ip.address.ip}}/{{command.parameters.url_path.lstrip('/')}}`
+2. Pass something like `{"auth": ["admin_user", "admin_password"]}` to use basic auth.
+3. SSL verification is turned off by default. You can turn it back on by specifying `{"verify": true}`
 
 
 ## Binding Pollers to Devices