Skip to content

Commit

Permalink
Develop (#6)
Browse files Browse the repository at this point in the history 
readme updates
add "smoke tests" -- basic helpful scripts i use when making changes
remove unused blocking method from transport
improve auth handling for system ssh
move some things from `scrape` -> `networkdriver` where they make more sense
add transport keepalive support
regen docs
  • Loading branch information
@carlmontanari
carlmontanari authored Feb 24, 2020
1 parent 95ee04c commit dca5373
Show file tree
Hide file tree
Showing 56 changed files with 5,226 additions and 1,528 deletions.
139 changes: 90 additions & 49 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,14 +67,15 @@ The final piece of scrapli is the actual "driver" -- or the component that binds
- [Native and Platform Drivers Examples](#native-and-platform-drivers-examples)
- [Platform Regex](#platform-regex)
- [Basic Operations -- Sending and Receiving](#basic-operations----sending-and-receiving)
- [Disabling Paging](#disabling-paging)
- [Login Handlers](#login-handlers)
- [Response Objects](#response-objects)
- [Handling Prompts](#handling-prompts)
- [Driver Privilege Levels](#driver-privilege-levels)
- [Sending Configurations](#sending-configurations)
- [TextFSM/NTC-Templates Integration](#textfsmntc-templates-integration)
- [Timeouts](#timeouts)
- [Disabling Paging](#disabling-paging)
- [Login Handlers](#login-handlers)
- [Keepalives](#keepalives)
- [SSH Config Support](#ssh-config-support)
- [Telnet](#telnet)
- [FAQ](#faq)
Expand Down Expand Up @@ -271,6 +272,39 @@ with IOSXEDriver(**my_device) as conn:
```


## Disabling Paging

scrapli `Scrape` (the base driver) does not know or care about disabling paging! If you use the base `Scrape` class
you will need to handle disabling paging yourself. The `NetworkDriver` and all of its sub-classes (i.e. iosxe/junos
drivers) will however attempt to disable paging for you.

All of the drivers, have a standard/default disable paging string already configured for you, however this is of
course user configurable. In addition to passing a string to send to disable paging, scrapli supports passing a
callable. This callable should accept the drivers reference to self as the only argument. This allows for users to
create a custom function to disable paging however they like. In general it is probably a better idea to handle
this by simply passing a string, but the goal is to be flexible so the callable is supported.

```python
from scrapli.driver.core import IOSXEDriver

def iosxe_disable_paging(cls):
cls.send_commands("term length 0")

my_device = {"host": "172.18.0.11", "auth_username": "vrnetlab", "auth_password": "VR-netlab9", "session_disable_paging": iosxe_disable_paging, "auth_strict_key": False}

with IOSXEDriver(**my_device) as conn:
print(conn.get_prompt())
```


## Login Handlers

Some devices have additional prompts or banners at login. This generally causes issues for SSH screen scraping
automation. scrapli (`NetworkDriver` and all sub classes) supports -- just like disable paging -- passing a string to
send or a callable to execute after successful SSH connection but before disabling paging occurs. By default this
is an empty string which does nothing.


## Response Objects

All read operations result in a `Response` object being created. The `Response` object contains attributes for the command
Expand Down Expand Up @@ -438,46 +472,46 @@ with IOSXEDriver(**my_device) as conn:

## Timeouts

scrapli supports several timeout options. The simplest is the `timeout_socket` which controls the timeout for... setting
up the underlying socket in seconds. Value should be a positive, non-zero number, however ssh2 and paramiko
transport options support floats. `timeout_scoket` is not used when using the `telnet` transport option.
scrapli supports several timeout options:

- `timeout_socket`
- `timeout_transport`
- `timeout_ops`

`timeout_transport` sets the timeout for the actual transport session setup (system|paramiko|ssh2|telnet) -- for
example for `system` transport this is the SSH `ConnectTimeout` value.
`timeout_socket` is exactly what it sounds where possible. For the ssh2 and paramiko transports we create our own
socket and pass this to the created object (paramiko or ssh2 object). The socket is created with the timeout value
set in the `timeout_socket` attribute. For telnet and system transports we do not create a socket ourselves so this
value is used slightly differently.

For telnet, the `timeout_socket` is used as the timeout for telnet session creation. After the telnet session is
created the timeout is reset to the `timeout_transport` value (more on that in a second).

For system transport, `timeout_socket` governs the `ConnectTimeout` ssh argument -- which seems to be very similar to
socket timeout in paramiko/ssh2.

`timeout_transport` is intended to govern the timeout for the actual transport mechanism itself. For paramiko and
ssh2, this is set to the respective libraries timeout attributes. For telnet, this is set to the telnetlib timeout
value after the initial telnet session is stood up. For system transport, this value is used as the timeout value
for read and write operations (handled by operation timeout decorator).

Finally, `timeout_ops` sets a timeout value for individual operations -- or put another way, the timeout for each
send_input operation.


## Disabling Paging

scrapli `Scrape` driver attempts to send `terminal length 0` to disable paging by default. In the future this will
likely be removed and relegated to the device drivers only. For all drivers, there is a standard disable paging
string already configured for you, however this is of course user configurable. In addition to passing a string to
send to disable paging, scrapli supports passing a callable. This callable should accept the drivers reference to
self as the only argument. This allows for users to create a custom function to disable paging however they like
. This callable option is supported on the native driver as well. In general it is probably a better idea to
handle this by simply passing a string, but the goal is to be flexible so the callable is supported.

```python
from scrapli.driver.core import IOSXEDriver

def iosxe_disable_paging(cls):
cls.send_commands("term length 0")

my_device = {"host": "172.18.0.11", "auth_username": "vrnetlab", "auth_password": "VR-netlab9", "session_disable_paging": iosxe_disable_paging, "auth_strict_key": False}

with IOSXEDriver(**my_device) as conn:
print(conn.get_prompt())
```


## Login Handlers

Some devices have additional prompts or banners at login. This generally causes issues for SSH screen scraping
automation. scrapli supports -- just like disable paging -- passing a string to send or a callable to execute after
successful SSH connection but before disabling paging occurs. By default this is an empty string which does nothing.
## Keepalives

In some cases it may be desirable to have a long running connection to a device, however it is generally a bad idea
to allow for very long timeouts/exec sessions on devices. To cope with this scrapli supports sending "keepalives
". For "normal" ssh devices this could be basic SSH keepalives (with ssh2-python and system transports). As scrapli
is generally focused on networking devices, and most networking devices don't support standard keepalives, scrapli
also has the ability to send "network" keepalives.

In either case -- "standard" or "network" -- scrapli spawns a keepalive thread. This thread then sends either
standard keepalive messages or "in band" keepalive messages in the case of "network" keepalives.

"network" keepalives default ot sending u"\005" which is equivalent of sending `CTRL-E` (jump to end (right side) of
line). This is generally an innocuous command, and furthermore is never sent unless the keepalive thread can acquire
a channel lock. This should allow scrapli to keep sessions alive as long as needed.

## SSH Config Support

Expand Down Expand Up @@ -622,11 +656,17 @@ As stated, this project is 100% type checked and will remain that way. The value

## Testing

I broke testing into two main categories -- unit and functional. Unit is what you would expect -- unit testing the code.
Functional testing connects to virtual devices in order to more accurately test the code. Unit tests cover quite a
bit of the code base due to mocking the FileIO that the channel reads/writes to. This gives a pretty high level of
confidence that at least object instantiation and channel read/writes will generally work... Functional tests
against virtual devices helps reinforce that and gets coverage for the transport classes.
Testing is broken into two main categories -- unit and functional. Unit is what you would expect -- unit testing the
code. Functional testing connects to virtual devices in order to more accurately test the code. Unit tests cover
quite a bit of the code base due to mocking the FileIO that the channel reads/writes to. This gives a pretty high
level of confidence that at least object instantiation and channel read/writes will generally work... Functional
tests against virtual devices helps reinforce that and gets coverage for the transport classes.

For more ad-hoc type testing there is a `smoke` folder in the tests directory -- for "smoke tests". These are simple
scripts that don't really "test" (as in no assertions or pytest or anything), but are useful for basic testing that
things have not gotten broken while working on new features. These have been handy for spot testing during
development so rather than leave them in a private directory they are included here in case they are useful for
anyone else!

### Unit Tests

Expand Down Expand Up @@ -754,24 +794,25 @@ This section may not get updated much, but will hopefully reflect the priority i

## Todo

- Paramiko Transport ssh config support -- at least for user/port/identity-file (to match ssh2-python current support)
- Continue to bring in features from `ssh2net` -- at the moment finishing/adding ssh config support is the priority
, however it would be good to add keep alives and some other widgets from `ssh2net` in a more optimized, cleaned up
fashion here in scrapli.
- Investigate blocking modes and timeouts further. This was reasonably fleshed out in `ssh2net` I think, but less so
here... it seems to not actually be needed (the ability to flip between blocking and non-blocking reads), so it may
be worth removing this from the base Transport class entirely to simplify things.
- need to test further against linux -- looks like may need to adopt some extra cruft (or so i thought) from ssh2net
to deal w/ rstripping all lines and rejoining them so the prompt matching works better. for network devices this
hasn't seemed to be an issue but for quick testing against ubuntu it may be...
- Add keepalive support to system, paramiko and telnet (for "out of band"/"standard" keepalives) to get parity w/ ssh2
- Add tests for keepalive stuff if possible (steal from ssh2net! :D)
- Create an actual useful init in base transport class and super it from the child transport classes -- setting
things like keepalive over and over in each child class is silly.
- Maybe worth breaking readme up into multiple pages and/or use a wiki -- don't like the fact that wiki pages are not
in the repo though...
- Investigate pre-authentication handling for telnet -- support handling a prompt *before* auth happens i.e. accept
some banner/message -- does this ever happen for ssh? I don't know! If so, support dealing with that as well.
- Remove as much as possible from the vendor'd `ptyprocess` code. Type hint it, add docstrings everywhere, add tests
if possible (and remove from ignore for test coverage and darglint).
- Improve testing in general... make it more orderly/nicer, retry connections automatically if there is a failure
(failures happen from vtys getting tied up and stuff like that it seems), shoot for even better coverage!
- Remove disable paging "stuff" from the base `Scrape` class -- thats really network specific and doesnt belog in a
base ssh type class.
- Add a dummy container (like nornir maybe?) to use for functional testing -- its very likely folks won't have a
vrnetlab setup or compute to set that up... it'd be nice to have a lightweight container that can be used for basic
testing of `Scrape` and for testing auth with keys and such.
- Improve logging -- especially in the transport classes.

## Roadmap

Expand Down
12 changes: 0 additions & 12 deletions docs/scrapli/channel/channel.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -197,16 +197,11 @@ <h1 class="title">Module <code>scrapli.channel.channel</code></h1>
&#34;&#34;&#34;
prompt_pattern = get_prompt_pattern(prompt, self.comms_prompt_pattern)

# disabling session blocking means the while loop will actually iterate
# without this iteration we can never properly check for prompts
self.transport.set_blocking(False)

while True:
output += self._read_chunk()
output = re.sub(b&#34;\r&#34;, b&#34;&#34;, output)
channel_match = re.search(prompt_pattern, output)
if channel_match:
self.transport.set_blocking(True)
return output

@operation_timeout(&#34;timeout_ops&#34;)
Expand All @@ -230,7 +225,6 @@ <h1 class="title">Module <code>scrapli.channel.channel</code></h1>
LOG.debug(f&#34;Write (sending return character): {repr(self.comms_return_char)}&#34;)
while True:
output = self._read_chunk()
output.rstrip(b&#34;\\&#34;)
channel_match = re.search(prompt_pattern, output)
if channel_match:
self.transport.set_timeout()
Expand Down Expand Up @@ -601,16 +595,11 @@ <h2 id="raises">Raises</h2>
&#34;&#34;&#34;
prompt_pattern = get_prompt_pattern(prompt, self.comms_prompt_pattern)

# disabling session blocking means the while loop will actually iterate
# without this iteration we can never properly check for prompts
self.transport.set_blocking(False)

while True:
output += self._read_chunk()
output = re.sub(b&#34;\r&#34;, b&#34;&#34;, output)
channel_match = re.search(prompt_pattern, output)
if channel_match:
self.transport.set_blocking(True)
return output

@operation_timeout(&#34;timeout_ops&#34;)
Expand All @@ -634,7 +623,6 @@ <h2 id="raises">Raises</h2>
LOG.debug(f&#34;Write (sending return character): {repr(self.comms_return_char)}&#34;)
while True:
output = self._read_chunk()
output.rstrip(b&#34;\\&#34;)
channel_match = re.search(prompt_pattern, output)
if channel_match:
self.transport.set_timeout()
Expand Down
6 changes: 0 additions & 6 deletions docs/scrapli/channel/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -233,16 +233,11 @@ <h2 id="raises">Raises</h2>
&#34;&#34;&#34;
prompt_pattern = get_prompt_pattern(prompt, self.comms_prompt_pattern)

# disabling session blocking means the while loop will actually iterate
# without this iteration we can never properly check for prompts
self.transport.set_blocking(False)

while True:
output += self._read_chunk()
output = re.sub(b&#34;\r&#34;, b&#34;&#34;, output)
channel_match = re.search(prompt_pattern, output)
if channel_match:
self.transport.set_blocking(True)
return output

@operation_timeout(&#34;timeout_ops&#34;)
Expand All @@ -266,7 +261,6 @@ <h2 id="raises">Raises</h2>
LOG.debug(f&#34;Write (sending return character): {repr(self.comms_return_char)}&#34;)
while True:
output = self._read_chunk()
output.rstrip(b&#34;\\&#34;)
channel_match = re.search(prompt_pattern, output)
if channel_match:
self.transport.set_timeout()
Expand Down
17 changes: 14 additions & 3 deletions docs/scrapli/decorators.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,14 +32,16 @@ <h1 class="title">Module <code>scrapli.decorators</code></h1>
LOG = logging.getLogger(&#34;scrapli&#34;)


def operation_timeout(attribute: str) -&gt; Callable[..., Any]:
def operation_timeout(attribute: str, message: str = &#34;&#34;) -&gt; Callable[..., Any]:
&#34;&#34;&#34;
Decorate an &#34;operation&#34; -- raises exception if the operation timeout is exceeded

Wrap an operation, check class for given attribute and use that for the timeout duration.

Args:
attribute: attribute to inspect in class (to set timeout duration)
message: optional message to pass when decorating a non-standard operation such as telnet
login (as opposed to &#34;normal&#34; channel operations)

Returns:
decorate: wrapped function
Expand All @@ -51,6 +53,8 @@ <h1 class="title">Module <code>scrapli.decorators</code></h1>
import signal # noqa

def _raise_exception(signum: Any, frame: Any) -&gt; None:
if message:
raise TimeoutError(message)
raise TimeoutError

def decorate(wrapped_func: Callable[..., Any]) -&gt; Callable[..., Any]:
Expand Down Expand Up @@ -82,7 +86,7 @@ <h1 class="title">Module <code>scrapli.decorators</code></h1>
<h2 class="section-title" id="header-functions">Functions</h2>
<dl>
<dt id="scrapli.decorators.operation_timeout"><code class="name flex">
<span>def <span class="ident">operation_timeout</span></span>(<span>attribute)</span>
<span>def <span class="ident">operation_timeout</span></span>(<span>attribute, message='')</span>
</code></dt>
<dd>
<section class="desc"><p>Decorate an "operation" &ndash; raises exception if the operation timeout is exceeded</p>
Expand All @@ -91,6 +95,9 @@ <h2 id="args">Args</h2>
<dl>
<dt><strong><code>attribute</code></strong></dt>
<dd>attribute to inspect in class (to set timeout duration)</dd>
<dt><strong><code>message</code></strong></dt>
<dd>optional message to pass when decorating a non-standard operation such as telnet
login (as opposed to "normal" channel operations)</dd>
</dl>
<h2 id="returns">Returns</h2>
<dl>
Expand All @@ -106,14 +113,16 @@ <h2 id="raises">Raises</h2>
<summary>
<span>Expand source code</span>
</summary>
<pre><code class="python">def operation_timeout(attribute: str) -&gt; Callable[..., Any]:
<pre><code class="python">def operation_timeout(attribute: str, message: str = &#34;&#34;) -&gt; Callable[..., Any]:
&#34;&#34;&#34;
Decorate an &#34;operation&#34; -- raises exception if the operation timeout is exceeded

Wrap an operation, check class for given attribute and use that for the timeout duration.

Args:
attribute: attribute to inspect in class (to set timeout duration)
message: optional message to pass when decorating a non-standard operation such as telnet
login (as opposed to &#34;normal&#34; channel operations)

Returns:
decorate: wrapped function
Expand All @@ -125,6 +134,8 @@ <h2 id="raises">Raises</h2>
import signal # noqa

def _raise_exception(signum: Any, frame: Any) -&gt; None:
if message:
raise TimeoutError(message)
raise TimeoutError

def decorate(wrapped_func: Callable[..., Any]) -&gt; Callable[..., Any]:
Expand Down
Loading

0 comments on commit dca5373

Please sign in to comment.