Releases: carlmontanari/scrapli
Releases · carlmontanari/scrapli
Moar testing, Packet Pushers, IOSXE enable fix
- Added Packet Pushers scrapli episode to the README!!
- Added NXOS and Junos mock ssh servers and created tests for open/close methods (silly tests but just ensures we send what we think we should be sending)
- Created a property
timeout_ops
on the driver class -- this property will also set thetimeout_ops
value of the channel as well, this is just to make it so users don't have to doconn.channel.timeout_ops
to set the timeout value... that was not super intuitive! - Update dev/test requirements to finally have pylama 2.6! This means that isort can be unpinned and free to update!
- Add
send_and_read
method toGenericDriver
-- this method allows you to send an input (at the current priv level) and wait for a prompt, an expected output, or a duration. - Add
eager
flag to the channelsend_input
method -- this probably should not be used by many folks, but can be used to not read until the prompt pattern is seen. In other words, this will send an input, read the input off the channel and then return. - All exceptions that are raised due to catching an internal exception should now be raising "from" the caught exception -- mostly this is to appease Pylama, but may end up being nicer on the eyes/easier to see whats going on in some scenarios.
- IOSXE now catches "Enable password:" for an escalation pattern from exec to privilege exec -- fixes #45
- The "requires open" decorator has been updated/fixed to play nice with asyncio
timeout_ops
has been converted from an int to a float to allow for more granular timeout control (the other timeouts remain as integers)- Few minor docstring fixes from copypasta issues :)
- Update black pin/re-run black
scrapli factory!
- Fixed the same
get_prompt
issue from the last release, but this time managed to actually fix it in async version! - Better handling of
read_until_input
-- stripping some characters out that may get inserted (backspace char), and compares a normalized whitespace version of the read output to the a normalized whitespace version of the input, fixes #36. - Improved system transport ssh error handling -- catch cipher/kex errors better, catch bad configuration messages.
- Now raise an exception if trying to use an invalid transport class for the base driver type -- i.e. if using asyncssh transport plugin with the "normal" sync driver class.
- Added links to the other projects in the scrapli "family" to the readme.
- Created first draft of the scrapli "factory" -- this will allow users to provide the platform name as a string to a single
Scrapli
orAsyncScrapli
class and it will automagically get the right platform driver selected and such. This is also the first support forscrapli_community
, which will allow users to contribute non "core" platforms and have them be usable in scrapli just like "normal". - Overhaul decorators for timeouts into a single class (for sync and async), prefer to use signals timeout method where possible, fall back to multiprocessing timeout where required (multiprocessing is slower/more cpu intensive so dont use it if we dont have to).
Internal improvements!
- Fixed a silly issue where
get_prompt
was setting the transport timeout to 10s causing user defined timeouts to be effectively ignored. - Improved telnet authentication handling -- previously if a return character was needed to get the auth prompts to kick into gear this could break auth.
- Added "auth_bypass" to telnet transport.
- Probably BUGFIX -- async functions were being decorated by the "normal"
operation_timeout
decorator -- created a mostly duplicated async version of the timeout decorator to wrap theAsyncChannel
methods. - Fixed a maybe regression that caused drivers to try to authenticate (via interactive methods) even if a
auth_secondary
is not set. Added tests to make sure that we raise a warning if there is no secondary password set, but try to increase privilege without authentication, and of course if there is an auth secondary set, we obviously try to auth in the normal fashion. - Started thinning down the PtyProcess stuff to simplify and and remove all unnecessary parts, as well as add typing and docstrings... not done yet, but some progress!
- Added additional asyncio example
- Added blurb about versioning in README
- Fixed a few README issues (incorrect methods/typos)
- Updated notes about auth_bypass to include telnet support
- Added
SSHNotFound
exception for system SSH/PtyProcess if ssh binary can't be found
Internal work - prep for scrapli_netconf!
Mostly a release for some internal work as well as scrapli_netconf
- Updated IOSXE base config to include netconf setup for consistency w/ scrapli_netconf
- Removed "pipes" authentication for system ssh -- this is mostly an internal change that simplifies the way that
system transport authenticates. We lose the ability to very easily read out of stderr what is going on so even if we auth with a key now we have to "confirm" that we are authenticated, but this removes a fair bit of code and unifies things as well as allows for the next line item... - Added support for
auth_private_key_passphrase
to system transport -- allows for entering ssh key passphrase to decrypt ssh keys - Added an example on how to deal with "weird" things like banners and macros -- these types of things change how the ssh channel works in that they are pseudo "interactive" -- meaning the prompt is modified/removed so scrapli can't ever "know" when a command is done inserting. It would be possible to support these types of config items more "natively" but doing so would lose some of the smarts about how scrapli enters/confirms inputs sent, so for now (and probably for forever) these will need to be configured in a "special" fashion
- Updated IOSXE for functional tests to use 16.12.03 -- this includes updates to the base config/expected configs... AFAIK there is some better netconf/restconf support in this version which may be handy for tests for scrapli-netconf
- Update channel/drivers to never decode bytes -- this now only happens in the response object; primary motivation for this is to not have to decode/re-encode in general, and in scrapli-netconf in particular
Async Support and Much more!
- Converted all priv levels to be kwargs instead of just args for setup -- simple thing but makes it more readable IMO.
- Added to the Juniper prompt pattern to include matching the RE prompt that is on the line "above" the "normal
" prompt as this was getting included in command output instead of being seen as part of the prompt by scrapli. - Convert driver privilege escalation prompts to use regex to match upper and lower case "P" in password prompt
- Fix core drivers to actually allow for users to pass
failed_when_contains
,textfsm_platform
,genie_platform
, anddefault_desired_privilege_level
- Add better exception/message for attempting to send command/config to a connection object that has not been opened
- Add testing for on open/close methods of core drivers
- Add
send_config
method to send a single configuration string -- this will automagically handle sending a full configuration, breaking it into a list of configs, sending that list withsend_configs
and then joining the responses into a singleResponse
object... or of course you can just send a single config line with it too! - Add better handling/logging for
SystemSSH
transport when key exchange cannot be negotiated - Convert the
_failed()
method ofMultiResponse
to be a property so users can check.failed
on aMultiResponse
object more intuitively/sanely - ASYNC ALL THE THINGS... basically only an internal change, but hugely modified the guts of scrapli to try to be able to best support asyncio while still having the same api for sync and async. Again, if you dont care about aysncio this probably doesnt matter at all as all the "public" stuff has not changed for sync versions of things.
- Completely overhaul unit tests -- unit tests now spin up an SSH server using asyncssh, this server is a very basic implementation of an IOSXE device. This fake IOSXE device allows for connecting/sending commands/handling log on stuff like disabling paging all in as close to the real thing as possible while being completely self contained and completely in python. Additionally since there was a lot of changes to break things out to be more granular with the async implementation the testing has evolved to support this.
- Increased all hostname patterns to match up to 63 characters -- this is the hostname length limit for Cisco IOSXE at least and should be a reasonable value that hopefully doesnt really ever need to be changed/expanded now
- Changing logging to create a logger associated with each object instance and include the name/ip of the host in the log name -- should make things a lot nicer with threads/asyncio/etc.
- Moved from tox to using nox for handling tests/linting; originally this was because of some of the unit testing failing when ran via tox (now I believe this was because there was no TERM env var set in tox), but at this point nox is quite nice so we'll stick with it!
- Added exception to be raised when users try to use system transport on Windows
- BUGFIX: Added underscores to hostname patterns for IOSXE, IOSXR, NXOS, and Junos (not valid in EOS at least in my testing)
- No more Windows testing, not worth the effort
- BUGFIX: Added functionality to merge less specific (but matching) host entry data for ssh config file hosts -- meaning that we can now merge attributes from a "*" entry into a more specific host entry (see #21)
- Add dependabot to see how we like having that friend around...
- Moved from tox -> nox; long story as to why, but very much like nox!
- fixed some docstrings and re-added darglint into the mix
- Added a "debug" workflow for connecting to runner containers for testing stuff... hopefully wont need to be used often
- Modified workflows a bit to use nox, add darglint back in, and lint more stuff on weekly build (next weekly build may fail if dev doesnt get pulled into master :))
- Updated readme to include async stuff
- Added a basic async example
- Regen docs
Improved config mode(s) handling, raise for status (responses), better.... things?
- Add underscores to EOS config prompt matching
- Actually fixed on_close methods that I could have sworn were fixed.... gremlins! (was sending prompt pattern instead of a return char... for copypasta reasons probably)
- No longer "exit" config mode... given that send_command like methods already check to ensure they are in the right priv level there is no reason to exit config mode... just leave it when you need to. Should be a minor speed up if using send_configs more than once in a row, and otherwise should be basically exactly the same.
- For NetworkDrivers we no longer set the channel prompt pattern depending on the priv level -- it is now always the combined pattern that matches all priv levels... this should make doing manual things where you change privileges and don't use scrapli's built in methods a little easier. Scrapli still checks that the current prompt matches where it thinks it should be (i.e. config mode vs privileged exec) though, so nothing should change from a user perspective.
- Improve (fix?) the abort config setup for IOSXR/Junos
- Add more helpful exception if ssh key permissions are too open
- Convert PrivilegeLevel from a namedtuple to a class with slots... better for typing and is also mutable so users can more easily update the pattern for a given privilege level if so desired
- Minor clean up stuff for all the core platforms and network driver, all internal, mostly just about organization!
- Add "configuration_exclusive" privilege level for IOSXRDriver, add "configuration_private" and "configuration_exclusive" for JunosDriver, modify some of the privilege handling to support these modes -- these can be accessed by simply passing privilege_level="configuration_exclusive" when using send_configs method
- Add support for configuration sessions for EOS/NXOS. At this time sessions need to be "registered" as a privilege level, and then are requestable like any other privilege level, and can be used when sending configs by passing the name of your session as the privilege level argument for send config methods
Add a space to EOS prompts -- it seems its very easy to add one to the prompts and scrapli did not enjoy that previously! - Give users the option to pass in their own privilege levels for network drivers, and also throw a warning if users try to pass comms_prompt_pattern when using network drivers (as this should all be handled by priv levels)
- Created MultiResponse object to use instead of a generic list for grouping multiple Response objects
Added raise_for_status methods to Response and MultiResponse -- copying the requests style method here to raise an exception if any elements were failed - BUGFIX: fixed an issue with IOSXEDriver not matching the config mode pattern for ssh pub key entries.
Logs Logs Logs! Transport Options, commands/configs from files, and did I mention better logging?
- Continued improvement around
SystemSSHTransport
connection/auth failure logging - Fix for very intermittent issue where pty fd is not available for reading on SystemSSH/Telnet connections, now we loop over the select statement checking the fd instead of failing if it isn't immediately readable
- Implement atexit function if keepalives are enabled -- this originally just lived in the ssh2 transport, but needs to be here in the base
Transport
class as the issue affected all transport types - Added
send_commands_from_file
method... does what it sounds like it does... - Added
send_configs_from_file
method (NetworkDriver
and sub-classes)... also does what it sounds like it does - Simplified privilege levels and overhauled how auth escalation/deescalation works. Its still probably a bit more complex than it should be, but its a bit more efficient and at least a little simpler/more flexible.
- Removed
comms_prompt_pattern
from Network drivers and now build this as a big pattern matching all of the priv levels for that device type. This is used only for initial connection/finding prompt then scrapli still sets the explicit prompt for the particular privilege level. - Implemented lru_cache on some places where we have repetitive tasks... probably unmeasurable difference, but in theory its a little faster now in some places
- Moved some Network driver things into the base
NetworkDriver
class to clean things up a bit. - Added an
_abort_config
method to abort configurations for IOSXR/Juniper, this is ignored on the other core platforms - BREAKING CHANGE: (minor) Removed now unneeded exception
CouldNotAcquirePrivLevel
- Made the
get_prompt_pattern
helper a little worse... should revisit to improve/make its use more clear - Fixed a screw up that had ridiculous transport timeouts -- at one point timeouts were in seconds, then milliseconds... went back to seconds, but left things setting millisecond values... fixed :D
- Added
transport_options
to baseScrape
class -- this is a dict of arguments that can be passed down to your selected transport class... for now this is very limited and is just for passing additional "open_cmd" arguments toSystemSSHTransport
. The current use case is adding args such as ciphers/kex to your ssh command so you don't need to rely on having this in an ssh config file.
Usability improvements, readability improvements, much better logging!
- Increase character count for base prompt pattern for
Scrape
,GenericDriver
, and core drivers. Example:r"^[a-z0-9.\-@()/:]{1,32}[#>$]$"
for the baseIOSXEDriver
comms_prompt_pattern
has been increased to:r"^[a-z0-9.\-@()/:]{1,48}[#>$]$"
- Improve the logging for
SystemSSHTransport
authentication - Fixed an issue where
SystemSSHTransport
auth would fail due to a login banner having the wordpassword
in the banner/text - Significantly increase the base
timeout_ops
value -- as this is not a timer that is going to cause things to block, it may as well be much higher for the default value to help prevent issues - Fixed an issue w/ ssh config file not parsing the last host entry
- Added super basic tests for most of the examples -- just making sure they don't blow up... in general that should keep them in decent shape!
- Removed cssh2 and miko transports from scrapli core. These have been migrated to their own repositories. From a users perspective nothing really should change -- you can still
pip install scrapli[paramiko]
to install the paramiko transport and the requirements (paramiko), and the actual usage (setting"transport" = "paramiko "
) remains the same! This is mostly about keeping the core of scrapli as simple as possible, and also will hopefully help to illustrate thatSystemSSH
is the development priority for scrapli. - Convert many function calls to use keyword args for better readability throughout
- Add a
comms_auto_expand
argument to theChannel
; for now this is mostly not used, but may be useful in the future. The purpose of this is to handle devices that auto expand input commands to their full canonical name. - Hopefully(?) fixed a bit of an idiosyncrasy where the
timeout_transport
was being used to decorate read/write operations for telnet/system transports. This is no longer the case, the read/write methods are NOT decorated now , instead we rely on thetimeout_ops
to time these operations out OR thetimeout_transport
being set to the timeout value (telnet) orServerAliveInterval
value for system ssh.
Better (any) Windows support, Overhauled "interactive", + More!
- BREAKING CHANGE: modify
send_interact
to just make more sense in general... now it supports 1->N "events" to interact with -- see the "handling prompts" section of README for updated example - Moved
record_response
ofResponse
object to be a private method, shouldn't really be needed publicly - Moved
authenticate
andisauthenticated
methods of ssh2/paramiko transports to private methods - Add
auth_bypass
option to ignore ssh auth for weird devices such as Cisco WLC -- currently only supported on system transport. - Bump timeout_transport up to 10 seconds after finding some issues for some users.
- Add example for "non-standard" device type (Cisco WLC) demo-ing the auth_bypass, custom on_open method, custom comms_prompt_pattern and just general non-standard device stuff.
- Add option (and make it the default) to have textfsm data returned in list of dict form with the headers being the keys and of course the row values as the values, should be much nicer on the eyes this way!
- Added terminal width settings for the core drivers to set things as wide as possible so long commands don't have issues
- Teeny tiny improvements that may make things a tick faster in Channel by using str methods instead of re
- Create a draft of public api status doc -- this should be useful on a quick glance to see if/when any public methods change, obviously as development simmers down things should be stable but inevitably stuff will change, so the goal here is to just document when methods were introduced and the last time they were changed
- Move some imports around so that scrapli works on windows (with paramiko/ssh2 transports)
Add `GenericDriver`, clean up Channel, system auth handling improvements and more!
- Add support for
parse_genie
to Response object; obviously really only for Cisco devices at this point unless there are parsers floating around out there for other platforms I don't know about! - Add an
atexit
function for the ssh2 transport which forcibly closes the connection. This fixes a bug where if a user did not manually close the connection (or use a context manager for the connection) the script would hang open until an interrupt. - Added a
GenericDriver
for those with non-core platforms. TheGenericDriver
has a really broad prompt pattern match, doesn't know about privilege levels or any other device specific stuff, but does provide thesend_command
,send_commands
,send_interact
, andget_prompt
methods just like the "core" drivers do. This should be a decent starting point for anyone working on non-core platforms! - Minor unit test improvement to cover send_commands (plural) and to cover the new
GenericDriver
- Improved auth failure handling for systemssh using pty auth (username/pass auth)
- Add "failed_when" strings to the core drivers; these are used in the response object to help indicate if the channel input failed or succeeded. For scrapli not super super helpful, but nornir_scrapli will benefit from this as well!
- Modify
NetworkDriver
to inherit fromGenericDriver
-- this allowed for some clean up of how/whereResponse
objects get created/returned from.Channel
now is much more de-coupled from whatever sits on top of it (this will be important for some netconf testing happening soon!). - Minor test de-duplication around ssh config/known hosts file gathering.
- Added a few simple examples for structured data (textfsm/genie) and updated existing examples a bit.