Replace telnetlib with scrapli #279
Conversation
- Add python3-pip, git and scrapli install via pip to all node - Dockerfiles. - Replace all nodes telnetlib functions with new adapated scrapli functions. (wait_write, expect, read_until) - Move Cisco and Juniper nodes to their own vendor subdirectories, fix makefiles to ensure functionality works. - Other things, which should be outlined in the PR.
|
It might be worth making another scrapli branch and then I can change this PR to merge into that branch (instead of master). This way other contributors can submit PRs for any changes to other nodes and once all/enough nodes work, that branch can be merged into master? |
|
sound idea @kaelemc |
…nfigurations - Startup and bootstrap configurations use scrapli with IOSXEDriver - Changed variable name from 'csr' to 'cat8kv' in cat8kv install function. - Reverted change in bootstrap_spin so that console output is evaluated against empty byte-string instead of regular string.
|
IOS-XE nodes (CSR1000V, Cat8kv and Cat9kv) the Scrapli XE driver is now being used. Had some issues with it working with IOS-XE 16.x but it was an error on my end.. got the configs (both bootstrap and startup) to work nicely. |
- existing XRv node uses vmdk image only. This one will use the qcow2 image.
|
Maybe this belongs as it's own PR but I've added an xrv_qemu directory. It is the same as XRv but with some modifications to make XRv work when the user provides the qcow2 image. The existing xrv directory requires vmdk images, most users who import images from CML or other places on the web most likely will get a qcow2, this makes it easier for users so they don't have to mess around with I tried one more time to use the scrapli |
- Alter the log levels for some logs from debug->error - Move the VM startup log message and add information about qemu smp and startup RAM.
|
Not a trivial change, but I would suggest to create 1 base Dockerfile for all of vrnetlab, and then derive all platform images from that It doesn't make sense to do 100x "install scrapli" in all those separate files - that becomes unmanageable |
|
yeah we can definitely create the base image hosted on ghcr.io with base pacakages that every vr system relies on |
|
@jbemmel Excellent idea, certainly is/was unmanageable for me |
- This means for all nodes, environment variables are displayed in the logs.
…-XE 16.x compatibility)
- Useful for lower-spec systems where some operations may take time.
|
So this is a difficult one to balance, in labs of different sizes the nodes take varying times to boot (depending on system specs etc.).. I'm wondering if we should bump all the scrapli timeouts to something very large to make the connection timing out a non-issue? Currently I have XE devices on 10 minutes, I have a feeling for people with lower-spec systems might hit time-outs meaning they could never boot the device. Should we increase to something very large just to be safe? The base Scrapli |
|
I agree, let's have a long enough timeout so that you don't have to guess the time down to minutes. We should leave the timeout configurable via the env var setting that can be then put in the topology file if needed to tune it up |
|
Ok 👍, having timeouts as an env var is a good idea |
- New env var SCRAPLI_TIMEOUT added, defaults to 3600 seconds (1 hour). - It's used to enable the user to modify the operation, transport and socket timeout for the Scrapli driver used to apply the config to the CSR.
|
I had to remove NETCONF configuration on CSR as different versions of IOS-XE 16.x and 17.x have different behaviors when this command is entered in the config. In 16.x a prompt will appear asking for some confirmation about the NETCONF configuration. I figured it's best to just remove this instead of adding extra complexity to handle the prompt for that single command. Thoughts? |
|
@kaelemc added genisoimage and reuploaded |
|
@hellt Thanks, I guess one more thing is if you could just change Scrapli to install from the latest changes via: Otherwise everything is broken and won't work.. (seems to be a random EOF sent at some point from either qemu or the node itself and Scrapli will think the connection hasn't been opened correctly) If/when a new release comes out with the fixes then we can pin the version to that release :) |
|
@kaelemc |
|
thanks Carl |
|
@carlmontanari @hellt Thanks! |
- Use format string in log colour formatting - Print newlines before log messages in wait_write() to improve visual clarity of log messages.
- Hopefully this is more intuitive to users, they see they have a .qcow2 file and can use the xrv_qcow dir?
|
@hellt New image (& scrapli release) are all working great, i've moved Cisco nodes over so far and tested all of them. Install time is much better without having to re-install all the packages (even better when everything is cached locally of course 😄). I have the Dockerfile changes ready for all the other nodes but I want to test as much as I can first to see if any key packages are missing, any incompatibilities etc. In terms of Dockerfiles, since the and in the recent commits i've just pushed you'll see on XRv9k I've switched it over to pull vcpu and ram from the env vars directly instead of via the flags/argparser. I figured if we want vrnetlab to be the 'source of truth' for those things (discussed in #2285.) there's no need to have the clab end to contain logic to pass vcpu/ram via the flags... do you agree with this approach, am I clouding this PR with too much other stuff, what are your thoughts? I've also implemented it this way with the cat8kv (for some reason there was no vcpu/ram knobs implemented). |
|
you can remove the maintainer label/field altogether. It is not up to date anyways. Yes, correct, the defaults for the cpu/mem should stay within the vrnetlab node definition, and containerlab can override this via QEMU |
👍
Alright, currently I have it set to use env vars called Didn't know about the |
|
it is not only the preferred naming, but also is generically loaded in the vrnetlab common The reason there are VCPU/MEM dangling in the code base is pure legacy. We should converged on the global QEMU_* env vars as they are read once by every node, without duplication |
|
Oh perfect, got it. Will make the changes 👍 |
- Added default SMP/RAM for all nodes, this can be overriden by the env var(s) `QEMU_SMP` & `QEMU_MEMORY`
|
Will convert as many of the nodes to use the Scrapli community drivers as possible. I've added a checklist in the original PR comment of nodes I think are 'working' for sure.
Every node is using the new base vrnetlab image, will have to iron out any missing packages or incompatibilities there. All Cisco nodes have default SMP/memory and I have tested that they are infact overriden correctly using the respective env vars as pointed out. I had to make a minor change for XRv9k as without the explicit single socket config it did nothing but kernel panic. This means like other nodes, for XRv9k users can just set the |
| self.tn.write("yes\r".encode()) | ||
| self.logger.trace("Read: %s" % res.decode()) | ||
| self.print(res) | ||
| self.logger.debug("writing to serial console: %s" % cmd) | ||
| self.tn.write("{}\r".format(cmd).encode()) | ||
|
|
There was a problem hiding this comment.
2024-11-15 02:18:02,877: launch DEBUG writing to serial console: root
Traceback (most recent call last):
File "/launch.py", line 483, in <module>
vr.start()
File "/vrnetlab.py", line 734, in start
vm.work()
File "/vrnetlab.py", line 524, in work
self.bootstrap_spin()
File "/launch.py", line 123, in bootstrap_spin
self.wait_write("root", wait=None)
File "/launch.py", line 250, in wait_write
self.tn.write("{}\r".format(cmd).encode())
^^^^^^^^^^^^^
AttributeError: 'Driver' object has no attribute 'write'
There was a problem hiding this comment.
Oops sorry about that, Still have to migrate to scrapli JunOS driver :)
| self.wait_write("restconf") | ||
| self.wait_write("netconf-yang") |
There was a problem hiding this comment.
Which version of CSR did you find had a problem with these? I only have csr1000v-universalk9.17.03.04a.qcow2 where it seems to work. You could use the same version condition that was in place for the ip domain name command if the NETCONF command is problematic on older.
There was a problem hiding this comment.
Was testing with both XE17 and 16.09.x versions (which were the problematic ones).
| if int(self.version.split('.')[0]) >= 16: | ||
| self.wait_write("ip domain name example.com") | ||
| else: | ||
| self.wait_write("ip domain-name example.com") |
There was a problem hiding this comment.
This should be part of the new config string too, if older CSR are still supported.
There was a problem hiding this comment.
👍 trying to get an older version to test this out with.
| # check if we are prompted to overwrite current keys | ||
| (ridx, match, res) = self.tn.expect( | ||
| [ | ||
| b"How many bits in the modulus", | ||
| b"Do you really want to replace them", | ||
| b"^[^ ]+#", | ||
| ], | ||
| 10, | ||
| ) | ||
| if match: # got a match! | ||
| if ridx == 0: | ||
| self.wait_write("2048", None) | ||
| elif ridx == 1: # press return to get started, so we press return! | ||
| self.wait_write("no", None) |
There was a problem hiding this comment.
If you restart a container it comes up with keys already configured:
RP/0/0/CPU0:vr-xrv#2024-11-15 13:31:22,158: vrnetlab INFO writing to console: 'crypto key generate rsa'
2024-11-15 13:31:22,258: vrnetlab INFO writing to console: '2048'
2024-11-15 13:31:22,258: vrnetlab INFO waiting for '#' on console.
crypto key generate rsa
Fri Nov 15 13:31:21.519 UTC
The name for the keys will be: the_default
% You already have keys defined for the_default
Do you really want to replace them? [yes/no]: 2048
% Please answer 'yes' or 'no'.
There was a problem hiding this comment.
Thanks, could you explain a little further, because I don't think I can see this behaviour?
Thanks for your review, appreciate it a lot.
I'll have to go over each platform and migrate them all to their Scrapli (and Scrapli Community) drivers. If it's ok I'll request another review when that's all good to go 👍
|
Replaced by #297 |
This PR replaces the usage of telnetlib in vrnetlab with scrapli.
Changes
Telnetlib has been removed and replaced and now the Scrapli Base Driver class is used for base telnet functionality.
debian:bookworm-slimfrom the Amazon ECR.gitandpipadded, as well as installation command of Scrapli via the git repo.Cisco and Juniper nodes are in their own vendor-specific subdirectory.
VENDOR_SUBDIRvariable in the Makefiles of each node as well as logic inmakefile.includeso that files get copied one level further (due to vendor subdirectory). IfVENDOR_SUBDIRis1then files are copied one subdirectory level further to account for the vendor subdirectory.Logger for VR class (inherited by all nodes) now uses a logger under the
vrnetlabinstance. This is because Scrapli uses the root logging instance, and outputs all channel output as 'debug' log level.self.loggerwill still write 'debug' logs to stdout (docker logs).Coloured the logging levels so it's easier to see, looks nicer visually too.
Functions
read_until()
telnetlib had a
read_until()function which was used bywait_write. This has been re-implemented under the VR class.Args:
match_str: a string of the text to wait fortimeout: a float of how long to wait before exiting the function even if no match is found. Defaults to None.Returns: All console output that was read up until the match, or function timeout.
wait_write()
Adapted to scrapli, functionality should be analogous to the telnetlib version.
timeout(float): This has been added, and is passed toread_until, Defaults to None.expect()
telnetlib had the
expect()function which was used in thelaunch.pyentrypoint. This new adapted version in the VR class should function the same as the telnetlib one.Args:
regex_list: a list of byte-strings which are used to match via regex on the console output.timeout: a float of how long before the function should just timeout and stop waiting. Defaults to None.Returns: a tuple of:
-1if there was no match. (int)print()
The VR class now has a print function which is used to simply write to stdout and instantly flush. It's used so that the telnet console can output nicely on
docker logs.If the console output was printed via the logger, the formatting would make it difficult to interpret the output.
Args:
bytes: byte-string of what to write to stdout.Usage
Relevant changes to make a node work:
Ensure git, pip and scrapli are installed in the Dockerfile
Replace
self.tn.expect()withself.expect()Use
self.print()to print any console output, do not use logging for this. There is no need todecode()/convert the returned byte-string ofexpect()asself.print()doesn't accept strings, this would result in malformed outputs.Logging levels for any log output should be changed to be more appropriate.
INFO is Green
WARNING is Yellow
ERROR/DEBUG is Red
Other
I don't expect other nodes to 'plug and play' and work perfectly from the get go, I'm sure I have made errors or overlooked something so some work is required on the other nodes to ensure functionality.
Collaboration from others is required so we can confirm all nodes work reliably (as well as to fix some other possible issues in the way i've implemented the changes). I'm open to feedback 😊.
Confirmed working platforms
subdirectory names are in brackets. Follows alphabetical/subfolder order of the directory tree.