Various Fixes and Updates for newer NetBox Versions #56

nahun · 2021-05-28T00:50:45Z

This PR is a little big (sorry about that), but I'll summarize the changes.

Uses OPTIONS to lookup choices. Requires Add Options Method python-netboxapi#9 to be merged.
Ignores IPv6 link local address which is anything in fe80::/10.
Will only send a PUT request for interfaces and devices when something is going to change.
Get all interfaces on a device first and create a map to lookup later. This allows just one bigger request vs many small ones.
Does not change the interface type to Other of an existing interface unless overwrite is specified.
Fixes IP assignment to the newer model with later NetBox versions (assigned_object_id and assigned_object_type).
QSFP interface type fixes.
Fixes VLAN tagged list processing.
Adds Loopback interface as a Virtual type.
Adds detailed LLDP neighbors method to Cisco IOS.

I haven't really looked at the testing. Mostly tested on newer Cisco Cat9Ks.

Fixes #39

…ice unless something changed.

…to Cisco IOS.

nahun · 2021-05-28T01:10:09Z

I'm not sure if my QSFP changes are correct, but it seems good for Cisco IOS at least. I know the testing is failing for Cisco Nexus though. Probably need to add some testing for QSFP28 100G interfaces.

aruhier · 2021-05-28T01:11:48Z

Hey,
Thanks a lot for this PR! I'm not working at Scaleway anymore (that's why I transferred them the ownership of these 2 repos), but I can still do some review.

It'll be however complicated for me to test it, but on the other hand, this tool doesn't work with the recent versions of Netbox, and that for a while, so I won't be picky. If anyone at @scaleway wants to test it, it would help, but otherwise I'll merge if things look ok.

Anyway, I already had a quick look, I'm going to do a review tomorrow. Overall it looks good!
There's one thing I thought about the Options. Maybe it can be integrated into the mappers a bit better (in netbox-api) so they handle the caching, and you wouldn't have to bother caring about it here. But your current solution is good enough and it's something that can be added later if there's some interests on it.

i4networks · 2021-05-28T12:52:50Z

Hi,

ERROR: netbox_importer: Error when polling device xx-xx--xx: Unexpected format in the OPTIONS response at dcim/

How can we help out?

Best,

I4Networks

aruhier · 2021-05-28T13:04:13Z

@i4networks: thanks for the testing. Can you run it again in debug mode so you could paste the OPTIONS request?

i4networks · 2021-05-28T14:11:05Z

@aruhier what parameter do u use for debug?

aruhier · 2021-05-28T14:18:58Z

--verbose debug should do the trick. It'll be very verbose but normally requests should print you every http requests it's doing.

i4networks · 2021-05-28T15:42:06Z

@aruhier
......
switch1#
DEBUG: netbox_importer: Interface VRF_0_vlan1015 has IP but was not listed
DEBUG: urllib3.connectionpool: Starting new HTTP connection (1): localhost:8001
DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/devices/?name=switch1&limit=50 HTTP/1.1" 200 1772
DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost
DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/interfaces/?device_id=31&limit=50 HTTP/1.1" 200 43113
DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost
DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/interfaces/?device_id=31&limit=50&offset=50 HTTP/1.1" 200 42974
DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost
DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/interfaces/?device_id=31&limit=50&offset=100 HTTP/1.1" 200 42927
DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost
DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/interfaces/?device_id=31&limit=50&offset=150 HTTP/1.1" 200 41119
DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost
DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/interfaces/?device_id=31&limit=50&offset=200 HTTP/1.1" 200 40691
DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost
DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/interfaces/?device_id=31&limit=50&offset=250 HTTP/1.1" 200 40749
DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost
DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/interfaces/?device_id=31&limit=50&offset=300 HTTP/1.1" 200 8231
DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost
DEBUG: urllib3.connectionpool: http://localhost:8001 "OPTIONS /api/dcim/ HTTP/1.1" 200 181
DEBUG: netmiko: write_channel: b'\n'
DEBUG: netmiko: read_channel:
i4n-cr1-ahm-dp#
DEBUG: netmiko: read_channel:
DEBUG: netmiko: read_channel:
DEBUG: netmiko: write_channel: b'exit\n'
ERROR: netbox_importer: Error when polling device switch1: Unexpected format in the OPTIONS response at dcim/
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:57<00:00, 57.62s/it]
[root@nms netbox-netprod-importer]#

aruhier · 2021-05-28T15:44:24Z

I don't get why it tries to do a query on api/dcim. @nahun can you investigate this issue? I will also try to do the same later on my side.

nahun · 2021-06-01T03:09:12Z

Interesting, I can't find any reason that would be happening.

@i4networks, can you post more information like your config.yml and devices.yml?

I thought maybe it was due to not using the latest changes in netboxapi, but it would throw a straight python exception about no options() method in the mapper if that was the case.

The only call to options() is in the _NetboxPusher.search_value_in_choices() method and it seems fine. Doesn't seem possible that it wouldn't use the model correctly for the route in netboxapi.

aruhier · 2021-06-04T13:05:41Z

@i4networks: Any news about the bug you encountered?

i4networks · 2021-06-05T10:19:34Z

Hi @aruhier ,

Sorry for the late response.
I tried both Cisco and Juniper MX with a fairly simple yaml.

test.yml
netboxhostname:
driver: ios

config.yml
cat config.yml
########################

Global options

########################

Be more verbose

verbose: None

Disable ssl warnings in urllib3

disable_ssl_warnings: False

################

Netbox

################

netbox:

Netbox API URL

url: "http://localhost:8001/api"

username: "user"

password: "password"

or to use a token instead

token: "letsnotleakthetokenongithub"

##########################

Interconnections

##########################

On some devices, LLDP will expose the host FQDN. If devices are stored on

Netbox only by their hostname, the interconnection process will not be able

to find them. Fill this list to strip the domain name from exposed names.

remove_domains:

"foo.tld"
"bar.tld"

vim: set ts=2 sw=2:

netbox version v2.11.4
Python 2.7.5

nahun · 2021-06-08T23:44:49Z

@i4networks, it happens on both Cisco and Juniper devices if you try one at a time? I don't have a Juniper device to test with unfortunately.

i4networks · 2021-06-11T11:40:53Z

@nahun i tested with only one device at the time in the test.yml

nahun · 2021-06-23T21:45:23Z

I can't reproduce that error. You'll need to be very specific on how you're able to reproduce.

Versions of everything like Python & Python packages (use pip freeze). Confirm using the latest netboxapi from the master branch.

Config and device files

Model of each device and what version they're running.

Provide anything else that might be useful.

aruhier · 2021-06-23T21:50:40Z

Lets not block this PR for too long, now this project is not working anymore for a lot of people anyway. I'm doing the review right now.

aruhier

Globally, in looks good, thanks a lot!

aruhier · 2021-05-28T01:24:24Z

netbox_netprod_importer/push.py

        for ip in ip_addresses:
+            if ip_interface(ip).ip in link_local_v6:


It should be in my opinion an option. I remember looking into netbox to know which switch had which LLA.

aruhier · 2021-06-23T21:56:34Z

netbox_netprod_importer/vendors/cisco/ios.py

+                    if line.startswith('System Name'):
+                        hostname = value
+                    elif line.startswith('Local Intf'):
+                        local_port = value
+                    elif line.startswith('Port id'):
+                        port = value
+                    elif line.startswith('Chassis id'):
+                        chassis_id = value
+
+                neighbours.append(
+                    {
+                        "local_port": local_port,
+                        "hostname": hostname,
+                        "port": port,
+                        "chassis_id": chassis_id
+                    }
+                )


I would prefer having the neighbours attributes (local_port, hostname, port, chassis_id) into a dictionary that you then append to the neighbours list, instead of different variables that are declared in the scope of the for (it's apparently working in python, but it's unintuitive).

Also, instead of appending all the neighbours in a list and yielding at the end, you can yield each neighbour at line 131.

VictorPavlushin · 2021-09-20T12:06:25Z

Hi,

ERROR: netbox_importer: Error when polling device xx-xx--xx: Unexpected format in the OPTIONS response at dcim/

How can we help out?

Best,

I4Networks

This is due to the fact that the latest changes to the library https://github.com/scaleway/python-netboxapi are not published in the pip repository. Delete the system one and install it from github.

i4networks · 2021-11-27T17:43:37Z

Hi,
ERROR: netbox_importer: Error when polling device xx-xx--xx: Unexpected format in the OPTIONS response at dcim/
How can we help out?
Best,
I4Networks

This is due to the fact that the latest changes to the library https://github.com/scaleway/python-netboxapi are not published in the pip repository. Delete the system one and install it from github.

pip uninstall netboxapi
python -m pip install git+https://github.com/scaleway/python-netboxapi.git

fixed the issue.

brianpr · 2021-12-03T19:22:54Z

I have the same problem

pip install git+https://github.com/scaleway/python-netboxapi.git

it didn't work for me, now it tells me:

ERROR: netbox_importer: Error when polling device xxx.xxx.xxx: 404 Client Error: Not Found for url: https://test-domain.com/api/dcim/_choices/?limit=50 -

any ideas?

VictorPavlushin · 2021-12-04T08:02:34Z

I have the same problem

pip install git+https://github.com/scaleway/python-netboxapi.git

it didn't work for me, now it tells me:

ERROR: netbox_importer: Error when polling device xxx.xxx.xxx: 404 Client Error: Not Found for url: https://test-domain.com/api/dcim/_choices/?limit=50 -

any ideas?

use this branch with commits:
https://github.com/nahun/netbox-netprod-importer/pull/1

appleoddity · 2023-09-01T13:42:15Z

So, what happened here? Why did these fixes not get merged? This project is broken and dead. Worthless to anyone.

@nahun How do I pull this fixed code base, it looks like you deleted it?

nahun added 4 commits May 27, 2021 17:18

qsfp additions and fixes

d58b905

Various fixes and updates. Now doesn't send PUT for interface and dev…

f04e732

…ice unless something changed.

Fix processing tagged VLANs in a list

5ba93ae

Add Loopback to virtual interface. Fix vlan typo. Add LLDP neighbors …

6c6f284

…to Cisco IOS.

nahun marked this pull request as ready for review May 28, 2021 01:07

aruhier suggested changes Jun 23, 2021

View reviewed changes

nahun closed this by deleting the head repository Jan 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Various Fixes and Updates for newer NetBox Versions #56

Various Fixes and Updates for newer NetBox Versions #56

nahun commented May 28, 2021 •

edited

Loading

nahun commented May 28, 2021

aruhier commented May 28, 2021

i4networks commented May 28, 2021

aruhier commented May 28, 2021

i4networks commented May 28, 2021

aruhier commented May 28, 2021

i4networks commented May 28, 2021

aruhier commented May 28, 2021

nahun commented Jun 1, 2021

aruhier commented Jun 4, 2021

i4networks commented Jun 5, 2021

nahun commented Jun 8, 2021

i4networks commented Jun 11, 2021

nahun commented Jun 23, 2021

aruhier commented Jun 23, 2021

aruhier left a comment

aruhier May 28, 2021

aruhier Jun 23, 2021

aruhier Jun 23, 2021

VictorPavlushin commented Sep 20, 2021

i4networks commented Nov 27, 2021

brianpr commented Dec 3, 2021

VictorPavlushin commented Dec 4, 2021 •

edited

Loading

appleoddity commented Sep 1, 2023

		for ip in ip_addresses:
		if ip_interface(ip).ip in link_local_v6:

Various Fixes and Updates for newer NetBox Versions #56

Various Fixes and Updates for newer NetBox Versions #56

Conversation

nahun commented May 28, 2021 • edited Loading

nahun commented May 28, 2021

aruhier commented May 28, 2021

i4networks commented May 28, 2021

aruhier commented May 28, 2021

i4networks commented May 28, 2021

aruhier commented May 28, 2021

i4networks commented May 28, 2021

aruhier commented May 28, 2021

nahun commented Jun 1, 2021

aruhier commented Jun 4, 2021

i4networks commented Jun 5, 2021

Global options

Be more verbose

Disable ssl warnings in urllib3

Netbox

Netbox API URL

username: "user"

password: "password"

or to use a token instead

Interconnections

On some devices, LLDP will expose the host FQDN. If devices are stored on

Netbox only by their hostname, the interconnection process will not be able

to find them. Fill this list to strip the domain name from exposed names.

vim: set ts=2 sw=2:

nahun commented Jun 8, 2021

i4networks commented Jun 11, 2021

nahun commented Jun 23, 2021

aruhier commented Jun 23, 2021

aruhier left a comment

Choose a reason for hiding this comment

aruhier May 28, 2021

Choose a reason for hiding this comment

aruhier Jun 23, 2021

Choose a reason for hiding this comment

aruhier Jun 23, 2021

Choose a reason for hiding this comment

VictorPavlushin commented Sep 20, 2021

i4networks commented Nov 27, 2021

brianpr commented Dec 3, 2021

VictorPavlushin commented Dec 4, 2021 • edited Loading

appleoddity commented Sep 1, 2023

nahun commented May 28, 2021 •

edited

Loading

VictorPavlushin commented Dec 4, 2021 •

edited

Loading