Data Integrity Errors on Import

[JamesonRGrieve]

I've used the indicated https://github.com/netbox-community/Device-Type-Library-Import to import the data and was presented with several integrity errors culminating in an unhandled exception. Reading through them it sounds as if they are problems in the data of this repository not the import script (but happy to migrate the issue if I'm incorrect).

Let me know if I'm doing something incorrectly to cause these, but 99.9% of the data up to the point of the unhandled exception imported fine.

Netbox version 4.4 found. Using new filters.
259 Vendors Found
4941 Device-Types Found
Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'PULTIMO', 'slug': 'pultimo'}"]} creating device type: PULTIMO 701BA8PP
Could not find Rear Port for Front Port: IN RG45 NET SURGE PROTECTION - 8p8c - 529
Error '[{"__all__":["Rear port (COM) must belong to the same device type","Front port template with this Rear port and Rear port position already exists."]}]' creating Front Port
Error {"manufacturer":["Related object not found using the provided attributes: {'name': 'HW-GROUP', 'slug': 'hw-group'}"]} creating device type: HW-GROUP HWG-STE
Could not find Rear Port for Front Port: IN - 8p8c - 665
Error '[{"__all__":["Rear port (COM) must belong to the same device type","Front port template with this Rear port and Rear port position already exists."]}]' creating Front Port
Error {"__all__":["Must specify a unit when setting a weight"]} creating device type: Cisco Catalyst 2960-12PC-L
Error '[{},{},{"description":["Ensure this field has no more than 200 characters."]},{"description":["Ensure this field has no more than 200 characters."]}]' creating Module Bay
Error '[{},{"description":["Ensure this field has no more than 200 characters."]},{"description":["Ensure this field has no more than 200 characters."]},{"description":["Ensure this field has no more than 200 characters."]},{},{}]' creating Module Bay
Error '[{},{},{"description":["Ensure this field has no more than 200 characters."]},{"description":["Ensure this field has no more than 200 characters."]},{"description":["Ensure this field has no more than 200 characters."]},{"description":["Ensure this field has no more than 200 characters."]},{},{}]' creating Module Bay
Error '[{},{},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]}]' creating Interface
Error '[{},{},{},{},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]}]' creating Interface
Error '[{},{},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]}]' creating Interface
Error '[{},{},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]}]' creating Interface
Error {"u_height":["Child device types must be 0U."]} creating device type: Fortinet Fortinet Interface Module
Error {"u_height":["Child device types must be 0U."]} creating device type: Fortinet FortiGate Module 7620F
Modules Enabled. Creating Modules...
83 Module Vendors Found
1611 Module-Types Found
Error creating manufacturers
Error '{"manufacturer":["Related object not found using the provided attributes: {'name': 'Qlogic', 'slug': 'qlogic'}"]}' creating module type: {'manufacturer': {'name': 'Qlogic', 'slug': 'qlogic'}, 'model': 'UCSC-PCIE-QD25GF', 'part_number': 'UCSC-PCIE-QD25GF', 'comments': 'Qlogic QL41212H dual-port 25G NIC', 'interfaces': [{'name': 'Ethernet{module}/1', 'type': '25gbase-x-sfp28'}, {'name': 'Ethernet{module}/2', 'type': '25gbase-x-sfp28'}], 'src': '/root/device-type-library/repo/module-types/Qlogic/UCSC-PCIE-QD25GF.yml'}
Traceback (most recent call last):
  File "/root/device-type-library/./nb-dt-import.py", line 54, in <module>
    main()
    ~~~~^^
  File "/root/device-type-library/./nb-dt-import.py", line 35, in main
    netbox.create_module_types(module_types)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/root/device-type-library/netbox_api.py", line 174, in create_module_types
    self.device_types.create_module_interfaces(curr_mt["interfaces"], module_type_res.id)
                                                                      ^^^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'module_type_res' where it is not associated with a value

[NeverBehave]

I am currently working on some import issue, e.g. #3754 #3764

Just a heads up that the README of the project saying Tested working with 2.9.4, 2.10.4, which is pretty outdated. Just be careful when you proceed.

[NeverBehave]

For importing device type, it seems the following error occurs on my side.

 [2981] cisco-ws-c2960c-12pc-l
      File: device-types/Cisco/WS-C2960C-12PC-L.yaml
      __all__: Must specify a unit when setting a weight

  [4453] fortinet-fim-7941f
      File: device-types/Fortinet/FIM-7941F.yaml
      u_height: Child device types must be 0U.

  [4465] fortinet-fpm-7620f
      File: device-types/Fortinet/FPM-7620F.yaml
      u_height: Child device types must be 0U.

[NeverBehave]

Another two duplicate entry on device-type:

Constraints: (manufacture, model) and (manufacture, slug) must be unique within same type.

Errors:
  ✗ Duplicate device-types for manufacturer='Apple', model='MacPro7,1' found in files: build/device-types/Apple/A1991.yaml, build/device-types/Apple/A2304.yaml
  ✗ Duplicate device-types for manufacturer='Juniper', model='MX304' found in files: build/device-types/Juniper/MX304-BASE.yaml, build/device-types/Juniper/MX304.yaml

[harryajc]

I am currently working on some import issue, e.g. #3754 #3764

Just a heads up that the README of the project saying Tested working with 2.9.4, 2.10.4, which is pretty outdated. Just be careful when you proceed.

Yeah the script hasn't had much development, although there was a change earlier this year to add support for 4.1.0+ as something changed which broke the old script

[NeverBehave]

@harryajc What's the standard for module-images? it seems to be pretty free styled and I could not locate documents around them.

[harryajc]

@harryajc What's the standard for module-images? it seems to be pretty free styled and I could not locate documents around them.

The only standard is that the images shouldn't have any white or empty space around the device and should be a standard jpg or png, apart from that there is no standards around pixel size or file sizes

[NeverBehave]

Good to know. I was actually trying to ask about the filename format. While the device front/rear image seems to be defined, module one doesn't appear to be.

[NeverBehave]

FYI I have tried to do a full import on commit: c24937c

And it seems all data fits in Netbox version v4.4.5-Docker-3.4.1, all available types(device, rack and module) with their corresponded interfaces (including bridge, a special one).

Exceptions would be the module images, which I have opened a different issue to address it.

[JamesonRGrieve]

Just for reference, I'm running Community 4.4.6 based on the Proxmox Community Scripts release.

[JamesonRGrieve]

Made it further, but different errors.

Package devicetype-library is already installed, updating /root/device-type-library/repo
Netbox version 4.4 found. Using new filters.
260 Vendors Found
4949 Device-Types Found
Could not find Rear Port for Front Port: IN - 8p8c - 665
Error '[{"__all__":["Rear port (COM) must belong to the same device type","Front port template with this Rear port and Rear port position already exists."]}]' creating Front Port
Error '[{},{},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]}]' creating Interface
Error '[{},{},{},{},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]}]' creating Interface
Error '[{},{},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]}]' creating Interface
Error '[{},{},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]},{},{"bridge":["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: backplane0"]}]' creating Interface
Modules Enabled. Creating Modules...
83 Module Vendors Found
1615 Module-Types Found
Error creating manufacturers
Error '{"manufacturer":["Related object not found using the provided attributes: {'name': 'Qlogic', 'slug': 'qlogic'}"]}' creating module type: {'manufacturer': {'name': 'Qlogic', 'slug': 'qlogic'}, 'model': 'UCSC-PCIE-QD25GF', 'part_number': 'UCSC-PCIE-QD25GF', 'comments': 'Qlogic QL41212H dual-port 25G NIC', 'interfaces': [{'name': 'Ethernet{module}/1', 'type': '25gbase-x-sfp28'}, {'name': 'Ethernet{module}/2', 'type': '25gbase-x-sfp28'}], 'src': '/root/device-type-library/repo/module-types/Qlogic/UCSC-PCIE-QD25GF.yml'}
Traceback (most recent call last):
  File "/root/device-type-library/./nb-dt-import.py", line 54, in <module>
    main()
    ~~~~^^
  File "/root/device-type-library/./nb-dt-import.py", line 35, in main
    netbox.create_module_types(module_types)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/root/device-type-library/netbox_api.py", line 174, in create_module_types
    self.device_types.create_module_interfaces(curr_mt["interfaces"], module_type_res.id)
                                                                      ^^^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'module_type_res' where it is not associated with a value

[NeverBehave]

I am not using the Device-Type-Library-Import script FYI

We build an internal tooling for data validation and import/sync into Netbox, and using this repo and a source of data.
Unfortunately I won't be able to share it right now. However I am pretty sure the error here is the import script didn't associate some relationships for the bridge interfaces/manufacture, not the data source conflicts.