[REQUEST] Standard "Open ZigBee Coordinator Backup Format" support for Zigbee network backups and restoring from and to ConBee/RaspBee (deCONZ based adapters)?

[Hedda]

@manup @SwoopX @Haerteleric @ChrisHae Posting this a public request asking you as dresden-elektronik's deCONZ/Phoscon developers to consider implementing support for Zigbee network backup/restore using the new "Open ZigBee Coordinator Backup Format" open standard and open file format in and by the deCONZ/Phoscon Zigbee gateway implementations.

https://github.com/zigpy/open-coordinator-backup

This new open standard and open file format for Zigbee network backup is a common unified file format for storing and restoring Zigbee network backups. It was initially invented as a shared project by developers from Zigbee2MQTT and zigpy (library used in Home Assistant's ZHA and other implementations), and have more recently been adopted by even more Zigbee gateway implementations.

https://github.com/zigpy/open-coordinator-backup/blob/main/README.md

By sharing the same open standard and open file format for Zigbee network backup in different Zigbee implementations/applications it makes it possible for end-users to upgrade, downgrade, and/or migrate between different hardware dongles and even between different Zigbee gateway implementations/applications without having to re-pair their Zigbee devices that they previously joined/paired to their Zigbee network.

While the deCONZ/Phoscon is made by a commercial company and already haing its own proprietary backup format for Zigbee networl backups, I believe that if deCONZ/Phoscon could also supporting exporting and importing backups in an open standard and open file format such as the "Open ZigBee Coordinator Backup Format" then that would allow ConBee or RaspBee users that today uses other implementations to restore and save backup Zigbee network backup files in the same or other Zigbee implementation using old or new hardware, including other implementations which already support ConBee or RaspBee, like Zigbee2MQTT, iOBroker, and zigpy-deconz (used by Home Assistant's ZHA integration, and Jeedom, etc.).

So far this open format standard has already has been adopted or is in the progress of being adopted by these Zigbee implementations:

• Home Assistant's ZHA integration component (via zigpy, zigpy-cli, and its libraries) https://www.home-assistant.io/integrations/zha

• Zigbee for Domoticz (the Zigbee plugin for Domoticz) -> Domoticz-Zigate developers looking into implementing a zigpy layer for Zigbee stack and/or radio hardware abstraction? zigpy/zigpy#865

• Jeedom Official Zigbee Plugin Jeedom Official Zigbee Plugin based on zigpy now listed as "stable" zigpy/zigpy#725

• Zigbee2MQTT (Z2M) https://www.zigbee2mqtt.io/ (ZNP Adapter Manager/Backup Refactor Koenkk/zigbee-herdsman#303)

• ioBroker Zigbee https://www.iobroker.net/#en/adapters/adapterref/iobroker.zigbee/README.md

Example usage in implementations:

• https://github.com/zigpy/zigpy-znp/blob/dev/TOOLS.md#backup-and-restore
• https://github.com/zigpy/bellows#nvram-backup-and-restore

FYI, this is already a popular upgrade, downgrade, and migration method among users in some communities as seen in examples here:

• https://www.home-assistant.io/blog/2022/09/07/release-20229/#zigbee-backup-and-restore--migration
• https://www.youtube.com/watch?v=ql58ZfXso1M&ab_channel=EverythingSmartHome
• https://community.home-assistant.io/t/backup-your-zha-husbzb-1-stick-and-even-seamlessly-migrate-to-a-new-stick-without-re-pairing/229044/

[Hedda]

Any chance Dresden Elektronik will add "Open ZigBee Coordinator Backup Format" export/import support to deCONZ/Phoscon?

[Hedda]

FYI, Home Assistant's ZHA integration now has features and GUI/UI for backup, restore, and migration between different adapters:

https://www.home-assistant.io/blog/2022/09/07/release-20229/#zigbee-backup-and-restore--migration

https://www.home-assistant.io/integrations/zha/#zigbee-backup-and-restore-in-zha

This includes migrating to and/or from ConBee/RaspBee based adapters if the backup was made from within the ZHA integration.

It is also possible to migrate from Zigbee2MQTT to Home Assistant's ZHA integration if using a Texas Instruments based adapter:

https://skyconnect.home-assistant.io/zigbee-migration-selection/

[ukmgranger]

This would definitely be a great option. The thought of re-pairing my 180 devices fills me with dread!

[t34wrj]

This would be a great addition.

[Hedda]

Also see this related feature request for zigbee-herdsman (used by Zigbee2MQTT and IoBroker) -> Koenkk/zigbee-herdsman#665

[ChrisHae]

This kinda slipped pass me. But yes I think this is not a big effort to implement and would be an improvement for the ConBee/deCONZ.

[Hedda]

This kinda slipped pass me. But yes I think this is not a big effort to implement and would be an improvement for the ConBee/deCONZ.

@ChrisHae / @manup any updates on this feature request?

[manup]

Hi, as I've mentioned in another thread (I can't find the link quickly) this can't work.

From the project description:

The primary use case for this specification is to provide a simple standard for exporting ZigBee network information to disk. Standardizing this format will allowing users to backup, restore, and migrate their networks between coordinator hardware and network management software without having to needlessly rejoin all of their devices to a new network.

Imho this description is very misleading for users since they will be left with with a half working network when migrating between projects: e.g. ZHA → Z2M.

The described data in https://github.com/zigpy/open-coordinator-backup#backup-structure is only a tiny fraction of what can be considered a backup. Bindings, reporting configuration, groups, scenes, automations and rules and other parts, can't be really transferred between "network management software" like ZHA, Z2M and deCONZ because they all have very different approaches how to implement these.

For example here is the Philips Hue motion sensor:

• deCONZ https://github.com/dresden-elektronik/deconz-rest-plugin/blob/master/devices/philips/sml004_motion_sensor.json
• Zigbee2Mqtt https://github.com/Koenkk/zigbee-herdsman-converters/blob/c1d45f17048205b1203a10bf855d1551e8f61110/src/devices/philips.js#L2228-L2254
• ZHA https://github.com/zigpy/zha-device-handlers/blob/dev/zhaquirks/philips/motion.py

When migrating just this single device, say from Z2M to ZHA, the on device configuration for bindings and reporting needs to be reconfigured to match what the device handlers expect: Remove bindings that aren't used on new project, verify and update ZCL reporting. Add/remove groups, .... For many sleeping devices this is only possible during the joining phase. Side note: It can also be done very slowly step by step when the device wakes up... but as far as I'm aware only the deCONZ DDF implementation is able to do this currently.

Z2M and deCONZ use a SQLite databases storing such information, perhaps ZHA too. These have custom schemas. In my opinion this is the only true backup, but it has the limitation that it only works for the given project. I don't see a way to migrate one format to another since also the logic in the code / device handlers using the data is very different.

---

TLTR

• Migrate from project X to Y not possible (too different, will end in half broken network)
• Migrate hardware X to Y possible, but use the projects database store for that

[secabeen]

I respectfully disagree here. Although not everything can be migrated, I think there is still value in supporting a basic device migration between projects. It's all about how complicated your network is, and what you are trying to do. Without some form of migration support, the only way to switch projects or devices is to do a complete factory reset and re-pair on every device, which is a lot of work. If it's possible to migrate devices to new projects or new hardware, that saves some work relative to the alternative. Yes, doing so would wipe out "Bindings, reporting configuration, groups, scenes, automations and rules and other parts", but the alternative (factory reset and re-pair) wipes out all of that and requires physical access and manual reset of every device. One of these takes longer than the other.

Please consider support for a migration path that at least allows users to adopt other projects without having to completely burn their existing network to the ground before starting over. Any amount of work saved is a win over that alternative.