Add migration from FFULM to FFMUC

[maurerle]

See https://github.com/freifunkMUC/gluon-packages/tree/main/ffulm-migration

seems like I need to create a PR to get the firmware bakery running.

[grische]

@maurerle We want to run the migration from older versions (that are based on <=OpenWRT 19.07) towards our legacy first and then let all devices that can migrate to stable auto-migrate to stable afterwards.

Manually triggered CI run: https://github.com/freifunkMUC/site-ffm/actions/runs/8239459590

[grische]

I replace the custom patch with a package, so that it's easier to work on it. This should also fix outstanding migration issues.

[T0biii]

when the x86-64 build is done i going to test the sysupgrade again

Update Logs: https://gist.github.com/T0biii/6417727bec661755f318d4dd9508cd1a

after upgradeing again with auotupdate to v2024.3.1

[T0biii]

Logs after sysupgrade and after autoupdater:
https://gist.github.com/T0biii/e86e9dc40c4e0c552d0de2d59fbb3446

After The Autoupdater there are still left some things that need to be cleand up:

https://gist.github.com/T0biii/e86e9dc40c4e0c552d0de2d59fbb3446#file-logread-after-autoupdater-L611-L634

• Hostname got not migraded is now OpenWrt but should be this

[T0biii]

https://gist.github.com/T0biii/1bae4151d1d2d911371e152232cbf7ca

• Hostname got not migraded
• DNS works now
• bat0 appears to be broken now (Error - interface bat0 is not present or not a batman-adv interface)

[T0biii]

Log Info after Upgrade:

Sat Apr  6 19:55:00 2024 cron.info crond[3093]: USER root pid 6220 cmd /usr/sbin/print_map.sh -p
Sat Apr  6 19:55:00 2024 cron.info crond[3093]: USER root pid 6221 cmd /usr/sbin/print_service.sh -p

seems to be a cronjob from ULM to publish services/map infos

Cronjobs

# Check for new firmware every 6 hours
0 */6 * * * /usr/sbin/autoupdater
# Publish map data
*/5 * * * * /usr/sbin/print_map.sh -p
# Publish service data
*/5 * * * * /usr/sbin/print_service.sh -p
# Reboot every 5 days at 04:05
5 4 */5 * * /sbin/reboot`

[grische]

@T0biii thanks for spotting this. I compared backups from a firstboot legacy and a ffulm-migrated legacy device.
These things still need to be addressed:

• Check what to do with /etc/opkg/keys/ Delete keys in /etc/opkg
• Probably delete /etc/sysupgrade.conf
• Check /etc/sysctl.conf if we want to keep it Delete /etc/sysctl.conf
• Delete /etc/firewall.user
• Purge node entries from /etc/hosts
• Delete /etc/crontabs folder
• Remove preconfigured root password from /etc/shadow
• Delete /etc/uhttpd.crt and /etc/uhttpd.key

[maurerle]

regarding /etc/opkg/keys/ there is a simple fix in freifunk-gluon/packages#263

you can copy the patch from the FFAC repo to the tree, to remove old unused keys:
https://github.com/ffac/site/blob/v2023.2.x/patches/gluon-packages.patch

The existing keys are only needed for opkg modules and surely don't work, but it is better to remove these entirely

[grische]

@maurerle thanks for clarifying! We don't use the upgrade for migration, but a custom rc.d path and can use a simple bash startup script for the cleanup.

[grische]

I updated the module to address the above cleanup, improved logging and cleaned up the code. The only open questions is how to migration ath79 images. ffulm provides both ar71xx as well as ath79 images for devices and we only have ar71xx currently in the legacy (Gluon v2021.1.x) firmware.

For example:

• https://firmware.freifunk-ulm.de/?version=ffulm-2.3.1&target=ath79%2Fgeneric&id=tplink_tl-wdr4300-v1
• https://firmware.freifunk-ulm.de/?version=ffulm-2.3.1&target=ar71xx%2Fgeneric&id=tl-wdr4300-v1

@maurerle any ideas?

[grische]

@DasSkelett @maurerle I won't be able to really working on this before the deadline anymore.

The basic framework is in place and a basic migration is possible with this. Currently we only migration:

• Node name
• Contact info
• Location

If there is more features (like the ones that were initially in @DasSkelett commit), you can add them to the existing S14 file or add an S15/S16/S17/S18 file (make sure it's executable chmod +x) like this one:
https://github.com/freifunkMUC/gluon-packages/tree/main/ffulm-migration/files/lib/gluon/ffulm-migration/rc.d

In the stages S14 to S18 you will be able to access ALL configurations from ffulm, even the network, system, etc configurations by using the prefix ${FFULM_PREFIX} (comes from . /lib/gluon/ffulm-migration/migration-tools.sh) in your uci get commands.

For example:

# shellcheck source=ffulm-migration/files/lib/gluon/ffulm-migration/migration-tools.sh
. /lib/gluon/ffulm-migration/migration-tools.sh

if what_i_need=$(uci -q get "${FFULM_PREFIX}network.whatever.you.need"); then
  # do some magic
  echo $what_i_need
else
  migration_log ffulm-migration warn "Failed to find 'what I needed'"
fi

Open ToDos

• Migrate to ffmuc_ulm domain instead of ffmuc_welt
• How to migrate meshes? We probably need to migrate more configs, like the the mesh-on-... options for that.
• How to migrate ath79 devices that are only ar71xx in legacy (e.g. WDR4300)? Can we somehow override this? Force-flashing seems to do the trick.
• Anything else that we missed?

[maurerle]

Eine kleine Test-Checkliste war ungefähr:

• kommt der Knoten nach migration korrekt online (test auf x86 vm)

• kommt der Knoten nach migration korrekt online (test auf wifi router)

• klappt Client wifi an dem Router ordentlich

• wird das root passwort korrekt gelöscht (empty login geht nicht)

• wird nodename/lat/lon/contact korrekt übernommen?

• sind eingetragene ssh keys weiterhin gültig?

• wird mesh_on_wan korrekt übernommen? (not tested yet)

• mesh lan und custom network config wird nicht übernommen (wontfix)

• How to migrate meshes? - we could build another ffulm openwrt image which includes this package https://github.com/freifunk-gluon/community-packages/tree/master/ffac-scheduled-sysupgrade

• How to migrate ath79 images - we can wait and see if this is an actual issue (if there is actually ath79 firmware in the wild) and if this is the case, we can build a openwrt-22 image which includes the migration package

This PR is ready from my side.

[GoliathLabs]

Should we add the schedules sysupgrade package before merging?