I finally upgraded my headless home server that was running Ubuntu 18.04 to 20.04 – I only run long-term support releases (LTS) builds as this is my main home server with functionality that we all depend on for managing our data and household digital storage. I used do-release-upgrade to do the upgrade. Overall, it was an ok experience but in this post I want to highlight the state of networking configuration in Ubuntu 20.04 server.
On rebooting the server after the upgrade, I found that it was no longer reachable on the network. I hate this part because it means breaking out the one USB wired keyboard that I retain just for this purpose and switching my monitors HDMI input to the normally headless server to figure out what is going on. As it turned out, it was stuck in the boot process in a step that was taking x seconds of unlimited (ouch) – more careful reading of the boot spew indicated it was stuck bringing up the network. A little search on the interwebz yielded the nugget that mashing esc, shift, f12 or some combination thereof will allow you to skip such steps – I don’t know which one did the trick but mashing the keyboard like a manic monkey did get the machine to boot. Of course, there was no network support. On checking networkctl status it was clear that the bridge network I had setup as part of my QEMU+KVM setup prior to the upgrade was interfering somehow – the bridge adapter and vlan were not showing up even though the ethernet adapter itself was fine. To add to it, my home server acts as the DNS for my network through pi-hole and is supposed to have a static address but that wasn’t the case for the ethernet adapter as it had a different address.
Instead of trying to troubleshoot the mess, I took the easier approach – since my QEMU+KVM setup is disposable (and in general can be recreated very easily), I decided to nuke the bridge network and vlan. To do this, you need to first bring down the bridge adapter and then delete it using this incantation:
ip link set dev br0 down
brctl del br0
After this, things started improving – a reboot brought the machine back on the network but with the wrong ip address. The static IP was configured on my AmplifiHD to the mac address of the bridge adapter – fixing this and rebooting brought things back up mostly normal – pi-hole was still not functional for some reason. Again, instead of fixing this, I decided to nuke pi-hole and reinstall it instead. At this point, I also decided to look a bit closer at how Ubuntu 20.04 network configuration worked and here’s what I found:
For some reason, Ubuntu 20.04 uses something called cloud-init in my server – this is nominally a great idea for headless servers provisioned in a data center, but in my case was simply adding one more complication to configuration in my server. Ubuntu is using netplan normally for network configuration and I was fairly familiar and comfortable with this – but my /etc/netplan had a 50-cloud-init.yaml file in it that said don’t edit me, I am generated. To simplify the configuration and disable cloud-init, I did the following:
- Created an empty file called cloud-init.disabled in /etc/cloud
- Created a file /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the contents
network: {config: disabled}
With this, cloud-init is disabled and network configuration is only managed by netplan. I copied the 50-cloud-init.yaml into 01-netcfg.yaml and added renderer: networkd to it.
After this, I nuked pi-hole and reinstalled it and everything seemed to be working fine. Just to be sure there are no hidden issues, I checked journalctl -r -b to look through the messages and saw "Could not acquire DHCPv6 lease Address already in use" messages from networkd. A quick check showed dhcpcd was running in the machine and I realized pi-hole installs and enables dhcpcd which I don’t need. Stopping the daemon and removing it from the system fixed that problem and got me back to a fully functional home server with Pi-Hole enabled.
I hope this helps anyone who might run into similar problems with networking on Ubuntu 20.04 + QEMU + KVM + Pi-Hole.