Sometimes it might be necessary to start in single-user mode to do some administration work, or even reset an existing password.

Normally this can be achieved via the GRUB boot loader.

CentOS / RedHat (with root account enabled)

Switch on your system.

Press Esc until the GRUB menu shows up.

This will bring up the GNU GRUB menu. If the CentOS/RedHat logo/boot messages show up you will need to restart (Ctrl-Alt-Del) and try again.

Select the OS/boot you want to edit. Normally the first line. Press e to edit it.

CentOS Linux (3.10.0-1160.53.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.45.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.42.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.41.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.36.2.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-d0401f7cdedb4955a0a262b3e0054323) 7 (Core)


Use the ↑ and ↓ keys to change the selection.
Press 'e' to edit the selected item, or 'c' for command prompt.

You will need to find the entry for the kernel. Normally starts with linux16.

[...]
linux16 /vmlinuz-3.10.0-1160.53.1.el7.x86_64 root=/dev/mapper/centos_centos7-root ro rd.lvm.lv=centos_centos7/root rd.lvm.lv=centos_centos7/swap rhgb quite LANG=en_US.UTF-8
[..]

At the end of the line you can choose to add one of the following:

single
systemd.unit=rescue.target
systemd.unit=emergency.target

So that it looks like this:

[...]
linux16 /vmlinuz-3.10.0-1160.53.1.el7.x86_64 root=/dev/mapper/centos_centos7-root ro rd.lvm.lv=centos_centos7/root rd.lvm.lv=centos_centos7/swap rhgb quite LANG=en_US.UTF-8 single
[..]

Type Ctrl-X to exit and the system will start in single- user mode.

If required, remount the root filesystem:

# mount -o remount,rw /

[If there are other filesystems you need to mount from fstab:]

# mount --all

CentOS / RedHat (without root account enabled)

It might be that your system didn’t have a root account enabled, in which case the above steps will fail. There is a workaround.

Switch on your system.

Press Esc until the GRUB menu shows up.

This will bring up the GNU GRUB menu. If the CentOS/RedHat logo/boot messages show up you will need to restart (Ctrl-Alt-Del) and try again.

Select the OS/boot you want to edit. Normally the first line. Press e to edit it.

CentOS Linux (3.10.0-1160.53.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.45.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.42.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.41.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.36.2.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-d0401f7cdedb4955a0a262b3e0054323) 7 (Core)


Use the ↑ and ↓ keys to change the selection.
Press 'e' to edit the selected item, or 'c' for command prompt.

You will need to find the entry for the kernel. Normally starts with linux16.

[...]
linux16 /vmlinuz-3.10.0-1160.53.1.el7.x86_64 root=/dev/mapper/centos_centos7-root ro rd.lvm.lv=centos_centos7/root rd.lvm.lv=centos_centos7/swap rhgb quite LANG=en_US.UTF-8
[..]

At the end of the line add this:

rd.break

So that it looks like this:

[...]
linux16 /vmlinuz-3.10.0-1160.53.1.el7.x86_64 root=/dev/mapper/centos_centos7-root ro rd.lvm.lv=centos_centos7/root rd.lvm.lv=centos_centos7/swap rhgb quite LANG=en_US.UTF-8 rd.break
[..]

Type Ctrl-X to exit and the system will start emergency mode in read-only mode.

Remount the root filesystem as read/write:

# mount -o remount,rw /sysroot

Switch to the sysroot jail.

# chroot /sysroot

Reset the password, or do any required tasks.

If you have SELinux enforcing mode enabled it will protect the OS from any changes. After you change the password type the following to overcome this:

# touch /.autorelabel

Restart.

# reboot -f

Ubuntu / Debian

Switch on your system.

Press and hold the Shift key.

In some instances pressing the Esc key several times (instead of holding it) achieves the same result. Just be aware that if you press it too many times it will bring you to the GRUB CLI. You can type normal and you will get to the menu described below.

This will bring up the GNU GRUB menu. If the Ubuntu logo/boot messages show up you will need to restart (ctrl-alt-del) and try again.

Select Advanced Options on the GRUB menu.

			GNU GRUB version 2.04

 Ubuntu 20.04.4 LTS
*Advanced options for Ubuntu 20.04.4 LTS
 History for Ubuntu 20.04.4 LTS
 UEFI Firmware Settings

And select the recovery mode option. Normally the latest kernel installed on your system.

			GNU GRUB version 2.04

 * Ubuntu 20.04.4 LTS, with Linux 5.13.0-37-generic
** Ubuntu 20.04.4 LTS, with Linux 5.13.0-37-generic (recovery mode)
 Ubuntu 20.04.4 LTS, with Linux 5.13.0-35-generic
 Ubuntu 20.04.4 LTS, with Linux 5.13.0-35-generic (recovery mode)

This will boot the system and show a series of options. Select root.

Recovery Menu (filesystem state: read only)

	resume			Resume normal boot
	clean			Try to make free space
	dpkg			Repair broken packages
	fsck			Check all file systems
	grub			Update grub bootloader
	network			Enable networking
	root			Drop to root shell prompt
	system-summary	System summary

					<OK>

This message will show. Press Enter.

Press Enter for maintenance
(or press Ctrl-D to continue)

If your / volume is ZFS it will be already read and write. Other filesystems might start in read only mode. If so, remount:

# mount -o remount,rw /

[If there are other filesystems you need to mount from fstab:]

# mount --all

Changing an user’s password

# passwrd <username>

Adding a new user

In the rare event of not having an user, you can add one and give it sudo privileges.

# useradd <username>

# usermod -a -G sudo <username>

# passwd <username>

Other OS

You can find description on how to get to single-user mode for other Linux distros in this Microsoft’s article.

https://docs.microsoft.com/en-us/troubleshoot/azure/virtual-machines/serial-console-grub-single-user-mode

On Ubuntu 20.04 after installing the NVIDIA driver 510 metapackage the system stopped booting.

It will either hang with a black screen and blinking cursor on the top left or show the following error message:

[FAILED] Failed to start Mark current ZSYS boot as successful.
See 'systemctl status zsys-commit.service' for details.
[  OK  ] Stopped User Manager for UID 1000.

Attempting to revert from a snapshot ends up with the same error message. This wasn’t the case on another separate system that had the same upgrade.

The “20.04 zys-commit.service fails” message is quite interesting and it seems that the overall cause is a mismatch of user/kernel zfs components.

These are the steps I followed to fix it. Many thanks to Lockszmith for his research in identifying the issue and finding a fix. He created two posts raising it, links provided here.

https://askubuntu.com/users/720005/lockszmith

https://askubuntu.com/questions/1388997/zsys-commit-service-fails-with-couldnt-promote-dataset-not-a-cloned-filesy

Fix

Restart Ubuntu and boot in recovery mode

[In GRUB]

*Advanced options for Ubuntu 20.04.3 LTS

[Select the first recovery option in the menu]
*Ubuntu 20.04.3 LTS, with Linux 5.xx.x-xx-generic (recovery mode)

[Wait for the system to load the menu and select:]
root

[Press Enter for Maintenance to get the CLI]

Check the reason for the error.

# systemctl status zsys-commit.service
[...]
 Feb 17 11:11:24 ab350 systemd[1] zsysctl[4068]: level=error msg="couldn't commit: couldn't promote dataset "rpool/ROOT/ubuntu_733qyk": couldn't promote "rpool/ROOT/ubuntu_733qyk": not a cloned filesystem"
 [...]

Attempting to promote it manually fails:

# zfs promote rpool/ROOT/ubuntu_733qyk

cannot promote `rpool/ROOT/ubuntu_733qyk` : not a cloned filesystem

Uninstall the NVIDIA drivers.

# dkms uninstall nvidia/510.47.03
# dkms remove nvidia/510.47.03 --all

Make sure you can connect to the internet. You can temporarily assign a DHCP address to one of the network interfaces.

# dhclient -v eno1
# ip address

Update the system and install a 3rd party ZFS set of tools.

# apt update
# apt upgrade
# apt autoremove

[Add 3rd party PPA for zfstools]
# add-apt-repository ppa:jonathonf/zfs
# apt update 

[Upgrade ZFS]
# apt upgrade

[If ZFS isn't upgraded, do it manually]
# apt install zfs-initramfs zfs-zed zfsutils-linux

It might take a bit to update. Reboot normally.

# reboot

It should boot normally.

If this doesn’t work for you, reboot in recovery mode again and promote the filesystem manually.

# zfs promote rpool/ROOT/ubuntu_733qyk

Sort graphical drivers

Revert to NVIDIA metapackage 470 (if this is what broke your system). Reboot, and fix resolution settings.

Upgrading back to 510 will bring the error back and make it even more difficult to fix. Don’t!

Things will only work if zfs and zfs-kmod match versions.

$ zfs --version
zfs-0.8.3-1ubuntu12.13
zfs-kmod-2.0.6-1ubuntu2

[boot in recovery mode]
# apt reinstall zfs-initramfs zfs-zed zfsutils-linux
# zfs promote rpool/ROOT/ubuntu_733qyk

[reboot in normal mode]
[Configure the 470 drivers]

Reverting to previous ZFS version

The system should now be back to normal, but you might want to revert to the mainline ZFS version despite the bug. After all, this was a hack to promote the filesystem and get it back to work.

# add-apt-repository --remove ppa:jonathonf/zfs

[Check that is has been removed]
$ apt policy

# apt update

[Pray]
# apt remove zfs-initramfs zfs-zed zfsutils-linux
# apt install zfs-initramfs zfs-zed zfsutils-linux

[Check the right version is installed]
# apt list --installed | grep zfs

# apt autoremove

[Pray harder]
# reboot

With that I managed to bring my system back to a working condition, but updating the drivers a second time made it fail again and I couldn’t fix it. A clean install of 20.04.3 doesn’t seem to exhibit this problem. Not sure what is the reason behind it but there are a few bugs open with Canonical regarding this.

I hope that 22.04 will bring a better ZFS version.