Ubuntu: apt error message “Key is stored in legacy trusted.gpg keyring”
After upgrading to Ubuntu 22.04 running apt shows an error message saying “Key is stored in legacy trusted.gpg keyring“:
# apt update
[..]
All packages are up-to-date.
W: https://apt.syncthing.net/dists/syncthing/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
The key needs to be exported from the legacy keyring and then imported back to the current system.
List the keys and find the key ID of the repository that is showing the error. In this case it is Syncthing.
Update the source file for the repository adding the exported key.
# vim /etc/apt/sources.list.d/syncthing.list
deb [arch=amd64 signed-by=/usr/share/keyrings/syncthing.gpg] https://apt.syncthing.net/ syncthing stable #Syncthing
Confirm that the error message is no longer showing.
# apt update
[...]
Hit:5 https://apt.syncthing.net syncthing InRelease
[...]
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up-to-date.
Finally, remove the old signature.
# apt-key del 00655A3E
Linux: Booting in single-user mode
Sometimes it might be necessary to start in single-user mode to do some administration work, or even reset an existing password.
Normally this can be achieved via the GRUB boot loader.
CentOS / RedHat (with root account enabled)
Switch on your system.
Press Esc until the GRUB menu shows up.
This will bring up the GNU GRUB menu. If the CentOS/RedHat logo/boot messages show up you will need to restart (Ctrl-Alt-Del) and try again.
Select the OS/boot you want to edit. Normally the first line. Press e to edit it.
CentOS Linux (3.10.0-1160.53.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.45.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.42.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.41.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.36.2.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-d0401f7cdedb4955a0a262b3e0054323) 7 (Core)
Use the ↑ and ↓ keys to change the selection.
Press 'e' to edit the selected item, or 'c' for command prompt.
You will need to find the entry for the kernel. Normally starts with linux16.
Type Ctrl-X to exit and the system will start in single- user mode.
If required, remount the root filesystem:
# mount -o remount,rw /
[If there are other filesystems you need to mount from fstab:]
# mount --all
CentOS / RedHat (without root account enabled)
It might be that your system didn’t have a root account enabled, in which case the above steps will fail. There is a workaround.
Switch on your system.
Press Esc until the GRUB menu shows up.
This will bring up the GNU GRUB menu. If the CentOS/RedHat logo/boot messages show up you will need to restart (Ctrl-Alt-Del) and try again.
Select the OS/boot you want to edit. Normally the first line. Press e to edit it.
CentOS Linux (3.10.0-1160.53.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.45.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.42.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.41.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.36.2.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-d0401f7cdedb4955a0a262b3e0054323) 7 (Core)
Use the ↑ and ↓ keys to change the selection.
Press 'e' to edit the selected item, or 'c' for command prompt.
You will need to find the entry for the kernel. Normally starts with linux16.
Type Ctrl-X to exit and the system will start emergency mode in read-only mode.
Remount the root filesystem as read/write:
# mount -o remount,rw /sysroot
Switch to the sysroot jail.
# chroot /sysroot
Reset the password, or do any required tasks.
If you have SELinux enforcing mode enabled it will protect the OS from any changes. After you change the password type the following to overcome this:
# touch /.autorelabel
Restart.
# reboot -f
Ubuntu / Debian
Switch on your system.
Press and hold the Shift key.
In some instances pressing the Esc key several times (instead of holding it) achieves the same result. Just be aware that if you press it too many times it will bring you to the GRUB CLI. You can type normal and you will get to the menu described below.
This will bring up the GNU GRUB menu. If the Ubuntu logo/boot messages show up you will need to restart (ctrl-alt-del) and try again.
Select Advanced Options on the GRUB menu.
GNU GRUB version 2.04
Ubuntu 20.04.4 LTS
*Advanced options for Ubuntu 20.04.4 LTS
History for Ubuntu 20.04.4 LTS
UEFI Firmware Settings
And select the recovery mode option. Normally the latest kernel installed on your system.
GNU GRUB version 2.04
* Ubuntu 20.04.4 LTS, with Linux 5.13.0-37-generic
** Ubuntu 20.04.4 LTS, with Linux 5.13.0-37-generic (recovery mode)
Ubuntu 20.04.4 LTS, with Linux 5.13.0-35-generic
Ubuntu 20.04.4 LTS, with Linux 5.13.0-35-generic (recovery mode)
This will boot the system and show a series of options. Select root.
Recovery Menu (filesystem state: read only)
resume Resume normal boot
clean Try to make free space
dpkg Repair broken packages
fsck Check all file systems
grub Update grub bootloader
network Enable networking
root Drop to root shell prompt
system-summary System summary
<OK>
This message will show. Press Enter.
Press Enter for maintenance
(or press Ctrl-D to continue)
If your / volume is ZFS it will be already read and write. Other filesystems might start in read only mode. If so, remount:
# mount -o remount,rw /
[If there are other filesystems you need to mount from fstab:]
# mount --all
Changing an user’s password
# passwrd <username>
Adding a new user
In the rare event of not having an user, you can add one and give it sudo privileges.
Linux / Unix: Comparing differences between folders
I had to check the file changes between two Backintime snapshots recently. You can always use rsync for that, but there is a more straightforward way by using diff.
$ diff -qr directoyr-1/ directory-2/
-q will display only the files that differ.
-r will make the comparison recursive.
There is a GUI application called Meld that provides similar functionality, but the diff approach will work anywhere and requires memorising less flags than rsync.
Linux / Ubuntu / hdparm: Identifying drive features and setting sleep patterns
Preparing the storage
Install hdparm and smartmontools
Install hdparm and the SMART monitoring tools.
# apt install hdparm smartmontools
Identify the right hard drive
Make sure you identify the correct drive, as some of the commands will destroy data. If you don’t understand the commands, then check them first. You have been warned.
Identify the block size
Knowing the block size of the device is important. It will help optimising writes, and in the case of SSD or flash drives avoid write amplification and wear and tear.
Pay attention to the physical/optimal size. This is the one that matters.
SSDs will hide the true size of the pages and blocks. Even the same drive models might be built with different components, so getting it right is tricky.
Use the drive’s sector physical size to match the ZFS ashift (block size).
Retrieve drive IDs
When setting ZFS pools or using disk tools it is best to avoid using device names as they can easily change their order. Using the drive ID or serial will ensure that no matter in which port or in which order the drives are plugged it will be the correct drive selected.
This matters with any disk accessing utility if you have several drives, or will be inserting external drives regularly.
$ ls -l /dev/disk/by-id/
[...]
lrwxrwxrwx 1 root root 9 Mar 9 13:16 usb-TOSHIBA_External_USB_3.0_20150612015531-0:0 -> ../../sda
[...]
You can also extract model and serial numbers with hdparm.
Even better, depending on the use of the drive, and if there is a plan to add mirror drives, is to partition the drive to ensure there is enough space if a different drive model is later added. Although I believe ZFS already does this and rounds down partitions using Mebibytes.
Test for damaged sectors
An additional and optional step is to test the hard drive for damaged sectors. This kind of test tends to be destructive so it is best if it is done before configuring the pools.
badblocks is a useful tool to achieve this.
It is installed by default, but if not you can do it manually.
# apt install e2fsprogs
A destructive test can be done with:
# badblocks -wsv -b 4096 /dev/sda
If you want to run the test while preserving the disk data you can run it in a non-destructive way. This will take longer.
# badblocks -nsv -b 4096 /dev/sda
ZFS has built-in checks and protection so in most cases you can skip this step.
Setting hard drive sleep patterns
Above I explained that using disk IDs is always a better idea. For simplicity, I will be using device names in several examples below, but I still advise using IDs or serials.
Check if the disk supports sleep
Check if the drive supports standby.
# hdparm -y /dev/sda
If supported the output will be:
/dev/sda:
issuing standby command
Any other output might indicate that the drive doesn’t support sleep, or that a different tool/setting might be required.
Next, check if the drive supports write cache:
# hdparm -I /dev/sda | grep -i 'Write cache'
The expected output is:
* Write cache
The * indicates that the feature is supported.
An example of a complete hdparm output from a drive is shown below for reference. Different drives, with different features, will show different output, or even none at all.
# hdparm -I /dev/sda
/dev/sda:
ATA device, with non-removable media
Model Number: TOSHIBA MD04ACA500
Serial Number: 55OBK0SPFPHC
Firmware Revision: FP2A
Transport: Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
Supported: 8 7 6 5
Likely used: 8
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 9767541168
Logical Sector size: 512 bytes
Physical Sector size: 4096 bytes
Logical Sector-0 offset: 0 bytes
device size with M = 1024*1024: 4769307 MBytes
device size with M = 1000*1000: 5000981 MBytes (5000 GB)
cache/buffer size = unknown
Form Factor: 3.5 inch
Nominal Media Rotation Rate: 7200
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec'd by Standard, no device specific minimum
R/W multiple sector transfer: Max = 16 Current = 16
Advanced power management level: 128
DMA: sdma0 sdma1 sdma2 mdma0 mdma1 *mdma2 udma0 udma1 udma2 udma3 udma4 udma5
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* SMART feature set
Security Mode feature set
* Power Management feature set
* Write cache
* Look-ahead
* Host Protected Area feature set
* WRITE_BUFFER command
* READ_BUFFER command
* NOP cmd
* DOWNLOAD_MICROCODE
* Advanced Power Management feature set
SET_MAX security extension
* 48-bit Address feature set
* Device Configuration Overlay feature set
* Mandatory FLUSH_CACHE
* FLUSH_CACHE_EXT
* SMART error logging
* SMART self-test
* General Purpose Logging feature set
* WRITE_{DMA|MULTIPLE}_FUA_EXT
* 64-bit World wide name
* WRITE_UNCORRECTABLE_EXT command
* {READ,WRITE}_DMA_EXT_GPL commands
* Segmented DOWNLOAD_MICROCODE
unknown 119[7]
* Gen1 signaling speed (1.5Gb/s)
* Gen2 signaling speed (3.0Gb/s)
* Gen3 signaling speed (6.0Gb/s)
* Native Command Queueing (NCQ)
* Host-initiated interface power management
* Phy event counters
* Host automatic Partial to Slumber transitions
* Device automatic Partial to Slumber transitions
* READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
DMA Setup Auto-Activate optimization
Device-initiated interface power management
* Software settings preservation
* SMART Command Transport (SCT) feature set
* SCT Write Same (AC2)
* SCT Error Recovery Control (AC3)
* SCT Features Control (AC4)
* SCT Data Tables (AC5)
* reserved 69[3]
Security:
Master password revision code = 65534
supported
not enabled
not locked
not frozen
not expired: security count
supported: enhanced erase
more than 508min for SECURITY ERASE UNIT. more than 508min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 500003964bc01970
NAA : 5
IEEE OUI : 000039
Unique ID : 64bc01970
Checksum: correct
An example of a complete smartctl output from a drive is shown below also for reference. As mentioned earlier, different systems will generate different outputs.
# smartctl --all /dev/sda
smartctl 7.1 2019-12-30 r5022 [aarch64-linux-5.4.0-1029-raspi] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Toshiba 3.5" MD04ACA... Enterprise HDD
Device Model: TOSHIBA MD04ACA500
Serial Number: 55OBK0SPFPHC
LU WWN Device Id: 5 000039 64bc01970
Firmware Version: FP2A
User Capacity: 5,000,981,078,016 bytes [5.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Mon Mar 8 15:02:10 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Status not supported: Incomplete response, ATA output registers missing
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.
General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 533) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0
3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 9003
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 9222
5 Reallocated_Sector_Ct 0x0033 100 100 050 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 084 084 000 Old_age Always - 6418
10 Spin_Retry_Count 0x0033 253 100 030 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 9212
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 482
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 104
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 9225
194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 37 (Min/Max 15/72)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 253 000 Old_age Always - 0
220 Disk_Shift 0x0002 100 100 000 Old_age Always - 0
222 Loaded_Hours 0x0032 085 085 000 Old_age Always - 6393
223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0
224 Load_Friction 0x0022 100 100 000 Old_age Always - 0
226 Load-in_Time 0x0026 100 100 000 Old_age Always - 214
240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 5617 -
# 2 Short offline Completed without error 00% 4702 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
More information about hdparm and smartctl is available on the following sites.
Depending on the drive manufacturer and model you might need to query the settings with different flags. Check the man page.
[Get/set the Western Digital Green Drive's "idle3" timeout value.]
# hdparm -J /dev/sd[a-e]
/dev/sda:
wdidle3 = 300 secs (or 13.8 secs for older drives)
/dev/sdb:
wdidle3 = 8.0 secs
/dev/sdc:
wdidle3 = 300 secs (or 13.8 secs for older drives)
/dev/sdd:
wdidle3 = 300 secs (or 13.8 secs for older drives)
/dev/sde:
wdidle3 = 300 secs (or 13.8 secs for older drives)
From the man page:
A setting of 30 seconds is recommended for Linux use. Permitted values are from 8 to 12 seconds, and from 30 to 300 seconds in 30-second increments. Specify a value of zero (0) to disable the WD idle3 timer completely (NOT RECOMMENDED!).
There are flags for temperature (-H for Hitachi drives), acoustic management (-M), measuring cache performance (-T), and others. Go on, read that man page. 🙂
The -S flag sets the standby/spindown timeout for the drive. Basically, how long the drive will wait with no disk activity before turning off the motor.
Value
Description
0
Disable the feature.
1 to 240
Five seconds multiples (a value of 120 means 10 minutes).
241 to 251
Thirty minutes intervals (a value of 242 means 1 hour).
Note that hdparm might wake the drive up when is queried. smartctl can query the drive without waking it.
# APM setting (-B)
apm = 127
# APM setting while on battery (-B)
apm_battery = 127
# on/off drive's write caching feature (-W)
write_cache = on
# Standby (spindown) timeout for drive (-S)
spindown_time = 120
# Western Digital (WD) Green Drive's "idle3" timeout value. (-J)
wdidle3 = 300
hdparm.conf method
Edit the configuration file:
# vim /etc/hdparm.conf
And insert an entry for each drive. Select only settings/features/values that are supported by that drive, otherwise the rest of the options won’t be applied. Test, test, test!
In my case, the above method works. I couldn’t get this one to work on my system, but it could be because of the OS. I am leaving it for reference in case it might be of help.
# vim /etc/udev/rules.d/69-disk.rules
Create an entry for each drive editing the serial number and hdparm parameters. Make sure that only supported flags are added or it will fail.
Linux: Configure locale and keyboard layout when remotely accessing from a Mac
At work I have to remote into several different Linux systems from a Mac and there is always the pain of having to handle different keyboard layouts if using Synergy or VMs.
The conversion from a Mac keyboard layout doesn’t translate correctly when the Linux system has the keyboard configured as a PC.
The XKBOPTIONS I have here are for Synergy to keep the Control and Alt keys on the Mac working the same on the Linux systems. You might not need or want it. Just remove it from the commands if that is the case.
You can also do a text GUI configuration of the keyboard with:
# dpkg-reconfigure keyboard-configuration
If your environment isn’t in English the menus won’t be either. You can force the language output of the application launched to be in the default one. That would be English in most cases. The same command as above but forcing the output to be in English:
You might need to restart if editing i18n, but the change should be automatic with loadkeys.
CentOS 7
Edit the following file:
/etc/locale.conf
with the following:
LANG="en_GB.UTF-8"
Or type the following command:
# localectl set-locale LANG=en_GB.UTF-8
Set the keymap:
# localectl set-x11-keymap gb macintosh mac lv3:alt_switch
In CentOS 7 it isn’t necessary to reboot, the above command automatically loads the key mappings.
CentOS 8
The same commands used for CentOS 7 fail. I suspect that there is a file or folder with the keyboard mappings that has been moved. It might be a bug or a deprecated feature.
Server and minimal installs are normally headless and have no graphical interface.
If needed you can add a GUI manually. The process is slightly different depending on the distro.
RedHat / CentOS 7.x
# yum update
# yum groupinstall "Server with GUI"
RedHat / CentOS 8.x
# dnf update
# dnf groupinstall workstation
Ubuntu 18.04.x LTS
# apt update
[Install minimum GNOME desktop]
# apt install --no-install-recommends ubuntu-desktop
[Install full desktop with associated applications]
(Long process and too many extras installed)
# apt install ubuntu-desktop
[There are other alternative desktops and installations possible:]
[Generic Gnome desktop]
# apt install vanilla-gnome-desktop
[Mate]
# apt install ubuntu-mate-desktop
[Xfce]
# apt install xubuntu-desktop
[KDE]
# apt install kubuntu-desktop
[LightDM]
# apt install --no-install-recommends lightdm
Debian 9.x
# apt update
# apt install gnome-core
Debian 10.x
# apt update
# apt install gnome-core
All the above distros use systemd as their init system and you set the default run level with the same set of commands.
[Enable run level 5 by default]
systemctl set-default graphical.target
[Enable run level 3 by default]
systemctl set-default multi-user.target
Despite systemd you can still use init to start the graphical interface without having to reboot.
# init 5
Linux: Initiating a CPU backtrace
Sometimes a process is taking a lot of CPU time and it isn’t clear what the cause is.
For example, at times it is common to see the kworker process consuming a lot of CPU. kworker is a placeholder process for kernel worker threads. These threads perform most of the actual processing for the kernel and you might want to see what device is involved.
You can run a CPU backtrace in Linux that records in dmesg what each one of the CPUs in the system are doing. This can be very useful to determine what specific process is hogging the CPU and in some cases to which module/driver it is related.
This is done using the magic SysRq key. This is a key combination that allows you to communicate directly with the kernel and perform several low level commands regardless of the state of the system. An exception would be a kernel panic, for obvious reasons.
When the magic SysRq key is enabled you can use the key combination Alt+SysRq+command key. There are many options and worth writing a future article just on it. The Wikipedia article explains some of them.
Because the magic SysRq key provides direct access to the kernel and the deep security implications of this it is disabled by default; if not in all, in most distros.
When the magic SysRq key is disabled and a backtrace is requested dmesg will not display any backtrace. So you can temporarily activate the magic SysRq key with:
# sysctl -w kernel.sysrq=1
or
# echo 1 > /proc/sys/kernel/sysrq
Be aware that this won’t persist between reboots. To turn it off you use:
# sysctl -w kernel.sysrq=0
or
# echo 0 > /proc/sys/kernel/sysrq
You can generate the backtrace with the Alt-SysRq-L key combination . If you need to script the backtrace or are accessing the system remotely, there is a CLI alternative to do the same:
The above example is from an idle system, but if the system was busier it would display more activity from other CPUs, system calls and what driver/module is involved.
If the same module, driver or hardware device keeps showing up in the backtraces you should check them as possible source of the high CPU utilisation.