Ubuntu: System freezing for a few seconds with iwlwifi microcode sw error

For a few months now my main system would momentarily freeze or stall (usually about 20-30 seconds) and then continue working. It was something that started after one system update and wasn’t fixed with any further updates.

I opened a bug with Debian without much luck.

The system would notify that one of the CPU cores timed out and for a few moments the computer would stall or freeze before resuming as if nothing had happened.

dmesg was showing timeouts related to iwlwifi:

[ 2313.312941] Timeout waiting for hardware access (CSR_GP_CNTRL 0x0c04000c)
[ 2313.312995] WARNING: CPU: 4 PID: 1424 at drivers/net/wireless/intel/iwlwifi/pcie/trans.c:2066 iwl_trans_pcie_grab_nic_access+0x1f9/0x230 [iwlwifi]

iwlwifi is the kernel driver for several Intel based wireless adapters.

It is possible to install a different versions of the driver manually but I don’t like to deviate too much from a standard installation. It can complicate maintenance and troubleshooting in the future.

The issue would happen several times throughout the day. The truth is that with some of the updates it became less frequent, but it was still happening often enough and filling the syslog with errors.

Sep 8 14:02:53 tuxedo kernel: [15317.424052] iwlwifi 0000:08:00.0: Microcode SW error detected. Restarting 0x0.
Sep 8 14:02:53 tuxedo kernel: [15317.467034] WARNING: CPU: 4 PID: 1350 at drivers/net/wireless/intel/iwlwifi/mvm/../iwl-trans.h:1180 iwl_mvm_dump_lmac_error_log+0x51d/0x570 [iwlmvm]
Sep 8 14:02:53 tuxedo kernel: [15317.467096] RIP: 0010:iwl_mvm_dump_lmac_error_log+0x51d/0x570 [iwlmvm]
Sep 8 14:02:53 tuxedo kernel: [15317.467120] iwl_mvm_dump_nic_error_log+0x20/0x70 [iwlmvm]
Sep 8 14:02:53 tuxedo kernel: [15317.467126] iwl_mvm_nic_error+0x35/0x40 [iwlmvm]
Sep 8 14:02:53 tuxedo kernel: [15317.467146] iwl_pcie_irq_handle_error+0xb3/0x110 [iwlwifi]
Sep 8 14:02:53 tuxedo kernel: [15317.467172] iwlwifi 0000:08:00.0: HW error, resetting before reading
Sep 8 14:02:53 tuxedo kernel: [15317.474328] iwlwifi 0000:08:00.0: Start IWL Error Log Dump:
Sep 8 14:02:53 tuxedo kernel: [15317.474407] iwlwifi 0000:08:00.0: Start IWL Error Log Dump:
Sep 8 14:02:53 tuxedo kernel: [15317.474510] iwlwifi 0000:08:00.0: 0xA5A5A5A2 | FSEQ_ERROR_CODE
Sep 8 14:02:53 tuxedo kernel: [15317.476063] iwlwifi 0000:08:00.0: FW error in SYNC CMD STATISTICS_CMD
[...]
Sep 8 16:38:15 tuxedo kernel: [24620.634090] iwlwifi 0000:08:00.0: Microcode SW error detected. Restarting 0x0.
Sep 8 16:38:15 tuxedo kernel: [24620.677672] WARNING: CPU: 4 PID: 1350 at drivers/net/wireless/intel/iwlwifi/mvm/../iwl-trans.h:1180 iwl_mvm_dump_lmac_error_log+0x51d/0x570 [iwlmvm]
Sep 8 16:38:15 tuxedo kernel: [24620.677672] Modules linked in: ccm btrfs xor zstd_compress raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs rfcomm vboxnetadp(OE) vboxnetflt(OE) xfrm_user xfrm_algo vboxdrv(OE) cmac algif_hash algif_skcipher af_alg bnep nls_iso8859_1 snd_hda_codec_hdmi snd_sof_pci snd_sof_intel_hda_common snd_soc_hdac_hda snd_sof_intel_hda snd_sof_intel_byt snd_sof_intel_ipc snd_sof snd_sof_xtensa_dsp snd_hda_ext_core snd_hda_codec_realtek snd_soc_acpi_intel_match snd_soc_acpi snd_hda_codec_generic ledtrig_audio snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel snd_intel_dspcfg snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_seq_midi mei_hdcp snd_seq_midi_event intel_rapl_msr snd_rawmidi iwlmvm mac80211 x86_pkg_temp_thermal btusb intel_powerclamp btrtl snd_seq uvcvideo btbcm libarc4 btintel kvm_intel snd_seq_device bluetooth videobuf2_vmalloc snd_timer videobuf2_memops kvm videobuf2_v4l2 videobuf2_common videodev ecdh_generic rapl mc ecc intel_cstate input_leds snd iwlwifi joydev
Sep 8 16:38:15 tuxedo kernel: [24620.677771] CPU: 4 PID: 1350 Comm: irq/149-iwlwifi Tainted: P W OEL 5.4.0-47-generic #51-Ubuntu
Sep 8 16:38:15 tuxedo kernel: [24620.677828] iwl_pcie_irq_handle_error+0xb3/0x110 [iwlwifi]
Sep 8 16:38:15 tuxedo kernel: [24620.677834] iwl_pcie_irq_msix_handler+0x180/0x4a0 [iwlwifi]
Sep 8 16:38:15 tuxedo kernel: [24620.677859] iwlwifi 0000:08:00.0: HW error, resetting before reading
Sep 8 16:38:15 tuxedo kernel: [24620.684849] iwlwifi 0000:08:00.0: Start IWL Error Log Dump:
Sep 8 16:38:15 tuxedo kernel: [24620.684853] iwlwifi 0000:08:00.0: Status: 0x00000040, count: -1156803901
Sep 8 16:38:15 tuxedo kernel: [24620.684854] iwlwifi 0000:08:00.0: Loaded firmware version: 46.6bf1df06.0
Sep 8 16:38:15 tuxedo kernel: [24620.684855] iwlwifi 0000:08:00.0: 0x7EEF56BF | ADVANCED_SYSASSERT
Sep 8 16:38:15 tuxedo kernel: [24620.684857] iwlwifi 0000:08:00.0: 0x570D67A4 | trm_hw_status0
Sep 8 16:38:15 tuxedo kernel: [24620.684858] iwlwifi 0000:08:00.0: 0xA73BAA53 | trm_hw_status1
Sep 8 16:38:15 tuxedo kernel: [24620.684858] iwlwifi 0000:08:00.0: 0x514FE288 | branchlink2
Sep 8 16:38:15 tuxedo kernel: [24620.684860] iwlwifi 0000:08:00.0: 0xF788EFED | interruptlink1
Sep 8 16:38:15 tuxedo kernel: [24620.684861] iwlwifi 0000:08:00.0: 0x6A36568B | interruptlink2
Sep 8 16:38:15 tuxedo kernel: [24620.684862] iwlwifi 0000:08:00.0: 0xED77E0BF | data1
Sep 8 16:38:15 tuxedo kernel: [24620.684863] iwlwifi 0000:08:00.0: 0x02150D02 | data2
Sep 8 16:38:15 tuxedo kernel: [24620.684864] iwlwifi 0000:08:00.0: 0xF7226FBF | data3
Sep 8 16:38:15 tuxedo kernel: [24620.684864] iwlwifi 0000:08:00.0: 0xAA60560C | beacon time
Sep 8 16:38:15 tuxedo kernel: [24620.684865] iwlwifi 0000:08:00.0: 0xC6D77EF3 | tsf low
Sep 8 16:38:15 tuxedo kernel: [24620.684867] iwlwifi 0000:08:00.0: 0x6007359C | tsf hi
Sep 8 16:38:15 tuxedo kernel: [24620.684868] iwlwifi 0000:08:00.0: 0x19FB8975 | time gp1
Sep 8 16:38:15 tuxedo kernel: [24620.684869] iwlwifi 0000:08:00.0: 0x007831E0 | time gp2
Sep 8 16:38:15 tuxedo kernel: [24620.684870] iwlwifi 0000:08:00.0: 0x63B3FF9E | uCode revision type
Sep 8 16:38:15 tuxedo kernel: [24620.684871] iwlwifi 0000:08:00.0: 0x050B9CF9 | uCode version major
Sep 8 16:38:15 tuxedo kernel: [24620.684872] iwlwifi 0000:08:00.0: 0xF195AF3A | uCode version minor
Sep 8 16:38:15 tuxedo kernel: [24620.684873] iwlwifi 0000:08:00.0: 0xFC8B56B7 | hw version
Sep 8 16:38:15 tuxedo kernel: [24620.684874] iwlwifi 0000:08:00.0: 0xDF4DF7DF | board version
Sep 8 16:38:15 tuxedo kernel: [24620.684875] iwlwifi 0000:08:00.0: 0xFFFFF764 | hcmd
Sep 8 16:38:15 tuxedo kernel: [24620.684876] iwlwifi 0000:08:00.0: 0x61242F92 | isr0
Sep 8 16:38:15 tuxedo kernel: [24620.684877] iwlwifi 0000:08:00.0: 0x6BF774FF | isr1
Sep 8 16:38:15 tuxedo kernel: [24620.684878] iwlwifi 0000:08:00.0: 0x2BF9DCB6 | isr2
Sep 8 16:38:15 tuxedo kernel: [24620.684879] iwlwifi 0000:08:00.0: 0xFFFFB9ED | isr3
Sep 8 16:38:15 tuxedo kernel: [24620.684880] iwlwifi 0000:08:00.0: 0xE50361AD | isr4
Sep 8 16:38:15 tuxedo kernel: [24620.684881] iwlwifi 0000:08:00.0: 0x7F5B434C | last cmd Id
Sep 8 16:38:15 tuxedo kernel: [24620.684882] iwlwifi 0000:08:00.0: 0x6235DCC3 | wait_event
Sep 8 16:38:15 tuxedo kernel: [24620.684883] iwlwifi 0000:08:00.0: 0xFDBBEBCF | l2p_control
Sep 8 16:38:15 tuxedo kernel: [24620.684884] iwlwifi 0000:08:00.0: 0xB018A732 | l2p_duration
Sep 8 16:38:15 tuxedo kernel: [24620.684885] iwlwifi 0000:08:00.0: 0x637FAB7A | l2p_mhvalid
Sep 8 16:38:15 tuxedo kernel: [24620.684886] iwlwifi 0000:08:00.0: 0xA4283522 | l2p_addr_match
Sep 8 16:38:15 tuxedo kernel: [24620.684887] iwlwifi 0000:08:00.0: 0xF6CBFCCA | lmpm_pmg_sel
Sep 8 16:38:15 tuxedo kernel: [24620.684888] iwlwifi 0000:08:00.0: 0x172A8B18 | timestamp
Sep 8 16:38:15 tuxedo kernel: [24620.684889] iwlwifi 0000:08:00.0: 0xF97EB79D | flow_handler
Sep 8 16:38:15 tuxedo kernel: [24620.684942] iwlwifi 0000:08:00.0: Start IWL Error Log Dump:
Sep 8 16:38:15 tuxedo kernel: [24620.684943] iwlwifi 0000:08:00.0: Status: 0x00000040, count: -179974165
Sep 8 16:38:15 tuxedo kernel: [24620.684944] iwlwifi 0000:08:00.0: 0x587791CE | ADVANCED_SYSASSERT
Sep 8 16:38:15 tuxedo kernel: [24620.684945] iwlwifi 0000:08:00.0: 0xEFE6FFBD | umac branchlink1
Sep 8 16:38:15 tuxedo kernel: [24620.684946] iwlwifi 0000:08:00.0: 0x68924F38 | umac branchlink2
Sep 8 16:38:15 tuxedo kernel: [24620.684947] iwlwifi 0000:08:00.0: 0xADDFFAAA | umac interruptlink1
Sep 8 16:38:15 tuxedo kernel: [24620.684948] iwlwifi 0000:08:00.0: 0x5F1F8880 | umac interruptlink2
Sep 8 16:38:15 tuxedo kernel: [24620.684949] iwlwifi 0000:08:00.0: 0xF9D3AF77 | umac data1
Sep 8 16:38:15 tuxedo kernel: [24620.684950] iwlwifi 0000:08:00.0: 0x1CFAD4EC | umac data2
Sep 8 16:38:15 tuxedo kernel: [24620.684951] iwlwifi 0000:08:00.0: 0x4ED9F44D | umac data3
Sep 8 16:38:15 tuxedo kernel: [24620.684952] iwlwifi 0000:08:00.0: 0x8B07B711 | umac major
Sep 8 16:38:15 tuxedo kernel: [24620.684954] iwlwifi 0000:08:00.0: 0x7BDAEF3C | umac minor
Sep 8 16:38:15 tuxedo kernel: [24620.684955] iwlwifi 0000:08:00.0: 0xA2D50203 | frame pointer
Sep 8 16:38:15 tuxedo kernel: [24620.684956] iwlwifi 0000:08:00.0: 0xF35DCF8D | stack pointer
Sep 8 16:38:15 tuxedo kernel: [24620.684956] iwlwifi 0000:08:00.0: 0xCA5E1060 | last host cmd
Sep 8 16:38:15 tuxedo kernel: [24620.684957] iwlwifi 0000:08:00.0: 0xFBF4D7E7 | isr status reg
Sep 8 16:38:15 tuxedo kernel: [24620.684981] iwlwifi 0000:08:00.0: Fseq Registers:
Sep 8 16:38:15 tuxedo kernel: [24620.685050] iwlwifi 0000:08:00.0: 0xA5A5A5A2 | FSEQ_ERROR_CODE
Sep 8 16:38:15 tuxedo kernel: [24620.685185] iwlwifi 0000:08:00.0: 0xA5A5A5A2 | FSEQ_TOP_INIT_VERSION
Sep 8 16:38:15 tuxedo kernel: [24620.685338] iwlwifi 0000:08:00.0: 0xA5A5A5A2 | FSEQ_CNVIO_INIT_VERSION
Sep 8 16:38:15 tuxedo kernel: [24620.685490] iwlwifi 0000:08:00.0: 0xA5A5A5A2 | FSEQ_OTP_VERSION
Sep 8 16:38:15 tuxedo kernel: [24620.685629] iwlwifi 0000:08:00.0: 0xA5A5A5A2 | FSEQ_TOP_CONTENT_VERSION
Sep 8 16:38:15 tuxedo kernel: [24620.685764] iwlwifi 0000:08:00.0: 0xA5A5A5A2 | FSEQ_ALIVE_TOKEN
Sep 8 16:38:15 tuxedo kernel: [24620.685899] iwlwifi 0000:08:00.0: 0xA5A5A5A2 | FSEQ_CNVI_ID
Sep 8 16:38:15 tuxedo kernel: [24620.686035] iwlwifi 0000:08:00.0: 0xA5A5A5A2 | FSEQ_CNVR_ID
Sep 8 16:38:15 tuxedo kernel: [24620.686170] iwlwifi 0000:08:00.0: 0xA5A5A5A2 | CNVI_AUX_MISC_CHIP
Sep 8 16:38:15 tuxedo kernel: [24620.686305] iwlwifi 0000:08:00.0: 0xA5A5A5A2 | CNVR_AUX_MISC_CHIP
Sep 8 16:38:15 tuxedo kernel: [24620.686455] iwlwifi 0000:08:00.0: 0xA5A5A5A2 | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
Sep 8 16:38:15 tuxedo kernel: [24620.686593] iwlwifi 0000:08:00.0: 0xA5A5A5A2 | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
Sep 8 16:38:15 tuxedo kernel: [24620.686616] iwlwifi 0000:08:00.0: Collecting data: trigger 2 fired.
Sep 8 16:38:43 tuxedo kernel: [24649.245319] Modules linked in: ccm btrfs xor zstd_compress raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs rfcomm vboxnetadp(OE) vboxnetflt(OE) xfrm_user xfrm_algo vboxdrv(OE) cmac algif_hash algif_skcipher af_alg bnep nls_iso8859_1 snd_hda_codec_hdmi snd_sof_pci snd_sof_intel_hda_common snd_soc_hdac_hda snd_sof_intel_hda snd_sof_intel_byt snd_sof_intel_ipc snd_sof snd_sof_xtensa_dsp snd_hda_ext_core snd_hda_codec_realtek snd_soc_acpi_intel_match snd_soc_acpi snd_hda_codec_generic ledtrig_audio snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine snd_hda_intel snd_intel_dspcfg snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_seq_midi mei_hdcp snd_seq_midi_event intel_rapl_msr snd_rawmidi iwlmvm mac80211 x86_pkg_temp_thermal btusb intel_powerclamp btrtl snd_seq uvcvideo btbcm libarc4 btintel kvm_intel snd_seq_device bluetooth videobuf2_vmalloc snd_timer videobuf2_memops kvm videobuf2_v4l2 videobuf2_common videodev ecdh_generic rapl mc ecc intel_cstate input_leds snd iwlwifi joydev
Sep 8 16:38:43 tuxedo kernel: [24649.245403] Workqueue: events iwl_fw_error_dump_wk [iwlwifi]
Sep 8 16:38:43 tuxedo kernel: [24649.245420] iwl_trans_pcie_release_nic_access+0x61/0x70 [iwlwifi]
Sep 8 16:38:43 tuxedo kernel: [24649.245424] iwl_trans_pcie_read_mem+0x94/0xc0 [iwlwifi]
Sep 8 16:38:43 tuxedo kernel: [24649.245429] iwl_fw_dump_mem.isra.0.part.0+0x50/0x90 [iwlwifi]
Sep 8 16:38:43 tuxedo kernel: [24649.245434] iwl_fw_error_dump_file.isra.0+0x438/0xfe0 [iwlwifi]
Sep 8 16:38:43 tuxedo kernel: [24649.245437] iwl_fw_dbg_collect_sync+0xe7/0x310 [iwlwifi]
Sep 8 16:38:43 tuxedo kernel: [24649.245444] iwl_fw_error_dump_wk+0x59/0x80 [iwlwifi]
Sep 8 16:38:44 tuxedo kernel: [24649.716905] iwlwifi 0000:08:00.0: Failing on timeout while stopping DMA channel 8 [0xa5a5a5a2]
Sep 8 16:38:44 tuxedo kernel: [24649.775318] iwlwifi 0000:08:00.0: Applying debug destination EXTERNAL_DRAM
Sep 8 16:38:44 tuxedo kernel: [24649.890159] iwlwifi 0000:08:00.0: Applying debug destination EXTERNAL_DRAM
Sep 8 16:38:44 tuxedo kernel: [24649.954830] iwlwifi 0000:08:00.0: FW already configured (0) - re-configuring
Sep 8 16:38:44 tuxedo kernel: [24649.972183] iwlwifi 0000:08:00.0: BIOS contains WGDS but no WRDS

Doing some reading it seems that Intel wireless drivers have some known issues. It seems that there isn’t much that can be done from that side. It is very likely that even newer drivers and firmware would behave the same.

But ignoring recurring issues is never good practice!

There was a note on the Debian’s wiki about how to disable driver options for troubleshooting. On the Arch Linux forum the user mkdy created a modprobe file to do that after experiencing similar freezes.

I tried his workaround and it also works on my Ubuntu system.

Create /etc/modprobe.d/iwl.conf and add the following content:

options iwlwifi 11n_disable=1 swcrypto=0 bt_coex_active=0 power_save=0
options iwlmvm power_scheme=1 
options iwlwifi d0i3_disable=1 
options iwlwifi uapsd_disable=1 
options iwlwifi lar_disable=1

After rebooting the system the errors and freezes stopped. It could be that not all of the options are needed. If I have time I will experiment and try to determine if one in particular is the one responsible for my freezes.

Addendum

A few weeks after I applied the patch and experienced no more entries in the log I noticed me experiencing lag online. I then noticed that my log had some new entries related to iwlwifi and to top it off I realised that the above settings made the connection slower.

I don’t know if this was caused by the last kernel and system updates, it could be. The system is on 5.4.0-47 which is currently the latest release on Ubuntu 20.04.

I ended up testing the different options on /etc/modprobe.d/iwl.conf. This entry seems to remove the syslog iwlwifi entries, the random freezes, the lag and the slow connection.

options iwlwifi bt_coex_active=0 swcrypto=0 power_save=0
options iwlmvm power_scheme=1

I am leaving all of the previous above for reference in case you are trying to troubleshoot a similar issue.

I will keep a look on future updates to make sure it doesn’t break again.