Project

General

Profile

Bug #3121

Can't boot after upgrading lvm2 from 2.03.13-1 to 2.03.14-1

laust - 11 months ago - . Updated 11 months ago.

Status:
fixed
Priority:
bug
Assignee:
% Done:

0%


Description

  • steps to reproduce:
    Upgrade system incl. lvm2.
  • expected result:
    Normal reboot.
  • actual result:
    Boot stuck after typing the encryption password (at kernel boot, not to unlock /boot).

Downgrading lvm2 fixes it without any other manual intervention.

On an x86_64 system, nonsystemd, LUKS+LVM.

History

#1

Updated by Jetpack 11 months ago

The exact same thing is happening with me as well. I saw some "command not found" udev errors during the mkinitcpios, but ignored them since it told me that the image generation was successful...

After booting via livecd, I tried to run mkinitcpio again, and had the same issue: it spams "add_udev_rule: command not found" when running the encrypt and lvm2 build hooks.

lvm2 version 2.03.14-1

#2

Updated by Jetpack 11 months ago

laust wrote:

Downgrading lvm2 fixes it without any other manual intervention.

I tried to downgrade lvm2 but I'm still getting the same errors when running the build hooks.

#3

Updated by Jetpack 11 months ago

Jetpack wrote:

laust wrote:

Downgrading lvm2 fixes it without any other manual intervention.

I tried to downgrade lvm2 but I'm still getting the same errors when running the build hooks.

Turns out I had to downgrade cryptsetup as well. My Linux-libre box now boots without this issue. Here's what I did:

  • downgraded lvm2 to version 2.03.13-1
  • downgraded cryptsetup to version 2.4.1-1

Package archives are available on the Arch Linux Archives.

#4

Updated by avalos 11 months ago

Jetpack wrote:
Turns out I had to downgrade cryptsetup as well. My Linux-libre box now boots without this issue. Here's what I did:

  • downgraded lvm2 to version 2.03.13-1
  • downgraded cryptsetup to version 2.4.1-1

Package archives are available on the Arch Linux Archives.

I can confirm your solution is working.

#5

Updated by avalos 11 months ago

So, now what? What is causing the issue with the new version? Is it a bug? When will we be able to upgrade those packages again?

#6

Updated by Drag0nFly 11 months ago

This is a little nasty and appears related to https://github.com/archlinux/mkinitcpio/pull/54 (systemd/elogind 249.5-3 being required for non-systemd systems)

It's the “add_udev_rule” function which is missing after the upgrade, causing the initcpios to fail–

/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found

Similar issue reported on Artix here: https://forum.artixlinux.org/index.php/topic,3217.0.html

Hopefully a fix will make it to Parabola as well when this gets some priority.

#7

Updated by dllud 11 months ago

Confirmed here too with systemd. Downgrading lvm2 and cryptsetup to the versions described above fixed it for the time being.
The proper fix that Parabola maintainers need to apply is just upgrading systemd in [libre] to 249.5-3 as pointed out by eworm-de

#8

Updated by GNUtoo 11 months ago

For me it also warns about add_udev_rule for [encrypt] as well.

I added . /usr/lib/initcpio/install/systemd at the beginning of /usr/lib/initcpio/install/lvm and /usr/lib/initcpio/install/encrypt and the warning goes away during mkinitcpio -p linux-libre.

I've then tried to boot the resulting linux-libre entry after that and it still didn't boot automatically.

On the machine I've tested it on, I've the following disk structure:

/dev/sda1 (LVM PV)
           |-> ext4 /boot/
           |-> LUKS (LVM PV)
                    |-> ext4 rootfs

And it worked before.

So maybe there is more than just not finding the add_udev_rule function.

If instead I don't add this source and just add lvm and cryptsetup in BINARIES in mkinitcpio.conf, lvchange -ay hangs when trying to activate LVM volumes, but with . /usr/lib/initcpio/install/systemd and without these additions in BINARIES I can still manually use them to boot my system.

edit1: fixed the disk structure syntax.

#9

Updated by GNUtoo 11 months ago

  • Status changed from unconfirmed to confirmed
#10

Updated by GNUtoo 11 months ago

I reinstalled the stock cryptsetup and lvm2 packages (to get unmodified /usr/lib/initcpio/install/ file) and I built and installed Arch Linux's systemd (from systemd/trunk from https://github.com/archlinux/svntogit-packages).

It didn't complain when running the pacman hook that created/updated the initramfs, and it booted.

Note that I used systemd before on this machine so I've no idea how this would behave with OpenRC.

Beside name changes (Arch -> Parabola, and more importantly Linux -> GNU/Linux) in systemd, we also have changes specifically to enable the peaceful cohabitation between openrc and systemd in Parabola.

So we also do need to make sure that it still works for OpenRC users.

#11

Updated by bill-auger 11 months ago

my main dev box uses openrc, and my build partition is LVM
(along with all of my VMs); so i need to address this also
before rebooting my dev box

#12

Updated by bill-auger 11 months ago

the openrc service files are in the 'lvm2-openrc' package - if anything needs to change for openrc, it is probably in one of these four files:

$ pkgfile -l lvm2-openrc | grep -vE '\/$' | cut -d '   ' -f 2
/etc/conf.d/lvm
/etc/init.d/lvm
/etc/init.d/lvm-monitoring
/etc/init.d/lvmetad
#13

Updated by bill-auger 11 months ago

my openrc system was not affected, at least i show no symptoms - neither my / nor /boot are not on an LVM volume, and my LVM volumes are not encrypted; but i can access them normally

#14

Updated by CommodoreCrunch 11 months ago

Just made a pull request with the fix for OpenRC installs: https://pagure.io/abslibre/pull-request/35

The same hook can be added to the libre systemd package to fix it there, but I don't currently have a systemd install to test it so I haven't pull requested that. It would be added to the initcpio-install-udev file.

#15

Updated by bill-auger 11 months ago

  • Assignee set to bill-auger
  • Status changed from confirmed to in progress

that PKGBUILD does not patch any files and does not install initcpio_install - where did you put your initcpio_install ?

why not patch the 'functions' file directly in the mkinitcpio package, using the arch patch exactly?

#16

Updated by CommodoreCrunch 11 months ago

The eudev package already installs the initcpio_install file to /usr/lib/initcpio/install/udev at line 80 of the PKGBUILD

#17

Updated by bill-auger 11 months ago

sry for the lazy question - im more interested in my second
question - we generally prefer to do exactly what arch does
(within the FSDG) - if that patch is due to be in a future
release of 'mkinitcpio', then it is most sensible to patch that
package now - that would need to happen later anyways, if the
patch is accepted upstream, but then also these changes today
would need to be reverted

#18

Updated by bill-auger 11 months ago

there is a patched 'mkinitcpio' package in [nonsystemd-testing] that people can try - nonsystemd people plz try it

  • nonsystemd-testing/mkinitcpio 30-2.nonsystemd1

here is the patch, if people want to apply it to the /usr/lib/initcpio/functions file manually:

https://github.com/archlinux/mkinitcpio/commit/9ee1333a5f3302d7ddb004cf0909c94b4cff60ba.diff

ani has confirmed that the patch works for systemd systems with FDE - nonsystemd-testing/mkinitcpio can probably be moved to [nonsystemd]; and a similar temporary package created for [libre]

#19

Updated by bill-auger 11 months ago

there is also a patched 'mkinitcpio' package in [libre-testing] if systemd people still have trouble

  • libre-testing/mkinitcpio 30-2.parabola2

ovruni maintains 'systemd' - i would prefer to wait for him to upgrade it, unless it is an emergency

#20

Updated by nona 11 months ago

bill-auger wrote:

there is a patched 'mkinitcpio' package in [nonsystemd-testing] that people can try - nonsystemd people plz try it

running OpenRC 0.42.1-7, enabling [nonsystemd-testing] (by uncommenting inside /etc/pacman.conf) and running pacman -Su mkinitcpio lvm2 cryptsetup works here
openrc 0.42.1-7
lvm2 2.03.14-1
cryptsetup 2.4.1-3
cryptsetup-openrc 20170927-3
lvm2-openrc 20170927-3
nonsystemd-testing/mkinitcpio 30-2.nonsystemd1

#21

Updated by Jetpack 11 months ago

bill-auger wrote:

there is a patched 'mkinitcpio' package in [nonsystemd-testing] that people can try - nonsystemd people plz try it

  • nonsystemd-testing/mkinitcpio 30-2.nonsystemd1

The patch is working here :)

  • openrc 0.42.1-7
  • lvm2 2.03.14-1
  • lvm2-openrc 20170927-3
  • cryptsetup 30-2.nonsystemd1
  • cryptsetup-openrc 20170927-3
  • mkinitcpio 2.4.1-3
#22

Updated by bill-auger 11 months ago

ok i will move it from nonsystemd-testing to nonsystemd

are any systemd users still having trouble? - maybe the package in [libre-testing] is not needed?

#23

Updated by Anonymous 11 months ago

systemd user here. I have not tested yet the new package in [libre-testing], but last time I did a full system upgrade (on 1 November), I could not boot because of both lvm and encrypt hooks and had to chroot to manually revert systemd, lvm2 and cryptsetup to their previous state. So, If I understood the issue, I do confirm that the package needs to be adapted for systemd users as well.

#24

Updated by GNUtoo 11 months ago

Hi,

With libre-testing, it worked.

For the record here's my pacman.conf:

#
# /etc/pacman.conf
#
# See the pacman.conf(5) manpage for option and repository directives

#
# GENERAL OPTIONS
#

[options]
Architecture = x86_64
# XferCommand = /usr/local/bin/curl-tor-get %u %o
HoldPkg = pacman glibc
CleanMethod = KeepInstalled

# Misc options
UseSyslog
Color
CheckSpace
VerbosePkgLists

SigLevel = Required DatabaseOptional
LocalFileSigLevel = Optional

###########################
# Parabola standard repos #
###########################

[libre-testing]
Include = /etc/pacman.d/mirrorlist

[libre]
Include = /etc/pacman.d/mirrorlist

#######################
# Arch standard repos #
#######################

[testing]
Include = /etc/pacman.d/mirrorlist

[core]
Include = /etc/pacman.d/mirrorlist

[extra]
Include = /etc/pacman.d/mirrorlist

[community-testing]
Include = /etc/pacman.d/mirrorlist

[community]
Include = /etc/pacman.d/mirrorlist

######################
# Supplemental repos #
######################

#[pcr-testing]
#Include = /etc/pacman.d/mirrorlist

[pcr]
Include = /etc/pacman.d/mirrorlist

[cross]
Include = /etc/pacman.d/mirrorlist

##############
# Over-rides #
##############

Include = /etc/pacman.d/*.conf

And here's the command I used to go back to Parabola's systemd packages:

pacman -S systemd systemd-common systemd-libs systemd-libsystemd systemd-libudev systemd-nss-myhostname systemd-nss-mymachines systemd-nss-resolve systemd-nss-systemd systemd-resolvconf systemd-sysvcompat systemd-udev

#25

Updated by bill-auger 11 months ago

And here's the command I used to go back to Parabola's systemd packages:

did `pacman -Syuu` not work? - to migrate systemd<>non-systemd should be simple (no conflicts, no breakage) - eg: any of these should work:

# pacman -C /etc/pacman.conf.x86_64.nonsystemd -Syuu  # prepare pacman.conf.systemd, manually beforehand, once
# pacman -C /etc/pacman.conf.x86_64.systemd    -Syuu  # prepare pacman.conf.nonsystemd, manually beforehand, once
# pacman -Syuu                                        # flip commented-state of [nonsystemd] in pacman.conf, 
                                                      # manually beforehand, each migration

FWIW, we could package pacman.conf.x86_64.{,non}systemd for the use case and recommend that method - alternatively, i made a tool to automate the last case, which could be libre/pacman or your-initfreedom install hooks - IIRC i have not mentioned that script to anyone yet; but it is part of the grand plan: that migrations should be easy and smooth (noob-friendly)

#26

Updated by Anonymous 11 months ago

Is there a way to enable [libre-testing] specifically for mkinitcipio? I don’t want to mess with other packages at the testing stage on my production machine here.

#27

Updated by theova 11 months ago

bill-auger wrote:

are any systemd users still having trouble? - maybe the package in [libre-testing] is not needed?

After today's update I got

$ sudo mkinitcpio -p linux-libre                                  
==> Building image from preset: /etc/mkinitcpio.d/linux-libre.preset: 'default'
  -> -k /boot/vmlinuz-linux-libre -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-libre.img
==> Starting build: 5.14.11-gnu-1
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [encrypt]
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
  -> Running build hook: [lvm2]
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/install/lvm2: line 33: add_udev_rule: command not found
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-libre.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux-libre.preset: 'fallback'
  -> -k /boot/vmlinuz-linux-libre -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-libre-fallback.img -S autodetect
==> Starting build: 5.14.11-gnu-1
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [encrypt]
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
  -> Running build hook: [lvm2]
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/functions: line 196: add_udev_rule: command not found
/usr/lib/initcpio/install/lvm2: line 33: add_udev_rule: command not found
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: /boot/initramfs-linux-libre-fallback.img
==> Image generation successful

So it is still needed.

boarim wrote:

Is there a way to enable [libre-testing] specifically for mkinitcipio? I don’t want to mess with other packages at the testing stage on my production machine here.

I updated only this package with

sudo pacman -U https://repo.parabola.nu/libre-testing/os/x86_64/mkinitcpio-30-2.parabola2-any.pkg.tar.zst

Now, mkinitcpio runs without problems.

#28

Updated by bill-auger 11 months ago

ok, so 'libre-testing/mkinitcpio' is necessary and sufficient - i
will move it to [libre]

#29

Updated by Anonymous 11 months ago

theova wrote:

I updated only this package with
sudo pacman -U https://repo.parabola.nu/libre-testing/os/x86_64/mkinitcpio-30-2.parabola2-any.pkg.tar.zst

Okay, thanks for the tip.

bill-auger: I can confirm this did the trick on systemd setups. I had no issue with the upgrade this time.

#30

Updated by bill-auger 11 months ago

  • Status changed from in progress to fixed

Also available in: Atom PDF