Project

General

Profile

Bug #2141

libretools - Feature Request #2403: [libremakepkg]: eradicate mksource() from abslibre

libremakepkg failing to build package due to read-only startdir in recent librechroots

GNUtoo - over 5 years ago - . Updated 2 months ago.

Status:
fixed
Priority:
discussion
Assignee:
% Done:

0%


Description

I've tried building it inside an i686 chroot with:

$ sudo libremakepkg -n parabola-i686

and this gives:
  INSTALL sound/usb/snd-usb-audio.ko
  INSTALL sound/usb/snd-usbmidi-lib.ko
  INSTALL sound/usb/usx2y/snd-usb-us122l.ko
  INSTALL sound/usb/usx2y/snd-usb-usx2y.ko
  INSTALL sound/x86/snd-hdmi-lpe-audio.ko
  INSTALL virt/lib/irqbypass.ko
  DEPMOD  4.19.8-gnu-1
  -> Installing hooks...
/startdir/PKGBUILD: line 466: /startdir/linux.install.pkg: Read-only file system
==> ERROR: A failure occurred in package_linux-libre().
    Aborting...

related: https://lists.parabola.nu/pipermail/dev/2019-May/007207.html


Files

PKGBUILD (300 Bytes) PKGBUILD GNUtoo, 2019-04-16 06:17 PM
hello.install (77 Bytes) hello.install GNUtoo, 2019-04-16 06:20 PM
hello.sh (29 Bytes) hello.sh GNUtoo, 2019-04-16 06:20 PM
v20180804.diff (515 Bytes) v20180804.diff GNUtoo, 2019-05-16 02:55 PM

History

#1

Updated by GNUtoo over 5 years ago

  • Subject changed from linux-libre not building with libremakepkg i686 to linux-libre-4.19.8_gnu-1-i686 not building with libremakepkg
#2

Updated by bill-auger over 5 years ago

  • Assignee set to Megver83
#3

Updated by Megver83 about 5 years ago

  • Assignee deleted (Megver83)
  • Status changed from open to not-a-bug

two things:

1) this issue is not related with linux-libre specifically

2) man, don't be that obvious, your chroot is broken or corrupted. Try building linux-libre from another one and delete parabola-i686.

#4

Updated by bill-auger about 5 years ago

Megver83 wrote:

Try building linux-libre from another one and delete parabola-i686.

yes, that is always a last resort option - i have needed to destroy the chroot a few times lately - and not only the working chroot, but the entire read-only replica like:

  # librechroot -n default delete
  # librechroot -n i686 delete
  # rn -rf /var/lib/archbuild/default
  # rn -rf /var/lib/archbuild/i686
#5

Updated by GNUtoo about 5 years ago

  • Status changed from not-a-bug to confirmed

Before reporting the bug I made sure to delete the chroot and recreate it before.

I tried again to delete and recreate the chroot, and I still have the exact same issue with the package I'm interested in building (linux-libre-x86_64).

 |  /startdir/PKGBUILD: line 334: /startdir/linux.install.pkg: Read-only file system
 |  ==> ERROR: A failure occurred in package_linux-libre-x86_64().

and I do have space:

$ df -h --output=avail /var/lib/archbuild/
Avail
  27G

#6

Updated by GNUtoo about 5 years ago

I've tried again with linux-libre and I've the same issue:

 |    DEPMOD  4.20.6-gnu-1
 |    -> Installing hooks...
 |  /startdir/PKGBUILD: line 466: /startdir/linux.install.pkg: Read-only file system
 |  ==> ERROR: A failure occurred in package_linux-libre().
 |      Aborting...
==> Copying log and package files out of the chroot...

I didn't rebuild the chroot since my last try yesterday with linux-libre_x86-64 though.

#7

Updated by GNUtoo about 5 years ago

  • Subject changed from linux-libre-4.19.8_gnu-1-i686 not building with libremakepkg to linux-libre-4.19.8_gnu-1-i686 not building with libremakepkg with newly created librechroots

I tried again on winston.parabola.nu and I had the exact same issue, so this is probably related to the creation of a new chroot. It may work on older chroot, which I don't have lying around.

#8

Updated by oaken-source about 5 years ago

this behavior is a more general problem, and can also be observed when building any package that relies on the pkgver() function to update the PKGBUILD. it appears that for some reason, the build processes in the chroot have lost write permissions to the files in /startdir.

this might be a recent change to our build toolchain, I distinctly remember this working before.

#9

Updated by GNUtoo about 5 years ago

A workaround for that work at least for pkgver() is to build the package twice:
  • once with makepkg so it updates the PKGBUILD
  • once with libremakepkg after having updated the PKGBUILD
I wonder what would be the best way to deal with that?
  • Add a function to makepkg so it would update the PKGBUILD?
  • Enable the chroot to modify the PKGBUILD and copy it back to the directory in abslibre?
  • don't care about the changes made to PKGBUILD or other files in abslibre?
#10

Updated by GNUtoo about 5 years ago

  • Subject changed from linux-libre-4.19.8_gnu-1-i686 not building with libremakepkg with newly created librechroots to libremakepkg failing to build package due to read-only startdir in recent librechroots
#11

Updated by GNUtoo about 5 years ago

I've made a quick and dirty test case that I uploaded:

$ sudo libremakepkg 
==> Initializing the chroot...
==> Starting pre-build activities...
 |  ==> Downloading blacklist of proprietary software packages...done
 |    -> Inspecting package pkgname=hello.sh (0-1)
==> Downloading sources...
 |  ==> Making package: hello.sh 0-1 (mar. 16 avril 2019 20:15:44 CEST)
 |  ==> Retrieving sources...
 |    -> Found hello.sh
 |  ==> Validating source files with sha512sums...
 |      hello.sh ... Skipped
 |  ==> Entering fakeroot environment...
 |  ==> Creating source package...
 |    -> Adding PKGBUILD...
 |    -> Generating .SRCINFO file...
 |    -> Adding hello.sh...
 |    -> Compressing source package...
 |  ==> Leaving fakeroot environment.
 |  ==> Source package created: hello.sh (mar. 16 avril 2019 20:15:46 CEST)
==> Starting to build the package...
 |  ==> Cleaning chroot...
 |    -> Creating a full list of packages...
 |    -> No packages to remove
 |    -> No packages to add
 |  ==> Making package: hello.sh 0-1 (Tue 16 Apr 2019 08:15:49 PM CEST)
 |  ==> Checking runtime dependencies...
 |  ==> Checking buildtime dependencies...
 |  ==> Retrieving sources...
 |    -> Found hello.sh
 |  ==> WARNING: Skipping all source file integrity checks.
 |  ==> Extracting sources...
 |  ==> Entering fakeroot environment...
 |  ==> Starting package()...
 |  /startdir/PKGBUILD: line 13: /startdir/hello.install.pkg: Read-only file system
 |  ==> ERROR: A failure occurred in package().
 |      Aborting...
==> Copying log and package files out of the chroot...

and with makepkg:

$ makepkg
==> Making package: hello.sh 0-1 (mar. 16 avril 2019 20:17:30 CEST)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Found hello.sh
==> Validating source files with sha512sums...
    hello.sh ... Skipped
==> Extracting sources...
==> Entering fakeroot environment...
==> Starting package()...
==> Tidying install...
  -> Removing libtool files...
  -> Purging unwanted files...
  -> Removing static library files...
  -> Stripping unneeded symbols from binaries and libraries...
  -> Compressing man and info pages...
==> Checking for packaging issues...
==> Creating package "hello.sh"...
  -> Generating .PKGINFO file...
  -> Generating .BUILDINFO file...
  -> Adding install file...
  -> Generating .MTREE file...
  -> Compressing package...
==> Leaving fakeroot environment.
==> Finished making: hello.sh 0-1 (mar. 16 avril 2019 20:17:33 CEST)

#12

Updated by GNUtoo about 5 years ago

#13

Updated by GNUtoo about 5 years ago

#14

Updated by eschwartz about 5 years ago

bill-auger: No, that PKGBUILD doesn't work -- the install= must be global OR in the package() function. Furthermore, makepkg installs it relative to $startdir, not $srcdir -- so you can declare it during package to ./hello.install, but when makepkg reads it, it sees it as "$startdir/./hello.install", so you're still using the unmodified version.

And if you try using install="${srcdir}/hello.install", makepkg will error out because it is not permitted to use an install file that isn't in startdir.

There's really only one solution, and that is, use a read/write mounted startdir. This fixes both the linux packages which modify the install file, and packages which use pkgver() and modify the PKGBUILD.

#15

Updated by bill-auger about 5 years ago

eli was referring to my suggestion of including the the install hook file in the sources() array then modifying it the prepare() stage - i deleted the suggestion at the same time eli was commenting, because it does not actually work

#16

Updated by GNUtoo almost 5 years ago

Here's a workaround:

$ cd dir/where/PKGBUILD/resides
$ git clone git://git.parabola.nu/packages/libretools.git
$ git -C libretools/ checkout 646ac0258c3295943778142468aadfe5b04ad6d1~1
$ git -C libretools/ apply v20180804.diff
$ ln -s libretools/src/chroot-tools/libremakepkg ./
$ sudo ./libremakepkg 
[...]
$ ls
$ ls 
hello.install      hello.sh                       hello.sh-0-1-i686.pkg.tar.xz  libremakepkg
hello.install.pkg  hello.sh-0-1-i686-package.log  hello.sh-0-1.src.tar.gz       PKGBUILD

Thanks a lot to elibrokeit on the #parabola IRC channel on Freenode for pointing the point where the statedir was mounted to me and bill-auger. This enabled me to very easily find the issue.

#17

Updated by bill-auger almost 5 years ago

  • Priority changed from bug to discussion
  • Assignee set to lukeshu
  • Description updated (diff)
#18

Updated by bill-auger almost 5 years ago

the log message for the commit that introduced this change indicates that it was necessary for some reason - i think it would be hasty to revert this until we know what the trade-off would be

"- We mount the temporary directory containing the extracted source package files read-only,
to be sure that makepkg doesn't modify the PKGBUILD. This is necessary because
--holdver only disables pkgver() if it's a VCS package."

right now, i think the only packages that are broken by this are the kernels - there is a single command that requires a writable /startdir, and it does not appear to have any value whatsoever - all that it does is write the 'pkgbase' into the .install script - 'pkgbase' will never change, so that could be hard-coded into the .install script anyways

#19

Updated by bill-auger almost 5 years ago

megver says he did not see this problem because his chroots use openrc/nonsystemd - apparently the read-only mount only relates to systemd-nspawn - dunno if that is a bug or a feature

#20

Updated by Megver83 almost 5 years ago

bill-auger wrote:

megver says he did not see this problem because his chroots use openrc/nonsystemd - apparently the read-only mount only relates to systemd-nspawn - dunno if that is a bug or a feature

I'm using https://www.parabola.nu/packages/nonsystemd/x86_64/libretools/ to build packages. Yesterday I built linux-libre-hardened and -pae successfully without having to do any extra arrangements. So please, do not (yet) do things like this (or at least with kernels)

#21

Updated by bill-auger almost 5 years ago

i made that change mainly to make the point clear that it served no purpose - it was a code smell - the only effect of reverting that change now, would be to make the package not able to build with the standard tools - anything like that is an indication that something is not well thought out and should be discussed

maybe that is how arch does it, but the goal of staying close to arch, does not mean we need to do any useless things they do - the change that broke these packages was already a deviation from arch, and a more drastic one - this change is completely benign - even if we reverted the change that made the /startdir read-only as gnutoo suggested, there would still be no justification for reverting this change - it is more of a curiosity why that ugly hack was there in the first place

#22

Updated by eschwartz almost 5 years ago

It's not an ugly hack. It is a curious appendage designed to be useful in the sole case where some user wants to create a custom kernel.

Your "(NOTE: hard-coded into linux.install)" is quite missing the point of why any of that exists in the libre/linux-libre or upstream core/linux PKGBUILDs at all -- it is now no longer possible to simply change the pkgbase and generate a forked custom kernel, so you should either revert the change, or do one better and strip out the rest of the workarounds intended solely for custom kernel derivations.

#23

Updated by bill-auger almost 5 years ago

i wonder if thats why i could not publish the first kernels i built - i originally did change the pkgbase intentionally, so that i would not replace any existing kernels; but the packages were rejected with "this package was not built in a chroot"

maybe that was not related, im not sure - it was confusing and i was in a rush, and i tried several crazy hacks to get it out; but that was the original reason i pushed the changes i made to the PKGBUILDs

#24

Updated by bill-auger almost 5 years ago

i noticed that libremakepkg has options to control mounting ro or
rw - maybe that was intended to be the solution all along

-w <PATH[:INSIDE_PATH[:OPTIONS]]>     Bind mount a file or
directory, read/write
-r <PATH[:INSIDE_PATH[:OPTIONS]]>     Bind mount a file or
directory, read-only
#25

Updated by lukeshu almost 5 years ago

There is a real issue where pkgver() isn't getting evaluated before the sourceball is created, which then causes problems with the read-only $startdir. But: the PKGBUILD should be doing it's work in $srcdir, not $startdir.

However, I would tend to be of the opinion that "work" should happen in $srcdir. Would using an absolute path to the modified script not work?

  sed "$subst" "$startdir/$install" > "$srcdir/$install.pkg" 
  true && install=$srcdir/$install.pkg
#26

Updated by bill-auger almost 5 years ago

no i tried that - see eli's comment # 14 https://labs.parabola.nu/issues/2141#note-14

#27

Updated by bill-auger over 4 years ago

gnutoo reverted the change that originally spurred this BR with a temporary patch in abslibre

https://git.parabola.nu/abslibre.git/tree/libre/libretools/0001-libremakepkg-rw-startdir.patch

Several packages require a read-write startdir:
- Some packages have a pkgver that is computed dynamically through a pkgver function. This is the case for many packages using git repositories. At the end of the package build, the pkgver is automatically updated in the PKGBUILD, however, without that fix that fails with libremakepkg as the PKGBUILD was set read-only.
- Some packages like linux-libre are modifying the install= script. This is done by creating a temporary install script in the startdir that is then modified with sed. Once this is done that install script is then dynamically selected. As this also require to have read-write access to the startdir to be read-write it fails to build the package if it's not the case.

In both cases it's possible to modify the PKGBUILDs to workaround the issue, however the Arch Linux distribution has a read-write startdir, and modifying each affected packages would significatively increase the cost (in time and efforts) of maintaining Parabola.

there has not been any problems with that yet, although lukeshu indicated that problem that the change was intended to solve will turn up someday - maybe this issue is fixed? - only time will tell :)

#28

Updated by bill-auger almost 4 years ago

  • Parent task set to #2403

if the problem that this change was intended to solve is related to the sourceball, it is possible that the proposal for fixing/eradicating mksource() (rolling the sourceball after prepare()) may also solve that problem - if it does, then lukeshu's patch could be re-introduced, and `libremakepkg -w` could be used for packages which need it

AFAIK, linux-libre is the only such package, and i offered a simple way to avoid that; so gnutoo's concern about the maintenance penalty is probably negligable

the libretools commit which introduced r/o /startdir and /srcdest:
https://git.parabola.nu/packages/libretools.git/commit/?id=646ac0258c3295943778142468aadfe5b04ad6d1

the linux-libre adaptation:
https://git.parabola.nu/abslibre.git/commit/?id=98431c0f2cedd313389b7120917643d825381ba7

OTOH, if the original motivation was only to accommodate PKGBUILDs with `date` in them, i would be more in favor of simply not doing that - i would also suggest to modify all packages built from VCS to use the checkout ID (git tag, etc) as the pkgver explicitly, whenever a versioned release is not available, eliminating the need for pkgver() - rather than accommodating lazy versioning, i would prefer to remove pkgver() from all PKGBUILDs - i find pkgver() to be a clumsy work-around for a sloppy maintenance habit; and it can be avoided in all cases, by checking out the explicit checkpoint in prepare(), with the very unlikely exception, when that commit has been deleted upstream - the most common forges (eg: github) will gladly generate a source-ball from any commit, using this form:

pkgver='XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
source=( ${pkgname}-${pkgver}.tar.gz::https://github.com/NAMESPACE/${pkgname}/archive/${pkgver}.tar.gz )

that avoids the `date` problem; and also avoids inadvertently rebuilding a package from possibly different sources, even without obtaining the parabola source package, which we still do not actually publish - currently, checking out the explicit checkpoint in prepare() would not affect the source package though - that is the problem which #2403 addresses

the original motivation:
https://lists.parabola.nu/pipermail/dev/2019-June/007217.html

#29

Updated by lukeshu 2 months ago

A few things

  • The rationale for mksource() being separate from prepare() is that sourceballs should never contain nonfree code--if they do, then we're distributing nonfree code, which FSDG says we're not allowed to do (even if our PKGBUILD build script immediately deletes it).
  • I generally think that it's a bug if a package needs to write to $startdir. In every instance I've seen, it either makes it so that the same package can't be rebuilt from the sourceball we publish (bad!), or is accommodating using date instead of git tags/hashes. That said, I recognize that we only have finite hours in the day, and that it may not be feasible to fix all such packages.

So, in v20240221, I've adjusted it to mount $startdir read-only by default, but added a -W flag as an escape hatch, the same as we have the -N escape hatch for networking. https://git.parabola.nu/packages/libretools.git/commit/?id=63d3993a320ee03c20da05d0e04ddbd3cc800335

#30

Updated by lukeshu 2 months ago

  • Status changed from confirmed to fixed

Closing as "fixed", but feel free to re-open if you have an example of a PKGBUILD that you feel is justified in writing to $startdir.

#31

Updated by bill-auger 2 months ago

thats a fine solution - as i remember the original debate, there was only one package that used a trick like that - it was linux-libre IIRC and IIRC it does not do that anymore - so since some years ago, AFAIK there are not PKGBUILDs which would make use of a writable startdir anyways - so the CLI option is great (for emergency use only)

just as a side note, it is a similar/parallel situation to the -N option - AFAIK, the keyring package was the only one which required the -N option; and i changed that - so it is also reserved now only for emergency use

Also available in: Atom PDF