Feature Request #2403

[libremakepkg]: eradicate mksource() from abslibre

bill-auger - 22 days ago - . Updated 16 days ago.

info needed
% Done:



related to forum thread

IIRC when discussing the 'ruby' package, we concluded that it was probably intended per the FSDG, to provide complete and pre-cleaned source-balls to the distro users; which entails the devs to keep some of the libre-fications tools private

the FSDG does not require support for versions of 'foo' that were not included in any distro release however; but OTOH, this PKGBUILD contains instructions for downloading known non-free software

on the third hand (the philosophical hand), the 'gist' of the four freedoms is not about possessing or sharing non-free software; but that the user is able to do with it as they see fit - if the only thing that one wishes to do with that software is to merely posses it or share it, then anything redistributable would satisfy the 'gist' of the four freedoms for that person

probably the simplest way to resolve this class of issue, is to add a note to the PKGBULID with sed above every instance of 'mksource()', asking the reader to ignore it and ask a parabola dev to publish the intermediate source-ball, if they are building ahead of the current release; then to publish the missing source-ball for any enthusiastic user who asks for one - a better solution to this class of issue, would be for the PKGBUILD to explaining how to run mksource() and have the mksource() function create the source-ball with expected libre name in the sources() array - if that does not conflict with the FSDG - an even better solution to this class of issue, is to eliminate it by eradicating mksource() from abslibre somehow, even if that is simply moving the PKGBUILD to a private git branch

this incident is not a peculiar though - it is currently systematic AFAIK; so i consulted the FSDG:

"A free system distribution must not steer users towards obtaining any nonfree information
for practical use, or encourage them to do so. The system should have no repositories
for nonfree software and no specific recipes for installation of particular nonfree programs."

there is another clause that might suggest that this use case is within the guidelines; although it is in the "Documentation" section

"In general, something that helps people who already use nonfree software
to use the free software better with it is acceptable,
but something that encourages users of the free software to install nonfree software is not."

libre-fications would seem to fit into that documentation exception - id say the key point is that such scripts do not install the program "for practical use" - the use-case in which the user is interested, is to download a non-free tarball, and to liberate it, without viewing or executing any of the files within it; for the sole purpose of using the cleaned sources in freedom - it would seems better to instruct people how to liberate non-free software rather than impeding their curiosity or ambition; though it may snag upon some FSDG caveat

some options:

  • should consult the FSDG mailing list about this grey area
  • put these on the wiki as documentation, instead of in PKGBUILDs
  • libretools could be made to download the file into memory instead of writing a hard copy
  • hide all such scripts at the cost of impeding enthusiastic users who want to help

Related issues

Related to Packages - Freedom issue #1035: [your-system-sanity]: Non-Free Software From Third-party Package Managersin progress

Related to libretools - Housekeeping #1973: [libremakepkg] Should extract the sourceball into the chroot, instead of a tmpdiropen




Updated by bill-auger 22 days ago

  • Description updated (diff)

Updated by bill-auger 22 days ago

  • Description updated (diff)

Updated by bill-auger 22 days ago

  • Description updated (diff)

Updated by bill-auger 22 days ago

  • Description updated (diff)

Updated by freemor 22 days ago

I agree this should be automatic instead of the semi clunky mksource()

The way I see it the issue is with Parabola trying to provide a source tarball
as that forces us to make a libre one as we can not provide the non-libre one.

The need to provide a libre tarball is unclear, if not false on its face as
building "version next" of something will always require pulling the non-free

It might be better to forget about source tarballs. PKGBUILDS pull the source
they need. The PKGBUILDS in abslibre will always clean that source making it

As bill-auger pointed out above PKGBUILDS in abslibre in no way induce people
to install non-free. The do the opposite and provide the path to freedom.

Providing the "How" (as in how to clean that source) is always more instructive
then just giving someone pre-cleaned source.

Must people that will be playing at the show me the source/pkgbuild level will
probably like myself see the libre package as an unnecessary hustle that I
never really have to deal with as all the work in liberating the source would
just move from prepare to mksource() but every version I'd end up pulling the
non-free, figuring out what needs fixing, and adjusting mksource. rolling a
tarball that no one will ever use, then building from the libre tarball instead
of just doing in all in prepare() and forgetting about creating a next to
unused libre source


Updated by bill-auger 22 days ago

  • Description updated (diff)

Updated by theova 20 days ago

As an curious user interested in building/updating packages I support Freemor's point of view.

Altough I see the question of letting users download non-free software, I think it is important that the download is not "for practical use" since it is dirrectly removed afterwards.

I'm strongly opposed to making a private git branch. This would hide the work/information to liberate software and makes it only available for a selected circle.

"All information for practical use in a free distribution must be available in source form.
(“Source” means the form of the information that is preferred for making changes to it.)" [FSDG]

For me, updating packages is a very "practical use" and the source to do that are the PKGBUILDs. So the entire PKGBUILDs should be free (and available).

Other question: If the mksource() step is automated, what is then the benefit over doing the steps in prepare()?


Updated by bill-auger 19 days ago

For me, updating packages is a very "practical use" 

updating packages is not something that the distro user should need to do - the very reason why distros exist, is to provide curated software to the user that is ready to use OOTB - i see "practical use" in terms of the FSDG as very literally referring to "using" the program for its intended end-user use-case - liberating it, so that the user can use it freely "in practice", is something more "meta"; and the distro user should not need to do that - im quite certain that we should be hosting only pre-cleaned sources with all four freedoms already present in every byte - i think this is is an over-sight in the tooling; but that is a different issue than what the PKGBUILD downloads from external sources - i think we could make that work transparently with the end-result being a FSDG-compliant source-ball

If the mksource() step is automated, what is then the benefit over doing the steps in prepare()?

probably, if the mksource() step was automated, it would all be done in prepare() - the main difference is that the prepare() function often has to treat the sources differently depending on the arch

freemor verified today that the source-ball does get created before prepare() fires - that would explain the extra, non--standard, manual mksource() step - that is apparently the only way to publish pre-cleaned source-balls currently - in that case it is curious why there are so few PKGBUILDs that have mksource() - it seems to me that everything in [libre] would deserve that same treatment

ive been thinking we should make some changes to libremakpkg anyways - aside from the sources being pre-cleaned, ive noticed that any source-balls made from VCS sources end up with the entire VCS history inside; which makes them much much larger than necessary - theyre not versioned release tarballs at all, they are git-balls; which would still include the non-free bits even if they were made after the prepare() step


Updated by freemor 18 days ago

Hmmm.. perhaps we are looking at this the wrong way.
The purpose of makepkg/libremakepkg is to build binaries from source. not to provide curated source.
We have all the patches. We know where the source lives.

what if we rolled a separate tool (libresource or some other nice name) That fetched the PKGBUILD and patches from abslibre and then just ran the equivalent of (or maybe actually) makepkg -o. resulting in a directory with the cleaned source.


Makes management of libre source a separate step from makepkg/libremakepkg
It creates a unified easy too for anyone wanting to see the clean source: libresource libre/iceweasel
It is not complicated to build with standard tools (git, makepkg)
It (might maybe hopefully) could eliminate Parabola needing to store massive source tarballs
It feels like a better fit for the Arch way of doing things.
The only retooling of libremakepkg would be to remove the rolling/uploading of source.



Updated by bill-auger 17 days ago

are you suggesting not to host any sources, or to continue
hosting the pristine upstream sources warts and all? - that is
the most important concern, regardless of what the tools do

probably the original intention was that users should never need
to download known non-free software; and to avoid publishing any
tools that automatically do so - on the one hand, that would be
even worse than what TPPMs do in terms of suggesting and leading
users toward non-free software - on the other hand, i think we
agree that this particular use-case is not "for practical use" ;
so it may not be a strict FSDG requirement - if the source-balls
are not clean though, as must be the case if they are created
before prepare() runs, then PKGBUILD with the mksource() function
are the only ones that produce pre-cleaned sources - everything
in [libre] should have had that same treatment; and perhaps that
was the goal - if nothing else in [libre] gets that treatment,
then those few are just a drop in the bucket; and moving the
mksource() procedure into prepare() for those few package which
have it, would be barely significant to the big picture - that
would be the easiest way to resolve the present issue; but its
not a very satisfying solution

if the suggestion was not to host any sources, the biggest
concern i have with that is the obvious one - im not sure if the
FSDG requires that; but it does require that we ensure all source
code to available for the current release; and the only way to
ensure that is to self-host it - that would make source code
storage to be essential infrastructure; and it is always unwise
to rely on third-parties to maintain your essential
infrastructure , especially with no contract - regardless of the
FSDG, the GPL alone extends that requrement to three years
beyond the current release, for GPL programs - the only
third-party code host that i would be comfortale using for that
purpose would be the software heritage repos - i think they are
a legitimate charity, so they do have some obligation to the
public - that can not be said of most individual projects or any
commercial freebie hosts; but it would be a lot of work to
change every PKGBUILD to fetch from that host - also, i dont
think they prioritize tracking the bleeding edge of changes; so
that cuod be a no-starter for parabola

aside from ensuring that sources are somehow available, there is
the FSDG "self-hosting" requirement; according to which, one
should be able to compile vN+1 of the distro using only vN of
the distro and tools that are provided by the distro - i dont
think that ubuquitous networking should be taken for granted as
an excuse to avoid supplying everything required to build the
distro - software curation is the one defining factor of what a
distro is and what distro devs do - ideally, i would like to
take that a step further and publish a sources ISO for
offline/clean-room use - currently, a user would need to do an
unreasonable amount of prep work, if they wanted to accomplish
that; but we already have most of the tooling to make that a

im pretty sure that is why parabola mirrors everything from
arch, rather than having users rely solely on the arch repos -
some users are very concerned about to which servers they expose
their IP - i would like to take that concern a step further and
modify the redirector to always use a parabola mirror - if
winston had a gigbit pipe with unlimited traffic, we could
remove the redirector entirely - the redirector is a necessary
compromise due to circumstances, in a similar way as relying on
the upstreams to feed the user facing PKGBUILDs; but we do have
the resources to avoid the latter because of the lesser damand
for source-ball

i think that complete self-reliance was a goal of the project
that is still incomplete; so if we need to modify libretools, i
would rather continue in that direction rather than removing
relevent functionality - i like the idea of downloading from the
upstream and running prepare() in memory, then creating the
source-ball after prepare() has done its work; resulting in a
cleaned tarball on the local filesystem - librestage and
librerelease already are expecting that libremakepkg has created
one - the only real problem with the current workflow is that the
source-balls are not clean and the VCS ones are overweight - the
latter can be solved simply by rm -rf the VCS data dir
(.svn/, .git/, .hg/)

an alternative idea, which i like better, would be to put those
pre-cleaned source-balls into use by modifying makepkg to
recognize, according to $pkgname-$pkgver-$pkgrel, that a
pre-cleaned source-ball exists on the parabola server and to
pull that instead of the upstream sources, bypassing the
prepare() function - if no pre-cleaned source-ball exists for
that version, then that is unsupported software, just like AUR,
and falling back on the upstream sources is the reasonable thing
to do - as with the AUR, we could suggest against doing so "for
practical use"; but endorese it only for the use-case of helping
parabola devs liberate it "for practical use" by others


Updated by eschwartz 17 days ago

The concept of "libre" tarballs is essentially the same concept as Debian's "orig" tarballs, where an upstream source release artifact is taken, but the distro decides it isn't "good enough" and determines to apply certain changes. In debian's case, that sometimes includes deleting lots of files that maybe aren't used by the package, for example if software contains a vendored copy of a library, with a configure switch to either use the vendored copy or a system copy... Debian might delete that. Debian might delete minified js from the source tarball. Debian might rename the directory it is stored in, Debian might just recompress the tarball so its checksums don't match but it takes less space on their mirror network.

The question is why this is necessary. Debian certainly seems to think it is so, but then, Debian hosts the sources for every package, no matter what. They also seem to be going to some lengths merely for the sake of prettiness.

Does the DFSG truly require you to not make noncompliant source code available at all? If all you are doing is providing a persistent mirror of the upstream tarball, but the canonical "help" that Parabola offers, guiding people towards the act of building and installing the software, is the PKGBUILD, then the PKGBUILD -- regardless of the state of the sources themselves -- does not encourage using non-DFSG software. The PKGBUILD will, when left to its own devices, always build a DFSG-compliant binary product, and extracting the build information but removing the DFSG bits is not a supported or recommended route.

From my outside perspective, it would seem like Parabola should have the option to just upload mirrors of the tarball, but have the PKGBUILD perform all DFSG compliance modifications in prepare() -- with suitable comments saying "# these changes, in accordance with the DFSG, protect the user by removing nonfree software".

The alternative is to make it difficult to use the PKGBUILD at all, causing users who want to use the PKGBUILD as a base for updating the version to give up and download something from Github or another upstream website and run ./configure; make; sudo make install without the benefit of a PKGBUILD or DFSG modifications.


Updated by bill-auger 17 days ago

  • Related to Freedom issue #1035: [your-system-sanity]: Non-Free Software From Third-party Package Managers added

Updated by bill-auger 16 days ago

  • Related to Housekeeping #1973: [libremakepkg] Should extract the sourceball into the chroot, instead of a tmpdir added

Updated by bill-auger 16 days ago

i just found a open issue (#1973) that suggests unpacking sources into memory that could tie into this issue

Also available in: Atom PDF