Project

General

Profile

Proposing a big change about packages for PCR

GNUtoo - 9 months ago -

Hi,

In the Parabola Wiki the Parabola developers (me included) are currently telling Parabola users who are not (yet) Parabola developers that they can open bug reports for adding new packages to Parabola1. As I understand not a lot of packages were added in this way.

In addition many packages are often outdated in general, and if we filter out the less important bugs in the bug trackers, some important bugs remain there for years, and this includes bugs about licensing or freedom issues.

And work is being done too, packages are being updated too and so on, but there aren't enough developers involved.

So I propose to change the policy about requesting packages for Parabola as it doesn't seem to work and instead clarify how it works in practice.

Users that are not yet Parabola developers would now have to send patches for adding the packages. In the long run this could bring in more Parabola developers.

The nice thing is that Parabola packages are relatively easy to work with: they consist of some functions like build() packages() with contains commands to build the software, along with some variables [2], for instance to set the license.

What is more complicated is to properly review software when adding it. People typically at least look at licenses and dependencies and so on but experience also helps knowing where to look for potentially nonfree software bundled in free software projects. Here as existing developers would have to review patches anyway, people submitting patches and other developers reading the mailing list could then learn that through the patch review process.

[1]https://wiki.parabola.nu/How_to_contribute_to_Parabola%27s_Repositories#Adding_new_packages_to_.5Bpcr.5D
[2]https://wiki.parabola.nu/PKGBUILD

Denis.


Replies (10)

RE: Proposing a big change about packages for PCR - nona - 9 months ago -

I usually create packages for the AUR, because it is very easy to just keep my own packages instead of downloading the whole database. In the past, I have had difficulties to create the infrastructure which is required to become a package manager (and created bug reports based on that). With the little time that we all have, I can only come back to start the process again several months later.

My suggestion is to (1) allow a repository which permits partial checkouts (as it is done with AUR packages), (2) to make sure that the steps which are shown on the Wiki can actually be followed and (3) to have some basic scripts to automatically sift some parts of the libre-verification process.

Also, it would be interesting to check the repositories from Trisquel and see if we can incorporate packages that they have and are not in Parabola yet. (At least for inspiration, e.g. Gmsh).

RE: Proposing a big change about packages for PCR - gap - 8 months ago -

As a user and potential future packager, I have to say that I am impressed by the sheer amount of software included in Parabola's repos in general and especially the PCR. It's the most on any system repo I've ever used, and I haven't even considered using a third-party repo (and installing potentially hidden or mislabelled proprietary software) for installing user software.

As for development libraries, the long-standing TPPM issue necessitates that I be vigilant and take responsibility when using language-specific PMs, although that's not something non-programmer users have to worry about.

For the PCR, I think removing some of the ancient and deprecated packages is a wise move, although packaging requests should be left intact, because it means users can use the native PCR instead of resorting to non-FSDG-compliant repos like the AUR, which are a slippery slope towards losing freedom. Such requests are also a valuable opportunity to thank people for getting involved, and potentially teaching them how to audit and package new programs.

I agree we need a comprehensive auditing guide for packages, both new, and already in the Parabola repos. Furthermore, we need to fix the root cause of this issue by reaching out to projects to care about freedom in order to reduce the frequency and severity of freedom issues, by not including nonfree components, or otherwise making the package useless in the free world, in the first place.

nona's idea for semi-automated libre verification is brilliant; perhaps someone could write a script integrating existing tools like f"os"sology?

RE: Proposing a big change about packages for PCR - bill-auger - 8 months ago -

every git repo permits checkout of single files or directories, via the `git sparse-checkout` command - that is how the proposed octopi enhancement would work - the arch PKGBUILD repos on github, make that one step simpler; by keeping each package on separate individual branches

nona's proposal would be a natural extension to that (something which has also been proposed several times in the past), a 'PUR' PKGBUILD repo (the parabola analog to AUR), but maintained by parabola users - i love the idea; but parabola team members can not endorse it nor support it directly - to do so would be the equivalent of vetting and maintaining the packages in PCR, but without compiling them - compiling them is not the time-consuming part though - 99% of the work is in the initial licensing audit, responding to bug reports and user support questions, and maintaining the build recipes for each new version

however, there are no automated "libre-verification" scripts; and we should not want any - that should always be done manually - unfortunately, that task requires a non-trivial (but possible for anyone) amount of licensing education - that training/on-boarding alone, would put a huge demand on the parabola team - it would simply not be worth the time, to train someone who is only willing to maintain a small number of packages - i have been asked to write a "how to audit software licensing" wiki article - even that would be a daunting task to do one time, and incomplete at best - we would want a permanent, well-trained community team, willing to work closely with the FSF licensing team (presuming that the FSF is willing and able to handle the additional querries), to vet all incoming packages, on-board new contributors, and maintain the PUR repo

WRT trisquel, it is not possible to use anything from trisquel directly - some of the LOC in some of the "build helpers" (as they are called) could be used indirectly (manually, tediously); but generally, debian packaging is 1% configuration and 99% magic (generally, no part of the build is scripted imperatively); where arch packaging is 1% configuration, 0% magic, and 99% imperative scripting

besides that, trisquel does not need to liberate very many packages (about an order of magnitude fewer) - that is because debian already separates libre from non-free (mostly in accordance with the FSDG) - arch OTOH, has no such policy; so parabola has many more packages to filter and/or liberate - also, the PCR is mostly packages requested by users - i do not believe that trisquel does anything like that; but only has packages (or liberated versions of them) which are also in debian - therefore, there are probably very few trisquel build helpers, for anything which parabola is missing - and lastly, (if that were not enough), even if a few such packages were found, the versions of trisquel packages are years behind parabola - its not unimaginable; but i dont suppose there are many low-hanging ripe fruits to be plucked there

RE: Proposing a big change about packages for PCR - bill-auger - 8 months ago -

in response directly to GNUtoo

they can open bug reports for adding new packages to Parabola1. As I understand not a lot of packages were added in this way.

AFAIK, nearly all PCR packages were added that way, or something similar - some user asked for it, then a parabola dev audited it (hopefully), wrote the PKGBUILD (or adapted one from elsewhere), and adopted the package (ie: added it to parabola's workload perpetually) - if users were expected to maintain all PKGBUILDs in PCR, there would be none now

there is exactly one package which is actively maintained by a parabola user - that is iceweasel - still, it takes me a significant amount of time, just to review the changes, and build the packages - usually, at least one of the arches does not build OOTB, and i spend more time fixing the PKBGUILD for that arch - o/c mozilla is a horrible example; and we are dammed lucky to have a parabola user maintaining it, or helping to any degree - but that example underlines the general concern - every new package comes with some non-zero cost to the parabola team, even if someone else is effectively maintaining it

how about this (sarcastic) counter proposal: for every new package which enters PCR; one existing package must be ejected - that would keep the workload a constant - packages are rarely removed, once entering the system - they may grow stale, but i have not seen any removed "just cause"

in practice, this proposal is no different than asking parabola users to put PKGBUILDs on the AUR, then ask on the mailing list for someone to pull the changes and build them routinely - in that case, it make no difference if the parabola user actually is the maintainer - the request to pull changes and rebuild, is no more helpful than flagging a PCR package "out-of-date" - as long as someone is maintaining an AUR package for that program, it is an equivalent amount of work for parabola - simply being handed a PKGBUILD (or notified that an upgarded one exists) is an insignificant portion of the overall workload - so in practice, there is very little difference in expecting each packaging request to be accompanied by a working PKGBUILD - in 99% of the cases, a working PKGBUILD exists on the AUR, ripe for the plucking - that is not the reason for the back-log - the reason for the back-log is that we can not trust that the AUR packager has done a thorough FSDG-fit job - but we can not trust that any rando on the parabola bug tracker has done so either

the desire to clean-up the bug tracker is a good one; but "packaging requests" are categorized as such - they are not "bug reports" - they can be filtered and sorted independently - "bug reports" against existing packages should be treated with a much higher priority - "packaging requests" are best considered as low-priority, and handled cautiously and very conservatively; because in reality, each one amounts to a negative benefit to parabola, by adding an extra workload burden, which was not present before - given the current amount of human resources parabola has available, each one must have a very high desirability/work-load ratio, and most programs do not, because parabola already has a tool for (almost) every job

i am leaning toward the plan to eliminate PCR - to let users maintain PKGBUILDs like the AUR, and to give AUR-helper-like functionality to octopi; so that users can easily compile their own whatever, from where-ever, maintained by whoever - that would, in one fell-swoop, reduce the workload on parabola, while also reducing the size of the parabola repos greatly, while also greatly increasing the amount of easily upgradable packages available (though only for those users, brave enough to venture outside parabola for recipes) - the only PKGBUILDs which the AUR-helper could access by default, are those already present in PCR now; and those would still need to be maintained and supported; so the workload reduction is not necessarily very much

the only manageable way to accelerate or proliferate new package adoption, is to get a few "trusted users" well-trained, to join the community team, to bear that extra workload, and to train others to do the same; and most importantly, to convince new contributors to maintain those new packages for the long term

if that happened, it would reduce the workload on parabola, quite a lot more than eliminating the PCR would - more-so if both happened - the plan to eliminate PCR would require exactly the same dedicated community team, and dedicated contributors, or else there would be no user support for those packages - from the user perspective, the only practical difference without PCR, would be no more binary packages - each user would need to build the PCR things with octopi or makepkg

RE: Proposing a big change about packages for PCR - bill-auger - 7 months ago -

i just re-read my comments; and realized that they tend not to favor the general suggestion - that was not intentional - i was mainly trying to diminish the urgency or severity of the proposal, not to dismiss it - in reality, i never liked getting packaging requests on the bug tracker - it really should not be allowed - they were always accepted on either mailing list too which is not good for reference/searching - i was further dismayed when this "New Software Suggestions" category of this forum was created - that spreads packaging requests across three places

if we are proposing to stop accepting packaging requests on the bug tracker, i am in favor of that

bug trackers should be for tracking work being done, not for proposing or discussing arbitrary new ideas that are possible - that is what the mailing list is for - as gnutoo points out, those packaging requests rarely get adopted - rather, it would be much better to have a single location/interface to add and list all packaging requests, even if that were one "sticky' ticket, which remains open forever - many people do seem to prefer the web over email; so alternatively we could direct packaging requests to pagure, which may encourage pull requests, even if those live forever as a WIP - that would be better than cluttering the main bug tracker with them

or most simply, direct packaging requests to this forum "New Software Suggestions" category - though i think it would be ideal to have some voting mechanism for these, so they could be prioritized in some meaningful order

RE: Proposing a big change about packages for PCR - bill-auger - 7 months ago -

TLDR: if we are proposing to stop accepting packaging requests on the bug tracker, i am in favor of that

most simply, direct packaging requests to this forum "New Software Suggestions" category - though i think it would be ideal to have some voting mechanism for these, so they could be prioritized in some meaningful order

RE: Proposing a big change about packages for PCR - Zuss - 7 months ago -

I was talking with bill-auger in the IRC channel about updating the guidelines to update/submit packages.
I do agree that packaging requests should be put onto the forum instead of putting them on the tracker, which is meant for issues & bugs. As mentioned earlier, they also don't have a high priority either.

The updated wiki page can be seen at: https://wiki.parabola.nu/How_to_contribute_to_Parabola's_Repositories

Main changes:
  • All new package requests and submissions should go into the "New Software Suggestions" category
  • Updating/patching packages should be done via a pull request on the pagure repo

The goal was to make it more clear on how to submit/request packages in the future, both for new and existing users.
Additional input would be nice as it was only bill-auger and me discussing the changes at the time.

RE: Proposing a big change about packages for PCR - gap - 7 months ago -

Shouldn't we do some sort of vote before adopting a proposal?
Has this been agreed-upon already?
I feel this decision-making process is not transparent if it has already gone ahead despite only being discussed by a couple of contributors on an ephemeral IRC chat and then not being communicated to anyone else.
In contrast, I have made several proposals on the forums which have been lightly discussed, and I wouldn't expect them to be adopted before going through a rigorous review and discussion process and then being communicated widely on the mailing lists in advance for example.

As for the packaging requests, I think this change is well intentioned, but poorly planned.
How is this going to be implemented?
Are we going to delete existing tickets or move them all to individual forum threads?
Who has authority to open a genuine package request tracking issue to document the progress of adding an agreed-upon package by a proposed voting mechanism?
How will the voting mechanism work?
This also leads into the next issue, namely: I didn't know we had a Pragure repo.
This isn't integrated or linked to on our homepage AFAICT.
The page says the repo was started two years ago.
Are there any other Parabola projects hosted in repos on external infrastructure that I don't know about?
If this is going ahead, it hasn't been communicated on the issue tracker or forum, but ephemeral IRC chats.

Moreover, I don't think hosting our infrastructure somewhere else and being reliant on the operator of that infrastructure is a particularly good idea.
We would lose a lot of control over it versus self-hosting.
I proposed we adopt a self-hosted instance of SourceHut on this thread: https://labs.parabola.nu/boards/6/topics/1297 on which I give many reasons I will not repeat here as to why I think self-hosting SourceHut is the best option for us.

SourceHut is also built from the ground up around the workflow this proposal intends to switch to, ie. publically discussing issues and then having a senior project member move a precise plan over to the issue tracker as a TODO item instead of having the web-based GitHub-inspired system of where anything and everything is discussed and agreed-upon via issue report threads themselves.
Better yet, all chats are done via email on mailing lists, meaning proposals will reach everyone and are not ephemeral.
In every way I can think of, SourceHut is literally purpose-built for use-cases like ours.

Prague is just yet another unpopular GitHub clone, not to mention that https://pragure.io violates the GNU FSDG by being branded "Fedora", a nonfree GNU+Linux distro.
The homepage is also labelled "Open Source".
To fix these issues, we would have to either beg Red Hat to rebrand Pragure upstream and then host the new rebranded version, or self-host our own cleaned-up version.
We would be totally at the mercy of Red Hat, who might just ignore us.
In other words, relying on someone else's website versus self-hosting a libre source forge program puts us in a similar disposition as running proprietary software; we are at the mercy of someone else who has power we don't, over the software we are using.
Of course, this is not unethical because Red Hat runs their own copy of a libre program on their own infrastructure and has every right to do what they want with it, and if we opt-in to using it, we would just have to go along whatever Red Hat wants to do.
This is a real example of how not being in control of our own infrastructure negatively impacts us.

At the end of the day, I'm happy to contribute to Parabola and engage with the community via any infrastructure so long as it is at a bare minimum freedom-respecting on the user's end, but I do not think this current plan is the way to go.

RE: Proposing a big change about packages for PCR - Zuss - 7 months ago -

The reason the package requests section was updated was to reflect what bill-auger had discussed. It is still open for discussion and isn't the end all because there is still points to iron out just as you listed.

As of right now there are 211 packaging requests on the issue tracker with ~50 of them being active in the past year. We could move them to the forum, although I would be in favour of starting from a clean slate.
The voting mechanism would be a nice feature, implementing it is another task. Perhaps something simple like adding a voting counter in which the users can cast a vote within the package request thread itself.

The Pagure repo was briefly mentioned in the How to contribute to Parabola's Repositories wiki page, that could have been easily missed. Though there would be no harm including it on the main page.
How would using Pagure violate the FSDG? It is setup by the Fedora Project. Which I don't think is considered as the distribution itself. More information about the project itself can be found here: https://docs.fedoraproject.org/en-US/project/
I agree that Parabola should have its own selfhosted infrastructure and it does seem to be in the works.
In fairness, it does take quite some time and resources to setup new things, which the devs may not have a lot of room for.
However I am interested in implementing things that can be done right now. Utilizing existing infrastructure to make it more clear for people on how to contribute to Parabola's repos, that was the main goal of updating the wiki page.
People need to have something they can work with while they wait for these big changes to happen, which may take some time.

RE: Proposing a big change about packages for PCR - bill-auger - 7 months ago -

there is a lot of confusion on this thread

pagure was never suggested because there was no general
agreement about it - i am the only one who moderates it closely
- i have cloned many of parabola repos to pagure over the years
as backups - some people wanted to use a web forge to send
patches to abslibre; so i told those people they could use the
pagure repo, because it was already there anyways - once people
started using it, i made it to send all activity to parabola's
mailing list - that way, none of the parabola devs would actually
need to use it - in effect, contributions to the pagure repo are
as if they were sent to the mailing list

it is not important for it to be "official" or whatever - people
could send me patches on floppy disks by snail mail, and they
would be just as valuable - its just another way to get
contributions into parabola; so it really did not require any
discussion or agreement

OTOH, packaging requests on the bug tracker, even if they
include complete working PKGBUILDs, do not make it to the mailing
list; so they are not so visible - so it is really not a "big
change" to send packaging requests to the forum instead - it is
actually the very same software - both the forum and the bug
tracker are the same running instance of redmine - redmine can
also be a git interface, and many other things - those features
were never enabled though

i did not expect so many people would start using the pagure
repo, because i only told a small few about it; but it was always
in mind that parabola could host an instance, if enough people
wanted to use a web forge - it was never intended to be a
primary tool, or "official" in any way, unless it was hosted by
parabola - it is something of an unwritten rule, that parabola
should be entirely self-sufficient

i would definitely add to the wiki that pagure is not the only
way to send patches - the primary way to send patches is the dev
mailing list; and it always has been - the mailing list is the
best way to communicate with the dev team; and im sure that it
always will be - the bug tracker, the forum, IRC, pagure, or any
of the others are not the primary communication channels - all
of those things are supplements, for people who prefer not to
use email for some reason

sourcehut specifically, would not add anything to parabola; which
it does not already have - parabola already has mailing lists for
development discussions and patches, a web forum and another
mailing list for community discussions, a bug tracker for bugs,
patches, and work-in-progress, and the IRC channel for anything
else - thats not to mention the other things people have setup
over the years, the reddit forum, the gnu-social forum, the
matrix channel

the only reason to have any web forge, is to accommodate people
who are more comfortable with the github-like workflow - people
prefer pagure, git-tea, and gitlab, precisely because those are
github-like - i think that anyone who would be content with
sourcehut, would be content with the tools parabola already
uses

if parabola needs a web forge too, pagure would be the best
option, by far - for one thing, the software is GPL-licensed -
also, the system was designed to be fully decentralized, using
only git - no other forge comes close to that level of
libre-friendliness

    (1-10/10)