Why are incomplete rebuilds pushed to stable (perl, python3.7)?

rogerthat · 2018-08-27 16:36:19

Thank you so much for the comprehensive reply! You've helped me understand better...

levi wrote:

I think the glib problems might have been resolved, but I need to refamiliarise myself with that issue.

Yes, actually the only current issue I still had during my last upgrade attempt was the python version causing some libraries to break, and that broke the rhythmbox application.

levi wrote:

It's always safe to run pacman -Syu with only the one -y - it'll check for strict dependency troubles. Scripting libs version misses are noted but can only be warned about once you've actually committed those to your file system. You'll need to make a call about whether you can live without some of those libs for the time being, else back out the new version of the language.

Are you saying that "pacman -Syu" is always safe (and check strict dependency troubles) but not "pacman -Syyu"?
And so if I see any problems during the upgrade, just rollback the affected packages (such as python) and all should be okay?
Would this rollback procedure have worked to avoid problems during the glib mismatch that apparently caused some people to be unable to boot?

levi wrote:

We could also do with more people on the bug tracker to be honest, to raise issues when they find them and maybe to try to reproduce problems to help validate reports, which are languishing slightly right now.

If you can point me to it please, I will sign up! I'd be glad to help there, as I [wrongly?] thought this was the place to report issues.

Last edited by rogerthat (2018-08-27 16:37:48)

levi · 2018-08-27 17:31:30

rogerthat wrote:

Are you saying that "pacman -Syu" is always safe (and check strict dependency troubles) but not "pacman -Syyu"?
And so if I see any problems during the upgrade, just rollback the affected packages (such as python) and all should be okay?
Would this rollback procedure have worked to avoid problems during the glib mismatch that apparently caused some people to be unable to boot?

Sorry, I had misunderstood what that second '-y' does. I had thought it skipped all 'do you want to upgrade?' questions, but actually it just forces a database update, so actually it's safe. I can't see an 'assume yes' option in pacman actually, so you'd actually have to do some piping to get it to do that, I assume.

rogerthat wrote:

levi wrote:
We could also do with more people on the bug tracker to be honest
If you can point me to it please, I will sign up! I'd be glad to help there, as I [wrongly?] thought this was the place to report issues.

It's linked in the heading of this page, as 'Bugs'. You will need a bit of help to get fully set up, unless it's changed since I signed up a few weeks ago now. You'll need someone to give you the relevant permissions to raise bugs, which you don't seem to get by default. I feel that just at the moment, highlighting these bugs via a forum post is valid, since it gets them a bigger visibility, but once we've got a process to handle these breaks more smoothly, the bug tracker would probably be the best way alone to raise them.

andreas_baumann · 2018-08-27 19:05:15

No, I'm running on a 64-bit with a proper VT-x emulation. The emulation of 32-bit processors (especially when
it comes to 'I really want to have a Pentium MMX' for instance) is not perfect, but it's enough to detect most
errors. The real testing is then done on 5-6 old 32-bit machines, ranging from an EEEPC 701 to an old AMD K-6
machine.

As I see it, the glibc 2.28 bug was a really dangerous one, as you could brick your system with it. All
the other incidents are - sorry - minor IMHO.

Things will only get worse: packages handling PCMCIA have to be maintained, Xorg drops old MACH64 drivers.
Software gets written to use SSE and SSE2 (say Qt, webkit). Software needs more than >4GB of RAM to compile (for instance
firefox) and some (especially) new software not even bothering about 32-bit anymore - especially new software
of startup companies - why should they?

levi · 2018-08-27 19:57:07

I tend to feel that if projects start to drop old functionality or depend on new features, unless third parties step forward to maintain forks of the old code, we shouldn't feel the need to support them; it's down to people with actually those features to maintain the old code if they think it's worth it. I'm fairly sure running a graphical environment on an old system without SSE2 isn't really practical any more anyway - I struggled to find a desktop environment that would run on my old build server which was a mid-era coppermine P3, although to be fair I was trying to use the built-in graphics, and ended up using tmux in text mode for my development purposes, and links for when I needed the internet.

I'm not sure how arch32 can best notify users of older hardware what to expect on their systems, in the case of stuff which just won't run because it's built to use newer instruction sets than they have support for. It already doesn't support anything pre-pentium pro, or anything newer than about 2008. If the kernel or bash, or filesystem drivers start to have developments that use SSE, we might need to get users to compile them themselves if they want to use those - it's usually just a compiler flag to enable/disable that stuff.

andreas_baumann · 2018-08-27 20:43:26

You can have a look at my i486 branch, which actually runs on a i486. But it's currently just base and base-devel with some stuff like
for instance gnat missing.

Xorg went the way to basically support Nvidia, Intel and AMD GPUs, and that's it. So for older machines alternative X implementations are most likely needed.
I never got anything to run on a MACH64 or a Matrox Millenium GPU anymore.

So, that's also what I think. Assuming MMX, SSE and SSE2 is around is a good choice and it allows roughly 10 year old hardware do it's job.
Anything older I consider vintage computing. :-)

rogerthat · 2018-08-27 20:59:50

This is a naive question, but I do think it might be relevant to this thread...

Can anyone give a basic explanation of the process behind the archlinux32 repo maintenance?
I'm wondering what is the process (automated or manual) that leads to broken builds being pushed to the non-test repos.

I am just guessing from bits and pieces I've seen mentioned... It sounds like when archlinux[64] pushes things to the repos, archlinux32 mirrors those pushes, and there is a recompile process that maintains the parallel codebase?

Last edited by rogerthat (2018-08-27 21:20:36)

levi · 2018-08-27 22:26:08

I was under the impression that the arch32 testing repos were intending to mirror the arch64 mainline repos, and then after testing things get migrated to the mainline arch32 repos. That doesn't seem to be strictly the case though; I've found dozens of differences between the a32 testing repos and the a64 mainline repos, and while some of those differences are listed as a build failure, not all are.

Edit: I'd have thought it would be relatively safe in this day and age to cross-compile everything for arch32 on a faster more modern system running arch64 and multilibs with more modern amount of RAM. I used to do a fair bit of cross compilation in and old job, but I should declare that I've not actually tried it on arch64 yet.

Last edited by levi (2018-08-27 22:47:57)

rogerthat · 2018-08-27 23:50:43

Thanks, Levi. That helps clarify.

On another topic, so I've been told that arch doesn't support partial upgrades. And I've seen people mention a pacman "ignore list." So, I'm confused. If someone encounters a system problem after doing an upgrade, how are they supposed to recover?

I thought you would do a pacman -U to rollback all the packages in the upgrade using the local cache files?

Then, is there an ability to ignore certain problematic packages (such as python recently, for example) and run an upgrade again?

levi · 2018-08-28 03:43:27

Just because it's not supported, it doesn't mean you can try it and run with it. Just don't expect any real support for that setup.

rogerthat · 2018-08-28 04:06:55

So, just tried an update...

1) Perl 5.26.6 -> 5.28.0 problem:

WARNING: '/usr/lib/perl5/5.26' contains data from at least 1 packages which will NOT be used by the installed perl interpreter.
Run the following command to get a list of affected packages: pacman -Qqo '/usr/lib/perl5/5.26'

And that generates:

$ pacman -Qqo '/usr/lib/perl5/5.26'
libproxy

I'm not sure what that is or if I need it.

2) Python 3.6.6 -> 3.7.0 problem:

When I run rhythmbox I get:

Failed to load module 'python3loader': libpython3.6m.so.1.0: cannot open shared object file: No such file or directory
Could not load plugin loader 'python3'

WHAT SHOULD I BEST DO?
I'm thinking to downgrade python and perl (and closely related packages) using the cache.

Last edited by rogerthat (2018-08-28 04:17:20)

levi · 2018-08-28 04:28:44

The perl failure is probably just noise. According to the package I have, it's required on my system by: glib-networking hexchat qt5-base

It's a big package with the c/c++ library plus *all* of the bindings in one package though, so if none of those are written in perl then I at least would be okay with that warning. You can check your own system by running 'pacman -Ql libproxy' and observing the 'Required by' line.

Your rhythmbox issue might be resolvable with a config change, but I'm not familiar with that app so can't advise. If you can find the string 'libpython3.6m.so.1.0' anywhere in its config, then changing that to the actual libpython3 filename you have should work around that problem.

rogerthat · 2018-08-28 12:53:17

So, I'm now running pacman with excludes like this... Is this the best way to cope until things are resolved?

sudo pacman -Syu --ignore perl,perl-parse-yapp,python,pygobject-devel,python-gobject

--------------------------------

levi wrote:

The perl failure is probably just noise. According to the package I have, it's required on my system by: glib-networking hexchat qt5-base

For me, it's required by libproxy (which I've seen others report as well).

$ pacman -Qqo '/usr/lib/perl5/5.26'
libproxy

--------------------------------

levi wrote:

It's a big package with the c/c++ library plus *all* of the bindings in one package though, so if none of those are written in perl then I at least would be okay with that warning. You can check your own system by running 'pacman -Ql libproxy' and observing the 'Required by' line.

It just spits out a list of files. There is no 'Required by' anywhere. E.g.:

$ pacman -Ql libproxy
libproxy /usr/
libproxy /usr/bin/
libproxy /usr/bin/proxy
libproxy /usr/include/
libproxy /usr/include/proxy.h
libproxy /usr/lib/
libproxy /usr/lib/libproxy.so
libproxy /usr/lib/libproxy.so.1
libproxy /usr/lib/libproxy.so.1.0.0
libproxy /usr/lib/libproxy/
libproxy /usr/lib/libproxy/0.4.15/
libproxy /usr/lib/libproxy/0.4.15/modules/
libproxy /usr/lib/libproxy/0.4.15/modules/config_gnome3.so
libproxy /usr/lib/libproxy/0.4.15/modules/config_kde.so
libproxy /usr/lib/libproxy/0.4.15/modules/network_networkmanager.so
libproxy /usr/lib/libproxy/0.4.15/modules/pacrunner_webkit.so
libproxy /usr/lib/libproxy/pxgsettings
libproxy /usr/lib/perl5/
libproxy /usr/lib/perl5/5.26/
libproxy /usr/lib/perl5/5.26/vendor_perl/
libproxy /usr/lib/perl5/5.26/vendor_perl/Net/
libproxy /usr/lib/perl5/5.26/vendor_perl/Net/Libproxy.pm
libproxy /usr/lib/perl5/5.26/vendor_perl/auto/
libproxy /usr/lib/perl5/5.26/vendor_perl/auto/Net/
libproxy /usr/lib/perl5/5.26/vendor_perl/auto/Net/Libproxy/
libproxy /usr/lib/perl5/5.26/vendor_perl/auto/Net/Libproxy/Libproxy.so
libproxy /usr/lib/pkgconfig/
libproxy /usr/lib/pkgconfig/libproxy-1.0.pc
libproxy /usr/lib/python2.7/
libproxy /usr/lib/python2.7/site-packages/
libproxy /usr/lib/python2.7/site-packages/libproxy.py
libproxy /usr/lib/python3.6/
libproxy /usr/lib/python3.6/site-packages/
libproxy /usr/lib/python3.6/site-packages/libproxy.py
libproxy /usr/share/
libproxy /usr/share/cmake/
libproxy /usr/share/cmake/Modules/
libproxy /usr/share/cmake/Modules/Findlibproxy.cmake

Last edited by rogerthat (2018-08-28 13:14:16)

levi · 2018-08-28 15:52:51

Sorry, I replace that lower case L with a lower case I. D'oh!

andreas_baumann · 2018-08-29 08:39:44

@rascholer: thanks for the list, I'm priorizing and pushing those packages to stable now. Just be patient, as I have still a backlog of 800 packages to be rebuild.

rogerthat · 2018-08-29 11:26:26

levi wrote:

Sorry, I replace that lower case L with a lower case I. D'oh!

$ pacman -Qi libproxy
Name            : libproxy
Version         : 0.4.15-6
Description     : A library that provides automatic proxy configuration management
Architecture    : i686
URL             : http://libproxy.github.io/libproxy/
Licenses        : LGPL
Groups          : None
Provides        : None
Depends On      : gcc-libs
Optional Deps   : networkmanager: NetworkManager configuration module
                  perl: Perl bindings [installed]
                  python2: Python 2.x bindings [installed]
                  python: Python 3.x bindings [installed]
                  glib2: gsettings configuration module [installed]
                  webkit2gtk: PAC proxy support (Webkit2gtk engine)
Required By     : glib-networking  libquvi
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 301.00 KiB
Packager        : Evangelos Foutras <evangelos@foutrelis.com>
Build Date      : Sat 26 Aug 2017 03:51:49 PM CEST
Install Date    : Sun 08 Oct 2017 10:42:29 PM CEST
Install Reason  : Installed as a dependency for another package
Install Script  : No
Validated By    : Signature

levi · 2018-08-29 16:27:28

Okay, gcc-libs shouldn't be a problem; perl scripts don't need compiling, so compiler libs won't be affected by perl libs in the wrong place. Of the optional deps you have installed glib2 could potentially be a problem - I don't know it. The rest don't use perl as far as I know.

Of your actual required lines I have glib-networking, but not libquvi.

Personally I'd try to live with that and see if it actually throws up any errors in your daily use. It's not likely to stop the machine booting.

rogerthat · 2018-08-29 23:35:35

Thanks, I guess glib2 might be part of the Rhythmbox UI rendering.

Anyway, I am using this command for now. Is something like this the proper/best way to cope until things are resolved?

sudo pacman -Syu --ignore perl,perl-parse-yapp,python,pygobject-devel,python-gobject

levi · 2018-08-30 01:29:19

That's reasonable, if you absolutely want to upgrade everything else. You could also edit your /etc/pacman.conf and write each package as a IgnorePkg= line under your [options] section, if you feel you're going to have to keep doing this for a little while.

Personally I'm still at the stage where I just don't upgrade, or upgrade specific packages as I become aware of patches in them. But it's getting closer, I'll admit.

Your RhythmBox problems seem to result from a scripting system it employs looking for the wrong version of perl. It's not a glib2 error for sure.

eschwartz · 2018-08-31 16:36:15

levi wrote:

Then again, I'm not entirely happy that simple dependency dropouts have been allowed into stable, let alone testing. This is possible to check for programatically, so could not be a surprise in the future.

It's more difficult than you think. Arch Linux upstream doesn't check for this programmatically, we do it by hand using the manpower of several dozen people and have no issues. Arch Linux 32 is powered by from one to three people on any given day, and is heavily reliant on programmatic checking, and gets it wrong sometimes.

It is very, very easy to check for shared library mismatches, and we have grep tools to aid us in our manual oversight of this process. For Arch Linux 32, that's mostly the only thing they do use.

Where does this fall down? For things like, you guessed it, python and perl, where there are no shared library links to check. Perl doesn't version their library at all, but stores it instead in /usr/lib/perl5/5.28/core_perl/CORE/libperl.so for hysterical raisins. Python does version the library, but that only helps for compiled extensions which link to libpython3.7m.so.1.0 and excludes the many .py files which are stored in version-specific PYTHONPATH locations.

This too can be scripted around, but I think everyone deserves one free opportunity to completely screw up their first ever major rebuild of a weird situation. I anticipate that abaumann and deep42thought will adapt their scripts to properly handle python and perl in the future.

As for the glibc (the GNU C library, not glib, the Gnome Library!) symbols error, this happened exclusively because they built things against testing/glibc, but then moved them to stable *before* glibc. More experienced distro maintainers will have long since learned their lesson about low-level infrastructure like this -- glibc is backwards-compatible but not forwards compatible, and no longer uses meaningful sonames. In theory this is to allow running software on any system newer than the one that it was built on, forever into the future. In practice this means you need to make sure you built against the older system...

rogerthat · 2018-08-31 16:53:26

Thanks to everyone working on fixing this!
Doesn't sound easy. I'm looking forward to getting updated again once things are resolved.

thx1138 · 2018-09-01 18:07:17

Questions:

Where, for instance, chromium and firefox are dependent upon the earlier version of libicui18n and become unusable with the "icu" package upgrade, and the package web pages show newer "Versions Elsewhere chromium-68.0.3440.106-1.0 [build-list] (i686)" and "Versions Elsewhere firefox-61.0.2-1.0 [build-list] (i686)", does that mean that a rebuild is scheduled but not yet complete?

In other words, is the availability of hardware to re-build large 32-bit packages quickly an issue here? Are more build machines needed?

Or, is there actually an already rebuilt newer package version literally "Elsewhere", and I just don't know where to look? I'm just trying to understand how the process works now with Arch 32-bit. I don't understand the meaning of "Versions Elsewhere" in this context.

Also, was the problem with the "icu" upgrade to icu 62.1-1.0 something everyone already knew about, or would it be helpful to report breakage somewhere, or to someone? I'd like to understand what actions are helpful, and which are not helpful.

James

andreas_baumann · 2018-09-01 19:05:07

For Mozilla stuff, if somebody finds a way to build around the 4 GB virtual address space exhaustion when linking with the gold LD, I'm happy to hear about.
(this is Thunderbird, Firefox, Seamonkey when building libxul.so).

Chromium had some linking issues and should now be fixed.

Detecting dependencies is not always easy, for instance icu is just a depends=(icu). The problem is that they change the ABI in every version
icu.so.62 vs icu.so.61. So, sometimes those things get missed. The trick is usually to install testing to a virtual machine, see what breaks (or
use a script), them if things are ok, report them back (there is a wiki entry on how to report bugs here - basically by script:
https://bbs.archlinux32.org/viewtopic.php?id=171).

Sometimes an important package breaks and all depending packages cannot be built. For instance this is true with recent glibc 2.28 updating
and gnulib trouble (__readahead, readdir_r, ftello, major/minor, statx). Authors were using hacks and unofficial interfaces in the C library itself.
Usually people tend to fix that by just disabling deprection warnings, till the ABI breaks - instead of fixing their code!

levi · 2018-09-01 19:07:19

The problem with firefox is 32-bit native machines generally top out at 2GB of RAM, and that turns out to not be enough to build firefox these days. Someone who does builds needs to set up a cross compiler I guess, so they can use newer hardware with more RAM to do it.

This thread on the other hand is mainly about perl and python packages which were slow in being updated to work with the new version of perl/python, which ended up being pushed to stable before everything was ready. I don't honestly know why that happened; perl and python builds generally don't need lots of RAM, so I think it may just have been the old problem between keyboard and chair. Mostly the packages which needed to be updated were already in testing. There a thread for the firefox and chromium issues I think, and the firefox one at least contains packages you can build which use the prebuilt 32-bit packages from mozilla to update your firefox.

Edit: Ninjaed by Andreas.

Last edited by levi (2018-09-01 19:09:27)

andreas_baumann · 2018-09-01 19:10:52

Well, the cross-compiling wiki of firefox tells me nothing about rust, because Gecko is now written in rust. So you need
a cross-compiling rust first. Tried to use the classic linker instead of gold - linking trouble.

Some of my trials are documented in https://bugs.archlinux32.org/index.php? … task_id=43.

Mozilla is source number one for bloated and ill-designed software, sorry. Find a new browser fast.
And don't start to program in rust..

deep42thought · 2018-09-01 21:43:06

thx1138 wrote:

Where, for instance, chromium and firefox are dependent upon the earlier version of libicui18n and become unusable with the "icu" package upgrade, and the package web pages show newer "Versions Elsewhere chromium-68.0.3440.106-1.0 [build-list] (i686)" and "Versions Elsewhere firefox-61.0.2-1.0 [build-list] (i686)", does that mean that a rebuild is scheduled but not yet complete?

Yes, exactly: This means, they are scheduled to be built but are not yet successfully built. (either not yet built at all or the build threw an error)

thx1138 wrote:

In other words, is the availability of hardware to re-build large 32-bit packages quickly an issue here? Are more build machines needed?

The number of available build slaves is sufficient (but we don't have too much head room). We rather lack the manpower to fix (in a timely manner) all the packages which do not compile on/for 32-bit. To put this in numbers: big rebuilds of python or perl (where a few thousand packages need to be rebuilt) are done in less than 24h - iff most of the packages build without issues; if we need to fix a lot of packages, then it'll take us a lot longer ...

thx1138 wrote:

Or, is there actually an already rebuilt newer package version literally "Elsewhere", and I just don't know where to look? I'm just trying to understand how the process works now with Arch 32-bit. I don't understand the meaning of "Versions Elsewhere" in this context.

The (built and not-yet-built) packages are tracked in a mysql database. "Version Elsewhere" simply means, that the mentioned version is found in that mysql database. Whether it corresponds to a real package or a virtual one, depends on the repository name: build-list and deletion-list are virtual repositories; core, extra, community, testing, community-testing, staging, community-staging and build-support are real repositories.

Here is also a thread describing how the build system works.

Regards, deep42thought

Arch Linux

#26 2018-08-27 16:36:19

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#27 2018-08-27 17:31:30

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#28 2018-08-27 19:05:15

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#29 2018-08-27 19:57:07

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#30 2018-08-27 20:43:26

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#31 2018-08-27 20:59:50

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#32 2018-08-27 22:26:08

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#33 2018-08-27 23:50:43

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#34 2018-08-28 03:43:27

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#35 2018-08-28 04:06:55

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#36 2018-08-28 04:28:44

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#37 2018-08-28 12:53:17

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#38 2018-08-28 15:52:51

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#39 2018-08-29 08:39:44

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#40 2018-08-29 11:26:26

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#41 2018-08-29 16:27:28

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#42 2018-08-29 23:35:35

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#43 2018-08-30 01:29:19

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#44 2018-08-31 16:36:15

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#45 2018-08-31 16:53:26

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#46 2018-09-01 18:07:17

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#47 2018-09-01 19:05:07

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#48 2018-09-01 19:07:19

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#49 2018-09-01 19:10:52

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#50 2018-09-01 21:43:06

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

Board footer