Why are incomplete rebuilds pushed to stable (perl, python3.7)?

rascholer · 2018-08-23 19:00:58

I'm really fed up with this. It feels like the ignore list in my pacman.conf is currently near the size of a telephone book.
Below are all packages built against python 3.6 and the woefully incomplete list of updates build against python 3.7.

A similar problem exists with perl 5.28. In my case "imagemagick" and "libproxy" weren't build against the new version. And perl 5.28 is available in core since a few days ago.

I'm afraid to run a system update at the moment.
This is a thing that should never happen.

pacman -Qoq /usr/lib/python3.6 wrote:

avahi
btrfs-progs
ipython
lensfun
libblockdev
libbytesize
libgexiv2
libibus
libixion
liborcus
libproxy
libreoffice-fresh
libxml2
perl
perl-error
perl-mailtools
perl-timedate
python
python-appdirs
python-cachecontrol
python-cairo
python-certifi
python-chardet
python-colorama
python-crypto
python-decorator
python-distlib
python-distro
python-docopt
python-feedparser
python-gobject
python-html5lib
python-idna
python-jedi
python-keyutils
python-lockfile
python-msgpack
python-mutagen
python-netifaces
python-packaging
python-parso
python-pew
python-pexpect
python-pickleshare
python-pip
python-pipenv
python-progress
python-prompt_toolkit
python-psutil
python-ptyprocess
python-pygments
python-pyparsing
python-pytoml
python-requests
python-retrying
python-setuptools
python-sgmllib
python-six
python-traitlets
python-urllib3
python-virtualenv
python-virtualenv-clone
python-wcwidth
python-webencodings
python-xdg
python-yaml
speech-dispatcher
udiskie
util-linux
xcb-proto
youtube-dl

checkupdates wrote:

btrfs-progs 4.17-1.0 -> 4.17.1-1.0
iproute2 4.17.0-1.0 -> 4.18.0-1.0
lensfun 0.3.2-6 -> 0.3.2-7.0
libbytesize 1.3-1.0 -> 1.3-2.0
libdrm 2.4.92-1.0 -> 2.4.93-1.0
libinput 1.11.2-1.0 -> 1.11.3-1.0
libixion 0.13.0-2.0 -> 0.13.0-3.0
liborcus 0.13.4-2.0 -> 0.13.4-3.0
libsoup 2.62.2-1.0 -> 2.62.3-1.0
libwps 0.4.9-1.0 -> 0.4.10-1.0
libxml2 2.9.8-2.0 -> 2.9.8-3.0
llvm-libs 6.0.1-1.0 -> 6.0.1-2.0
man-db 2.8.3-2.0 -> 2.8.4-1.0
pygobject-devel 3.28.3-1.0 -> 3.28.3-2.0
pygobject2-devel 2.28.7-1.0 -> 2.28.7-2.0
python 3.6.6-1.0 -> 3.7.0-3.0
python-appdirs 1.4.3-1 -> 1.4.3-2.1
python-cairo 1.17.0-1.0 -> 1.17.0-2.0
python-certifi 2018.4.16-1.0 -> 2018.8.13-1.0
python-chardet 3.0.4-1 -> 3.0.4-2.0
python-crypto 2.6.1-5 -> 2.6.1-6.0
python-gobject 3.28.3-1.0 -> 3.28.3-2.0
python-idna 2.7-2.0 -> 2.7-3.0
python-packaging 17.1-1.0 -> 17.1-2.1
python-pyparsing 2.2.0-1 -> 2.2.0-2.1
python2-cairo 1.17.0-1.0 -> 1.17.0-2.0
python2-gobject2 2.28.7-1.0 -> 2.28.7-2.0
shared-mime-info 1.9-1.1 -> 1.10-1.0
vulkan-icd-loader 1.1.73.0-1.0 -> 1.1.82+2958+1f9a54573-1.0

Last edited by rascholer (2018-08-23 19:04:52)

bulletmark · 2018-08-23 23:20:15

yes, there has gotta be a better way than this ...

levi · 2018-08-24 05:19:21

It does seem at present it's less hassle to add the testing repos to your pacman.conf. Those are based on the stable branch of the mainline arch64 project, so tend to have much less dependency dropouts than stable seems to at the moment. It's not without troubles; around a month ago I held off updating for a couple of weeks because of a perl dependency failure, but I've avoided many problems reported by users of static in the past few weeks.

I'm not sure if a significant proportion of the maintainers of this project are on holiday at the moment, it being August and all. Certainly the bugs I've raised have not been touched yet.

As I mentioned in another thread, I'm currently working on a script that can check the dependency tree of all packages in your currently configured repos. I hope to have a beta version out in the next few days.

rascholer · 2018-08-24 17:32:18

levi wrote:

It does seem at present it's less hassle to add the testing repos to your pacman.conf. Those are based on the stable branch of the mainline arch64 project, so tend to have much less dependency dropouts than stable seems to at the moment.

The testing repos are certainly more complete but are still missing packages that need to be rebuilt. I could somewhat understand why [testing] is incomplete, but why and how those partial rebuilds ended up in stable escapes me.

I'm using Arch64 for 11 years and don't think I've ever had problems (at least in this scale) before.

jetfrog28 · 2018-08-25 16:07:34

rascholer wrote:

The testing repos are certainly more complete but are still missing packages that need to be rebuilt.

Yeah. qutebrowser broke so I updated to the testing repos, and even there I'm missing the following:

pacman -Qqo /usr/lib/python3.6 wrote:

avahi
hplip
libffado
libibus
libproxy
libreoffice-fresh
libxml2
python-pyqt5
python-sip-pyqt5
python-six
qutebrowser
xcb-proto

timsong · 2018-08-25 17:33:25

I totally agree with you rascholer, doing pacman -Syu lately is like playing Russian roulette on Archlinux 32. Thankfully I keep my system partition regularly backed up (with partimage on bootable puppy linux cd) so I can go back in time if needs be. I worry for all the newbees to Archlinux 32.
The one reason I stuck with Arch Linux for the last 10 years was the system upgrade stability.
I am now frightened to do pacman -Syu, without making a partimage of my system first.

The recent "GLIBC2.28 not found" error completely f***ed my system, I couldn't even get passed GRUB. I was very lucky to have another computer.

I thank all the people involved with Archlinux 32 for giving there time so we can have a 32bit Arch, but I must express my unhappiness of late with updates.

Last edited by timsong (2018-08-25 17:36:18)

tommi · 2018-08-25 19:39:33

To backup a system is one thing. To learn about it is another. So i'm not afraid of Archlinux 32. I don't worry for all the newbees. This is no bad system, thus they realy learn something. Tala!
Help is necessary and present. But if you want a rolling system without any learning you can use Microsoft's Windows for example. Give them control so you never get problems
Good luck!
Tommi

P.S.: for a good system it's not the job to protect the user from himself (:)

rascholer · 2018-08-25 20:26:15

tommi wrote:

To backup a system is one thing. To learn about it is another. So i'm not afraid of Archlinux 32. I don't worry for all the newbees. This is no bad system, thus they realy learn something.

I don't think you understand what the problem is.
Since a few weeks the stable repositories are in an incorrect state. Meaning that some packages are rebuilt against a new library version, but the library version they now depend on is not offered as an update (glibc). This led to machines that couldn't be booted, if you overlooked the error messages when the ALPM hooks were run.
And downgrading X amount of packages is not always easy.

Also incomplete rebuilds of python and perl packages are currently offered which leads to applications not working. This is especially troubling, since no error message was shown during updates.

And how can the users learn something when it is impossible to install a correct and working Arch Linux 32 system at the moment?

I can't remember when the stable repositories of any distribution which where in such an incorrect state over such a long period.

P.S.: for a good system it's not the job to protect the user from himse[lf (:)

In this particular case it's the user that needs to be protected from the distribution.

Edit:
There are currently 30 updated for my system available that I can't install. It's also impossible at the time to install new packages, since partial updates are not supported and some may depend on python/perl packages that are already rebuilt for the new python/perl version. This is simply unacceptable.

Last edited by rascholer (2018-08-25 21:22:16)

timsong · 2018-08-25 22:22:12

Sorry Tommi, but you are typing out of your rear end.
pacman -Syu should be effortless and safe, as it was for 99.9% of the time on Arch Linux.

tommi wrote:

But if you want a rolling system without any learning you can use Microsoft's Windows for example. Give them control so you never get problems wink

Windows is not a rolling system like the arch repos. Does windows update also update every non windows app installed on the system and check their dependancies ? No it doesn't.

tommi wrote:

I don't worry for all the newbees

Hmmmmm, that says all we need to know about you.

Last edited by timsong (2018-08-25 22:29:30)

rogerthat · 2018-08-26 01:55:48

Problems here too. I ran pacman system upgrade, and it broke Rhythmbox 3.4.2 because of Python 3.7.0 .
I rolled back python, python-devel, and python-gobject, and that restored things.

Like others, this is my second time in as many weeks that pacman broke my system (glibc2.28 last time). Never had this before in a few years of arch usage.

I'm now trying to figure out how to verify if everything on my system is in a valid internal state. For example, warning in pacman.log that perl 5.26 "contains data from the [libproxy package] which will not be used by the installed perl interpreter."

Any advice?
Thanks

rogerthat · 2018-08-26 02:19:09

levi wrote:

As I mentioned in another thread, I'm currently working on a script that can check the dependency tree of all packages in your currently configured repos. I hope to have a beta version out in the next few days.

That would be so useful!

tommi · 2018-08-26 03:21:55

I am sorry. You are right.

levi · 2018-08-26 03:38:55

rogerthat wrote:

levi wrote:
As I mentioned in another thread, I'm currently working on a script that can check the dependency tree of all packages in your currently configured repos. I hope to have a beta version out in the next few days.
That would be so useful!

It's out now; see: https://bbs.archlinux32.org/viewtopic.php?id=2565

tommi · 2018-08-26 03:52:54

Excuse me, please!

levi · 2018-08-26 05:04:28

You're right in that it's been a big learning experience for us all though!

andreas_baumann · 2018-08-26 18:12:56

The detection mechanism for stable has problems with:
- ABI changes as in glibc with tagged symbols
- breaks for scripting languages like perl and python which have a directory somewhere
like /usr/xxxperl/site/5,26 and /usr/xxxperl/site/5.28 and are completly incompatible between
minor versions (actually, are they?). This also relates to pacman changes, where scripts
for dependency calcutations inside depends directives where used in the past and those had
to go away.

The dependencies are only partially modelled (or not modelled at all) in the PKGBUILD
dependencies. For instance imagemagick depends on A perl, not on a specific one.

Currently there are other major issues bugging us:

- python being a mess in terms of bootstrapping and circular dependencies everywhere.
- toolchain changes and severe bugs in the toolchain (should I mention the LTO LLVM disaster
when linking?)
- upstream packages making more and more of Intel-fine-tuning using dubious
constructs (for example I doubt that parsing your command line options gets
any fast - at least relevantly faster - by improving the strcmp with SSE2).
- severe performance issues on the buildmaster itself

Archlinux32 is a free effort of basically 3 persons. Official support has been dropped (though
I have to say that upstream - thanks eli - and downstream - thanks all parabola, manjaro32 guys)
are still helping out as often as they can.

This all leads to a backlash currently.

There is the plugbuild approach, where downstream (ArchArm in this case) just builds, whatever
gets pushed upstream, fixing the bugs. But this also leads to package breakage (I still have
some krb5 issues in postfix for instance).

The current build system also has another purpose: it tries to produce reproducable and automatized
packages, less human intervention and trickery (as human labour is a scarse resource ATM in Archlinux32).

I'm working currently on all the issues, deep42thought is in his well deserved holidays, so the number
of hands is limited.

My two cents. :-)

timsong · 2018-08-26 19:36:00

And now tonight we have libx264 and ffmpeg errors on upgrade !!!!!
https://bbs.archlinux32.org/viewtopic.php?id=2553

Please stop pushing all these new updated packages and changes to the stable mirrors until all the dependancies etc are satisfied !!!!!!

Is there anybody managing ArchLinux32 ?

Last edited by timsong (2018-08-26 19:43:51)

levi · 2018-08-26 20:18:04

At least you can say 'n' to that upgrade and it all works fine. But sure it could be better.

I am tempted to push back against the idea that 'pacman -Syu' should always be safe and simple to do. Even the most secure and functional upgrades might degrade the performance of your hardware in this day and age of speculative execution flaws. But I've long been slightly disappointed that I never got as much information about what was in my upgrades than I got on an old debian stable branch, although on the other hand I'm sure it wouldn't take me too long to get burned out if they did tell me about every little change they were making - that's the risk you play by being closer to the cutting edge in a linux like Arch unlike a nicer old (but older) Debian stable branch.

Then again, I'm not entirely happy that simple dependency dropouts have been allowed into stable, let alone testing. This is possible to check for programatically, so could not be a surprise in the future.

andreas_baumann · 2018-08-27 08:56:09

@timsong: deep42thought, tyzoid and me are managing Archlinux32. If you prefer something more stable, you are free to
compile your own Gentoo for 32-bit Intel or use Fedora or Debian.

rogerthat · 2018-08-27 15:53:37

andreas_baumann wrote:

The detection mechanism for stable has problems with:
- ABI changes as in glibc with tagged symbols
- breaks for scripting languages like perl and python which have a directory somewhere
like /usr/xxxperl/site/5,26 and /usr/xxxperl/site/5.28 and are completly incompatible between
minor versions (actually, are they?). This also relates to pacman changes, where scripts
for dependency calcutations inside depends directives where used in the past and those had
to go away.
The dependencies are only partially modelled (or not modelled at all) in the PKGBUILD
dependencies. For instance imagemagick depends on A perl, not on a specific one.
Archlinux32 is a free effort of basically 3 persons. Official support has been dropped (though
I have to say that upstream - thanks eli - and downstream - thanks all parabola, manjaro32 guys)
are still helping out as often as they can.

Hello. I have no desire to complain. This is no doubt a big project to manage with so few hands. And it is hugely helpful to me to be able to run 32-bit ArchLinux on some old hardware I have.

I just would like to know, in practical terms, what can I do to avoid having my hardware and/or applications break during system upgrades?

I work in enterprise software development, so I have good technical comprehension, but I do not have the specific skills needed to understand linux system administration in a deep way. I would offer my help, except that I can barely write an ugly bash script, so I'm guessing I won't be much other than a burden.

Specifically, I'd like to know:

1) Since pacman -Syu doesn't apparently validate what is in the repo, how can I verify some packages haven't been released that will create unmet dependencies? I've gotten errors and/or warnings recently related to glib, perl, and python. How can I catch this type of thing BEFORE running pacman and breaking things?

2) Since I've now done a manual rollback procedure (pacman -U from local cache) twice, how can I check if my current installation is actually valid, meets all dependencies, and is built against proper library verisons?

3) Is it safe right now actually to run pacman -Syu and get things synced and valid?

Thank you.

Last edited by rogerthat (2018-08-27 16:09:45)

timsong · 2018-08-27 16:00:26

andreas-baumann wrote:

deep42thought, tyzoid and me are managing Archlinux32

Thanks for all your hard work in keeping ArchLinux32 alive, and please excuse my frustrations.

andreas_baumann wrote:

If you prefer something more stable, you are free to
compile your own Gentoo for 32-bit Intel or use Fedora or Debian.

I love the arch system and use many AUR builds, and my own builds so I would like to stick with the arch system as I think it's the best.

andreas_baumann · 2018-08-27 16:05:15

Don't update the most important systems first, ideally have some virtual machines and do
test installations there. If they fail, you just wait till you update the real system.

That's what I'm doing. :-)

andreas_baumann · 2018-08-27 16:07:37

The current status is not satisfactory, I know: the backlog is huge: https://buildmaster.archlinux32.org/
and we are not fast enough, if whole universes like Python, Haskel, C-library, toolchain, Perl are changed
at once.

levi · 2018-08-27 16:18:53

rogerthat wrote:

Specifically, I'd like to know:
1) Since pacman -Syu doesn't apparently validate what is in the repo, how can I verify some packages haven't been released that will create unmet dependencies? I've gotten errors and/or warnings recently related to glib, perl, and python. How can I catch this type of thing BEFORE running pacmand an breaking things?

pacman validates all of your dependencies, strictly speaking - it'll only install things that can be installed along with all of their defined dependencies. There are other troubles currently, namely your python one, whereby certain python plugin packages actually install their scripts to specific versioned folders (the intent is you can run multiple different versions of scripting languages for testing purposes, and you sometimes need different code to work around issues in different versions of the core interpreter). It can only tell you about these changes after installing the packages unfortunately.

The perl issues are pacman dependency troubles, namely quite a lot of perl library packages specify they won't work with the new version of perl. It can and does tell you about that and will refuse to apply an upgrade if you actually need those libs. Perl itself is fine and functional, but if you particularly need any of those libs (and its safe to assume you probably did once upon a time at least) you'll have dependency troubles which will stop you applying anything.

I think the glib problems might have been resolved, but I need to refamiliarise myself with that issue.

rogerthat wrote:

2) Since I've now done a manual rollback procedure (pacman -U) twice, how can I check if my current installation is actually valid, meets all dependencies, and is built against proper library verisons?

If pacman installed everything and didn't warn about bad libs you should be reasonably safe to assume everything'd good in your system, although it's always the case that the only way to be sure is to use it. There's no linux I know of that guarntees anything is going to be fully operational for you - maybe some of the commercial ones have that though.

rogerthat wrote:

3) Is it safe right now actually to run pacman -Syu and get things synced and valid?

It's always safe to run pacman -Syu with only the one -y - it'll check for strict dependency troubles. Scripting libs version misses are noted but can only be warned about once you've actually committed those to your file system. You'll need to make a call about whether you can live without some of those libs for the time being, else back out the new version of the language.

To avoid these problems in stable, I guess we need more people using the testing repos, so the maintainers at least have valid packages to push forward. We could also do with more people on the bug tracker to be honest, to raise issues when they find them and maybe to try to reproduce problems to help validate reports, which are languishing slightly right now.

levi · 2018-08-27 16:29:20

andreas_baumann wrote:

Don't update the most important systems first, ideally have some virtual machines and do
test installations there. If they fail, you just wait till you update the real system.
That's what I'm doing. :-)

Do you have a 32-bit only machine with VT-x support? In my experience running virtual machines in purely software modes is pretty slow and frustrating, and a lot of those old machines aren't exactly speedy to begin with. I personally only run stuff on the bare metal, and back out upgrades if they break my processes, and only try upgrades every few days if they're failing.

I only really get and care about kernel issues with all of the speculative execution attacks being found at the moment. But thus far the upgrade blocks haven't stopped me installing any of those for very long, touch wood.

Arch Linux

#1 2018-08-23 19:00:58

Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#2 2018-08-23 23:20:15

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#3 2018-08-24 05:19:21

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#4 2018-08-24 17:32:18

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#5 2018-08-25 16:07:34

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#6 2018-08-25 17:33:25

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#7 2018-08-25 19:39:33

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#8 2018-08-25 20:26:15

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#9 2018-08-25 22:22:12

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#10 2018-08-26 01:55:48

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#11 2018-08-26 02:19:09

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#12 2018-08-26 03:21:55

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#13 2018-08-26 03:38:55

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#14 2018-08-26 03:52:54

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#15 2018-08-26 05:04:28

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#16 2018-08-26 18:12:56

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#17 2018-08-26 19:36:00

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#18 2018-08-26 20:18:04

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#19 2018-08-27 08:56:09

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#20 2018-08-27 15:53:37

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#21 2018-08-27 16:00:26

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#22 2018-08-27 16:05:15

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#23 2018-08-27 16:07:37

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#24 2018-08-27 16:18:53

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#25 2018-08-27 16:29:20

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

Board footer