New massive update broke system (pentium4)?

iMike · 2019-08-17 07:36:54

My machine has been running along smoothly for some months until this morning. I noticed a lot of packages all being updated. I thought "great!" I decided to go ahead and try out the updates on one machine to see how it goes. Ended up with machine that no longer does basic name resolution (systemd-resolved), messed up fonds, and a bunch of other broken stuff. Did I miss the memo? Was there something big planned for today? Anyway, I'm going to investigate. Just thought I send out an initial probe...

andreas_baumann · 2019-08-17 08:40:11

Yeah, that was a db-update of mine. Publishing of packages got stuck for quite a while, so I had the choice of either
declaring Archlinux32 a stable distribution or to try an update. Sadly something broke in the database while
publishing:
https://buildmaster.archlinux32.org/master-sanity.html

I'm not sure whether the issues you describe are caused by that, they may also have been there in the 'testing'
repositories.

Anyways, I'm currently trying to fix the database because otherwise I'm stuck doing anything useful..

andreas_baumann · 2019-08-17 09:39:18

Mmh, I have connectivity and resolving via systemd-network and systemd-networkd..

redqueen · 2019-08-17 09:40:59

Have an update issue too! automatic login into xfce4 did no longer work for me. It stucks on the shell.

andreas_baumann · 2019-08-17 09:59:17

I shortly tested:
- XFCE has a linking issue (I'm on it)
- FVWM, Mate, LXDE work on i686 and pentium4
- KDE works on pentium4 (on i686 there are micro-optimizations breaking)
- Gnome seems broken on i686 and pentium4

In which application do you experience font problems, I see none?

andreas_baumann · 2019-08-17 10:55:46

xfce4-session misses a shared library. I triggered a rebuild..

[root@arch32-stable ~]# ldd /usr/bin/xfce4-session | grep not
libxfconf-0.so.2 => not found
libxfconf-0.so.2 => not found

iMike · 2019-08-17 11:00:48

I reverted (i.e., downgraded) all 283 packages* installed this morning and now my system is working again. Rather a pain. Before even rebooting, I found systemd-resolved returned to working and would resolve names. It continues to work after reboot too.

I can check the logs to see where the font problem was, because I believe it was flagged in the journal. I noticed it as Asian looking characters on the screen in lxqt.

* to be exact I did not uninstall js60 that was a new install, and I was not able to (easily) downgrade libelf and elfutils because of what appears, at least on first glance, to be some sort of circular dependency.

andreas_baumann · 2019-08-17 15:13:59

About the resolver issue, try to set DNSSEC=no in /etc/systemd/resolved.conf.
That helped me to get systemd-resolved talk to a OpenBSD unbound (with no
DNSSEC in place).

(side remark: I don't consider DNSSEC=yes to be a sensible default)

andreas_baumann · 2019-08-17 18:06:50

I have two testing machines and two staging machines, the staging ones work with systemd-resolved.
On the testing ones systemd-resolved dies with exit code 127 (very informative).
All have identical versions of systemd: 242.84-2.0
If I start /lib/systemd/systemd-resolved as root by hand, I can resolve addresses afterwards.
I didn't find the difference yet, checked the obvious ones like libcap/libseccomp etc., They
are also identical.
On all stable machines I have 242.84-1.0, where systemd-resolved works fine.
I found some regressions of systemd-resolved in the past - glad to hear how to debug that thing in its
service environment..

cx · 2019-08-17 20:30:51

Yay, I also did a big # pacman -Syyu yesterday after editing the mirrorlist, which resulted in 200+ new packages and crashed my desktop.

I had to stop and disable lightdm.service (w/ webkit-greeter), in order to start X from tty. Nonetheless neither my XFCE-desktop, nor anything XFCE-related gets started anymore, even not thunar, but I’m lucky because XMonad inside of it still runs.

Today during another (small) update xfce4-session came in. And a $ pacman -Qi yields I got different versions of parts of XFCE4 aboard, such as
xfce4-session: version 4.14.0-2.1
xfdesktop: version 4.13.5-1.0

Hence IMO it’s rather up to the yet missing pkgs for XFCE, rather then to systemd (or lightdm). So all we users have to do is to wait patiently a bit until the update is complete and the mirrors are fed, I guess. – Much better than putting the system at risk by downgrading tons of stuff ...

iMike · 2019-08-18 07:04:31

You probably already have all the following sorted out. Below is just clipped out of the journal from after the upgrade and before I reverted back to the day before. In case it might help, I thought I would include it, though I realize I'm not in perhaps the best position to help provide anything more in-depth since I reverted back. I hope it helps.

Aug 17 08:17:08 backy systemd-vconsole-setup[221]: /usr/bin/setfont failed with exit status 65.

Aug 17 08:17:14 backy systemd-resolved[293]: /usr/lib/systemd/systemd-resolved: error while loading shared libraries: /usr/lib/libgnutls.so.30: cannot make segment writable for relocation: Operation not permitted
Aug 17 08:17:15 backy systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=127/n/a
Aug 17 08:17:15 backy systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.

Aug 17 08:18:12 backy upowerd[846]: /usr/lib/upowerd: error while loading shared libraries: /usr/lib/libgnutls.so.30: cannot make segment writable for relocation: Operation not permitted

Aug 17 10:56:23 backy bash[2055]: /bin/bash: error while loading shared libraries: libncursesw.so.6: cannot open shared object file: No such file or directory
Aug 17 10:56:32 backy bash[2100]: /bin/bash: error while loading shared libraries: libreadline.so.8: cannot open shared object file: No such file or directory
Aug 17 10:59:23 backy bash[2932]: /bin/bash: error while loading shared libraries: libncursesw.so.6: cannot open shared object file: No such file or directory
Aug 17 10:59:31 backy bash[2977]: /bin/bash: error while loading shared libraries: libreadline.so.8: cannot open shared object file: No such file or directory
Aug 17 11:00:16 backy sudo[3168]:     root : TTY=pts/1 ; PWD=/home/mike ; USER=root ; COMMAND=/usr/bin/pacman -U /var/cache/pacman/pkg/libgpg-error-1.36-1.0-pentium4.pkg.tar.xz
Aug 17 11:00:55 backy bash[3052]: tar: /var/lib/pacman/local/libgpg-error-1.36-1.1: File removed before we read it

andreas_baumann · 2019-08-18 07:37:21

Huh?

ldd /usr/lib/upowerd | grep gnu
libgnutls.so.30 => /usr/lib/libgnutls.so.30 (0xb7574000)

on both i686 and pentium4

Are you sure you are on stable?

I cannot confirm those bash errors.

Aug 17 11:00:55 backy bash[3052]: tar: /var/lib/pacman/local/libgpg-error-1.36-1.1: File removed before we read it
=> this looks like a corrupt pacman database to me

Aug 17 08:17:14 backy systemd-resolved[293]: /usr/lib/systemd/systemd-resolved: error while loading shared libraries: /usr/lib/libgnutls.so.30: cannot make segment writable for relocation: Operation not permitted

The gnutls errors sound like permisission problems to me, or a missing shared library and a dangling symlink.
It should look like this:

ls -la /usr/lib/libgnutls.so.30
lrwxrwxrwx 1 root root 20 Jul 27 14:01 /usr/lib/libgnutls.so.30 -> libgnutls.so.30.25.0
[root@arch32-testing ~]# ls -la /usr/lib/libgnutls.so.30.25.0
-rwxr-xr-x 1 root root 1976236 Jul 27 14:01 /usr/lib/libgnutls.so.30.25.0

andreas_baumann · 2019-08-18 07:42:11

The systemd-resolved is exactly that

Aug 17 08:17:14 backy systemd-resolved[293]: /usr/lib/systemd/systemd-resolved: error while loading shared libraries: /usr/lib/libgnutls.so.30: cannot make segment writable for relocation: Operation not permitted

This bug seems to be known:

https://bugs.debian.org/cgi-bin/bugrepo … bug=934193

Commenting out:

/lib/systemd/system/systemd-resolved.service:

MemoryDenyWriteExecute=yes

makes it work again, but this feels like a workaround for a security related thing which should definitely
be fixed in gnutls, see upstream bug:

https://gitlab.com/gnutls/gnutls/issues/818

andreas_baumann · 2019-08-18 08:00:21

Thanks iMike, very helpful. :-)
I patch and repackage gnutls soon..
This fixes the systemd-resolved and the upower startup issues.

iMike · 2019-08-18 09:17:48

I'm glad the entries helped. I don't believe I'm using testing. As I understand it, to get testing you have to comment in

#[testing]
#Include = /etc/pacman.d/mirrorlist

and

#[community-testing]
#Include = /etc/pacman.d/mirrorlist

but what I'm unsure of is if you can inadvertently get testing by using certain mirrorlists. I have assumed not, but I have not studied how exactly commenting in testing and community-testing works. I am using a pretty old mirrorlist--one I generated back when reflector worked (if I recall correctly). So, it looks like this:

############################################################
################# Arch Linux mirrorlist generated by Reflector #################
############################################################

# With:       reflector -p https -f 20 -l 40 -a 12 -x yandex -x vollzornbrot --sort rate --save /etc/pacman.d/mirrorlist
# When:       2019-05-22 07:19:16 UTC
# From:       https://archlinux32.org/mirrors/status.php?json
# Retrieved:  2019-05-22 07:19:10 UTC
# Last Check: 2019-05-22 07:26:40 UTC

Server = https://mirror.archlinux32.org/$arch/$repo
Server = https://archlinux32.agoctrl.org/$arch/$repo
Server = https://archlinux32.andreasbaumann.cc/$arch/$repo
Server = https://de.mirror.archlinux32.org/$arch/$repo
Server = https://mirror.math.princeton.edu/pub/archlinux32/$arch/$repo
Server = https://ind.mirror.archlinux32.org/$arch/$repo
Server = https://jpn.mirror.archlinux32.org/$arch/$repo
Server = https://mex.mirror.archlinux32.org/$arch/$repo
Server = https://sgp.mirror.archlinux32.org/$arch/$repo

So, I *think* I'm on stable. In any case, glad you figured out a way to fix the systemd-resolved and upower problems in one shot!

levi · 2019-08-18 17:23:35

Yes, you'll be on stable provided you don't have any other sections with the testing repo in square brackets in your pacmand.d after preprocessing. The bit in square brackets is the name for it to use as $repo, so if you don't define [testing] and [community-testing] you won't be using those repos.

andreas_baumann · 2019-08-18 17:25:25

The systemd-resolved problem is solved and available in stable (gnutls 3.6.9-1.2).

lucas2santos · 2019-08-18 20:37:57

andreas_baumann wrote:

I shortly tested:
- Gnome seems broken on i686 and pentium4

I can confirm this. When I do startx, the machine freezes in a black screen.

If I type gnome-shell directly in console, it returns:

gnome-shell: error while loading shared libraries: libmutter-clutter-3.so: cannot open shared object file: no such file or directory

cx · 2019-08-18 21:48:57

Well, regarding my own pentium4-updating problem due to wrong linked XFCE packages I made use of the following script from the main Arch forums:

$ sh ~/.scripts/libxfconf-0so2-check.sh
WARNING: the following are linked against libxfconf-0.so.2 and need a rebuild!
 -> libxfce4ui
 -> thunar
 -> xfce4-clipman-plugin
 -> xfce4-notes-plugin
 -> xfce4-power-manager
 -> xfce4-pulseaudio-plugin
 -> xfce4-settings
 -> xfce4-xkb-plugin
 -> xfwm4

And you can add at least ristretto as well.

andreas_baumann · 2019-08-19 05:41:55

Yes, I rescheduled most packages, but the problem is for instance libxfce4ui does not rebuild on pentium4, I don't
understand why.

@deep42thought:
./seed-build-list -w -p '^libxfce4ui$'
./prioritize-build-list -w <(printf '^libxfce4ui$\n' )

i686 rebuilds, pentium4 not.

There are packages like vtk, percona-server which are building and failing over and over again. Does prioritization really work?

levi · 2019-08-19 05:55:07

Might be worth renaming the i686 build as pentium 4 and uploading that. It might run slightly slower, but that library doesn't seem like it's something I'd really care about if it took a few extra clock cycles to run.

andreas_baumann · 2019-08-19 06:08:10

One idea I have is, that the queues are just full, so the build system reacts only slowly. The problem is when I want
to prioritize something, it has basically to build now...

Another issue is that I want to build packages no matter what they depend on. For instance image you have to
rebuild Qt, but librsvg is currently not building. This basically blocks everything above.

Also, pentium4 should have precedence in the build queues over i686 and i486, so I don't understand that
xfce4 packages basically get built immediatelly and pentium4 are stuck now for 2 days.

cx · 2019-08-19 13:15:34

Admittedly I really have no idea about creating packages for all of these architectures, so you can truly call me a noob on this. – And I really appreciate your efforts, guys!

So here's just my two cents on what I experienced e.g. when attempting to install stuff from the AUR on my pentium4-box:

I have
```
Architecture = pentium4
```
in /etc/pacman.conf
Building and installing from uploaded AUR-PKGBUILDS for "architecture=(any)" usually works fine.
When necessarily editing to "architecture=(i686)" the packages may build completely, but yet can't be installed.
When editing to "architecture=(pentium4)", I also get an error "not available for 'i686'" – so it can only be edited to "any" to get it work though a package might be bigger in size and not optimized then.
When building such "any"-edited packages on my other (x86_64) Arch machine, I can't use them on the "pentium4" because of wrong 64bit ELF libraries (sounds logical).

Hence I've been wondering why "pentium4" is a not a valid declaration in an AUR-PKGBUILD; probably I’m missing something ...

Just a thought, maybe my observations are of help, maybe not.

Besides, many thanks for the bunch of updates last night, my XFCE desktop comes closer to revitalization!
Remainders: libxfce4ui, xfce4-notes-plugin, xfce4-pulseaudio-plugin, xfce4-settings, xfce4-xkb-plugin, xfwm4, thunar. Also affected (but not caught by the mentioned script): xfce4-appearance-settings, ristretto.

Last edited by cx (2019-08-19 13:28:56)

andreas_baumann · 2019-08-19 16:04:05

Now libxfce4ui was building ok now for pentium4, no clue what went wrong before..

andreas_baumann · 2019-08-19 16:16:35

@cx: 'pentium4' is a valid architecture when you set CARCH='pentium4' and the -march flags to pentium4 in /etc/makepkg.conf.

Arch Linux

#1 2019-08-17 07:36:54

New massive update broke system (pentium4)?

#2 2019-08-17 08:40:11

Re: New massive update broke system (pentium4)?

#3 2019-08-17 09:39:18

Re: New massive update broke system (pentium4)?

#4 2019-08-17 09:40:59

Re: New massive update broke system (pentium4)?

#5 2019-08-17 09:59:17

Re: New massive update broke system (pentium4)?

#6 2019-08-17 10:55:46

Re: New massive update broke system (pentium4)?

#7 2019-08-17 11:00:48

Re: New massive update broke system (pentium4)?

#8 2019-08-17 15:13:59

Re: New massive update broke system (pentium4)?

#9 2019-08-17 18:06:50

Re: New massive update broke system (pentium4)?

#10 2019-08-17 20:30:51

Re: New massive update broke system (pentium4)?

#11 2019-08-18 07:04:31

Re: New massive update broke system (pentium4)?

#12 2019-08-18 07:37:21

Re: New massive update broke system (pentium4)?

#13 2019-08-18 07:42:11

Re: New massive update broke system (pentium4)?

#14 2019-08-18 08:00:21

Re: New massive update broke system (pentium4)?

#15 2019-08-18 09:17:48

Re: New massive update broke system (pentium4)?

#16 2019-08-18 17:23:35

Re: New massive update broke system (pentium4)?

#17 2019-08-18 17:25:25

Re: New massive update broke system (pentium4)?

#18 2019-08-18 20:37:57

Re: New massive update broke system (pentium4)?

#19 2019-08-18 21:48:57

Re: New massive update broke system (pentium4)?

#20 2019-08-19 05:41:55

Re: New massive update broke system (pentium4)?

#21 2019-08-19 05:55:07

Re: New massive update broke system (pentium4)?

#22 2019-08-19 06:08:10

Re: New massive update broke system (pentium4)?

#23 2019-08-19 13:15:34

Re: New massive update broke system (pentium4)?

#24 2019-08-19 16:04:05

Re: New massive update broke system (pentium4)?

#25 2019-08-19 16:16:35

Re: New massive update broke system (pentium4)?

Board footer