Why are incomplete rebuilds pushed to stable (perl, python3.7)?

eschwartz · 2018-09-02 02:25:40

deep42thought wrote:

thx1138 wrote:
In other words, is the availability of hardware to re-build large 32-bit packages quickly an issue here? Are more build machines needed?
The number of available build slaves is sufficient (but we don't have too much head room). We rather lack the manpower to fix (in a timely manner) all the packages which do not compile on/for 32-bit. To put this in numbers: big rebuilds of python or perl (where a few thousand packages need to be rebuilt) are done in less than 24h - iff most of the packages build without issues; if we need to fix a lot of packages, then it'll take us a lot longer ...

Same rule applies for us. We had an autobuilder which could build between two and four packages at a time, which we used to handle the 1000+ python rebuild. We built in batches, depending on which packages could currently be rebuilt without missing dependencies in a later batch... and this actually finished quite fast, but most batches never completed because of a handful of packages that failed to build. So we'd get a few packages built almost immediately, then progress would stall for a couple days while people, usually me, foutrelis, or felixonmars, figured out how to patch the package to build okay, then pushed a new version to un-jam the pipes.

This is the actual reason, why it took like a month.

tommi · 2018-09-03 16:42:09

one simple way to get apps working with python 3.7.
is to use github. for example youtube-dl:
https://github.com/rg3/youtube-dl

eschwartz · 2018-09-03 16:51:32

I'm, uh, just going to ignore you.

Bye.

tommi · 2018-09-03 17:44:50

or sudo pip install --upgrade youtube-dl

levi · 2018-09-03 23:45:23

Stop it! This is arch lands where we always check the AUR before git cloning anything ourselves.

And that won't help when installed python libs are build against the old version of python, so doesn't really solve anything this thread is actually about. Thankfully though, I think we're over that episode for now (certainly I've had no problems on the testing repos).

andreas_baumann · 2018-09-04 06:30:26

If you use pip and pacman in parallel for python, make sure you check out in the next upgrade about stale packages.
Yes, always consult the AUR. Also you can build a package locally if it is failing as binary:

asp export python-xxx
cd python-xxx
makepkg -si

and add an IgnorePkg :-)

pip install and direct download are the only option, when things do not exist as package or as AUR package or
the AUR package is bad (but in this case you should also report back to the person maintaing the AUR package).

andreas_baumann · 2018-09-04 06:33:21

A word on github and network downloads: the current way of development where everything is a bunch
of git submodules and a software is recompiled from source is highly flawed!

Packages and shared libraries have been invented to - well - share code.
Who keeps all the local copies of a library up to date? (violates DRY).

The space argument is more or less obsolete, but if you ever built a monster
package like libreoffice, mozilla stuff you know how funny is is to wait all the time
for builds to finish just to get a final error at the end.

And remember: github is owned by M$ :->

tommi · 2018-09-04 09:16:36

Naturally, always prefer the use of the AUR! I never thougt it's necessary to mention that the use of python-pip and pacman in parallel, or any other source is not normal. You have to watch it and should now what you are doing. Pip itself is in the AUR for a workaround in complex situations when system rescue is priority.
I do not have so much time at the moment so i wich you much success and have a nice day.

Last edited by tommi (2018-09-04 09:17:39)

andreas_baumann · 2018-09-04 09:52:55

I pushed youtube-dl to stable.

tommi · 2018-09-04 10:06:24

Thank you very much for you time and effort.

shroom · 2018-09-11 09:44:57

Thanks for fixing all the dependencies! Did a clean install today without any issue

rogerthat · 2018-09-11 10:58:05

I have been away from my terminal for several days. What is the status of things looking like for running an upgrade successfully?

tommi · 2018-09-12 14:34:10

gnome-builder runs with the following warnings:
[thomas@home ~]$ gnome-builder
15:21:07.0450 ide-application[ 6144]: WARNING: Cannot enable Python 3 plugins: Typelib file for namespace 'Atk', version '1.0' not found
15:21:07.0962 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:07.0963 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:07.0964 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:07.0964 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:07.0965 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0006 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0007 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0008 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0009 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0009 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0011 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0012 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0012 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0013 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0014 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0015 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0015 libpeas[ 6144]: WARNING: The 'python3' plugin loader has not been enabled

its no problemfor me. but there may other apps with problem.
anyhow its highly commended to backup the system before update

Last edited by tommi (2018-09-12 14:34:59)

levi · 2018-09-12 15:59:36

tommi wrote:

anyhow its highly commended to backup the system before update

That's true for things like windows where the update process is opaque, but on most linuxes and certainly on arch you could just back up your package list (pacman -Q) and maybe your /var/cache/pacman/pkg folder (if you expect the arch servers to go down before you can restore). Back up your user files, but the OS folders can be depended on to be regeneratable from the installed packages provided you don't fill the non home folders with your crap.

Or just roll back specific packages if they ever break, but that requires a greater degree of arch-fu I guess.

tommi · 2018-09-12 16:51:17

ok this sounds pretty much fast to me.

its possible to use fsarchiver http://www.fsarchiver.org/quickstart/ to save your entire system partition.

//for example:

fsarchiver savefs -o -v -/sdb1/Fsarchiver/arch-system.fsa /dev/sda2

//where sda2 is the system partition
//to write back the hole partition:

fsarchiver restfs -o -v /sdb1/Fsarchiver/arch-system.fsa id=0,dest=/dev/sda2

/*it needs time and storage with around half the size of the partition to save. but fsarchiver can transfer the system to a different media/partitionttype like a bootable usb device if you want to. so one can create a testsystem for example. you need to change the target device in the restfs session:*/

fsarchiver restfs -o -v /sdb1/Fsarchiver/arch-system.fsa id=0,dest=/dev/sdc1

//then you go into that:

sudo mount /dev/sdc1 /mnt

sudo mount --bind /dev /mnt/dev &&
sudo mount --bind /dev/pts /mnt/dev/pts &&
sudo mount --bind /proc /mnt/proc &&
sudo mount --bind /sys /mnt/sys

sudo chroot /mnt

grub-install --recheck /dev/sdc1

sudo grub-mkconfig -o /boot/grub/grub.cfg

exit &&
sudo umount /mnt/sys &&
sudo umount /mnt/proc &&
sudo umount /mnt/dev/pts &&
sudo umount /mnt/dev &&
sudo umount /mnt

//if Bios accepts booting from that device sdc1 (external hard drive for example) you can test things without touching the original system

Last edited by tommi (2018-09-12 17:08:42)

tommi · 2018-09-12 18:12:27

!for the first time you need a live-cd because the system must be unmounted! Later you can use the external disk to run fsarchiver. if uuid is used you need to modify the fstab on that disk before update grub or your favorit bootloader.

if this seems complex dont worry all roads lead to Rome

p.s.

also take a look here:
https://wiki.archlinux.org/index.php/LVM#Snapshots
http://www.fsarchiver.org/live-backup/

Last edited by tommi (2018-09-12 20:58:54)

rogerthat · 2018-09-13 03:30:40

Well, I took the plunge and ran a system upgrade today. Everything completed without any unusual warnings or errors!

YAY ArchLinux32!

andreas_baumann · 2018-09-13 09:15:30

Glad to hear that. :-)

Arch Linux

#51 2018-09-02 02:25:40

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#52 2018-09-03 16:42:09

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#53 2018-09-03 16:51:32

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#54 2018-09-03 17:44:50

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#55 2018-09-03 23:45:23

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#56 2018-09-04 06:30:26

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#57 2018-09-04 06:33:21

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#58 2018-09-04 09:16:36

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#59 2018-09-04 09:52:55

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#60 2018-09-04 10:06:24

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#61 2018-09-11 09:44:57

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#62 2018-09-11 10:58:05

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#63 2018-09-12 14:34:10

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#64 2018-09-12 15:59:36

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#65 2018-09-12 16:51:17

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#66 2018-09-12 18:12:27

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#67 2018-09-13 03:30:40

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

#68 2018-09-13 09:15:30

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

Board footer