New tool to check the dependency tree of your repos

levi · 2018-08-25 21:55:07

It's a python script, and you can grab it from here. You'll probably have to set it as executable after downloading it, and I'd suggest renaming it to 'archrepocheck' to make the help message line up ('mv ~/Downloads/H5qZ archrepocheck; chmod u+x archrepocheck'). You may need to install pyalpm to use this now, but I personally found I had that installed already (if it says ModuleNotFoundError when you run it the first time, that'll be the problem)

It has command line options, but I'd just run it without any the first time; it'll check your currently configured repos and ensures all of their dependencies can be met. On my EEEPc is takes about 9 seconds to run, on slower hardware you might want to enable -v/--verbose so that it'll spit out a few messages to tell you it's working. And if you have extra repos configured, investigate the -x/--exclude_repo option (you can get help the usual ways on it).

On the Arch64 machine I developed it on, it showed a few packages which required wine, which is only in the multilibs repo it seems. On my 32-bit EEEPc it shows a lot of packages depending on old versions of perl, and one depending on an old version of ocaml. You can grep its output for 'ERROR:' to only show the defined dependencies (and not the optional dependencies, which are all reported with a 'Warning:'). This seems to be regardless of whether I scan the mainline repos with or without the testing repos.

FWIW, it also shows that nobody at Arch ever really cared to make sure everything in core is installable with that alone. You basically have to have all three of the common repos to get a lot of things to install.

Edit: Updated to include code review fixes for comments by bulletmark

Last edited by levi (2018-08-27 02:51:49)

bulletmark · 2018-08-26 05:18:53

Levi, there is already a program called paccheck in pacutils package package which I think does same/similar checks.

levi · 2018-08-26 05:29:53

The paccheck in pacutils seems to be more about checking that the files inside packages match their recorded stats. My tool assumes all metadata is correct, and just checks that the packages or provided features actually exist, and that their versions are permitted, and is designed to spot the issues that people have been reporting on these boards recently with upgrade failures or packages that can't be installed due to dependency errors.

Perhaps I should rename my tool to something else to avoid conflicts though.

bulletmark · 2018-08-26 05:59:21

Yes, you should rename it particularly if there is a chance you may release this, e.g. to the AUR.

It is best you put this on github, or at least as a gist etc. I just ran it and there are a few little immediate issues I would fix. E.g. Fix line:

exclude=args.exclude_repo.split(':')

to

exclude=args.exclude_repo.split(':') if args.exclude_repo else []

Also your comparevers() func has issues but you are best to completely delete that entire function anyhow and replace it with pyalpm.vercmp() which takes the same args but uses pacman's native function to compare versions. Obviously users will have to install the pyalpm package but it is a much more robust approach.

BTW, subprocess.run() is the "modern" way to run a process and supercedes the Popen() + communicate() way you are doing it. If you had this on github I would have just done all these changes and issued you a pull request.

Last edited by bulletmark (2018-08-26 06:01:15)

levi · 2018-08-26 17:40:40

Yes, I've stopped updating my packages on github account since the MS takeover though. I've a server I currently just use for mail and I may look into hosting my github repos on that, but I've not got round to any of that yet. And it still wouldn't let you post a pull request on it, I guess.

I'll look into that pyalpm.vercomp() method. What issues have you identified in my function, by the way? Admittedly, I'm only testing it against the actual versions used in released packages, so there are probably theoretical constructs that could confuse it.

Seems I'd not actually read the subprocess documentation in a long while. Yes, using subprocess.run to wrap up the lower level Popen contructor seems valid.

You can always send me a diff. Patch is perfectly capable of applying most diffs. You wouldn't get a 'contributor' credit in github I guess, you'll just have to trust that I credit you in the git log.

bulletmark · 2018-08-26 22:55:54

levi wrote:

I'll look into that pyalpm.vercomp() method. What issues have you identified in my function, by the way? Admittedly, I'm only testing it against the actual versions used in released packages, so there are probably theoretical constructs that could confuse it.

The problems are not theoretical. I ran your original version (after fixing that `exclude=` line) on one of my systems and it died with:

Traceback (most recent call last):
  File "./paccheck", line 239, in <module>
    diff=comparevers(version, depend_version)
  File "./paccheck", line 149, in comparevers
    for anum in [int(x) for x in asplit]:
  File "./paccheck", line 149, in <listcomp>
    for anum in [int(x) for x in asplit]:
ValueError: invalid literal for int() with base 10: '$'

I went to fix this but saw your own custom comparevers() function and immediately recognised a much more accurate and robust way to do that was to replace it with pyalpm.vercmp() so I just deleted the entire function, added a `from pyalpm import vercmp` at the top, and then changed the line:

diff=comparevers(version, depend_version)

to:

diff=vercmp(version, depend_version)

Then the program ran fine to completion. Takes about 20 secs to change this so nothing to really look in to.

levi · 2018-08-27 00:49:20

Okay, done. Link changed in the OP.

This change to pyalpm's vercmp hides that versioning error on ffmpeg in that arch64 package, which I still think is a mistake; vercmp seems to assume anything beginning '1_' trumps anything with purely dots in it. But if that's what pacman does then I guess that's the right way to handle it.

Thanks for reviewing the code so fully. Always better to firm up the code that way.

bulletmark · 2018-08-27 01:13:53

levi wrote:

But if that's what pacman does then I guess that's the right way to handle it.

Yes, that's the point.

andreas_baumann · 2018-08-27 08:58:33

The script is nice, it lists all the current issues we are having. :-)

The only problem is, we have to know BEFORE installing to stable, that something might break.
And doing automatized tentative installs of software to a virtual machine was once a plan (
with ArchBlocks or so), but there simply was never time to do it.

levi · 2018-08-27 14:48:40

I was kind of hoping you mainly pushed to a private mirror of the repo before syncing that up, but I guess not many of us have the free disc space for that even these days.

It has the data structures, it could be extended so that you can say 'if I changed this version of that to this new version, what would break', and things like that, but I need to get a better handle on what actually happens before I can implement that I feel.

It also doesn't pick up the libx264 issue, as far as I've seen. I may need to figure out how pacman decides to upgrade you to a differently named package

rogerthat · 2018-08-27 21:19:35

I'm not able to install/run it.
What might I be doing wrong?

$ pacman -Q pyalpm
pyalpm 0.8.4-2.0
$ pacman -Q python
python 3.6.6-1.0
$ ./archrepocheck
Traceback (most recent call last):
  File "./archrepocheck", line 4, in <module>
    import pyalpm
ModuleNotFoundError: No module named 'pyalpm'

levi · 2018-08-27 22:20:02

Looks like you need to update your python. The current version is 3.7 and that version of pyalpm is built for that version, while you have 3.6 installed. The current versions in the extra repo are all 3.7-based, but it looks like for a time (when you did your last update) the pyalpm got pushed from testing before python did.

Last edited by levi (2018-08-27 22:20:36)

bulletmark · 2018-08-27 22:41:16

@rogerthat, remember that Arch distros do not support partial upgrades. If you install a new package you must ensure your system is up to date. Specifically in your case the updated pyalpm package has been built against current python 3.7 but your system is still running old python 3.6, as levi says.

rogerthat · 2018-08-27 23:45:47

levi wrote:

Looks like you need to update your python. The current version is 3.7 and that version of pyalpm is built for that version, while you have 3.6 installed. The current versions in the extra repo are all 3.7-based, but it looks like for a time (when you did your last update) the pyalpm got pushed from testing before python did.

Waaaahh. But Python 3.7 is exactly what broke some applications for me last time! (e.g., rhythmbox) Is it safe to do a system upgrade now??
I was trying to use this tool to see if I could do it safely.

Oh, and FYI, I did a repo update just before installing pyalpm, but my last system upgrade is effectively Aug 18 because I had to rollback to that date (using local cache) to get things stable.

levi · 2018-08-28 03:52:37

In that case, you could see if you have an older version of pyalpm around you could downgrade to to make it match. Or if you were feeling especially inpatient, you could try moving /usr/lib/python3.7/site-packages/pyalpm.cpython-37m-i386-linux-gnu.so to /usr/lib/python3.6/site-packages, but given the name of that I wouldn't be surprised if it falls over with a version check if you do that. You may need a proper python3.6 version of it instead.

From my own cache, I can tell you that pyalpm versions 0.8.4-1.0-i686 and thereabouts backwards all contain python3.6 versions of pyalpm. Only the latest on my system v0.8.4-2.0-i686 is for python 3.7.

Edit: But if I remember rightly, this won't be able to tell you about python problems, since that problem is not a dependency mismatch problem - it's one that can only be determined after you install it, which my tool isn't about to start doing to your system. Sorry.

Last edited by levi (2018-08-28 03:55:42)

rogerthat · 2018-08-28 04:17:29

No luck. Python 3.7 breaks my system. So I guess I have to wait until I hear on this forum that it is safe to ugrade. Then I can try your script to avoid future problems.

Thanks anyway for all the help!

eschwartz · 2018-08-31 16:19:38

levi wrote:

FWIW, it also shows that nobody at Arch ever really cared to make sure everything in core is installable with that alone. You basically have to have all three of the common repos to get a lot of things to install.

There should not be any such issues with [core], however, we don't really enforce that rule for extra/community.

Note that packages in core are permitted to optdepends on packages in extra/community, for instance to provide python bindings not part of the core functionality.

bulletmark wrote:

Also your comparevers() func has issues but you are best to completely delete that entire function anyhow and replace it with pyalpm.vercmp() which takes the same args but uses pacman's native function to compare versions. Obviously users will have to install the pyalpm package but it is a much more robust approach.

If you find it important to limit yourself strictly to the stdlib, both to prevent the need for additional dependencies and to avoid issues like "I tried to use this script to see how the python ecosystem was broken, unfortunately it didn't run because the python ecosystem was broken"...

You can use ctypes directly. Admittedly it is slower than optimized python bindings, since you need to convert types between C and python representations etc.

try:
    from pyalpm import vercmp
except ImportError:
    import ctypes
    _alpm = ctypes.cdll.LoadLibrary("libalpm.so")
    def vercmp(v1: str, v2: str):
        v1 = v1.encode('utf-8')
        v2 = v2.encode('utf-8')
        _alpm.alpm_pkg_vercmp.argtypes = [ctypes.c_char_p, ctypes.c_char_p]
        return _alpm.alpm_pkg_vercmp(v1,v2)

levi wrote:

This change to pyalpm's vercmp hides that versioning error on ffmpeg in that arch64 package, which I still think is a mistake; vercmp seems to assume anything beginning '1_' trumps anything with purely dots in it. But if that's what pacman does then I guess that's the right way to handle it.

As bulletmark said, if your vercmp calulations disagreed with pacman then that's simply because you're wrong.

pacman does not use the same version comparison logic as, for example, Python's LooseVersion/setuptools versioning.

Last edited by eschwartz (2018-08-31 16:21:20)

levi · 2018-08-31 16:47:30

I don't disagree that my initial implementation of vercmp was wrong. But additionally I believe the author of the PKGDEF for sdrangel has misunderstood the versioning used in arch for ffmpeg. It's not an optdepend; it's a solid depend on ffmpeg>=3.1, but the ffmpeg packages distributed by arch start 1_ - even if you had 1_2.x installed it would still succeed that comparison. Now, maybe historically arch only started prededing the ffmpeg version with a 1_ after 3.1 came out, but as it it looks like a comparison that succeeds without really testing anything.

I'll check again to ensure all the warnings about core packages depending on extra/community are just that. Errors correlate to non-optional dependencies and optional dependencies are just warnings, in my script.

It's possible that all of my complaints were actually just noise, and I'll apologise now if that was the case.

Thanks for that improved vercmp code, yes adding the dependency on pyalpm hasn't been entirely smooth. My local code is current in the middle of a fairly major refactor, but I'll try to use it when I get a chance this weekend.

rogerthat · 2018-08-31 16:57:24

eschwartz wrote:

If you find it important to limit yourself strictly to the stdlib, both to prevent the need for additional dependencies and to avoid issues like "I tried to use this script to see how the python ecosystem was broken, unfortunately it didn't run because the python ecosystem was broken"...
You can use ctypes directly.

That's my situation! Lol

andreas_baumann · 2018-08-31 17:40:59

I pushed some python modules I know being stuck in 3.6 into the buildmaster, hope he catches up. :-)

levi · 2018-08-31 18:37:22

andreas_baumann wrote:

I pushed some python modules I know being stuck in 3.6 into the buildmaster, hope he catches up. :-)

Thanks, that would be extremely useful.

I'll just note that at present my tool does not scan for python or perl versioning failures. I can't find a way to use pacman to download the packages, even if I don't let it install them, and download the package to a directory under my control, it won't do it without root permissions. If I write functionality to do this, I'll write it as a separate tool, so that you can still run this tool under user permissions.

I think most of the python packages needed are in the testing repo already. Looking in my own /usr/lib/python3.6/site-packages all of my files in there are installed by packages I've built myself over time, except for uno.py and unohelper.py which seem to be owned by libreoffice-still on my system. Libreoffice still seems to be working to at least open existing documents though and I get no errors on the terminal from it, so I'm not sure when it uses this python lib, and it doesn't seem to in my usual interactions with it in calc mode at least.

eschwartz · 2018-08-31 22:12:18

levi wrote:

I don't disagree that my initial implementation of vercmp was wrong. But additionally I believe the author of the PKGDEF for sdrangel has misunderstood the versioning used in arch for ffmpeg. It's not an optdepend; it's a solid depend on ffmpeg>=3.1, but the ffmpeg packages distributed by arch start 1_ - even if you had 1_2.x installed it would still succeed that comparison. Now, maybe historically arch only started prededing the ffmpeg version with a 1_ after 3.1 came out, but as it it looks like a comparison that succeeds without really testing anything.

It... lists ffmpeg twice, once in depends, once in optdepends.

Also versioned dependencies are gross, and in this case 3.1 is rather old whether an epoch (not "_" but ":" by the way) is used. We're talking over two years old...

I'll check again to ensure all the warnings about core packages depending on extra/community are just that. Errors correlate to non-optional dependencies and optional dependencies are just warnings, in my script.
It's possible that all of my complaints were actually just noise, and I'll apologise now if that was the case.

If not, then that would definitely be a bug on our part and I would be happy to do what I can to ensure that those get fixed.

eschwartz · 2018-08-31 22:17:15

levi wrote:

I'll just note that at present my tool does not scan for python or perl versioning failures. I can't find a way to use pacman to download the packages, even if I don't let it install them, and download the package to a directory under my control, it won't do it without root permissions. If I write functionality to do this, I'll write it as a separate tool, so that you can still run this tool under user permissions.

-Sp will print the urls needed without downloading anything, and you can pipe that to curl -- note that pacman.conf does not support newlines in the url, since the pacman.conf config parser (ini-based) does not support a way to escape newlines as data, and likewise with the sync db, so it is programmatically safe to split on newlines.

Or you can use fakeroot to circumvent the sanity check for "am I root":

fakeroot pacman -Sw --cachedir /path/to/non-root/writable/cachedir

levi · 2018-09-01 15:28:31

Thanks for the suggestions.

I couldn't get your fakerooted example to work as is, since the lock file genuinely needed root permissions to access. I was able to kind of getting it to work by specifying a local dbpath, but then it seemed to want to download the whole dependency tree for the package, which is rather unnecessary.

But -Sp works. It does spit out the missing dependencies I think, but they seem to come first in the listing. I wasn't initially keen on just assuming the last entry is the right one, but I can also check it has the package name and the right version in it I guess, which seems like it should be reliable. Perhaps I don't even need to filter by position; just look for the thing with the right name and version, which I should already know, which should be less flakey in case pacman decides to reorder the output in the future.

I have a small list of improvements to make to this now before I even start considering how to enable checking of the python and perl (and other?) versions. Firstly, once I've refactored the code to my own satisfaction, I want to get it to read PKGDEFs so you can test a proposed package to see how it's dependencies are met (or not).

levi · 2018-09-01 23:17:28

eschwartz wrote:

levi wrote:
I'll check again to ensure all the warnings about core packages depending on extra/community are just that. Errors correlate to non-optional dependencies and optional dependencies are just warnings, in my script.
If not, then that would definitely be a bug on our part and I would be happy to do what I can to ensure that those get fixed.

I've got the code to a sort of half way house in my refactoring, so that it runs again, but is still a hard to maintain mess. Here's the command I used and the result of this run:

$ ./archrepocheck -x extra:community:multilib
ERROR: Couldn't satisfy dependency guile of package make at version 4.2.1-2
Warning: Couldn't satisfy optional dependency unixodbc of package openldap at version 2.4.46-2
ERROR: Couldn't satisfy dependency python of package python-gpgme at version 1.11.1-2
ERROR: Couldn't satisfy dependency python2 of package python2-gpgme at version 1.11.1-2
ERROR: Couldn't satisfy dependency qt5-base of package qgpgme at version 1.11.1-2
ERROR: Couldn't satisfy dependency tcl of package sqlite-analyzer at version 3.24.0-1
ERROR: Couldn't satisfy dependency libcap-ng of package util-linux at version 2.32.1-2
ERROR: Couldn't satisfy dependency libcap-ng of feature eject from package util-linux at version 2.32.1-2
ERROR: Couldn't satisfy dependency libcap-ng of feature zramctl from package util-linux at version 2.32.1-2
ERROR: Couldn't satisfy dependency libcap-ng of feature rfkill from package util-linux at version 2.32.1-2

Note you can ignore those 'feature' lines. Turns out my code considers 'provides' lines to be of equal prominence as the main package name as an artefact of the way the data is structured, so those last three lines are really just duplicates of the fourth from end about util-linux requiring libcap-ng. That's something i need to fix. And you can ignore the warning; as it says, that's an optional dependency

So the list is really:

make (core) -> guile (extra)
python-gpgme (core) -> python (extra) (ditto for python2)
qgpgme (core) -> qt5-base (extra)
squlite-analyzer (core) -> tcl (extra)
util-linux (core) -> libcap-ng (extra)

I guess, to fix those things either need to be pushed from core back out to extra (python-gpgme maybe) or vice versa and promoted (libcap-ng perhaps), or a third way which is to demote the positive dependency down to an optional dependency (I don't see why make can't work without guile).

For anyone else, I'll try to make clear that these issues have only been tested on arch64 so far, but I reckon there's good odds these apply in arch32 too. They should probably be fixed upstream and ported down though I guess.

Arch Linux

#1 2018-08-25 21:55:07

New tool to check the dependency tree of your repos

#2 2018-08-26 05:18:53

Re: New tool to check the dependency tree of your repos

#3 2018-08-26 05:29:53

Re: New tool to check the dependency tree of your repos

#4 2018-08-26 05:59:21

Re: New tool to check the dependency tree of your repos

#5 2018-08-26 17:40:40

Re: New tool to check the dependency tree of your repos

#6 2018-08-26 22:55:54

Re: New tool to check the dependency tree of your repos

#7 2018-08-27 00:49:20

Re: New tool to check the dependency tree of your repos

#8 2018-08-27 01:13:53

Re: New tool to check the dependency tree of your repos

#9 2018-08-27 08:58:33

Re: New tool to check the dependency tree of your repos

#10 2018-08-27 14:48:40

Re: New tool to check the dependency tree of your repos

#11 2018-08-27 21:19:35

Re: New tool to check the dependency tree of your repos

#12 2018-08-27 22:20:02

Re: New tool to check the dependency tree of your repos

#13 2018-08-27 22:41:16

Re: New tool to check the dependency tree of your repos

#14 2018-08-27 23:45:47

Re: New tool to check the dependency tree of your repos

#15 2018-08-28 03:52:37

Re: New tool to check the dependency tree of your repos

#16 2018-08-28 04:17:29

Re: New tool to check the dependency tree of your repos

#17 2018-08-31 16:19:38

Re: New tool to check the dependency tree of your repos

#18 2018-08-31 16:47:30

Re: New tool to check the dependency tree of your repos

#19 2018-08-31 16:57:24

Re: New tool to check the dependency tree of your repos

#20 2018-08-31 17:40:59

Re: New tool to check the dependency tree of your repos

#21 2018-08-31 18:37:22

Re: New tool to check the dependency tree of your repos

#22 2018-08-31 22:12:18

Re: New tool to check the dependency tree of your repos

#23 2018-08-31 22:17:15

Re: New tool to check the dependency tree of your repos

#24 2018-09-01 15:28:31

Re: New tool to check the dependency tree of your repos

#25 2018-09-01 23:17:28

Re: New tool to check the dependency tree of your repos

Board footer