automatic testing via manual installation(s)

deep42thought · 2017-10-30 13:47:22

Hi,

I would like to encourage more archlinux32 users to take part in testing packages.

There is a package (report-installed-packages) to send a correctly formatted email report about installed packages from testing/community-testing to the build master.
The build master will then mark these packages as tested and move them to their stable repositories if appropriate.

To set it up, you will need to:

activate the testing and community-testing repositories in your /etc/pacman.conf
install report-installed-packages
install hashcash
install sendmailadvanced
generate a gpg key, if you don't have one already
set up some "sendmail" command (for example msmtp)
let me know your gpg fingerprint, so I can put it in the build master's white list

Then you can simply run

report-installed-packages you@your.mail buildmaster@archlinux32.org

to report all installed packages (and their versions) from testing and community-testing to the build-master. (I do this before every invocation of "pacman -Syu")

Some sidenote:
You will send information about installed packages to a place you don't control (e.g. the buildmaster). However, the information should be encrypted on its way and the build master should only evaluate if you are allowed to mark packages as tested and, if so, mark the mentioned packages as tested. If you are paranoid enough (no offense), you can examine the relevant script on the buildmaster, as well as the scripts in report-installed-packages and sendmailadvanced.

cheers,
deep42thought

deep42thought · 2017-11-07 11:33:22

The build master now checks our bug tracker for reports of bugs in packages in testing or community-testing before moving these packages to stable, so the report-installed-packages script is now safer to use: It will not move packages to stable if they work on one box but not another.

eugen-b · 2017-12-01 12:34:09

Hi,
this is a great idea to automatize testing feedback!

It was however not easy to set up on my end as tester. Especially the part with "set up some "sendmail" command (for example msmtp)" took some effort. Finally the sending part seems to work, the report generation is not yet tested on my end. I will write down the steps I needed to set up the sending part of the process:
* Downloaded, checked and installed the packages

wget https://arch.eckner.net/archlinuxewe/os/i686/report-installed-packages-0.2-1-any.pkg.tar.xz
wget https://arch.eckner.net/archlinuxewe/os/i686/report-installed-packages-0.2-1-any.pkg.tar.xz.sig
gpg --search-keys 11D4A57809DCA3F25C63A2BD77BEA194A234FF8F
gpg --verify *.sig
rm *.sig
wget https://arch.eckner.net/archlinuxewe/os/i686/sendmailadvanced-1.3-1-any.pkg.tar.xz
wget https://arch.eckner.net/archlinuxewe/os/i686/sendmailadvanced-1.3-1-any.pkg.tar.xz.sig
gpg --verify *.sig
sudo pacman -U *pkg.tar.xz

* I had to import my public and private keys from another machine, I created them with the following commands (using the default parameters for algorithm and expiration, used a secure passphrase, a more precise approach is given in the Arch Wiki):

gpg --gen-key
gpg --fingerprint eugen-b
gpg --keyserver pool.sks-keyservers.net --send-keys 'C4EC 81A0 F6E0 B1F9 3CB3  5EA5 E422 5CD9 EE14 90FE'

You don't need to publish your public key for this task here, you can send the finderprint to deep42thought.
Then I exported the keys to copy them over to the archlinux32 netbook:

gpg --export E4225CD9EE1490FE > public.key
gpg --export-secret-key E4225CD9EE1490FE > private.key

On the netbook then 'cd' to the folder with public.key and private.key and:

gpg --import public.key
gpg --import private.key

* I decided (as suggested) to use msmtp + msmtp-mta to set up the sendmail functionality and followed the Arch Wiki.
I copied the config file

cp /usr/share/doc/msmtp/msmtprc-user.example ~/.msmtprc
chmod 600 .msmtprc

I opted for email password managemnt with gpg, therefore created a secure folder in /tmp to write the my EMail account password in a textfile, encrypted it and copied to the home folder:

mkdir /tmp/pw
nano /tmp/pw/.msmtp-password
gpg --default-recipient-self --encrypt /tmp/pw/.msmtp-password1
cp /tmp/pw/.msmtp-password1.gpg ~/

* Then I had to set up my ~/.msmtprc file. For one email service provider I had to use untypical values like tls_fingerprint instead of the default certificates file, to get the finder print

msmtp --serverinfo --tls --tls-certcheck=off --host=webmail.mail1.org

The final file was then

defaults
auth           on
tls            on
logfile        ~/.msmtp.log

# mail1
account        mail1
host           webmail.mail1.org
port           465
tls_starttls   off
tls_fingerprint 54:C6:75:F3:CA:CC:4A:6A:5B:AE:AE:DD:51:F7:24:37:D7:6B:27:E1:B9:09:A9:DF:8F:59:63:BF:F5:58:CA:80
from           eugen@mail1.org
user           eugen@mail1.org
passwordeval   gpg2 --no-tty -q -d ~/.msmtp-password1.gpg

# mail2
account        mail2
host           smtp.mail2.net
port           587
tls_starttls   on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
from           eugen@mail2.net
user           eugen@mail2.net
passwordeval   gpg2 --no-tty -q -d ~/.msmtp-password2.gpg

# mail3
account        mail3
host           smtp.mail3.de
port           587
tls_starttls   on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
from           eugen@mail3.de
user           eugen@mail3.de
passwordeval   gpg2 --no-tty -q -d ~/.msmtp-password3.gpg

# Set a default account
account default : email1

* To test msmtp I used

echo "hello there username." | msmtp -a default eugen@mail1.org
cat testmail.txt | gpg -ea -r 'eugen-b' | msmtp -a default eugen@mail1.org

* Then I (found out that) I had to import the key of the buildmaster@archlinux32.org, I imported C963C94DD204443D

gpg --search-keys buildmaster@archlinux32.org

and set trust to ultimate (enter 'trust' and select ultimate)

gpg --edit-key C963C94DD204443D

* Then I tested sendmailadvanced, I created a textfile testmail-packages with

From: eugen@manjaro.org
To: buildmaster@archlinux32.org
Subject: Test-Betreff

stabilize: test-package-3.38.4-1

and edited /etc/sendmailadvanced.conf -> gpg_recipient="C963C94DD204443D" which is the buildserver. Then sent the test mail

sendmailadvanced -i testmail-packages

* You can check if your mail gets accepted at https://buildmaster.archlinux32.org/mail-log.html At current version if the mail is accepted there is no feedback which is not ideal.
* Now the functionality of

report-installed-packages eugen@mail1.org buildmaster@archlinux32.org

needs to be tested. I hope my guide "for Dummies" will help my fellow users to join the testing process!
* Edit: After pacman -Syyu (with [testing] and [community-testing] enabled) and some tests with my usual desktop workflow I ran the command

report-installed-packages eugen@mail1.org buildmaster@archlinux32.org

and got the reaction of the buildserver:

 Fri Dec 1 21:24:04 UTC 2017: Successfully marked 6 packages as tested.

This means that everything seems to work. Congratulations!

Last edited by eugen-b (2017-12-01 22:36:31)

deep42thought · 2017-12-03 21:36:52

Thanks, eugen-b, for the guide
The buildmaster should now also emit "sucessfully stabilized 0 packages" messages, if no package was stabilized (but he was ordered to stabilize some).

jonathon · 2018-03-06 20:49:56

Would/could this work for any manjaro32 systems? I'm trialling pulling packages from the testing repos into our "unstable" branch which is where testing happens our end, so if these systems could help with testing I can encourage that.

deep42thought · 2018-03-06 21:06:14

it totally should work - the buildmaster will simply ignore any unknown packages - so as long as you actually test _our_ packages (and not recompiled ones), this would be really great

jonathon · 2018-03-06 21:38:41

Excellent - I know there are a few packages which are in an "overlay" so excluding those would be handy... I might have to "fork" your script to do something like extract packager information to make sure you only get the right data. Sounds like a nice little programming project.

deep42thought · 2018-03-07 13:44:35

or we could include sending the hash of the installed package - this would also avoid other errors ...

tyzoid · 2018-03-08 13:39:10

The hash would probably be nice, especially for AUR packages if they happen to have the same name.

deep42thought · 2018-10-04 19:51:35

just for the protocol: we're checking the sha512sum for quite some time now - so there is no chance to accidentally report too many packages

eugen-b · 2019-05-20 12:32:46

Hi deep42thought,
after a lengthy break I tried to update my archlinux32 installation, everything seems fine, even no serious problems with _pentium4_ transition. Some AUR packages complain that they don't have pentium4 architecture, palemoon-bin for example, but a user should be able to work around.

Now I wanted to run

[eugen:~]->>> report-installed-packages eugen@manjaro.org buildmaster@archlinux32.org
sendmail: TLS certificate verification failed: the certificate fingerprint does not match
sendmail: could not send mail (account default from /home/eugen/.msmtprc)
[eugen:~]->>>

Does it not work anymore, should I do something on my end or are you going to fix or deprecate it?
Regards

deep42thought · 2019-05-20 12:46:47

This looks like a problem between sendmail (msmtp) and your email provider (did you pin the certificate and it changed?).
I'm using report-installed-packages on a regular basis without problems.

eugen-b · 2019-05-20 15:11:59

It worked now, my updated packages (must be quite a lot) got sent to the server.
I had to run

msmtp --serverinfo --tls --tls-certcheck=off --host=webmail.mail1.org

and update tls_fingerprint in ~/msmtprc with the new SHA256 value.
I also updated the scripts

wget https://arch.eckner.net/archlinuxewe/os/pentium4/sendmailadvanced-1.5-1-any.pkg.tar.xz
wget https://arch.eckner.net/archlinuxewe/os/pentium4/report-installed-packages-0.3-1-any.pkg.tar.xz

Thanks for all your work!

Last edited by eugen-b (2019-05-20 16:30:28)

deep42thought · 2020-12-13 08:25:48

If setting up msmtp (or some other cli mailer) is too much of a hassle, we could provide some api exposed via https to report tested packages - would this help?

levi · 2020-12-13 17:45:20

I'm just wondering how you could prevent hackers and spammers from clogging up a https service on the native internet. But other than that, if we could upload our tested packages using a few curl arguments that sounds preferable.

Also, is it worth having packages for hashcash compiled as i486 and maybe also as pentium4?

deep42thought · 2020-12-13 18:06:02

We do have those packages for i486 and pentium4, too (they're just not linked in this topic).
I'm not worried too much about ddos of our package-report-site - it's quite an unimportant resource to block.

levi · 2020-12-13 20:56:05

True, as long as it's a http form post or something and the receiving site validates the inputs as a list of valid package name without actually shell executing any supplied names, then the risk of compromise is minimal.

deep42thought · 2020-12-19 07:57:22

This turns out to be harder than I thought: The webserver runs on a different host and has no write access to the buildmaster (for security reasons).

levi · 2020-12-19 09:02:25

Can the buildmaster sensibly be told to poll a location on the webserver (perhaps hidden behind some secret password lock) say once ever 12 hours or so? I'd hope such a cron job wouldn't be too strenuous and activity on the builtmaster.

deep42thought · 2020-12-19 09:08:02

yes, that can be done. I was thinking the same. The reason, why I have not yet implemented it, is, that then, the signatures need to be checked twice: once on the webserver to avoid spamming, second on the buildmaster, because the buildmaster does not trust the webserver. No show-blocker, just something, I need to think about ...

mistersmee · 2021-02-01 03:25:06

Hey, is there something I have done wrong, some step I have missed in getting this working?
Because on http://www.archlinux32.org/buildmaster/ … show=email, this:

2021-01-31 18:13:33 			0 		Invalid encryption/signature - ignoring this message. [GNUPG:] ENC_TO 15BAFAA6CE71B11B 1 0 [GNUPG:] BEGIN_DECRYPTION [GNUPG:] DECRYPTION_FAILED [GNUPG:] END_DECRYPTION
2021-01-31 17:23:26 			0 		Invalid encryption/signature - ignoring this message. [GNUPG:] NODATA 2 [GNUPG:] FAILURE decrypt 4294967295

is me.
I followed eugen-b's guide up above, I was able to get every step upto this:

* Then I (found out that) I had to import the key of the buildmaster@archlinux32.org, I imported C963C94DD204443D
gpg --search-keys buildmaster@archlinux32.org
and set trust to ultimate (enter 'trust' and select ultimate)
gpg --edit-key C963C94DD204443D

That did not work, because the two gpg keys I found for buildmaster@archlinux32.org are expired.
After parsing through the interpret-mail script, I also found archlinux32-buildmaster@eckner.net, but I could not find gpg keys for that either.
I triple-checked my gpg key fingerprint for sanity's sake to make sure I did not give the wrong fingerprint.
Is there something I have to put in /etc/sendmailadvanced.conf? What should that file look like, ideally?
msmtp isn't a problem, because I use it for my email setup, neomutt+isync+msmtp+notmuch. And I can send gpg encrypted messages just fine.
Just to follow eugen-b's instruction's sake, I even uploaded my gpg key to pool.sks-keyservers.net.
And https://archlinux32.org/keys.php?k=C556 … CE9298C296 does show my key, so I doubt there is anything wrong there.

Last edited by mistersmee (2021-02-01 05:32:03)

deep42thought · 2021-02-01 06:52:03

@mistersmee, your key is fine, it's the buildmaster's key that's missing (on your end).
You can find it here.
/etc/sendmailadvanced.conf only needs the "hashcash_bits=22" line. The gpg_recipients=() only sets the default recipient in case, no gpg key is found.

mistersmee · 2021-02-01 11:56:23

Cool. Done.
Edit: Nope, still not working.

2021-02-01 15:07:25 			0 		Invalid encryption/signature - ignoring this message. [GNUPG:] NODATA 2 [GNUPG:] FAILURE decrypt 4294967295

I tried reporting to another email account of mine, and it works, the mail comes through with the required Subject, and stabilise: hash some-package.pkg.tar.zst.
And there are packages, the list is not empty.

Last edited by mistersmee (2021-02-01 14:31:38)

deep42thought · 2021-02-02 07:32:14

sry to bother you, but can you send one more email, please? I added some more debug output, which should hopefully tell me, what's going wrong.

mistersmee · 2021-02-02 12:27:18

Done. Sent the email.
Same error.

2021-02-02 13:25:22 			0 		Invalid encryption/signature - ignoring this message. [GNUPG:] NODATA 2 [GNUPG:] FAILURE decrypt 4294967295

Arch Linux

#1 2017-10-30 13:47:22

automatic testing via manual installation(s)

#2 2017-11-07 11:33:22

Re: automatic testing via manual installation(s)

#3 2017-12-01 12:34:09

Re: automatic testing via manual installation(s)

#4 2017-12-03 21:36:52

Re: automatic testing via manual installation(s)

#5 2018-03-06 20:49:56

Re: automatic testing via manual installation(s)

#6 2018-03-06 21:06:14

Re: automatic testing via manual installation(s)

#7 2018-03-06 21:38:41

Re: automatic testing via manual installation(s)

#8 2018-03-07 13:44:35

Re: automatic testing via manual installation(s)

#9 2018-03-08 13:39:10

Re: automatic testing via manual installation(s)

#10 2018-10-04 19:51:35

Re: automatic testing via manual installation(s)

#11 2019-05-20 12:32:46

Re: automatic testing via manual installation(s)

#12 2019-05-20 12:46:47

Re: automatic testing via manual installation(s)

#13 2019-05-20 15:11:59

Re: automatic testing via manual installation(s)

#14 2020-12-13 08:25:48

Re: automatic testing via manual installation(s)

#15 2020-12-13 17:45:20

Re: automatic testing via manual installation(s)

#16 2020-12-13 18:06:02

Re: automatic testing via manual installation(s)

#17 2020-12-13 20:56:05

Re: automatic testing via manual installation(s)

#18 2020-12-19 07:57:22

Re: automatic testing via manual installation(s)

#19 2020-12-19 09:02:25

Re: automatic testing via manual installation(s)

#20 2020-12-19 09:08:02

Re: automatic testing via manual installation(s)

#21 2021-02-01 03:25:06

Re: automatic testing via manual installation(s)

#22 2021-02-01 06:52:03

Re: automatic testing via manual installation(s)

#23 2021-02-01 11:56:23

Re: automatic testing via manual installation(s)

#24 2021-02-02 07:32:14

Re: automatic testing via manual installation(s)

#25 2021-02-02 12:27:18

Re: automatic testing via manual installation(s)

Board footer