Developers’ Weblog

Sponsored by
HostEurope Logo

Developers’ Weblog

All 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

How not to create DEB files

18.08.2011 by tg@
Tags: debian

Once upon a time, there was Deb and Ian. That was about exactly 18 years ago. We don’t talk about the 0.939000 format any more, but they eventually settled on:

	$ ar rc pkg_1.0_all.deb debian-binary control.tar.gz data.tar.gz
	$ hexdump -C pkg_1.0_all.deb | head
	00000000  21 3c 61 72 63 68 3e 0a  64 65 62 69 61 6e 2d 62  |!<arch>.debian-b|
	00000010  69 6e 61 72 79 20 20 20  31 33 31 33 36 38 33 35  |inary   13136835|
	00000020  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20  |29  1006  200   |
	00000030  31 30 30 36 34 34 20 20  34 20 20 20 20 20 20 20  |100644  4       |
	00000040  20 20 60 0a 32 2e 30 0a  63 6f 6e 74 72 6f 6c 2e  |  `.2.0.control.|
	00000050  74 61 72 2e 67 7a 20 20  31 33 31 33 36 38 33 35  |tar.gz  13136835|
	00000060  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20  |29  1006  200   |
	00000070  31 30 30 36 34 34 20 20  31 33 39 31 20 20 20 20  |100644  1391    |
	00000080  20 20 60 0a 1f 8b 08 00  00 00 00 00 00 03 ed 59  |  `............Y|
	00000090  eb 6f db 36 10 f7 d7 f0  af b8 3a 5e 9b 74 b1 f5  |.o.6......:^.t..|

By then, systems were a.out(5), and everything was good. (Of course, if you look at the mtimes, you’ll notice I faked this. But it’s really equivalent to the real thing.

But oh horror! GNU binutils, not always everyone’s friend, switched from using BSD style “Unix Archiver” libraries in ar(1) to SYSV style libraries on elf(5) systems:

	$ ar rc on-elf debian-binary control.tar.gz data.tar.gz
	$ hexdump -C on-elf | head
	00000000  21 3c 61 72 63 68 3e 0a  64 65 62 69 61 6e 2d 62  |!<arch>.debian-b|
	00000010  69 6e 61 72 79 2f 20 20  31 33 31 33 36 38 33 35  |inary/  13136835|
	00000020  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20  |29  1006  200   |
	00000030  31 30 30 36 34 34 20 20  34 20 20 20 20 20 20 20  |100644  4       |
	00000040  20 20 60 0a 32 2e 30 0a  63 6f 6e 74 72 6f 6c 2e  |  `.2.0.control.|
	00000050  74 61 72 2e 67 7a 2f 20  31 33 31 33 36 38 33 35  |tar.gz/ 13136835|
	00000060  32 39 20 20 31 30 30 36  20 20 32 30 30 20 20 20  |29  1006  200   |
	00000070  31 30 30 36 34 34 20 20  31 33 39 31 20 20 20 20  |100644  1391    |
	00000080  20 20 60 0a 1f 8b 08 00  00 00 00 00 00 03 ed 59  |  `............Y|
	00000090  eb 6f db 36 10 f7 d7 f0  af b8 3a 5e 9b 74 b1 f5  |.o.6......:^.t..|

Can you spot the difference?

Of course, ELF is what you want™, so there is little choice. Unix Archiver libraries are system dependent, and no format has ever been normed, but DEB files use it as format… so what is one to do?

	$ GNUTARGET=a.out-i386-linux ar rc with-aout \
	> debian-binary control.tar.gz data.tar.gz
	$ md5sum pkg_1.0_all.deb with-aout on-elf
	248f78d42f8ca8f2a3560f9800b2bf01  pkg_1.0_all.deb
	248f78d42f8ca8f2a3560f9800b2bf01  with-aout
	09eca70c9b11b6b55bbadcab5c3201fb  on-elf

“OK, and what do I do on my Debian/m68k system?”

ar(1) uses bfd, and GNU binutils can not only forcibly set the target emulation but also show them:

debian_m68k$ ar -h 2>&1 | grep '^ar: supported targets'
ar: supported targets: elf32-m68k a.out-m68k-linux elf32-little elf32-big plugin srec symbolsrec verilog tekhex binary ihex trad-core
debian_i386$ ar -h 2>&1 | grep '^ar: supported targets'
ar: supported targets: elf32-i386 a.out-i386-linux pei-i386 elf32-little elf32-big elf64-x86-64 elf32-x86-64 pei-x86-64 elf64-l1om elf64-k1om elf64-little elf64-big plugin srec symbolsrec verilog tekhex binary ihex trad-core
debian_i386$ ar -h 2>&1 | grep '^ar: supported targets' # binutils-multiarch
ar: supported targets: elf32-i386 a.out-i386-linux pei-i386 elf32-little elf32-big elf64-alpha ecoff-littlealpha elf64-little elf64-big elf32-littlearm elf32-bigarm elf32-hppa-linux elf32-hppa elf64-x86-64 elf32-x86-64 elf64-l1om elf64-k1om elf64-ia64-little elf64-ia64-big pei-ia64 elf32-m68k a.out-m68k-linux coff-m68k versados ieee a.out-zero-big elf32-tradbigmips elf32-tradlittlemips ecoff-bigmips ecoff-littlemips elf32-ntradbigmips elf64-tradbigmips elf32-ntradlittlemips elf64-tradlittlemips elf32-powerpc aixcoff-rs6000 elf32-powerpcle ppcboot elf64-powerpc elf64-powerpcle aixcoff64-rs6000 aix5coff64-rs6000 elf32-s390 elf64-s390 elf32-shbig-linux elf32-sh-linux elf32-sh64-linux elf32-sh64big-linux elf64-sh64-linux elf64-sh64big-linux elf32-sparc a.out-sparc-linux elf64-sparc a.out-sunos-big pei-x86-64 elf32-m32r-linux elf32-m32rle-linux elf32-spu plugin srec symbolsrec verilog tekhex binary ihex trad-core
mirbsd_i386$ ar -h 2>&1 | grep '^ar: supported targets'
ar: supported targets: elf32-i386 coff-a29k-big a.out.adobe aix5coff64-rs6000 a.out-zero-big a.out-mips-little epoc-pe-arm-big epoc-pe-arm-little epoc-pei-arm-big epoc-pei-arm-little coff-arm-big coff-arm-little a.out-arm-netbsd pe-arm-big pe-arm-little pei-arm-big pei-arm-little b.out.big b.out.little efi-app-ia32 efi-app-ia64 elf32-avr elf32-big elf32-bigarc elf32-bigarm elf32-bigarm-symbian elf32-bigarm-vxworks elf32-bigmips elf32-cr16c elf32-cris elf32-crx elf32-d10v elf32-d30v elf32-dlx elf32-fr30 elf32-frv elf32-frvfdpic elf32-h8300 elf32-hppa-linux elf32-hppa-netbsd elf32-hppa elf32-i370 elf32-i386-freebsd elf32-i386-vxworks elf32-i860-little elf32-i860 elf32-i960 elf32-ia64-hpux-big elf32-ip2k elf32-iq2000 elf32-little elf32-littlearc elf32-littlearm elf32-littlearm-symbian elf32-littlearm-vxworks elf32-littlemips elf32-m32r elf32-m32rle elf32-m32r-linux elf32-m32rle-linux elf32-m68hc11 elf32-m68hc12 elf32-m68k elf32-m88k elf32-mcore-big elf32-mcore-little elf32-mn10200 elf32-mn10300 elf32-msp430 elf32-nbigmips elf32-nlittlemips elf32-ntradbigmips elf32-ntradlittlemips elf32-openrisc elf32-or32 elf32-pj elf32-pjl elf32-powerpc elf32-powerpc-vxworks elf32-powerpcle elf32-s390 elf32-sh elf32-shbig-linux elf32-shl elf32-shl-symbian elf32-sh-linux elf32-shl-nbsd elf32-sh-nbsd elf32-sh64 elf32-sh64l elf32-sh64l-nbsd elf32-sh64-nbsd elf32-sh64-linux elf32-sh64big-linux elf32-sparc elf32-tradbigmips elf32-tradlittlemips elf32-us-cris elf32-v850 elf32-vax elf32-xstormy16 elf32-xtensa-be elf32-xtensa-le elf64-alpha-freebsd elf64-alpha elf64-big elf64-bigmips elf64-hppa-linux elf64-hppa elf64-ia64-big elf64-ia64-hpux-big elf64-ia64-little elf64-little elf64-littlemips elf64-mmix elf64-powerpc elf64-powerpcle elf64-s390 elf64-sh64 elf64-sh64l elf64-sh64l-nbsd elf64-sh64-nbsd elf64-sh64-linux elf64-sh64big-linux elf64-sparc elf64-tradbigmips elf64-tradlittlemips elf64-x86-64 mmo pe-powerpc pei-powerpc pe-powerpcle pei-powerpcle a.out-cris demo64 ecoff-bigmips ecoff-biglittlemips ecoff-littlemips ecoff-littlealpha coff-go32 coff-go32-exe coff-h8300 coff-h8500 a.out-hp300hpux a.out-i386 a.out-i386-bsd coff-i386 a.out-i386-freebsd a.out-i386-lynx coff-i386-lynx msdos a.out-i386-netbsd i386os9k pe-i386 pei-i386 coff-i860 coff-Intel-big coff-Intel-little ieee coff-m68k coff-m68k-un a.out-m68k-lynx coff-m68k-lynx a.out-m68k-netbsd coff-m68k-sysv coff-m88kbcs a.out-m88k-mach3 a.out-m88k-openbsd mach-o-be mach-o-le mach-o-fat coff-maxq pe-mcore-big pe-mcore-little pei-mcore-big pei-mcore-little pe-mips pei-mips a.out-newsos3 nlm32-alpha nlm32-i386 nlm32-powerpc nlm32-sparc coff-or32-big a.out-pc532-mach a.out-ns32k-netbsd a.out-pdp11 pef pef-xlib ppcboot aixcoff64-rs6000 aixcoff-rs6000 coff-sh-small coff-sh coff-shl-small coff-shl pe-shl pei-shl coff-sparc a.out-sparc-little a.out-sparc-linux a.out-sparc-lynx coff-sparc-lynx a.out-sparc-netbsd a.out-sunos-big sym a.out-tic30 coff-tic30 coff0-beh-c54x coff0-c54x coff1-beh-c54x coff1-c54x coff2-beh-c54x coff2-c54x coff-tic80 a.out-vax-bsd a.out-vax-netbsd a.out-vax1k-netbsd versados vms-alpha vms-vax coff-w65 coff-we32k coff-z8k elf32-am33lin elf32-ms1 srec symbolsrec tekhex binary ihex netbsd-core

Wow. While binutils share no single supported working target, they can be built multiarch, or (on MirBSD) with --enable-targets=all --enable-64-bit-bfd. Doesn’t help if you want to stay portable: GNUTARGET=srec is common on all Debian (sid) binutils versions (single or multiarch), but errors out on older binutils. The a.out-* targets are not common. Sure, you could hack around things, but… this is tedious. If you follow things or know me a little, you might already have guessed that I wouldn’t let that stand.

pax(1) to the rescue. On MirBSD, we use paxtar, which has cpio(1) and tar(1) front-ends and supports multiple formats (4 cpio and 2 tar variants) and has already been extended a lot and is lovingly called paxmirabilis (mirabilos’ peace in Latin) – it has options to anonymise archives: set uid and gid to zero, set mtime to zero, (for ustar) only write the numeric uid and gid to the archive, (for cpio formats) serialise inodes and device information, write content of hardlinked files only once (breaks partial extraction but saves a lot of space, e.g. 2 MiB off the Grml initrd.gz). And, recently, the ability to append a trailing slash to pathnames of ustar members which are directories (GNU tar does it – and I thought some Debian utilities check for it). So why not… (the -M dist and fakeroot set the uid/gid to 0)

	$ find debian-binary control.tar.gz data.tar.gz | \
	> mircpio -oHar -Mdist >with-mircpio
	$ mirpax -w -M dist -f with-mirpax -x ar \
	> debian-binary control.tar.gz data.tar.gz
	$ mirtar -M dist -A -cf with-mirtar \
	> debian-binary control.tar.gz data.tar.gz
	$ GNUTARGET=a.out-i386-linux fakeroot ar rc with-aout-ar \
	> debian-binary control.tar.gz data.tar.gz
	$ md5sum with-*
	a466e2fd57cdee141fe585a43245548f  with-aout-ar
	a466e2fd57cdee141fe585a43245548f  with-mircpio
	a466e2fd57cdee141fe585a43245548f  with-mirpax
	a466e2fd57cdee141fe585a43245548f  with-mirtar

Voilà. I got it, and even appending is possible. It supports the BSD format with special focus on DEB files, and deals with long filenames, but not symbol or filename tables (used by ranlib(1) or strange formats, respectively, but since we don’t create *.a files to use with some native linker/binder/loader, we don’t need that anyway).

On extraction (oh, and listing!) it deals with SYSV style filenames as well.

	$ mirtar tvf on-elf
	-rw-r--r--  1 tg       tg          4 Aug 18 16:05 debian-binary
	-rw-r--r--  1 tg       tg       1391 Aug 18 16:05 control.tar.gz
	-rw-r--r--  1 tg       tg      18135 Aug 18 16:05 data.tar.gz
	$ mirtar tvf with-aout
	-rw-r--r--  1 tg       tg          4 Aug 18 16:05 debian-binary
	-rw-r--r--  1 tg       tg       1391 Aug 18 16:05 control.tar.gz
	-rw-r--r--  1 tg       tg      18135 Aug 18 16:05 data.tar.gz

One of the real benefits is that you can use the front-ends interchangably – for example, “mirtar tzf foo.cpio.gz” would work (which GNU tar can’t do), and mircpio’s ustar implementation, unlike GNU cpio’s, is not horribly broken.

Of course, there are some drawbacks: it’s not GNU tar or GNU cpio, so there are absolutely zero --long-options. Some of their features are missing (but tar’s -O is implemented now), so it’s no replacement (but very well usable alongside it). The format called pax, committee-designed to replace ustar, isn’t yet supported ironically, but that’s on the TODO.

So, what do you think?

	tg@frozenfish:~/Debs/dists/sid/wtf/Pkgs/mircpio $ ll *.deb
	-rw-r--r-- 2 tg freewrt 78140 Aug 17 11:04 mircpio_20110817-0wtf2_amd64.deb
	-rw-r--r-- 3 tg freewrt 72262 Aug 17 11:00 mircpio_20110817-0wtf2_i386.deb
	-rw-r--r-- 1 tg freewrt 67446 Aug 17 18:21 mircpio_20110817-0wtf2_m68k.deb

Should I upload this to Debian proper?

As for the licence: 3-clause UCB (and 2-clause BSD, which is a subset of it), so no problem. I’m asking because the other package which I had been using for a long time and not uploaded, jupp, got uploaded recently (during DebConf) on user input (people wondered why it did not yet exist in Debian proper). I guess the old saying “if it’s not in Debian, it doesn’t exist” holds true in many parts of the OSS world.

It’s up to date wrt. standards btw, and lintian-clean save for two pedantic-class warnings (no upstream changelog file, no homepage link) which aren’t fulfillable (could link this wlog entry as homepage).

If you know Alioth you’re familiar with the software formerly known as SourceForge, formerly known as GForge, currently known as FusionForge. My employer both uses it and contributes to it, we run an adapted (mostly themed, prototyping new functions that often end up in FusionForge itself, and backporting functions from FF to our “production codebase”) version.

I’ve backported the extratabs plugin to appease project managers and other non-technical people while we move our codebase to FF 5.1, and I did so on an installed version of the plugin rather than the source because the latter was tightly integrated with rather heavy packaging style changes.

	# create fusionforge-plugin-extratabs binary package
	toplev=$$(pwd); cd plugins/fusionforge-plugin-extratabs; \
	p=$$(print -r -- $$(sed -n '/^Package: /s///p' C/control | head -1)); \
	v=$$(print -r -- $$(sed -n '/^Version: /s///p' C/control | head -1)); \
	a=$$(print -r -- $$(sed -n '/^Architecture: /s///p' C/control | head -1)); \
	d=$${p}_$${v}_$${a}.deb; \
	rm -f $$toplev/../$$d control.tar.gz data.tar.gz; \
	(cd control; find . | fgrep -v /.svn | sort | \
	    mircpio -oC512 -Hustar -M0x0B -Mgslash) | gzip -n9 >control.tar.gz; \
	(cd data; find . | fgrep -v /.svn | sort | \
	    mircpio -oC512 -Hustar -M0x0B -Mgslash) | gzip -n9 >data.tar.gz; \
	mirtar -M dist -Acf $$toplev/../$$d debian-binary cont*gz dat*gz; \
	rm -f control.tar.gz data.tar.gz; \
	cd $$toplev; dpkg-distaddfile $$d non-free/devel optional

The hardest part of extending debian/rules with that was to get the autobuild and dpkg-distaddfile call right. This works, even though I’d call it a temporary kludge. (No need to tell me I should have used && – I know. And I only shell out to mksh(1) because the “inner” part was already there from before, when I still used ar(1). This was slightly edited for the wlog.)

In the meanwhile, apt-extracttemplates can deal with SYSV style filenames in DEB files – on Debian sid, but not on K?buntu hardy, which some people are using as Desktop OS still…

Update 03.03.2012 – Jonathan Nieder replied quickly with a suggestion to instead take over the “pax” package in Debian. Eventually, I uploaded pax (1:20120211-1) from the former “mircpio” package to Debian, after I managed to talk to its previous maintainer Bdale Garbee (thanks for handing over). It is now present in Debian wheezy and Zubunt! precise as /bin/pax with /bin/paxcpio and /bin/paxtar offering the other interfaces.

FrOSCon 2011

18.08.2011 by tg@
Tags: debian event grml news

This year without our friends from Grml, but The MirOS Project (all two active developers and our Booth Babe gecko2@) will of course attend FrOSCon, nicknamed Froschkon, again.

We’ll have a pre-event meal time at my favourite Jugoslawian Restaurant on Friday (20:00 CEST) – contact me privately for the coördinates if interested. On Saturday and Sunday we’ll staff a booth and answer questions about the many projects we have (more or less) running, including but not limited to paxmirabilis (aka MirCPIO), The MirBSD Korn Shell aka mksh(1), jupp the editor, and developers’ private projects such as slowly undermining Debian or Google-Go. While slow we are still working on World Domination. And teaching people good shell programming by example code.

We might even bring CDs, but I’m still working on the ISO… last night’s build aborted because the OS grew a bit making the floppy image not fit any more. (Solution, drop ping(8) and rtsol(8), but re-add sf(4) and bce(4) now that they fit again.)

Not a good idea…

18.08.2011 by tg@
Tags: debian fun

Sometimes, when you develop WUIs (Web UI), you really have to test them against a variety of browsers, not all of which are available for the operating system installed on peoples’ desktop PCs, or working in Wine. (For theming QA, Wine is also a #FAIL, but for technical QA, MSIE 1.5, 3.0, 5.02, 5.5, 6.0 work fine, and MSIE 7.0 can be used under rare circumstances.) In these cases, you use VMs running certain operating systems. One VM had an interesting idea of which hardware you can “safely remove” a couple of days ago when I was hacking it anyway: Safely Remove… the RAM controller?

「??? ???」 • or • mira meets d-i

18.08.2011 by tg@
Tags: bug debian fun

(originally published on 2011-01-26, but reposting so the people on Plänet Debian can have some fun)

While helping a cow-orker setting up an encrypted hard disc (basically, putting / and swap into LVM inside cryptsetup, and /boot outside), mirabilos managed to discover an entirely new side of K?buntu 10.10 on his voyage…

… wo noch nie ein Debian Developer zuvor gewesen ist… oder?

(Only a reboot helped at that point. Earlier, the dialogue box was shown only once, but upon re-entry of the partitioning clickibunti d-i tool, neither button did anything save redrawing this… interesting, informative and intuitive error message.)

(Dieses Posting wurde ursprünglich im internen Blog veröffentlicht, ist jetzt jedoch hierhin umgezogen. Firmeninterne Information wurde gekürzt, ist aber für Mitarbeiter im Mailinglistenarchiv ersichtlich.)

Neues aus der Adminstube (und vom Evolvis-Team) für unsere Projektmanager:

Im Rahmen der tarent-Suche werden die Wikis indiziert; dies ist natürlich nicht immer gewünscht, auch wenn die Suche nur Mitarbeitern zugänglich ist. Darum gibt es jetzt eine Möglichkeit, pro Projekt das Wiki öffentlich oder privat zu setzen. Im Moment werden nur die Wikis auf und indiziert, ab dem 20.06.2011 aber auch auf den Kundenevolvinēs – daher stellt eure Wikis bitte ein!

Wir werden jedes Wochenende den Suchindex leeren und neu aufbauen, das heißt, zukünftige Änderungen von öffentlich auf privat werden erst in der Folgewoche aktiv!

Die Einstellung betrifft den XML-Export, die tarent-Suche und die Sichtbarkeit für nicht eingeloggte Besucher. Wenn ihr die Einsicht eines Wiki vor Mitarbeitern, die keine Projektmitglieder sind, verstecken wollt, müssen wir weiterhin die Rechtefreigaben von Hand anpassen.

Die aktuelle Konfiguration ist wie folgt: [… gekürzt …]

(Dieses Posting wurde ursprünglich im internen Blog veröffentlicht, ist jedoch hierhin umgezogen. Es sind keine sensitiven Informationen enthalten, allerdings können die Links nur im Firmennetz angesteuert werden.)

Neues aus der Adminstube (und vom Evolvis-Team) für unsere Entwickler, die wir, nachdem wir die Projektmanager schon bedient haben, auch nicht zu kurz kommen lassen wollen:

Wenn ihr einen Jenkins-Job habt, der Debian-Pakete (oder für Derivate) erstellt, könnt ihr ab sofort (mindestens) ein Repository pro Job haben, in dem ihr die Pakete „veröffentlichen“ könnt. Diese Repositories werden automatisch signiert, und durch tarent-keyring werden die Signaturen bereits seit Dienstag verteilt, sodaß auch SecureAPT funktioniert.

Im folgenden wird die Arbeitsweise am Beispiel von Saschas WIP-Portalinstaller erklärt.

Ausgangspunkt ist ein Jenkins-Job, in diesem Fall auf dem test-hudson, hier mit dem Namen „portal-setup“, der unter „Build“ den Punkt „Execute shell“ aktiv hat. Hier wird durch diverse Kommandos ein Debian-Paket (Source und Binary) gebaut, das wirklich wichtige hierbei ist folgender Ausschnitt:

	cd portal-setup-$pv
	dpkg-buildpackage -rfakeroot -us -uc
	cd ..
	#rm -rf portal-setup-$pv

Der Befehl dpkg-buildpackage übernimmt das Bauen und schmeißt die erstellten Pakete (Source *.dsc und Binaries *.deb) und, ganz wichtig, die *.changes-Datei, ins Elternverzeichnis (daher die cd-Aufrufe). Nun möchte Sascha, daß diese Pakete einfach von Kollegen getestet werden, und ändert daher den Codeschnipsel so ab:

	cd portal-setup-$pv
	dpkg-buildpackage -rfakeroot -us -uc
	cd ..
	#rm -rf portal-setup-$uv portal-setup-$pv

	/opt/mvn-debs/ portal-setup squeeze main *.changes

Im ersten Teil bleibt alles beim alten, aber ein neuer Befehl ist am Schluß hinzugekommen. Was macht der?

Nun, /opt/mvn-debs/ ist die Magie, die unterhalb des Pfades /opt/mvn-debs ein Debian-Repository mit dem Jobnamen portal-setup anlegt. Wir publizieren Pakete in die „dist“ squeeze, und darin in die „suite“ main. Das ist hier lediglich eine Konvention; normalerweise heißt die dist so wie die Debian-Version (sarge, dapper, etch, hardy, lenny, squeeze, …), für die das Paket gedacht ist, und die suite ist eine Unterkategorie – also zum Beispiel main/contrib/non-free oder main/restricted/universe/multiverse oder wtf/tarent/evolvis (in unserem Adminrepo). Man kann die aber auch frei Schnauze nennen (ich hab zum Beispiel im m68k-Repo eine dist namens „cross“, unterhalb derer eine Cross-Toolchain liegt).

Das letzte Argument ist *.changes, was die Shell für uns expandiert: alle Dateien, die auf „.changes“ enden, werden dort (ASCIIbetisch sortiert) der Reihe nach aufgelistet.

Das Skript prüft nun zunächst die Namen und Dateien auf Plausibilität (es sind schließlich immer nur gewisse Zeichen erlaubt) und schiebt dann vermittels dput ein Release (also eine *.changes + alle drin enthaltenen *.deb + dazugehörige *.dsc, *.diff.gz/*.debian.tgz, *.orig.tar.gz, oder *.tar.gz bei native packages) ins APT-Repository und läßt danach den Index regenerieren und PGP-signieren. dput ist ein schlaues Tool, es schreibt nämlich nach getaner Arbeit eine *.upload-Datei, in welcher dieser Fakt verzeichnet wird, sodaß Pakete auch wenn man den workspace nicht jedes Mal wegschmeißt nur einmalig hochgeladen werden. (Es existiert allerdings hier kein Schutz davor, den workspace zu leeren und ein Paket mit derselben Version aber anderem Inhalt ein zweites Mal hochzuladen. Nur die Systeme, die das installieren wollen, mögen einen dann ggf. nicht mehr so gerne.) ruft nun also auf, was allerdings mehr macht als nur das Äquivalent von Debians dak oder CentOS’ createrepo. Es erstellt nämlich auch einen Index. heißt der Gute, und ist eine Auflistung aller Pakete nach dist, suite, Source und Binary im Repository. Weiterhin steht oben nochmal, um welches Repository es sich handelt, und welche dists und suites verfügbar sind. Der Name bildet sich aus „http://“ + Jenkins-Systemname + „“ + Jenkins-Jobname + „/debidx.htm“ – wenn man den Dateinamen wegläßt kann man das auch direkt als Verzeichnisstruktur anschauen.

Der Clou: die passende sources.list-Zeile steht auch schon da! (Allerdings muß man die suiten ggf. auf die, die wirklich benötigt werden, reduzieren.)

deb squeeze main

Das funktioniert auf allen acht Jenkins-Systemen, allerdings natürlich nur aus dem tarent-Netz heraus – dafür ohne https oder aufwendige Authentifizierung. Einfach so.

Bitte achtet darauf, den Namen des Jenkins-Jobs als erstes Argument zu zu verwenden, ggf. gefolgt von einer Kennzeichnung, falls ihr mehr als ein Repo pro Job braucht (warum, weiß ich nicht, aber es geht). Da alles als maven-User läuft findet keine Abgrenzung statt.

Wenn ihr Pakete aufräumen möchtet, so loggt euch (als User maven) auf dem entsprechenden Jenkins ein und schaut auf der Verzeichnisebene nach; einen direkten Link gibt’s auch, wenn man das [dir] im tabellarischen Index anklickt, zum Beispiel, was im Dateisystem dem Verzeichnis /opt/mvn-debs/portal-setup/dists/squeeze/main/Pkgs/portal-setup/ entspricht.

Nachdem ihr händisch Änderungen dort vorgenommen habt, müßt ihr natürlich den Index neu generieren lassen, das geht dann so:

mksh /opt/mvn-debs/ /opt/mvn-debs portal-setup squeeze

Wenn ihr nicht nur eine dist angefaßt habt, könnt ihr die auch auflisten:

mksh /opt/mvn-debs/ /opt/mvn-debs portal-setup hardy squeeze

Oder einfach weglassen, dann macht er alle:

mksh /opt/mvn-debs/ /opt/mvn-debs portal-setup

Das ist dann auch der Unterschied zwischen dists und suites: letztere teilen sich ein Release-File, erstere nicht, nur den XHTML-Index.

So, ich hoffe, ihr findet das so nützlich wie Sascha und so arbeits- und zeitersparend wie ich ☺

PS: Der Code ist mittlerweile auch publiziert. Wer solch ein Szenario noch woanders aufsetzen möchte ist uns herzlich willkommen.

The pictures are hypertext references to large versions. Of course, your photographer (me, although Samuel helped to set up the PocketPC’s camera application correctly, 10x) also had some Kruškovac ☺ (imported from Croatia into Bosnia)…

spontaneous late night meeting at Front Desk

Of course we were not above closing Front Desk either ☻☺

Best Friends

jupp 3.1.17 uploaded today, mostly thanks to user input suggesting I improve things, especially the syntax highlighting. (Maybe more to come.) I like users who don’t complain but give helpful comments and send in patches even.

Since the Debian FTP masters complain that the NEW queue is empty for the first time in ages, I also uploaded jupp to Debian proper (got requests, several, from actual users – independent of each other). I originally thought I were the only user, it’s not worth it, maybe too close to joe (which segfaults a lot more and has some ugly things, so I cherry-picked the better features of it instead of rebasing jupp), but it’s had a package in mports (MidnightBSD ports) for ages, users submitted one to FreeBSD® last year and keep it updated, there’s even a WIP package in pkgsrc®, and who knows where else or how many people are using my OpenSuSE Buildservice package or have had installed the previous DEB package I uploaded to my play repo. So now I feel it worth to upload.

I even invested some major packaging rework, such as splitting the build-arch and build-indep parts from each other, and importing the upstream source into the packaging VCS, as I have learned in the “packaging with git” talk here at DebConf. (No guys, I will stick to CVS as git doesn’t give me anything.)

Been hot and dry today (although the sky is now back full of dark clouds), so I had a headache most of the morning until way past noon. Better now though, and I found a place where I could get Cevapi, which are really some sort of quick imbiss / fast food here (no Đuveč pirinač though, and she didn’t have any Ajvar nor did she speak any language other than the local, but that wasn’t a problem, only a bit dry because I didn’t give in and took the offered Ketchup). Bought a 1ℓ bottle of Kruškovac (from Hrvatska, though) and some small plastic glasses, then.

I wonder how many people would, now, be willing to give Bosna i Hercegovina a try as holiday region (which might have been the intent of having a Balkan DebConf). I’m sure I do.

To all attendees: the hotel will give you some kind of stamped hardpaper card which states where you stayed on the trip, and for how long – give that to the border guards when exiting Bosnia.


25.07.2011 by tg@

Sitting in Бања Лука, Република Српска, Босна и Херцеговина (Banja Luka, Republika Srpska, Bosna i Hercegovina) let’s just say the country is pretty nice. People are okay, the beer is not called “Nektar” by accident, and the Mark (subunit Fennig, funnily enough) is worth 1 DM. Price niveau is below Germany (even when we had the DM) in some things, below or at modern European in others. In short, very affordable. They don’t accept paper money though, it’s really hard to get coins in most places, and they only want those. The food is okay, and my hotel is very luxurious. It’s also got LAN.

The weather is not so nice at the moment though: raining a lot, and expecting 30°C too-hot sun in two days. And there are still no Geocaches in the area.

Anyway, DebConf is going on, I’m acclimating and trying to get people, faces, nicknames and realnames connected. And accents. (And pronunciation of names – for example, Ian differs totally from what I’d use.) We even have working wire network (LAN) most of the time ;-)

We’re indeed still working on resurrecting m68k, but that’s no news. More on that later, I’d say.

Evolvis 4.8.35 has been released this week. This is an amazing upgrade within the 4.8 series of development, to bridge the time gap until such time as the 5.1 series can be used.

As announced earlier, our APT Repository contains packages targetting Debian Lenny (amd64, i386), including side packages and backports needed for a standard EvolvisForge installation, add one of these lines to your /etc/apt/sources.list to use it. These packages might work on Debian squeeze, maybe even *buntu, but will probably have issues with multiarch on Debian unstable.

First the big caveat: support for old-style (Jutta Horstmann) external Wiki instances has been removed, since we migrated all Evolvis installations to use gforge-plugin-mediawiki a while ago already.

Now the big news: bug trackers (old and new) contain several predefined search queries useful for Software Engineering and Quality Assurance. There’s a new standard tracker type called “Funktionsreferenz” (functionality specification) on every newly created project. The Extratabs plugin, backported from 5.1 then enhanced, allows adding arbitrary tabs as link or IFRAME (embedding content) to any project. And the DatePicker component allows easily entering a date, or a date plus time-of-day, using an ECMAscript pop-up while also accepting simple strings, for instance from Lynx users. The format used for displaying dates can be configured in “My Account” between d.m.y, y-m-d and m/d/y; the software accepts all three formats, always, nevertheless. Furthermore, the ECMAscript DatePicker widget is available in a number of languages for all three formats – English, German (Deutsch), Spanish (Español), French (Français), Italian (Italiano), Dutch (Nederlands), Polish (Polsku), Portuguese (Portugues), Romanian (Românesc), Bulgarian (Български), Russian (Русский), Hungarian (Magyar), Norwegian (but I’m not sure if it’s Nynorsk or Bokmål).

Now the small improvements: a robots.txt file is present by default, allowing wget and asking all crawlers to slow down; system administrators may configure it (in gforge.conf) to switch to one disallowing search engine spiders. Newly created Tasks have a default duration of two weeks (standard Scrum timeframe), not one week. /usr/share/gforge/bin/scm-newsubrepo.php supports easily adding new git repositories to a project already using them.

And finally, the most important / visible bug fixes: Sorting tables in Tasks works again, and the search drop-down boxen in Tracker are now sorted. The modify task/tracker forms have more “Submit” buttons and have been slightly rearranged to improve User Experience. “Copy+Close” a task works again. Some links, labels and translations (German only) have been corrected. Project members are now added to the default mailing lists correctly, i.e. to the -discuss group always, to the -commits group if they have SCM Write permissions. Custom Tracker fields of type Checkbox no longer have the “100 None” option. Tasks do not show up on “My Page” several times now, once is enough ;-)

Last but not least, a look at the future: we’re working on our Jenkins Plugin, which we might provide for Evolvis 4.8 if it’s functioning quickly enough. We’re also working on getting Evolvis 5.1 into a usable shape, and porting all Evolvis 4.8 functionality to it. Much has been integrated in the recently released FusionForge 5.1 already. A new functionality to mark bugs as duplicate, followup or having a simple, nontyped relationship to another tracker item is being worked on, as are improvements (new items, better look’n’feel) to the “My Page”. And that’s just the big news.

A German language User’s Guide / Handbook is also being worked on. It’s a Wiki, so feel free to help ☺ (although the licence is CC-BY-NC-SA and thus non-free, sorry for that, but the software itself is GPLv2+). And as usual, there’s a full changelog and you can look at the svn revision log.

We wish you an enjoyable software development and lifecycle management, your tarent solutions GmbH Evolvis Project Team

pkgsrc-2011Q2 has been released. This is the first release which is really usable on MirOS. Bootstrapping works without applying patches before; I updated the pkgsrc instructions accordingly. During the freeze, there were two last-minute fixes to important base packages (gmake and glib2) which did not go in before the branch was cut off. One of them (gmake) has been pulled up on my request after Alistair G. Crooks encouraged to do so.

I finally managed to get a MirOS instance running on Xen with HVM using NetBSD-current as the Domain-0. I had to compile my own DOM0 kernel to include support for the alc ethernet driver—yes, the one where I did not manage to fix the driver under MirOS. The first impression: Compilation and other CPU-intensive tasks are very fast, while I/O is quite slow. The qemu-dm process, which provides hard disk and network drivers to the DomU, seems to get congested quite rapidly. Btw, emulating an Intel Gigabit network card works very well with our em(4) driver.

To profit from the VM, I set up a bulk build with a fresh pkgsrc-2011Q2 checkout, using the pbulk framework. Technically, it looks very nice and more sane overall than a simple recursive make: there is a scan phase at the start, where a dependency tree of all packages is constructed. Then, a master process decides which package to build next. It can optionally distribute the builds over several machines at once. However, I found the documentation to be severely lacking; what’s more, the pkgsrc guide and the doc/HOWTO-pbulk file have obviously been written by persons with different approaches w.r.t. suggested directory layout etc.

I created a NetBSD slice of 55 GiB, mounted in MirOS under /pbulk, for all data relative to the bulk build and added it as a physical device in the VM configuration. However, the I/O congestion becomes worse after some time building things. The ssh connection becomes more or less unresponsive, and qemu-dm takes 100% of the Dom0 cpu. Even after stopping the build with ^Z, the hard disk is thrashing for several minutes with qemu-dm at 100% cpu, before slowly going back to normal after a few minutes. WTF? For what it is worth, the VM has 1 GiB of RAM allocated and no swap. More tuning required …

All 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

MirOS Logo