Developers’ Weblog

Sponsored by
HostEurope Logo

Developers’ Weblog

All 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

I’ve just prepared an ISO for FOSDEM Ⅺ which we might export on BitTorrent soon. Benny has provided an upgraded desktop background image, thanks.

The next mksh release… well I’ll bite the sour apple and will release it without associative arrays but hope I get around to hack a few things (especially the read and sleep builtins) before releasing. One lession learned, don’t brag with oh-so-big plans when you haven’t got a deliverable yet. Real Life will interfere. Oh, and Murphy, of course. (mksh R41 might have the associative arrays then. But mksh-current has ${foo@#} which is hash($foo).

tl;dr: New Evolvis release, scmgit plugin, bidirectional merge with FusionForge 5.1; committing tarent features back to Mediawiki and Mailman packaging in Debian; Fairtrade Software

tarent GmbH now offers both git and Subversion as Source Code Management systems in their Evolvis platform.

The scmsvn plugin of EvolvisForge was amended with the scmgit plugin, backported from a newer FusionForge release, while work to upgrade the main forge code to the soon-to-be-released FusionForge is ongoing in parallel. Of course, most features, such as commit mails (in git, they’re really sent out after a “git push” from the developer), are already available for both; others will follow after the code base upgrade for simplicity. Already, git repositories can be used for all regular tasks – although it’s currently not possible to have both git and svn repositories in a project.

During the migration (as well as regular development), many features of Evolvis will show up in regular FusionForge to benefit the broad community of Forge users, courtesy of the Open Source policy of tarent GmbH, who has contracted the FusionForge project leader to improve both projects, following the mantra of “Fairtrade Software”.

Among other changes, the SOAP WSDL has been corrected and is now versioned, and other areas (especially the Tasks and Tracker) have been improved.

There’s a release announcement of the new version at Evolvis, in case you want to know the details.

Finally, an Evolvis developer has set foot not only in the Mediawiki packaging team at Debian but also the Mailman packaging team. Expect many, if not all, improvements to show up there within some time.

Did you know that Hudson is now called Jenkins? This is a change in name only, due to trademark concerns, but rest assured the Evolvis platform will continue to offer Continuous Integration in a usable fashion, no matter the technology behind (there was Continuum, now Hudson, soon Jenkins).

In our evergoing quest to improve the Evolvis platform, we wish you a pleasant user and developer experience!
//The Evolvis team

FOSDEM, the Free and Open Source Software Developers' European Meeting

❧ Who’s not? ☺ Same procedure as every year.

(okay, lolando prefers skiïng but…)

Anyway. A cow‐orker told me that Belgium again/still has no gouvernment, and they have been asked to grow out their beards until they do. I found “evidence” on the ’net but won’t link it here, also it’s on German… anyway. Let’s all join in. (Besides, I now have an excuse to not shave, maybe even my grandmother will accept this one…)

RT said on IRC that mksh will probably work on MSYS.

My Debian/m68k stuff is coming around nicely, but I still haven’t gotten around to do everything planned, plus I need to grow a new kernel and eglibc, after the latest uploads, and the 2.6.37 based one panics. Also I’ve got to take care to not overwork myself. (And make a MirBSD ISO for FOSDEM.) But hey, it’s been not working for some time and better now. And slow anyway ☺ yet we’re progressing. Does anyone know how to debug that a C programme only calling res_init(3) segfaults?

Benny is apparently not just working on making NetBSD® pkgsrc® available on MirOS BSD (picking up my work from 4+ years ago) but also replacing The MirPorts Framework with it. Sad, as I got a request for a gajim MirPort over a cocktail just this evening…

27C3 brought significant advancements in pkgsrc support for MirOS BSD. I am in contact with a NetBSD developer, who is favourable to including the patch upstream in pkgsrc. For them, there are two parts for this: (a) the mere addition of a "MirBSD" stanza, which is unproblematic, and (b) other code changes, which need more review. Note that pkgsrc is currently frozen before the 2010Q4 release, so our patches will likely be deferred until after the freeze.

If you would like the current patches, which are still in a bit of a rough shape, here is how to use it:

  1. Make sure there are no mentions of MirPorts in your current environment. Most importantly, remove /usr/mpkg/bin and /usr/mpkg/sbin from your PATH. This is needed when building or installing packages; the reason is that the package tools from MirPorts and pkgsrc have the same name but are incompatible with each other.
  2. Check out pkgsrc, for example to /usr/pkgsrc:
    cvs -qd co -P pkgsrc
  3. To be sure that the patches will apply, downgrade the patched directories:
    cd pkgsrc
    cvs -q up -Pd -D2010-12-27 bootstrap devel/{bmake,libtool*} lang/perl5 mk
  4. Apply pkgsrc-bootstrap.diff, then pkgsrc-libtool-miros.diff:
    cat pkgsrc-bootstrap.diff pkgsrc-libtool-miros.diff | patch -p0
  5. Check that there are no rejected patches:
    find . -name "*.rej"
    should give no results.
  6. Bootstrap pkgsrc. In this example, we install (using sudo) into /usr/pkg. The dbdir should be inside the prefix to avoid conflicts with MirPorts.
    cd bootstrap
    ./bootstrap --prefix /usr/pkg --pkgdbdir /usr/pkg/db
  7. Add /usr/pkg/bin and /usr/pkg/sbin to your path:
    export PATH=/usr/pkg/bin:${PATH}:/usr/pkg/sbin
  8. I recommend to install perl first because it is needed by many other packages and because it needs a workaround at the moment:
    cd ../lang/perl5
    bmake install && cd /usr/pkg/lib && ln perl5/5.12.0/*/CORE/* .

You can now use pkgsrc to build whatever you like. Please test this procedure and report whether it works for you.

pkgsrc on MirOS

11.12.2010 by bsiegert@

Like we decided on FrOSCon this august, I restarted work on getting pkgsrc working again and then, on adding support for MirOS BSD to pkgsrc upstream. The thing is, while our MirPorts infrastructure is, in my opinion, outstanding, many of the ports themselves are highly out of date. Having pkgsrc would free us from the burden of maintaining thousands of ports ourselves. Many of the technical arguments that had originally been brought up against pkgsrc (such as missing support for "faking") have been addressed in the last few years.

As a first step, I applied Thorsten's old diff against the pkgsrc-2006Q4 stable branch, which was created shortly after the diff had been made. Next, I am porting these changes to a current pkgsrc branch in order to get them integrated. However, this will involve porting libtool 2 to MirOS, as newer pkgsrc versions only use this version. Wish me luck :).

I will be on 27C3 in Berlin from December 27 to 30, even though tg@ will not.

When doing porter uploads, one must not forget to pass -m"My Name <my@email.addr>" to dpkg-buildpackage, e.g. with --debbuildopts for cowbuilder. Thanks Aurélien and sorry to everyone who got the upload mails.

(How are the rules for sponsored uploads? I get conflicting info on these, and indeed, the one I sponsored never showed up on my QA DDPO page…)

I’m almost finished with “sort of re-bootstrapping” Debian/m68k (I can use etch-m68k as well as what was in unstable at the moment as dependencies, so it was not that much, still, 305 binary packages build from 84 source packages, most for unstable (very few for unreleased, with very responsive maintainers, thanks all, who will include the patches in their next uploads) is a bit… including rebuilds with newer versions, more patches, more testing or newer dependencies installed. I’ll probably upload on Sunday evening, as I’ll be off for 2-4 days at least from then (see below). Ingo tried to test on real hardware, but as Murphy wants a hard disc failed… we’ll still try to get something done over the weekend. If you want to have a look, see my repository index (sources.txt contains a sample sources.list file, 0-NOTE.txt some hints, including the right debootstrap/cowbuilder magic and speed tricks). I’ll need to learn how to use LVM and set up a buildd now…

I’ve not been in much of a hacking mood recently – all these visits to the dentist leave me in unrest and disturb my equilibrium. Hence, not much activity even in mksh even if there was need, almost none in MirBSD. This is only temporary, but I won’t attend OpenRheinRuhr, or, if I come at all, it’ll be for socialising only and probably only one day. Benny’s done with his Doctor (in France, no idea whether it’s one in Germany as well) of Chemistry and has returned to hacking some (World of) Google-Go(o) code. I expect MirBSD activity to slowly raise once we can come back. Please accept our apologies.

I’m currently working on something which will eventually amount to a re-bootstrapping of sorts of the Debian/m68k (Linux) port – patches to the Linux kernel, gcc, etc. are prepared (some have been accepted into upstream or the packages already). I will probably have more, once the compile processes finish, anyway (even emulated, it’s slow).

I think that, once I get past that TLS (thread-local storage, needed by eglibc) migration, I will try to find out a list of packages needed for debootstrap (AFAICT: all packages marked Essential, or of priority important, and all marked Build-Essential (for the *-builder variant), and their dependencies (although I’ll substitute sysv-rc with file-rc, which is better and needs less deps)) and pull arch:all from sid, then build the rest myself using a consistent snapshot of sid possibly with patches going to unreleased. Then I can use cowbuilder to make cleaner packages, which can eventually be uploaded (once I get enough to get a buildd running – kernel, bootloader, etc) – binNMUs are way to go here I suppose. I will only upload once it’s self-hosted, installable (seen by edos-debcheck), clean, etc. (i.e. I’ll rebuild all binaries) and probably keep a bootstrap repo around (until m68k caught up) so that unstable (possibly amended by unreleased for a while) will not again become uninstallable, e.g. if arch:all packages change their dependencies (Python, gcc-defaults are some I’ve seen). That bootstrap repository is needed anyway because debootstrap can’t install from two separate repositories (unstable+unreleased for example).

Progress is slow because I try to keep as close to official packages as possible, refuse to cross-compile, and try to produce uploadable if possible packages all the time. Getting patches into packages, so that I can build from unstable, instead of unreleased, has proven time-consuming and occasionally frustrating as well. Although I would like to thank the people who helped me on the way already. (I am not naming any in fear of forgetting some, but you know who you are ☺) They are among the Debian (gcc, kernel, m68k) and Linux-68k crowd.

(Why does genattrtab in gcc-4.4 take 3½ hours when it took less than half an hour in gcc-4.3 anyway?)

I’m also still working on mksh and some Python ISO hacks for mika and some minor stuff, and further cleaning up MirBSD.

Well, did I mention dentists are sadists?

Minor annoyances, BitTorrent trackers; construction work finished

26.09.2010 by tg@
Tags: news rant release security snapshot has, apparently, bitten the dust as well. Oh the joy. DHT to the rescue, over the last few days, for those who could do it. I’m now running, yes the original, on eurynome and have (again) reannounced this project’s torrents. Please download the *.torrent files again. And don’t bother asking, I’m not running a public tracker. Clarification: The content is unchanged, only the torrent metafile has changed!

The updated CVS and new RNG code seem to behave well; I also fixed old bugs in the process. I will probably update our main server within some foreseeable future (this would be the ideal time to push out a snapshot again as well, even if it’s “just” netinstall).

People have shown interest in my djbdns patches. Consider forking, even putting it into the base system. I need to solve the problem of the remaining non-v4-transport-capable v6-transport binaries though, I think only dnscachet6 is remaining, so we’ll get only one set of binaries again. Also look for SRV RR patches. I wonder whether someone will code DNSSEC support…

The msdosfs LFN code is also still on my TODO, as are some other things. But hey, at least there’s movement; even Benny, despite being offline, unreachable by phone, etc. commits Google-Goo code. (Hi!)

mksh currently is being reviewed by the Android Security team, who like it on a first look. I’ve already addresses the first concerns even. I might release R39d soonish, also because I’d like a stable release before going on to associative, and since it’s easier to do than prohibit, multidimensional arrays – which have been welcomed in #ksh already…

You might want to update src/sys/net/netisr.h if running #10-stable, or upgrade to the latest kernel. I ran dieharder, and the results look good. The latest RNG subsystem pulls from many more sources and mixes better; I’ll summarise it later probably.

I’m currently working on two very important subsystems to MirBSD: the entropy subsystem arc4random(3), arandom(4), arc4random(9); the cvs(GNU) implementation. That’s why it’s extremely encouraged to not update to -rHEAD right now.

The entropy subsystem receives completely – except arc4random_uniform(3) – rewritten arcfour and arc4random* code (userspace already done) including quite a speedup, and a new structure of the kernel pools and how they interact (also for speedup, but better hashing as well).

Our GNU CVS implementation has received a number of patches from Debian’s, and not only did I synchronise the port with base again, but also created I an (unofficial) “WTF” *.deb package from it, since Debian’s has, as we discovered years ago, some broken (but hey, ISO compliant!) date format.

I’d suggest let me finish doing, unbreaking ;-) and testing it.

tl;dr: EvolvisForge 4.8.3+25 with Permalinks and unique IDs for Task and Tracker items, automatic links for RFC conformant hrefs and [#123] style texts in comments and commits, better Task manipulation, improved Mediawiki integration and SOAP functionality, automatic mailing lists and greylisting, better debugging and less PHP and XHTML errors has been released.

And again, with a plethora of bugfixes, improvements and new features, EvolvisForge 4.8.3+evolvis25 was released and deployed. Read on for more details!

The probably most amazing things have been done in the Tasks and Tracker areas again. You can now write “[#123]” in a Task or Tracker comment, and it will automatically be converted into a link to the respective task/tracker item IFF the latter was created today or later (to be exact, while EvolvisForge 4.8.3+evolvis25 or later was installed); this stipulation is valid for all such references. When you write “[#123]” in an SVN commit message, the same applies in both directions – links to the task/tracker item from SCM → ViewVC (web-based Subversion browser) and links to the changeset from the task/tracker item as Related Commit (similar to how Related Tasks work). Similar to the t_follow.php based Permalinks for Tasks, Trackers now also have permalinks that are displayed. In both Tasks and Tracker, the ID column in the subproject browse view links to them for easy grabbing. The browse views also have configurable paging for logged-in users now, the setting of which is stored as a global preference.

In task/tracker item comments (as well as the initial comment, named “detailed description”), hyperlinks are made “clickable” automatically as well, using RFC-compliant matching (i.e. http://foo/möp is not a valid link – to be exact, it’ll link to http://foo/m – but http://foo/m%C3%B6p is).

It is now possible to change the “percent complete” column of Tasks in a bulk update. The Copy+Closed functionality now works almost correctly (a minor detail for better tarent-Activity usability will be changed later). The comments are now displayed in chronological order, instead of antichronological, for better human readability.

The state and percent complete of linked tasks are shown in the detail view of a Tracker item (bug, feature request, …), and related tasks can be unlinked there as well, instead of just in the tasks’ detail view.

The search now ignores double quotes and searches for the combination of all words by default.

The MediaWiki plugin sees Interwiki capability: the Interwiki table is global (per-forge) instead of per-wiki and filled with all præficēs automatically, using the Project’s unix name. It can also be edited with the new Special:Interwiki MediaWiki extension. The mw-wrapper.php script works correctly again, and nightly XML dumps of every Wiki are available, for example for backups. It is now possible to use both English (File:foo.jpg, #REDIRECT, Special:Weird) and German (Datei:foo.jpg, #WEITERLEITUNG, Spezial:Seltsam) syntax in the Wikis. Pages like Login/Logout, Create Account and Lost Password, that have no value in the MediaWiki plugin, redirect to the Forge’s equivalent functionality.

The SOAP WSDL now compiles cleanly and sees a few bugfixes as well as a new API “addUploadedFile” which works with the File Release System and manual (SFTP) uploads, to facilitate automatic deployment, e.g. from a Hudson running Maven.

The Document Manager has received some fixes and the ability to use manual (SFTP) upload like the FRS.

All projects will have a -commits and a -discuss mailing list; all members with SCM commit access will be added to the former, all proect members no matter what to the latter automatically upon joining the project (you can, of course, unsubscribe). Postings to the -commits list bear the -discuss list as followup. The Postfix MTA will be configured to use Postgrey for greylisting to reduce spam income.

There’s now a suggestion list for project tags. In most places where a history was kept (project, task, tracker), not only the old but also the new value are stored and shown now, and the sort order has been reversed to chronological as well.

EvolvisForge will now work better with Python 2.6 and PHP 5.3 on Debian unstable, although Debian Lenny (with backports) is still the only supported platform; *.deb files are available on request, we’ll allow others to install EvolvisForge and related software by and for themselves now (although most of the functionality is supposed to be achievable by using FusionForge, our upstream Open Source project). (In fact, in between writing this Release Announcement for the Evolvis Project and formatting it for the Blog, Roland Mas has begun merging improvements into FusionForge trunk. Nevertheless, contact evolvistodo {klammeraffe} tarent {punkt} de if you want to set up an EvolvisForge instance yourself. Remember that Evolvis is more than just the Forge, it’s also Wikis (now included in the Forge though), Blogs, Planet, Continuous Integration (soon to be integrated in the Forge), Domisol, and more.)

Some minor things have been fixed too, for example, there was a tremendous effort to fix all PHP warnings found and make all pages XHTML 1.0/Transitional compliant (if one isn’t, it can now be considered a bug), which led to the redesign of some pages, such as the Task “Select Columns” facility and the project Admin page. Wrong output, texts and translations in some places have been fixed. Developers of EvolvisForge now have better help in debugging: the “pink pop-up” can display calls to db_query{,_params} and backtraces when PHP warnings/errors occur. The licence information of a project will not be shown, as it currently cannot be set (we’re working on that). Several theme issues have been corrected, and to Mozilla™ Firefox® users (and users of other Gecko-based browsers), fields’ elements will now appear white-on-black by default with black-on-white (instead of white-on-blue) for selected items, due to a bug in that browser series. And, of course, the bugfixes from our upstream, FusionForge, have been merged when applicable as well.

The complete changelog and our MediaWiki Plugin demonstration page are still available for your convenience.

We wish you a pleasant experience using the new, improved EvolvisForge, as well as the rest of our Evolvis platform! -- The Evolvis team

PS: Sorry for the very long posting, but I’ve already tried to go down on detail and only mention the important things, mostly from a user’s PoV, and condensed… it’s just a lot happened and some co-workers are really excited about those features yet.

mksh, encodings, MirBSD, BitTorrent, WinCE

28.08.2010 by tg@
Tags: android debian geocache hardware mksh news release snapshot

mksh was merged into Android (both AOSP and Google’s internal master tree) in the night 24/25th August, and is expected to be the one shell to rule them all, for Gingerbread.

mksh(1) now also has a cat builtin, for here documents mostly. It calls the cat(1) command if it receives any options. The shell is nevertheless smaller than yesterday because of improved string pooling.

There’s another reason to use the MirOS OPTU-16 encoding instead of PEP 383, on which I already wrote: try passing a wide-char filename to a function such as MessageBoxW, or create a filename on a system using wide chars, such as FAT’s LFN or ISO 9660’s Joliet, or one that only allows Unicode (canonically decomposed – ü → ü – out of all things) like HFS+. OPTU-8 at least maps to somewhat reserved codepoints (would, of course, be better to get an official 128 codepoint block, but the chance’s small of getting that in the BMP). Still.

Oh well, the torrents. I’ve remade them all, using one DHT seed node and OpenBitTorrent as tracker and put them on a very rudimentary BT page that will be completely redone soonish. Please re-download them. I currently do not believe will return.

Finally, I fell victim to a selling-out and may have just bought a Windows Mobile 6 based phone (Glofiish X650) and an SDHC card and an extra battery with double capacity. Well, at least it’s said to run CacheWolf well. I still would like to have something like Interix, Cygwin, UWIN, coLinux, or maybe some qemu-for-WinCE variant that runs Android, Maemo, Debian/armhf (or armel or arm) at near-native speed (and is usable – the device sadly doesn’t have a hardware keyboard, but it comes with SiRFstar Ⅲ GPSr). It only has 64 MiB RAM, like the Zaurus SL-C3200 and the jesusPhone, though. ☹ Any chance to get MirWorldDomination onto that device as well?

Tomorrow, eight years ago, is the date we now use as birthing point for MirOS. The thing is, we did not really want to create a BSD of our own, fork, or whatnot. We were mostly happy OpenBSD users (really happy before the first eMail exchange with its developers, where Theo de Raadt did indeed stand out but was not the only one – just the one with the authority to deny us), improved it locally and submitted patches and ports. We were flamed for that or, worse, ignored. I begun putting up my “OpenBSD patchkit” on my homepage (back then, at Tripod) and still tried to feed things to upstream and OpenBSD. Then, at some point, Theo de Raadt made it clear he did not want me and the patch kit had grown (from one 4M file into several of them), so I ended up doing a “cvs -d /cvs init” and went from there. Benny’s story is similar – he laughed at me while trying to get ports added to OpenBSD, then discovered his ports were added to the MirPorts Framework and getting commit access there was easier than getting some random developer to commit something of his to OpenBSD. (This trend ended there though… every single person I approached since has become OpenBSD ports committer – I wonder whether they used my invitation letter to blackmail Theo?) It’s often thought that there was a clash of opinions between Theo and me. I think while we might disagree in certain aspects or priorities things should have, in the end we both wanted the same thing, I just was promised to never become a member of the OpenBSD project, so it’s really just “them” being uncooperative. (They (Henning and others) did burn the T-Shirt I gave Theo as a gift some day for making OpenBSD what it was. I won’t comment on that, again, now.)

All 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

MirOS Logo