Developers’ Weblog

Sponsored by
HostEurope Logo

Developers’ Weblog

All 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

mksh R40b (nowadays with filled in user’s caveats (for R40, too!) and packager’s upgrade hints) has just been released. This is a should-have upgrade, fixing a number of – admittedly some obscure – bugs, changing things begun in R40, improving upon others. Thanks to the PLD Linux guys for spotting all these errors; thanks to them and phpnet.org both for adopting mksh so well.

I have also fixed a bug in nroff(1) which will lead to an even nicer looking HTML manpage mksh(1) (after the next rebuild and upload of a MirBSD snapshot – scheduled RSN).

jupp 3.1.16 took on the task of merging Debian joe changes (aiming at an upload). I also split the jupprc file into three versions (2.8 generic/DOS, 3.1+jupp and 3.7/Unix) because of the differences in the baseline executables making rc files partially mutually incompatible (think Insert key), annoyingly warning (think syntax, hmsg), or less usable (joe’s new menu system).

jupp 2.8.2 is a companion to jupp 3.1.16 – mostly because of the new help window “character map” ☺

Binaries for jupp should be updated RSN too.

Considering Banja Luka is arriving quickly, the “r” in RSN should be taken with a few grains of salt. I’ve also scheduled working on the pcc Debian package for the next future; updating lynx and maybe others like OpenSSH in MirBSD is also due; cvs(1) will receive more of my time, but before the next Upload I’d like to fix LP#12230 once verified.

Builds for Debian/m68k are also still running. I note I did in fact not manage to make a new base image, yet (but 2.6.39 kernels miss a patch, anyway, so waiting for 3.0 is ok). It’s still using gcc-4.4 because nobody tests gcc-4.6 and gcj-4.6 FTBFS due to SIGSEGV, but that’s ok in my books. rsyslog is broken but sysklogd works.

The #ksh|Freenode page finally got a well-deserved link to Planet Commandline. Throw more my way!

Acronyms and translations, too. (Got Norwegian and Rumanian covered in the meantime. No idea whether any RTL languages will work in that beast. But I’m young and need the money)

Since I’m writing a wlog entry anyway… let me thank Gunnar for a nice summary on the current Free Culture discussion; my comments on Nina’s site seem to be eaten, but let me support it fully, although, of course, I normally use a copycenter style licence, which is specifically written for general works of authorship under copyright law, not limited to software. I did in fact have that in mind. Maybe some people will like it (it’s less than one Kibibyte long) either generally or just for their everyday random musings (they can then keep CC-BY-SA for the “big works” if they so desire).

Wouter, grass background makes green headlines illegible. I’ve never liked, and never installed manually, cups either. (Benny tells me that Apple’s new version refuses to talk with a non-Apple cups, kinda defeating the whole idea I think.) Port 9100 is JetDirect (probably with an HP in front and some subset of ©®™ trailing) and just nice. (Being able to talk ESC/P with your printer like print '\033K\x07\0\x3E\x81\x99\xA5\xA5\x81\x3E' >/dev/lpa too rocks though, IMHO. Yes, mine can, and I still can. /dev/lpa is BSD.)

Kai, thanks for your vimrc lines:

	:highlight TrailWhitespace ctermbg=red guibg=red
	:match TrailWhitespace /\s\+$\| \+\ze\t/
 

Automatic removal is harmful, though – I just fell into the trap since jupprc contains needed whitespace at EOL… but manual removal (bound to ^K] in jupp) rocks. And I like that your solution uses such strong a colour – vim users are the single most represented offender group for actually leaving the redundant whitespace at EOL there, and it should hurt their eyes. (Sadly there is some vehement disagreement preventing them from inclusion in grml-etc-core – but that’s why I re-post them here.) Ah, and jupp can of course display whitespace visibly (although it uses ‘·’/‘→’, replacing the arrow with ‘¬’ if no UTF-8, not ‘»’), accessible with ^Ov.

Steve, want to put up a checklist for sites? We can “crowdsource” the… testing… to maybe get some interesting results…

Some other people would get more comments if they were idling in IRC (Freenode) or allow comments on their blog, specifically without too high an entrance barrier – OpenID is ok, but many other things, and ECMAscript, are not; but I can’t really say that loud because our wlog is static HTML compiled from a flat plaintext data source so it doesn’t allow such either. I often forget what I wanted to add if I can’t get it out quickly enough (especially at work). Sowwy…

Me like the cat picture postings (Amayita, Tiago, ¡Gracias!).

New releases

11.07.2011 by tg@
Tags: debian mksh

You might have noticed the release of mksh R40 recently, after more than a year of development. Well, stay tuned for both R40b (with accumulated fixes) and R41 (intent to speed up array handling a lot and prepare for what we postponed to mksh R42 now – associative, multi-dimensional arrays).

You should also upgrade, if you have not yet done so, to kwalletcli 2.11.

Finally, jupp 3.1.15 was left out to the world, including Minix 3 users this time, by special request of one of these on our mailing list. In addition to the MidnightBSD mport – which has been there in like forever – and the MirPort and the FreeWRT package, in December 2011 a user submitted it to FreeBSD® ports, and Benny is going to add it to NetBSD® pkgsrc® soon, he said. (He also updated their mksh source package. Thanks!) I’ve been asked by two people, independent from each other, when I’ll upload it to Debian proper, instead of the private-repo packaging. Maybe I should indeed do that, comments?

  • √ Agreement to pay from company
  • √ Going to drive with some apparently speed-loving brits
  • √ Registration accepted
  • √ Dienstreiseantrag prepared
  • √ Sent that beast to the office ticket queue

So yes, this means I’m going to DebConf 11 to what used to be Yugoslawia when I was there the last time, although in the Poreč region of Istria, Hrvatska.

mksh R40 has been released Sunday 12nd June. So, what can you do with The MirBSD Korn Shell?

Have a look at the collection of shell snippets, which admittedly is only the beginning – most scripts don’t make use of the cool new features yet. But they eventually will. The Shell-Toolkit project is definitely worth a visit! (Admittedly not using my most ❦ dear SCM, but with a DVCS, every checkout is a clone, i.e. a backup, and none of the contributors must fear a central repo being taken down, which, for such a loose collection, is desirable.)

(First posting to Plänet Commandline! Tag: pcli)

Vutral asked in IRC how to synchronise two shells’ environment while they’re running. As you may know, POSIX systems cannot change a process’ environment vector after it has been started, only the process itself can. Well, the shell can, and we’ll use a variety of things for this.

This trick assumes you have $HISTFILE set to the same pathname in both shells (obviously, they run under the same user). It uses export -p to render the current list of exported variables, then transforms the list from newline-separated to a single big one-line export statement.
Then it transforms all remaining newlines (which will be part of a single-quoted string, since that’s mksh(1)’s export format) into the sequence '$'\n'' which means: terminate current single-quoted string, append $'\n' and open up a new single-quoted string immediately; concatenate these three.
Now, $'\n' is just a fancy way of saying newline, and part of mksh because David Korn (yes, the Korn in Korn Shell) strongly suggested to me that this functionality be included – but, as we can see here, it pays off.
Finally, the so transformed string is prepended by unset \$(export); which, when executed, will cause the shell to unset (and unexport) all currently exported variables. The shell parameters that are not exported, i.e. not in the environment, are not affected by this code (except for $x and $nl, but… whatever).
This string is then passed to read -s (plus -r and clearing IFS to enable raw mode), which means, read into the parameter $REPLY (which we conveniently don’t use – but it’s trashed too, thus) but store into history at the same time.

Ah hah! Now, the persistent history feature comes into effect! After running the below statement in the “source” shell, switch into the terminal running the “destination” shell, press Enter once on the empty line (Ctrl-U to empty it if it wasn’t), then Cursor-Up (↑) to recall… voilà, an insanely large line with the previously created string sorta expanded… and press Enter again to run it. Now your set of exported parameters is the exact same (minus if you exported IFS, nl, x or REPLY) as in the “source” shell.

I’ve added extra spaces and a linewrap below, this is really just one big line:

nl=$'\n'; x=$(export -p); x=${x//${nl}export/}; IFS= read -rs <<<"unset \\\$(export);${x//$nl/\'\$\'\\\\n\'\'}"

Of course, this makes a nice function, for your ~/.mkshrc or somesuch.

I was just tracking down why some mails seem to have garbled Subjects.

It looked like this in Alpine:
Subject: [BOFH commits] r2040: fix directory struct =?UTF-8?Q?ure=E2=86=B5=20unix?=/mirror/=?UTF-8?Q?=20=E2=86=92=20mirror?=. bonn

The raw header sight was this:
Subject: [BOFH commits] =?utf-8?q?r2040=3A__fix_directory_struct__=3D=3FUT?=
=?utf-8?b?Ri04P1E/dXJlPUUyPTg2PUI1PTIwdW5peD89L21pcnJvci89P1VURi04P1E/?=
=?utf-8?b?PTIwPUUyPTg2PTkyPTIwbWlycm9yPz0uIGJvbm4=?=

Ein Schelm, wer Böses dabei denkt… First suspect was, of course, Mailman – after decoding, this showed the classical signs of a double-encode. (After ruling out general header brokenness, but no, 76 chars is ok.) I thus hand-crafted an eMail with the correct header line and sent that out:
Subject: [BOFH commits] =?utf-8?q?r2040=3A_fix_directory_structure?=
=?utf-8?b?4oa1IHVuaXgvbWlycm9yLyDihpIgbWlycm9yLmJvbm4=?=

Huh? Mailman and Python weren’t the culprit, thus – this is correct mangling. Okay, let’s dive into the Perl code that actually sends out the eMails. To make a long story short, have a look at this, then RFC 2047:
tglase@tglase:~ $ perl -MEncode -e '$subject = "r2040: fix directory structure↵ unix/mirror/ → mirror.bonn"; Encode::from_to($subject, "UTF-8", "MIME-Q"); print "{Subject: ".$subject."}\n";'
{Subject: r2040:=?UTF-8?Q?=20fix=20directory=20struct?=
=?UTF-8?Q?ure=E2=86=B5=20unix?=/mirror/=?UTF-8?Q?=20=E2=86=92=20mirror?=.
bonn}

Amazingly enough, PHP’s mb_encode_mimeheader, despite being talked to trash in the comments on its online documentation, does manage to get it right:
tglase@tglase:~ $ php -r 'mb_internal_encoding("UTF-8");echo "{".mb_encode_mimeheader("Subject: r2040: fix directory structure↵ unix/mirror/ → mirror.bonn", "UTF-8", "Q")."}\n";'
{Subject: r2040: fix directory =?UTF-8?Q?structure=E2=86=B5=20unix/mirror/?=
=?UTF-8?Q?=20=E2=86=92=20mirror=2Ebonn?=}

Wow. Now, the Perl guys I know told me to use Perl’s Mail tools… which are much too high-level though, for all I have and want is the subject string and an RFC 822 header line. I told them I’m not above doing this, and so I did. The 3P languages can really be annoying.

Why’s Perl’s output wrong anyway? I don’t know for sure, but I think the atoms must be separated, so unquoting /mirror/ in the middle, with no spaces around it, are wrong. (Besides, Encode::from_to can’t do the job right anyway, as it misses the name of the header, which is included in the 76 chars allowed for the first line. BAD • Broken As Desdigned.)

Disclaimer: I don’t really know any Perl, I fight my way through PHP and, barely, Python. (But I can code.)

I’m still working on mksh and doing some MirBSD core and ports work in between. On the other hand, since a lot of things suck so much, and other things become unacceptable, I’m seriously considering writing things like a libc (probably not complete, not totally from scratch – no way I’m going to code sunrpc – and not fast but at least correct. I’ve written quite some code for the MirBSD libc already, as well as kernel and bootloader. Most of it is shared between these, and I’ve digged in enough others (klibc, dietlibc, µClibc, musl, glibc) to have seen more. I was thinking of static linkage only at first, mostly for bootloader and compiler support code and, later, to link mksh statically against it on some OSes. But cnuke@ wanted a “correct” libc with dynamic linkage. Hmm… (On the other hand, we all know that these are dreams and a RL job can be time and will consuming.)

Roland Mas is hacking on Alioth this weekend, which runs code from me and some coworkers via FusionForge – more coverage ☺ I’m also doing okay with Debian/m68k (still slowly of course, but then, as BSD developer, I’m used to that *g*) and one of the old m68k buildds may very well be resurrected soonish. (Still need to port elfutils…) By the way, XHTML sucks, some things are hard to get right, and writing a DTD should be obsolete…

I took over cvs(1) in Debian and replaced it with my previously private package of MirPorts’ cvs. Way to go! (Thanks to Steve for handing over maintainership.) Now, everyone, please test it and recheck whether your old bugs still apply. Oh, and send patches. But pserver, PAM etc. are gone for good, don’t bother.

PS: It’s still Google Go, Benny. (World of Google’s goo?)

Progress with pkgsrc

19.05.2011 by bsiegert@
Tags: pkgsrc

The MirOS support for pkgsrc is progressing, albeit very slowly. This is because I spend more time traveling for work than in the office. After a long day of meetings and conferences, I am no longer motivated to code in my free time. What’s more, several different projects at work have tight deadlines.

Having said this, I committed MirOS support for libtool-2.2.6b in pkgsrc the other day, after a positive review by agc. As for the bmake patch, which is still available from our pkgsrc page, I was told to redo the patch, this time for src/usr.bin/make directly. When this patch will have gone in, somebody else (joerg?) will sync pkgsrc/devel/bmake, and MirOS will be able to bootstrap pkgsrc without patches. Let’s try to get to this point before the 2011Q2 branch.

The new “showcase” machine that I bought a while ago, does not have working network connectivity and SATA under MirOS. tg@ recommended an installation into an HVM guest under Xen. I am currently trying to set this up, however I must first succeed to get a stable NetBSD Dom0 system. I am using NetBSD-current because it has the right network driver and a custom kernel, however the console seems flakey, and accessing /kern/xen (even just ls!) leaves the process hanging in the “tstile” state. pkgsrc has four versions of Xen, so I should just test them all until I find one that works.

PS: My Go package, image/tiff is now part of the standard library. Yay!

:(

19.05.2011 by bsiegert@

I hope I will never again have to attend a funeral for a friend who committed suicide.

That is all.

tl;dr: Full JSON encoder/emitter and decoder/parser in Pure PHP now available as part of FusionForge/Evolvis under GPLv2. Do not use PHP’s built-in code, it’s broken. Read on for details and links.

I’ve pimped the minijson code in EvolvisForge to be a full-blown JSON encoder and decoder (lexer/parser) now. You wouldn’t believe just how much is broken in PHP (its own json_encode handles floats wrong in most locales, and mb_check_encoding doesn’t check the encoding of a multibyte string… at least, unlike Python, PHP manages to get 8bit okay though).

Anyway, thought you want to know. If you ever need a Pure PHP JSON encoder/decoder, you know where to look. It’s 100% my code (although written during dayjob, so the exploitation rights are with tarent GmbH). Current licence is GPLv2+ or AGPLv3+. If you spot any bugs, you know where to report them. I think it should be good though. (Do note that JSON is case-sensitive, so “NULL”, “True” and “\N” or “\U20AC” are not valid, “null”, “true”, “\n” and “\u20AC” or “\u20ac” are.)

I plan to store the “art_cust123” information in the user_prefs table in JSON now, instead of ‘|’-separated values, to avoid problems like these we had with broken database format and subsequent corruption of values, by using a dictionary (JSON Object) instead.

The ECMA 262 standard (ECMAscript and JSON) is freely available, though. JSON is also additionally specified in RFC 4627 which differs slightly (see my notes for details, mostly the goal element).

Update: fixed links

The blogs have been moved from *.blogs.evolvis.org to *.blog.tarent.de to provide proper SSL certificates. This also means that the use of blogs for people who are not employees of tarent GmbH, tarent AG or one of its subsidiaries is currently not available. (If that should become necessary, please contact us.

The feed has moved to http://evolvisforge.blog.tarent.de/feed/ for this blog (similarily to the others), but just in case, you’ll get redirected (although only to the https version, which Planetplanet doesn’t seem to like).

Planet Evolvis will be set up anew using Planet Venus software shortly.

All in all, the Greater Evolvis Platform is undergoing updates and improvements.

in-target: E: Kaputte Pakete

11.04.2011 by tg@
Tags: bug debian rant

*buntu Hardy kann zur Zeit nicht installiert werden (der Kernel (in main) dependet auf Pakete aus restrictet, das ist aber zum Installationszeitpunkt nicht aktiv und sowieso unfrei; und wieso ist eigentlich das hardy-updates Repo im d-i eingeschaltet und nicht erst hinterher?).

Lustiger aber: „Einige Pakete konnten nicht installiert werden. Das kann bedeuten, dass[sic!] Sie eine unmögliche Situation angefordert haben oder dass[sic!], wenn Sie die Unstable-Distribution verwenden, […]“

gecko2s Kommentar dazu nur, daß unstable bei *buntu stable heiße. Ich habs dann auf LTS korrigiert (ist nicht das erste Mal – und sowieso, wieso tauschen die in einer stabilen Version PostgreSQL-Majorversionen aus?) und dabei haben wir’s belassen: Debian unstable = *buntu LTS.

Naja, wie wir das letztens Simon gesagt haben (Upgrade innerhalb einer Version von *buntu auf einem Server hat grub durch grub2 ausgetauscht): Mit Debian wär’ das nicht passiert!

All 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

MirOS Logo