mksh tag cloud

Sponsored by
HostEurope Logo

mksh tag cloud

All 1 2 3 4 5 6 7 8 9

The MirBSD Korn Shell R52c was published today as bugfix-accumulating release of low upto medium importance. Thanks to everyone who helped squashing all those bugs; this includes our bug reporters who always include reproducer testcases; you’re wonderful!

MirCPIO was also resynchronised from OpenBSD, to address the CVE-2015-{1193,1194} test cases, after a downstream (wow there are so many?) reminded us of it; thanks!
This is mostly to prevent extracting ../foo – either directly or from a symlink(7) – from actually ending up being placed in the parent directory. As such the severity is medium-high. And it has a page now – initially just a landing page / stub; will be fleshed out later.

Uploads for both should make their way into Debian very soon (these are the packages mksh and pax). Uploading backports for mksh (jessie and wheezy-sloppy) have been requested by several users, but none of the four(?) DDs asked about sponsoring them even answered at all, and the regular (current) sponsors don’t have experience with bpo, so… SOL ☹

I’ve also tweaked a bug in sed(1), in MirBSD. Unfortunately, this means it now comes with the GNUism -i too: don’t use it, use ed(1) (much nicer anyway) or perlrun(1) -p/-n…

Finally, our PDF manpages now use the PA4 paper size instead of DIN ISO A4, meaning they can be printed without cropping or scaling on both A4 and US-american “letter” paper. And a Бодун from the last announcement: we now use Gentium and Inconsolata as body text and monospace fonts, respectively. (And à propos, the website ought to be more legible due to text justification and better line spacing now.) I managed to hack this up in GNU groff and Ghostscript, thankfully. (LaTeX too) Currently there are PDF manpages for joe (jupp), mksh, and cpio/pax/tar.

And we had Grünkohl today!

Also, new console-setup package in the “WTF” APT repository since upstream managed to do actual work on it (even fixed some bugs). Read its feed if interested, as its news will not be repeated here usually. (That means, subscribe as there won’t be many future reminders in this place.)

The netboot.me service appears to be gone. I’ll not remove our images, but if someone knows what became of it drop us a message (IRC or mailing list will work just fine).

PS: This was originally written on 20160304 but opax refused to be merged in time… Happy Birthday, gecko2! In the meantime, the Street Food festival weekend provided wonderful food at BaseCamp, and headache prevented this from being finished on the fifth.

Update 06.03.2016: The pax changes were too intrusive, so I decided to only backport the fixes OpenBSD did (both those they mentioned and those silently included), well, the applicable parts of them, anyway, instead. There will be a MirCPIO release completely rebased later after all changes are merged and, more importantly, tested. Another release although not set for immediate future should bring a more sensible (and mksh-like) buildsystem for improved portability (and thus some more changes we had to exclude at first).

I’ve also cloned the halfwidth part of the FixedMisc [MirOS] font as FixedMiscHW for use with Qt5 applications, xfonts-base in the “WTF” APT repo. (Debian #809979)

tl;dr: mksh R52c (bugfix-only, low-medium); mircpio 20160306 (security backport; high) with future complete rebase (medium) upstream and in Debian. No mksh backports due to lacking a bpo capable sponsor. New console-setup in “WTF” APT repo, and mksh there as usual. xfonts-base too. netboot.me gone?

Our PDF manpages will, starting from now, be generated with Inconsolata instead of Bitstream Vera Mono as monospace font. The body font is still Gentium, of course.

To be more exact: the Teχ flavour of Inconsolata Regular and Bold, with the varl and varqu flags, is used, and because GNU groff also requires an Italic or at least Oblique font (also in its bold variant, which the mksh(1) manpage doesn’t use though), Inconsolata LGC (both Italic and Bold Italic) are plugged in there. I added them as PFA Type 1 fonts to GNU groff, so I had to make some fixes in FontForge (merging the variants into the main font, removing unused glyphs (not for LGC), fixing the validation (mostly, and not so much for LGC), autohinting where FontForge expressed a need for that, renaming glyphs to the names expected by afmtodit, …), but it works.

I’m not regenerating older PDF manpages though.

Inconsolata is also not all I wish for a monospaced font (and even bsiegert@ says nothing goes over FixedMisc) but it has, at least, a 0 (digit zero) with a correct stroke through it ☺

I just published the first version of git find on gh/mirabilos/git-find for easy collaboration. The repository deliberately only contains the script and the manual page so it can easily be merged into git.git with complete history later, should they accept it. git find is MirOS licenced. It does require a recent mksh (Update: I did start it in POSIX sh first, but it eventually turned out to require arrays, and I don’t know perl(1) and am not going to rewrite it in C) and some common utility extensions to deal with NUL-separated lines (sort -z, grep -z, git ls-tree -z); also, support for '\0' in tr(1) and a comm(1) that does not choke on embedded NULs in lines.

To install or uninstall it, run…

	$ git clone git@github.com:mirabilos/git-find.git
	$ cd git-find
	$ sudo ln -sf $PWD/git-find /usr/lib/git-core/
	$ sudo cp git-find.1 /usr/local/share/man/man1/
	… hack …
	$ sudo rm /usr/lib/git-core/git-find \
	    /usr/local/share/man/man1/git-find.1

… then you can call it as “git find” and look at the documentation with “git help find”, as is customary.

The idea behind this utility is to have a tool like “git grep” that acts on the list of files known to git (and not e.g. ignored files) to quickly search for, say, all PNG files in the repository (but not the generated ones). “git find” acts on the index for the HEAD, i.e. whatever commit is currently checked-out (unlike “git grep” which also knows about “git add”ed files; fix welcome) and then offers a filter syntax similar to find(1) to follow up: parenthesēs, ! for negation, -a and -o for boolean are supported, as well as -name, -regex and -wholename and their case-insensitive variants, although regex uses grep(1) without (or, if the global option -E is given, with) -E, and the pattern matches use mksh(1)’s, which ignores the locale and doesn’t do [[:alpha:]] character classes yet. On the plus side, the output is guaranteed to be sorted; on the minus side, it is rather wastefully using temporary files (under $TMPDIR of course, so use of tmpfs is recommended). -print0 is the only output option (-print being the default).

Another mode “forwards” the file list to the system find; since it doesn’t support DOS-style response files, this only works if the amount of files is smaller than the operating system’s limit; this mode supports the full range (except -maxdepth) of the system find(1) filters, e.g. -mmin -1 and -ls, but it occurs filesystem access penalty for the entire tree and doesn’t sort the output, but can do -ls or even -exec.

The idea here is that it can collaboratively be improved, reviewed, fixed, etc. and then, should they agree, with the entire history, subtree-merged into git.git and shipped to the world.

Part of the development was sponsored by tarent solutions GmbH, the rest and the entire manual page were done in my vacation.

izabera did make a good point in IRC the other day for why we will need to have two locales at the very least in MirBSD – C and C.UTF-8 (the latter being widespread enough by now, thanks to me, interestingly enough. He uses code which leads to unexpected results…

	$ generate() { tr -dc "[:alnum:]" < /dev/urandom | dd bs="$len" count=1; }
	$ len=10; echo $(generate 2>/dev/null)
	Ut流54Ȫf

… because tr(1) was the first utility I converted to Unicode, to explore possibilities and craft the OPTU encoding and, thus, “流” is, indeed, an alphanumeric character.

This implies two things: we need to change MirBSD libc locale functions back to support two charsets (and make setlocale(3) match), and mksh(1) should implement locale tracking (to change set ±U whenever one of the relevant parameters (${LC_ALL:-${LC_CTYPE:-${LANG:-C}}}) changes in the session; users could still set utf8-mode manually though). For this to not break anything, we’ll have to audit scripts in MirBSD though (usually adding export LC_ALL=C at their begin is enough, and we need this for portable scripts anyway) and remove all occurrences of #ifndef __MirBSD__ before setlocale(3) calls in applications. This will take a while.


Secondly, I opened an issue with POSIX about handling of the (deprecated, and for good reason) `-style command substitutions. The GNU autoconf texinfo manual gives good advice for portable shell scripts, and we all knew that foo="bar `echo \"baz\"`" wasn’t portable due to use of more than one set of double quotes, but my (and the yash authors’) reading of the standard (and mksh R52’s POSIX mode) make it set $foo to bar "baz" instead of the historic bar baz now, and I wish to get this clarified (and, possibly, the standard changed to match historic practice, as this breaks at least the Acrobat Reader 5 start script). Nothing has been decided yet (due to the holidays, I’m sure), but we got input from some other people involved in shell.

So, if any #!/bin/sh scripts break or behave weirdly with R52, you now know why. I’m waiting for an official statement.

mksh R52 released

12.12.2015 by tg@
Tags: mksh pcli

The MirBSD Korn Shell R52 was published today. While there are still several known bugs, this is a release that primarily fixes lots of these, and, as with R51, we have no known regressions. Some of the itinerary for R52 has moved to R53 instead, as some bugfixes change the shell language and thus warrant a new major version, which is why this is not R51b, and they accumulated and could use some testing ;-)

This release has a nōnzero chance to break existing scripts that use some extension features – I had to quote some tildes in dot.mkshrc and a variable expansion in ${x/y"$z"} in MirWebseite (the $z) – twice!. As usual, test!

In less related news, a new release of the FixedMisc [MirOS] font is available (in BDF form and no conflict with the system Fixed [Misc] font); our CVS has the sources in bdfctool(1) format.

The MirBSD Korn Shell R51 was published today. This is a feature release clearly, but still something a lot of people would wish to use. It contains several known severe bugs, but they all are no regressions, i.e. they exist in R50f already.

This one is kinda an early release, as I wish to have those known issues all fixed, but the changes – both deep down and enduser-visible – already warrant people looking for breakages, plus it makes synchronisation with mksh-os2 easier.

mksh R52 will follow, as bugfix release, pretty soon. Itinerary:

  • Fixes for as much of these known bugs as possible (code rewrites)
  • Unicode 8
  • New feature: print -a
  • Fixes for bugs reported against R51
  • Possibly more EBCDIC and OS/2 code synchronisation
  • Maybe a dead useful debug tool…

Once that’s out, I’ll roll up the fixes into R50g, so that particular code branch is not dead yet either ☺

And afterwards, at least mksh(1)-wise – I have got a lot of other things on my plate after all – we can attempt getting EBCDIC and maybe OS/2 to a state where the code is included in CVS.

carstenh asked in IRC how to make a shebang for mksh(1) scripts that works on both regular Unix and Android.

This is not as easy as it looks, though. Most Unicēs will have mksh installed, either manually or by means of the native package system, as /bin/mksh. Some put it into package manager-specific directories; I saw /sw/bin/mksh, /usr/local/bin/mksh and /usr/pkg/bin/mksh so far. Some systems have it as /usr/bin/mksh but these are usually those who got poettering’d and have /bin a symlink anyway. Most of these systems also have env(1) as /usr/bin/env.

Android, on the contrary, ships with precisely one shell. This has been mksh for a while, thankfully. There is, however, neither a /bin nor a /usr directory. mksh usually lives as /system/bin/mksh, with /system/bin/sh a symlink(7) to the former location. Some broken Android versions ship the binary in the latter location instead and do not ship anything that matches mksh on the $PATH, but I hope they merge my AOSP patch to revert this bad change (especially as some third-party Android toolkits overwrite /system/bin/sh with busybox sh or GNU bash and you’d lose mksh in the progress). However, on all official Android systems, mksh is the system shell. This will be important later.

The obvious and correct fix is, of course, to chmod -x the scripts and call them explicitly as mksh scriptname. This is not always possible or desirable; sometimes, people will wish it to be in the $PATH and executable, so we need a different solution.

There’s a neat trick with shebangs – the absence of one is handled specifically by most systems in various ways. I remember reading about it, but don’t remember where; I can’t find it on Sven Mascheck’s excellent pages… but: the C shell variants run a script with the Bourne Shell if its first line is a sole colon (‘:’), the Bourne family shells run it with themselves or ${EXECSHELL:-/bin/sh} in those cases, and the kernel with the system shell, AFAIK. So we have a way to get most things that could call the script to interpret it as Bourne/POSIX shell script on most systems. Then we just have to add a Bourne shell scriptlet that switches to mksh iff the current shell isn’t it (lksh, or something totally different). On Android, there is only ever one shell (or the toolkit installer better preserve mksh as mksh), so this doesn’t do anything (I hope – but did not test – that the kernel invokes the system shell correctly despite it not lying under /bin/sh) nor does it need to.

This leaves us with the following “shebang”:

	:
	case ${KSH_VERSION-} in
	*MIRBSD\ KSH*) ;;
	*)	# re-run with The MirBSD Korn Shell, this is an mksh-specific script
		test "${ZSH_VERSION+set}" = set && alias -g '${1+"$@"}'='"$@"'
		exec mksh "$0" ${1+"$@"}
		echo >&2 E: mksh re-exec failed, should not happen
		exit 127 ;;
	esac
 

The case argument not only does not need to, but actually should not be quoted; the expansion is a set -u guard; the entire scriptlet is set -e safe as well; comments and expansions are safe. exec shall not return, but if it does (GNU bash violates POSIX that way, for example), we use POSIX’ appropriate errorlevel. zsh is funny with the Bourne shell’s way of using "$@" properly. But this should really be portable. The snippet is both too short and too obvious (“only way to do it”) to be protected by copyright law.

Thanks to carstenh and Ypnose for discussing things like this with us in IRC, sending in bugfixes (and changes we decline, with reason), etc. – it feels like we have a real community, not just consuments ☺

mksh R50f coming soon

11.04.2015 by tg@
Tags: mksh pcli

Please test mksh-current from CVS (or the inofficial git mirror)! There are security-related fixes I’ll MFC in a few days, for which I’d prefer for them (and the other changes) to not introduce any regressions. Thanks!

exciting news, or so

07.04.2015 by tg@
Tags: debian event fun geocache mksh news personal pkgsrc plan rant security work

I implemented <? support (including <?php…) script embedding support for *.inc in MirWebseite today; the specific syntax was explicitely requested by Natureshadow. Ugh.

My own hacking activities are progressing, even if slowly. I do some other interesting, funny, social, beneficial, etc. stuff in between, though. I’ll even have to get some of my DD buddies to sponsor me some QA uploads of packages I formerly maintained, whereever changes are queued up… such as better old-format repo compatibility in cvs(GNU) ☺ Though some of the stuff I do at work is currently done only there… sorry.

Also: prepare to be fully enlightened about just what evil (nice picture) Docker is. I especially liked the comparison of containers to a herd of cattle, mere numbers, replaceable, whereas VMs are cats, each with their individual name, lovely petted each day, etc.

ObHint: Some may have noticed I do have a Twitter account now. I do not really use it much. I got it because I wanted to rant at someone who only gave Twitter as means to contact them (a European company running a lottery for USA citizens only). But I found one nice thing: @HourlyCats (though @FacesPics and @BahnAnsagen are funny too, and the Postillon anyway). The internet is there for cat content, anyway.
Ahem. Do not contact me there, use IRC, more specifically, the Freenode network, and possibly memoserv to mirabilos instead, I can’t fit things into 140 chars, that’s just ridiculous. Also, don’t follow me. It may contain rants, it’s NSFW, and I’m not censoring there. As I said: I do not use it. So should you. (But kudos for having a mostly functional “fallback” site (the “mobile” one), which even works in PocketIE (Windows Mobile) and Opera 9, though not so much lynx(1)…)

odc (from #!/bin/mksh on IRC) is hacking support to use mksh instead of GNU bash for bootstrapping pkgsrc® (e.g. on Solaris). Nice! Good luck!

… à propos mksh(1), dear Debian armel and armhf buildd maintainer colleagues, pretty please with strawberries and chocolate ice on top (I just had that on waffles at my favourite ice salon, so I may be biased), do like s390x and update your chroots and wanna-build give-back mksh, as we requested, so the privacy fix makes it into jessie. Thanks in advance!

Oh, and Y_Plentyn and I both have been putting more and updated packages into my APT repository. XTaran held a talk at CLT 2015 mentioning it… maybe I should write up some docs about how to use it for which purposes (e.g. how to avoid systemd but not get the other packages from it, or how to use it with systemd (trivial but has to be stated, it’s freedom of choice after all), etc.)?

Besides decent fanfiction (the stories in the Uzumaki Naruto universe seem, on average, to be much longer than those in the Harry Potter one), the weather is becoming good, so I’ve already been enjoying going out for some geocaching and will have the bike fixed at the shop RSN (it suffers a bit each winter, as it stands outside, since our basement is mouldy, which is worse than a bit of rust IMHO) to get more activity in. Also planning to head to the GPS Maze in Mainz and, besides what time FrOSCon (including preparation) allows, heading to DebConf for a while.

mirabilos’ Waypoints

… to my shame I must admit I fucked up, and we still do not have support in libssl for SHA2-signed X.509 certificates. Also, StartSSL fucked up, so currently https for www.mirbsd.org is toast.

Also more on the rant side, services offered by web-based platforms, be they web (e.g. Groundspeak’s GC.COM) or not (Googlemail, which $orkplace switched to against my express veto some time ago) are getting worse and worse over time. I had hoped they realise that and improve, especially when seeing small signs (such as GC.COM pages shrinking to 20% of the formerly served bloat) but… no.

Bernhard’s article on Plänet Debian about the “colon” command in the shell could use a clarification and a security-relevant correcture.

There is, indeed, no difference between the : and true built-in commands.

Stéphane Chazelas points out that writing : ${VARNAME:=default} is bad, : "${VARNAME:=default}" is correct. Reason: someone could preset $VARNAME with, for example, /*/*/*/*/../../../../*/*/*/*/../../../../*/*/*/* which will exhaust during globbing.

Besides that, the article is good. Thanks Bernhard for posting it!

PS: I sometimes use the colon as comment leader in the last line of a script or function, because it, unlike the octothorpe, sets $? to 0, which can be useful.

Update: As jilles pointed out in IRC, “colon” (‘:’) is a POSIX special built-in (most importantly, it keeps assignments), whereas “true” is a regular built-in utility.

mksh R50d released

07.10.2014 by tg@
Tags: bug debian mksh news pcli

The last MirBSD Korn Shell update broke update-initramfs because I accidentally introduced a regression in field splitting while fixing other bugs – sorry!

mksh R50d was just released to fix that, and a small NULL pointer dereference found by Goodbox on IRC. Thanks to my employer tarent for a bit of time to work on it.

mksh R50c released, security fix

03.10.2014 by tg@
Tags: android bug debian mksh news pcli release security

The MirBSD Korn Shell has got a new security and maintenance release.

This release fixes one mksh(1)-specific issue when importing values from the environment. The issue has been detected by the main developer during careful code review, looking at whether the shell is affected by the recent “shellshock” bugs in GNU bash, many of which also affect AT&T ksh93. (The answer is: no, none of these bugs affects mksh.) Stephane Chanzelas kindly provided me with an in-depth look at how this can be exploited. The issue has not got a CVE identifier because it was identified as low-risk. The problem here is that the environment import filter mistakenly accepted variables named “FOO+” (for any FOO), which are, by general environ(7) syntax, distinct from “FOO”, and treated them as appending to the value of “FOO”. An attacker who already had access to the environment could so append values to parameters passed through programs (including sudo(8) or setuid) to shell scripts, including indirectly, after those programs intended to sanitise the environment, e.g. invalidating the last $PATH component. It could also be used to circumvent sudo’s environment filter which protected against the vulnerability of an unpatched GNU bash being exploited.

tl;dr: mksh not affected by any shellshock bugs, but we found a bug of our own, with low impact, which does not affect any other shell, during careful code review. Please do update to mksh R50c quickly.

All 1 2 3 4 5 6 7 8 9

MirOS Logo