planet-cli tag cloud

Sponsored by
HostEurope Logo

planet-cli tag cloud

All 1 2 3 4

Beltane Snapshot and Mainframe Korn Shell…ish

05.05.2017 by tg@
Tags: mksh pcli plan snapshot

I was planning to do an mksh R56 release and then a full MirBSD snapshot (i386, sparc — due to actual user request — and possibly even a Live CD or at least baselive) but this got stones on my way.

I’m not quite finished with what I originally had planned for R56 (basically, the Debian postfix package’s maintainer scripts started using character classes in bracket expressions, and this required not only careful planning and design but also quite some rewriting and thinking, fixing other bugs, reading the specs, and considering EBCDIC) which led to me asking the EBCDIC porter some things again, which led to trying to merge his outstanding patches and make R56 the Mainframe Korn Shell release (also mksh ;-) but we’re not quite there yet.

The MirBSD snapshot was planned to be started from CVS as of Beltane (Walpurgis) 2017 except the latest and greatest mksh is also kinda a requirement, and CVE fixes are tricking in, to add insult to injury for stuff I had just updated. I’d also love to have the latest sendmail and lynx in it but that’ll have to wait.

I’ll also do a new CVS snapshot tarball at the same time, so keep your eyes open for the new rolling MirBSD snapshot.

Somehow, spring weather does not agree with me anyway. I didn’t get much done, nor much good sleep. Private life’s looking up but also more busy. I do manage to still help out others even in code I don’t really understand… must be good karma. Vacation would be nice… but then, I know I wouldn’t get all out of it.

Updates to the last two posts

16.03.2017 by tg@
Tags: bug debian grml news pcli rant snippet tip work

Someone from the FSF’s licencing department posted an official-looking thing saying they don’t believe GitHub’s new ToS to be problematic with copyleft. Well, my lawyer (not my personal one, nor for The MirOS Project, but related to another association, informally) does agree with my reading of the new ToS, and I can point out at least a clause in the GPLv1 (I really don’t have time right now) which says contrary (but does this mean the FSF generally waives the restrictions of the GPL for anything on GitHub?). I’ll eMail GitHub Legal directly and will try to continue getting this fixed (as soon as I have enough time for it) as I’ll otherwise be forced to force GitHub to remove stuff from me (but with someone else as original author) under GPL, such as… tinyirc and e3.

My dbconfig-common Debian packaging example got a rather hefty upgrade because dbconfig-common (unlike any other DB schema framework I know of) doesn’t apply the upgrades on a fresh install (and doesn’t automatically put the upgrades into a transaction either) but only upgrades between Debian package versions (which can be funny with backports, but AFAICT that part is handled correctly). I now append the upgrades to the initial-version-as-seen-in-the-source to generate the initial-version-as-shipped-in-the-binary-package (optionally, only if it’s named .in) removing all transaction stuff from the upgrade files and wrapping the whole shit in BEGIN; and COMMIT; after merging. (This should at least not break nōn-PostgreSQL databases and… well, database-like-ish things I cannot test for obvious (SQLite is illegal, at least in Germany, but potentially worldwide, and then PostgreSQL is the only remaining Open Source database left ;) reasons.)

Update: Yes, this does mean that maintainers of databases and webservers should send me patches to make this work with not-PostgreSQL (new install/, upgrade files) and not-Apache-2.2/2.4 (new debian/*/*.conf snippets) to make this packaging example even more generally usable.

Natureshadow already forked this and made a Python/Flask package from it, so I’ll prod him to provide a similarily versatile hello-python-world example package.

Since I use this as base for other PHP packages like SimKolab, I’ve updated my packaging example with:

  • PHP 7 support (untested, as I need libapache2-mod-php5)
  • tons more utility code for you to use
  • a class autoloader, with example (build time, for now)
  • (at build time) running a PHPUnit testsuite (unless nocheck)

The old features (Apache 2.2 and 2.4 support, dbconfig-common, etc.) are, of course, still there. Support for other webservers could be contributed by you, and I could extend the autoloader to work at runtime (using dpkg triggers) to include dependencies as packaged in other Debian packages. See, nobody needs “composer”! ☻

Feel free to check it out, play around with it, install it, test it, send me improvement patches and feature requests, etc. — it’s here with a mirror at GitHub (since I wrote it myself and the licence is permissive enough anyway).

This posting and the code behind it are sponsored by my employer ⮡ tarent.

Please use the correct (perma)link to bookmark this article, not the page listing all wlog entries of the last decade. Thank you.</update>

Some updates inline and at the bottom.

The new Terms of Service of GitHub became effective today, which is quite problematic — there was a review phase, but my reviews pointing out the problems were not answered, and, while the language is somewhat changed from the draft, they became effective immediately.

Now, the new ToS are not so bad that one immediately must stop using their service for disagreement, but it’s important that certain content may no longer legally be pushed to GitHub. I’ll try to explain which is affected, and why.

I’m mostly working my way backwards through section D, as that’s where the problems I identified lie, and because this is from easier to harder.

Note that using a private repository does not help, as the same terms apply.

Anything requiring attribution (e.g. CC-BY, but also BSD, …)

Section D.7 requires the person uploading content to waive any and all attribution rights. Ostensibly “to allow basic functions like search to work”, which I can even believe, but, for a work the uploader did not create completely by themselves, they can’t grant this licence.

The CC licences are notably bad because they don’t permit sublicencing, but even so, anything requiring attribution can, in almost all cases, not “written or otherwise, created or uploaded by our Users”. This is fact, and the exceptions are few.

Anything putting conditions on the right to “use, display and perform” the work and, worse, “reproduce” (all Copyleft)

Section D.5 requires the uploader to grant all other GitHub users…

  • the right to “use, display and perform” the work (with no further restrictions attached to it) — while this (likely — I didn’t check) does not exclude the GPL, many others (I believe CC-*-SA) are affected, and…
  • the right to “reproduce your Content solely on GitHub as permitted through GitHub's functionality”, with no further restructions attached; this is a killer for, I believe, any and all licences falling into the “copyleft” category.

Note that section D.4 is similar, but granting the licence to GitHub (and their successors); while this is worded much more friendly than in the draft, this fact only makes it harder to see if it affects works in a similar way. But that doesn’t matter since D.5 is clear enough. (This doesn’t mean it’s not a problem, just that I don’t want to go there and analyse D.4 as D.5 points out the same problems but is easier.)

This means that any and all content under copyleft licences is also no longer welcome on GitHub.

Anything requiring integrity of the author’s source (e.g. LPPL)

Some licences are famous for requiring people to keep the original intact while permitting patches to be piled on top; this is actually permissible for Open Source, even though annoying, and the most common LaTeX licence is rather close to that. Section D.3 says any (partial) content can be removed — though keeping a PKZIP archive of the original is a likely workaround.

Affected licences

Anything copyleft (GPL, AGPL, LGPL, CC-*-SA) or requiring attribution (CC-BY-*, but also 4-clause BSD, Apache 2 with NOTICE text file, …) are affected. BSD-style licences without advertising clause (MIT/Expat, MirOS, etc.) are probably not affected… if GitHub doesn’t go too far and dissociates excerpts from their context and legal info, but then nobody would be able to distribute it, so that’d be useless.

But what if I just fork something under such a licence?

Only “continuing to use GitHub” constitutes accepting the new terms. This means that repositories from people who last used GitHub before March 2017 are excluded.

Even then, the new terms likely only apply to content uploaded in March 2017 or later (note that git commit dates are unreliable, you have to actually check whether the contribution dates March 2017 or later).

And then, most people are likely unaware of the new terms. If they upload content they themselves don’t have the appropriate rights (waivers to attribution and copyleft/share-alike clauses), it’s plain illegal and also makes your upload of them or a derivate thereof no more legal.

Granted, people who, in full knowledge of the new ToS, share any “User-Generated Content” with GitHub on or after 1ˢᵗ March, 2017, and actually have the appropriate rights to do that, can do that; and if you encounter such a repository, you can fork, modify and upload that iff you also waive attribution and copyleft/share-alike rights for your portion of the upload. But — especially in the beginning — these will be few and far between (even more so taking into account that GitHub is, legally spoken, a mess, and they don’t even care about hosting only OSS / Free works).

Conclusion (Fazit)

I’ll be starting to remove any such content of mine, such as the source code mirrors of jupp, which is under the GNU GPLv1, now and will be requesting people who forked such repositories on GitHub to also remove them. This is not something I like to do but something I am required to do in order to comply with the licence granted to me by my upstream. Anything you’ve found contributed by me in the meantime is up for review; ping me if I forgot something. (mksh is likely safe, even if I hereby remind you that the attribution requirement of the BSD-style licences still applies outside of GitHub.)

(Pet peeve: why can’t I “adopt a licence” with British spelling? They seem to require oversea barbarian spelling.)

The others

Atlassian Bitbucket has similar terms (even worse actually; I looked at them to see whether I could mirror mksh there, and turns out, I can’t if I don’t want to lose most of what few rights I retain when publishing under a permissive licence). Gitlab seems to not have such, but requires you to indemnify them… YMMV. I think I’ll self-host the removed content.

And now?

I’m in contact with someone from GitHub Legal (not explicitly in the official capacity though) and will try to explain the sheer magnitude of the problem and ways to solve this (leaving the technical issues to technical solutions and requiring legal solutions only where strictly necessary), but for now, the ToS are enacted (another point of my criticism of this move) and thus, the aforementioned works must go off GitHub right now.

That’s not to say they may not come back later once this all has been addressed, if it will be addressed to allow that. The new ToS do have some good; for example, the old ToS said “you allow every GitHub user to fork your repositories” without ever specifying what that means. It’s just that the people over at GitHub need to understand that, both legally and technically¹, any and all OSS licences² grant enough to run a hosting platform already³, and separate explicit grants are only needed if a repository contains content not under an OSI/OKFN/Copyfree/FSF/DFSG-free licence. I have been told that “these are important issues” and been thanked for my feedback; we’ll see what comes from this.

① maybe with a little more effort on the coders’ side³

② All licences on one of those lists or conformant to the DFSG, OSD or OKD should do⁴.

③ e.g. when displaying search results, add a note “this is an excerpt, click HERE to get to the original work in its context, with licence and attribution” where “HERE” is a backlink to the file in the repository

④ It is understood those organisations never un-approve any licence that rightfully conforms to those definitions (also in cases like a grant saying “just use any OSS² licence” which is occasionally used)

Update: In the meantime, joeyh has written not one but two insightful articles (although I disagree in some details; the new licence is only to GitHub users (D.5) and GitHub (D.4) and only within their system, so, while uploaders would violate the ToS (they cannot grant the licence) and (probably) the upstream-granted copyleft licence, this would not mean that everyone else wasn’t bound by the copyleft licence in, well, enough cases to count (yes it’s possible to construct situations in which this hurts the copyleft fraction, but no, they’re nowhere near 100%).

How to use the subtree git merge strategy

20.12.2016 by tg@
Tags: debian grml pcli tip work

This article might be perceived as a blatant ripoff of this Linux kernel document, but, on the contrary, it’s intended as add-on, showing how to do a subtree merge (the multi-project merge strategy that’s actually doable in a heterogenous group of developers, as opposed to subprojects, which many just can’t wrap their heads around) with contemporary git (“stupid content tracker”). Furthermore, the commands are reformatted to be easier to copy/paste.

To summarise: you’re on the top level of a checkout of the project into which the “other” project (Bproject) is to be merged. We wish to merge the top level of Bproject’s “master” branch as (newly created) subdirectory “dir-B” under the current project’s top level.

	$ git remote add --no-tags -f Bproject /path/to/B/.git
	$ git merge -s ours --allow-unrelated-histories --no-commit Bproject/master
	$ git read-tree -u --prefix=dir-B/ Bproject/master
	$ git commit -m 'Merge B project as our subdirectory dir-B'

	Later updates are easy:
	$ git pull -s subtree Bproject master

(mind the trailing slash after dir-B/ on the read-tree command!)

Besides reformatting, the use of --allow-unrelated-histories recently became necessary. --no-tags is also usually what you want, because tags are not namespaced like branches.

Another command you might find relevant is how to clean up orphaned remote branches:

	$ for x in $(git remote); do git remote prune "$x"; done

This command locally deletes all remote branches (those named “origin/foo”) that have been deleted on the remote side.

Update: Natureshadow wishes you to know that there is such a command as git subtree which can do similar things to the subtree merge strategy explained above, and several more related things. It does, however, need the præfix on every subsequent pull.

“I don’t like computers”

13.11.2016 by tg@
Tags: debian pcli personal rant tip

cnuke@ spotted something on the internet, and shared. Do read this, including the comments. It’s so true. (My car is 30 years old, I use computers mostly for sirc, lynx and ssh, and I especially do not buy any product that needs to be “online” to work.)

Nice parts of the internet, to offset this, though, do exist. IRC as a way of cheap (affordable), mostly reliant, communication that’s easy enough to do with TELNET.EXE if necessary. Fanfiction; easy proliferation of people’s art (literature, in this case). Fast access to documentation and source code; OpenBSD’s AnonCVS was a first, nowadays almost everything (not Tom Dickey’s projects (lynx, ncurses, xterm, cdk, …), nor GNU bash, though) is on a public version control system repository. (Now people need to learn to not rewrite history, just commit whatever shit they do, to record thought process, not produce the perfect-looking patch.) Livestreams too, I guess, but ever since went dead due to a USA law change on 2016-01-02, it got bad.

Please save GMane!

28.07.2016 by tg@
Tags: debian news pcli rant

GMane has been down for a day or two, and flakey for a day before that. MidnightBSD’s laffer1 just linked the reason, which made me cry out loud.

GMane is really great, and I rely on the NNTP interface a lot, both posting and especially reading — it gives me the ability to download messages from mailing lists I don’t receive in order to be able to compose replies with (mostly) correct References and In-Reply-To headers. Its web interface, especially the article permalinks, are also extremely helpful.

This is a request for a petition to save GMane. Please, someone, do something! Thanks in advance!

The MirBSD Korn Shell R52c was published today as bugfix-accumulating release of low upto medium importance. Thanks to everyone who helped squashing all those bugs; this includes our bug reporters who always include reproducer testcases; you’re wonderful!

MirCPIO was also resynchronised from OpenBSD, to address the CVE-2015-{1193,1194} test cases, after a downstream (wow there are so many?) reminded us of it; thanks!
This is mostly to prevent extracting ../foo – either directly or from a symlink(7) – from actually ending up being placed in the parent directory. As such the severity is medium-high. And it has a page now – initially just a landing page / stub; will be fleshed out later.

Uploads for both should make their way into Debian very soon (these are the packages mksh and pax). Uploading backports for mksh (jessie and wheezy-sloppy) have been requested by several users, but none of the four(?) DDs asked about sponsoring them even answered at all, and the regular (current) sponsors don’t have experience with bpo, so… SOL ☹

I’ve also tweaked a bug in sed(1), in MirBSD. Unfortunately, this means it now comes with the GNUism -i too: don’t use it, use ed(1) (much nicer anyway) or perlrun(1) -p/-n…

Finally, our PDF manpages now use the PA4 paper size instead of DIN ISO A4, meaning they can be printed without cropping or scaling on both A4 and US-american “letter” paper. And a Бодун from the last announcement: we now use Gentium and Inconsolata as body text and monospace fonts, respectively. (And à propos, the website ought to be more legible due to text justification and better line spacing now.) I managed to hack this up in GNU groff and Ghostscript, thankfully. (LaTeX too) Currently there are PDF manpages for joe (jupp), mksh, and cpio/pax/tar.

And we had Grünkohl today!

Also, new console-setup package in the “WTF” APT repository since upstream managed to do actual work on it (even fixed some bugs). Read its feed if interested, as its news will not be repeated here usually. (That means, subscribe as there won’t be many future reminders in this place.)

The service appears to be gone. I’ll not remove our images, but if someone knows what became of it drop us a message (IRC or mailing list will work just fine).

PS: This was originally written on 20160304 but opax refused to be merged in time… Happy Birthday, gecko2! In the meantime, the Street Food festival weekend provided wonderful food at BaseCamp, and headache prevented this from being finished on the fifth.

Update 06.03.2016: The pax changes were too intrusive, so I decided to only backport the fixes OpenBSD did (both those they mentioned and those silently included), well, the applicable parts of them, anyway, instead. There will be a MirCPIO release completely rebased later after all changes are merged and, more importantly, tested. Another release although not set for immediate future should bring a more sensible (and mksh-like) buildsystem for improved portability (and thus some more changes we had to exclude at first).

I’ve also cloned the halfwidth part of the FixedMisc [MirOS] font as FixedMiscHW for use with Qt5 applications, xfonts-base in the “WTF” APT repo. (Debian #809979)

tl;dr: mksh R52c (bugfix-only, low-medium); mircpio 20160306 (security backport; high) with future complete rebase (medium) upstream and in Debian. No mksh backports due to lacking a bpo capable sponsor. New console-setup in “WTF” APT repo, and mksh there as usual. xfonts-base too. gone?

The things you find in upstream code…

13.02.2016 by tg@
Tags: archaeology bug pcli rant security snapshot

I had just gotten an eMail from the nightly /etc/security cronjob that the mailbox from the user foo.lock belongs to the user foo (name changed to protect the… innocent? well, I know that guy from #OpenBSD on IRC, so… YMMV… anyway). Of course, I wanted to change that to exclude mbox lockfiles…

	# Mailboxes should be owned by user and unreadable.
	ls -l /var/mail | sed 1d | \
	awk '$3 != $9 \
		{ print "user " $9 " mailbox is owned by " $3 }

… oh wow. Needless to say I fixed that, although you must update your stat(1) first; it now has a possibility to generate NUL-terminated output (or any separator, really) which I used for this. (And no, Schily, I’m still of the opinion that NUL termination, even when one has to add it to each utility separately, is the better way to go.)

Dear OpenBSD developers, repeat after me:
Do n̲o̲t̲ parse ls(1) output!
Or write 100 lines of it, or something, until it sinks in.

(It can take some writing for it to sink in… just yesternight the fanfiction I was reading was at the point where Dolores Umbridge uses her Blood Quill on the students. Coincidence.)

Our PDF manpages will, starting from now, be generated with Inconsolata instead of Bitstream Vera Mono as monospace font. The body font is still Gentium, of course.

To be more exact: the Teχ flavour of Inconsolata Regular and Bold, with the varl and varqu flags, is used, and because GNU groff also requires an Italic or at least Oblique font (also in its bold variant, which the mksh(1) manpage doesn’t use though), Inconsolata LGC (both Italic and Bold Italic) are plugged in there. I added them as PFA Type 1 fonts to GNU groff, so I had to make some fixes in FontForge (merging the variants into the main font, removing unused glyphs (not for LGC), fixing the validation (mostly, and not so much for LGC), autohinting where FontForge expressed a need for that, renaming glyphs to the names expected by afmtodit, …), but it works.

I’m not regenerating older PDF manpages though.

Inconsolata is also not all I wish for a monospaced font (and even bsiegert@ says nothing goes over FixedMisc) but it has, at least, a 0 (digit zero) with a correct stroke through it ☺

expect turmoil

08.02.2016 by tg@
Tags: archaeology bug hardware news pcli personal plan rant

My network at home is unstable. NetCologne suggests to switch to fibre network, but that only comes with a dynamic IPv6 address and NAT64; completely unsuitable to running a server. (I could arguably tunnel a static IPv4 address from a dedicated server to home, but that would completely foil my plans for redundancy.) So I may need an ISP (phone isn’t important) that provides me with connectivity where a static IPv4 (and, ideally, a static IPv6 /64 or /48 – but only if the reverse DNS gets delegated to me, otherwise that’s unusable) ends up at a device of my choosing (and not a plastic router which can then “forward ports”; I require full internet to end up at my own device).

HostEurope is relocating the other server, both physically and network-wise. Their plan seems fool-proof so far, though.

gecko2@ is decommissioning the server on which eurynome is hosted, shortly. This will also be no small amount of fun for everyone involved. Expect old links, SSH host keys, etc. to break. This explicitly includes /etc/ssh/*known_hosts.

During all those moves, I will downsize my DNS zones and change some entries, so that old or duplicate records will be gone.

I’ll likely generate and publish completely new hostkeys (both gzsig(1) and PGP clearsigned) once this is all over. The current gzsig(1) key is at the end of /usr/share/doc/README in any installed system. (Do note MD5 is considered insecure.) My current PGP key is 9031955E7A97A4FDA32B2B8676B534B2E99007E0 but this requires GnuPG, so check both.

My seeming inability to remember rarely-used “secure” passwords, i.e. those not fitting into my normal schemata, led to me not attempting to run a CA myself any more. While, thanks to rsc, we have an official certificate for now, I probably will get StartSSL for “all” other systems (i.e. herc, as I appear to be downsizing), despite it lacking the SSL client purpose (important e.g. to SMTP). This shouldn’t affect anyone.

PS: I still hate Karneval!

I just published the first version of git find on gh/mirabilos/git-find for easy collaboration. The repository deliberately only contains the script and the manual page so it can easily be merged into git.git with complete history later, should they accept it. git find is MirOS licenced. It does require a recent mksh (Update: I did start it in POSIX sh first, but it eventually turned out to require arrays, and I don’t know perl(1) and am not going to rewrite it in C) and some common utility extensions to deal with NUL-separated lines (sort -z, grep -z, git ls-tree -z); also, support for '\0' in tr(1) and a comm(1) that does not choke on embedded NULs in lines.

To install or uninstall it, run…

	$ git clone
	$ cd git-find
	$ sudo ln -sf $PWD/git-find /usr/lib/git-core/
	$ sudo cp git-find.1 /usr/local/share/man/man1/
	… hack …
	$ sudo rm /usr/lib/git-core/git-find \

… then you can call it as “git find” and look at the documentation with “git help find”, as is customary.

The idea behind this utility is to have a tool like “git grep” that acts on the list of files known to git (and not e.g. ignored files) to quickly search for, say, all PNG files in the repository (but not the generated ones). “git find” acts on the index for the HEAD, i.e. whatever commit is currently checked-out (unlike “git grep” which also knows about “git add”ed files; fix welcome) and then offers a filter syntax similar to find(1) to follow up: parenthesēs, ! for negation, -a and -o for boolean are supported, as well as -name, -regex and -wholename and their case-insensitive variants, although regex uses grep(1) without (or, if the global option -E is given, with) -E, and the pattern matches use mksh(1)’s, which ignores the locale and doesn’t do [[:alpha:]] character classes yet. On the plus side, the output is guaranteed to be sorted; on the minus side, it is rather wastefully using temporary files (under $TMPDIR of course, so use of tmpfs is recommended). -print0 is the only output option (-print being the default).

Another mode “forwards” the file list to the system find; since it doesn’t support DOS-style response files, this only works if the amount of files is smaller than the operating system’s limit; this mode supports the full range (except -maxdepth) of the system find(1) filters, e.g. -mmin -1 and -ls, but it occurs filesystem access penalty for the entire tree and doesn’t sort the output, but can do -ls or even -exec.

The idea here is that it can collaboratively be improved, reviewed, fixed, etc. and then, should they agree, with the entire history, subtree-merged into git.git and shipped to the world.

Part of the development was sponsored by tarent solutions GmbH, the rest and the entire manual page were done in my vacation.

All 1 2 3 4

MirOS Logo