Russ Allbery: pam-krb5 4.8

This is the default Kerberos PAM module for Debian and Ubuntu systems, and supports both MIT Kerberos and Heimdal. I'm not sure how many people still use straight Kerberos PAM modules these days, with sssd taking off, but I'm still maintaining it.

This release fixes a somewhat obscure bug: if you configure the module to do expired password changes properly, it checks to see that the expired credentials can still get kadmin/changepw credentials to do the password change. However, it was setting credential options improperly on that call, which could cause it to spuriously fail if, say, krb5.conf is configured to request proxiable credentials but kadmin/changepw doesn't support proxiable credentials. Thanks to Florian Best for the excellent bug report.

The test suite in this version also works properly with Heimdal 7.0.1 and later, which changed a bunch of the messages (at the cost of skipping tests with earlier versions of Heimdal), and reports richer error messages on PKINIT failures with Heimdal. It also includes documentation fixes and lots of warning fixes, and now builds properly with tons of warnings enabled with GCC 7, Clang, and the Clang static analyzer.

You can get the latest version from the pam-krb5 distribution page.

Russ Allbery: rra-c-util 7.0

This is my collection of utility libraries and support code for (mostly) C software.

The major version bump is due a backwards-incompatible change: dropping the SA_LEN macro from portable/macros.h, including all the Autoconf machinery to probe for it. This macro came from INN's old portability code when porting to IPv6, but INN turned out to not really need it and it's never caught on. It was causing some warnings with GCC 7 that would otherwise have been hard to fix, so it was time for it to go.

There are a couple of other changes to function signatures that shouldn't matter for backward compatibility: network_sockaddr_sprint now takes a socklen_t for better type compatibility with other networking functions, and bail_krb5 and diag_krb5 in the TAP add-ons take a long as the error code argument so that they can take either a krb5_error_code or a kadm5_ret_t.

The remaining changes are all about enabling more warnings. rra-c-util now builds with intensive warnings enabled in both GCC 7 and Clang, and the warning options have been refreshed against GCC 7. It also reports clean from the Clang static analyzer, which included changing reallocarray and the vector implementations to always allocate memory of at least the minimum size.

You can get the latest version from the rra-c-util distribution page.

Russ Allbery: C TAP Harness 4.2

The functional change in this release of my test framework for C programs is the addition of a new is_blob test function. This is equivalent to ok(memcmp(...)) but it reports where the two memory regions differ as a diagnostic. This was contributed by Daniel Collins.

Otherwise, the changes are warning fixes and machinery to more easily test with warnings. C TAP Harness now supports being built with warnings with either GCC or Clang.

You can get the latest version from the C TAP Harness distribution page.

Gregor Herrmann: RC bugs 2017/30-52

for some reason I'm still keeping track of the release-critical bugs I touch, even though it's a long time since I systematically try to fix them. & since I have the list, I thought I might as well post it here, for the third (& last) time this year:

  • #720666 – src:libxml-validate-perl: "libxml-validate-perl: FTBFS: POD coverage test failure"
    don't run POD tests (pkg-perl)
  • #810655 – libembperl-perl: "libembperl-perl: postinst fails when libapache2-mod-perl2 is not installed"
    upload backported fix to jessie
  • #825011 – libdata-alias-perl: "libdata-alias-perl: FTBFS with Perl 5.24: undefined symbol: LEAVESUB"
    upload new upstream release (pkg-perl)
  • #826465 – texlive-latex-recommended: "texlive-latex-recommended: Unescaped left brace in regex is deprecated"
    propose patch
  • #851506 – cpanminus: "cpanminus: major parts of upstream sources with compressed white-space"
    take tarball from github (pkg-perl)
  • #853490 – src:libdomain-publicsuffix-perl: "libdomain-publicsuffix-perl: ftbfs with GCC-7"
    apply patch from ubuntu (pkg-perl)
  • #853499 – src:libopengl-perl: "libopengl-perl: ftbfs with GCC-7"
    new upstream release (pkg-perl)
  • #867514 – libsolv: "libsolv: find_package called with invalid argument "2.7.13+""
    propose a patch, later upload to DELAYED/1, then patch included in a maintainer upload
  • #869357 – src:libdigest-whirlpool-perl: "libdigest-whirlpool-perl FTBFS on s390x: test failure"
    upload to DELAYED/5
  • #869360 – slic3r: "slic3r: missing dependency on perlapi-*"
    upload to DELAYED/5
  • #869576 – src:gimp-texturize: "gimp-texturize: Local copy of intltool-* fails with perl 5.26"
    add patch, QA upload
  • #869578 – src:gdmap: "gdmap: Local copy of intltool-* fails with perl 5.26"
    provide a patch
  • #869579 – src:granule: "granule: Local copy of intltool-* fails with perl 5.26"
    add patch, QA upload
  • #869580 – src:teg: "teg: Local copy of intltool-* fails with perl 5.26"
    provide a patch
  • #869583 – src:gnome-specimen: "gnome-specimen: Local copy of intltool-* fails with perl 5.26"
    provide a patch
  • #869884 – src:chemical-mime-data: "chemical-mime-data: Local copy of intltool-* fails with perl 5.26"
    provide a patch, upload to DELAYED/5 later
  • #870213 – src:pajeng: "pajeng FTBFS with perl 5.26"
    provide a patch, uploaded by maintainer
  • #870821 – src:esys-particle: "esys-particle FTBFS with perl 5.26"
    propose patch
  • #870832 – src:libmath-prime-util-gmp-perl: "libmath-prime-util-gmp-perl FTBFS on big endian: Failed 2/31 test programs. 8/2885 subtests failed."
    upload new upstream release (pkg-perl)
  • #871059 – src:fltk1.3: "fltk1.3: FTBFS: Unescaped left brace in regex is illegal here in regex; marked by <-- HERE in m/(\${ <-- HERE _IMPORT_PREFIX}/lib)(?!/x86_64-linux-gnu)/ at debian/fix-fltk-targets-noconfig line 6, <> line 1."
    propose patch
  • #871159 – texlive-extra-utils: "pstoedit: FTBFS: Unescaped left brace in regex is illegal here in regex; marked by <-- HERE in m/\\([a-zA-Z]+){([^}]*)}{ <-- HERE ([^}]*)}/ at /usr/bin/latex2man line 1327."
    propose patch
  • #871307 – libmimetic0v5: "libmimetic0v5: requires rebuild against GCC 7 and symbols/shlibs bump"
    implement reporter's recipe (thanks!)
  • #871335 – src:smlnj: "smlnj: FTBFS: Unescaped left brace in regex is illegal here in regex; marked by <-- HERE in m/~?\\begin{ <-- HERE (small|Bold|Italics|Emph|address|quotation|center|enumerate|itemize|description|boxit|Boxit|abstract|Figure)}/ at mltex2html line 1411, <DOCUMENT> line 1."
    extend existing patch, QA upload
  • #871349 – src:ispell-uk: "ispell-uk: FTBFS: The encoding pragma is no longer supported at ../../bin/ line 12."
    propose patch
  • #871357 – src:packaging-tutorial: "packaging-tutorial: FTBFS: Unescaped left brace in regex is illegal here in regex; marked by <-- HERE in m/\\end{ <-- HERE document}/ at /usr/share/perl5/Locale/Po4a/ line 643."
    analyze and propose a possible patch
  • #871367 – src:fftw: "fftw: FTBFS: Unescaped left brace in regex is deprecated here (and will be fatal in Perl 5.30), passed through in regex; marked by <-- HERE in m/\@(\w+){ <-- HERE ([^\{\}]+)}/ at texi2html line 1771."
    propose patch
  • #871818 – src:debian-zh-faq: "debian-zh-faq FTBFS with perl 5.26"
    propose patch
  • #872275 – slic3r-prusa: "slic3r-prusa: Loadable library and perl binary mismatch"
    propose patch
  • #873697 – src:libtext-bibtex-perl: "libtext-bibtex-perl FTBFS on arm*/ppc64el: t/unlimited.t (Wstat: 11 Tests: 4 Failed: 0)"
    upload new upstream release prepared by smash (pkg-perl)
  • #875627 – libauthen-captcha-perl: "libauthen-captcha-perl: Random failure due to bad images"
    upload package with fixed ong prepared by xguimard (pkg-perl)
  • #877841 – src:libxml-compile-wsdl11-perl: "libxml-compile-wsdl11-perl: FTBFS Can't locate XML/Compile/ in @INC"
    add missing build dependency (pkg-perl)
  • #877842 – src:libxml-compile-soap-perl: "libxml-compile-soap-perl: FTBFS: Can't locate Test/ in @INC"
    add missing build dependencies (pkg-perl)
  • #880777 – src:pdl: "pdl build depends on removed libgd2*-dev provides"
    update build dependency (pkg-perl)
  • #880787 – src:libhtml-formatexternal-perl: "libhtml-formatexternal-perl build depends on removed transitional package lynx-cur"
    update build dependency (pkg-perl)
  • #880843 – src:libperl-apireference-perl: "libperl-apireference-perl FTBFS with perl 5.26.1"
    change handling of 5.26.1 API (pkg-perl)
  • #881058 – gwhois: "gwhois: please switch Depends from lynx-cur to lynx"
    update dependency, upload to DELAYED/15
  • #882264 – src:libtemplate-declare-perl: "libtemplate-declare-perl FTBFS with libhtml-lint-perl 2.26+dfsg-1"
    add patch for compatibility with newer HTML::Lint (pkg-perl)
  • #883673 – src:libdevice-cdio-perl: "fix build with libcdio 1.0"
    add patch from doko (pkg-perl)
  • #885541 – libtest2-suite-perl: "libtest2-suite-perl: file conflicts with libtest2-asyncsubtest-perl and libtest2-workflow-perl"
    add Breaks/Replaces/Provides (pkg-perl)

Chris Lamb: Favourite books of 2017

Whilst I managed to read just over fifty books in 2017 (down from sixty in 2016) here are ten of my favourites, in no particular order.

Disappointments this year included Doug Stanhope's This Is Not Fame, a barely coherent collection of bar stories that felt especially weak after Digging Up Mother, but I might still listen to the audiobook as I would enjoy his extemporisation on a phone book. Ready Player One left me feeling contemptuous, as did Charles Stross' The Atrocity Archives.

The worst book I finished this year was Adam Mitzner's Dead Certain, beating Dan Brown's Origin, a poor Barcelona tourist guide at best.

Year of Wonders

Geraldine Brooks

Teased by Hilary Mantel's BBC Reith Lecture appearances and not content with her short story collection, I looked to others for my fill of historical fiction whilst awaiting the final chapter in the Wolf Hall trilogy.

This book, Year of Wonders, subtitled A Novel of the Plague, is written from point of view of Anna Frith, recounting what she and her Derbyshire village experience when they nobly quarantine themselves in order to prevent the disease from spreading further.

I found it initially difficult to get to grips with the artificially aged vocabulary — and I hate to be "that guy" — but do persist until the chapter where Anna takes over the village apothecary.

The Second World Wars

Victor Davis Hanson

If the pluralisation of "Wars" is an affectation, it certainly is an accurate one: whilst we might consider the Second World War to be a unified conflict today, Hanson reasonably points out that this is a post hoc simplification of different conflicts from the late-1910s through 1945.

Unlike most books that attempt to cover the entirety of the war, this book is organised by topic instead of chronology. For example, there are two or three adjacent chapters comparing and contrasting naval strategy before moving onto land armies, constrasting and comparing Germany's eastern and western fronts, etc. This approach leads to a readable and surprisingly gripping book despite its lengthy 720 pages.

Particular attention is given to the interplay between the various armed services and how this tended to lead to overall strategic victory. This, as well as the economics of materiel, simple rates-of-replacement, combined with the irrationality and caprice of the Axis would be an fair summary of the author's general thesis — this is no Churchill, Hitler & The Unnecessary War.

Hanson is not afraid to ask "what if" questions but only where they provide meaningful explanation or provide deeper rationale rather than as an indulgent flight of fancy. His answers to such questions are invariably that some outcome would have come about.

Whilst the author is a US citizen, he does not spare his homeland from criticism, but where Hanson's background as classical-era historian lets him down is in contrived comparisons to the Peloponnesian War and other ancient conflicts. His Napoleonic references do not feel as forced, especially due to Hitler's own obsessions. Recommended.

Everybody Lies

Seth Stephens-Davidowitz

Vying for the role as the Freakonomics for the "Big Data" generation, Everybody Lies is essentially a compendium of counter-arguments, refuting commonly-held beliefs about the internet and society in general based on large-scale observations. For example:

Google searches reflecting anxiety—such as "anxiety symptoms" or "anxiety help"—tend to be higher in places with lower levels of education, lower median incomes and where a larger portion of the population lives in rural areas. There are higher search rates for anxiety in rural, upstate New York than in New York City.


On weekends with a popular violent movie when millions of Americans were exposed to images of men killing other men, crime dropped. Significantly.

Some methodological anecdotes are included: a correlation was once noticed between teens being adopted and the use of drugs and skipping school. Subsequent research found this correlation was explained entirely by the 20% of the self-reported adoptees not actually being adopted...

Although replete with the kind of factoids that force you announce them out loud to anyone "lucky" enough to be in the same room as you, Everybody Lies is let down by a chronic lack of structure — a final conclusion that is so self-aware of its limitations that it ready and repeatedly admits to it is still an weak conclusion.

The Bobiverse Trilogy

Dennis Taylor

I'm really not a "science fiction" person, at least not in the sense of reading books catalogued as such, with all their indulgent meta-references and stereotypical cover art.

However, I was really taken by the conceit and execution of the Bobiverse trilogy: Robert "Bob" Johansson perishes in an automobile accident the day after agreeing to have his head cryogenically frozen upon death. 117 years later he finds that he has been installed in a computer as an artificial intelligence. He subsequently clones himself multiple times resulting in the chapters being written from various "Bob's" locations, timelines and perspectives around the galaxy.

One particular thing I liked about the books was their complete disregard for a film tie-in; Ready Player One was almost cynically written with this in mind, but the Bobiverse cheerfully handicaps itself by including Homer Simpson and other unlicensable characters.

Whilst the opening world-building book is the most immediately rewarding, the series kicks into gear after this — as the various "Bob's" unfold with differing interests (exploration, warfare, pure science, anthropology, etc.) a engrossing tapestry is woven together with a generous helping of humour and, funnily enough, believability.

Homo Deus: A Brief History of Tomorrow

Yuval Noah Harari

After a number of strong recommendations I finally read Sapiens, this book's prequel.

I was gripped, especially given its revisionist insight into various stages of Man. The idea that wheat domesticated us (and not the other way around) and how adoption of this crop led to truncated and unhealthier lifespans particularly intrigued me: we have an innate bias towards chronocentrism, so to be reminded that progress isn't a linear progression from "bad" to "better" is always useful.

The sequel, Homo Deus, continues this trend by discussing the future potential of our species. I was surprised just how humourous the book was in places. For example, here is Harari on the anthropocentric nature of religion:

You could never convince a monkey to give you a banana by promising him limitless bananas after death in monkey heaven.

Or even:

You can't settle the Greek debt crisis by inviting Greek politicians and German bankers to a fist fight or an orgy.

The chapters on AI and the inexpensive remarks about the impact of social media did not score many points with me, but I certainly preferred the latter book in that the author takes more risks with his own opinion so it's less dry and more more thought-provoking, even if one disagrees.

La Belle Sauvage: The Book of Dust Volume One

Philip Pullman

I have extremely fond memories of reading (and re-reading, etc.) the author's Dark Materials as a teenager despite being started on the second book by a "supply" English teacher.

La Belle Sauvage is a prequel to this original trilogy and the first of another trio. Ms Lyra Belacqua is present as a baby but the protagonist here is Malcolm Polstead who is very much part of the Oxford "town" rather than "gown".

Alas, Pullman didn't make a study of Star Wars and thus relies a little too much on the existing canon, wary to add new, original features. This results in an excess of Magesterium and Mrs Coulter (a superior Delores Umbridge, by the way), and the protagonist is a little too redolent of Will...

There is also an very out-of-character chapter where the magical rules of the novel temporarily multiply resulting in a confusion that was almost certainly not the author's intention. You'll spot it when you get to it, which you should.

(I also enjoyed the slender Lyra's Oxford, essentially a short story set just a few years after The Amber Spyglass.)

Open: An Autobiography

Andre Agassi

Sporting personalities certainly exist, but they are rarely revealed by their "authors" so upon friends' enquiries to what I was reading I frequently caught myself qualifying my response with «It's a sports autobiography, but...».

It's naturally difficult to know what we can credit to Agassi or his (truly excellent) ghostwriter but this book is a real pleasure to read. This is no lost Nabokov or Proust, but the level of wordsmithing went beyond supererogatory. For example:

For a man with so many fleeting identities, it's shocking, and symbolic, that my initials are A. K. A.


I understand that there's a tax on everything in America. Now, I discover that this is the tax on success in sports: fifteen seconds of time for every fan.

Like all good books that revolve around a subject, readers do not need to know or have any real interest in the topic at hand, so even non-tennis fans will find this an engrossing read. Dark themes abound — Agassi is deeply haunted by his father, a topic I wish he went into more, but perhaps he has not done the "work" himself yet.

The Complete Short Stories

Roald Dahl

I distinctly remember reading Roald Dahl's The Wonderful Story of Henry Sugar and Six More collection of short stories as a child, some characters still etched in my mind; the 'od carrier and fingersmith of The Hitchhiker or the protagonist polishing his silver Trove in The Mildenhall Treasure.

Instead of re-reading this collection I embarked on reading his complete short stories, curious whether the rest of his œuvre was at the same level. After reading two entire volumes, I can say it mostly does — Dahl's typical humour and descriptive style are present throughout with only a few show-off sentences such as:

"There's a trick that nearly every writer uses of inserting at least one long obscure word into each story. This makes the reader think that the man is very wise and clever. I have a whole stack of long words stored away just for this purpose." "Where?" "In the 'word-memory' section," he said, epexegetically.

There were a perhaps too many of his early, mostly-factual, war tales that were lacking a an interesting conceit and I still might recommend the Henry Sugar collection for the uninitiated, but I would still heartily recommend either of these two volumes, starting with the second.

Watching the English

Kate Fox

Written by a social anthropologist, this book dissects "English" behaviour for the layman providing an insight into British humour, rites of passage, dress/language codes, amongst others.

A must-read for anyone who is in — or considering... — a relationship with an Englishman, it is also a curious read for the native Brit: a kind of horoscope for folks, like me, who believe they are above them.

It's not perfect: Fox tediously repeats that her "rules" or patterns are not rules in the strict sense of being observed by 100% of the population; there will always be people who do not, as well as others whose defiance of a so-called "rule" only reinforces the concept. Most likely this reiteration is to sidestep wearisome criticisms but it becomes ponderous and patronising over time.

Her general conclusions (that the English are repressed, risk-averse and, above all, hypocrites) invariably oversimplify, but taken as a series of vignettes rather than a scientifically accurate and coherent whole, the book is worth your investment.

(Ensure you locate the "revised" edition — it not only contains more content, it also profers valuable counter-arguments to rebuttals Fox received since the original publication.)

What Does This Button Do?

Bruce Dickinson

In this entertaining autobiography we are thankfully spared a litany of Iron Maiden gigs, successes and reproaches of the inevitable bust-ups and are instead treated to an introspective insight into just another "everyman" who could very easily be your regular drinking buddy if it weren't for a need to fulfill a relentless inner drive for... well, just about anything.

The frontman's antics as a schoolboy stand out, as are his later sojourns into Olympic fencing and being a commercial pilot. These latter exploits sound bizarre out of context but despite their non-sequitur nature they make a perfect foil (hah!) to the heavy metal.

A big follower of Maiden in my teens, I fell off the wagon as I didn't care for their newer albums so I was blindsided by Dickinson's sobering cancer diagnosis in the closing chapters. Furthermore, whilst Bruce's book fails George Orwell's test that autobiography is only to be trusted when it reveals something disgraceful, it is tour de force enough for to distract from any concept of integrity.

(I have it on excellent authority that the audiobook, which is narrated by the author, is definitely worth one's time.)

Sune Vuorela: Aubergine – Playing with emoji

Playing with emojis

At some point, I needed to copy paste emojis, but couldn’t find a good way to do it. So what does a good hacker do?
Scratch an own itch. As I wrote about in the past, all these projects should be shared with the rest of the world.
So here it is:

It looks like this with the symbola font for emojis: Screenshot

It basically lets you search for emojis by their description, and by clicking on a emoji, it gets inserted into the clipboard.

As such, I’m not sure the application is really interesting, but there might be two interesting bits in the source code:

  • A parser for the unicode data text files in /usr/share/unicode/NamesList.txt is placed in lib/parser.{h,cpp}
  • A class that tries to expose QClipboard as QML objects placed in app/clipboard.{h,cpp}. I’m not yet sure if this is the right approach for that, but it is the one that currently makes most sense in my mind. If I’m receiving proper feedback, I might be able to extend/finish it and submit it to Qt.

And of course, now it is simple to describe fancy cooking:

🍆 🔪 🔥
(aubergine) (hocho) (fire)

I ❣ emoji

Russ Allbery: Review: Saving Francesca

Review: Saving Francesca, by Melina Marchetta

Series: Francesca #1
Publisher: Alfred A. Knopf
Copyright: 2003
Printing: 2011
ISBN: 0-307-43371-4
Format: Kindle
Pages: 245

Francesca is in Year Eleven in St. Sebastian's, in the first year that the school opened to girls. She had a social network and a comfortable place at her previous school, but it only went to Year Ten. Most of her friends went to Pius, but St. Sebastian's is a better school. So Francesca is there, one of thirty girls surrounded by boys who aren't used to being in a co-ed school, and mostly hanging out with the three other girls who had gone to her previous school. She's miserable, out of place, and homesick for her previous routine.

And then, one morning, her mother doesn't get out of bed. Her mother, the living force of energy, the one who runs the household, who pesters Francesca incessantly, who starts every day with a motivational song. She doesn't get out of bed the next day, either. Or the day after that. And the bottom falls out of Francesca's life.

I come at this book from a weird angle because I read The Piper's Son first. It's about Tom Mackee, one of the supporting characters in this book, and is set five years later. I've therefore met these people before: Francesca, quiet Justine who plays the piano accordion, political Tara, and several of the Sebastian boys. But they are much different here as their younger selves: more raw, more childish, and without the understanding of settled relationships. This is the story of how they either met or learned how to really see each other, against the backdrop of Francesca's home life breaking in entirely unexpected ways.

I think The Piper's Son was classified as young adult mostly because Marchetta is considered a young adult writer. Saving Francesca, by comparison, is more fully a young adult novel. Instead of third person with two tight viewpoints, it's all first person: Francesca telling the reader about her life. She's grumpy, sad, scared, anxious, and very self-focused, in the way of a teenager who is trying to sort out who she is and what she wants. The reader follows her through the uncertainty of maybe starting to like a boy who may or may not like her and is maddeningly unwilling to commit, through realizing that the friends she had and desperately misses perhaps weren't really friends after all, and into the understanding of what friendship really means for her. But it's all very much caught up in Francesca's head. The thoughts of the other characters are mostly guesswork for the reader.

The Piper's Son was more effective for me, but this is still a very good book. Marchetta captures the gradual realization of friendship, along with the gradual understanding that you have been a total ass, extremely well. I was somewhat less convinced by Francesca's mother's sudden collapse, but depression does things like that, and by the end of the book one realizes that Francesca has been somewhat oblivious to tensions and problems that would have made this less surprising. And the way that Marchetta guides Francesca to a deeper understanding of her father and the dynamics of her family is emotionally realistic and satisfying, although Francesca's lack of empathy occasionally makes one want to have a long talk with her.

The best part of this book are the friendships. I didn't feel the moments of catharsis as strongly here as in The Piper's Son, but I greatly appreciated Marchetta's linking of the health of Francesca's friendships to the health of her self-image. Yes, this is how this often works: it's very hard to be a good friend until you understand who you are inside, and how you want to define yourself. Often that doesn't come in words, but in moments of daring and willingness to get lost in a moment. The character I felt the most sympathy for was Siobhan, who caught the brunt of Francesca's defensive self-absorption in a way that left me wincing even though the book never lingers on her. And the one who surprised me the most was Jimmy, who possibly shows the most empathy of anyone in the book in a way that Francesca didn't know how to recognize.

I'm not unhappy about reading The Piper's Son first, since I don't think it needs this book (and says some of the same things in a more adult voice, in ways I found more powerful). I found Saving Francesca a bit more obvious, a bit less subtle, and a bit more painful, and I think I prefer reading about the more mature versions of these characters. But this is a solid, engrossing psychological story with a good emotional payoff. And, miracle of miracles, even a bit of a denouement.

Followed by The Piper's Son.

Rating: 7 out of 10

Antoine Beaupré: An overview of KubeCon + CloudNativeCon

The Cloud Native Computing Foundation (CNCF) held its conference, KubeCon + CloudNativeCon, in December 2017. There were 4000 attendees at this gathering in Austin, Texas, more than all the previous KubeCons before, which shows the rapid growth of the community building around the tool that was announced by Google in 2014. Large corporations are also taking a larger part in the community, with major players in the industry joining the CNCF, which is a project of the Linux Foundation. The CNCF now features three of the largest cloud hosting businesses (Amazon, Google, and Microsoft), but also emerging companies from Asia like Baidu and Alibaba.

In addition, KubeCon saw an impressive number of diversity scholarships, which "include free admission to KubeCon and a travel stipend of up to $1,500, aimed at supporting those from traditionally underrepresented and/or marginalized groups in the technology and/or open source communities", according to Neil McAllister of CoreOS. The diversity team raised an impressive $250,000 to bring 103 attendees to Austin from all over the world.

We have looked into Kubernetes in the past but, considering the speed at which things are moving, it seems time to make an update on the projects surrounding this newly formed ecosystem.

The CNCF and its projects

The CNCF was founded, in part, to manage the Kubernetes software project, which was donated to it by Google in 2015. From there, the number of projects managed under the CNCF umbrella has grown quickly. It first added the Prometheus monitoring and alerting system, and then quickly went up from four projects in the first year, to 14 projects at the time of this writing, with more expected to join shortly. The CNCF's latest additions to its roster are Notary and The Update Framework (TUF, which we previously covered), both projects aimed at providing software verification. Those add to the already existing projects which are, bear with me, OpenTracing (a tracing API), Fluentd (a logging system), Linkerd (a "service mesh", which we previously covered), gRPC (a "universal RPC framework" used to communicate between pods), CoreDNS (DNS and service discovery), rkt (a container runtime), containerd (another container runtime), Jaeger (a tracing system), Envoy (another "service mesh"), and Container Network Interface (CNI, a networking API).

This is an incredible diversity, if not fragmentation, in the community. The CNCF made this large diagram depicting Kubernetes-related projects—so large that you will have a hard time finding a monitor that will display the whole graph without scaling it (seen below, click through for larger version). The diagram shows hundreds of projects, and it is hard to comprehend what all those components do and if they are all necessary or how they overlap. For example, Envoy and Linkerd are similar tools yet both are under the CNCF umbrella—and I'm ignoring two more such projects presented at KubeCon (Istio and Conduit). You could argue that all tools have different focus and functionality, but it still means you need to learn about all those tools to pick the right one, which may discourage and confuse new users.

Cloud Native landscape

You may notice that containerd and rkt are both projects of the CNCF, even though they overlap in functionality. There is also a third Kubernetes runtime called CRI-O built by RedHat. This kind of fragmentation leads to significant confusion within the community as to which runtime they should use, or if they should even care. We'll run a separate article about CRI-O and the other runtimes to try to clarify this shortly.

Regardless of this complexity, it does seem the space is maturing. In his keynote, Dan Kohn, executive director of the CNCF, announced "1.0" releases for 4 projects: CoreDNS, containerd, Fluentd and Jaeger. Prometheus also had a major 2.0 release, which we will cover in a separate article.

There were significant announcements at KubeCon for projects that are not directly under the CNCF umbrella. Most notable for operators concerned about security is the introduction of Kata Containers, which is basically a merge of runV from and Intel's Clear Containers projects. Kata Containers, introduced during a keynote by Intel's VP of the software and services group, Imad Sousou, are virtual-machine-based containers, or, in other words, containers that run in a hypervisor instead of under the supervision of the Linux kernel. The rationale here is that containers are convenient but all run on the same kernel, so the compromise of a single container can leak into all containers on the same host. This may be unacceptable in certain environments, for example for multi-tenant clusters where containers cannot trust each other.

Kata Containers promises the "best of both worlds" by providing the speed of containers and the isolation of VMs. It does this by using minimal custom kernel builds, to speed up boot time, and parallelizing container image builds and VM startup. It also uses tricks like same-page memory sharing across VMs to deduplicate memory across virtual machines. It currently works only on x86 and KVM, but it integrates with Kubernetes, Docker, and OpenStack. There was a talk explaining the technical details; that page should eventually feature video and slide links.

Industry adoption

As hinted earlier, large cloud providers like Amazon Web Services (AWS) and Microsoft Azure are adopting the Kubernetes platform, or at least its API. The keynotes featured AWS prominently; Adrian Cockcroft (AWS vice president of cloud architecture strategy) announced the Fargate service, which introduces containers as "first class citizens" in the Amazon infrastructure. Fargate should run alongside, and potentially replace, the existing Amazon EC2 Container Service (ECS), which is currently the way developers would deploy containers on Amazon by using EC2 (Elastic Compute Cloud) VMs to run containers with Docker.

This move by Amazon has been met with skepticism in the community. The concern here is that Amazon could pull the plug on Kubernetes when it hinders the bottom line, like it did with the Chromecast products on Amazon. This seems to be part of a changing strategy by the corporate sector in adoption of free-software tools. While historically companies like Microsoft or Oracle have been hostile to free software, they are now not only using free software but also releasing free software. Oracle, for example, released what it called "Kubernetes Tools for Serverless Deployment and Intelligent Multi-Cloud Management", named Fn. Large cloud providers are getting certified by the CNCF for compliance with the Kubernetes API and other standards.

One theory to explain this adoption is that free-software projects are becoming on-ramps to proprietary products. In this strategy, as explained by InfoWorld, open-source tools like Kubernetes are merely used to bring consumers over to proprietary platforms. Sure, the client and the API are open, but the underlying software can be proprietary. The data and some magic interfaces, especially, remain proprietary. Key examples of this include the "serverless" services, which are currently not standardized at all: each provider has its own incompatible framework that could be a deliberate lock-in strategy. Indeed, a common definition of serverless, from Martin Fowler, goes as follows:

Serverless architectures refer to applications that significantly depend on third-party services (knows as Backend as a Service or "BaaS") or on custom code that's run in ephemeral containers (Function as a Service or "FaaS").

By designing services that explicitly require proprietary, provider-specific APIs, providers ensure customer lock-in at the core of the software architecture. One of the upcoming battles in the community will be exactly how to standardize this emerging architecture.

And, of course, Kubernetes can still be run on bare metal in a colocation facility, but those costs are getting less and less affordable. In an enlightening talk, Dmytro Dyachuk explained that unless cloud costs hit $100,000 per month, users may be better off staying in the cloud. Indeed, that is where a lot of applications end up. During an industry roundtable, Hong Tang, chief architect at Alibaba Cloud, posited that the "majority of computing will be in the public cloud, just like electricity is produced by big power plants".

The question, then, is how to split that market between the large providers. And, indeed, according to a CNCF survey of 550 conference attendees: "Amazon (EC2/ECS) continues to grow as the leading container deployment environment (69%)". CNCF also notes that on-premise deployment decreased for the first time in the five surveys it has run, to 51%, "but still remains a leading deployment". On premise, which is a colocation facility or data center, is the target for these cloud companies. By getting users to run Kubernetes, the industry's bet is that it makes applications and content more portable, thus easier to migrate into the proprietary cloud.

Next steps

As the Kubernetes tools and ecosystem stabilize, major challenges emerge: monitoring is a key issue as people realize it may be more difficult to diagnose problems in a distributed system compared to the previous monolithic model, which people at the conference often referred to as "legacy" or the "old OS paradigm". Scalability is another challenge: while Kubernetes can easily manage thousands of pods and containers, you still need to figure out how to organize all of them and make sure they can talk to each other in a meaningful way.

Security is a particularly sensitive issue as deployments struggle to isolate TLS certificates or application credentials from applications. Kubernetes makes big promises in that regard and it is true that isolating software in microservices can limit the scope of compromises. The solution emerging for this problem is the "service mesh" concept pioneered by Linkerd, which consists of deploying tools to coordinate, route, and monitor clusters of interconnected containers. Tools like Istio and Conduit are designed to apply cluster-wide policies to determine who can talk to what and how. Istio, for example, can progressively deploy containers across the cluster to send only a certain percentage of traffic to newly deployed code, which allows detection of regressions. There is also work being done to ensure standard end-to-end encryption and authentication of containers in the SPIFFE project, which is useful in environments with untrusted networks.

Another issue is that Kubernetes is just a set of nuts and bolts to manage containers: users get all the parts and it's not always clear what to do with them to get a platform matching their requirements. It will be interesting to see how the community moves forward in building higher-level abstractions on top of it. Several tools competing in that space were featured at the conference: OpenShift, Tectonic, Rancher, and Kasten, though there are many more out there.

The 1.9 Kubernetes release should be coming out in early 2018; it will stabilize the Workloads API that was introduced in 1.8 and add Windows containers (for those who like .NET) in beta. There will also be three KubeCon conferences in 2018 (in Copenhagen, Shanghai, and Seattle). Stay tuned for more articles from KubeCon Austin 2017 ...

This article first appeared in the Linux Weekly News.

Shirish Agarwal: The VR Show

One of the things If I had got the visa on time for Debconf 15 (Germany) apart from the conference itself was the attention on VR (Virtual Reality) and AR (Augmented Reality) . I had heard the hype so much for so many years that I wanted to experience and did know that with Debianities who might be perhaps a bit better in crystal-gazing and would have perhaps more of an idea as I had then. The only VR which I knew about was from Hollywood movies and some VR videos but that doesn’t tell you anything. Also while movie like Chota-Chetan and others clicked they were far lesser immersive than true VR has to be.

I was glad that it didn’t happen after the fact as in 2016 while going to the South African Debconf I experienced VR at Qatar Airport in a Samsung showroom. I was quite surprised as how heavy the headset was and also surprised by how little content they had. Something which has been hyped for 20 odd years had not much to show for it. I was also able to trick the VR equipment as the eye/motion tracking was not good enough so if you put shook the head fast enough it couldn’t keep up with you.

I shared the above as I was invited to another VR conference by a web-programmer/designer friend Mahendra couple of months ago here in Pune itself . We attended the conference and were showcased quite a few success stories. One of the stories which was liked by the geek in me was framastore’s 360 Mars VR Experience on a bus the link shows how the framastore developers mapped Mars or part of Mars on Washington D.C. streets and how kids were able to experience how it would feel to be on Mars without knowing any of the risks the astronauts or the pioneers would have to face if we do get the money, the equipment and the technology to send people to Mars. In reality we are still decades from making such a trip keeping people safe to Mars and back or to have Mars for the rest of their life.

If my understanding is correct, the gravity of Mars is half of earth and once people settle there they or their exoskeleton would no longer be able to support Earth’s gravity, at least a generation who is born on Mars.

An interesting take on how things might turn out is shown in ‘The Expanse

But this is taking away from the topic at hand. While I saw the newer generation VR headsets there are still a bit ways off. It would be interesting once the headset becomes similar to eye-glasses and you do not have to either be tethered to a power unit or need to lug a heavy backpack full of dangerous lithium-ion battery. The chemistry for battery or some sort of self-powered unit would need to be much more safer, lighter.

While being in the conference and seeing the various scenarios being played out between potential developers and marketeers, it crossed my mind that people were not at all thinking of safe-guarding users privacy. Right from what games or choices you make to your biometric and other body sensitive information which has a high chance of being misused by companies and individuals.

There were also questions about how Sony and other developers are asking insane amounts for use of their SDK to develop content while it should be free as games and any content is going to enhance the marketability of their own ecosystem. For both the above questions (privacy and security asked by me) and SDK-related questions asked by some of the potential developers were not really answered.

At the end, they also showed AR or Augmented Reality which to my mind has much more potential to be used for reskilling and upskilling of young populations such as India and other young populous countries. It was interesting to note that both China and the U.S. are inching towards the older demographics while India would relatively be a still young country till another 20-30 odd years. Most of the other young countries (by median age) seem to be in the African continent and I believe (might be a myth) is that they are young because most of the countries are still tribal-like and they still are perhaps a lot of civil wars for resources.

I was underwhelmed by what they displayed in Augmented Reality, part of which I do understand that there may be lot many people or companies working on their IP and hence didn’t want to share or show or show a very rough work so their idea doesn’t get stolen.

I was also hoping somebody would take about motion-sickness or motion displacement similar to what people feel when they are train-lagged or jet-lagged. I am surprised that wikipedia still doesn’t have an article on train-lag as millions of Indians go through the process every year. The one which is most pronounced on Indian Railways is Motion being felt but not seen.

There are both challenges and opportunities provided by VR and AR but until costs come down both in terms of complexity, support and costs (for both the deployer and the user) it would remain a distant dream.

There are scores of ideas that could be used or done. For instance, the whole of North India is one big palace in the sense that there are palaces built by Kings and queens which have their own myth and lore over centuries. A story-teller could use a modern story and use say something like Chota Imambara or/and Bara Imambara where there have been lots of stories of people getting lost in the alleyways.

Such sort of lore, myths and mysteries are all over India. The Ramayana and the Mahabharata are just two of the epics which tell how grand the tales could be spun. The History of Indus Valley Civilization till date and the modern contestations to it are others which come to my mind.

Even the humble Panchtantra can be re-born and retold to generations who have forgotten it. I can’t express it much better as the variety of stories and contrasts to offer as bolokids does as well as SRK did in opening of IFFI. Even something like Khakee which is based on true incidents and a real-life inspector could be retold in so many ways. Even Mukti Bhavan which I saw few months ago, coincidentally before I became ill tells of stories which have complex stories and each person or persons have their own rich background which on VR could be much more explored.

Even titles such as the ever-famous Harry Potter or even the ever-beguiling RAMA could be shared and retooled for generations to come. The Shiva Trilogy is another one which comes to my mind which could be retold as well. There was another RAMA trilogy by the same author and another competing one which comes out in 2018 by an author called PJ Annan

We would need to work out the complexities of both hardware, bandwidth and the technologies but stories or content waiting to be developed is aplenty.

Once upon a time I had the opportunity to work, develop and understand make-believe walk-throughs (2-d blueprints animated/bought to life and shown to investors/clients) for potential home owners in a society (this was in the hey-days and heavy days of growth circa around y2k ) , it was 2d or 2.5 d environment, tools were lot more complex and I was the most inept person as I had no idea of what camera positioning and what source of light meant.

Apart from the gimmickry that was shown, I thought it would have been interesting if people had shared both the creative and the budget constraints while working in immersive technologies and bringing something good enough for the client. There was some discussion in a ham-handed way but not enough as there was considerable interest from youngsters to try this new medium but many lacked both the opportunities, knowledge, the equipment and the software stack to make it a reality.

Lastly, as far as the literature I have just shared bits and pieces of just the Indian English literature. There are 16 recognized Indian languages and all of them have a vibrant literature scene. Just to take an example, Bengal has been a bed-rock of new Bengali Detective stories all the time. I think I had shared the history of Bengali Crime fiction sometime back as well but nevertheless here it is again.

So apart from games, galleries, 3-d visual interactive visual novels with alternative endings could make for some interesting immersive experiences provided we are able to shed the costs and the technical challenges to make it a reality.

Filed under: Miscellenous Tagged: #Augmented Reality, #Debconf South Africa 2016, #Epics, #framastore, #indian literature, #Mars trip, #median age population inded, #motion sickness, #Palaces, #planet-debian, #Pune VR Conference, #RAMA, #RAMA trilogy, #Samsung VR, #Shiva Trilogy, #The Expanse, #Virtual Reality, #VR Headsets, #walkthroughs, Privacy

Steinar H. Gunderson: Compute rescaling progress

My Lanczos rescaling compute shader for Movit is finally nearing usable performance improvements:

BM_ResampleEffectInt8/Fragment/Int8Downscale/1280/720/640/360         3149 us      69.7767M pixels/s
BM_ResampleEffectInt8/Fragment/Int8Downscale/1280/720/320/180         2720 us      20.1983M pixels/s
BM_ResampleEffectHalf/Fragment/Float16Downscale/1280/720/640/360      3777 us      58.1711M pixels/s
BM_ResampleEffectHalf/Fragment/Float16Downscale/1280/720/320/180      3269 us      16.8054M pixels/s

BM_ResampleEffectInt8/Compute/Int8Downscale/1280/720/640/360          2007 us      109.479M pixels/s  [+ 56.9%]
BM_ResampleEffectInt8/Compute/Int8Downscale/1280/720/320/180          1609 us      34.1384M pixels/s  [+ 69.0%]
BM_ResampleEffectHalf/Compute/Float16Downscale/1280/720/640/360       2057 us      106.843M pixels/s  [+ 56.7%]
BM_ResampleEffectHalf/Compute/Float16Downscale/1280/720/320/180       1633 us      33.6394M pixels/s  [+100.2%]

Some tuning and bugfixing still needed; this is on my Haswell (the NVIDIA results are somewhat different). Upscaling also on its way. :-)