Martin Probst

AI wrote the compiler, the test suites planned it

2026-02-06T03:54:48+00:00

Anthropic recently described how a team of Claude agents built a C compiler, presented as a project where they handed the task to the agents and then mostly stepped back: about 100,000 lines of Rust that can build a bootable Linux kernel and pass 99% of most compiler test suites, including GCC's torture tests.

But: the agents had access to extremely detailed and comprehensive test suites and execution harnesses, both written by humans, with the harness built specifically for the AI to consume. And they got a working compiler, but no commercial-grade product.

This is still quite the achievement, don't get me wrong.

But I'd expect the test suites to go a long way not just in validating the result, but also in structuring the task. The AI didn't solve "how do I compile Linux", but "there's a test with this description, part of the built-ins suite, to correctly handle GCC's __attribute__((constructor)) declaration attribute; get the compiler to emit this specific assembly for this input".

That is, some of the input wasn't just what to do, but also how to structure the compiler, how to break the overall goal down into jobs, and how precisely to validate them.

I think the "mostly walked away" framing could have been more honest about that. I guess "we got Claude to follow along these test suites, until finally getting Linux to compile" is a bit less impressive, though.

Deployment frequency is not a goal

2025-09-13T10:11:41+00:00

Developer productivity efforts like to measure software delivery with the DORA metrics, among them deployment frequency and lead time for changes. These metrics are real and useful, but what's annoying is the common use of deployment frequency as a goal metric.

Deployment frequency has a hard cut-off, beyond which more frequent deployments aren't useful, or are even harmful. Monthly releases are a problem. Weekly - or even daily - deployments are fine, as long as they are automated or low toil. Releasing every push is harmful.

It's just not beneficial to anyone to deploy every five minutes. It doesn't meaningfully accelerate delivery (no customer ever wanted that button all of five minutes earlier!). And it's really harmful for production stability, as it becomes increasingly difficult to attribute breakages to particular releases.

I'd argue similarly for lead time to production. For a significant product, you want a careful and measured rollout across dev, QA, and then individual canary stages and production clusters. You cannot do meaningful canarying in five minutes, so a shorter lead time to production will be strictly worse (assuming you don't waste time on manual QA).

Why Go's garbage collector isn't generational

2020-02-05T13:06:50+00:00

Originally posted as a comment on Hacker News on Why Discord is switching from Go to Rust. Another commenter explained Discord's garbage collection problems like this:

The problem is that garbage collectors are optimized for applications that mostly have short-lived objects, and a small amount of long-lived objects.

I think it's not quite that.

Applications typically have a much larger old generation than young generation, i.e. many more long-lived objects than short-lived objects. So GCs do get optimized to process large heaps of old objects quickly and efficiently, e.g. with concurrent mark/sweep.

However, there is an additional observation: once an application has reached steady state, most newly allocated objects die young (think: the data associated with processing a single HTTP request or user interaction in a UI).

So as an additional optimization, GCs often split their heap into a young and an old generation, where garbage collecting the young generation earlier/more frequently overall reduces the amount of garbage collection done (and offsets the effort required to move objects around).

In the case of Go, though, the programming language allows "internal pointers", i.e. pointers to members of objects. This makes it much harder (or much more costly) to implement a generational, moving garbage collector, so Go doesn't actually have a young/old generation split, nor the additional optimization for young objects.

Opportunistic fixing vs. long-term planning

2018-09-13T08:59:09+00:00

Originally posted as a comment on Hacker News on Ending PHP Support, and the Future of Hack, the announcement that Facebook's HHVM would drop PHP support and focus on Hack, Facebook's own language derived from PHP.

It's interesting to see HHVM drop PHP support, particularly if you look at it from an organizational decision-making standpoint. Hypothetically, because I'm not involved in any of this, but still:

Facebook was initially implemented using PHP, and you can read this process as continued iteration based on this very early decision. From the outside, Facebook seems to have always taken what they had, taken the problems they currently had, and made the most incremental change they could to fix whatever problem they had. From PHP to HHVM to Hack to dropping PHP support, they continued iterating on whatever code base and development chain they currently had.

If you take a step back, though, Facebook is gradually moving to a different programming environment, from a dynamic language with very loose semantics to a statically typed system with (from code I've seen in e.g. Phabricator) a very different programming style than run-of-the-mill PHP. While iterating incrementally, they still ended up forking PHP and moving to a different language.

The approach is quite different from e.g. Twitter, though, who started off using Ruby on Rails, and then moved everything to an entirely different toolchain (Scala) when they found Rails couldn't keep up with their problem set.

This difference in approaches isn't quite the same as the good old "rewrite vs. refactor" dichotomy. Twitter's change was certainly incremental too, moving individual services to Scala bit by bit and setting them live incrementally, instead of a "big bang" rewrite where everything moves to Scala one day.

I think the main difference is planning horizon: do you opportunistically fix whatever is broken right now, or do you make a long-term plan for what you want your future platform to be, and then set out to get there?

Opportunistic fixing has the advantage that you can be pretty certain you're fixing the right problem, and don't invest in YAGNI features. On the other hand, you risk ending up in local optima. Pouring a lot of effort into PHP makes sense if you plan to continue using it - but if you end up forking the language anyway, you lose a lot of the advantages of a shared ecosystem (and arguably damage the existing ecosystem in the process through fragmentation).

You could equally well imagine Facebook picking a different future tech stack five years ago, be it Rust or Go or whatever, and adding the features they miss from PHP to it. I think it's an interesting thought experiment where that'd have taken them. I have no idea if taking that path would have been better for any of {Facebook, PHP, target language X, world}.

Why a different language can't replace JavaScript

2018-09-02T13:27:34+00:00

Originally posted as a comment on Hacker News on Evan Martin's post TypeScript at Google. At the time, Evan and I were working on TypeScript at Google together, and this describes the thinking behind our approach. The comment replied to this passage from the post:

The first tempting option is to just abandon this ruined planet and settle a new one that doesn't even involve JavaScript. If only we invested more in GWT (a Google project that compiles Java to JavaScript) or Dart (a Google project that compiles a new language to JavaScript) or WASM or [insert your favorite language here — Clojure? Haxe? Elm?] we wouldn't need to worry about JavaScript at all!

I thought about this for quite a while when we started on TypeScript at Google. I've convinced myself that using a different language cannot solve the problem that we're looking at.

The argument goes like this: if you use a different language than JavaScript for front end development, you're (hopefully) doing it for a reason. That is, you (a) want the other language's semantics (e.g. bounds checks on array accesses, runtime types, method dispatch, you name it), or you (b) might want the programming language's ecosystem: editors, IDEs, etc., but most importantly libraries and frameworks. Most likely, you're looking for a combination of (a) and (b).

The problem with (a) is that the more different the runtime semantics of your programming language are, the more emulation code you need. Emulation code is costly in code size, performance, interop with JS, and often understandability (at some point you will need to debug the output, believe me).

The problem with (b) is that you typically opt into a large ecosystem that was originally written for a very different environment. This was a big issue with GWT: people would pull in Java libraries, but those libraries, amazing as many are, were not written for the web or with code size in mind. This meant using a common library like Guava required a lot of engineering effort to keep code size in check. This also applies to WASM, possibly even more so.

These mechanisms work together to make it very hard to hit the sweet spot between performance, developer experience, and interoperability. And ironically, the better you get at the goals you're aiming for (different semantics, different ecosystem), the farther you get away from a working solution.

I don't think it's impossible to achieve a transpile-to-JS language that works really well with enough investment, but keeping in mind the other reasons Evan lists (mind share, existing code base), incrementally improving JavaScript where needed is IMHO the much more promising approach.

Software techniques are tradeoffs, not beliefs

2018-07-01T21:45:55+00:00

Originally posted as a comment on Hacker News in a discussion of Lost in Math. Another commenter argued that software development isn't really engineering: we can measure little about it, and debates about "clean" or "beautiful" code come down to subjective positions, fanatically defended by "true believers".

True believers are a problem indeed - people who seem to think software development is about applying firmly held beliefs out of context can really screw things up royally.

That being said, by now software engineering has a large set of techniques with reasonably well-understood properties, and we can absolutely reason through whether a technique is useful to solve a particular problem.

For example, polymorphism is a useful technique to invert the locality of implementation details, making it easier to extend software in a distributed way. Its advantage is its drawback: it makes control flow harder to follow, and generally makes the software more difficult to understand, precisely due to the distributed extensibility.

Similarly, mocking is a technique to reduce testing "surface" by removing dependencies on specific components from a test. It's a great tool to make tests easier to write, improve test performance, and avoid ambient test dependencies and associated complexity. On the other hand, it reduces test depth by ignoring potentially important interactions, and often leads to overly brittle, hard-to-maintain tests that add little value in actually testing the software.

It's easy to go on. None of these techniques are silver bullets or axioms to be followed religiously; they are specific solutions to specific problems. They aren't necessarily measurable in an analog fashion, such as the load-bearing capability of a steel beam; they are often more binary (quite fitting, given the domain). But you can judge whether they are useful to solve a specific problem.

That is, I don't follow the "it's all just opinion" kind of software engineering defeatism. Yes, there's a ton of extremely questionable evangelism, often spread with huge pomp and circumstance by people with limited insight. But that's really just that: limited insight.

Why vendoring every dependency doesn't scale

2015-07-10T09:15:47+00:00

Originally posted as a comment on Hacker News on Go 1.5's vendor/ experiment. Go 1.5 added experimental support for vendor/ directories, in which a package keeps its own copies of the packages it depends on.

Go's vendoring is similar to node_modules + npm, in essence.

Having used node and npm on both small (~200 transitive deps) and large (~1000 transitive deps) projects, I think this is an approach that's conceptually not a good idea (outside of possible implementation issues with npm).

With vendoring, you end up with a very large tree of transitive vendored dependencies. You have no idea of, or control over, what's in it, and what versions of what packages are in it.

Nearly all of the time, the two allegedly incompatible versions of a library that you install separately turn out to be 2.0.13 and 2.0.14, and you only get two because, well, somebody hasn't upgraded yet.

This might seem trivial, but in the end you have a project tree where every dependency gets installed separately for everything that depends on it, so you end up with something like O(number of deps * number of deps) installed libraries, which obviously doesn't scale.

You also end up with the versioning problem. Imagine there's an important reason to upgrade some package P, e.g. a security issue. Now you have to hunt down every single project in your transitive dependencies that depends on it, and ask them to upgrade or fork.

You also still have version issues. If you transitively depend on the same library C in versions Cv1 and Cv2, and you want to pass an object/data structure/... from Cv1 to Cv2 through some path, you open yourself up to all kinds of hilarious version mismatches and failures.

This isn't new - e.g. Eclipse/OSGi did the same thing with separated class loaders, and it was a mess.

I think flattening your dependency/vendor tree to only ever have one version of library C, and hoping/testing for minor version mismatches to work out, is the most viable thing to do.

clang-format is available via npm

2015-04-11T19:24:34.097742+00:00

Thanks to the effort of my colleague Alex Eagle (mostly), clang-format is now available via npm. You can install it via npm install -g clang-format, and then just use the clang-format command normally. Binaries are available for Mac OS X and Linux, both 64 bit.

Beyond JavaScript, clang-format now does a pretty decent job at formatting TypeScript, too, including support for classes, interfaces, generics, and of course type annotations. While TypeScript comes with a simple indenter, that doesn't really support full formatting - e.g. it won't wrap overlong lines, doesn't align function parameters, etc. This is of course highly subjective, but I think clang-format does a better job, and it's nice to be able to use the same formatter across more or less all your languages (C++, Java, JS, TypeScript).

See my clang-format for JavaScript for more details on how to use it.

clang-format for JavaScript

2015-01-15T16:24:54.985878+00:00

clang-format is a very neat code formatter developed as part of the Clang project by my colleague Daniel Jasper here at Google. It's so good that many projects have decided to just completely rely on clang-format and never indent code manually nor use a linter. clang-format is reasonably well known in the C++ community by now, but it's much less well known that it also works for a bunch of other vaguely ALGOL-shaped programming languages. I've been working a bit with Daniel to make sure it works on JavaScript, which sorely lacks a robust formatter.

Two days ago my pull request to Homebrew was accepted (yay!), so you can now easily install clang-format on any Mac and enjoy awesome code formatting.

Install homebrew
brew install clang-format
There's no step three

Now when you run clang-format -style=Google somefile.js, it'll automatically spit a nicely indented version back at you. For example this terrible mess:

function Foo() {
     this.bar =
    function() {this.doX(); this.doY();
  for (var x =
0; x < 12; x++) { if (this.z) 
    new 
 Foo(). callWithArgs   (
        {arg1: 'hello',
arg2: 'world'});}
    };          this.baz = 123;
           this.bam = {some: 'object literal', including: function() { calls(); }, asdSomeMoreKeysToWrap: 123};
}

With one keypress turns into this nicely indented piece:

function Foo() {
  this.bar = function() {
    this.doX();
    this.doY();
    for (var x = 0; x < 12; x++) {
      if (this.z) new Foo().callWithArgs({arg1: 'hello', arg2: 'world'});
    }
  };
  this.baz = 123;
  this.bam = {
    some: 'object literal',
    including: function() { calls(); },
    asdSomeMoreKeysToWrap: 123
  };
}

I really like it - it takes a lot of the tedium out of writing code, and its indentation is really beautiful - usually much nicer than what I would bother with on a typical day.

clang-format already has integration with many editors, e.g. vim, or Sublime Text, and the command line is straight forwad to use.

Discuss on HN

Go on App Engine and GoSublime (golang, code completion)

2014-05-26T19:54:46.090525+00:00

I'm re-writing this blog using Go on App Engine, just for the fun. It took me quite a while to wrap my head around the available tool options, and get them set up correctly.

It turns out Sublime Text (my favorite text editor as of late) has a really nice plugin called GoSublime that provides various Go tools, including auto-completion through go code.

Sadly it also turns out that GoSublime is a bit fickle to get to work with App Engine's Go libraries, possibly because of a bug. I think at this time, it's not possible to have multiple Go workspaces with GoSublime that use different GOROOT paths (e.g. plain go + App Engine go).

But at least it's possible to get consistent code completion and other GoSublime features working, with these global configuration settings in Preferences > Package Settings > GoSublime > Settings - User:

{
  "installsuffix": "appengine",
  "env": {
    // Your GOPATH for App Engine projects, e.g.
    "GOPATH": "/Users/martin/goae",
    // Your GOROOT, e.g.
    "GOROOT": "/usr/local/Cellar/go-app-engine-64/1.9.5/share/go-app-engine-64/goroot"
  },
  // ... any other settings you have ...
}

The source code running this puny blog is available at http://github.com/mprobst/blog by the way. It has zero features and is the first thing I ever wrote in Go, so probably not a good idea to take as an example for much.

Blog on AppEngine, Java, Jersey / JAX-RS

2012-03-07T21:02:22.574000+00:00

Continuing the theme that I mostly use this blog to experiment with software, not so much for writing an actual blog, I moved over the software from Django/Python to Java, both running on AppEngine.

Django Out

First of all, I'll have to admit that I'm not a Python programmer. I can program in the language, but I don't feel at home. I've never really used it long enough at a time to grow familiar with the APIs and the pythonic way of doing things. I still struggle to remember the basic string manipulation APIs and such. I've always felt that Python is a nice, clean language (except for the constant "self." that is), I've just never gotten around to use it enough.

That being said, coming from Rails, Django is a big step back, and Django nonrel, the patch to make Django run on AppEngine, is worse. Django itself lacks a lot of things taken for granted in Rails, and Django non-rel is not only cumbersome to install, it also lacks many of the compelling features of Django (e.g. the automated administration backend).

So I didn't really see a compelling reason to stay with Django, and as I wanted to move away anyway, I could go back to Java as well. I also wanted to play around with a couple of Java technology pieces that seem nice.

Jersey In

I've used Jersey from time to time before. It's a web framework that revolves mostly around HTTP. Jersey (and JAX-RS) is not trying to abstract HTTP away, it's trying to model HTTP well and give a good framework to program on the web, which resounds with me a lot. It's geared towards writing RESTful APIs more than towards writing actual dynamic web sites, so a bit of setup/customization is required, but nothing too bad.

Jersey of course only gives you the basic web interaction layer, i.e. the controller. I'm using Objectify to persist Plain Old Java Objects (POJOs) for the (trivial) models, Closure Templates to render views, and Gson to convert model objects into JSON like data structures for the views (Closure doesn't support accessing POJOs directly so that it can support Java and JavaScript from the same templates).

Validation

I haven't really found any compelling library for validation. JSR 303's model driven approach through annotations seems nice, Spring Validation is more of a (quite trivial?) utility library, and I'm not going to touch Apache Commons Validator and its XML configuration with a 10 foot pole. I ended up just rolling something similar to Spring's support library on my own; after all it's only a trivial blog.

Form Support

The other thing missing from a productive toolkit is converting from submitted form data to objects (application/x-www-url-encoded, not multipart/form-data). Spring MVC offers something like that with its Command or Form objects, but that seems not very useful if not combined with all the rest of Spring. Given that we have an excellent library for JSON->POJO conversion with Gson, maybe it'd just be a naming convention for form fields that allows translating them to JSON (a bit like Objectify translates Datastore's simple maps to objects). Again, not really needed for a blog.

Webstack

Overall I'm quite happy with it. If you don't opt into Java's traditional AbstractViewHandlerControllerFactory patterns (Spring, I'm looking at you) you can end up with pretty productive environments. None of the components I use are overly ceremonial, nothing needs "configuration", and no XML involved in any form, you can even (mostly) get around web.xml and appengine-web.xml :-) I do realize that all of this is complete overkill for a tiny blog, as is writing your own blog in the first place. On the other hand, I kind of enjoy putting these pieces together and writing something on AppEngine, and I'm always interested in seeing how hard it is or how easy it should be to build certain things.

It's always hard to get started from nothing, and I suspect this might be interesting for others. I'll write a couple of blog posts about the different parts of the system, including putting it together using Guice.

Exporting from iPhoto to Picasa Web

2012-01-14T11:16:41.905000+00:00

I have a significant collection of photos on my home machine, all stored in iPhoto. I back them up to an external hard drive, using time machine, and copy them to a second machine using rsync every now and then. However all of that is within my apartment. That's not only somewhat dangerous (think fire, burglary, ... - at least if you're paranoid enough), it also means I cannot access the photos from anywhere.

So my current weekend project is to upload them to Picasa Web. I actually like iPhoto, and I don't really want to switch to Picasa to manage my photos, so I have to get the pictures out of iPhoto into Picasa Web.

Google has two tools for that, a standalone uploader, and an export plugin for iPhoto (that curiously lives in a completely different place of the UI than iPhoto's other export plugins ...). Both are available in a single download here. Neither of them does what I want though - the standalone uploader doesn't understand iPhoto or its albums at all, and the iPhoto plugin is a manual process. You select individual albums and upload them. That's probably useful if you want to export a single album, but not for a complete export. I need a background task that automatically does the uploading, preferably a restartable one.

Background process, automated? Command line to the rescue! :-)

It turns out there's a very straightforward iPhoto export tool that also supports incremental exports, exportiphoto on Github. Downloading that and running exportiphoto -d "$HOME/Pictures/iPhoto Library" $HOME/tmp/iphotoexport worked like a charm (-d to suppress dates prepended to the event name). Some ~5 of my pictures were referenced in the album metadata but inexplicably missing on the hard disk, I simply deleted them.

This leaves you with a directory structure of "$event/$picture" in tmp/iphotoexport for each of your events (unrelated: why is this not a built-in feature of iPhoto?). Now for the uploading part - uploading to Picasa is surprisingly easy using the Google command line. It requires the Google client libraries, but both are available in Python's easy_install repository, so simply running sudo easy_install googlecl gives you all you need. Creating an album from a set of pictures is as simple as google picasa create "My Album" my_album/* --date=2012-01-03. We can get the date for the album from the minimum last modified time of all exported files in the album using some ruby trickery, the dates of the pictures themselves are taken from the photo metadata. The command line uses OAuth 2 to log you in, which is a bit spooky - it pops up a browser in which you give the permissions, and then proceeds on the command line. Nice to see OAuth working so well, I really have to find some time and understand that.

So TL;DR:

curl https://raw.github.com/BMorearty/exportiphoto/master/exportiphoto.py > exportiphoto.py
sudo easy_install googlecl

python exportiphoto.py "Pictures/iPhoto Library" "tmp/iphotoexport"
cd tmp/iphotoexport

for album in *; do
  echo Uploading $album...
  min_last_modified=`ruby -e 'require "time"; puts Dir[ARGV[0] + "/*"].map { |x| File.mtime(x) }.min.strftime("%F")' "$album"`
  google picasa create "$album" $album/* --date=$min_last_modified
done

As for the incremental, resumable part, I still have no idea. I'll poke around with the Google command line and see if I can get last modified timestamps or something similar out of it.

Lion's Address Book and Calendar

2011-12-10T10:45:29.136000+00:00

After using them for a couple of months, I'm more and more annoyed by Mac OS Lion's productivity apps, the address book and calendar. At first, I was a bit surprised by the gratuitous real world analogies (ripping of sheets in calendar, the pseudo notebook style in address book), but thought hey, whatever.

The more I use it, the more annoyed I am. For example, navigating between dates in calendar is a disaster. The controls are hidden on the top right, far away from the pill buttons that control the timespan shown, which are much more prominent but most of the time not what you actually want to use. The "rip off" animation when navigating back and forth between months not only doesn't make sense, it just takes a long time and disorients users. The main user interactions for me, "what's happening this week" and "plan next month" are unnecessarily cumbersome. Google's calendar really does a way better job there, with navigation elements in the top left, the whole month overview, and a much better attention flow from top left to bottom right.

The page flipping in Address Book has the same issue. It doesn't make sense (the same content appears on the front and back of the same physical page, the book only has three pages), it's full UI change is disorienting, and it makes it impossible to interact with groups and contacts efficiently.

I wonder what this is all about. Apple clearly should know better. And surprisingly, as Google (and IMHO much of the web) moves to cleaner, more minimalist UI design, Apple goes into the direct opposite direction, with real world analogies that don't work, and just plain bad UI design. Are they loosing their mojo in desktop UI apps?

Subsetting a language is a red herring

2011-02-23T15:14:24+00:00

Originally posted as a comment on Hacker News on Switching to Plan J, a post about moving from Scala back to Java. Earlier in the thread, the discussion had turned to a language's "complexity budget": languages have only so much room for concepts before their interactions become unpredictable and too much to keep in mind at once. The comment replied to another commenter, who wrote:

Scala can be used strictly as a "better Java" without using any of the advanced type features or idioms if that is what you want. […]

I offer the hypothesis that when using any new and "large" language on a real life project you have to work with a subset. This is not a bad thing and "subset" is not a dirty word […]

I think the "subsetting the language" approach is a red herring. Yes, you can decide you'll only use features X, Y, and Z. Many people do that in similarly complex languages like C++.

But the programmers will, one way or the other, get exposed to the features you considered esoteric, be it through libraries, or sample code, or whatever. And then they have a piece of code in the language they use for daily work that they don't understand and can't debug, and they're alienated by strange compiler errors - not good.

It's IMHO also quite hard to come up with the subset you like. I mean, programmers like to have endless discussions about completely irrelevant topics, like brace placement; wait and see the discussions about actual language features to (not) use! Most features are in themselves completely reasonable, so it's hard to make a cut here.

I also think the "mediocre programmer" argument is a red herring. Not that there aren't mediocre programmers who will have trouble understanding certain language features at all - there certainly are.

But I'd wager that even for very good programmers, having many complex language features (that interact!) in a code base will be problematic. Of course you can understand what's going on with some effort on your side, but the problem is the effort. Programming languages are IMHO a bit like user interfaces in this regard: "don't make me think". Having code in a code base that's complex and/or hard to understand is a really, really bad thing. People will shy away from reading and modifying it, and over time it'll grow into an unmaintained monstrosity.

Again, this is about the complexity budget. Original Java code was dead easy to understand; now, with generics, it's a bit harder (but overall I think generics are less bad than you hear all the time). That was, and still is, one of the really killer Java features. Not only for mediocre programmers, but for general readability, and thus usability of the language.

Blog on Google App Engine

2011-02-04T13:56:58.891000+00:00

Over the last couple of days, I took up another recreational programming project: I ported this blog to Google App Engine, using Python and django-nonrel. Partially this is of course preparation for my next job, but App Engine also looks like a good solution for a personal blog. You can run your own code, hosting is free for small projects, and you don't have to worry about most system administration minutiae.

Whenever I look at a new web framework, I estimate how much work it would be to create a weblog in it - I think weblogs are sort of the hello world of web applications. If a weblog isn't easy to implement, then something about the framework is a bit fishy.

Porting my blog from Ruby on Rails to App Engine was a bit more work than I originally expected, but I think it's not necessarily the platform's fault. After all, I tried to learn Python, App Engine, and Django all at once, with the added bonus of using a non-mainline, patched version of Django. Also, like I guess most software engineers, I consistently underestimate the work associated with more or less anything. I mean, writing out articles from a database, how hard could it be, right?

In the end, I spent 4 or 5 days on this, mostly browsing through the Django documentation to find out how things are done over here, and cursing that in Python all the standard library functions have The Wrong Name™ (when coming from Java and Ruby). Some things are still a bit hacky (I insisted on putting the required libraries in a subfolder, for which I had to patch django-nonrel...).

It's also been a bit of a pain putting the individual pieces together. The individual App Engine APIs are explained properly, but the built-in "webapp" framework is clearly lacking, so you have to pick up something like django-nonrel. Getting that to work nicely with App Engine however requires quite some reading and fiddling around. Not sure how that looks on the Java side of App Engine, but for the Python framework, the batteries are definitely not included. Python is a nice language though, and I already love using the datastore instead of a relational database. The framework also has a lot of nifty features, like the bulkloader system, or the XMPP support.

Rails vs Django

Compared to Rails, Django seems to be influenced by the Python spirit of doing things correctly. Rails is "anything that works and looks cool is ok", Django seems to be searching for the right way to do things a bit more. For example, Django has another level of indirection with separate classes handling forms, where Rails just uses the model classes directly. Both works, Rails a bit less verbose, Django is a bit stricter. It's probably easier to customize Django's form handling through the added level of abstraction, and obviously easier to get different behaviour in different places. Rails on the other hand wins with simplicity and a clear "this is the way to do things".

I'm not sure which approach I like better. Rails is clearly easier to pick up, and solving the simple, expected problems is probably easier.

Another thing one notices is that Rails has a lot more mindshare, a lot more features, documentation, plugins, etc. I'm sure there is something comparable to Ruby's XML builder, AJAX handlers, etc., but I couldn't find an easy way to do that. Django is also missing a bunch of features you take for granted in Rails, like migrations, RESTful handlers, easy support for content types and so on. Django users can certainly get all of that, but need to put in more effort.

I wonder how these things are done internally at Google. I guess I'll see in a month ;-)

Features!

While being at it, I added some nice features: I finally put a proper anti-spam solution together (the one before didn't require a CAPTCHA, but was a bit flaky - I suspect it had a bunch of false positives, and it effectively disallowed talking about Viagra ;-)). When someone leaves a comment, a chat bot now informs me about it using Jabber. And for authentication I can rely on Google Accounts through the User service.

What's still missing is support for pingbacks and trackbacks. Both specs are somewhat annoying (trackback with its pseudo RDF-embedding in articles, pingback with its XML-RPC requirement). Before doing that, I'd like to put at least some rudimentary testing system together; at the moment, this is all just a hack without any quality control whatsoever.

I guess I should get back to posting a bit more often on this blog to set off the investment. Currently I feel I wrote more code for the blogging system than text on the blog itself - somewhat like having an oldtimer car where you spend more time lying under the car than sitting inside of it.

PS: excuses to the feed subscribers, I had to change the permalinks for entries once again, so your readers probably marked all old entries as new... I promise this will be the very last time! Until I migrate everything to my new domain probst.io, that is ;-)

Google += Martin

2011-02-01T15:00:47.205000+00:00

As I've already announced quite some time ago on Twitter, I will be joining Google as a Software Engineer, starting 1st of March in Munich.

Leaving EMC/X-Hive has not been an easy decision. I joined X-Hive in April 2005, and working there has in many aspects shaped my life. I've been very lucky to end up at such a liberal, easy going company, with great colleagues. I was able to do wonderful things (working on the core of an excellent product, in a very international team, business trips to both Americas, participating in the W3C XQuery Working Group, working with a diverse set of technologies...) which I guess wouldn't have been easy at other companies. Some of the greatest things about X-Hive are the relaxed attitude, the "bottom-up" organization, the freedom for engineers to make their own calls, and the diligence and care for product quality. X-Hive feels very much like family.

I'll be missing the company lunch at the big kitchen table, the snarky discussions, the Dutch (and Brazilian! and Czech! and Russian! and French! and Belgian!) humour. I might even be missing the Dutch food, with some exceptions (I'm looking at you, frikandel).

I don't exactly know what I'll be doing at Google yet, but I'm sure it will be interesting. It might sound cliché, but I'm looking forward to new challenges, learning new technologies, and a new team.

HTML5 vs the W3C?

2010-11-04T11:37:44+00:00

I agree with Michael Kay's annoyance about the browser vendors unilateralism, expressed here.

However I think this has a bit of history. The browser vendors started their own thing after the W3C first went off with XHTML into a direction that created lots of problems for browser vendors for no return, and then completely grinded to a halt due to the W3C's process being gambled and prone to blockading.

So first we had one side ignoring the other side's concerns, and now we largely have it the other way around, with a lot more open conflicts.

I'd hope we can get over the artificial conflicts that are created here, and get both sides to acknowledge the other's legitimate concerns. It doesn't quite look like that's going to happen soon, though.

Division of Labour

2010-04-29T13:59:04+00:00

Weird: everyone who has worked in small companies/startups and large companies knows that you spend a lot more time in large companies on overhead. That can be all sorts of things, like buying a piece of software, taking time off, etc.

The strange thing is that in theory, one would think that a large organisation is structured better and caters to many more people, thus has a better division of labour, and thus the overhead for individuals goes down.

But it's exactly opposite. In a startup, you spend time on things that might not be your exact role, like administrating a web server when you are strictly speaking a developer. But that actually contributes to the company's success. And even if you count that time as overhead, I still feel I had less overhead in smaller organisations.

I wonder if this is just for software companies and bureau type workers, as opposed to assembly line workers. Still seems like a weird failure; economies of scale would suggest the opposite effect.

Reducing XPath

2010-01-05T14:43:51+00:00

Michael Kay writes on his blog: "Could XPath have been better", suggesting XPath would have been a nicer language without all the little inconsistencies. Instead, he'd rather map more or less everything to built in functions and their application to sequences, including the axes, predicates, and so on.

This very much sounds like an implementors pipe dream: remove all the annoying inconsistencies and make it easier to create fast implementations.

If you are done replacing all implicit syntax with function calls, I think you might find that you have written a LISP interpreter with built in functions (some with funny or punctuation names) for DOM navigation. Not that that would be a bad thing.

Though this makes one wonder what the actual proposed value of XPath is, once you reduced it to a LISP dialect. Probably the restricted expressiveness and from that the ability of analyzing the function applications to produce a clever execution strategy.

This always reminds me of Erik Meijer and his presentation on LINQ at VLDB 2005 (?), where he demonstrated how LINQ effectively maps certain function applications (selection, projection) to different repositories. I still like the approach: provide a somewhat unified syntax, hand over an Abstract Syntax Tree at run time to the data source/repository, and let that find a good way of executing the query. Integrating the query language into the programming language very much reduces the pain for users, and creates a uniform interface for many different data sources.

This is of course limited to the .NET platform and effectively SQL only, afaik, and I have never actually used LINQ, so I have no idea how good it works out in practice. I might imagine that tool support (profiling! indexes!) can be difficult.

Generating Eclipse build files with XQuery

2009-11-16T20:27:38+00:00

A friend of mine had a problem today. He was trying to make a huge Ant-based project usable from within Eclipse. The build file would manage dependencies through XML property files within about 30 subdirectories, each declaring which sub-projects need to be compiled before itself.

Writing all those .project and .classpath files is a very tedious task, and clicking them together in Eclipse is even more tedious. XQuery to the rescue!

I simply imported all the property files into xDB and ran this query:

import module namespace fw = 'java:java.io.FileWriter';
import module namespace xn = 'java:com.xhive.dom.interfaces.XhiveNodeIf';

for $project in /project
let $name := substring-before( substring-after(document-uri($project/root()), '/build/'), '/')
where $name != 'swami'
return
(:let $name := if ($name = 'daogen') then 'DaoGenerator' else $name:)
let $dep-internal := tokenize($project/property[@name='${module.name}.depend.internal']/@value, ',')
let $dep-common := tokenize($project/property[@name='${module.name}.depend.internal']/@value, ',')[not(ends-with(., '.jar'))]
let $deps := ('libs', distinct-values(($dep-internal, $dep-common)))
let $fw := fw:new(concat('/Users/martin/tmp/build/', $name, '/.classpath'))
let $pw := fw:new(concat('/Users/martin/tmp/build/', $name, '/.project'))
let $classpath :=
<classpath>
{ comment { $name } }
	<classpathentry kind="src" path="src"/>
	<classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER"/>
{ for $dep in $deps return
	<classpathentry combineaccessrules="false" kind="src" path="/SW {$dep}"/>
}
	<classpathentry kind="output" path="bin"/>
</classpath>
let $project :=
<projectDescription>
	<name>SW {$name}</name>
	<comment></comment>
	<projects>
	</projects>
	<buildSpec>
		<buildCommand>
			<name>org.eclipse.jdt.core.javabuilder</name>
			<arguments>
			</arguments>
		</buildCommand>
	</buildSpec>
	<natures>
		<nature>org.eclipse.jdt.core.javanature</nature>
	</natures>
</projectDescription>

return (
  fw:write($fw, xn:to-xml($classpath)),
  fw:write($pw, xn:to-xml($project)),
  fw:close($fw),
  fw:close($pw)
)

What does this do? It iterates through all project descriptions, takes the project name from the document URI, takes out the dependency information (those are the tokenize calls), and create a .project and .classpath XML snippet for each sub-project.

xDB does not include functions to write to the file system out of the box. We could create our own custom extension functions in Java and put them on the classpath, but there is a much easier way through the Java Module Import functionality.

We simply import java.io.FileWriter, create a FileWriter through the fw:new(...) calls for the right file location, serialize the XML snippets using xn:to-xml(...), and then make sure to close the file writers.

We still needed to do some fixups in Eclipse (mainly adding .jar files to the build path and fixing some wrong circular dependencies), but this certainly saved us hours. The world is a much nicer place when you have effective XML tools at your hands :-)