Martin Probst's weblog

Go on App Engine and GoSublime (golang, code completion)

Monday, May 26, 2014, 19:54 — 0 comments Edit

I’m re-writing this blog using Go on App Engine, just for the fun. It took me quite a while to wrap my head around the available tool options, and get them set up correctly.

It turns out Sublime Text (my favorite text editor as of late) has a really nice plugin called GoSublime that provides various Go tools, including auto-completion through go code.

Sadly it also turns out that GoSublime is a bit fickle to get to work with App Engine’s Go libraries, possibly because of a bug. I think at this time, it’s not possible to have multiple Go workspaces with GoSublime that use different GOROOT paths (e.g. plain go + App Engine go).

But at least it’s possible to get consistent code completion and other GoSublime features working, with these global configuration settings in Preferences > Package Settings > GoSublime > Settings - User:

{
  "installsuffix": "appengine",
  "env": {
    // Your GOPATH for App Engine projects, e.g.
    "GOPATH": "/Users/martin/goae",
    // Your GOROOT, e.g.
    "GOROOT": "/usr/local/Cellar/go-app-engine-64/1.9.5/share/go-app-engine-64/goroot"
  },
  // ... any other settings you have ...
}

The source code running this puny blog is available at http://github.com/mprobst/blog by the way. It has zero features and is the first thing I ever wrote in Go, so probably not a good idea to take as an example for much.

Blog on AppEngine, Java, Jersey / JAX-RS

Wednesday, March 7, 2012, 21:02 — 0 comments Edit

Continuing the theme that I mostly use this blog to experiment with software, not so much for writing an actual blog, I moved over the software from Django/Python to Java, both running on AppEngine.

Django Out

First of all, I’ll have to admit that I’m not a Python programmer. I can program in the language, but I don’t feel at home. I’ve never really used it long enough at a time to grow familiar with the APIs and the pythonic way of doing things. I still struggle to remember the basic string manipulation APIs and such. I’ve always felt that Python is a nice, clean language (except for the constant “self.” that is), I’ve just never gotten around to use it enough.

That being said, coming from Rails, Django is a big step back, and Django nonrel, the patch to make Django run on AppEngine, is worse. Django itself lacks a lot of things taken for granted in Rails, and Django non-rel is not only cumbersome to install, it also lacks many of the compelling features of Django (e.g. the automated administration backend).

So I didn’t really see a compelling reason to stay with Django, and as I wanted to move away anyway, I could go back to Java as well. I also wanted to play around with a couple of Java technology pieces that seem nice.

Jersey In

I’ve used Jersey from time to time before. It’s a web framework that revolves mostly around HTTP. Jersey (and JAX-RS) is not trying to abstract HTTP away, it’s trying to model HTTP well and give a good framework to program on the web, which resounds with me a lot. It’s geared towards writing RESTful APIs more than towards writing actual dynamic web sites, so a bit of setup/customization is required, but nothing too bad.

Jersey of course only gives you the basic web interaction layer, i.e. the controller. I’m using Objectify to persist Plain Old Java Objects (POJOs) for the (trivial) models, Closure Templates to render views, and Gson to convert model objects into JSON like data structures for the views (Closure doesn’t support accessing POJOs directly so that it can support Java and JavaScript from the same templates).

Validation

I haven’t really found any compelling library for validation. JSR 303’s model driven approach through annotations seems nice, Spring Validation is more of a (quite trivial?) utility library, and I’m not going to touch Apache Commons Validator and its XML configuration with a 10 foot pole. I ended up just rolling something similar to Spring’s support library on my own; after all it’s only a trivial blog.

Form Support

The other thing missing from a productive toolkit is converting from submitted form data to objects (application/x-www-url-encoded, not multipart/form-data). Spring MVC offers something like that with its Command or Form objects, but that seems not very useful if not combined with all the rest of Spring. Given that we have an excellent library for JSON->POJO conversion with Gson, maybe it’d just be a naming convention for form fields that allows translating them to JSON (a bit like Objectify translates Datastore’s simple maps to objects). Again, not really needed for a blog.

Webstack

Overall I’m quite happy with it. If you don’t opt into Java’s traditional AbstractViewHandlerControllerFactory patterns (Spring, I’m looking at you) you can end up with pretty productive environments. None of the components I use are overly ceremonial, nothing needs “configuration”, and no XML involved in any form, you can even (mostly) get around web.xml and appengine-web.xml :-) I do realize that all of this is complete overkill for a tiny blog, as is writing your own blog in the first place. On the other hand, I kind of enjoy putting these pieces together and writing something on AppEngine, and I’m always interested in seeing how hard it is or how easy it should be to build certain things.

It’s always hard to get started from nothing, and I suspect this might be interesting for others. I’ll write a couple of blog posts about the different parts of the system, including putting it together using Guice.

Exporting from iPhoto to Picasa Web

Saturday, January 14, 2012, 11:16 — 1 comment Edit

I have a significant collection of photos on my home machine, all stored in iPhoto. I back them up to an external hard drive, using time machine, and copy them to a second machine using rsync every now and then. However all of that is within my apartment. That’s not only somewhat dangerous (think fire, burglary, … - at least if you’re paranoid enough), it also means I cannot access the photos from anywhere.

So my current weekend project is to upload them to Picasa Web. I actually like iPhoto, and I don’t really want to switch to Picasa to manage my photos, so I have to get the pictures out of iPhoto into Picasa Web.

Google has two tools for that, a standalone uploader, and an export plugin for iPhoto (that curiously lives in a completely different place of the UI than iPhoto’s other export plugins …). Both are available in a single download here. Neither of them does what I want though - the standalone uploader doesn’t understand iPhoto or its albums at all, and the iPhoto plugin is a manual process. You select individual albums and upload them. That’s probably useful if you want to export a single album, but not for a complete export. I need a background task that automatically does the uploading, preferably a restartable one.

Background process, automated? Command line to the rescue! :-)

It turns out there’s a very straightforward iPhoto export tool that also supports incremental exports, exportiphoto on Github. Downloading that and running exportiphoto -d “$HOME/Pictures/iPhoto Library” $HOME/tmp/iphotoexport worked like a charm (-d to suppress dates prepended to the event name). Some ~5 of my pictures were referenced in the album metadata but inexplicably missing on the hard disk, I simply deleted them.

This leaves you with a directory structure of “$event/$picture” in tmp/iphotoexport for each of your events (unrelated: why is this not a built-in feature of iPhoto?). Now for the uploading part - uploading to Picasa is surprisingly easy using the Google command line. It requires the Google client libraries, but both are available in Python’s easy_install repository, so simply running sudo easy_install googlecl gives you all you need. Creating an album from a set of pictures is as simple as google picasa create “My Album” my_album/* –date=2012-01-03. We can get the date for the album from the minimum last modified time of all exported files in the album using some ruby trickery, the dates of the pictures themselves are taken from the photo metadata. The command line uses OAuth 2 to log you in, which is a bit spooky - it pops up a browser in which you give the permissions, and then proceeds on the command line. Nice to see OAuth working so well, I really have to find some time and understand that.

So TL;DR:

curl https://raw.github.com/BMorearty/exportiphoto/master/exportiphoto.py > exportiphoto.py
sudo easy_install googlecl

python exportiphoto.py “Pictures/iPhoto Library” “tmp/iphotoexport” cd tmp/iphotoexport

for album in ; do echo Uploading $album… min_last_modified=ruby -e 'require "time"; puts Dir[ARGV[0] + "/*"].map { |x| File.mtime(x) }.min.strftime("%F")' "$album" google picasa create “$album” $album/ –date=$min_last_modified done

As for the incremental, resumable part, I still have no idea. I’ll poke around with the Google command line and see if I can get last modified timestamps or something similar out of it.

Lion's Address Book and Calendar

Saturday, December 10, 2011, 10:45 — 0 comments Edit

After using them for a couple of months, I’m more and more annoyed by Mac OS Lion’s productivity apps, the address book and calendar. At first, I was a bit surprised by the gratuitous real world analogies (ripping of sheets in calendar, the pseudo notebook style in address book), but thought hey, whatever.

The more I use it, the more annoyed I am. For example, navigating between dates in calendar is a disaster. The controls are hidden on the top right, far away from the pill buttons that control the timespan shown, which are much more prominent but most of the time not what you actually want to use. The “rip off” animation when navigating back and forth between months not only doesn’t make sense, it just takes a long time and disorients users. The main user interactions for me, “what’s happening this week” and “plan next month” are unnecessarily cumbersome. Google’s calendar really does a way better job there, with navigation elements in the top left, the whole month overview, and a much better attention flow from top left to bottom right.

The page flipping in Address Book has the same issue. It doesn’t make sense (the same content appears on the front and back of the same physical page, the book only has three pages), it’s full UI change is disorienting, and it makes it impossible to interact with groups and contacts efficiently.

I wonder what this is all about. Apple clearly should know better. And surprisingly, as Google (and IMHO much of the web) moves to cleaner, more minimalist UI design, Apple goes into the direct opposite direction, with real world analogies that don’t work, and just plain bad UI design. Are they loosing their mojo in desktop UI apps?

Blog on Google App Engine

Friday, February 4, 2011, 13:56 — 1 comment Edit

Over the last couple of days, I took up another recreational programming project: I ported this blog to Google App Engine, using Python and django-nonrel. Partially this is of course preparation for my next job, but App Engine also looks like a good solution for a personal blog. You can run your own code, hosting is free for small projects, and you don’t have to worry about most system administration minutiae.

Whenever I look at a new web framework, I estimate how much work it would be to create a weblog in it - I think weblogs are sort of the hello world of web applications. If a weblog isn’t easy to implement, then something about the framework is a bit fishy.

Porting my blog from Ruby on Rails to App Engine was a bit more work than I originally expected, but I think it’s not necessarily the platform’s fault. After all, I tried to learn Python, App Engine, and Django all at once, with the added bonus of using a non-mainline, patched version of Django. Also, like I guess most software engineers, I consistently underestimate the work associated with more or less anything. I mean, writing out articles from a database, how hard could it be, right?

In the end, I spent 4 or 5 days on this, mostly browsing through the Django documentation to find out how things are done over here, and cursing that in Python all the standard library functions have The Wrong Name™ (when coming from Java and Ruby). Some things are still a bit hacky (I insisted on putting the required libraries in a subfolder, for which I had to patch django-nonrel…).

It’s also been a bit of a pain putting the individual pieces together. The individual App Engine APIs are explained properly, but the built-in “webapp” framework is clearly lacking, so you have to pick up something like django-nonrel. Getting that to work nicely with App Engine however requires quite some reading and fiddling around. Not sure how that looks on the Java side of App Engine, but for the Python framework, the batteries are definitely not included. Python is a nice language though, and I already love using the datastore instead of a relational database. The framework also has a lot of nifty features, like the bulkloader system, or the XMPP support.

Rails vs Django

Compared to Rails, Django seems to be influenced by the Python spirit of doing things correctly. Rails is “anything that works and looks cool is ok”, Django seems to be searching for the right way to do things a bit more. For example, Django has another level of indirection with separate classes handling forms, where Rails just uses the model classes directly. Both works, Rails a bit less verbose, Django is a bit stricter. It’s probably easier to customize Django’s form handling through the added level of abstraction, and obviously easier to get different behaviour in different places. Rails on the other hand wins with simplicity and a clear “this is the way to do things”.

I’m not sure which approach I like better. Rails is clearly easier to pick up, and solving the simple, expected problems is probably easier.

Another thing one notices is that Rails has a lot more mindshare, a lot more features, documentation, plugins, etc. I’m sure there is something comparable to Ruby’s XML builder, AJAX handlers, etc., but I couldn’t find an easy way to do that. Django is also missing a bunch of features you take for granted in Rails, like migrations, RESTful handlers, easy support for content types and so on. Django users can certainly get all of that, but need to put in more effort.

I wonder how these things are done internally at Google. I guess I’ll see in a month ;-)

Features!

While being at it, I added some nice features: I finally put a proper anti-spam solution together (the one before didn’t require a CAPTCHA, but was a bit flaky - I suspect it had a bunch of false positives, and it effectively disallowed talking about Viagra ;-)). When someone leaves a comment, a chat bot now informs me about it using Jabber. And for authentication I can rely on Google Accounts through the User service.

What’s still missing is support for pingbacks and trackbacks. Both specs are somewhat annoying (trackback with its pseudo RDF-embedding in articles, pingback with its XML-RPC requirement). Before doing that, I’d like to put at least some rudimentary testing system together; at the moment, this is all just a hack without any quality control whatsoever.

I guess I should get back to posting a bit more often on this blog to set off the investment. Currently I feel I wrote more code for the blogging system than text on the blog itself - somewhat like having an oldtimer car where you spend more time lying under the car than sitting inside of it.

PS: excuses to the feed subscribers, I had to change the permalinks for entries once again, so your readers probably marked all old entries as new… I promise this will be the very last time! Until I migrate everything to my new domain probst.io, that is ;-)

Google += Martin

Tuesday, February 1, 2011, 15:00 — 5 comments Edit

As I’ve already announced quite some time ago on Twitter, I will be joining Google as a Software Engineer, starting 1st of March in Munich.

Leaving EMC/X-Hive has not been an easy decision. I joined X-Hive in April 2005, and working there has in many aspects shaped my life. I’ve been very lucky to end up at such a liberal, easy going company, with great colleagues. I was able to do wonderful things (working on the core of an excellent product, in a very international team, business trips to both Americas, participating in the W3C XQuery Working Group, working with a diverse set of technologies…) which I guess wouldn’t have been easy at other companies. Some of the greatest things about X-Hive are the relaxed attitude, the “bottom-up” organization, the freedom for engineers to make their own calls, and the diligence and care for product quality. X-Hive feels very much like family.

I’ll be missing the company lunch at the big kitchen table, the snarky discussions, the Dutch (and Brazilian! and Czech! and Russian! and French! and Belgian!) humour. I might even be missing the Dutch food, with some exceptions (I’m looking at you, frikandel).

I don’t exactly know what I’ll be doing at Google yet, but I’m sure it will be interesting. It might sound cliché, but I’m looking forward to new challenges, learning new technologies, and a new team.

HTML5 vs the W3C?

Thursday, November 4, 2010, 11:37 — 0 comments Edit

I agree with Michael Kay’s annoyance about the browser vendors unilateralism, expressed here.

However I think this has a bit of history. The browser vendors started their own thing after the W3C first went off with XHTML into a direction that created lots of problems for browser vendors for no return, and then completely grinded to a halt due to the W3C’s process being gambled and prone to blockading.

So first we had one side ignoring the other side’s concerns, and now we largely have it the other way around, with a lot more open conflicts.

I’d hope we can get over the artificial conflicts that are created here, and get both sides to acknowledge the other’s legitimate concerns. It doesn’t quite look like that’s going to happen soon, though.

Division of Labour

Thursday, April 29, 2010, 13:59 — 1 comment Edit

Weird: everyone who has worked in small companies/startups and large companies knows that you spend a lot more time in large companies on overhead. That can be all sorts of things, like buying a piece of software, taking time off, etc.

The strange thing is that in theory, one would think that a large organisation is structured better and caters to many more people, thus has a better division of labour, and thus the overhead for individuals goes down.

But it’s exactly opposite. In a startup, you spend time on things that might not be your exact role, like administrating a web server when you are strictly speaking a developer. But that actually contributes to the company’s success. And even if you count that time as overhead, I still feel I had less overhead in smaller organisations.

I wonder if this is just for software companies and bureau type workers, as opposed to assembly line workers. Still seems like a weird failure; economies of scale would suggest the opposite effect.

Reducing XPath

Tuesday, January 5, 2010, 14:43 — 1 comment Edit

Michael Kay writes on his blog: “Could XPath have been better”, suggesting XPath would have been a nicer language without all the little inconsistencies. Instead, he’d rather map more or less everything to built in functions and their application to sequences, including the axes, predicates, and so on.

This very much sounds like an implementors pipe dream: remove all the annoying inconsistencies and make it easier to create fast implementations.

If you are done replacing all implicit syntax with function calls, I think you might find that you have written a LISP interpreter with built in functions (some with funny or punctuation names) for DOM navigation. Not that that would be a bad thing.

Though this makes one wonder what the actual proposed value of XPath is, once you reduced it to a LISP dialect. Probably the restricted expressiveness and from that the ability of analyzing the function applications to produce a clever execution strategy.

This always reminds me of Erik Meijer and his presentation on LINQ at VLDB 2005 (?), where he demonstrated how LINQ effectively maps certain function applications (selection, projection) to different repositories. I still like the approach: provide a somewhat unified syntax, hand over an Abstract Syntax Tree at run time to the data source/repository, and let that find a good way of executing the query. Integrating the query language into the programming language very much reduces the pain for users, and creates a uniform interface for many different data sources.

This is of course limited to the .NET platform and effectively SQL only, afaik, and I have never actually used LINQ, so I have no idea how good it works out in practice. I might imagine that tool support (profiling! indexes!) can be difficult.

Generating Eclipse build files with XQuery

Monday, November 16, 2009, 20:27 — 0 comments Edit

A friend of mine had a problem today. He was trying to make a huge Ant-based project usable from within Eclipse. The build file would manage dependencies through XML property files within about 30 subdirectories, each declaring which sub-projects need to be compiled before itself.

Writing all those .project and .classpath files is a very tedious task, and clicking them together in Eclipse is even more tedious. XQuery to the rescue!

I simply imported all the property files into xDB and ran this query:

import module namespace fw = 'java:java.io.FileWriter';
import module namespace xn = 'java:com.xhive.dom.interfaces.XhiveNodeIf';

for $project in /project
let $name := substring-before( substring-after(document-uri($project/root()), '/build/'), '/')
where $name != 'swami'
return
(:let $name := if ($name = 'daogen') then 'DaoGenerator' else $name:)
let $dep-internal := tokenize($project/property[@name='${module.name}.depend.internal']/@value, ',')
let $dep-common := tokenize($project/property[@name='${module.name}.depend.internal']/@value, ',')[not(ends-with(., '.jar'))]
let $deps := ('libs', distinct-values(($dep-internal, $dep-common)))
let $fw := fw:new(concat('/Users/martin/tmp/build/', $name, '/.classpath'))
let $pw := fw:new(concat('/Users/martin/tmp/build/', $name, '/.project'))
let $classpath :=
<classpath>
{ comment { $name } }
    <classpathentry kind="src" path="src"/>
    <classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER"/>
{ for $dep in $deps return 
    <classpathentry combineaccessrules="false" kind="src" path="/SW {$dep}"/>
}
    <classpathentry kind="output" path="bin"/>
</classpath>
let $project := 
<projectDescription>
    <name>SW {$name}</name>
    <comment></comment>
    <projects>
    </projects>
    <buildSpec>
        <buildCommand>
            <name>org.eclipse.jdt.core.javabuilder</name>
            <arguments>
            </arguments>
        </buildCommand>
    </buildSpec>
    <natures>
        <nature>org.eclipse.jdt.core.javanature</nature>
    </natures>
</projectDescription>

return (
  fw:write($fw, xn:to-xml($classpath)), 
  fw:write($pw, xn:to-xml($project)),
  fw:close($fw),
  fw:close($pw)
)

What does this do? It iterates through all project descriptions, takes the project name from the document URI, takes out the dependency information (those are the tokenize calls), and create a .project and .classpath XML snippet for each sub-project.

xDB does not include functions to write to the file system out of the box. We could create our own custom extension functions in Java and put them on the classpath, but there is a much easier way through the Java Module Import functionality.

We simply import java.io.FileWriter, create a FileWriter through the fw:new(…) calls for the right file location, serialize the XML snippets using xn:to-xml(…), and then make sure to close the file writers.

We still needed to do some fixups in Eclipse (mainly adding .jar files to the build path and fixing some wrong circular dependencies), but this certainly saved us hours. The world is a much nicer place when you have effective XML tools at your hands :-)


New Post