Martin Probst's weblog

Threads vs. Processes in Java & Co.

Friday, April 11, 2008, 08:13 — 5 comments Edit

Erik Engbrecht posted a very interesting article on Multiprocess versus Multithreaded System design. It includes a lot of insight, but just one quote:

One one hand on most operating systems threads involve less overhead than processes, so it is more efficient to use multiple threads than multiple processes. On the other hand multiple processes ultimately will give you better reliability because they can be spawned and killed independently from one another.


My rule of thumb is to look at the amount of shared data or messaging required between concurrent execution paths and balance against how long the “process” (not OS process) is expected to live.

I think one important point is missing here. It’s the question of what your underlying framework is, and how you access it.

Early on, C-written CGI applications were an acceptable solution as the applications mostly used the underlying UNIX system as their framework and API. Spawning a simple process that more or less only accesses the basic C library in a short lived transaction is fast and safe.

But eventually applications get more and more complex, and you will have more and more shared functionality that is accessed in a library fashion. You will want to write this in the same language and environment as your application, as this makes debugging, lifecycle management etc. much easier.

As these libraries start to get more complex, their initialization will start to take significant time and resources. And at some point, you will no longer be able to load them every time a request hits your CGI program.

So now you can either put those into some kind of daemon that is accessed by IPC, or you will have to create a long running server process that handles multiple requests, either sequentially (FCGI) or even truly multithreaded like todays Java servers.

It’s also not just the cost of initializing these libraries, it’s also their footprint. As I’ve written before, running a herd of mongrels can get really expensive, memory wise.

MVM a solution?

That being said, losing the simplicity and reliability of processes is indeed highly problematic. I’m not sure about the state of Sun’s Multi VM project, which was designed to solve some of these issues.

Maybe this thing could be overcome if our programming languages allowed us to define application parts that are effectively stateless, ambient libraries, executing more or less purely functional (well, except for logging and such…) once they are initialized.

If you could define such constraints on program parts, you could have your cake and eat it, too. Multiple instances of your application could share a possibly very large part of their code base, but you could still kill single instances at will, as they don’t share state with other instances.

The problem is with the “more or less” part of “purely functional”. I’m not sure if it’s actually possible to find that niche where you can still have some state. And of course you would have to be ably to statically prove those constraints, or else everything is moot.

Also highly interesting in this context is Microsofts Singularity, which achieves a runtime that doesn’t need processes but still has all the nice properties of processes (isolation, ‘kill -9’), due to some carefully chosen, statically verifiable constraints.

Memory usage, Java vs. Rails

Wednesday, April 9, 2008, 09:52 — 3 comments Edit

This blog is running on a virtual server at Hosteurope. The server is running a plain Apache 2, my Subversion repository, this blog, and since yesterday a Tomcat instance for the XQuery pretty printer.

The interesting part in getting Tomcat to run was (apart from some mod_jk problems) that I exhausted the 128 MB of memory my virtual server has. Which surprised me a bit - the JVM instance should not consume more than 64 MB, so who was taking all that memory?

This is the current list, according to RSS, which is not precise, but still:

The interesting point is that due to Ruby not supporting kernel threads and Rails generally being single threaded (boo!), I end up with a lot more memory consumed for e.g. a simple blog running 3 mongrel instances plus ferret. RSS includes shared libraries and non-writable memory, but pmap -d reports something like 41592K non shared, private writable memory for the mongrel process.

I had to reduce the number of mongrels to one, so that I can run a Java VM on this host. Thus, my server can only process one concurrent request from one user at this time. When the Ruby process is waiting for filesystem or database operations, nothing else gets processed, even though CPU time would be available.

This is indeed pretty annoying. Some 50 MB don’t seem much memory on a regular desktop machine these days, but for (shared) hosting, it still is a large issue. In the case of my puny blog this isn’t really a problem, plus I’ve set up decent caching so most requests will be handled directly by Apache, but still…

Maybe switching to Merb and Ruby 1.9 (or JRuby) would be useful, though I have no idea if that really runs concurrently in one process, or if it’s just generally possible.

XQuery Pretty Printer

Tuesday, April 8, 2008, 13:14 — 0 comments Edit

As readers of this blog might know, I’ve spent quite some time implementing and optimizing XQuery engines in the last three years.

Something that has always bugged me were my attempts at XQuery parsers. XQuery is a keyword free language that has a truly complex lexical structure, so writing a parser is much more complicated for XQuery than one for, say, Java.

A nice example is this valid (!) query from Building a Tokenizer for XQuery:

declare namespace namespace = “";
declare union <union>for gibberish {
   for $for in for return <for>div div</for>
if(if) then then else else- +-++-- instance
of element() * * **—++div- div -div

The * and + signs change meaning in different contexts, and so do the keywords like ‘if’ themselves. Within XML literals, the lexical structure changes completely. Function calls, type checks and the like are hard to disambiguate. And whitespace is sometimes significant, sometimes not. Quite a mess.

I wrote the first XQuery parser in my Bachelor’s project in fall 2004/ spring 2005. As it was my first parser, and XQuery isn’t really easy, and it was in C++, it ended up being quite a mess. The lexer was stateful, using a stack to switch contexts. These contexts would however not be controlled by the lexer, but from the parser. As the parser was implemented with ANTLR (2.7), syntactic predicates, lookahead etc. would bring the lexer out of context. The whole thing was pretty slow and generally fragile.

The next one was an existing parser at X-Hive which I extended and maintained. In this case, state was managed by the handwritten lexer, and parsing was again done using ANTLR. This one was really fast and worked pretty well, but maintaining the handwritten lexer with it’s 20+ different states, the stack etc. was quite annoying. Adding support for XQuery Update to it was painful. I think code like a lexer should always be written in a domain specific language, otherwise you keep writing lots and lots of repeating Java code that essentially means nothing.

So in a case of extreme procrastination, I started to write another XQuery parser, using ANTLR 3. ANTLR 3 brings a new analysis algorithm, called LL(*). That allows to make parsing decisions with arbitrary lookahead, depending on what context is needed. This is ideal for a language, where ‘if’ vs ‘if (’ makes a huge difference.

I again chose to manage state from the parser. But due to the LL(*) parsing this time I only needed two different lexical contexts: XML literals and plain XQuery. This simplified the whole thing a lot - I more or less wrote the parser in 3 or 4 days.

I’m pretty pleased with the end result. It’s fast enough (about 0.2 ms for simple queries, might test some more soon) and it parses all the ~20000 test cases from the XQTS suite successfully (except for 37 false positives where my parser allows constructs like ‘foo(: comment :): bar’ as QNames, but that will be a simple fix).

Currently the parser is little more than a language acceptor. But I’ve always wanted to have a decent XQuery formatter/pretty printer. So I managed to get it to produce parse trees (like ASTs, but with a node for every rule), and use those for query formatting.

I basically construct the parse tree, use it as the input for an XSLT transformation that inserts <indent/> tags and <br/>s for line handling and puts some HTML spans with classes around the keywords. The result is piped through a SAX filter that converts the indentation tags into real indentation.

You can find the result here: XQuery Pretty Printer / Formatter.

Any bug reports or suggestions are very welcome. I will probably make the parser open source, once it’s finished (it still has an issue with the full character range in QNames). I’ll also probably make the formatter a web service accessible using plain HTTP, so I can use it to format my blog posts.


Installing/importing a root certificate in Java

Friday, March 14, 2008, 13:30 — 0 comments Edit

I just completed the nice exercise of installing a root certificate into a Java installation on a Debian server. Somebody I need to communicate with uses a certificate signed by a root CA ( that is apparently not included in the JVM 1.5.0_14 that ships with my Debian.

The only documentation I could find after lots of Googling is extremely verbose and obfuscated, so here comes the short hand version.

Basically you download their certificate, in this case here. Then you convert the certificate into the PEM format (whatever that is…), and install it into the system wide cacerts keystore. You’ll be asked for the password to that store, it is changeit (and please don’t change it!).

Oh and of course in between you perform extensive measures to make sure the certificate is valid, such as, well, … you can compare the fingerprint to the one on the webpage where you downloaded it, tremendously secure ;-)

# convert:
openssl x509 -in a-cert-globaltrust.crt -out a-cert-globaltrust.pem -outform PEM
# install, type "changeit" for the password
keytool -import -file a-cert-globaltrust.pem -keystore /etc/java-1.5.0-sun/security/cacerts

You might also want to install it for openssl clients, as documented here.

API usability

Friday, March 7, 2008, 19:51 — 0 comments Edit

Alex on API usability

Sun is definitely bad on the developer usability part. Really.

Eh Alex, when was the last time you had to use C++/MFC or similar?

Or use some random Ruby functionality from the standard library, trying to find the documentation for it?

Or actually do something in JavaScript, with the awkward set of interpreters, and no proper documentation for any libraries?

There are certainly issues with Java (and in particular a lot of the Java libraries), but it’s certainly not that bad. Though I sympathize with your frustration, I know that very well ;-)

Shell script to restore .svn directories in Mac OS X Bundles

Monday, February 11, 2008, 10:00 — 1 comment Edit

Checking in Mac OS X bundles, such as the files created by or Keynote, into Subversion breaks, as Pages removes the metadata directories (‘.svn’) Subversion uses to keep track of file status.

Pretty annoying, and it doesn’t seem to get fixed very soon. Following various links I found a script by Daniel Sadilek that recovers the missing .svn directories. While the script from the blog article itself doesn’t work with directories that contain whitespace, the one posted in the comments by “Percy” does.

Basically you edit your checked-in bundle using Pages/Keynote/…, then run ‘svnrecover <bundlepath>’, and afterwards you can check your stuff in.

I modified it a bit to better handle folder switches by using pushd/popd, so I’m reproducing it here:


# must be dir!
if [ ! -d "$1" ]; then
  echo "Path is not a directory."
  echo "Usage: $0 [bundle directory]"
  exit 1

# get info
dir=`dirname "$1"`
base=`basename "$1"`

# go to dir
pushd "$dir">/dev/null

# prefix

# does it exist?
while [ -x "$prefix.$base" ]; do
  prefixi=$(( prefixi + 1 ))

# move to temp
echo "* moving to temp location..."
mv "$base" "$prefix.$base"

# update
echo "* svn update..."
svn update -q "$base"

# check to make sure it's a dir!
if [ ! -d "$base" ]; then
  echo "Updated path is not a folder! Restoring original!"
  mv "$prefix.$base" "$base"
  exit 2

# go to dir
echo -n "* moving .svn directories"
pushd "$base" >/dev/null

# find .svn dirs
for foo in `find . -type d -name .svn`; do
  if [ -d ../"$prefix.$base"/"`dirname "$foo"`" ]; then
    # only move it if the folder exists in the modified version
    mv "$foo" ../"$prefix.$base"/"`dirname "$foo"`"
    echo -n "."

popd >/dev/null

# replace
echo "* replacing..."
rm -Rf "$base"
mv "$prefix.$base" "$base"
popd >/dev/null
echo "* done."

Silent errors from JSTL

Thursday, January 31, 2008, 16:15 — 0 comments Edit

The JavaServer Pages Standard Tag Library appears to have a habit of silently failing to work.

Boring technical details

Today I ran into a very nice issue: I tested and coded my webapp against Tomcat 6.0 with a version 2.5 web.xml. Deploying this on Tomcat 5.5 on the target system seemed to work: no errors, database was filling from webservice, Spring worked, I could access webpages, etc. But due to the niceness of JSTL a c:choose statement containing an EL expression was always false, so the page always came up with no content. Very nice to debug, as no error or anything shows up and the developer of course assumes some failure to load data.

End of the story: JSTL silently fails if the web.xml version tag and Tomcat combination doesn’t fit JSTLs expectations. I needed to change my web.xml to 2.4.

Slightly less boring rant

Another example of really great software: include a versioning scheme, but don’t give any errors if versions don’t match, but rather silently fail.

While Java has it’s deficiencies, I’m beginning to think that much could be fixed by replacing the totally broken toolchain (yes, Maven, Tomcat & Co.; I’m looking at you!) with proper software.

My current hit list would be:

And there’s still a lot missing on that list.

The JVM is an awesome platform, and Java as a language is totally tolerable, and most of the platforms/libraries even sound rather sane in theory, but are a real pain to work with in practice. It’s interesting to see that a small group of Ruby guys was able to craft a tool chain that’s much easier to use in so little time, with no big vendor support. I think Java web applications are looking for some sort of a Rails like renovation, or they will fade into irrelevance.

SOAP with Axis, 4D and something called multiRefs

Friday, January 25, 2008, 10:47 — 1 comment Edit

SOAP really totally sucks. I just wasted several hours trying to access a certain Webservice using Axis. The server is apparently implemented using 4D.

It turns out that by default Axis (at least 1.x) uses a feature called “sendMultiRefs”. This turns regular, unsuspecting messages into this:

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv=""
    <ns1:SomeAction xmlns:ns1="some-namespace"
      <SID xsi:type="xsd:string">some param1</SID>
      <startID href="#id0"/>
    <multiRef xmlns:soapenc=""
      id="id0" soapenc:root="0" soapenv:encodingStyle=""
      xsi:type="xsd:int">some param2</multiRef>

The fun thing is that this happens somewhere really deep down in the Axis code - if you debug the Axis message send, all you see of the SOAP message is a regular message (without that multiRef abomination), even when you get down really close to the HTTP socket send. I found this in the end using Wireshark/Ethereal to inspect the actual packages.

Now what happens is that 4D silently drops/ignores the second parameter and apparently sets the value of that input parameter to 0. It doesn’t give any error whatsoever, thanks. Additionally, due to the close coupling to the programming language, the implementation cannot make out if the parameter is not present or set to zero, because it comes out as an int value.

There are a lot of questions here. Why does Axis use a bizarre encoding scheme, that is known to cause problems with a variety of platforms, where it’s totally superfluous? It even makes the messages larger.

And why is this multiRef thing there, anyways? The only thing I can find about it on is a mention that it’s included in SOAP 1.2, but not in SOAP 1.1. I can’t find any mention of it in the spec itself, though. It only appears to be useful for serializing Object graphs that are self-referencing.

Why a messaging protocol needs something like that, which is totally linked to the complex semantics different meanings of identity in different programming languages etc., is beyond me. And actually most people don’t seem to implement it anyways…

It’s quite frustrating to use all this complex, bizarre stuff when all you’d really need is a webservice that takes 1 parameter and returns some XML-encoded list, i.e. http://foo/bar?startID=x.

Tomcat startup timeouts in Eclipse

Monday, January 21, 2008, 18:44 — 1 comment Edit

Today starting Tomcat from Eclipse using WTP suddenly stopped working for me. Tomcat would start up all fine and nice, but then after some time Eclipse would complain that Tomcat didn’t come up properly and kill it again. The strange thing: everything was actually working, and I could access the webapp with a browser.

After some intense digging I found out that Eclipse tries to connect to Tomcat via HTTP to find out if it works. I then found out using netstat -Af inet that Eclipse was indeed connecting to localhost:8080, and Tomcat was accepting the connection, but then apparently nothing happened. Eclipse kept spamming Tomcat with new connections, and after some time it hit the timeout:

Active Internet connections
Socket   Proto Recv-Q Send-Q  Local Address      Foreign Address    (state)
       0 tcp4       0      0    ESTABLISHED
       0 tcp4       0      0     ESTABLISHED
       0 tcp4       0      0    ESTABLISHED
       0 tcp4       0      0     ESTABLISHED
       0 tcp4       0      0    ESTABLISHED
       0 tcp4       0      0     ESTABLISHED
       0 tcp4       0      0    ESTABLISHED
       0 tcp4       0      0     ESTABLISHED
       0 tcp4       0      0    ESTABLISHED
       0 tcp4       0      0     ESTABLISHED
       0 tcp4       0      0    ESTABLISHED
       0 tcp4       0      0     ESTABLISHED
       0 tcp4       0      0    ESTABLISHED
       0 tcp4       0      0     ESTABLISHED
       … and so on …

No idea why this was exactly happening, but I fixed it by deleting all server related configuration from my workspace. That is, delete all Tomcat related configuration, exit Eclipse, and then within your workspace:

rm -r .metadata/.plugins/server

Of course this is just cargo cult programming, but it fixed my problem.

Why is String.format() static?

Wednesday, January 16, 2008, 11:01 — 0 comments Edit

I really wonder why they made java.lang.String.format(String format, Object… args) a static method. Compare:

String message = String.format(“Failed to quizzle the fibble: %s”, fibble);
String message = “Failed to quizzle the fibble: %s”.format(fibble);

The latter isn’t only more object-oriented (static methods are in effect global procedures), it’s also less typing.

New Post