Archive for the ‘Broken’ Category

Outlandish newlines

Monday, August 18th, 2014

A file recently turned up from an external partner. Our software was having problems parsing it. I opened it up in TextPad (text editor for Windows) and everything looked fine. Obviously I consider our software to be perfect so I was a little perplexed as how this file could be causing problems…

It turned out there were newline issues. Opening the file in a hex editor revealed the file used nnr for newlines, i.e. 3 bytes.

I am amazed by (at least) the following facts:

  1. The fact that a file uses such newlines.
  2. The fact that any text editor can just open files using this newline scheme.

My PC’s door

Tuesday, August 5th, 2014

pcToday I wanted to use some software i have on CD on a new computers. Obviously modern computers don’t have CD drives. One “mini PC” I have at one of my offices, however, does. My plan was, today, as I’m at this office, to use the CD drive to create an ISO of the disk. (It’s my computer, doesn’t belong to an employer.)

However, for some time, the “door” of my little mini PC has been stuck. It worked fine at my old office sektor5 (= i could open it to get to the CD drive) but it never worked at the new company hiQ. I’ve tried it a few times (to get to the reset button) but could never get it open, it seems stuck (in the case of the reset button i just turned it off and on then, using the main button which is outside the door).

Today, i decided, it’s only a PC, it’s only plastic. If it really doesn’t open then, well, I might as well force it and break the door if necessary. There’s no point having a nice-looking PC door, if it doesn’t work, and you can’t actually use the equipment inside.

So, basically, today I tried really hard to persuade it to open without breaking it, but it just wouldn’t open it. So i forced it, now the plastic broke, now i can use the CD drive, so in a way everything’s good.

I now realize that the problem was I was trying to open the door the wrong way; the hinge was on the other side. If only it’d occurred to me to try and open it the other way, I would still have a beautiful door, nothing would have been broken. I don’t know why this just didn’t occur to me at all.

Now my PC looks like shit with a broken door hanging off it, and all completely needlessly.

That makes me sad.

2 of 30

Monday, January 27th, 2014

How would you diagnose the following bug?

  • A number of checkboxes representing user interests (Football? Music?); user can select/deselect their interests.
  • Software has worked well for years in production.
  • Suddenly intermittent reports start coming in that sometimes some checkboxes get unchecked by themselves.
  • You test live, you test on the test server, all is good.
    But the reports keep on trickling in, leading you to suspect that it isn’t just user foolishness (e.g. not understanding how to use a checkbox, which I wouldn’t normally put past users..)

This happened at Uboot around 2005.

I’ve forgotten how it came to pass that we fixed the bug. But the bug was this. Perhaps it sounds obvious once explained, but it was anything other than obvious at the time, especially given the fact it was completely unreproducable.

The software was coded in Perl5, and I hadn’t coded this particular screen myself. But, I had no reason to suspect that anything was wrong with the code, as, as I say, it had worked well live for years.

In Perl, “everything is a string”, apart from that, in fact, that’s not true at all. “everything appears to be a string; but might not be” describes the situation better. If you have a string “foo” then it’s stored as a string. If you have a string like “45” then it’s stored as an integer internally and converted to/from a string as needed. If you have a string like “45.6” then it’s stored as a double internally and converted to/from a string as needed. (Or it might be more complex than that, I’m not sure, perlnumber)

Supposedly this makes everything “easier” if you treat everything as a string. But I have no anger towards the junior developer who coded this screen who believed, as things appeared to be strings, things actually were stings. I mean, why wouldn’t you think that?

The checkboxes were stored in the software as a huge bit field. Perhaps this wasn’t the best representation, as that doesn’t scale (if you want to store 100 interests, you’re going to have to change the approach). But, that was the representation that had been chosen and, as I said, this had been online and worked well for years.

At some point, someone had decided to augment our 64-bit servers with some 32-bit servers. So you can imagine the rest. 2 out of 30 servers were 32-bit, 2 out of 30 clicks went to those servers, our dev server was the older 64-bit server, all our software had been developed on the old 64-bit servers. So 2 out of 30 clicks, all interests apart from 32 of then got lost.

Lesson learned (or not, as the lesson would have to be learnt by the programming language community): If things appear to be strings, make sure they actually are strings. Or, make sure it’s obvious that they’re not strings.

Had the software been written in Java, this couldn’t have happened as, independent of machine word size (32-bit or 64-bit) each Java numeric data type has a defined width, and is guaranteed to behave identically on any JVM implementation.

Constraint name visibility on MySQL and PostgreSQL

Thursday, February 7th, 2013

Can one have two tables with constraints of the same name? Is that even consistent between types of constraints? What about between database vendors?

It turns out that it’s neither consistent between types, nor is the way in which it’s inconsistent consistent between database vendors.

  MySQL 5.5 PostgreSQL 9.2
Unique constraint names Local to table Global to DB
Foreign Key constraint names Global to DB Local to table

From MySQL:

mysql> CREATE TABLE foo (x INTEGER, 
    ->   CONSTRAINT foo_unique UNIQUE(x));
Query OK, 0 rows affected (0.01 sec)

-- Second UNIQUE constraint CAN be created with same name
mysql> CREATE TABLE foo2 (x INTEGER, 
    ->   CONSTRAINT foo_unique UNIQUE(x));
Query OK, 0 rows affected (0.01 sec)

mysql> CREATE TABLE foo3 (x INTEGER, 
    ->   CONSTRAINT foo_fk FOREIGN KEY (x) REFERENCES foo(x));
Query OK, 0 rows affected (0.01 sec)

-- Second FOREIGN KEY constraint CANNOT be created with same name
mysql> CREATE TABLE foo4 (x INTEGER, 
    ->   CONSTRAINT foo_fk FOREIGN KEY (x) REFERENCES foo(x));
ERROR 1005 (HY000): Can't CREATE TABLE 'test.foo4' (errno: 121)

From PostgreSQL:

postgres=# CREATE TABLE foo (x INTEGER, 
postgres-#   CONSTRAINT foo_unique UNIQUE (x));

-- Second UNIQUE constraint CANNOT be created with same name
postgres=# CREATE TABLE foo2 (x INTEGER,
postgres-#   CONSTRAINT foo_unique UNIQUE (x));
ERROR:  relation "foo_unique" already exists

postgres=# CREATE TABLE foo3 (x INTEGER,
postgres-#   CONSTRAINT foo_fk FOREIGN KEY (x) REFERENCES foo(x));

-- Second FOREIGN KEY constraint CAN be created with same name
postgres=# CREATE TABLE foo4 (x INTEGER,
postgres-#   CONSTRAINT foo_fk FOREIGN KEY (x) REFERENCES foo(x));

Do not use automatic code reformatting

Thursday, January 24th, 2013

Beautiful source code is about a communication between the author of a program and the reader of a program. It’s a communication between two people. A computer cannot communicate as well with a human as a human can.

I’ve often heard it asserted that people cannot read other people’s code without auto-formatting. However, I think that reflects poorly on both the author (who isn’t thinking enough about the reader), and the reader (reading code often requires as much effort as writing code; often readers are lazy). If the authoring programmer cannot lay out code so that it’s understandable, they’re probably writing bad code; reformatting their code won’t suddenly make it good code.

Further to the reasons why auto-formatting do not help, here are some examples where it actually hinders. There are some examples from a customer I went back to work with in 2007, who I hadn’t visited for a few months.

In the following example, there is a clear structure to the statement. A restriction is added which is an “or”, composing of a left-hand-side and a right-hand-side. I formatted each one of those sides as on a single line indented from the surrounding “or”.

    Restrictions.ltProperty("wonUnitCount", "desiredUnitCount"),
    Restrictions.eq("reserveMet", false)));

The reformatted code does not have this structure. The “.eq” on the second line is a method on the Restrictions class of the right-hand-side. From the structure one might think that it’s one of the most structurally significant methods of the statement, but it’s not.

result.add(Restrictions.or(Restrictions.ltProperty("wonUnitCount", "desiredUnitCount"), Restrictions
    .eq("reserveMet", false)));

I needed to protect a bunch of parameters against being null. The eye scanning the following code can clearly see the “==” and the “=” in all the statements, thus one can see that all the statements are related and do similar things.

if (offeredUnitCount     == null) offeredUnitCount     = 1;
if (startCentsPerUnit    == null) startCentsPerUnit    = 1;
if (reserveCentsPerUnit  == null) reserveCentsPerUnit  = startCentsPerUnit;
if (autoBidIntervalCents == null) autoBidIntervalCents = 1;

However, the reformatted code is not only twice as long (thus meaning that some statements and methods which one would otherwise be able to see now scroll off the bottom of the monitor) but the reader is forced to examine each statement to see what they do as “==” and “=” visual pattern is no longer there.

if (offeredUnitCount == null)
    offeredUnitCount = 1;
if (startCentsPerUnit == null)
    startCentsPerUnit = 1;
if (reserveCentsPerUnit == null)
    reserveCentsPerUnit = startCentsPerUnit;
if (autoBidIntervalCents == null)
    autoBidIntervalCents = 1;

A method was going to throw a bunch of Exceptions but didn’t do so yet. This was just “in progress code”.

* @throws CentsPerUnitBeyondSystemMaximumException if either startCentsPerUnit
* or autoBidMaxCentsPerUnit are greater than the system maximum.
* TODO - throws DesiredUnitCountTooLowException(min)
* TODO - throws DesiredUnitCountGreaterThanOfferedUnitCountException
* TODO - throws StartCentsPerUnitTooLowException(min)
* TODO - throws CannotBidOnOwnItemException
* TODO - throws MustIncreaseUnitCountOrChangeMaxBid

Afterwards, one doesn’t know what’s going on at all.

* @throws CentsPerUnitBeyondSystemMaximumException if either startCentsPerUnit or autoBidMaxCentsPerUnit are greater than the
* system maximum. TODO - throws DesiredUnitCountTooLowException(min) TODO - throws
* DesiredUnitCountGreaterThanOfferedUnitCountException TODO - throws StartCentsPerUnitTooLowException(min) TODO - throws
* CannotBidOnOwnItemException TODO - throws MustIncreaseUnitCountOrChangeMaxBid

The method here takes few arguments. Listed in this way it’s clear that the function takes four arguments.

public static List<AuctionForSeller> getAuctionListForSeller(
    AuctionListTypeForSeller type,
    Member seller,
    int firstIndex,
    int itemsPerPage
) {

Afterwards there are 3 arguments on the first line and 1 argument on the last line. It’s almost as if the 4th argument is somehow of special significance; that’s the way it appears to the eye (even before one thinks about it with the conscious brain). But that’s not the case.

public static List<AuctionForSeller> getAuctionListForSeller(AuctionListTypeForSeller type, Member seller, int firstIndex,
    int itemsPerPage) {

Standards vs Fools

Thursday, December 13th, 2012

This is another rant found in my blog drafts folder from some time in 2006, when I was working at Uboot integrating a third party SMS provider.

Standards are quite useless if fools do not adhere to them; unfortunately fools are everywhere. Therefore I conclude standards are quite useless.

Specifically I implemented a data transfer protocol based on GET requests, and as specified in RFC 3986 “Uniform Resource Identifier” section 2.5.

… the data should first be encoded as octets according to the UTF-8 character encoding [STD63]; then …. should be percent-encoded. For example, … the character LATIN CAPITAL LETTER A WITH GRAVE would be represented as “%C3%80″ ….

However, they required me to send data Latin-1 encoded, rather than UTF-8 as according to the standard. I explained to them that this was wrong, that the character set is not a choice that is theirs to make. They simply replied:

We are sending with ISO-8859-1(Latin-1).

What can I do? (1) Refuse to work with this company? (But management want me to “find a solution”.) (2) Change my code to be non-standards compliant?

Obviously what I did was the second, but it felt so wrong.

How to generate “svn info −−xml” programmatically

Wednesday, December 12th, 2012

In Java, there is the great SVNKit which I have used for many a customer project successfully. You can do all Subversion operations in a few lines of Java, with a great object-oriented interface, exceptions, and so on. It’s easy to use, and it just works.

Today I had to produce certain XML files programmatically based on “svn info –xml” information. I was glad to see there was a SVNXMLInfoHandler which allows you to write the XML into any SAX ContentHandler. Again, great software design, it’s exactly what you want.

Alas, it didn’t quite work. At least in my setup of using SAXON and Xerces for XML processing, which we are already using for the project in hand. (In a different part of the software, XSLT 2.0 processing is done, and SAXON’s about the only library I know for any language that can do it.)

The problems were:

  1. That SAX events were written to the stream but the document was never started/ended,
  2. The source provided no namespace information, but SAXON requires namespace information (even if just to explicitly say that the “empty namespace” is used.

Unhelpfully, regarding point (2), I got the error

Parser configuration problem: namespace reporting is not enabled

I didn’t really know what this meant, especially in the context of SVNKit producing “svn info –xml” information. Looking at the source of SAXON, it turns out that the wording of the error makes sense if you’re using an XML parser to feed the XML tags into SAX, as opposed to e.g. SVNKit. In that case, the XML parser would have a document with namespace information, and could “report” it to SAX, or not. So, in other words, SVNKit wasn’t producing tags with namespace information, and SAX didn’t like that.

Looking at the source of SVNKit we can see lines like:

getHandler().startElement("", "", tagName, mySharedAttributes);

So startElement is being called e.g. like

startElement("", "", "info").

What SAXON needs is something like

startElement("my-namespace-url", "info", "ns:info").

So alas I had to develop the following class, which can be used as follows:

class SvnKitDomCreator extends IdentityForwardingSaxHandler {

  public SvnKitDomCreator(ContentHandler destination)
  throws SAXException {
    startPrefixMapping("svn", ns);
    startElement("", "",, new AttributesImpl());

  @Override public void startElement(
    String uri, String localName, String qName, Attributes noNsAttr
  ) throws SAXException {
    AttributesImpl nsAttr = new AttributesImpl();
    for (int i = 0; i < noNsAttr.getLength(); i++)
      nsAttr.addAttribute("", noNsAttr.getQName(i),
        "svn:" + noNsAttr.getQName(i), noNsAttr.getType(i),
    super.startElement(ns, qName, "svn:" + qName, nsAttr);

  @Override public void endElement(
    String uri, String localName, String qName
  ) throws SAXException {
    super.endElement(ns, qName, "svn:" + qName);

  public void close() throws SAXException {
    endElement("", "",;

SvnKitDomCreator domWriterWithLocalName = 
  new SvnKitDomCreator(domWriter);
  SVNRevision.HEAD, SVNRevision.HEAD, SVNDepth.EMPTY,
  new SVNXMLInfoHandler(domWriterWithLocalName));

The source for the referenced IdentityForwardingSaxHandler is LGPL: Download Databases & Life Util ZIP.

P.S. Hooray for open source! I’m sure I’d never have managed to get to the bottom of this if I hadn’t had the source for SVNKit and SAXON. I would have just been stuck with the conclusion that SVNKit produced XML, SAXON consumed XML, and “something” to do namespaces was going wrong.

SVNKit 1.7

Why am I using Hibernate?

Tuesday, December 11th, 2012

This was a blog post I found in my “drafts” folder from 2009. I am happy to report that I am no longer using Hibernate, and thus my stuff works.

Something is going wrong in my life. I read the book about Hibernate by its author, Gavin King, and I was really impressed, for the first time in my life, by an object-relational mapping system. Although I doubted it would really save development time over writing SQL manually, I thought it wouldn’t significantly increase development time either, and it had a few nice features (like being able to have a key-value Map in an object, and that Map getting persisted to its own table automatically).

But alas that was where my life started to go wrong. I’ve now completed a number of projects using Hibernate, and it’s all been hell, really.

  • I spent a whole work day trying to get the system to reconnect to the database after the connection was lost – this is 1 line in WebTek (Perl framework written by a friend, I have deployed 2 apps using it), or in my hand-written Uboot code, etc. (blog post)
  • If you use the Query API to say “give me any objects in the following set” and pass an empty set, rather than no objects getting returned, invalid SQL gets generated. I filed a bug report, but was told that generation of invalid SQL wasn’t an error. (And he, the lead developer of Hibernate, was hardly polite, see the last comment.)
  • If you persist a key-value Map to a database table (e.g. with a “key” col and a “value” col), if you overwrite a value in the Map (e.g. was “foo”->”bar” but now is “foo”->”baz”) then you’ll get a unique constraint violation as inserts of new values are done before deletes of old values. According to the bug report, this is expected behavior. My solution: drop the constraint, which I’m not too happy about. (Constraints are not only useful for data integrity, but also act as assertions which help database to choose the fastest statement execution plan)

I mean basically everything I try, no matter how simple, just doesn’t work.

Hibernate has 150 KLOC (150,000 lines of code) and you still need an extra “connection pool software” (to solve the problem of reconnecting to the database, and also for performance reasons apparently). The recommended one seems to be C3P0 which is another 50 KLOC (and what a useless name!). You can imagine how many bugs such a piece of software must have, due to its size. I can only imagine all frameworks like e.g. “Gorm uses Spring uses Hibernate uses C3P0 uses database client uses database”. Then your software has the union of the set of bugs which each layer contains – In addition to your own bugs.

Just now I’ve looked on a live server, which is under no load, and looking at the access log there were only been a few web requests at 9am, yet the web server log shows:

2009-06-02 01:53:41,889 DEBUG:
com.mchange.v2.c3p0.impl.NewPooledConnection@bb21ab closed by a client.
at com.mchange.v2.c3p0.impl.NewPooledConnection.close

Nothing like “shouting” (all caps) in an Exception message to inspire confidence. Or using the word “debug” when throwing an Exception. Or for that matter throwing the baseclass Exception and not creating a specific Exception subclass.

Incidentally this 2009-06-02 log was written to the 2006-06-01 logfile, and there were no web requests at any point on 2006-06-01. (Not sure if Jetty or log4j is at fault there.)

I need to consider alternative technologies: basically anything I deploy with this stack just isn’t going to work…

“−−” in XML comments

Wednesday, October 10th, 2012

I was slightly surprised today to check XML containing the following for validity,

<!-- "svn info --xml" and "svn log --xml" command -->

and to see the following error from the validity check:

Low-level XML well-formedness and/or validity processing output
Error: -- in comment

I mean basically this error does what it says on the tin, I mean, the string “−−” is indeed not allowed in comments. To quote Wikipedia:

The string “−−” (double-hyphen) is not allowed inside comments; this means comments cannot be nested.

I mean while technically correct, the implication seems to be that the objective was to disallow the nesting of comments so therefore “−−” had to be disallowed. Ignoring the reason why nesting of comments should be disallowed (why?) and the fact that one can disallow nested comments without disallowing “−−” (one need only disallow “<!−−”). (Did I mention Wikipedia is a site I don’t really like, and I have a tendency to believe it is written by fools?)

Also, the error that I originally encountered, I mean I just so got the feeling there was an “if” statement to detect exactly that situation. You know, if you’d removed that “if” statement, everything would have worked fine. There was no need for it. But they had to put it in there, because that’s what the XML specification demanded. I am glad I was not the programmer who had to implement that “if” statement, for I fear I would have become annoyed (i.e. more annoyed than I am now, simply encountering it.) explains “why”, at least from an SGML perspective:

To put it simply, the double dash at the start and end of the comment do not start and end the comment. Double dash indicates a change in what the comment is allowed to contain. The first −− starts the comment, and tells the browser that the comment is allowed to contain > characters without ending the comment. The second −− does not end the comment. It tells the browser that if it encounters a > character, it must then end the comment. If another −− is added, then it goes back to allowing the > characters:

<!-- this can contain > characters -- this can not,so the comment ends here>

So I tried exactly that example comment in my XML file:

Error: -- in comment


XML “nearly works”. In the sense of “doesn’t work”. See CDATA.

Mistake in the Javadoc for “Iterable”

Thursday, May 31st, 2012

Java has the keyword “for” to do “for each” semantics, which I always thought was strange. I mean

for ( ... ; ... ; ... )  { .. }  # standard C stuff
for (Object x : list) { ... }    # new "for each" syntax

I mean they are two different things, most languages use the keyword “foreach” for the second. I think “foreach” is more readable. I don’t see what Java stands to gain by “overloading” the “for” keyword.

Anyway, the second form takes either an array or an “iterable”. Check out the documentation for the “iterable” interface. It is succinct, but wrong.

I conclude from this that they weren’t quite sure themselves how to name the keyword, and that they made their decision quite late in the day.