Firstly, I love assigning a list to a list of lvalues i.e. variables; this is possible in both PHP and Perl which I use regularly; and no doubt many other languages. (But not Java: why not!?)

    ($a, $b) =      ($b, $a);  // Perl
list($a, $b) = array($b, $a);  // PHP

PHP, as always, wins in inelegance, having the left side syntactically different to the right side. While it’s obviously the case that a list of values and a list of lvalues are technically different, I don’t think this difference should be expressed in the syntax.

I mean, in most languages you write e.g. $foo=4 and $bar=$foo; in both those cases you write $foo but yet they do something different (lvalue and rvalue); given that you write them the same there i think the same should apply to lists.

But I digress – What I want to mention is using “underscore” to mean “any variable”. I first saw this in Prolog.

For example, imagine you have to implement an interface (e.g. in Java), it requires you to write a function taking two parameters, but one of the parameters you don’t care about. Wouldn’t it be nice to write

interface ExistingApi {
   public void createObject(String name, Object otherInformation);
}

class MyInstanceOfTheApi {
   public void createObject(String name, _) {
      ...
   }
}

i.e. this shows clearly you do not care about the second parameter.

In current Java (and all languages I program including Perl, PHP) you have to give all variables a name even if you don’t use them, either in function definitions or in “assign to a list” scenarios mentioned above. It is then left as an exercise to the reader to determine if these variables are used or not, and indeed an exercise to the writer to name the variable they are never going to use.

I mean yes, technically you can actually just call variables “_” (or “$_” (except in Perl where “$_” already means something)) but that would then be a coding convention as opposed to a language feature, and who knows if the coding convention is actually used correctly by a programmer. (If “_” is a variable there’s nothing stopping someone from using its value.)

And then you have the problem if you have two variables you don’t care about, you can’t call them both “_”.

By the way, Programming in Prolog is an excellent book.

In Java 1.4 the authors of Java saw it fit to introduce a method to split strings,

String csvData = "field1,field2,field3";
String[] fields = csvData.split(",");

However they did not introduce a method to “join” strings! Even in Java 7 there is no way to do this, e.g. via a static String.join method (2002-now).

OK I realize this is not “rocket science”, and I appreciate it exists in various versions in various third-party libraries, but still, it’s something every program needs to do at some point, it’s annoying to have to re-define it or think about it for each application.

For example, in one project I was working on in the last 6 months, such a function was created, and then had a bug! (OK but to be fair, that was not the only bug in the application!)

Come on, I mean this is a totally trivial function, totally necessary, available in all scripting lanuags why is it still not in Java?!

(P.S. Want to see an “enterprise java” solution to this problem? Check out the number of methods on this class)

String[] myArray = new String[] { "foo", "bar" };
List myList = Arrays.asList(myArray);

In Java 1.5 this was retrofitted for varargs; you could simply pass elements to the function

List<String> myList = Arrays.asList("foo", "bar");

I never really understood how that worked in a backwards-compatible way; I mean either the function takes an array of stuff, or it takes individual elements, surely?

It turns out, that with the varargs syntax, the caller is not forced to pass individual elements, the caller can instead pass an array of elements.

List<String> myList = Arrays.asList("foo", "bar");
List<String> myList = Arrays.asList(new String[] { "foo", "bar" });

The above two calls are identical, both return a List<String>.

But surely this is really dangerous? I mean Arrays.asList does not make any assumptions about what types of arguments it accepts; the list can be composed of any object.

How can it be certain that you want to have an List of Strings, and not a List containing a single element which is a String array? (An array is an object.)

To demonstrate this inconsistency:

String[] arr = new String[] { "foo", "bar" };
Arrays.asList(arr);            // returns List<String>
Arrays.asList(arr, arr);       // returns List<String[]>
Arrays.asList(arr, arr, arr);  // returns List<String[]>

Be aware of this limit. I was unaware of it when I started this project. But it makes Java completely unsuitable for a whole class of problem.

My customer supplies me with a config file from time to time, this specifies a certain algorithm. When the user enters data, this algorithm must be applied. The algorithm is complex, so performance is an issue.

The solution I chose was to generate code to execute the algorithm, based on the information in the config file. This is a valid computer-science approach, and is used for similar problems. For example, language parsers are often expressed as a grammar, and code to parse documents in the grammar are generated. JSPs are turned into Java classes which are then compiled and executed. WebTek pre-compiles HTML templates containing macros into code which produces the resulting HTML when executed.

However, don’t try this in Java, unless you are only working with small problems. A single method in Java can only be 64KB in size, once compiled.
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4262078

This means, JSPs can only be of a certain length, parsers can only parse languages of a certain complexity, if WebTek were written in Java then templates could only be of a certain length and complexity, and so on. Do you want to place such restrictions on the software you produce?

My specific problem involves simulating one million variations to a particular solution. How can I fit that into 64K?

• That is 0.06 bytes per solution variation; yet the simulation of a single variation involves many lines of code (i.e. in total compiling to more than 0.06 bytes!).
• I could put each variation into its own method, and have a big method which calls them all—but a method call takes more than 0.06 bytes!
• I could have a hierarchy of methods: one main method which calls, say, 100 sub-methods, each of those call 100 sub-sub-methods, and finally those methods call the methods for the individual variations.

It’s not even possible to know how many bytes a method will generate to! So, as the complexity of the simulation of a variation is expressed in the config file, I would have to essentially have to do a “trial and error” approach: generate a method, compile it, if I get the error concerning the 64KB limit, split the problem up into slightly smaller methods, try the compilation again, repeat, etc. (And the Java compiler is not even very fast.)

This is all so wrong! This is complexity, which isn’t solving the customer’s problem. This complexity costs me time (and thus my customer money), complexity leads to bugs and difficulty of maintenance, etc.

So I have changed the language. Rather than generate Java, I generate C and compile it using the GNU gcc compiler. From the GNU coding standards:

Avoid arbitrary limits on the length or number of any data structure, including file names, lines, files, and symbols, by allocating all data structures dynamically.

This is a good standard! I like it. All programs should be written with this in mind. Your program may well be online in 10 or 20 years, and the hardware may well have changed: a 64KB limit may seem reasonable one year but is a real limitation 10 or 20 years later in software which would otherwise still be useful.

So, if you are solving this type of problem, don’t use Java.

P.S. On a separate project I used a similar approach using Perl, and that worked out fine too.

1. A server may serve multiple applications; each application may use different libraries or even different versions of the same library, “side by side”.
2. XML parsers, transformers (XSLT), etc., have a standard interface, and there may be different implementations of this interface from different vendors, open-source projects, etc.
3. Which XML parser, transformed etc. is actually used depends on a global system variable.

And it’s point 3 that’s the problem really. Points 1 and 2 are debatable: they certainly bring advantages, but they certainly bring complexity too.

I just had the problem that one of my web applications stopped working, but only intermittently. Restarting the server led to everything being OK, but later things would not be OK. I do hate environments where everything appears to work, yet in fact doesn’t. I mean how do you know when you’re “done” in such an environment? (Or how do you even know you are in such an environment?)

The bug was caused by:

1. Application one used the default XML parser, and didn’t have any extra JARs (libraries) for reading XML
2. Application two required a special XML parser, set the global variable so it would be used, and included the JARs necessary for the special XML parser

So when a request came to application 1, after a request had come to application 2, then the system would try to instantiate the special XML parser within application 1 (specified in the global variable set by application 2), but wouldn’t find it, as it wasn’t deployed in application 1 (and applications can’t use one another’s libraries, due to feature #1).

This seems obvious when one describes it, but looking at the logs, on a live server, with the system down and the clock ticking? – Far from obvious.

So now, I assert, every time you want to create an XML parser, do the following:

If you require a special XML library, use:

System.setProperty("javax.xml.parsers.DocumentBuilderFactory",
    "org.apache.xerces.jaxp.DocumentBuilderFactoryImpl");
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
...

If you require the standard XML library, use:

Properties systemProperties = System.getProperties();
systemProperties.remove("javax.xml.parsers.DocumentBuilderFactory");
System.setProperties(systemProperties);
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
...

There is also the possibility to pass a parameter to DocumentBuilderFactory to specify which XML parser technology to use. That’s a good option too, as it wouldn’t “corrupt” this global variable (“system property”). However I think one should be defensive, and always delete the global variable if one wishes to use the standard XML parser, and therefore it doesn’t matter if this global variable gets corrupted or not.

Never do the following:

DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();

This simply relies on whichever XML parser is currently set in the global variable. You have no way to guarantee that some other application running on the same server won’t set the global variable to use an XML parser you don’t have installed in your application. Even if you have control of the server and all applications, you don’t know what software you’ll be writing in the future. (In this case I installed a new application to a server which’d been running fine for 1 year, but due to setting the global variable, the old application broke..)

The same applies for all those other “factory” situations such as TransformerFactory.newInstance() etc.

This feels all quite inelegant to me, and has just cost me a lot of time, and it’s not as if I’m so new to programming Java. I am wondering if there is a better way to approach it? Or is Java just broken in this particular respect?

P.S. This is not the only thing that went wrong with the old application today. I upgraded from Java 5 to Java 6 and suddenly some XML was not compliant against a schema according to Java – I had hit this error.

The Java application webserver (Jetty) was taking a long time to restart each time I did a change, so for some reason I thought I’d experiment with some new command-line options. Probably not the right time to do that.

Normally I would type

$ sudo  /etc/init.d/jetty6 restart
Stopping Jetty: OK
Starting Jetty: OK

and everything would be good. I tried typing

$ sudo /etc/init.d/jetty6 supervise

Then some stuff happened that I didn’t really understand. Rather than try and work out what it did I tried to restart it again using the old restart mechanism

$ sudo  /etc/init.d/jetty6 restart
...
Starting Jetty: FAILED

OK I mean that was basically what was going on, it just wrote FAILED. How helpful! There was no info in the logfile. I searched Google but didn’t come up with anything.

A reboot later, and about half an hour of looking into /etc/init.d/jetty6 with vi and randomly making changes and printing more stuff out yielded the fact that the “supervise” command had evidently run Jetty “as me” and not as the “jetty” user. So when the normal “restart” command came along and tried to run the program as “jetty” then there were files it couldn’t write to.

Solution:

$ sudo chown jetty /var/log/jetty6/2009_07_12.stderrout.log

The XML

<one-to-many type=”OtherClass”/>

delivers the error

Error parsing XML: Attribute “type” must be declared for element type “one-to-many”.

This looks like a perfectly self-explanatory error, however looking at the file, the element does have a “type” attribute. What should one do?

Thinking about it, I only just introduced the “type” attribute to the <one-to-many> element in my config. What happens if I change the attribute name to “fsdjkfdk”?

<one-to-many fsdjkfdk=”OtherClass”/>

The error is now:

Error parsing XML: Attribute “fsdjkfdk” must be declared for element type “one-to-many”.

What the error means is that the attribute must not be declared, as opposed to must.

It’s amusing to read even people on the Hibernate team get confused by this error, and can’t find a solution.

(Hibernate 3.3.1 – the most current version – although I encountered this error within the first hour of ever using Hibernate in Q1/2006.)

• Background jobs (without user-interface)
• Server-side web applications (many web servers & web frameworks available)
• In the web browser (applets)
• In the web browser (translation from Java to Javascript by GWT)
• On the desktop (using platform-independent Swing, or with native Apple UI using Cocoa)
• On some mobile phones (J2ME, Google Android) – although I haven’t tested how good that really works?

Writing the previous version of a certain application website in Perl, there was no easy way to give the customer a “tool” to test out new versions of the configuration file. These files would normally be installed on the server, were multiple megabytes in size, and the Perl would parse and use them. For testing, it was not ideal to have to upload potential new files to the test server, due to their size.

The new version is in Java and also takes a configuration file, but I have written a Swing (desktop) tool which simply allows the tester to select a new potential configuration file from their local hard disk, and the desktop tool reuses all the processing logic 1:1 that the web server in production would use.

That wouldn’t have been possible with the old version of the logic written in Perl. (I know there are windowing libraries for languages like Perl but they are hardly as easy to deploy – i.e. install on a Windows workstation – as a Java application – simply double-click the .jar file once Java is installed)

I am writing the web front-end for the new version in GWT so I can reuse certain (mainly user validation) code between the web browser (giving the user instant feedback in case of errors) and the web server (necessary for security in case someone bypasses the client and sends HTTP requests directly.) And simply pass Java objects between the web server and the web client, without having to worry about how that transfer works (JSON, XML, etc.)

Other mainstream candidates for languages which run on multiple places:

• Objective-C is not too bad, running on Apple desktops, Apple iPhones, and on the web server via WebObjects (does the current version of WO still use Objective-C or is it Java-only these days?) – but not in the browser
• Javascript runs on the web browser and server, and no doubt mobile browsers (but not desktops as far as I know)
• Perhaps C#? Certainly good desktop, web server integration, no doubt IE/Windows integration via ActiveX

Starting Jetty: OK

Yet connecting to the port (telnet localhost 8080) showed that the port was not open. Looking in the logfile (/var/log/jetty6) showed the following error:

2008-11-20 12:08:47.477::WARN:
  failed SelectChannelConnector@null127.0.0.1:8080
...
Caused by: java.nio.channels.UnresolvedAddressException

That meant that the web server was unable to open the socket on port 8080.

Googling for this error showed only 1 result which lead to a 404. Javadocs and Jetty docs did not mention any problem like that.

Thankfully I had a working system to compare it with. The working system wrote in its logfile:

2008-11-03 13:34:41.283::INFO:
  Started SelectChannelConnector@0.0.0.0:8080

Notice the difference between 0.0.0.0 (works) and null127.0.0.1 (doesn’t work). It turned out that on the good system, the config file /etc/jetty6/jetty.xml had the line

<Set name="host"><SystemProperty name="jetty.host" /></Set>

and the non-working system had:

<Set name="host"><SystemProperty name="jetty.host" />127.0.0.1</Set>

I have absolutely no idea how this non-working line came into the config. I didn’t change it, and the operations department surely didn’t change it either. I have absolutely no idea.

Why is Java so difficult !?

Jetty 6.1.11, Debian Etch, Linux 2.6.24, Sun Java 1.5

• Executing a SQL statement from my program

Because, as everyone knows, I don’t suffer from NIHS, I used standard object-relational mapping software Hibernate, with a standard programming language Java, using the standard web-application server Tomcat, and now I am using the standard “connection pooling” software C3P0 (which I didn’t know I needed to execute a SQL statement, see below..)

The program is, in fact, already completed, and is nearly deployed. On the test server it works fine and even on the (future) live server it worked fine. But the customer noticed that if one installed it one day, the next day it didn’t work. I’ve had such symptoms many times before, so I know immediately what was going on:

• MySQL drops a connection after 8 hours (configurable)
• The software is used during the day, but isn’t used during the night, therefore the connection times out in the night
• Therefore in the morning, the program one installed the day before no longer works

Perhaps I exaggerated the simplicity above of what I was really trying to achieve. It should really be expressed as the following:

• Executing a SQL statement from my program, even if a long time has passed since the last one was executed

But that amounts to the same thing in my opinion! It isn’t rocket science! (But in fact is, see below..)

A obvious non-solution is to increase the “connection drop after” time on the MySQL server from 8 hours to e.g. “2 weeks” (“wait_timeout” in “mysql.cnf”). But software has got to be capable of reconnecting after a connection drops. The database server may need to be reset, it may crash, it may suffer hardware failure, etc. If, every time one restarts one particular service, one has to restart a thousand dependent services (maybe some Java, some Perl, some PHP, some robots, ..) and then maybe restart services which are dependent on them – that’s a maintenance nightmare. So the software has to be altered to be able to handle connection drops automatically, by reconnecting. Once the software has been so altered, one no longer needs to alter the “wait_timeout” on the server.

The error was:

org.hibernate.util.JDBCExceptionReporter: The last packet successfully received from the server was 56697 seconds ago. The last packet sent successfully to the server was 56697 seconds ago, which  is longer than the server configured value of ‘wait_timeout’. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property ‘autoReconnect=true’ to avoid this problem.

Quite a helpful error message, don’t you think? But

• I’m not going to increase “wait_timeout” as discussed above,
• “testing validity” in the application – well I was using standard software Hibernate which should take care of this sort of thing automatically, but evidently wasn’t
• and we were already using ?autoReconnect=true in the JDBC URL (this evidently wasn’t working).

I figured I really needed to get to the bottom of this. Googling just showed (many) people with the same problem, but no solutions. The only way to get to the bottom of software is to read the source. (It has been the way to resolve issues of simple things simply not working in MySQL before.)

I stopped looking in the MySQL source for why “autoReconnect=true” didn’t work when I saw the following text in the source describing the autoReconnect parameter:

The use of this feature is not recommended, because it has side effects related to session state and data consistency

I have no idea what particular side-effects are meant here? I guess that’s left as an exercise for the reader, to test their imagination.

And anyway, I figure that a reconnect-facility belongs in the “application” (Hibernate in my case) as opposed to in database-vendor specific code. I mean the exactly the same logic would be necessary if one were connecting to PostgreSQL or Oracle, so it doesn’t make sense to build it in to the database driver.

So then I looked in the Hibernate code. To cut a long story short, the basic connection mechanism of Hibernate (as specified in all the introductory books and websites, which is probably how most people learn Hibernate) doesn’t support reconnecting, one has to use H3C0 connection pool (which itself didn’t always support reconnecting)

(I don’t want to use container/Tomcat-managed connections, as I have some command-line robots which do some work, and I don’t want to use different code for the robots as the web application. Although another company defined Servlets which did “robot work”, and the robot was just a “wget” entered into Tomcat – to get the user of container-managed connections – but this seems a too-complex solution to my taste..

But once one’s used H3C0, the default behavior seems to be that to process a request, if the connection is dead then the user sees and error – but at least it reconnects for the next request. I suppose one error is better than infinite errors, but still not as good as zero errors. It turns out one needs the option testConnectionOnCheckout - which the documentation doesn’t recommend because testing the connection before a request might lead to lower performance. Surely the software firstly has to work, only secondly does it have to work fast.

So, to summarize, to get a connection to “work” (which I define as including handling dropped connections by reconnecting without error): In “hibernate.cfg.xml”:

<!-- hibernate.cfg.xml -->
<property name="c3p0.min_size">5</property>
<property name="c3p0.max_size">20</property>
<property name="c3p0.timeout">1800</property>
<property name="c3p0.max_statements">50</property>
<property name="connection.provider_class">
   org.hibernate.connection.C3P0ConnectionProvider</property>
<!-- no "connection.pool_size" entry! -->

Then create a file “c3p0.properties” which must be in the root of the classpath (i.e. no way to override it for particular parts of the application):

# c3p0.properties
c3p0.testConnectionOnCheckout=true

Amazing, that that stuff doesn’t just work out of the box. Programming the solution myself in Uboot took, I think, 1 line, and I’m sure it’s not more in WebTek either.

That was an amazing amount of effort and research to get the simplest thing to work. Now if only this project had been paid by the hour…..

[Update 28 May 2009] More Java hate today. Starting a new application, deployed it, and it didn’t work. In the morning, the application was down. Reason: The new project used Hibernate 3.3, and upgrade from 3.2 to 3.3 requires the “connection.provider_class” property to be set. Previously the presence of “c3p0.max_size” was enough.