A friend said this to me today:

it’s really odd, why do fans of dynamic languages often have this cult-like approach?

It’s an interesting thought. I know many people who love dynamic typing and reject static typing without knowing much about it. (Of course there are many people who love dynamic typing but who do know lots about static typing.)

It got me to thinking, I reckon if you only know X and know nothing about non-X, you might start to fear non-X. Your fear might express itself by reinforcing your belief that X is the best, that non-X is no good.

I’m sure this is a standard human emotion, but I’ve absolutely no idea what it’s called. “Fear of the unknown” perhaps.

Anyway, an hour later I experienced a similar emotion when reading this:
http://blogs.msdn.com/b/oldnewthing/archive/2013/03/20/10403718.aspx

I have absolutely no idea what this article is on about. I know so little about Windows. I always deploy on UNIX or Linux. I always have. There’s so much to know about Windows. And all that stuff that there is to know: I don’t know any of it.

I reckon, when being hit with a volley of emotional, religious, and cult-like arguments about why all things other than X are bad, one should take into account the possibility that the arguer doesn’t know anything other than X, and is afraid of everything which would expose that lack of knowledge.

Talking to someone who doesn’t work in IT, and who isn’t a native English speaker; She was talking about various people in software development, and referred to them doing “programmizing” (as opposed to “programming”).

That’s a great concept. “Progammizing” reminds me of “terrorizing”. As in: before you have a bunch of processes involving paper and people and it all works. Then, some people come in and do some programmizing. Afterwards, nothing works, everything’s slow, crashes, has bugs, loses data, you have to turn stuff off and on a lot….

My mate has given me a laptop (which in itself is a strange occurrence, but that’s not the subject of this blog entry). It comes with two OSs, which you can choose from at boot:

• Android
• Windows 7

I mean, sure enough, this Acer Aspire netbook literally runs Android. But the hardware is different from any other device I’ve used Android on in the following respects:

• It dosen’t have a touch-screen
• It doesn’t have a SIM card
• It has an x86 processor (Intel Atom), not ARM
• It does have a keyboard
• It does have a trackpad (so a pointer appears on the screen like any other desktop OS)
• The screen is much bigger, and horizontal (so e.g. on the bar you can pull down from the top, each row is just ridiculously wide)

I mean, it does work, but I’m not sure why you’d want to use it?

Perhaps this was the inspiration for Windows 8.

I’ve heard the phrase “Let’s rewrite this in Java!” uttered in various meetings at various companies at various times in my career. Often by managers. All such projects to completely re-write the company’s software in a new language or framework inevitably end in disaster. Why is this? I must confess I don’t really know.

I suppose it doesn’t have to be Java, although it often is. Java is the classic. For example, a big company comes along and sees, oh my god, the entire code-base is written in a scripting language!! and feels the need to “clean up” the source code. Without further thought, the decision is taken to write everything. I think at least such companies inevitably choose to transition to Java.

It’s the same with frameworks. Oh my god, this is written with web framework X which is no longer trendy, or with no framework at all!!! The decision is taken to “introduce” a new front-end framework, often trendy, which necessitates a re-write of at least the entire front-end, if not the entire system.

Consequences

By “end in disaster,” I mean, mainly, that the system is never completed, and never goes online. At some point the code just gets deleted. (Or stored in some backup system, or on a branch of the VCS, where nobody ever looks at it, which are basically equivalent, if one judges the success of software by it going online and the benefit it brings to its users, and/or the money it brings to its sponsoring company, etc)

There is a danger to re-writes, beyond just costing the sponsoring organization money, without any benefit. In a small team it seems too much to concentrate on two projects at once (the old and the new). And the tasks associated with the new (conception, programming afresh) are much more fun than the tasks associated with the old (bug fixing, performance issues), so the programmers tend to prefer them. They also feel the work on the new system is a better investment of their time, as that work will be online a year from now, in contrast to work on the old system. With weak technical management programmers may do what they prefer as opposed to what is important. But if this takes a year or two, what about existing customers? The live system can just stagnate, while their competitors add new features and progress. I think in this case it’s important to have people assigned to the old project, and people assigned to the new. It might suck to be the guy assigned to the old project but what can you do? The work has to be done.

Reasons for failure

Perhaps it’s just a nature of re-writing software, as described by Joel on Software. But I think it’s more than that. I’ve seen re-writes or refactoring projects be successful. Classically the re-write of the Mac operating system was very successful, and the re-writing of the Netscape browser into what became Mozilla Firefox. The re-writing of Windows into Windows NT. At Uboot we re-wrote many pieces of software and they all worked out fine, and were much better for having been re-written.

I think it could have something to do with how the project starts out. What is trying to be achieved? I think if the project starts out with the objective “we need these completely new features” and then it turns out that a re-write is necessary, as the current code-base can’t support these new features, then such a re-write can be successful. If it starts out with the only objective being “we need to re-write it”, for example because the programming language is considered wrong, or the software design is considered wrong, i.e. things which are more faults of the old system than benefits of the new, then things will start to get into trouble.

I think if the objective is simply “we need a re-write” then everyone will chime in with all the problems the old system had and how they can be fixed. For example the software design can be improved (for its own sake). Newer frameworks can be used (for their own sake, or some vague reason such as they are more modern.) Radical new features, which have been on the roadmap for a while, but never really seemed to be totally necessary, can be introduced. Perhaps this is just too much at once, combined with re-writing a system which might have had years of work already put into it without all those new features? But I’m not sure it’s just that.

I also notice that, while the old system is still online, such re-writes do not have any feeling of urgency. (“The old system is online anyway. It’s important that we get it right this time, even if that takes longer.”) Perhaps that too is bad – urgency is often a good motivator for simplicity, and simplicity an attribute of systems with desirable properties such as ease of maintain, having fewer bugs, fewer scalability problems. Related: Perhaps people feel more allowance to act as an architecture astronaut; after all, the objective is to have an architecture, not to have features for customers.

Perhaps it’s because if there are no clearly expressed objectives beyond “we need a re-write” then the other objectives which haven’t been clearly stated (such as which new features to add) can change over time? Half way through the project a new framework is released and is decided to be used, on the grounds, well, we’re doing a re-write so we should use the most modern framework available. Or the same with new requirements which crop up during the project. If one introduces a new framework or feature sufficiently often, and these introductions require re-writing part of the already written re-write, then perhaps the re-write will never get a chance to be finished.

Perhaps it’s that people have a problem conceiving of a new system (which, on the day it’s released, must already have lots of features and scale to lots of users)? Perhaps people find it easier to go one feature at a time, and scale the existing system when necessary? Which is how an initial system often starts out.

Perhaps the projects fail because they take too long? One the one side you want to have people on the old team producing new features for the customers and keeping up with the competition. On the other side you don’t want the new project to work against a moving target and keep on re-writing bits of the re-write. These are contradictory objectives. Perhaps the trick is to get the re-write out the door in 3-6 months, before the old system has had a chance to change too much? Which is a problem if the system is beyond a certain size, or the re-write too ambitious.

Perhaps implementing a system in a language and/or framework with which the development team, and perhaps also architects, are not familiar, is a really bad idea too? This will inevitably lead to mistakes being made. Perhaps that can be mitigated by having an external consultant review the designs, perform design, do code reviews, or by using a language/framework with which senior members of the team are already familiar.

Conclusion

I don’t really know why, when I hear the words “Let’s rewrite it in X!” (where X is nearly always Java!), I just know the project is going to cost the company 1-2 years and most of the development team, get thrown away at the end, bring no benefit to either the users or the development team, and cause the live product to stagnate behind the competitors for this time.

And I don’t really know what you can do about it. Not re-writing things is clearly not acceptable either (would you like your primary system to be the latest incarnations of Netscape 4 on Windows ME?). I think you need to have clear objectives (which features do you need that you can’t implement on the old system, which technology do you want to move to and why?), keep separate teams for old and new, have at least one person with years of experience in the new technology, and make sure the new vision gets wrapped up within 3-6 months to avoid the live system having acquired too many new features.

Shameless self-promotion: I work as a software architect and have been involved in many successful re-writes. We re-wrote the UCP Media Album from Perl into Java J2EE and deployed it an an American mobile operator. For Offer Ready we’ve completely re-written our calculation engine. Uboot was in a constant state of flux e.g. introducing payments into messaging, etc. At easyname I oversaw the re-write of many key features such as the way money was handled. If you are hitting a wall with your current system and want to know your options, contact me.

This year has been an amazing year professionally. Together with my employee Martin, and a contractors I’ve been using for visual design and HTML, I’m proud to report we’ve taken the following features for the following customers online 2012.

Companies can give their employees mobile phones. Companies pay the bill for these mobile phones. Companies with thousands of such phones get PDF bills which would be inches thick if one were to print them out. mobilreport imports the data electronically and allows various reports over the data: which employee phoned the most? To which countries? During their free-time or during business hours? etc.

Programmed using Java Wicket, we use MySQL to store our data, which then gets exported to XML, transformed via XSLT to create the reports that you see in the browser, and which you can download as PDF, RTF or XLS. My colleague Martin Schmidt at Onestop Concept used Altova Stylevision is used to make designing the reports a case of WYSIWYG.

In 2012 we released:

• Monthly emails. We send employees monthly emails informing them of how much their mobile phone has cost the company. These are also designed in a WYSIWYG fashion with Stylevision.
• Complete new visual design. Design by Mandy, HTML from my people in the Philippines, implementation by Martin.
• Login administration. Whereas which users existed were previously done by altering configuration files, now a full GUI exists to allow customers to do this themselves. Implemented by Martin.
• More providers. Importing data from providers is a messy process! The files are not well-formed (e.g. CSV files where the “;” field separator appears unescaped in the middle of a cell, making the CSV file technically unparseable). We always supported A1, now we also support Telering AT, T-Mobile AT, A1 Landline AT, COLT AT, Vodafone DE, T-Mobile DE. (Want to see some nastiness? Check out EDIFACT. Want some changes? Apply to the UN.) Implemented by Martin.
• Master-detail reports. For example: On a table e.g. to which countries calls were made. Clicking [+] by one country opens an expansion where all users who made calls to that country were made, sorted by the cost of those calls. Which data forms the master, which data forms the detail, and how they are connected, is specified by configuration.
• Subsidiaries. Previously it was necessary to log out and log back in again as a different “customer” if one company had multiple subsidiaries. Aggregated data over all subsidiaries was this impossible. Now the model has been changed; each customer has 1-n subsidiaries.
• PostgreSQL. Ported the system from MySQL to PostgreSQL; although this was finished, it never hit production. (The issues we had with MySQL were resolved.)

I’ve been working for Offer-Ready on-and-off since March 2007. Operators can integrate this software into their workflow and find the optimal tariff for their customers. With this optimal tariff, documents such as offers can be generated.

This is great software that throws up a number of interesting technical challenges. We consider millions (no exaggeration!) of mobile phone tariff variations in well under a second, to find the optimal one.

This year we designed three new pieces of software from scratch, they are now all online:

• Publisher. When a change is made to the configuration defining the mobile phone tariffs, this software takes those changes, and does what needs to be done so that the new configuration is online and available. This is a non-trivial process involving many steps and much wall-clock time, this software ties all those steps together. While publishing is happening, the old version of the configuration is online and can be used; in case it fails, the old version stays online.
• Profile Editor. Data entry for the system. User may enter their usage data, which then calls the existing system (API) to work out the best tariff for these data. Programmed by Martin.
• Document Viewer. Results of calculations may be displayed in the browser; options to download PDF, RTF, XLS exist.

HR Gamification System

New product, not released yet. Website should support Human Resource departments to use the talent already within the company to find new talented employees. Using appropriate current techniques such as gamification.

I will be providing the software development for this product. Currently supporting the customer with requirements, doing software design. HTML is being done by my employees in Philippines. Next year we’ll get on to the software development (with Martin) and taking the system into production.

Between July 2011 and February 2012 inclusive I was the sole developer for a charity web platform called United Youth for Kurt Braunhofer. Although development was continued by a new team from March onwards, in December 2012 the new team took their platform online and included one piece of software from me: the Symbol Page.

After about 6 months of work we took the new design for mobilreport online a week or two ago.

This software is used by a number of big-name Austrian companies such as Austrian Gaming Industries, Stihl, GlobeAir, Blum, Fohrenburger, Stiegl, neudoerfler, and various others we’re not at liberty to name.

I have been working for mobilreport from the beginning. Onestop Concept designed the requirements together with the customer and I did the software design and implementation. For this new design project, I also was responsible for the visual design (Photoshops), the HTML (outsourced to Philippines, which was very successful), and my employee Martin integrated the HTML with our software.

A friend recently asserted that he doesn’t bother with security updates on his computer and for his software. This is wrong. Security updates are important and should always be installed.

This sort of thing will happen to you if you don’t update your computer. (Not that this attack was directly related to lack of updates, but the principle is the same..)

http://www.wired.com/gadgetlab/2012/08/apple-amazon-mat-honan-hacking/

To which he replied:

I don’t believe that updates play that much a roll. So i wont update my OS X to 10.7 (I use 10.6) because I think my machine won’t be able to handle it.

It’s important to differentiate between upgrades and security updates. Upgrades are e.g. from 10.6 to 10.7, which give you new features etc., but each version of OS has its own security updates (e.g. security updates are produced for 10.6). The only time when you need to upgrade for security reasons, is when the OS you’re using is so old it no longer gets security updates (e.g. Apple don’t produce security fixes for 10.4 any more).

Security updates are important because, if a researcher finds a security hole, they’ll inform the OS or software vendor (e.g. Apple for OS X), but not yet the rest of the world. Apple will fix it and produce a security update. After everyone has the security update, the researcher announces it to the world. From this moment on, hackers and virus writers have access to the information and can incorporate this information into their viruses. If you are running a system without security updates, the virus can infect your system.

Once a virus is on your system, it can record your keystrokes, see everything you’re submitting to the internet (e.g. email username/password), can alter webpages that you see (e.g. when you click “login” the information is submitted to their servers, not to the website you want), and so on. With this information attackers can slowly take over your identity, use your money, delete your data, and so on.

Not only operating systems need to be updated, also browsers, and all other software.

There are also “zero day attacks”, where a security researcher releases the information about a security hole immediately (after “zero days”), or where a hacker themselves find a security hole. There’s not much you can do about them, just install the updates as soon as they’re available, but there’s still a window between when the virus is out there and when your system is no longer vulnerable, when you can be attacked. But even then, better make this window as small as possible by installing the update as soon as it’s available.

Bought a new computer, decided to try out Ubuntu Linux. Here’s my report of things i liked, things that worked, things I disliked, and things that didn’t work.

Things that are improvements over Windows and Mac:

• Single update manager! No more Java, Adobe, Windows etc updates. My experience with Mac (at least 10.4) wasn’t much better. The Debian/Ubuntu update manager is extensible in the sense you can add random internet addresses for third-party or proprietary software updates if you so choose, and all updates appear together. With dependencies. Of course. I mean this is something Linux has got right for a long time, and Windows and Mac OS X haven’t.
• Moving windows with “alt”. Holding down “alt” allows you to just click on windows and move them around, I’ve totally got used to that, and am sad that Windows at work doesn’t offer that.
• UI stays responsive even in the face of slow disk activity (this was definitely not the case with XP64, where one app would slowly stop responding after another, until they all came back to life once the external HD had spun up, although that could have been some aspect of my broken computer of course..) My experience with Mac hasn’t been much better.
• Indicators. I think the things along the top of the screen are the most helpful (and visually appealing) of any OS i’ve used. I like the fact apps can attach themselves to the relevant bit (e.g. music player under the sound control, IM programs under the IM status control.)

And improvements over Windows (but these things work on a Mac):

• File Locking. Never again “This action can’t be completed because the file is open”. This is one of my most major problems with Windows. Happens nearly every time you want to delete something, the only solution, often, is a reboot.
• Windows screen update bug gone. I’ve often had the experience that Windows doesn’t update the screen correctly (e.g. torrent program, behind Chrome, just paints a progress bar in the middle of the Chrome window, i.e. something doesn’t realize that progress bar isn’t visible as the window isn’t on top). But I never had this problem on a Mac.
• The horizontal scroll wheel on my Apple mouse works, which it didn’t in XP. For example in VLC it moves the video forward and backwards, other places it scrolls. (Not sure about Windows 7.)

Things that aren’t so good:

• When you turn the computer off, it just kills all applications! Doesn’t give the applications a chance to complain about unsaved documents, or do a clean exit (e.g. torrent programs updating trackers). I mean Windows 3.0 from 1990 could do that, RISC OS 2 from 1989, etc., could do that…
• Resizing windows with a rubber-band box. I can’t believe that this is not solid. Not the biggest practical problem in the world but still, I mean RISC OS 2.0 had that in 1989. (NEXTSTEP only acquired this feature with Mac OS X released around 2000, not sure when Windows acquired this feature.)
• Menu system on Unity is ridiculous. Menus are on the top of the screen like the Mac, but they’re hidden! Unless you move the mouse to the top menu bar, menus don’t appear, you’ve no way to know they’re there. I searched for menus for quite some time (really!) Would you know where are the menus on this screen?
• Want to start a program on start-up? Rather than just right-clicking on the program in the program bar and saying “load on startup” (Mac, best) or moving it the start-up folder (Windows, also not bad), there is a facility buried deep in some menu where you can just type in command-line actions to be executed upon start-up.
• TortoiseSVN on Windows is, in my opinion, the best interface to any version control system ever. Integrates with the system’s file manager (isn’t an app which needlessly has its own file manager.) Right click using your normal file interface, “Update” or “Commit”, no complexity, just works.
• It suggests getting music etc from the “Ubuntu One Store”, I mean it uses the fact you’ve chosen their OS in order to get you to pay for stuff, a strategy reminiscent of a certain computer manufacturer named after a piece of fruit. But at least it’s not the case that one paid for the OS which they then use to get you to pay for stuff.
• I like the “minimize all” or “show desktop” Windows-M feature, haven’t found any equivalent on Ubuntu yet.

Things that would have worked fine on Windows or Mac, but surprised me by also just working on Ubuntu:

• It recognized my webcam and I was able to use it to do video calls on Skype. (“Creative Live! Cam Chat HD Webcam”)
• Printer just worked (Canon MP510)
• I have a PCI WLAN card in the PC, that just worked.
• Can read/write my NTFS disks without any problem.
• The “eject” button on my Apple keyboard ejects the CD. That didn’t work on my old XP computer, nor on my Sony Laptop which itself has an eject button on its internal keyboard. (But obviously works on a Mac.)

Things that didn’t work

• Chome. The most surprising thing, perhaps, is that Chrome doesn’t really work well? Often parts of the screen are not updated, and if you type things into forms you often can’t see what you’re typing. So I have to use Firefox.
• Unpausing CD. Insert a music CD, it plays, but as soon as you press pause, it can’t restart, it just doesn’t do anything when you try. You have to eject/re-insert the CD.
• Sound is funny (crackly) upon startup, but thankfully goes away after a minute or two

To be fair even with the negative things, I mean, although not everything works, one mustn’t forget that Windows doesn’t just work either, it’s just one’s used to it (e.g. when I start my work PC, some random Tomcat error always comes up, which you just have to click away..)

Annoyingly I’ve now transitioned myself into a state where I prefer some things about Ubuntu and some things about Windows, therefore neither Ubuntu nor Windows (nor Mac) appears to me to be the perfect operating system. Before I’d tried Ubuntu I was happy using Windows. Arguably this state transition was a mistake in terms of maximizing my operating system satisfaction.

Ubuntu 12.04, vs WIndows XP-7, Mac OS 10.4

For various reasons I have decided to leave s IT Solutions. The company was good, I stand by everything I said in my previous blog posts about working for large companies.

(1) I felt my position there was too junior. I was constantly having to justify the things I wanted to do in large meetings with various “senior” people who’d been working there for 10 or more years. But these people were younger than me, and had been working at that company all their life, and simply didn’t have the variety of experience that I had. They felt things had to be done a certain way, but I had seen various ways including the way they were proposing, but they weren’t interested in listening to my input on the consequences of that way. Basically the reason for that was people didn’t want to dilute their power; I don’t hate them for this, I just don’t need to work there either.

(2) Various things were far too complex. It took me about a month to learn how to deploy something on one particular environment (this wasn’t the live environment!) which involved asking a project manager for a release number (which involved justifying to him why the deployment was necessary, often he didn’t agree..), writing an email to dept X, they would forward that to dept Y, then there was some Windows tool with multiple tabs and multiple fields per tab, where you had to enter various magic data, plus there was something on the Windows tray you had to right-click and configure, then there was the web-based deployment tool and you had to know where to click in that.. Plus I didn’t have permission for any of these systems, I didn’t know that I needed to have asked in the past for permissions to these systems as I only discovered “just in time” that they existed at all. No documentation, nobody told me they existed as that would reduce their power that the bosses needed to go to them to get something done.

Anyway, I was told at least once, “well, you might not like it, but you’ll have to get used to it. That’s the way your life is now!!”. Wrong ;-)

For the next months I have the following things lined up:

(1) Implementation of a Java Wicket based system of analyzing mobile phone usage data. Usage data is uploaded via CSV files, I parse them and put them into a database, then the main screen involves a set of reports, defined over a configuration file, with snippets of “where” clauses in them. The user may select further restrictions via drop-downs, these are added to the “where” clause. The “select” against the DB is performed and an XML is created of the results. XSLT created by Altova Stylevision, which the customer has created and checked into my SVN, is then performed on the XML to produce PDF, RTF or HTML reports. I don’t need to change any code to change the layout of the report, it’s all in the XSLT file. To give the user a nice interface for managing the CSV files Windows Explorer and TortoiseSVN is used, I read the reports from SVN and insert the data programmatically via svnkit which is nice and easy to use, and I keep the database and configuration in sync like this. Onestop Concept provided analysis of the requirements, and I did all the software architecture and programming. Deployed at Nessus. All in all, a nice project!

(2) Further implementation on Offer Ready for Onestop Concept (Martin Schmidt) which involves using some cool algorithms to determine which mobile phone tariff (incl. packs) is the best option given a set of usage data. Written in Java and C. (Not written only in Java!)

(3) Working for firstload (hopefully!), a platform where you can download binary files acquired from Usenet. I will be working on their e-commerce software, for example the invoice-generation software, introducing tax rules, extending/re-writing the subscriptions logic, modernizing the software, and so on. PHP and MySQL. This is a domain I’ve worked in a few times before, I did the e-commerce software for Uboot’s SMS functionality, I wrote the Subscriptions system for easyname for Nessus, was hired to write a pay-out and tax logic module for TheContentMarket then FatFoogoo, and so on.

I’ve just got back from a 4-day course on working in teams (“Erfolgreich Zusammenarbeit im Team” by Coverdale for my employer s IT Solutions).

First conclusion -> Top food at Hotel Wende in Neusiedl am See.

I am so tired now, it was much more tiring than I had imagined it would be. Lots of “exercises” which really required a lot of concentration, especially if one was to learn the maximum possible from them. I must admit I went into it not expecting much (it was a mandatory course) but I tried my best anyway and did actually learn more than I thought i would (i.e. more than zero). (It did start a bit like S01E04 of the Office UK, I must admit..)

So here are my learnings, in no particular order -  (as much for my benefit in the future as anything else)

• Individual creativity. If you ask everyone in the room for ideas, once the first person suggests an idea, everyone tends to discuss and build on that. If you ask everyone individually to write ideas down, in silence, then you have a much greater range of suggestions.
• Need for moderation in meetings. In the exercises, it became clear that without a moderator / leader, the subject would wander randomly, and away from what should be discussed. This is rather obvious, but seems to contradict “scrum” (or at least the way it’s done where I work), where the belief is that if you put a range of people together, they’ll “work it out” and magically great ideas will be formed (instead this leads to “design by committee”).
• In meetings, write on a whiteboard not on paper. When dealing with a common idea, I am the one who tends to write the things down. Although it sounds obvious, I had never really noticed that nobody else can see what I’m writing it down. Writing the same thing on a white-board means everyone can see what’s going on and can comment on it.
• Dealing with blocking people. When dealing with people who are excessively negative and blocking in meeting, where no one can suggest anything without being shot down by the “expert”, it was suggested to try and actively split the process into two parts (1) Creation and collection of ideas, (2) Feedback/Risks to those ideas. I shall give this a try.
• If a new team member joins your team, try and treat them as equal to the existing team members as soon as possible. I certainly welcome when this is done. (I caught myself saying I was “new” the other day, although I’ve been at the company for 9 months. I wondered why that was. I realize now it’s because the people who’ve been there longer guard their role, don’t treat new people as equals, therefore I still feel new.)
• Write not only the answers but also the question on the white-board – When writing answers to a question on the white-board, if you don’t include the question (i.e. just ask it verbally at the start of the meeting) people forgets that the answers increasingly don’t fit to the question as the discussion progresses and the topic wanders.