Databases and Life » Uncategorized

Self-employed again

adrian — Wed, 30 Mar 2011 15:11:40 +0000

For various reasons I have decided to leave s IT Solutions. The company was good, I stand by everything I said in my previous blog posts about working for large companies.

(1) I felt my position there was too junior. I was constantly having to justify the things I wanted to do in large meetings with various “senior” people who’d been working there for 10 or more years. But these people were younger than me, and had been working at that company all their life, and simply didn’t have the variety of experience that I had. They felt things had to be done a certain way, but I had seen various ways including the way they were proposing, but they weren’t interested in listening to my input on the consequences of that way. Basically the reason for that was people didn’t want to dilute their power; I don’t hate them for this, I just don’t need to work there either.

(2) Various things were far too complex. It took me about a month to learn how to deploy something on one particular environment (this wasn’t the live environment!) which involved asking a project manager for a release number (which involved justifying to him why the deployment was necessary, often he didn’t agree..), writing an email to dept X, they would forward that to dept Y, then there was some Windows tool with multiple tabs and multiple fields per tab, where you had to enter various magic data, plus there was something on the Windows tray you had to right-click and configure, then there was the web-based deployment tool and you had to know where to click in that.. Plus I didn’t have permission for any of these systems, I didn’t know that I needed to have asked in the past for permissions to these systems as I only discovered “just in time” that they existed at all. No documentation, nobody told me they existed as that would reduce their power that the bosses needed to go to them to get something done.

Anyway, I was told at least once, “well, you might not like it, but you’ll have to get used to it. That’s the way your life is now!!”. Wrong ;-)

For the next months I have the following things lined up:

(1) Implementation of a Java Wicket based system of analyzing mobile phone usage data. Usage data is uploaded via CSV files, I parse them and put them into a database, then the main screen involves a set of reports, defined over a configuration file, with snippets of “where” clauses in them. The user may select further restrictions via drop-downs, these are added to the “where” clause. The “select” against the DB is performed and an XML is created of the results. XSLT created by Altova Stylevision, which the customer has created and checked into my SVN, is then performed on the XML to produce PDF, RTF or HTML reports. I don’t need to change any code to change the layout of the report, it’s all in the XSLT file. To give the user a nice interface for managing the CSV files Windows Explorer and TortoiseSVN is used, I read the reports from SVN and insert the data programmatically via svnkit which is nice and easy to use, and I keep the database and configuration in sync like this. Onestop Concept provided analysis of the requirements, and I did all the software architecture and programming. Deployed at Nessus. All in all, a nice project!

(2) Further implementation on Offer Ready for Onestop Concept (Martin Schmidt) which involves using some cool algorithms to determine which mobile phone tariff (incl. packs) is the best option given a set of usage data. Written in Java and C. (Not written only in Java!)

(3) Working for firstload (hopefully!), a platform where you can download binary files acquired from Usenet. I will be working on their e-commerce software, for example the invoice-generation software, introducing tax rules, extending/re-writing the subscriptions logic, modernizing the software, and so on. PHP and MySQL. This is a domain I’ve worked in a few times before, I did the e-commerce software for Uboot’s SMS functionality, I wrote the Subscriptions system for easyname for Nessus, was hired to write a pay-out and tax logic module for TheContentMarket then FatFoogoo, and so on.

Teamwork training in Neusiedl

adrian — Fri, 26 Nov 2010 15:54:54 +0000

I’ve just got back from a 4-day course on working in teams (“Erfolgreich Zusammenarbeit im Team” by Coverdale for my employer s IT Solutions).

First conclusion -> Top food at Hotel Wende in Neusiedl am See.

I am so tired now, it was much more tiring than I had imagined it would be. Lots of “exercises” which really required a lot of concentration, especially if one was to learn the maximum possible from them. I must admit I went into it not expecting much (it was a mandatory course) but I tried my best anyway and did actually learn more than I thought i would (i.e. more than zero). (It did start a bit like S01E04 of the Office UK, I must admit..)

So here are my learnings, in no particular order - (as much for my benefit in the future as anything else)

Individual creativity. If you ask everyone in the room for ideas, once the first person suggests an idea, everyone tends to discuss and build on that. If you ask everyone individually to write ideas down, in silence, then you have a much greater range of suggestions.
Need for moderation in meetings. In the exercises, it became clear that without a moderator / leader, the subject would wander randomly, and away from what should be discussed. This is rather obvious, but seems to contradict “scrum” (or at least the way it’s done where I work), where the belief is that if you put a range of people together, they’ll “work it out” and magically great ideas will be formed (instead this leads to “design by committee”).
In meetings, write on a whiteboard not on paper. When dealing with a common idea, I am the one who tends to write the things down. Although it sounds obvious, I had never really noticed that nobody else can see what I’m writing it down. Writing the same thing on a white-board means everyone can see what’s going on and can comment on it.
Dealing with blocking people. When dealing with people who are excessively negative and blocking in meeting, where no one can suggest anything without being shot down by the “expert”, it was suggested to try and actively split the process into two parts (1) Creation and collection of ideas, (2) Feedback/Risks to those ideas. I shall give this a try.
If a new team member joins your team, try and treat them as equal to the existing team members as soon as possible. I certainly welcome when this is done. (I caught myself saying I was “new” the other day, although I’ve been at the company for 9 months. I wondered why that was. I realize now it’s because the people who’ve been there longer guard their role, don’t treat new people as equals, therefore I still feel new.)
Write not only the answers but also the question on the white-board – When writing answers to a question on the white-board, if you don’t include the question (i.e. just ask it verbally at the start of the meeting) people forgets that the answers increasingly don’t fit to the question as the discussion progresses and the topic wanders.

1k reputation on stackoverflow.com

adrian — Fri, 05 Nov 2010 14:18:01 +0000

A few weeks ago I got really into stackoverflow.com, the question/answer site for programming:

But now I’ve got a bit bored of it. If you want to achieve reputation, it seems you have to take the following things into account:

People will only vote up your answers who are able to see that your answer is correct; therefore common topics or easy topics (such as simple SQL questions) will get more votes and therefore reputation points than something which requires more detailed knowledge such as Oracle optimization tips or Lucene full-text search indexing strategies. (Example of a popular answer, example of non-popular answer)
Obviously once a question has been correctly answered, there’s no point in answering it again. Therefore people tend to surf the newest questions, looking for things to answer. Therefore if you answer a really old question, even if it’s still valid and hasn’t been answered, nobody will see it and vote up your answer and you won’t get points.
People looking at a question, if there are 3 of 4 answers, they will vote up one of them if they are correct. If yours isn’t there yet, you don’t get voted up (yet). Therefore you have to answer quickly. Answering quickly is probably the exact opposite of answering in a reflected and considered way, with a lot of detail, etc.

So basically you have the situation that, in order to gain points, you have to keep on refreshing the “newest questions” page, find a really easy question which anyone could answer, and then answer it in a really non-detailed way.

But still, for a week or so, I was obsessed with my status. So it’s been definitely a good learning experience for me, that “game mechanics” with arbitrary rewards (which don’t cost the site owner anything) can really get your users to get obsessed with whatever you want them to do and make them do free work for you (in this case, build up a set of questions and answers on a website, which does have value to the site owner).

XSS attack via unchecked image uploads

adrian — Tue, 17 Aug 2010 11:45:20 +0000

If you allow users of your website to upload data (e.g. images), and you display this data to other users, you need to open the file on the server to examine it and check that it really is what it should be (e.g. an image).

Most website software will need to examine the image anyway, to extract thumbnails, determine width/height, etc. In which case, this security comes for free. But I’ve seen software which doesn’t have any such needs, and thus server-side examination is not done.

The reason is:

An uploading user (attacker) may upload a file such as “xxx.jpg” whose contents are in fact HTML/Javascript rather than an image
The web server may serve this file with the extension in the URL and the Content-Type: image/jpeg, however…
Internet Explorer may under some circumstances ignore these two pieces of information, instead preferring to look at the bytes of the file, and work out what type of content the file contains
Even if the image looks right in the tag, if the viewing user right-clicks and views the image in its own window, then the data will be interpreted as HTML/Javascript
This gives the file the capability to read the viewing user’s cookies, look into forms in other windows, etc., as the HTML/Javascript appears to have been served by the site the viewing user is viewing.
Through having e.g. tags (not affected by single-source policy) the file can send e.g. GET requests to an external site and transport this cookie and form information.
Thus the attacker’s external site now has cookie/form information, and the attacker can use this to impersonate the user at the site.

I was unaware of this before @ch2500 brought this my attention, thanks! More information.

Letters

adrian — Mon, 16 Aug 2010 08:19:03 +0000

The UNIX program “mail”, not the mail client one would use by choice under most circumstances. However today I have the pleasure of using a Solaris box which seems to be filled only with legacy tools like original “vi”, a “tail” where if you do “tail -f *” it only tails the first matching file and not all files.. but I digress.

$ mail
mail: Too many letters, overflowing letters concatenated

I mean ignoring the error itself and the disrespect which it forces me to show this program, look at the wording.

The error involves the word “letters”, i.e. the program was written before the word “mail” was the standard word for internet mail.

Humans reading source is not a strategy to defeat viruses

adrian — Thu, 12 Aug 2010 10:23:49 +0000

Apple’s policy of reviewing all apps is officially (amongst other reasons probably) to make sure apps don’t contain viruses etc.

However here we clearly see that having humans look over source code of all apps does not catch all policy violations (violations include writing viruses, and include other things as in this case):
http://www.mobilecrunch.com/2010/08/12/apple-pulls-camera-from-the-app-store-after-its-developers-reveal-a-contraband-feature/

If they can’t catch this policy violation, they won’t be able to catch all viruses either. (And they would need to catch all viruses to make it worth looking for viruses at all; catching all but one viruses is not sufficient. A single virus can do a lot of damage and infect widely.)

Apple not allowing one to just write and distribute code on their iPhone platform is an outrageous limitation of freedom. Presumably they have their reasons for doing so (e.g. so they make money on every commercial purchase of an application), but at least that reason they claim for doing it (protecting the public) is demonstrably not true.

(P.S. I’m not proposing a strategy which will defeat viruses; I’m just saying that Apple’s app store policy is not one.)

(P.P.S. Game console manufacturers have always had similar restrictions, those limitations of freedom are just as outrageous.)

Sudoku Solver

adrian — Sat, 15 May 2010 13:45:16 +0000

The above is a Sudoku solver I wrote in Java a while back. Here it is running in the browser. The Java has been converted to in-browser Javascript by Google Web Toolkit.

Please feel free to type in the hardest Sudokus you can find anywhere, and let me know if it can solve them, I haven’t found any it can’t yet.

I assume the in-page Javascript will not work in RSS readers, email notifications, etc. If that’s the case, click here to get to the original Sudoku solver page on my blog.

The algorithm works by backtracking, i.e. simply going through all squares from top-left to bottom-right, and finding the set of “candidate numbers” for that square (i.e. which numbers are not already used on that row, column or 3×3 block). The first candidate number is “applied” by placed it into the square, and the algorithm moves to the next square. If there is a square where the set of candidate numbers is empty (i.e. no number would work, given the candidates applied in previous squares) then the system returns to the previous square and tries the next candidate for that square. Once the system has gone “beyond” the bottom-right square, then the set of applied candidates consitutes a solution. However, backtracking continues, if the system reaches beyond the bottom-right square again, then the previously found solution
was not unique, and the algorithm terminates. Once the system has backtracked back to the start, if a single solution was found then it is a “unique solution” (success case), otherwise if there are no solutions then the Sudoku is unsolvable.

By the way, Internet Explorer really isn’t that fast; I appreciate that is well-known to anyone who’s ever read Google Chrome marketing material! But the Sudoku here takes 708ms to solve on IE8, 334ms on Firefox 3.6, about 100ms on Safari 4 and Opera 10, and 49ms on Chrome 4, on this 2.3GHz/core Intel Windows Vista computer.

Update: Alas there were Sudokus which took too long for the original synchronous approach (i.e. user clicks, do calculation, update the screen with the results). So now, in the case the calculation takes too long, a “worker” is spawned, which does the calculation, and the spinner is displayed in that case.

Download the source here.

My mate came across the following HTML

adrian — Fri, 30 Apr 2010 19:03:32 +0000

/etc/cron.daily scripts may not have dots in their name

adrian — Sun, 18 Apr 2010 16:20:01 +0000

I just had the problem I’d placed a script into /etc/cron.daily on my Debian Lenny system but it wasn’t getting run (or at least it didn’t seem so).

Two things I learned:

(1) Execute the following to see which scripts would get run:

run-parts --test /etc/cron.daily

(2) In my case, the reason the script was never executed was that I called it “xyz.sh” as it was a shell script; the dot in the filename was the problem. I removed the dot and now it runs.

“me” vs “you” in user-interface dialogs

adrian — Mon, 15 Feb 2010 12:20:13 +0000

Computers, if they are addressing the user, should address the user as “you”, not as “me”.

Computers need, from time to time, to address the user, for example “You have updated your setting successfully”.

Some programs use the word “you” to address the user, some use the word “me” on the grounds that the user is reading it, and to them, they are “me”.

However, using “me” to address someone is ridiculous! That’s as logical as a human using the word “I” or “me” to refer to the recipient of a piece of communication, “hey, do I fancy going to the pub?” on the grounds that, to the recipient, they are “I”.

“me” is the source of communication, “you” is the destination of communication; if a computer is communicating, the user is the recipient of the communication so should be called “you”.

Specifically Gmail is the main culprit for this in my life, it lists conversations between “me, Joe”; having “you, Joe” or “Adrian, Joe” would be much better! Gmail even, on the same screen that it lists conversations between “me” and other people, says “You are using 25% of your 7000MB”, so it’s not even consistent!