Archive for the ‘uboot’ Category

2 of 30

Monday, January 27th, 2014

How would you diagnose the following bug?

  • A number of checkboxes representing user interests (Football? Music?); user can select/deselect their interests.
  • Software has worked well for years in production.
  • Suddenly intermittent reports start coming in that sometimes some checkboxes get unchecked by themselves.
  • You test live, you test on the test server, all is good.
    But the reports keep on trickling in, leading you to suspect that it isn’t just user foolishness (e.g. not understanding how to use a checkbox, which I wouldn’t normally put past users..)

This happened at Uboot around 2005.

I’ve forgotten how it came to pass that we fixed the bug. But the bug was this. Perhaps it sounds obvious once explained, but it was anything other than obvious at the time, especially given the fact it was completely unreproducable.

The software was coded in Perl5, and I hadn’t coded this particular screen myself. But, I had no reason to suspect that anything was wrong with the code, as, as I say, it had worked well live for years.

In Perl, “everything is a string”, apart from that, in fact, that’s not true at all. “everything appears to be a string; but might not be” describes the situation better. If you have a string “foo” then it’s stored as a string. If you have a string like “45″ then it’s stored as an integer internally and converted to/from a string as needed. If you have a string like “45.6″ then it’s stored as a double internally and converted to/from a string as needed. (Or it might be more complex than that, I’m not sure, perlnumber)

Supposedly this makes everything “easier” if you treat everything as a string. But I have no anger towards the junior developer who coded this screen who believed, as things appeared to be strings, things actually were stings. I mean, why wouldn’t you think that?

The checkboxes were stored in the software as a huge bit field. Perhaps this wasn’t the best representation, as that doesn’t scale (if you want to store 100 interests, you’re going to have to change the approach). But, that was the representation that had been chosen and, as I said, this had been online and worked well for years.

At some point, someone had decided to augment our 64-bit servers with some 32-bit servers. So you can imagine the rest. 2 out of 30 servers were 32-bit, 2 out of 30 clicks went to those servers, our dev server was the older 64-bit server, all our software had been developed on the old 64-bit servers. So 2 out of 30 clicks, all interests apart from 32 of then got lost.

Lesson learned (or not, as the lesson would have to be learnt by the programming language community): If things appear to be strings, make sure they actually are strings. Or, make sure it’s obvious that they’re not strings.

Had the software been written in Java, this couldn’t have happened as, independent of machine word size (32-bit or 64-bit) each Java numeric data type has a defined width, and is guaranteed to behave identically on any JVM implementation.

If you’re going to change databases, do it in one go

Tuesday, June 25th, 2013

At Uboot, there was the suggestion that we should change database technology. This was because we would save money with by moving away from Oracle to e.g. MySQL.

It was further decided, as we had 200k+ lines of code, and 100+ tables, that if a migration was to occur, it shouldn’t happen  ”all at once”, but instead one would have some parts of the system migrated with other parts of the system not yet migrated.

But the strategy of migrating databases piece by piece has the following problems:

  1. Data consistency. Initially, all important data are within the source database. It is possible to guarantee certain consistency constraints (e.g. all photos are owned by existing users). However, if e.g. photos are migrated to MySQL, while users remain in Oracle, no such consistency can be guaranteed. Inconsistency has the consequence that software must accept not only valid data, but also invalid data. It is impossible to test the software for all combinations of invalid data, and with the rule that “software which is not tested does not work”, one can deduce that the software will then not work. The set of inconsistencies will increase over time, so the set of users who experience bugs will also increase over time. Afterwards, going from an inconsistent state to a consistent state is nearly impossible.
  2. Reporting over multiple system areas. With one database, one can ask a question such as “how many photos belong to users logged in in the last month”. A “SQL join” is used. However if the tables reside in different database, no such query can be asked. One would have to export the data from one database to another, in order to ask the query. Or export all data into a centralized data warehouse. Or write software to do what a single line of “join” would do. All this effort is saved, by using a single database.
  3. Point-in-time recovery. With one system, if the database crashes or an error is made (e.g. “drop table account” is executed by an employee), a “point-in-time recovery” can be performed. Data is restored to its consistent state as it was at a certain point in time. If one uses more than one database, a point-in-time recovery will be more difficult. For example if the databases are out-of-sync by 1 second, then some photos will exist for which there are no users, or vice versa, and data consistency problems exist as described in point 2.
  4. Spending money on migration; no saving in license fees. If a migration is done, time/money is spent. Not only in doing the migration, but fixing performance problems after the system is done. If money can be saved on the Oracle license, perhaps this time is worth it, however if one is using multiple databases then one is still paying for Oracle.

The solution is either to change databases in one go, or simply don’t change databases. You want to have your data in one place, at all times.

SMS trailers at Uboot

Tuesday, January 8th, 2013

We developed a cool algorithm to add informational trailer texts to SMS at Uboot. It was online from mid 2000 to mid 2011 (when Uboot stopped doing SMS). I developed this algorithm together with my colleague Mike Weinzettl. So that it doesn’t get lost forever, here is a description of it :-)

The general situation was this:

  • The users may send SMS from Uboot to phones
  • SMS have fixed length (e.g. single part SMS contains 160 characters)
  • The user may well type fewer than the maximum number of characters (e.g. 100 characters, resulting in a 1-part SMS with a maximum length of 160 characters)
  • We can use this space to promote Uboot (e.g. add “Sent by www.uboot.com” text)
  • We realized we needed a trailer system to define what text should be added to which SMS under which circumstances

Requirements:

  • Texts should be stored in a separate file/files or DB (i.e. not in the middle of the source code)
  • Different SMSs have a different number of free characters at the end (from zero characters upwards). It would be nice to put longer texts on those SMS which have more characters free.
  • Users speak different languages (English, German, ..)
  • It would be nice to put variables in the text (e.g. “Sent by USERNAME”)
  • The text depends on various other factors: Is the recipient the telephone number of a registered user?

We came up with the following solution.

  • We would have a set of candidate texts, combined with the conditions under which they may be used (e.g. recipient is known user, which language)
  • Conditions could either be a single value (e.g. “en” language), a list of acceptable values, or an indication that any value is acceptable (no condition)
  • Inspired by crontab, we used a file, one line of the file per rule.
  • Each line had a set of columns for the conditions (language,..) with values such as “de” (single value), or “en,de” (multiple values), or * (any value is acceptable)
  • The last column of the file was the text
  • The text may contain variables such as ${USER}

For example:

language  recipient-registered    text
en        yes                     Sent by ${USER} on Uboot
en        no                      Sent by ${USER} on www.uboot.com
en        yes                     Sent by Uboot
en        no                      Sent by www.uboot.com 
*         *                       Uboot
*         no                      www.uboot.com
de        yes                     Verschickt von ${USER} auf Uboot

The algorithm would then do the following:

  • Search through the file, finding all rules that matched the condition, take the texts
  • Expand variables in the texts
  • See how many characters are available in the user’s SMS
  • Throw away any texts (with variables expanded) which are longer than the available space
  • Take the longest text out of the remaining text
  • If there are more than one texts with the same length, choose one at random
  • If no text matches, simply use no trailer (e.g. if only 1 character is free, it’s unlikely a useful trailer will be defined with that length)

The advantage of this algorithm is, given two users who send an SMS with the same amount of space free, different trailers may be the longest, depending on how long their username is.

I’m proud of this algorithm :-)

Store users’ birth dates, not ages, in the database

Wednesday, January 2nd, 2013

If you store your users’ ages in the database, in one year’s time their ages will be wrong. If you store your users’ birth dates in the database, and calculate their age from that, this age will always be correct.

Some sites wish to ask the user their age and display it. It would seem simplest to just store this number in an integer field alongside the user in the database. But you can be sure that, in one year, this value will be wrong.

Instead you should ask the user their birth date. Store that in the database. Always calculate their age by seeing how many years have passed between their stored birth date and the current date. As they get older, their age will always be displayed correctly.

In case you really wish to ask the user their age only, one trick we did at uboot, was to calculate an approximate birth date. Assuming they’re half way between their birthdays, and they say they are n years old, assume their birth date was n+½ years ago (or 12n+6 months ago). Calculate the age to display as described above. On the day they enter their age, the display will be correct. One year thence it’ll be correct as well. In between it’ll be, well, an approximation. But better than displaying their entered age forever—showing them never ageing.

Standards vs Fools

Thursday, December 13th, 2012

This is another rant found in my blog drafts folder from some time in 2006, when I was working at Uboot integrating a third party SMS provider.

Standards are quite useless if fools do not adhere to them; unfortunately fools are everywhere. Therefore I conclude standards are quite useless.

Specifically I implemented a data transfer protocol based on GET requests, and as specified in RFC 3986 “Uniform Resource Identifier” section 2.5.

… the data should first be encoded as octets according to the UTF-8 character encoding [STD63]; then …. should be percent-encoded. For example, … the character LATIN CAPITAL LETTER A WITH GRAVE would be represented as “%C3%80″ ….

However, they required me to send data Latin-1 encoded, rather than UTF-8 as according to the standard. I explained to them that this was wrong, that the character set is not a choice that is theirs to make. They simply replied:

We are sending with ISO-8859-1(Latin-1).

What can I do? (1) Refuse to work with this company? (But management want me to “find a solution”.) (2) Change my code to be non-standards compliant?

Obviously what I did was the second, but it felt so wrong.

aezpskin Wörterbuch

Wednesday, February 2nd, 2011

To those who were there, this is classic stuff, to the rest of the world it will be meaningless even if I try and explain so I won’t. (And I wasn’t even there.)

The old Uboot has been replaced by a new Uboot today, and what better way to celebrate the old uboot than the re-release of the official language guide.

The original Wörterbuch mysteriously went offline, but I knew I had to somehow find it. It couldn’t die. It had to be saved. It has taken some effort to find and save; I’ve been searching for it on and off for the last few months.  In the end I had to guess URLs on the internet archive, until I found an old copy of http://www.aezpskin.com/. However this didn’t have the content, but “view source” revealed a “CMS location” which the Wayback machine had mangled to point to itself but it couldn’t serve (as it was was dyanmic). So i turned the CMS link into a non-Wayback web link, then replacing www. with sms. (prefix for old uboot) and some Javascript appeared on my screen which contained the content.

aezpskin optimiert translation
joe joker sign
sk selbst kündigung
ae aera
zp zielpunkt
in inder
bimu billa
wokness restaurant asia wok
lutschn, itness la sizilia
sg small game
sc, mure small cigarette
ssc, 90er small speed coffee
hg home going
rh raushauing (gefeuert werden)
5 minutes not exactly defined amount of time
sw small working
sws small working simulation
lks leberkässemmel
fls fleischlaberlsemmel (old), so ein scheiss (modern)
morgen jungs standardgruß (den ganzen tag anwendbar)
fuck selection absolute riesen fls (absolut scheiße)
feeling einen kater haben
riesen feeling einen mächtigen kater haben
i weiss auf einen spritzer (weinschorle) gehen
weisser spritzer
bejoen, weissen betrinken
eject erbrechen
hack eject absichtliches erbrechen
murenfabrik zigarettenautomat
selection alles ist eine selection
*mu mäßig unterwegs
possible? sg possible?
possible! sg possible!
yes yes yes alles geht klar, alles wird gut …
oh noooooooooooo alles geht schief
error, errn irgendwas geht schief
explosion riesiger error
brain error gehirn funktioniert kurzfristig nicht
sj small joe, z.b. kleines bier
bj big joe, z.b. großes bier
wiesoooooooooo alles hat sich verändert, wieso?
sk waffe dem chef drohen zu kündigen
og office going
sos small office sleeping
upjoen in der firma aufsteigen
hf erster chef
hf2, usw … nachfolger des ersten chefs
sbs small bauernschnapsen
massen sk, sk-lation das ganze team kündigt
auf der brutwiese
sg/ sg possible?, riss, i-riss aufferderung zum tischfussball
standes j (frügher auch stand sj) vom stand schräg rein ins tor
kerzerl ball joed senkrecht in die luft
bachner von der ecke – i ins tor
schwachner sarniger bachner
kuri eigentor
pralle mit druck gegen die bande
quetscherl mit druck einquetschen und hoffen …
eyejoe in die eggs, oder ins aug
horvath von hinten mit bande
hintrischer horvath
bei eich ball ist immer bei euch
sound check zu viel druck, ball joed aus tor raus
sound on richtiger druck – laut
sarne, sarnig slow spielen
druck, “oft liegts am druck” hart, gewalt
classic (mit essig) schnell anschieben und rein
essig schräg
sekkn schnell ziehen und rein
backsekkn, (professor becksekker) sekkn von hinten – hr. professor
muster druck, laut, stönen, tor
antonitsch druck, laut, stönen, kein tor
skoff schwach, leise, stöhnen, kein tor
sarbatini weiblich
david, (lockness-konzept) locker spielen mit umdrehen
“tsch” (reload, doorclosing, middlejoe) 1:1 – neubeginn, tür zu, middljoe
reload 1:1 – neubeginn
middlejoe von der mitte rein – gilt nicht
dev, pre, golden joe 7:9 – 8:9 – 9:9
esad tor durch reine willenskraft
resad aktivierter esad
walchhütter hektisch herumfuchteln und drauf
der alte trick ho speed wackler
thorsten hinterm tor im kreis vorbei
flying-konzept direkt annehmen und draufjoen
“re!” spielende – neues spiel
change request position tauschen
pe zu null
silent mode leise spielen (investoren im kasten)
brut kraft, gewalt, druck, laut
bruthochdruck (wenns am druck glegn is) siehe oben
leodolter mit tormann ins gegner tor joen
“i sekk da´n eine” jemanden eine (angesagte) sekkn machen
“schick erm” reiss ihm den ball hinein
“gspürt | g´sehn” sauber gejoet, aber auch sauber verhütet
spruchjoes und redewendness
“und i” aber sofort
“bei euch” hände weg – ned bei mir
“da kann man echt danke sagen” danke
   

NoSQL becuase SQL is too slow?

Monday, March 8th, 2010

I just read this excellent article relating to the reasoning for using NoSQL databases being performance problems.

http://thoughts.j-davis.com/2010/03/07/scalability-and-the-relational-model/

However, I wonder if it’s even reasonable to search for solutions to performance problems with the relational model; NoSQL, etc.

I think as computers get faster and faster (CPUs, SSDs, more memory, …) the set of problems which are “too slow” for a particular technology (or mindset) get fewer and fewer.

I used to have the pleasure of having to optimize systems (based on SQL, some of our solutions involved leaving the relational model); e.g. community with 4M users (uboot.com), but nowadays I have no customer where even an open-source database installed on reasonably inexpensive hardware is insufficient.

Of course, that’s just my experience, and there certainly are many problems, companies, etc. today which require solving performance problems in the database, but I assert a lot of the people proposing NoSQL as a solution to performance problems with SQL databases don’t have the performance problems in the first place; I mean not everyone is implementing Facebook, Google, …

We took uboot.com online 10 years ago

Sunday, February 21st, 2010

At approx 6:30am on Monday 21st Feb 2000 my boss, my colleague and I took the first version of Uboot online.

It certainly didn’t have all the features it needed back then, it took us one extra week to add an address book to the messaging functionality, for example. And it didn’t have any photo sharing, video sharing, blogging, or the e-commerce functions that it has now.

It’s actually amazing how fast we did develop the first version: a team of three software developers, I started at the company on 3rd Jan 2000 and completed two other projects before starting Uboot; the first commit was on 19th Jan 2000.

With a few exceptions, I have been involved with Uboot software development, sometimes more, sometimes less, since then.

The Internet: It’ll get slower before it gets faster

Thursday, January 31st, 2008

For the 3 weeks I was in the UK recently I used a UMTS modem (i.e. like a 3G phone) to surf the web and do all my work. Going round to my friend Robin’s house, who also works in IT, he does all his surfing through a cable from his phone to his computer: i.e. also UMTS.

At least in the UK, this is extremely popular. Also in Asia it makes a lot of sense; they have excellent high-speed mobile phone networks there and all ones preconceptions about the Asians having the latest handset devices: I can confirm first-hand that they’re all true.

As we all know and have been experiencing since about 2000, more and more phones are going to get more powerful and have larger screens. Full browsers will (and do) run on them. They will also be UMTS devices.

And for those people who don’t surf via UMTS, nearly everyone I know surfs at home using WLAN. A lot of offices use WLAN too. And obviously all the surfing at airports, coffee houses, hotels, conferences etc. all goes on via WLAN.

UMTS and WLAN have high bandwidth, but they have extremely high latency compared with a cable connection. That means that although the bytes flow fast once they’ve started, it takes a long time for the first byte to arrive.

I am quite proud of the fact that when I designed the “Uboot Joe” software (Windows software which ran on the user’s PC, sat on the notification area by the clock, and communicated with Uboot) I took this into account. Every action you do with the Joe is at most one client-server round trip. For example to view all the thumbnails in a folder, there is a single request from Joe to the server like “get all data in folder_id” and the return structure is a) information about the folder, b) information on all the photos within the folder, and c) all the binary JPEG data for the thumbnails of all those images. You can try using the Uboot Joe on a UMTS link, and it works faster than any website.

Contrast this design with HTML. The first response from the server contains <img src=xx> tags and only once that has been received can the browser make the further requests necessary to retrieve the images. If the first bytes of every response take a long time to arrive, then the user experiences that “long time” twice before they see the data they requested; first to get the HTML page then second to get the images.

In fact it’s worse. If a page has 50 embedded images, it doesn’t open up 50 concurrent connections to the server (for good reason). Instead it opens e.g. 4 connections. Which means that e.g. image number 5 has to wait for the “long time” of fetching image 1 to complete. (Some sites try to get around this by having lots of servers with different names e.g. img341.domain.com and distributing the images over these servers.)

And it’s even worse than that. Even if the application only does one round-trip to the server, the underlying protocols might do more round-trips, for example firstly to contact the DNS server to get the IP address for the domain name used in the URL; and then secondly to request the data from the server.

In addition to this being a problem with UMTS and WLAN, one also has to take into account that the Internet is global. When I’m in Macau accessing European servers I get a round trip of about 300ms. So if one adds three “long times” to an otherwise extremely fast request—easily done—one has added a whole second on to the time the user has to wait. And Jakob Nielsen says that after 1 second in total, users start to lose focus on what they’re doing.

So to design applications in this age, one needs to be aware of the number of serial server round-trips (i.e. the number of times you need to ask the server for something, and only once it’s been delivered, must you ask the server for something else).

For example:

  1. An HTML page which contains an external CSS file, and this CSS file contains URLs to images.
  2. Pages with many images. The browser only requests a few files from the same server at once, so again the response to image number 1 must be finished before the request for image number 3 can begin.
  3. Javascript software which does multiple serial calls to the server, e.g. “get session token for username/password” then “get info to display on page for session token”.
  4. A form which submits data to a piece of software. The software does something but instead of returning a result page, returns a redirection command to a “real” result page. Often done to allow one to hit “refresh” safely on the result page, or make the URL of the result page look nicer.

GWT is excellent in this regard. It has the ability to download lots of those small icon-size images in one request (it makes one big image on the server and chops them up again on the client) and it makes you explicitly aware of the number of server round trips by forcing you to define interfaces for client-server interactions – as opposed to some automated scheme where you write code and the framework decides when to insert client-server round-trips. (Wicket makes client-server round trips easy with AjaxLink; my fear is it might be too easy, and one might do them too often, and lose the overview of how many are happening).

Pre-caching is a good idea too. E.g. if you are a photo viewer application, with a photo shown full screen with a “next” button, it makes sense to load the image on the “next” page even before the user’s clicked on it. That download won’t interfere with the rest of the activity the user is doing, as the bandwidth is not the bottleneck, just the time between starting the download and the bytes starting to arrive at the client. (Although one can’t download too much without the user noticing, as some people pay per MB!)

But the most important point, I think, is: these days, one must test ones web applications on a high latency connection. Generally speaking, historically I have tended to develop locally (everything installed on my laptop), or I develop in an office with a network cable and high-speed Internet and a link to the data center where the test server sits—and the office is in the same country as the server. Maybe this sounds strange, but I think one should develop web applications while using a UMTS card.

New Year, New Blog

Tuesday, January 8th, 2008

I shall be blogging here henceforth. I have moved all my old articles over from my previous uboot.com blog.

The reason for the move was multiple:

  • It’s important to use the software you write, to experience its successes and limitations. I am a contractor for uboot and have been using their blog; however I am also a contractor for easyname and in December we took the hosting features online we’d been developing in 2006. I’m glad to report they work just great!
  • I wanted more control over the design. The text was small at the uboot blog and didn’t invite reading.
  • I have discovered hierarchical categories! So one can view just my software design posts for example. They are cool. Did uboot have them? If so, I never found them.
  • I wanted a facility for seeing new comments without checking all posts over all pages, and comparing the current number of comments on that post with my memory of the previous number of comments.
  • Trackback wasn’t fully working with uboot. Although I suppose in the time it took me to set up my own blog, I could have repaired the uboot feature!

It wasn’t easy easy as I had hoped to set up this blog. I imagined just FTPing over a WordPress installation and that would be it. While I don’t want to sound ungrateful to the open-source programmers who made both the blogging software, and the hosting software possible, there are a sufficient number of small problems – both in implementation and in architecture – with the internet, web servers, web protocols, and in every piece of software, as to make the seemingly simple process of installing some blogging software annoying and time-consuming. I am writing this at the end of the second day of full-time work creating this blog.

  • My plan to import my own data was to import the RSS feed from the old Blog. However, that RSS import software had three separate bugs. I have fixed these problems now in the source, and will submit them in due course.
    • The RSS importer didn’t use an XML parser, but instead regular expressions. Thus it required an <item> tag to look exactly like “<item>”; whereas the uboot RSS feed includes attributes of the item tag, i.e. <item x=”y”>. So it didn’t match and simply reported that it had “successfully” imported 0 posts.
    • Newlines in the HTML content were turned into <br> characters. My HTML content had a lot of newlines (that’s how the gmail WYSIWYG editor produces the content). These are ignored by the browser, so shouldn’t be turned into <br>s which are not ignored by the browser. I solved the problem by replacing all newlines in the HTML with spaces, before the <br> conversion happened.
    • HTML escaping was being performed needlessly on the article titles. The titles were already in HTML in the RSS file. So “&quot;” text was introduced into the user-visible titles. I know the RSS feed is correct in this regard as it renders correctly on Google Reader, Bloglines, etc. I have removed this conversion.
  • Alas I lacked sufficient knowledge of CSS so getting the style correct was a great pain. Yet I didn’t have particularly ambitious style requirements, as any viewer of the new blog can confirm.
    • One problem that took ages was the removal of some two-coloured vertical lines. Were they images? Clever CSS borders? I couldn’t find the border commands in the CSS file but also couldn’t find any image commands. Nor were they referenced from the HTML file. Finally I checked the images directory and found an image; then full-text searched all files. I found the image referenced in a <style> element at the <head> of one of the HTML files.
    • Various IE7 problems. I even had to insert a “if browser=IE” Javascript in one place.
  • All embedded image references and intra-blog links had to be changed. (I couldn’t even just leave them pointing to the old blog, as they were relative links, i.e. <img src=”/x/y.jpg”> so didn’t work after the new content had been imported.)
  • I created a “.htaccess” file to password-protect the website while it was under development. Later I deleted the files. However WordPress had written some rules into the file (without it being obvious to me) so that URLs like “/post-name” would be mapped to the correct PHP files. So after I deleted the “.htaccess” file to give everyone access, the blog no longer worked (it took me some time before I discovered that, as the URL “/” still worked; so it was not obvious which action had led to the pages stopping working)
    • Let us not forget that the syntax of “.htaccess” and “.htpasswd” files is far from obvious in the first place (But thankfully my hoster has a tool to write this files – actually I wrote that piece of software!)
  • I tested the RSS feeds from the new blog in Google Reader just at the moment when the .htaccess file was broken. Thus, Google Reader cached an non-working version of the page with 0 posts. And as Google Reader shares that cache between its users, I knew that anyone trying to subscribe to the feed would see the same thing. It’s fixed now though (by time).
  • I’m sure there were more problems but I can’t remember. I should have written them down as I was working; after all, the probability of me not writing a blog entry about the difficulties of installing the new blog software were clearly not particularly high.

So essentially had I not been deeply familiar with PHP, HTML, Javascript, .htaccess files, FTP, XML and (to an extent) CSS, I would not have made it. This is not something I would recommend for “Aunt Tillie”.

Anyway, now it’s done, and as I now maintain this software in contrast to before, I look forward to also having to fix it when it breaks randomly in the future (as inevitably software always does).