Mistakes I have made building web applications

March 2010

This blog is a write-up of the talk that I gave at dev8d in 2010.

When I was thinking about what to talk about at dev8d, I started to think about what I had actually learned since I'd started doing development. It's easy to list all the technical skills you've acquired and so on, but I realised looking back over old projects and cringing a lot, that one of the main things I've learned is to gradually eliminate mistakes that I've made. So this is going to be a list of some of the things that I've learned not to do. Coming from real life, it's both slightly esoteric and rather personal. So for example I learned about web security pretty early on in my web dev career and I don't think ever not used version control as a professional programmer, but I'm sure both would come high on a list of common web dev mistakes.

To give you a bit of context, I'm one of a team of four developers in a university department of around 100 people. Most of my projects are research projects but I've also worked on internal projects. Our projects are small with just one or two developers and we do absolutely everything from server and database administration through to user interface design, writing research papers and a good chunk of what might often be a project manager's job.

1. Not thinking about character encoding right from the start

If you want to make me break out in a cold sweat, mention character encoding to me. Normally I'm a fan of getting working software out there as quickly as possible and then adding functionality incrementally, but this is one place where it will come back to bite you if you don't spend just a tiny bit of time upfront. If you're a web developer, it's your job to understand character encodings and know how they affect your application.When I've psychologically recovered, I may even write something up on the topic. In the meantime, if nothing else, read this article by Joel Spolsky.

2. Not establishing upfront which browsers to support

You need to decide at the start of the project which browsers you are supporting. I include support for mobile devices here. If you don't have an established policy, then you'll end up having to support Netscape 0.3. I used to include browser resolutions here too, though that's become less of an issue. I've never met a non-technical person with opinions on which browsers to support, so let me give you a hint: if IE6 isn't a default browser at your institution, make certain it's not on the list.

3. Making bad choices of third-party code (and not using third-party code when I should have)

If I look back on old projects, an awful lot of the pain I have experienced has been associated with bad choices of 3rd party code, libraries and frameworks. However, this is a hard topic to give advice on since until you use something in anger, it's really hard to tell how good it is, by which point it's often too late. The most dangerous examples tend to be code with just about enough bugs or niggles to cause you lots of headaches, but not quite enough to justify switching to something else. All I can say is that when you start using other people's code, you need to put it throughs its paces quickly and be ruthless at the start about chaning your mind if there's the slightest hint it's not going to be right for you. Along similiar lines, you can waste a lot of time reinventing the wheel. In this day and age, you should have picked your favourite MVC framework. You shouldn't be writing your own search code and you should be using one of the great javascript libraries out there. Likewise for lots of other things. 

4. Underestimating the time for legal and acquiring domain names

Getting the terms and conditions and privacy policies for a site sorted can take a lot longer that you might imagine. It's not the job of lawyers to 'get' the web, and it might take quite a lot of iterations to get something both you and legal are happy with. Likewise, don't undestimate how long it can take to get a domain name sorted. I think from start to finish, getting a .ac.uk domain name took us about two months, including negotiating both our internal process and the JANET process. If a friend hadn't done me a big favour fast-tracking the process internally and giving me some useful tips, it might have taken much longer than that.

5. Not dotting i’s and crossing t’s

It's a bit mundane, but looking back on past projects, it really does pay not to be lazy. It's very tempting when you've got an exciting new feature working, not to put the time into proper error handling, form validation, handling edge cases properly and testing. But it's doing this that separates the  casual person who can get something working on the web from the good programmer. You need to take pride in your work and be constantly trying to make the best software that you possibly can.

6. Not clarifying expectations about admin interfaces and statistics

These are the two areas where I get the most what I call 'Can you just ...?' requests. I'm not sure about the psychology behind it, but there's a tendency for people to not think that changes to these areas require work or that an admin interface should let change absolutely anything on the site. So set expectations here early and get clarification. Statistics tends to be a big issue with research projects where people want data for evaluation or other research purposes and you often need to know early on what exactly you need to be recording even if there isn't a nice interface to view the data. My other big tip here is not leaving off the Google Analytics code after a major site redesign...

7. Asking permission to spend time on security, accessibility, backups and refactoring

There are some things that non-technical people just don't have the knowledge to make sensible judgements on, or will be tempted not to take seriously enough because they either aren't going to be the person feeling the consequences or haven't worked on enough web projects to make sensible judgements.  Part of your job to make the decisions on these. They are an overhead - you generally don't have anything to show for the work you've done, but you still need to just get on with them. At the same time, it's important to keep a level of visible progress on projects, to maintain trust, so you need to figure out how to fit these things in with 'visible progress' items.

8. Not checking the colour contrast of graphic design work

This is a little bit of a random mistake but it's something that's caused me quite a bit of a pain. For accessibility reasons you need to make sure that the colour contrast on websites is sufficient. If you're hiring a graphic designer for your site, most accessibility mistakes they make, you can fix quite easily yourself, but not this. As a general rule, unless you're hiring a designer who has made accessibility their speciality, you should be doing at least a minimal check of the acccessibility of their work yourself. There are great designers out there but there also ones who will promise a WCAG-compliant site and not deliver one.

9. Continually putting off usability testing

I don't know what it is about usability testing but it is so easy to put it off again and again. I guess it is 'important but not urgent'. It's one of those areas where the perfect is the enemy of the good, and especially in a university environment, it's easy to think that it needs to be treated as scientifically valid research, whereas what you're actually trying to do most of the time is look for the blindingly obvious ways to improve your site that you've missed because you're too close to it. For a side project, I was working on, we managed to do basic usability testing and get lots of really useful changes to make in less than 24 hours from deciding to do it, using a voice recorder and colleagues kindly volunteering their lunchbreak.

10. Underestimating the problem of spam

It's easy to imagine that spam doesn't exist or that captchas are all you need. Most of the spam on our site has clearly been generated by humans and it's embarrassing to have spam on prominent parts of the site. We've used Mollom which is probably great for blog comments but has given us too many false positives for the type of content we get on our site. Bottom line is that you need to decide how you're going to deal with spam before it becomes too big a problem. It's a frustrating thing to have to spend time, but it's part of making your site work well. 

11. Not protecting my programming time fiercely enough

Onto meta stuff now. If you're a developer, you don't need me to tell you that your productively isn't linearly related to the time you spend programming. You'll get a lot more done in a long stretch of uninterrupted time than in the same total duration divided into smaller intervals. There are two main threats to your programming time: interesting things and meetings.

Interesting things includes things like Twitter and e-mail and Hacker News. You absolutely need to turn off Twitter when you are coding. If currently have Twitter on when you are coding, you can think of this as a postiive thing: it means that you can become a better developer overnight just by turning off Twitter. If you work at a university like I do, there are also always lots of interesting seminars and other events going on. Sometimes you just need to be ruthless and not go to these even though they may sound fascinating.

The other problem is meetings. If you haven't read Paul Graham's article Maker's Schedule, Manager's Schedule then do. The way I've found to deal with meeting is to block out three days a week in my calendar for coding, and put all my meetings in the other two days. This means those days tend to be back-to-back with meetings but it's so much better than having them scattered throughout the week. I also try and do any admin jobs on those days. Occasionally, there will be something that absolutely has to happen one of the blocked out days, but as long as you only let that happen in emergencies then it's not too much of a disaster.

 12. Expecting non-developers to appreciate the work of a developer

When I worked at a tech company, you were respected for your technical prowess. If you work as a developer as a department where there are only a few developers, then that's not the case. Most people will never understand what you do or only in a limited way (and I include in that folk who have done the odd bit of coding here and there) and you have to accept that rather than try to explain what is hard about your job. You need to be doing development because you think it is worthwhile, not to get other people's appreciation.Counter-intuitively, once you start doing this, people will start appreciating you more. A quote I like is 'It is amazing what you can accomplish if you do not care who gets the credit' by Harry S Truman. Enjoy the privilege of being able to do the job that you do making things!