james mckay dot net

I’ve been fiddling with Django a bit in my spare time recently, with a view to possibly using it for some side projects. It’s the Python web application framework, similar in many respects to Ruby on Rails but with a distinctive flavour of its own.

It looks fairly promising. I particularly like its built in administration interface — it’s better than Rails’s scaffolding, giving you an integrated interface to pretty much all the models that are mapped to the database straight out of the box. It also seems more flexible than Rails — there are a few decisions that it leaves up to you, such as which JavaScript framework to use, or how exactly to structure your application.

I first started learning Python about three years ago when I used it to write some maintenance scripts for the Kingdom Faith podcasts, and since then I’ve used it for various random odds and ends (our daily build script is written in Python for instance) but nothing majorly extensive. As far as scripting languages go it has a lot to offer — it is relatively fast, has pretty good Unicode support, and fully supports procedural, object oriented, aspect oriented and functional programming paradigms, though it could do with better support for multi-statement lambdas like in JavaScript, Ruby and C#. I also like the way it forces you to write code that is clean and easy to read and follow, by making indentation syntactically significant and by making package and module names follow filesystem names. It is also fairly well documented, though the index in the .chm help file on Windows is a bit quirky at times.

Another good thing about Python is that it is very much an all-rounder: like .net or Java, you can write web applications, console applications, services, GUI applications, maintenance scripts or whatever else takes your fancy with it. This is in contrast to PHP and Ruby, which tend to be dominated to a much greater extent by web development, though they are capable of being used for other things. Python also has fairly mature support in the .net ecosystem — IronPython is the most mature of the DLR languages, so integrating Django with .net framework code is a real possibility. It also seems to have a much smarter contingent of users on average than either .net, Java or PHP. I think this is because it is only infrequently the first programming language that people learn, and most Python developers already have quite a bit of experience with two or even three other languages.

Comment on this »

Posted at 00:48 in Programming

Okay, folks, here’s a little exercise for those of you who think that closures are a pointless, computer-science-y concept of little or no relevance to real-world programming. This is a very practical snippet of code that I had to implement this afternoon, in PHP.

You have to write a function that takes two parameters: a template string containing placeholders such as [[foo]] and [[bar]], and a hashtable containing the values that are to be substituted into the placeholders, and returns a string carrying out the substitution. Your exercise is to write such a function in as few lines as possible.

In JavaScript, you can take advantage of the fact that anonymous functions have access to the arguments passed to the function in which they are declared, to produce a very elegant solution:

function do_template(template, values) {
    return template.replace(/\[\[(.*?)\]\]/g, 
        function(key) { return values[key.slice(2, -2)]; }
    );
}

In PHP, unfortunately, it is nowhere near as straightforward — while you can create functions on the fly using the create_function method, they don’t have access to the scope in which they were created, so I couldn’t use that particular trick here. The result? Twice as many lines of code to achieve the same result:

function do_template_substitute($part) {
    global $tmp_values;
    return $tmp_values[$part[1]];
}

function do_template($template, $values) {
    global $tmp_values;
    $tmp_values = $values;
    return preg_replace_callback('/\[\[(.*?)\]\]/',
        'do_template_substitute', $template);
}

Oh well, I guess PHP is a better language if you think that productivity can be measured in lines of code per day…

Comment on this »

Posted at 18:07 in Programming

It’s been about thirteen years now since I was last this far north at this time of year. Dad always used to tell us that it never gets properly dark at midsummer in the north of Scotland, but since I’ve spent nearly all my life in England, and we normally only head this way in August, I’d never realised just how not properly dark it doesn’t get, even though it is nine degrees south of the Arctic Circle.

This photograph, taken in Alford, Aberdeenshire just after 1am this morning, should give you some idea though. It was the point in the night when it gets darkest, and as you can see there is still quite a bit of light in the northern sky:

Technical details for the photo-geeks among you: f/2.8, two second exposure, ISO 80 film speed on a Canon PowerShot A720 IS digital camera. This is the same scene taken just over an hour earlier with the same settings:

Comment on this »

Posted at 11:42 in Miscellany

I’ve noticed recently that some websites have a very elegant solution to the problem of login recovery. If you forget your password, rather than sending you an e-mail with either your existing password or a new one, they send you a link that you can click on, which takes you straight to a page that logs you in automatically and allows you to choose a new password.

This works particularly well because it fixes the problems of both the “password reset” and “password reminder” approaches. Password reminders are bad because they require you to store the users’ passwords in plain text in the database, but password resets are also bad because they are completely user-unfriendly.

Not long ago we deployed a website for a client that used the ASP.NET membership provider for authentication and generating passwords. Unfortunately, we had to change it, because the ASP.NET membership provider generates seriously ugly passwords that look like “aFi$#3-Il1=+2x\{zZ14^” or something, prompting at least one user to send in an e-mail that said this:

I tried starting again from scratch and this time I was assigned a 21-character (!) password - the sort of thing you would expect to use if you were trying to get into Fort Knox … I find your site definitely “user-unfriendly”. What can I do?

This is why some teams settle for password reminders, even though they may be aware of the security risks. It’s also one thing that I dislike about the ASP.NET membership provider.

The login link approach gives you the best of both worlds and offers additional advantages on top of each. It bypasses both the login page and the process of navigating to the page that lets you change your password (which many users find confusing), making it much more user friendly than either. Certainly you won’t be asking your users to faff about copying and pasting “aFi$#3-Il1=+2x\{zZ14^” from their e-mail client to the login page. Furthermore, because your password is not reset until you actually change it, your old one will continue to work if you manage to dig it out in the meantime. And from a security point of view, you can still store passwords as a salted hash in the database.

Comment on this »

Posted at 14:00 in Security

If I can give one single piece of advice to ASP.NET developers anywhere, it will be this:

Learn another web development environment.

I really can not emphasise this strongly enough. From what I’ve observed, developers who only work with ASP.NET seem to have quite a bit of difficulty thinking outside of the Microsoft box. I am frequently confronted with indiscriminate and even inappropriate use of aspects of the .NET framework that don’t scale, such as DataSets, view state, or drag-and-drop programming. There’s nothing wrong with all these per se, but one of the most important things you need to know about how to use them is when not to use them. When all you have is a hammer, everything starts to look like a nail.

The ASP.NET Web Forms model in particular was originally designed to make web development look like Windows development, and ease the transition for VB6 developers from programming for rich Windows clients to the web. The result of this is that it has made the easy aspects of web development almost brain dead, while introducing a horrendously leaky abstraction layer that makes the hard things even harder, with masses of gotchas and pitfalls to trip you up if you venture outside it.

Languages such as PHP, Ruby on Rails or Python don’t have the same leaky abstractions, so developers tend to not only program “closer to the metal” but to think closer to the metal as well. This is why most of the cool sites, with stunning Ajax effects, tend to be written in these languages and target these platforms, while ASP.NET is largely languishing in the enterprisey world of Dilbert-esque cubicle farms.

I recommend you choose your alternative carefully, however. Rails and Python are the best choices. They will teach you patterns, practices, conventions, O/R mapping, MVC, and all round agile and pragmatic programming, and they tend to be taken up by smart and experienced developers who know what they’re doing. I have mixed feelings about Java: while you can learn a lot from it, like .NET it is very enterprisey, and at a time when everyone is getting excited about dynamic languages, Java is heading in completely the opposite direction. And I certainly don’t recommend PHP as a learning exercise: it is a beginners’ language — and a mind-bogglingly badly designed one at that — and while PHP guys are generally pretty enthusiastic and some of them are quite smart, and there are some decent PHP frameworks such as CakePHP and Symfony, the overwhelming majority of the PHP community simply don’t have what it takes to be programmers. Having said that, you need to know it, simply because it’s so pervasive.

You should also learn Linux if you can. It will teach you about modular design and the value of scripting everything that can be scripted. This is right at the heart of why Unix is Unix: a large part of its philosophy involves chaining text-based programs where the output of one can be passed as the input to another, to produce some fairly powerful command-based functionality, and scripting repetitive tasks so that their outcomes can be reliably reproduced. These are philosophies that seem largely lost in the world of Windows, which relies much more heavily on the visual, drag, drop and click approach of dialog boxes and wizards, even though they are every bit as essential if you want to have robust procedures and practices in place.

And whichever platform you take on board, you simply must familiarise yourself thoroughly with CSS, DHTML, JavaScript and Ajax, and at least one JavaScript framework such as Prototype or jQuery.

Personally, I still think that ASP.NET is technically the best platform on which to develop scalable, high performance, reliable web applications. However, in order to make the most of it, you need to have a good feel for what approaches you can import and learn from other platforms. Otherwise you will be stuck with the limitations and leaky abstractions of Web Forms.

Comment on this »

Posted at 14:00 in Programming

If you’ve ever had anything to do with modern church music, chances are you’ll have come across an organisation called Christian Copyright Licensing International. Their website has the strap line “encouraging the spirit of worship” and the idea is that rather than paying royalties to individual songwriters and their agents, you just pay one licence fee and that lets you sing whatever you like as often as you like in your church for a whole year. It helps with administration and makes it easier for your church to operate in righteousness, so it saves some time and hassle, though maybe not money. It’s a vast improvement over what we had in the early 80s with songbooks like this one that had a dozen or so entries that said “This song has been omitted for copyright reasons.”

However, it only covers church services, so if you are organising evangelistic events, or conventions like Faith Camp, or making your own worship album, or streaming your meetings live over the Internet, or making a mashup for something or other, or even playing tracks from your favourite Christian albums in a coffee shop, you need to go through the rigmarole of getting whatever other additional licences you need. And of course, all this costs more in terms of both money and time, and what might otherwise only take a couple of days can end up taking several weeks or even months while you’re waiting for permission to come through — if it comes through at all.

Now compare this “Christian” approach to copyright with the concepts that developers and geeks have come up with. I am talking, of course, about open source and Creative Commons.

If you’ve never heard of Creative Commons, you may want to take a look at this video, which explains it very simply and clearly:

The idea is for copyright owners to allow greater freedom and flexibility in what is done with their own intellectual property. Take my blog for example. I could put a notice on it saying you’re not allowed to copy it without paying me a fat fee, period, but I have deliberately chosen not to do so. Instead, I’ve released it under a licence that lets you reproduce it wherever you like as long as you aren’t doing so for profit, you acknowledge me as the original author, and if you make a derivative work, you grant others the same rights. You don’t even have to ask me — though it would of course be nice to know. The Creative Commons website allows you to choose a licence tailored to your needs from several different options.

The entire concept could have been lifted straight out of the New Testament, yet Christianity has had little involvement in it. It is a practical outworking of Jesus’ words, “Freely you have received, freely give” — indeed, in recent years, Bram Cohen, who is pretty much a poster child of the whole free content movement, made “Give and ye shall receive” the slogan for Bittorrent. It is a slight rewording of Luke 6:38.

Or what about Paul’s words in 2 Corinthians 2:17? “Unlike so many, we do not peddle the word of God for profit. On the contrary, in Christ we speak before God with sincerity, like men sent from God.”

So where on earth is the Body of Christ in all of this? Why are we dragging our heels when we should be forging ahead?

Worship leaders, church musicians and Christian authors have a lot in common with software developers such as myself. We tend to be very creative individuals, and what we do is often very much a labour of love. We write songs, books, blogs or computer code even if we’re not getting paid for it, and while it is nice to earn something from it, that is only a secondary consideration.

Yet while there are some people producing resources such as books, Bible studies and worship songs who have taken the concept of Creative Commons on board, they are very much on the fringes. Most, if not all, widely used Christian resources — including most modern translations of the Bible and nearly all songs that have a circulation beyond the songwriter’s home church — are only made available under restrictive commercial licences.

Is this encouraging the spirit of worship, or the spirit of mammon?

I would love to see some notable Christian songwriters distributing their compositions under licences similar to Creative Commons. I would love to see major ministries jumping on board, open sourcing their Bible study resources, and actively encouraging others to do the same.

I simply can’t accept the excuses that “it can’t be done” or “it’s impractical” or “worship leaders have to make money somehow.” The whole open source movement blows these claims completely out of the water. Some open source software packages have taken far longer to write than all the time that Graham Kendrick, Martin Smith, Tim Hughes, Matt Redman and the entire Hillsongs crowd have spent on all their songs put together — yet they are still made available for free, despite being mature and stable enough to power business critical servers. If software developers can do it, why can’t the Church?

Comment on this »

Posted at 14:00 in Miscellany

Now when I saw what this guy had to say about Colemak, my initial reaction was that he was being a jerk. Four days is nowhere near enough time to come to a reasonable conclusion about whether or not you’re going to get anywhere with an alternative keyboard layout, as even the most diehard fanboy would admit. Colemak actually has a lot going for it — it is easy to learn, and well supported by a vibrant online community, which comes in handy when you’re doing something as off-beat as using a different computer keyboard layout to everyone else.

But you can’t say the same thing about someone who draws exactly the same conclusions after having been at it for several hours a day for four months — by that time you should certainly be able to tell whether it’s going somewhere or whether you’re wasting your time. And in the past week or two, I have done exactly that.

My switch back to qwerty was partly prompted by our recent recruitment drive — as part of the interview process I’ll be wanting to do a little pair programming exercise with potential developers, and this is the kind of situation where an alternative keyboard layout would get in the way. However, much more significantly: I have found that Colemak has failed to meet my expectations.

My top Colemak speed of 71 words per minute may sound pretty impressive, but when you consider that my top qwerty speed on the same test was 90, the picture looks quite different. My typical results for Colemak have stuck stubbornly in the 62-64 range without budging an inch in three months, occasionally even dropping down into the 50s.

I’m sorry, but a net speed loss of 20% must be some new meaning of the word “fast” of which I was not previously aware.

I haven’t noticed any significant difference in comfort or accuracy either. Colemak initially gives the impression of being more disciplined and comfortable, but after four months of it, I was still making just as many typos and mistakes, and when switching back to qwerty, I did not notice any difference in long term comfort whatsoever.

Psychologistst talk about something called “cognitive dissonance.” This is where you get into something at considerable personal expense, then eventually, further down the road, it begins to dawn on you that you may be barking up completely the wrong tree. At this point, what many people do is to start rationalising their decision, and even defending it vigorously — the classic attributes of fanboyism. I sometimes wonder if this is what we see to a certain extent among devotees of alternative keyboard layouts, leading to the advantages of their layouts and the disadvantages of qwerty being exaggerated. They certainly would have you believe that qwerty is a total disaster area. They love to quote statistics about how much less your fingers travel on their layouts, how much more you use the home row, and so on. Frequency usage diagrams are all very well, but to be honest, that’s just theory, and unless you can demonstrate that this translates into a clear and obvious advantage in practice, which outweighs the disadvantages involved in using a non-standard layout, these statistics become no more meaningful than lines of code as a metric of developer productivity.

There have never been any scientific studies that have demonstrated significant advantage to alternative keyboard layouts, and even those that demonstrate relatively minor advantages are disputed. “The Fable of the Keys” by Liebowitz and Margolis is the well known paper here: its bottom line was that there were conflicts of interest behind wartime studies showing an advantage to Dvorak, and while it has seen one or two rebuttals from Dvorak fans, these don’t seem to have been given any serious consideration whatsoever by ergonomics researchers.

To be honest, I think this is why alternative keyboard layouts simply aren’t going to take the world by storm. Colemak is probably about as close as you’re going to get to attaining that goal, and sure, it’s easy to learn, and yes, its lively, friendly online community is fantastic, and yes, it’s maybe better than Dvorak, but its advantages are simply not sufficient to present a convincing case for its widespread adoption.

So sorry to disappoint any of you alternative keyboard fans out there. If you’re already a satisfied Colemak user, don’t let any of this put you off, of course. If you’ve found that it works for you, that’s fine — it’s just that it hasn’t worked out for me as I’d hoped.

Nothing personal…

(Update 4 June 2008: added a note on cognitive dissonance. Hat tip: Joel Spolsky and Jeff Atwood, who discuss the topic in their latest podcast on stackoverflow.com.)

Comments (21) »

Posted at 08:07 in Miscellany

I came across this article today when I was googling for a link for another blog entry. I was flabbergasted to see that it was written by someone with a PhD, appears in a professional engineering journal, and is currently linked from their home page:

Over time, there have been many attempts to define metrics that effectively measure software development productivity. Most of the ones that I have seen are amazingly complicated and very difficult to apply.

I think there is a simpler productivity metric which should be used across the industry: the total number lines of code in the organization divided by the number of people who are working on that code (including QA as well as development). For short, I will call this metric the LOC per head.

I propose that this measurement is an excellent representation of the development organization’s true productivity. If the number rises, it means that the development organization is more productive. If it decreases, it means that the organization is less productive

Ah, the old lines of code chestnut again. For some reason, managers seem to love it. The only problem is, it’s totally brain-dead. Like government targets, any formal productivity metric can and will be gamed — usually with disastrous results, as Joel Spolsky points out.

You want lines of code? Be prepared for your code base to be poisoned with endless copy and paste code and needless repetition, which, as any competent developer will tell you, is a nightmare to maintain. Or you may even end up with a joker on your team who decides to script the process and produce a million lines of code a second without even turning up at the office.

Besides, some frameworks such as Ruby on Rails or jQuery allow you to accomplish much more with far fewer lines of code. The first release of 37 Signals’ Ta-Da List — a full-blown commercial product — contained less than 600 lines of Ruby code. So does that make DHH and colleagues unproductive? Of course not! On the contrary — it makes them brilliant.

You want lots of check-ins to source control? Fine, you’ll end up with dozens of them just to correct a single spelling mistake — and as a side effect, a version history that leaves everyone totally confused as to exactly what’s been going on.

You want lots of bug fixes in the issue tracker? Expect your developers to deliberately write bugs into their code so that they can “fix” them.

You want to compensate for this by penalising bug reports? You’re asking your developers to mislead your testers about what functionality is actually in the code base so they’ll pick up on fewer bugs.

And so on, and so on.

As the old computing adage goes, garbage in, garbage out.

Comment on this »

Posted at 23:25 in Programming

Now if things carry on the way they are going, one of these days, we are probably going to get an application for our developer position from Zefram Cochrane. He’d be more than welcome — I’m sure that someone smart enough to invent the warp drive should have C# pretty much figured out by now even though he hasn’t been born yet, though I shudder to think what his penchant for loud heavy metal music would do to our score on the Joel Test.

Of course, Dr Cochrane is just trekkie fantasy, but even so, reality at the beginning of the 21st century does occasionally send us applications such as one (with no CV attached) from someone claiming “Ihave 34 year experience in asp.net c#” (sic). Given that in the absence of time travel and warp drives, no-one will have 34 years of experience of C# until 2035 at the earliest, I think we’ll wait until then before sending that guy the coding exercise we use as a screener. However, by that time, chances are that C# will be the new COBOL, having been replaced by something more esoteric.

It also sends us ones such as this e-mail the other day that was simultaneously funny, annoying and at the same time rather sad:

This Email is to introduce my company and to ask, if you can give us a chance to prove ourselves and provide our recruitment services to your company.

My name is ______, I represent a Recruitment consultancy called __________. I am attaching my company’s Terms of Business for your consideration and rates wise we are flexible like 12 - 15%.

We mainly work in IT sector e.g, (Web Developers / Designers; Software Developers, Testers, Business Analysts and Project Managers).

You have written that you wont accept calls from AGENCIES so thats why I am emailing you to try my luck.

Please consider and respond positively & if you have any questions please feel free to ask.

In other words, “I see you’ve said no agencies, so I thought I’d write to offer the services of my … agency.”

It boggles my mind to think what was going through this guy’s mind when he drafted this e-mail up. Did he think that because we aren’t taking calls from agencies that e-mails are fine? Sorry, we don’t say “no calls from agencies” — we say “Strictly NO AGENCIES please.” That means no phone calls, no e-mails, no letters, no carrier pigeons, no agencies, period.

Or does he think that because he’s called his company a “recruitment consultancy” that somehow exempts it from being an agency? Sorry, it doesn’t.

If you are a recruitment consultancy, whether you like it or not, you are an agency.

If you are a headhunter, you are an agency.

If you are enquiring on behalf of anyone other than yourself, you are an agency.

(Strictly speaking, that means that even if you are somebody’s girlfriend, calling on behalf of your better half, you are an agency, though that is admittedly probably stretching the point. Okay — strike that, if you are getting paid to enquire on behalf of anyone other than yourself, you are an agency.)

I’m not saying there’s anything wrong with agencies per se, other than that the quality of developers that they come up with can be pretty unpredictable, but as with all things such as these, we have a strict company policy in regard to these things of “don’t call us, we’ll call you.”

However, that aside, does someone who doesn’t understand that “no agencies” means “no agencies” really have the right stuff in his head to find us a competent developer? Methinks not, somehow…

Comment on this »

Posted at 14:00 in ASP.NET, Miscellany

We got an application in from a seemingly very talented web designer the other day in response to our job posting. With some pretty impressive artwork on her online portfolio, she might be a serious consideration if we were looking for someone to fulfil a role involving primarily graphic design.

However, there is just one question. We are looking for a developer, rather than a designer — so will she make the grade in that particular department?

I get the impression that the difference between web developers and web designers is somewhat lost on many people. This is probably quite understandable — the edges between the two is a rather blurry one, with a good deal of overlap, and both require a lot of creativity — and many people manage to handle both roles remarkably well. However, they involve completely different skill sets and aptitudes.

Designers tend to focus very much on the front end. They are (or at least they should be) good at art and graphic design, and if they are designing for the web, they should know HTML and CSS. They will be able to produce great WordPress themes, Flash animations and other eye candy. They most likely also know some basic PHP, MySQL and JavaScript.

The great unknown, however, is how well they can handle the more technical aspects of building a web application. Some of them are good at this, some are not so good. It is all too easy to forget that web development is software development — as a web developer, you are concerned with the much more technical aspects of the job. You need to understand database normalisation and object oriented design patterns, for starters, otherwise you will end up producing unnecessary duplication and bad code. You also need to have a firm grasp of security — at the very least you should understand topics such as SQL injection, cross site scripting and defence in depth. Then there are other aspects such as data structures, string manipulation, regular expressions, web services, scalability, caching, threading, concurrency, transactions, and so on. If any of that sounds like Klingon to you, then either you are not a developer or else you need to mug up on a few basic essentials.

Indeed, since you have to understand fairly difficult concepts such as concurrency, scalability and threading, web development can actually be harder to get right than traditional desktop development.

I sometimes wonder if web development gets such a bad reputation for the quality of code sometimes because there are a lot of people out there describing themselves as web developers when actually they are better suited to working as web designers. In order to be a good developer you need to be able to think at multiple levels of abstraction at the same time, pick up on patterns in things, and so on. Not everyone has the brain circuitry that enables them to do this.

By all accounts, a good test of this is how you handle recursion. Many people — even some computer science students — simply can’t understand it, viewing it purely as a bug that causes a stack overflow and therefore needs to be avoided. However, being able to use recursion effectively is a fundamental skill that crops up over and over again in programming. Traversing a directory tree, the nodes in a DOM document, or the page structure in a hierarchical content management system, should be second nature to all developers everywhere.

Comment on this »

Posted at 14:00 in Programming