A controversial topic

Language choice is a topic filled with opinion. Most programmers have a favorite language, and most programmers have languages they hate. Programming in a given language engenders similar responses to doing plumbing repairs with a given box of tools.

If you have every tool you need right there organized where you expect it, then plumbing is a manageable task with a satisfying result. When the box of tools has nothing but a hammer and a screwdriver it's a whole different story.

So what makes a good language choice? Really the answer is whatever language gives you the best bang for your buck on the problem you are trying to solve. Many companies stick to a strict set, C/C++, Perl, and Java for example. The reasons are usually that "everyone else is doing it, and we need to be able to support our code in the future."

This is a valid concern, but I seriously question whether the final decision is reasonable. In general the goal of computing should be to produce the most powerful and reliable tools possible with the least amount of effort and resources consumed. The fact that you can quickly hire someone who has used C before does not mean that you can quickly hire someone who will rapidly fix your software maintenance problem.

In essence restricting yourself to a few "popular" languages is an attempt to avoid training costs by hiring people who are pre-trained. However, there is a big difference between understanding how to create well formed statements in a programming language, and knowing how to create well designed and implemented programs to solve problems. Training an enthusiastic programmer who knows a little about one popular language to be able to program in your special language may in fact be the highest productivity method available if the special language fits your task.

Paul Graham has an excellent article on his web page about language choice, and the reason he chose Lisp to program Yahoo! Store. He also has written excellent books on Lisp.

Generally I am a polyglot when it comes to programming languages. I am competent to varying degrees in languages both big and little: C, C++, Python, Perl, Common Lisp, Scheme, Awk, sed, lex, yacc, bourne shell, SQL, Make, regular expressions, M4, forth, Postscript, LaTeX, and a variety of others. Generally depending on what my task is I choose a language that will solve the task as quickly and efficiently as possible. When it comes to producing software for a client I generally recommend that the client NOT focus on the language, but the problem. And therein lies a major stumbling block for many companies.

The choice of language should be made based on overall goals. In fact, before starting a major project, it would be wise to create several independent proof-of-concept programs in various candidate languages. This will let you determine which one will be best for your task, and will certainly save you time over choosing a language without enough expressive power.

In fact, often your task can be broken down into separate sub-projects with different languages for each sub project. For example, if you have a task that involves processing a large amount of text data to extract names and numbers and compute averages on the numbers, then fitting a model to the averages to make a prediction (which is a classic scenario for a company doing stock market analysis) then you might choose a language like SAS to do the whole project (though I wouldn't recommend it).

Or, you might choose to process the text file with a lexical analyzer made with flex (considerably more powerful than SAS input statements) then compute the simple averages inside the analyzer, and feed the output of this pre-processor into your statistical routines in SAS or R.

Which would be best for you? It depends on who you have available to work on the project, how long you have, whether the investment in learning flex and C/C++ (which work together) would pay off in flexibility of later enhancements, and how much speed and power you need. But reflexively reaching for a given tool will never let you take advantage of more powerful tools that simplify your task.

I'll give you another example. Suppose you want to create graphs of bandwidth usage on your web site without having any manual intervention. Excel is reasonably competent at making graphs, but doing it by hand is obviously out. Microsoft has a language for automating Excel, but it isn't terribly well suited to running automatically on a web server. What would be a good language choice?

For the graphics themselves creating Postscript is a very powerful way to render graphical information. You can insert postscript into paper reports, create images for web sites (with the help of the Free Software Ghostscript) and postscript can be generated by many different graphical data analysis programs. To process the data? Well it depends on what you know, what sort of data you have, and how much processing power you have. Python, or Perl can chew through most data relatively easily, if not ultra-quickly, Awk might be a good candidate if the data was already tabular. If you had a lot of data to process quickly perhaps a custom Common Lisp or C++ program would be best, both of which compile to native code.

My point is, the plethora of languages available gives you an opportunity to save time, money, effort, and confusion by using the best tool. And sticking to one small set of languages because of maintenance reasons is only reasonable if your total costs really do increase by using the extra languages. A little training combined with flexibility of tools goes a long way toward keeping your employees happy, and productive. And there's no better way to get value out of software projects than with happy, productive, well trained employees who are empowered to make decisions about the best way to solve a problem.