Ruby Procs And Lambdas (And The Difference Between Them)

LambdaAs you know, I am a big believer in knowing the basics. I find it is especially valuable to go back to the fundamentals of whatever skill you're trying to master periodically, as you gain more experience. Somehow you're always able to extract something valuable from the experience due to the extra insight you've acquired along the way. Recently I've been looking more closely at some Ruby concepts that I may have glossed over when I was learning the language initially, so I will likely be writing about this in some of my upcoming posts. I thought I would begin with one of the most powerful features of Ruby – procs (and lambdas :)).

What's So Good About Procs?

You know how everything in Ruby is an object, well, as it turns out that's not quite true. Ruby blocks are not objects! We can discuss the implications of that, but let's refrain, accept it as fact and move on. So, blocks are not objects, but you can turn them into objects without too much trouble. We do this by wrapping our block in an instance of the Proc class (more on this shortly). This is really great since it turns our block into a first class function, which in turn allows Ruby to support closures and once a language has closures, you can do all sorts of interesting things like making use of various functional concepts. That however is a story for another time, before we dive deep into those areas, let us really understand how Procs work.

Procs And Lambdas In Detail

There are three ways to create a Proc object in Ruby. Well, actually there are four ways, but the last one is implicit. Here they are:

  1. Using Proc.new. This is the standard way to create any object, you simply need to pass in a block and you will get back a Proc object which will run the code in the block when you invoke its call method.
    proc_object = Proc.new {puts "I am a proc object"}
    proc_object.call
    [email protected]:~/tmp$ ruby a.rb
    I am a proc object
  2. Using the proc method in the Kernel module. Being in the Kernel module, this method is globally available. It is basically equivalent to Proc.new in Ruby 1.9, but not in Ruby 1.8. In Ruby 1.8 the proc method is equivalent to lambda. As you may have guessed there is a difference between procs and lambdas (see below), so you need to be aware of whether you're using Ruby 1.8 or 1.9. If you're using 1.9 there is a way to find out if you're dealing with a proc or a lambda. The Proc object lambda? method:
    proc_object = proc {puts "Hello from inside the proc"}
    proc_object.call
    puts "Is proc_object a lambda - #{proc_object.lambda?}"
    
    [email protected]:~/tmp$ rvm use 1.9.1
    Using ruby 1.9.1 p378
    [email protected]:~/tmp$ ruby a.rb
    Hello from inside the proc
    Is proc_object a lambda - false

    If you're in 1.8 this method will not exist at all. By the way, did you notice how I had to switch Rubies and was able to do so painlessly through the awesome power of RVM, hope you're already using it. As far as I am concerned, if you need to switch between different versions of Ruby, don't bother with the proc method, just use Proc.new and you will never have an issue.

  3. Using the Kernel lambda method. Once again, this is globally available being in the Kernel module, using this method will create a Proc object, but it will be a lambda as opposed to a proc (if you really need to know what the differences are at this point you can skip to the explanation :)).
    proc_object = lambda {puts "Hello from inside the proc"}
    proc_object.call
    puts "Is proc_object a lambda - #{proc_object.lambda?}"
    [email protected]:~/tmp$ ruby a.rb
    Hello from inside the proc
    Is proc_object a lambda - true
  4. The implicit way. When you write a method which will accept a block, there are two ways you can go about it. The first is to simply use the yield keyword inside your method. If there is a yield in a method, but there is no block you will get an error. 
    def my_method
      puts "hello method"
      yield
    end
    
    my_method {puts "hello block"}
    my_method
    [email protected]:~/tmp$ ruby a.rb
    hello method
    hello block
    hello method
    a.rb:7:in `my_method': no block given (yield) (LocalJumpError)
        from a.rb:11:in `<main>'

    The other way is to use a special parameter in your method parameter list. This special parameter can have the same name but you will need to precede it with an ampersand. The block you pass into this method will then be associated with this parameter, you will infact no longer have a block – it will be a Proc object. You will simply need to call the Proc call method to execute the code in the block you passed in. The difference is, you can now return your block (or rather the Proc object) as the return value of the method, thereby getting your hands on a Proc object without having to use any of the above three methods explicitly.

    def my_method(&my_block)
      puts "hello method"
      my_block.call
      my_block
    end
    
    block_var = my_method {puts "hello block"}
    block_var.call
    [email protected]:~/tmp$ ruby a.rb
    hello method
    hello block
    hello block

Those are all the different ways to get a Proc object in Ruby (either a proc or a lambda).

Fun And Interesting Trivia About Procs And Lambdas

If you have a method to which you pass a block, but instead of calling yield, you call Proc.new without giving it a block of its own, it will return a proc that represents the block you passed in to the method e.g.:

def some_method
  Proc.new
end

my_proc = some_method{puts "I am a happy proc"}
my_proc.call
[email protected]:~/tmp$ ruby a.rb
I am a happy proc

In Ruby 1.9, there is actually another way to create lambdas. It isn't really a different way, it is simply some syntactic sugar, but it is crazy looking, so worth knowing :). The normal way to create a lambda is using the lambda keyword, and if you want your lambda to take parameters, you simply pass them in the normal block way e.g.:

normal_lambda = lambda {|param1, param2| puts "param1: #{param1} and param2: #{param2}"}
normal_lambda.call(10, 20)
[email protected]:~/tmp$ ruby a.rb
param1: 10 and param2: 20

With the new syntax it is slightly different. We replace the lambda method with the -> method (as in minus followed by greater than sign). The parameters move out of the block, and move into some brackets right after the -> method (so they look like method parameters, but in reality are still block parameters). Everything else is the same, so our previous example would look like:

normal_lambda = ->(param1, param2) {puts "param1: #{param1} and param2: #{param2}"}
normal_lambda.call(10, 20)
[email protected]:~/tmp$ ruby a.rb
param1: 10 and param2: 20

Everything still works fine (in Ruby 1.9, don't try this in 1.8), but if you're using Netbeans, this syntax is way too crazy for it and it will barf all over itself. This syntax is an interesting curiosity, but I recommend sticking to plain old lambda for your own sanity (especially if you're using Netbeans) and that of everyone else who has to read your code later.

Procs Are First Class (Functions That Is)

First class

Once we have created our first class function (in Proc form), there are a few things we may want to do with it, such as calling it or comparing it to another Proc. As usual, Ruby makes all of this interesting. Let's look at Proc equality first. You can compare procs or lambdas to each other, using the == method (just like with any other object). Unlike many other objects though, defining exactly the same proc twice will not make them equal. For example:

string1 = "blah"
string2 = "blah"

puts "#{string1 == string2}"

proc1 = Proc.new{"blah"}
proc2 = Proc.new{"blah"}

puts "#{proc1 == proc2}"
[email protected]:~/tmp$ ruby a.rb
String are equal? - true
Procs are equal? - false

There are only two ways that two procs or lambdas will be equal to each other. One way is to define two really simple procs that have the same object in them e.g.:

proc1 = Proc.new{string1}
proc2 = Proc.new{string1}

puts "Procs are equal? - #{proc1 == proc2}"
[email protected]:~/tmp$ ruby a.rb
Procs are equal? - true

This is not really very useful. The other way that two procs will ever be equal is if they are clones of each other (one was created by using the clone or dup method on the other).

proc1 = Proc.new{|x|"blah1"*x}
proc2 = proc1.dup

puts "Procs are equal? - #{proc1 == proc2}"
[email protected]:~/tmp$ ruby a.rb
Procs are equal? - true

Let us now look at executing procs. We have already seen the call method; this is the most common way to execute a proc or lambda.

puts Proc.new{|x|"blah1"*x}.call(2)
[email protected]:~/tmp$ ruby a.rb
blah1blah1

But, there are actually two other ways. The first is to use the array syntax, this works in both Ruby 1.8 and 1.9:

puts Proc.new{|x|"blah1"*x}[2]
[email protected]:~/tmp$ ruby a.rb
blah1blah1

The other is the dot syntax, which only works in 1.9.

puts Proc.new{|x|"blah1"*x}.(2)
[email protected]:~/tmp$ ruby a.rb
blah1blah1

This syntax being 1.9 only will once again cause Netbeans to barf all over itself :). Both of those ways are nice and succinct, but I still recommend you use call; it is not that much longer and is much more readable, both for yourself and for others. The array syntax could be especially confusing, consider:

some_var[2,3]

That could refer to an array, a string, a proc – without context it's a bit too ambiguous for my tastes, stick to call.

Even More Trivia About Procs And Lambdas

You may not know this, but procs and lambdas have an arity method which allows you to find out how many arguments a Proc objects expects to receive (Method objects also have this method). For example:

my_proc = Proc.new{|x|"blah1"*x}
puts "I need #{my_proc.arity} arguments"
[email protected]:~/tmp$ ruby a.rb
I need 1 arguments

Pretty simple – the fun bit starts when the block you use to create the proc expects an arbitrary number of arguments (or requires some, but sponges up the rest using a * prefixed final parameter) e.g.:

my_proc = Proc.new{|x, *rest|"blah1 #{x} - #{rest}"}
puts "I need #{my_proc.arity} arguments"
[email protected]:~/tmp$ ruby a.rb
I need -2 arguments

What the hell? When you get a negative number for you arity, it means there is potentially an arbitrary number of arguments involved. In this case, all you can do is find out how many arguments are required (the rest will be optional). You do this by using the ~ operator on the return value of the arity method:

my_proc = Proc.new{|x, *rest|"blah1 #{x} - #{rest}"}
puts "I actually require #{~my_proc.arity} arguments, the rest are optional"
[email protected]:~/tmp$ ruby a.rb
I actually require 1 arguments, the rest are optional

I reckon that's interesting stuff to know, it won't save your life on a daily basis, but there will be that one time when knowing this will make you look like an absolute superstar :).

So What IS The Difference Between Procs And Lambdas?

It took a while, but we finally got here. Long story short, the real difference between procs and lambdas has everything to do with control flow keywords. I am talking about return, raise, break, redo, retry etc. – those control words. Let's say you have a return statement in a proc. When you call your proc, it will not only dump you out of it, but will also return from the enclosing method e.g.:

def my_method
  puts "before proc"
  my_proc = Proc.new do
    puts "inside proc"
    return
  end
  my_proc.call
  puts "after proc"
end

my_method
[email protected]:~/tmp$ ruby a.rb
before proc
inside proc

The final puts in the method, was never executed, since when we called our proc, the return within it dumped us out of the method. If, however, we convert our proc to a lambda, we get the following:

def my_method
  puts "before proc"
  my_proc = lambda do
    puts "inside proc"
    return
  end
  my_proc.call
  puts "after proc"
end

my_method
[email protected]:~/tmp$ ruby a.rb
before proc
inside proc
after proc

The return within the lambda only dumps us out of the lambda itself and the enclosing method continues executing. The way control flow keywords are treated within procs and lambdas is the main difference between them. Or rather, the way that the return keyword and the break keyword are treated is the difference since most of the others are treated in exactly the same way in both procs and lambdas. Let's have a look at break. If you have a break keyword within a lambda it is treated just like a return:

def my_method
  puts "before proc"
  my_proc = lambda do
    puts "inside proc"
    break
  end
  my_proc.call
  puts "after proc"
end

my_method
[email protected]:~/tmp$ ruby a.rb
before proc
inside proc
after proc

However, if we change the lambda to a proc:

def my_method
  puts "before proc"
  my_proc = proc do
    puts "inside proc"
    break
  end
  my_proc.call
  puts "after proc"
end

my_method
[email protected]:~/tmp$ ruby a.rb
before proc
inside proc
a.rb:64:in `block in my_method': break from proc-closure (LocalJumpError)
    from a.rb:66:in `call'
    from a.rb:66:in `my_method'
    from a.rb:70:in `<main>'

We get a LocalJumpError, what happened? Well, the break keyword is used to break out of iteration, and we didn't have an iterator around our proc, so we couldn't break out of anything, so Ruby tries to punish us with an error. If we put an iterator around our proc, everything would be fine:

def my_method
  puts "before proc"
  my_proc = proc do
    puts "inside proc"
    break
  end
  [1,2,3].each {my_proc.call}
  puts "after proc"
end

my_method
[email protected]:~/tmp$ ruby a.rb
before proc
inside proc
after proc

So, how come the lambda didn't have the same error, we didn't have an iterator there either. Well, this is the other difference between procs and lambdas. Having a break keyword within a lambda will cause us to break out of the lambda itself, but within a proc we get an error if there is no iterator around the proc and we break out of the iterator if we do have one.

There you go, the difference between Ruby procs and lambdas. There are other control flow keywords (next, redo, retry, raise), but they behave the same in both procs and lambdas, that behaviour, however, is quite interesting, I urge you to have a play with it. More interesting Ruby fundamentals soon.

Images by rofi and Richard Moross

Why I Love Reading Other People’s Code And You Should Too

HateIt occurs to me, that many programmers hate reading code – c'mon admit it. Just about everyone loves writing code – writing code is fun. Reading code, on the other hand, is hard work. Not only is it hard work, it is boring, cause let's face it, any code not written by you just sucks (oh we don't say it, but we're all thinking it). Even your own code begins to look increasingly sucky mere hours after you've finished writing it and the longer you leave it the suckier it seems. So, why should you waste hours looking at other people's crappy code when you can be spending this time writing awesome code of your own? Weren't we just over this; give it a couple of hours and then come back and see if your code still looks awesome. You're never going to become a master of your craft if you don't absorb the knowledge of the masters that came before you. One way to do that is to find a master in person and get them to teach you everything they know. Is this possible – certainly, is it probable – not so much, you'd have to be extremely lucky. You don't really need luck though, we're fortunate to be in a profession where the knowledge and skill of all the masters is right there for us to absorb, embedded in the code they have written. All you have to do is read it, sure it might take you a bit longer without someone sitting there explaining it to you, but it's eminently possible. To put it in perspective, try becoming a great carpenter just by looking at a bunch of well constructed furniture.

I love reading code, I have always intuitively felt that you get a lot from it, yes it can be annoying and boring, but the payoff is well worth the effort. Consider this, if you wanted to become a great writer, would you focus exclusively on writing? You could try it, but you're not going to get far. It is an accepted fact that most great writers are also voracious readers. Before you can hope to write anything decent you need to read other great writers, absorb different styles, see what others have tried before you and feed your creative self. Your knowledge will slowly grow and eventually, your own writing will begin to exhibit some maturity, you will develop a 'feel' for it. Coding is no different, why would you expect to write anything decent if you've never read any great code? The answer is you shouldn't. Reading great code is just as important for a programmer as reading great books is for a writer (I can't take credit for this thought, it belongs to Peter Norvig and he is awesome, so take heed).

Even if all of this is unconvincing, there is one fact which is undeniable. Being good at reading code is important to your survival as a professional developer. Any non-trivial project these days will be a team effort and so there will always be large chunks of code you had no hand in which you have to work with, modify and extend. And so, code reading will likely be the most used and most useful skill you can have; better bite the bullet and get good at it – fast.

How To Read Code Like … Some Kind Of Code-Reading Guy

I can't tell you how many times I've seen programmers scroll up and down through an unfamiliar piece of code, for minutes on end, with a sour expression on their face. They would later declare the code to be unreadable crap and why waste the time anyway; we can just work around the problem somehow. I am not sure what the expectation was here, absorb the meaning of the code by osmosis, or maybe intensely stare your way to enlightenment? You don't read code by just looking at it for ages, you want to understand it and make it your own. Here are some techniques I like to use, it is not an exhaustive list, but I have found these to be particularly helpful.

Stare

  1. Try to build and run it. Often this is a simple one step process, like when you're looking at work code (as opposed to random code). However this is not always the case and you can learn a lot about the high level structure of the code from getting it to build and execute. And, on the subject of work code, you are intimately familiar with how to build your current project are you not? Builds are often complex, but there is a lot of understanding to be gleaned from knowing how the build does its thing to produce the executable bits.
  2. Don't focus on the details just yet. The first thing you want to do is get a bit of a feel for the structure and style of the code you're reading. Start having a browse and try to figure out what the various bits of the code are trying to do. This will familiarise you with the high level structure of the whole codebase as well as give you some idea of the kind of code you're dealing with (well factored, spaghetti etc.). This is the time where you want to find the entry point (whatever it happens to be, main function, servlet, controller etc.) and see how the code branches out from there. Don't spend too long on this; it's a step you can come back to at any time as you gain more familiarity with the code.
  3. Make sure you understand all the constructs. Unless you happen to be the premier expert on your programming language there are probably some things you didn't know it could do. As you're having a high level fly-through of the code, note down any constructs you may be unfamiliar with. If there are a lot of these, your next step is obvious. You're not going to get very far if you have no idea what the code is doing syntactically. Even if there are only a few constructs you're unfamiliar with, it is probably a good idea to go look them up. You're now discovering things about your language you didn't know before, I am happy to trade a few hours of code reading just for that.
  4. Now that you've got a good idea about most of the constructs, it is time to do a couple of random deep-dives. Just like in step 2, start flying though the code, but this time, pick some random functions or classes and start looking through them line by line. This is where the hard work really begins, but also where you will start getting the major pay-offs. The idea here is to really get into the mindset (the groove) of the codebase you're looking at. Once again don't spend too much time on this, but do try and deeply absorb a few meaty chunks before moving on. This is another step to which you can come back again and again with a bit more context and get more out of it every time.
  5. There were undoubtedly things in the previous step you were confused about, so this is the perfect time to go and read some tests. You will potentially have a lot less trouble following these and gain an understating of the code under test at the same time. I am constantly surprised when developers ignore a well-written and thorough test suite while trying to read and understand some code. Of course, sometimes there are no tests.
  6. No tests you say, sounds like the perfect time to write some. There are many benefits here, you're aiding your own understanding, you're improving the codebase, you're writing code while reading it, which is the best of both worlds and gives you something to do with your hands. Even if there are tests already, you can always write some more for your own benefit. Testing code often requires thinking about it a little differently and concepts that were eluding you before can become clear.
  7. Extract curious bits of code into standalone programs. I find this to be a fun exercise when reading code, even if just for a change of pace. Even if you don't understand the low level details of the code, you may have some idea of what the code is trying to do at a high level. Why not extract that particular bit of functionality into a separate program. It will make it easier to debug when you can execute a small chunk by itself, which – in turn – may allow you to take that extra step towards the understanding you've been looking for.
  8. The code is dirty and smelly? Why not refactor it. I am not suggesting you rewrite the whole codebase, but refactoring even small portions of the code can really take your understanding to the next level. Start pulling out the functionality you do understand into self-contained functions. Before you know it, the original monster function is looking manageable and you can fit it in your head. Refactoring allows you to make the code your own without having to completely rewrite it. It helps to have good tests for this, but even if you don't have that, just test as you go and only pull out functionality you're sure of. Even if the tests seem totally inadequate – learn to trust your own skill as a developer, sometimes you just need to go for it (you can always revert if you have to).
  9. If nothing seems to help, get yourself a code reading buddy. You’re probably not the only person out there who can benefit from reading this code, so go grab someone else and try reading it together. Don't get an expert though, they'll just explain it all to you at a high level and you will miss all the nuances that you can pick up by going though the code yourself. However, if nothing works, and you just don't get it, sometimes the best thing you can do is ask. Ask your co-workers or if you're reading open source code, try and find someone on the interwebs. But remember, this is the last step, not the first one.

If I was pressed for time and needed to understand some code reasonably quickly and could only pick one of the above steps, I would pick refactoring (step 8). You will not be able to get across quite as much, but the stuff you do get across you will know solid. Either way, the thing you need to keep in mind is this. If you're new to a significant codebase, you will never get across it instantly, or even quickly. It will take days, weeks and months of patient effort – just accept it. Even having an expert right there with you doesn't significantly cut the time down (this is what the last post in my teaching and learning series will be about). If however, you're patient and methodical about your code reading (and writing) you can eventually become intimately familiar with all aspects of the project and become the go-to man when it comes to the codebase. Alternatively you can avoid reading code and always be the guy looking for someone to explain something to you. I know which one I would rather be.

Seek Out Code Reading Opportunities – Don't Avoid Them

Code

We love to write new code, it's seductive cause this time we're going to get it just right. Ok, maybe not this time, but next time for sure. The truth is, you're always evolving in your craft and you're never going to get it just right. There is always value in writing new code, you get to practice and hone your skills, but there is often just as much (if not more) value in reading and playing with code written by others. You not only get some valuable tech knowledge from it, but often domain knowledge as well (after all, the code is the ultimate form of documentation) which is often more valuable still.

Even code that is written in an esoteric fashion, without following any kind of convention, can be valuable. You know the code I am talking about, it almost looks obfuscated, but it wasn't intended to be (for some reason it is often Perl code :)). Whenever I see code like that, I think of it this way. Just imagine the kinds of stuff you will learn if you can only decipher this stuff. Yeah, it's a major pain, but admit it, there is that niggling secret desire to be able to write such naturally obfuscated code yourself (no use denying it, you know it's true). Well, if you invest some time into reading code like that, you're much more likely to eventually be able to write it – doesn't mean you will write it, but you want to be ABLE to. Lastly, attitude is always paramount. If you view code reading as a chore then it will be a chore and you will avoid it, but if you choose to see it as an opportunity – good things will happen. 

Images by mafleen, Scott Ableman and LisaThumann

Closures – A Simple Explanation (Using Ruby)

ClosureThere are a great many decent developers out there who don't know what a closure is. I don't really have any concrete stats on this matter, it is simply an intuitive assessment based on experience. But, you know what – that's fair enough, considering that the most popular languages that are in use right now don't support closures (Java, C++). When the language you use day in day out doesn't support a concept, that concept is not going to be high on your agenda; infact you may not even be aware of it. And yes, I agree, good developers will know several (or perhaps many) different languages and there are plenty out there that do support closures – you're bound to stumble across one at some point. Which just goes to show that when most developers learn new programming languages, they go about it completely the wrong way and so when you hear someone say they know a dozen languages, they are likely overstating the situation by quite a margin. But, I'll leave that discussion for another time – today it is all about closures.

You probably came across closures when you were at uni, you just don't remember. Over the last 10 years or so, the curriculum of software related degrees has evolved to the point where concepts like closures are not emphasized, it is predominantly a functional concept and functional languages are out – Java is in. They were trying to make the degrees more industry-relevant and as a result there is a generation of programmers who have been professionally crippled (to some extent) through no fault of their own. You trust the education system to do right by you and there isn't much you can do when this trust is misplaced. I need go no farther than myself for an example. I did Java in the first year of my degree, C in the second. The first time we were exposed to functional programming, was during an AI subject where we had to do some Lisp along with having to learn a whole slew of other concepts. Needless to say, functional programming wasn't the emphasis in that subject; many people struggled and struggled badly. We certainly had some exposure to closures during that subject, but who had time to think about that, it was all most people could do to try and wrap their heads around a new style of programming when they had just spent two years getting used to a completely different one. By the time I found myself in industry, closures – along with most other functional concepts – were long forgotten. It took me years of self-study before I rediscovered this stuff – stuff that should have been the foundation of my education as a software developer. I once had a go at my CS degree for not teaching some things that would have made it more relevant. Were I to write a similar post these days, I would be a lot harsher – I still might.

The bitter irony is that while Java is still well and truly in, the functional style is coming back with a vengeance. Fully functional languages (Clojure) and functional hybrids (Scala) are in vogue and so the concepts that underpin them are once again relevant. But, the damage has already been done and will continue to be done – academia does not adjust itself in a hurry. You can't afford to be ignorant of functional concepts any more, even JavaScript – that Web 2.0 language that we all love to hateis a functional hybrid. And so developers scramble to get their heads around these concepts and as they scramble they are presented with stuff like this:

"In computer science, a closure is a first-class function with free variables that are bound in the lexical environment."

What the hell does that even mean! Wikipedia does go on to explain it a little bit better, but without some functional context, it is still tough going. Here is another one you commonly hear:

"A closure is a function that is said to be "closed over" it’s free variables"

Really! A 'closure' closes over, why, that makes it all much clearer. Seriously, the next question is always bound to be a plea for clarification. An explanation like that is worse than no explanation at all, although it certainly does make someone sound smart. But, when you're trying to teach someone something it is not about self-aggrandisement – we can do better. Let's give it a go.

A Concept Explained Simply

A closure is basically a function/method that has the following two properties:

  • You can pass it around like an object (to be called later)
  • It remembers the values of all the variables that were in scope when the function was created. It is then able to access those variables when it is called even though they may no longer be in scope.

Let's fill in some more details. As you may have guessed, you don't get closures for free; they must be explicitly supported by the language. In order for the language to be able to support closures, it must support first-class functions. A first class function is a function that can be treated like an object in that you can store it in collections and pass it as a parameter to other functions. As I said, the ability to be passed around is the first property of a closure.

A normal function is defined in a particular scope (i.e. in a class) and can only be called within that scope. This function has access to all the variables in the scope that it is defined, like the parameters that are passed into it as well as class variables. A closure on the other hand may be defined in one scope and be called in a completely different scope (since we can pass it around before calling it). Because of this, when a closure is created, it retains the values of all the variables that were in scope when the closure was defined. Even if the variables are generally no longer in scope when the closure is called, within the closure they still are. In other words, the closure retains knowledge of its lexical environment at the time it was defined.

Hopefully that "lexical environment" sentence above is starting to make a little bit more sense now, but I am sure it would make a lot more sense if we had an example. 

An Example In Ruby

No explanation is complete without some examples, that is usually what it takes to make things start to fall into place. We will use Ruby since it supports closures and I like it.

In Ruby, closures are supported through procs and lambdas. These constructs are very similar, but there are some subtle differences (I might do a post about that soon). Let's create a closure and see how it fulfils the two properties described above.

class SomeClass
  def initialize(value1)
    @value1 = value1
  end

  def value_printer(value2)
    lambda {puts "Value1: #{@value1}, Value2: #{value2}"}
  end
end

def caller(some_closure)
  some_closure.call
end

some_class = SomeClass.new(5)
printer = some_class.value_printer("some value")

caller(printer)

When we execute we get the following output:

[email protected]:~/tmp$ ruby closures.rb
Value1: 5, Value2: some value

As you can see, the value_printer function creates a closure, using the lambda construct, and then returns it. We then assign our closure to a variable and pass that variable to another function, which then calls our closure. This satisfies the first property of a closure – we can pass it around. Notice also that when we called our closure, we printed out "5" and "some value". Even though both the @value1 and value2 variables were both well and truly out of scope in the rest of the program when we finally called the closure; inside the closure they were still in scope as it retained the state of all the variables that were in scope when it was defined. And so, our lambda satisfies the second property also which makes it a closure. I hope the example has made things a bit clearer.

Of course, we can look into closures a little bit more deeply. For example, how do they retain the values of the variables that were in scope when the closure was defined? This must be supported by the language and there are two ways to do that.

  1. The closure will create a copy of all the variables that it needs when it is defined. The copies of the variables will therefore come along for the ride as the closure gets passed around.
  2. The closure will actually extend the lifetime of all the variables that it needs. It will not copy them, but will retain a reference to them and the variables themselves will not be eligible for garbage collection (if the language has garbage collection) while the closure is around.

If the language supports the first way, then if we create two or more closures which access the same variables, each closure will have its own distinct copy of those variables when it is called. If a language supports the second way, then all closures will reference the same variables, i.e. they will in effect be dealing with exactly the same variable. This is how Ruby does things. Here is an example:

class SomeClass
  def initialize(value1)
    @value1 = value1
  end

  def value_incrementer
    lambda {@value1 += 1}
  end

  def value_printer
    lambda {puts "value1: #{@value1}"}
  end
end

some_class = SomeClass.new(2)

incrementer = some_class.value_incrementer
printer = some_class.value_printer

(1..3).each do
  incrementer.call
  printer.call
end

This produces the following output:

[email protected]:~/tmp$ ruby closures.rb
value1: 3
value1: 4
value1: 5

We create two closures this time, one to increment a value and the other to print it out. When we then call both of the closures three times, we can see that both are operating on the same variable as the value is incremented on every iteration. If Ruby handled retaining variables for closures by copying them, we would have had the original value of 2 printed out on every iteration since our incrementer and printer closures would each have had a distinct copy of the @value1 variable.

Why Are Closures Useful

Closure

Now that we understand closures a bit better, what's the big deal about them anyway? Well, it depends on what kind of language you're using. In a functional language they are a very big deal. Functional languages are inherently stateless, but we can use closures to essentially store some state which will persist as long as our closure lives on (i.e. if the closure changes the value of a variable it will retain the new value the next time the closure is invoked). I hope it is reasonably self-evident how this can be useful. The existence of constructs such as closures along with several others, allow functional languages to be very terse in expressing logic which means you can do more with less code.

With non-functional languages things are a little murkier. There are better ways to represent state when it comes to imperative languages, so the only thing that makes closures compelling in that situation is the fact that you can use them to write terser code while still taking advantage of the imperative style. The existence of closures is partly the reason why people can often do more with less code in languages like Ruby (which support closures) than they can in languages like Java (which do not). If you have other things to add about the awesomeness of closures in any context, then do leave a comment, the more information we share the better off everyone will be.

Images by Tattooed JJ and ~Oryctes~ (Off)

Who Deserves The Credit For Software Craftsmanship and Great Design?

Common SenseHow did people design great software before OO and how did we ever manage to run a successful project before Agile came along? Those are the questions young programmers undoubtedly ask themselves at some point in their career (I know I did, early on). You hear about and read horror story after horror story from the dark ages of software development (sometimes the dark ages are years ago, other times they include the project before this one) and then you hear about how things like OO or Agile are transforming (or have transformed) the industry. The truth is, projects didn't fail due to lack of Agile or OO (or any other 'revolutionary' practice) and there were plenty of successful projects even in the bad old days (it's just that you don't tend to hear about the good ones). Don't get me wrong, I believe in what Agile stands for and I haven't even seen non-OO software in my career :), but I don't subscribe to the buzzwords, I subscribe to what these concepts stand for, the core ideals if you like. Sometimes, you need to take a step back and consider that while the moniker may be shiny and new, the concepts it tries to package are often time honoured and fundamental (or as time honoured and fundamental as you can get considering the relative youth of our industry).

I was thinking about the Unix Philosophy the other day and it struck me that pretty much all of the ideas seem really familiar. So, I went and cracked open a copy of "The Art Of Unix Programming", just to make sure I wasn't getting myself mixed up. And what do you know, it was just as I remembered, but this time, I had a bit more experience to put it all in perspective for me. Here are a couple of quotes by Doug McIlroy:

"Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new features."

Why, he is simply espousing making your software modular as well as giving a nod to separation of concerns. I dunno about you, but I keep something similar in mind when designing my classes, small, self-contained, I thought it was all OO principles :).

"Design and build software, even operating systems, to be tried early, ideally within weeks. Don't hesitate to throw away the clumsy parts and rebuild them"

Release early, release often. Refactor and leave the code better than you found it.

"Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you've finished using them"

Automate everything. Does it smell like agile in here or is it just me.

Let's keep going we don't even need to turn the page. Here is what Rob Pike had to say

"Rule 1. You can't tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you've proven that's where the bottleneck is.
 Rule 2. Measure. Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest."

Hmmm, avoid premature optimisation.

"Rule 4. Fancy algorithms are buggier than simple ones, and they're much harder to implement. Use simple algorithms as well as simple data structures."

YAGNI. Even more agile wisdom. You have to remember that this stuff is from the 70s, years before OO got any traction and decades before the agile movement was even on the horizon. And these quotes are a distillation of thought, after these people had time to mull it over; they were actually doing this stuff years before.

Eric S. Raymond, the author of "The Art Of Unix Programming", summarised the Unix philosophy in the following 17 tenets.

1. Rule of Modularity: Write simple parts connected by clean interfaces.

2. Rule of Clarity: Clarity is better than cleverness.

3. Rule of Composition: Design programs to be connected to other programs.

4. Rule of Separation: Separate policy from mechanism; separate interfaces from engines.

5. Rule of Simplicity: Design for simplicity; add complexity only where you must.

6. Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do.

7. Rule of Transparency: Design for visibility to make inspection and debugging easier.

8. Rule of Robustness: Robustness is the child of transparency and simplicity.

9. Rule of Representation: Fold knowledge into data so program logic can be stupid and robust.

10. Rule of Least Surprise: In interface design, always do the least surprising thing.

11. Rule of Silence: When a program has nothing surprising to say, it should say nothing.

12. Rule of Repair: When you must fail, fail noisily and as soon as possible.

13. Rule of Economy: Programmer time is expensive; conserve it in preference to machine time.

14. Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.

15. Rule of Optimization: Prototype before polishing. Get it working before you optimize it.

16. Rule of Diversity: Distrust all claims for "one true way".

17. Rule of Extensibility: Design for the future, because it will be here sooner than you think.

Let’s pick out some choice tidbits. How about number 4, separate interface from implementation, I think we're all well drilled in that one these days :). I am really fond of number 8, this is the stuff that most agile developers live by, robust, simple and clear code (you can take robust to mean well tested if that is your preference). Number 15 is a good one, anyone do spikes when they code – I know I do? Number 16 is my favourite, I interpret it as "if something doesn't work, stop doing it and do something else", reminds me of Scrum (or as Conan the barbarian would say it "SCrom" :)). Much of it is about keeping it simple, i.e. the KISS principle, just like many of the concepts behind agile. But of course, just because we're keeping it simple, doesn't mean we are foolish about it (it's keep it simple stupid not keep it stupid stupid), which is where number 17 comes in.

So where am I going with all of this, did the dudes who wrote Unix invent agile (or at least the underlying ideals), or software craftsmanship, or great software design? It's unlikely, after all Don Knuth said:

"Premature optimization is the root of all evil."

He said this in 1974 (the link is pdf) – but was likely thinking it for years and there were probably people before him as well. It all had to start somewhere, of course, as I said; software is a relatively young industry. But, it doesn't really matter who the first person was that came up with these ideas, the fact that the ideas continue to figure prominently in newer concepts tells me that they are just basic common sense. When programming was born as a career, smart people came along, did the first project or two and quickly twigged on to what seemed to make sense and what didn't. All the nonsense, confusion and failure was mostly from a bunch of other people trying to fit software development into moulds it wasn't suited for – refusing to realise that this was an industry fundamentally unlike any that has come before and must have its own "rules" (rather than trying to inherit some from manufacturing or random other industries that "looked" similar). The saddest thing was that these people had power and could impose their will and make their ideas stick, which is why the whole industry went to hell for quite a while (even to this day), with signs of recovery only now beginning to emerge.

So Does Agile Get Credit For Anything?

Christianity and Islam are some of the most popular religions in the world. This should surprise you, since they are much younger than many other religions (especially Islam). Do you know why these religions are so popular? Because they were the first missionary religions, the first to actively seek out new converts for their faith. Most of the religions that came before, either did not actively welcome outsiders or were actively hostile to them. These religions are a testament to the fact that if you actively try to seek out converts and don't give up, even due to major setbacks, you will slowly gain more and more followers and will eventually spread your message far and wide.

The agile movement was the beginning of the "missionary work" for these common sense software development concepts that we have now all come to know and love. Up until then, there was no focus, no organisation, and even when there was organisation, there was no attempt to seek out converts (other developers) and actively spread the message about how software should be built. The agile movement changed all that, which just goes to show that developers are just people and are willing to buy into something that seems sensible and hits close to home. But, the message itself was never revolutionary, just like the religious messages were never revolutionary. It was simply a bunch of common sense concepts that many good developers have followed for years.

What I am trying to say (besides some curious factoids and a history lesson :)), is this. Don't faddishly follow "best practices", remember rule 16 – there is no "one true way". It doesn't matter what tech you're using, it doesn't matter what the configuration of your task board is. The key is to surround yourself with good people (this really is key and deserves a separate post) and to always remain aware of the situation – then figure out what makes sense and just do that (even if what makes sense is to follow best practice or to do the exact opposite :)). Good programmers have always known the score.

Image by RobertBasil

Software As A Destination vs Software As A Journey

I am feeling a little existential and metaphorical today, so forgive me if this post has more than just a slight tinge of existential metaphor about it. I do hope it still makes some sense. 

Journey

There are two fundamental ways of looking at software development. One is all about the final product and the direct benefits you can get from it. The other is all about the lessons you learn from the process of building the software. I call them software as a destination and software as a journey. Historically speaking and even into the present day, the majority of companies that build any kind of software are 'software as a destination' companies. They want to extract some direct financial benefit from the software they build (which is fair enough), until this benefit is realised, the process of building the software is just a resource (time, money) drain. On the other hand, most individuals who build software for those companies would instinctively like to be (and in many cases ARE) 'software as a journey' people. We want to enjoy our work, we want to learn as much as we can, we want to interact with those around us, we want to build code that we can be proud of and we want to get paid. All of this will happen throughout the project rather than just at the end, and so it is all about the journey, the final success of the product is almost icing on the cake. It seems that there is a fundamental disconnect between what a company wants to get out of building software and what the developers will try to extract from the experience.

This is why the software development industry is in a sort-of screwed-up state, or at least it has been for many years (things are getting better). Companies see the destination and want to arrive there as quickly as they can, the sooner they get there the sooner they see the benefits from all that money they spend and the sooner they can start getting their money back, either directly or indirectly. Anything that stands in the way of this goal must be swept aside and this includes the developers who want to enjoy the journey and 'smell the roses' on their way to the final product. Of course, the companies hold all the cards (they employ you, pay you – which indirectly, feeds and clothes' your children) and so can usually get their way if they really want to. Developers can try to make noise about the value of paying attention to the journey, but if the company is dead set on doing things their way, that's the way things will be done, developers can either fall into line or there will be other developers to replace them. However, just like a real, physical journey, if you have your eyes set firmly on the destination and are not looking where you're putting your feet, there is a good chance you will trip over some roots, or a snake will bite you, or you won't notice that cliff which is right between you and where you're going. This is how we get failed projects.

Once the project and the company are at the bottom of the cliff, you can kinda tell that everyone is in trouble. But it is difficult to figure out exactly how you ended up where you are, you had your eyes firmly on the destination and forgot to pay attention to what was happening around you, so it is hard to pinpoint the mistakes that landed you at the bottom of a cliff. Companies will do a post-mortem, come up with some vague reasons for why things went to hell, everyone feels better that they have an explanation (leaving everyone none the wiser in reality). And so next time they go out and do the same thing all over again. All the while, the answer is staring everyone right in the face. If only they had listened to the developers who told them to pay attention to the journey and not worry so much about the destination things would have been different. Even novice programmers soon learn, from personal experience, that when we slow down and pay attention to what we are doing right now, we get a lot more out of it, we learn more and the final product is improved as a result. There are only benefits, no downsides, alright, you lose some time in the short term, but gain it all back with interest in the long term. But, at the macro level where committees make decisions, this same lesson takes years and decades to permeate the hive-mind. Not to mention the fact that seeing only the destination is much easier than constantly paying attention, and so even the occasional developer gets seduced by the 'software as a destination' mentality (it's really more than just the occasional), which sets everyone back by introducing conflicting opinion and muddying the waters unnecessarily.

There was a bunch of people who were all 'software as a journey' people who basically got sick of all the back and forth. They recognised the fact that the journey is what really matters not just at the personal level, where each individual developer is concerned, but at the team, project and company level as well. They decided to give a united voice to the message that as long as you take care and extract the maximum benefit along the way, the destination will take care of itself – you will arrive somewhere you want to be. It may not be exactly the same place where you thought you would end up, but it will be a nice place nevertheless and best of all, you will have a very good idea of precisely how you got where you are, despite not having a narrow focus on the final objective from the start. More like-minded people joined the initial bunch and the movement got more organised. Some of these people started companies of their own; others brought their message to larger organisation. And now that the 'software as a journey' message had a united voice (albeit still a comparatively small one), the success of looking at software development this way could be demonstrated, but more importantly, disseminated to the industry at large.

It is a major undertaking for a company to develop any significant piece of software. Not just from a financial perspective, but as far as airing out the dirty baggage and discovering the process inefficiencies that every significant company will have. After all, in order to put something logically into code you need to figure out exactly what that something is and how it works. This is a golden opportunity for the company to streamline and improve how their business works, figure out if some processes are unnecessary. You're building software, but you're paying attention to what you're doing not just at the technical level but at the organisational level as well. As a result, not only do you get more robust software that better fits your needs, but you also improve the day to day operations of the company; all because you were paying attention to the journey rather than focusing on the destination.

Destination

But, it is so much easier to simply look towards the goals and on paper it seems like it SHOULD be more efficient. If you're in management you want to seem more efficient and focused rather than being wishy-washy and touchy-feely, preaching self-awareness and a less-rapid pace just isn't strategic enough. And so companies create artificial deadline pressures (like end of financial cycle of some sort), to make everything neater and make the people in charge look good and it feels like for every step the agile movement takes towards running software projects better, some company somewhere, takes two steps back.

Robert A Heinlein once said:

"The supreme irony of life is that hardly anyone gets out of it alive."

Infact, he had many other quotes which I think are quite relevant to the software profession, maybe I'll do a post about that at some point. To me this quote says that it's the moment that really matters, enjoy the life you have now rather than constantly looking forward to something better. I am not sure if there is any irony to be found in software, but if you direct all your focus towards your goal without paying due attention to the nitty gritty of what you're doing every day, you're likely to not get any useable software out of it. As long as you have a reasonable idea of where you want to end up, you just need to get the details right and the bigger picture will tend to sort itself out. On the other hand, you can have the clearest possible goal in mind, but if you let the details slide, bad things will almost certainly happen.

Images by Norma Desmond and alicepopkorn

Sort Files Like A Master With The Linux Sort Command (Bash)

SortIf you do your development work in Linux, there are certain commands that you owe it to yourself to master fully. There are a number of these with the main ones being grep, find and sort. Just about everyone has at least a passing familiarity with these commands, but with most people the knowledge is superficial, they don't even realise how powerful those commands can be. So, if you really put in the effort to master them, not only will you make your own life much easier, but you will also be able to impress all you friends with your elite Linux skills when you pair with them :). I will cover grep and find (as well as other valuable commands) in subsequent posts – here we will concentrate on sort

Note: I am using bash, so your mileage might vary if you're using a different shell.

Sorting is a fundamental task when it comes to programming, if you have a decent knowledge of various sorting algorithms, their advantages and disadvantages, you will be a better software developer for it. However, often enough you just don't need to draw on this deeper knowledge. Whether you're answering an interview question about sorting or simply need to quickly sort some data in you day to day work – the Linux sort command is your friend.

The extent of most people's knowledge ends with:

sort some_file.txt

Which is fair enough, you rarely need to dig deeper, the default behaviour will usually do what you need and when it doesn't – we have Ruby or Perl, we can hack something together. Well I hope that we're somewhat more curious than your average developer :). As much is we like hacking things together, if a tool can already do all the work for us, we want to know about it. We can look at the man page for sort and discover all sorts of interesting bits, but even the man page is not really clear on the more advanced aspects of sort usage. It helps to have examples to truly grok the kinds of stuff you can do with sort, so let's have a look.

Sorting Basics

To start with we have the following file:

[email protected]:~/tmp/sort$ cat letters.txt
b
D
c
A
C
B
d
a

We'll do the most basic sort first:

[email protected]:~/tmp/sort$ sort letters.txt
a
A
b
B
c
C
d
D

Looks good, how about doing it in reverse:

[email protected]:~/tmp/sort$ sort -r letters.txt
D
d
C
c
B
b
A
a

Also easy, but what if we want to be case insensitive? Hang on a sec, according to the output it's already case insensitive. But the man page has an option for this:

-f, --ignore-case
              fold lower case to upper case characters

If sort is case insensitive by default, what is this option for. We'll that one is a bit of a gotcha, it looks like GNU sort is case insensitive by default, but the man page also contains the following:

*** WARNING *** The locale specified by the  environment  affects  sort
       order.  Set LC_ALL=C to get the traditional sort order that uses native
       byte values.

What this means that we need to set the LC_ALL environment variable to get the behaviour that we would expect from sort (i.e. capital letters before non-capitals). Let's try that:

[email protected]:~/tmp/sort$ export LC_ALL=C
[email protected]:~/tmp/sort$ sort letters.txt
A
B
C
D
a
b
c
d

That's better, and now our -f option is actually useful:

[email protected]:~/tmp/sort$ sort -f letters.txt
A
a
B
b
C
c
D
d

That looks ok, but something still seems a little funny, all the capitals appear before all the non-capitals every time. That's because the sort is not stable, but we can make it stable:

[email protected]:~/tmp/sort$ sort -f -s letters.txt
A
a
b
B
c
C
D
d

Now that's exactly what we wanted, it's case insensitive and stable, i.e. if the small letter appeared before the capital when unsorted (and the letters are the same), this order will be the same in the sorted list.

Ok, but what if we have numbers:

[email protected]:~/tmp/sort$ cat numbers.txt
5
4
12
1
3
56

A normal sort is not what we want:

[email protected]:~/tmp/sort$ sort numbers.txt
1
12
3
4
5
56

But we can fix that:

[email protected]:~/tmp/sort$ sort -n numbers.txt
1
3
4
5
12
56

And, if our lines happen to have some leading blanks:

[email protected]:~/tmp/sort$ cat blank_letters.txt
b
D
   c
A
C
    B
d
a

We can easily ignore those and still sort correctly (using the -b flag):

[email protected]:~/tmp/sort$ sort -f -s -b blank_letters.txt
A
a
b
    B
   c
C
D
d

Of course none of this actually writes the sorted output back to the file, we only get it on standard output. If we want to write it back to the file, we have to redirect the output to a new file and then replace the old file with the new file:

[email protected]:~/tmp/sort$ sort -f -s -b blank_letters2.txt > blank_letters2.sorted
[email protected]:~/tmp/sort$ cat blank_letters2.sorted
A
a
b
    B
   c
C
D
d
[email protected]:~/tmp/sort$ mv blank_letters2.sorted blank_letters2.txt
[email protected]:~/tmp/sort$ cat blank_letters2.txt
A
a
b
    B
   c
C
D
d

As an alternative to redirection, sort also has the -o option:

[email protected]:~/tmp/sort$ sort -f -s -b blank_letters2.txt -o blank_letters2.sorted
[email protected]:~/tmp/sort$ cat blank_letters2.sorted
A
a
b
    B
   c
C
D
d

Alright, this is all pretty standard stuff, let's see how we can do something a little bit more fancy.

Advanced Sort Usage

One of the most common use cases when it comes to sort is to pipe its output to uniq, in order to remove any duplicate lines, but this is often not necessary as sort has a uniq-type option built right in (-u):

[email protected]:~/tmp/sort$ cat blank_letters.txt
b
D
c
A
C
B
d
a
[email protected]:~/tmp/sort$ sort -f -s -u blank_letters.txt
A
b
c
D

It even took into account the fact that we wanted to be case insensitive. Compare that to a case-sensitive sort with a uniq option set:

[email protected]:~/tmp/sort$ sort -s -u blank_letters.txt
A
B
C
D
a
b
c
d

Nothing is removed as there are no duplicate lines.

But what if my lines are a little bit more complicated than just a single letter, what if there are multiple columns and I want to sort by one of those columns (not the first one). This is also possible:

[email protected]:~/tmp/sort$ ls -al | sort -k5
total 36
-rw-r--r-- 1 alan alan   14 May  9 00:12 numbers.txt
-rw-r--r-- 1 alan alan   16 May  9 00:00 letters.txt
-rw-r--r-- 1 alan alan   16 May  9 00:40 blank_letters.txt
-rw-r--r-- 1 alan alan   20 May  9 00:33 blank_letters2.sorted
-rw-r--r-- 1 alan alan   20 May  9 00:33 blank_letters2.txt
-rw-r--r-- 1 alan alan   20 May  9 00:37 blank_letters3.txt
-rw-r--r-- 1 alan alan   84 May  8 23:15 file1.txt
drwxr-xr-x 3 alan alan 4096 May  8 23:13 ..
drwxr-xr-x 2 alan alan 4096 May  9 00:40 .

As you can see we sorted the output of ls by the size (the 5th column). This is what the -k option is for. Basically -k tells sort to start sorting at a particular column given a particular column separator. The column separator is, by default, any blank character. So in the above example, we told sort to sort by the 5th column given the fact that the column separator is blanks.

But we don't have to be restricted by the default separator, we can specify our own using the -t option. Let's sort the first 10 lines of my /etc/passwd file (promise you won't hack my machine since I am giving it away like that :)) by the 4th column – the group id. As you know the /etc/passwd file uses the : (colon) character as the separator (for more info on the format see here). Here is the output unsorted:

[email protected]:~/tmp/sort$ cat /etc/passwd | head                 
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/bin/sh
man:x:6:12:man:/var/cache/man:/bin/sh
lp:x:7:7:lp:/var/spool/lpd:/bin/sh
mail:x:8:8:mail:/var/mail:/bin/sh
news:x:9:9:news:/var/spool/news:/bin/sh

There is a group id of 65534 in there which should appear last. Let's sort it:

[email protected]:~/tmp/sort$ cat /etc/passwd | head | sort -t: -k4 -n
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
lp:x:7:7:lp:/var/spool/lpd:/bin/sh
mail:x:8:8:mail:/var/mail:/bin/sh
news:x:9:9:news:/var/spool/news:/bin/sh
man:x:6:12:man:/var/cache/man:/bin/sh
games:x:5:60:games:/usr/games:/bin/sh
sync:x:4:65534:sync:/bin:/bin/sync

We had to do a numeric sort since we're dealing with numbers, and we specified : (colon) as the column separator. The output is sorted correctly with 65534 being on the last line. Pretty cool! But the fun doesn't stop there, we can sort by multiple columns, one after the other. Consider this list of IP addresses:

[email protected]:~/tmp/sort$ cat ips.txt
192.168.0.25
127.0.0.12
192.168.0.1
127.0.0.3
127.0.0.6
192.168.0.5

Let's sort it by the first column, so that all the addresses starting with 127 go together, and then sort it by the 4th column, to make sure that the IPs are sorted by the last column within each range.

[email protected]:~/tmp/sort$ cat ips.txt | sort -t. -k 2,2n -k 4,4n
127.0.0.3
127.0.0.6
127.0.0.12
192.168.0.1
192.168.0.5
192.168.0.25

We specified the dot as the separator. The -k 2,2n syntax has the following meaning. Do a sort by column (-k), start at the beginning of column 2 and go to the end of column 2 (2,2). The n on the end is to indicate that we want to do a numeric sort since we are dealing with numbers. That is some powerful stuff, wouldn't you agree?

Cool/Useful Stuff

There is still more we can do with the sort command. Have you ever wanted to randomize the lines in a file? It is not a common use case, but does come in handy once in a while (if only for testing purposes, sometimes). Well, the sort command has you covered here also with the -R option (that's capital R):

[email protected]:~/tmp/sort$ cat numbers.txt
5
4
12
1
3
56
[email protected]:~/tmp/sort$ cat numbers.txt | sort -R
5
4
1
3
12
56
[email protected]:~/tmp/sort$ cat numbers.txt | sort -R
3
4
1
56
5
12

We get a different order every time, which is what we would expect from randomizing the lines.

If you give sort multiple files on the command line, it will combine the contents of all the files and sort it as a whole:

[email protected]:~/tmp/sort$ sort -n numbers.txt numbers2.txt
1
1
3
4
4
5
7
8
10
12
22
23
26
56
56
68

This is really handy, but sometimes, the files you have are already sorted, you just want to merge them. Sort provides the -m options just for this purpose. The output of using sort on two files will be exactly the same whether you use -m or not, but merging should be faster:

[email protected]:~/tmp/sort$ sort -n -m numbers1.sorted numbers2.sorted
1
1
3
4
4
5
7
8
10
12
22
23
26
56
56
68

Lastly, if you just want to check if a file is sorted or not, without actually performing the sort, you have the -c option:

[email protected]:~/tmp/sort$ sort -n -c numbers.txt
sort: numbers.txt:2: disorder: 4
[email protected]:~/tmp/sort$ sort -n -c numbers1.sorted
[email protected]:~/tmp/sort$

There you go, the total awesomeness of the sort command laid bare. If you know any other handy things you can do with sort, do leave a comment. And remember – only use your new-found sort powers for good instead of evil :).

Image by SewPixie (so far behind!)

The Secret To Getting More Ideas (And Better Ones Too)

IdeaA lot of people have asked me recently how/where I get my ideas (e.g. like here) – in reference to the diverse topics that I write about. After what must have been the 10th time someone asked me that question, I thought to myself "…gee, what a good idea, why not write about that…", so perhaps I am not the best person to answer the question after all :), but I do have some thoughts, so I'll give it a whirl regardless.

Ideas are an interesting beast, the more of them you have; the more you tend to come up with. That might seem somewhat counterintuitive or even circular, "…so, ummm, to have more ideas you need to come up with more ideas? Why, that's pure diabolical genius! …” but bear with me for just a minute and I'll explain. Here is the way it goes with most people. You need to come up with an idea – business idea, article idea, book idea, conversational topic – whatever. So, you strain for a few seconds trying to have a flash of brilliance and when the flash inevitably fails to materialize, you shrug, say something self-deprecating like, "… man, I really wish I was better at thinking up ideas. If only I could have a brilliant business idea, I could become a millionaire. Nay, a billionaire! Oh well …". At this point you go back to doing whatever you were doing before and life goes back to normal – which is precisely the worst thing you could have done.

The Idea-Generating Muscle

When I first started blogging, there were a few opinions I wanted to share, I was excited. It seemed like I had hundreds of ideas, but in reality, it was more like half a dozen (the brain is good making stuff seem completely out of proportion when you're excited). Once my initial idea pool was exhausted, I found myself stuck for things to write about. Me, stuck for something to say – completely unacceptable, I have a reputation to uphold :). So, I just started writing about crazy stuff that popped into my head, all the while doing my best to pay attention to the random thoughts that I was having and examining each one for potential (to be turned into a post). This had the effect of keeping my need for ideas at the forefront of my mind and you know what, after a while my brain started to respond.

Here is how I see it. The needs that you have send signals to your brain, like the need to have a good idea about something. When your mind senses the need, it will do its best to respond, but its first try will likely be a bit of a failure. It's understandable, have you ever been brilliant at something the very first time YOU tried it? If at this point you remove the need (i.e. you give up since you couldn't get a good idea instantly) your mind will just go back to its default state (of not generating ideas). But, if the need remains, if it is persistent, your mind will keep trying to fulfil it and will likely generate another idea for you after a while (which will likely also be pretty dismal). The difference here is that you're now training your mind to give you what you require, by maintaining your need for it. Try visualising your brain as a set of muscles, there is an idea-generating muscle in there somewhere. But, like any muscle, if you don't consistently exercise it, you won't be able to get much out of it and are likely to overstrain it at the merest effort. So, keep your need for ideas constant, force yourself to come up with stuff, even if it is complete trash.  Keep sending the messages to your brain and over a period of days, weeks and months you will find yourself having more and more ideas about your topic of interest. More than that, as you keep going, your ideas WILL get better – that's a promise. Your mind will adapt, as you discard the bad ideas and adopt the good ones, by producing more ideas along the lines of what you consider "good". The more ideas you have, the better "trained" your mind becomes, the more chance it has to adapt to your style; which means you'll keep having better and better ideas (and more of them as well). It is really quite a fascinating process to observe in yourself.

What effect did it have on me? Well these days, my biggest problem, at least when it comes to my writing, is a kind of analysis paralysis where I often have several ideas that I am equally keen to write about at any given moment, it is surprisingly difficult to choose :). Of course, I do make a decision eventually – one can't procrastinate forever, but my list of "stuff/thoughts/ideas to write about" just gets longer and longer – kinda like my reading list.

Confirming The Principle

The interesting thing is that this training doesn't seem to work for ideas in general, only ideas regarding the actual need that you focus your mind on. When I developed an interest in how to conduct a good interview to find decent coders, I needed to come up with some interesting coding interview questions. And you know what, I was drawing blanks once more, it was like starting the mental training process all over again. I forced myself to come up with anything and produced some crazy ideas at first, then had some better thoughts and now (a few months later) am coming up with what I consider to be some pretty decent questions and ideas (which end up on that long list of things I want to write about :)). I've also gone through the same exercise for personal projects to practice your coding – with similar results. You start seeing the pattern after a while. Start thinking about whatever interests you; immerse yourself in it to create a personal need for ideas. Force yourself to examine everything your mind throws at you – even if it seems like nonsense – to allow your "mental-muscle" to adapt. Given enough time, good ideas will follow.

What About Business Ideas?

You know those guys who always seem to have hundreds of brilliant business ideas while you struggle to come up with even one that seems viable. Well, you can BE one of those guys; more specifically you can train your mind into making you one of those guys. You just need to persist and not give up, like most things in life that are worth striving for. The only issue is the fact that the cost of trying out a business idea (to really evaluate its worth), is potentially way too high for most people to be able to afford to do it every time. However once you have trained yourself to have some good ideas in several areas, you start to get a feel for how long it takes for your brain to start coming up with some great stuff more or less regularly. That time frame will likely be similar when it comes to ideas in other areas – like business ideas. The key is to pay attention and be a little bit more self-aware than your average bear :).

Winner

Finally, don't forget to write down your ideas when you have them, it doesn't matter how you do it, dictate them into your phone, carry a pen and paper, send yourself an SMS. Just make you sure you don't lose them; there is really no point in training yourself to have ideas if you're just going to let them flow away into the ether. Especially considering that given enough time, any one of them could be a real winner.

Images by Cayusa and sarah …

8 Types Of Software Consulting Firms – Which One Do You Work For?

ConsultantI have strong opinions regarding software consulting and consulting companies, which is probably no surprise as I seem to have a strong opinion about most things :). It's not a bad thing, makes for some lively conversation over drinks, but it is also a bit of a pain as it makes it hard to choose what to write about. I'll regale you with my serious thoughts regarding consulting in another post (don't forget to subscribe to my feed so you don't miss it) – today we'll try to maintain a bit of a lighter tone.

A little while ago I read an interesting book called "The Nomadic Developer", if you are considering going into consulting I recommend you read it (although you would appreciate it more if you've already worked in software consulting). The book has some decent advice and information for budding/existing software consultants, but the thing that really caught my eye was the humorous classification of consulting firms found at the beginning of the book. It was titled "The Seven Deadly Firms" and I thought I would share it with you for a bit of fun as well as adding one more classification of my own to bring the total up to eight. I will quote and paraphrase as necessary – here we go.

BOZO Consulting

BOZO is a place with well-intentioned people who really want to please their clients. The only downside to BOZO is it doesn't have the slightest clue about how to achieve that goal. BOZO says yes to everything: "Yes, of course we will cut the estimate." "Yes, of course, we will lower our bill rate." The idea is to get the deal at all costs. To summarize, BOZO has a sales-driven culture that lacks the ability to leverage any sort of delivery capability it accidentally hires (thought from Skorks: that would have to be one of the most buzzword driven sentences ever, how can we "leverage" that).

FEAR Consulting

Firm founded on its ability to motivate software developers by fear. Micromanagement, abuse and/or Machiavellian management techniques are the tools this firm uses to generate mundane and uninspired results for clients. The best way to describe how FEAR stays in business despite such deplorable working conditions and treatment is to reference what is known as the Stockholm Syndrome (thought from Skorks: the funny/sad thing is that any consultancy can accidentally/purposely take on aspects of FEAR Consulting in certain situation (financial/project pressures etc.)).

The Body Shop

Contracting firm that pretends to be a consultancy. Sadly, not all consultants involved have been informed of this "minor" detail. The Body Shop has zero corporate culture. No events, no infrastructure and no interaction with fellow "consultants". Working for a Body Shop you run significant risk of getting the sack after completing your first gig if there is no work immediately available (thought from Skorks: does anyone else find it funny that the body shop is an actual company, unfortunately all they do is sell skin care products, but who knows software consulting could be in their future).

CHEAP Consulting

Some firms compete on quality. CHEAP consulting isn't one of those firms. This is a great place to work as long as eating and having a roof over your head are not huge priorities for you. CHEAP Consulting firms execute entire projects using hordes of low-cost developers, rather than small numbers of good ones. Ironically, this makes their projects, after all is said and done, come in far more over budget than the firms they competed against in the first place. The main feature of CHEAP is the heavy use of the word resource. (thought from Skorks: and you know how I feel about the word resource, anyway, probably not too many of these around in the first world even cheap developers are reasonably expensive if the cost of living is high enough).

Personality Cult Consulting

The only drink that this firm has in the refrigerator is kool-aid. And if you refuse to drink it – that is, decide not to indulge in the cult of local hero worship – you might not have a future here. Critical thinking skills are not the highest priority. Frequently, a person who combines charisma with a good measure of talents finds himself able to start a consulting company that capitalizes on these traits. What starts out as a small group of like-minded people with a mission grows into a creature called PC Consulting. Discussions in such a place are rife with appeals to authority; that is, if the cult leader says something, it not only is given a lot of weight, but is treated as gospel (thought from Skorks: you know what I said about me having strong opinions, I cringe at how quickly I, and most decent developers I know, would get their ass handed to them in a place like this).

Smelzer and Melzer Accounting

Accountant

Even the owners of this place, an accounting partnership, wonder how they ended up in the "computers" business. They know they make money from it but are unsure why, and frankly, they are not sure they trust these "kids" who have a lot of scary ideas. In most cases, accounting firms are run by accountants. And accountants frequently have significant issues with the way software tends to be developed. Software development methodology is chosen, based on what is perceived as predictable and developers often become the scapegoats when the expectations of predictability fail to materialize (thought from Skorks: this one is easy, Big Four anyone?).

"Push the SKU" Consulting

"Services" arm of a product company that ends up acting as a de-facto sales force for the product, while at the same time, getting paid by the hour from the client to sell…ahem, I mean… "provide independent advice". The incentive system tends to make this not the place for development of software that works independent of a particular product. Such an organization is not always a good place for software developers who have the interest of the client at heart first and foremost (thought from Skorks: does your software do X? "Sure, sure it does, it just needs some minor configuration …").

Band Of Freelancers (My Contribution)

A bunch of friends decide to form a company to build the next great product. But of course building products takes time and you have to live in the meantime, so everyone picks up some occasional consulting work to "keep the lights on". The only problem is, occasional means pretty much permanent and it is difficult to write a product while working full time on something else. So, all the "founders" keep on consulting and no product ever materializes, so the company turns into a perpetual band-of-freelancers consulting company. The real danger is that the "founders" actually do well as consultants and are able to hire more people at which point there is a real danger of this company turning into BOZO, Body Shop or Personality Cult consulting.

There you go, humorous – yes, but also rather accurate, wouldn't you agree? Of course all those are examples of potentially dysfunctional consulting companies. There are plenty of big and small consultancies (admittedly mostly smaller) that are great places to work, treat their people well and approach their software development with skill and wisdom. The thing to be aware of is that no matter how good a consulting company is, most will exhibit at least some aspects of the above 8 at one point or another. The trick is to know whether or not, it is just a bad patch – soon corrected and forgotten – or whether it is becoming more of a permanent situation. Anyways, go check out the book if you want to get the full descriptions as well as tips for recognising if you might, infact, be working for one of these companies. If you do work for a software consultancy, how would you classify the company you work for? Does it by any chance fall into one of the 8 categories above?

On an even lighter note, is it just me or does anyone else ever misspell the word consulting as consluting, can be a little embarrassing if you do it in any kind of semi-official document. The other one is conlusting which also works but is not quite as good :).

Images by orkboi and Venn Diagram

What Every Developer Should Know About URLs

I have recently written about the value of fundamentals in software development. I am still firmly of the opinion that you need to have your fundamentals down solid, if you want to be a decent developer. However, several people made a valid point in response to that post, in that it is often difficult to know what the fundamentals actually are (be they macro or micro level). So, I thought it would be a good idea to do an ongoing series of posts on some of the things that I consider to be fundamental – this post is the first instalment.

Being a developer this day and age, it would be almost impossible for you to avoid doing some kind of web-related work at some point in your career. That means you will inevitably have to deal with URLs at one time or another. We all know what URLs are about, but there is a difference between knowing URLs like a user and knowing them like a developer should know them.

As a web developer you really have no excuse for not knowing everything there is to know about URLs, there is just not that much to them. But, I have found that even experienced developers often have some glaring holes in their knowledge of URLs. So, I thought I would do a quick tour of everything that every developer should know about URLs. Strap yourself in – this won't take long :).

The Structure Of A URL

Structure

This is easy, starts with HTTP and ends with .com right :)? Most URLs have the same general syntax, made up of the following nine parts:

<scheme>://<username>:<password>@<host>:<port>/<path>;<parameters>?<query>#<fragment>

Most URLs won't contain all of the parts. The most common components, as you undoubtedly know, are the scheme, host and path. Let's have a look at each of these in turn:

  • scheme – this basically specifies the protocol to use to access the resource addressed by the URL (e.g. http, ftp). There are a multitude of different schemes. A scheme is official if it has been registered with the IANA (like http and ftp), but there are many unofficial (not registered) schemes which are also in common use (such as sftp, or svn). The scheme must start with a letter and is separated from the rest of the URL by the first : (colon) character. That's right, the // is not part of the separator but is infact the beginning of the next part of the URL.
  • username – this along with the password, the host and the port form what's known as the authority part of the URL. Some schemes require authentication information to access a resource this is the username part of that authentication information. The username and password are very common in ftp URLs, they are less common in http URLs, but you do come across them fairly regularly.
  • password – the other part of the authentication information for a URL, it is separated from the username by another : (colon) character. The username and password will be separated from the host by an @ (at) character. You may supply just the username or both the username and password e.g.:
    ftp://[email protected]/
    ftp://some_user:[email protected]/
    

    If you don't supply the username and password and the URL you're trying to access requires one, the application you're using (e.g. browser) will supply some defaults.

  • host – as I mentioned, it is one of the components that makes up the authority part of the URL. The host can be either a domain name or an IP address, as we all should know the domain name will resolve to an IP address (via a DNS lookup) to identify the machine we're trying to access.
  • port – the last part of the authority. It basically tells us what network port a particular application on the machine we're connecting to is listening on. As we all know, for HTTP the default port is 80, if the port is omitted from an http URL, this is assumed.
  • path – is separated from the URL components preceding it by a / (slash) character. A path is a sequence of segments separated by / characters. The path basically tells us where on the server machine a resource lives. Each of the path segments can contain parameters which are separated from the segment by a ; (semi-colon) character e.g.:
    http://www.blah.com/some;param1=foo/crazy;param2=bar/path.html

    The URL above is perfectly valid, although this ability of path segments to hold parameters is almost never used (I've never seen it personally).

  • parameters – talking about parameters, these can also appear after the path but before the query string, also separated from the rest of the URL and from each other by ; characters e.g.:
    http://www.blah.com/some/crazy/path.html;param1=foo;param2=bar

    As I said, they are not very common

  • query – these on the other hand are very common as every web developer would know. This is the preferred way to send some parameters to a resource on the server. These are key=value pairs and are separated from the rest of the URL by a ? (question mark) character and are normally separated from each other by & (ampersand) characters. What you may not know is the fact that it is legal to separate them from each other by the ; (semi-colon) character as well. The following URLs are equivalent:
    http://www.blah.com/some/crazy/path.html?param1=foo&param2=bar
    http://www.blah.com/some/crazy/path.html?param1=foo;param2=bar
  • fragment – this is an optional part of the URL and is used to address a particular part of a resource. We usually see these used to link to a particular section of an html document. A fragment is separated from the rest of the URL with a # (hash) character. When requesting a resource addressed by a URL from a server, the client (i.e. browser) will usually not send the fragment to the server (at least not where HTTP is concerned). Once the client has fetched the resource, it will then use the fragment to address the relevant part.

That's it, all you need to know about the structure of a URL. From now on you no longer have any excuse for calling the fragment – "that hash link thingy to go to a particular part of the html file".

Special Characters In URLs

Special Character

There is a lot of confusion regarding which characters are safe to use in a URL and which are not, as well as how a URL should be properly encoded. Developers often try to infer this stuff from general knowledge (i.e. the / and : characters should obviously be encoded since they have special meaning in a URL). This is not necessary, you should know this stuff solid – it's simple. Here is the low down.

There are several sets of characters you need to be aware of when it comes to URLs. Firstly, the characters that have special meaning within a URL are known as reserved characters, these are:

";" | "/" | "?" | ":" | "@" | "&" | "=" | "+" | "$" | ","

What this means is that these characters are normally used in a URL as-is and are meaningful within a URL context (i.e. separate components from each other etc.). If a part of a URL (such as a query parameter), is likely to contain one of these characters, it should be escaped before being included in the URL. I have spoken about URL encoding before, check it out, we will revisit it shortly.

The second set of characters to be aware of is the unreserved set. It is made up of the following characters

"-" | "_" | "." | "!" | "~" | "*" | "'" | "(" | ")"

The characters can be included as-is in any part of the URL (note that they may not be allowed as part of a particular component of a URL). This basically means you don't need to encode/escape these characters when including them as part of a URL. You CAN escape them without changing the semantics of a URL, but it is not recommended.

The third set to be aware of is the 'unwise' set, i.e. it is unwise to use these characters as part of a URL. It is made up of the following characters

"{" | "}" | "|" | "\" | "^" | "[" | "]" | "`"

These characters are considered unwise to use in a URL because gateways are known to sometimes modify such characters, or they are used as delimiters. That doesn't mean that these characters will always be modified by a gateway, but it can happen. So, if you include these as part of a URL without escaping them, you do this at your own risk. What it really means is you should always escape these characters if a part of your URL (i.e. like a query param) is likely to contain them.

The last set of characters is the excluded set. It is made up of all ASCII control characters, the space character as well the following characters (known as delimiters)

"<" | ">" | "#" | "%" | '"'

The control characters are non-printable US-ASCII characters (i.e. hexadecimal 00-1F as well as 7F). These characters must always be escaped if they are included in a component of a URL. Some, such as # (hash) and % (percent) have special meaning within the context of a URL (they can really be considered equivalent to the reserved characters). Other characters in this set have no printable representation and therefore escaping them is the only way to represent them. The <, > and “ characters should be escaped since these characters are often used to delimit URLs in text.

To URL encode/escape a character we simply append its 2 character ASCII hexadecimal value to the % character. So, the URL encoding of a space character is %20 – we have all seen that one. The % character itself is encoded as %25.

That's all you need to know about various special characters in URLs. Of course aside from those characters, alpha-numerics are allowed and don't need to be encoded :).

A few things you have to remember. A URL should always be in its encoded form. The only time you should decode parts of the URL is when you're pulling the URL apart (for whatever reason). Each part of the URL must be encoded separately, this should be pretty obvious, you don't want to try encoding an already constructed URL, since there is no way to distinguish when reserved characters are used for their reserved purpose (they shouldn't be encoded) and when they are part of a URL component (which means they should be encoded). Lastly you should never try to double encode/decode a URL. Consider that if you encode a URL once but try to decode it twice and one of the URL components contains the % character you can destroy your URL e.g.:

http://blah.com/yadda.html?param1=abc%613

When encoded it will look like this:

http://blah.com/yadda.html?param1=abc%25613

If you try to decode it twice you will get:

  1. http://blah.com/yadda.html?param1=abc%613

    Correct

  2. http://blah.com/yadda.html?param1=abca3

    Stuffed

By the way I am not just pulling this stuff out of thin air. It is all defined in RFC 2396, you can go and check it out if you like, although it is by no means the most entertaining thing you can read, I'd like to hope my post is somewhat less dry :).

Absolute vs Relative URLs

Absolut

The last thing that every developer should know is the difference between an absolute and relative URL as well as how to turn a relative URL into its absolute form.

The first part of that is pretty easy, if a URL contains a scheme (such as http), then it can be considered an absolute URL. Relative URLs are a little bit more complicated.

A relative URL is always interpreted relative to another URL (hence the name :)), this other URL is known as the base URL. To convert a relative URL into its absolute form we firstly need to figure out the base URL, and then, depending on the syntax of our relative URL we combine it with the base to form its absolute form.

We normally see a relative URL inside an html document. In this case there are two ways to find out what the base is.

  1. The base URL may have been explicitly specified in the document using the HTML <base> tag.
  2. If no base tag is specified, then the URL of the html document in which the relative URL is found should be treated as the base.

Once we have a base URL, we can try and turn our relative URL into an absolute one. First, we need to try and break our relative URL into components (i.e. scheme, authority (host, port), path, query string, fragment). Once this is done, there are several special cases to be aware of, all of which mean that our relative URL wasn't really relative.

  • if there is no scheme, authority or path, then the relative URL is a reference to the base URL
  • if there is a scheme then the relative URL is actually an absolute URL and should be treated as such
  • if there is no scheme, but there is an authority (host, port), then our relative URL is likely a network path, we take the scheme from our base URL and append our "relative" URL to it separating the two by ://

If none of those special cases occurred then we have a real relative URL on our hands. Now we need to proceed as follows.

  • we inherit the scheme, and authority (host, port) from the base URL
  • if our relative URL begins with /, then it is an absolute path, we append it to the scheme and authority we inherited from the base using appropriate separators to get our absolute URL
  • if relative URL does not begin with / then we take the path of from base URL, discarding everything after the last / character
  • we then take our relative URL and append it to the resulting path, we now need to do a little further processing which depends on the first several characters of our relative URL
  • if there is a ./ (dot slash) anywhere in a resulting path we remove it (this means our relative URL started with ./ i.e. ./blah.html)
  • if there is a ../ (dot dot slash) anywhere in the path then we remove it as well as the preceding segment of the path i.e. all occurrences of "<segment>/../" are removed, keep doing this step until no more ../ can be found anywhere in the path (this means our relative path started with one or more ../ i.e. ../blah.html or ../../blah.html etc.)
  • if the path ends with .. then we remove it and the preceding segment of the path, i.e. "<segment>/.." is removed (this means our relative path was .. (dot dot))
  • if the path ends with a . (dot) then we remove it (this most likely means our relative path was . (dot))

At this point we simply append any query string or fragment that our relative URL may have contained to our URL using appropriate separators and we have finished turning our relative URL into an absolute one.

Here are some examples of applying the above algorithm:

1)
base: http://www.blah.com/yadda1/yadda2/yadda3?param1=foo#bar
relative: rel1

final absolute: http://www.blah.com/yadda1/yadda2/rel1

2)
base: http://www.blah.com/yadda1/yadda2/yadda3?param1=foo#bar
relative: /rel1

final absolute: http://www.blah.com/rel1

3)
base: http://www.blah.com/yadda1/yadda2/yadda3?param1=foo#bar
relative: ../rel1

final absolute: http://www.blah.com/yadda1/rel1

4)
base: http://www.blah.com/yadda1/yadda2/yadda3?param1=foo#bar
relative: ./rel1?param2=baz#bar2

final absolute: http://www.blah.com/yadda1/yadda2/rel1?param2=baz#bar2

5)
base: http://www.blah.com/yadda1/yadda2/yadda3?param1=foo#bar
relative: ..

final absolute: http://www.blah.com/yadda1/

Now you should be able to confidently turn any relative URL into an absolute one, as well as know when to use the different forms of relative URL and what the implications will be. For me this has come in handy time and time again in my web development endeavours.

There you go that's really all there is to know about URLs, it's all relatively simple (forgive the pun :)) so no excuse for being unsure about some of this stuff next time. Talking about next time, one of the most common things you need to do when it comes to URLs is recognise if a piece of text is infact a URL, so next time I will show you how to do this using regular expressions (as well as show you how to pull URLs out of text). It should be pretty easy to construct a decent regex now that we've got the structure and special characters down. Stay tuned.

Images by jackfre2, saucebomb and Steve Nilsen

Executing Multiple Commands – A Bash Productivity Tip

MultipleI love shell productivity hacks. I believe that if you work in a Linux environment, you owe it to yourself to become as good as you can be at working with your shell of choice (e.g. bash). You see, most people who have worked with Linux for any length of time have some level of skill with the shell, that level is usually mediocre at best. But, you do meet the occasional person who wows you with what they can accomplish and I will tell you right now that their skill does not come from superior knowledge (or at least not fully from superior knowledge). It is all about maximizing the effect of the knowledge you do have, finding little tricks and hacks that will save you, potentially less than a second, every time you perform a particular action. Thing is, some actions you might do hundreds of times per day, so those seconds really start to add up, especially once you have accumulated dozens of these hacks. Anyway, here is one such tip.

Running Multiple Commands

Whenever you work with the shell you almost always need to run several commands in a row. I am not talking about piping commands to each other, but just running several in sequence. Surprisingly, people usually tend to wait for each command to execute before running the next one. Why not execute all the commands at once? Bash has some decent support for this, and if you train yourself to do it this way, not only will it potentially save you some time, but it will also force you to think further ahead; all of which will make you that little bit more productive. Let's have a look.

Firstly we need to create some commands to make it easier to see what is happening. The first one will be called cmd1, and all it will do is sleep for 2 seconds then output something to the screen.

#!/bin/bash
sleep 2
echo "cmd1"

The second command will be called long-cmd and it will sleep for 30 seconds before outputting something to the screen.

#!/bin/bash
sleep 30
echo "long-cmd"

The third command will be called cmd-fail and it will exit with a non-zero exit status to simulate a failing command.

#!/bin/bash
echo "Failing..."
exit 2

We're now ready to run some commands in sequence by kicking them off at the same time. All you need to do is separate the commands with semicolons and they will execute one after another.

[email protected]:~/tmp$ ./cmd1 ; ./cmd1 ; date
cmd1
cmd1
Sat May  1 23:29:49 EST 2010

As you can see, the two cmd1 commands executed first and then the date command ran – as expected. The only problem here is this, if a preceding command fails, the subsequent ones will still run:

[email protected]:~/tmp$ ./cmd1 ; ./cmd-fail ; date
cmd1
Failing...
Sat May  1 23:32:33 EST 2010

This may be the behaviour we desire, in which case all is well, but what if we do care about the success of the preceding commands. In that case, separate the commands with double ampersand instead of semicolon.

[email protected]:~/tmp$ ./cmd1 && ./cmd-fail && date
cmd1
Failing...

As soon as one of the commands fails, no subsequent command will be executed – handy.

Here is another thought, if you don't need to run multiple commands in sequence, but simply want to kick off multiple commands, why not execute all of them at once by putting them all in the background. All you need to do is separate each command with a single ampersand:

[email protected]:~/tmp$ ./long-cmd & ./long-cmd & ./long-cmd &
[1] 2643
[2] 2644
[3] 2645
[email protected]:~/tmp$

All three of our long commands have kicked off in the background. This works because as soon as we background the first command by putting an ampersand after it, the shell gives us another prompt and is ready to accept more input, so we can keep going and execute as many commands as we want on the same line, by backgrounding them all. It is a slightly quicker way (than waiting for each command to finish, or backgrounding each one separately) to execute multiple commands if they don't rely on each other. If the commands produce output, all the output will still go to the screen, so the output from all the commands can potentially get mixed up, but you can, of course, still redirect the output of each command somewhere else. You can also easily check on the status of each of the backgrounded commands:

[email protected]:~/tmp$ jobs
[1]   Running                 ./long-cmd &
[2]-  Running                 ./long-cmd &
[3]+  Running                 ./long-cmd &

The number in square brackets is the id of the job, and you can use it to perform actions on the running job, such as bringing it to the foreground, or killing it:

[email protected]:~/tmp$ ./long-cmd & ./long-cmd & ./long-cmd &
[1] 2679
[2] 2680
[3] 2681
[email protected]:~/tmp$ kill %2
[email protected]:~/tmp$ jobs
[1]   Running                 ./long-cmd &
[2]-  Terminated              ./long-cmd
[3]+  Running                 ./long-cmd &
[email protected]:~/tmp$  fg %1
./long-cmd

If you foreground a job by using fg, and want to put it back in the background after, all you need to do is press Ctrl-Z and then background the job again by using its id.

[email protected]:~/tmp$ fg %1
./long-cmd
^Z
[1]+  Stopped                 ./long-cmd
[email protected]:~/tmp$ bg %1
[1]+ ./long-cmd &
[email protected]:~/tmp$ jobs
[1]   Running                 ./long-cmd &
[2]-  Running                 ./long-cmd &
[3]+  Running                 ./long-cmd &
[email protected]:~/tmp$

There you go, not revolutionary, but some handy tips to keep in mind to shave off a few seconds here and there. And believe you me, those seconds really do start to add up after a while. More about shell stuff soon.

Image by Roger Smith