Comments on: The Most Common Things You Do To A Large Data File With Bash

By: lre

lre — Mon, 12 Sep 2011 17:57:10 +0000

Just stumbled across this post, and have to add

awk ‘NR==1 { next} # Do not hold first line
hold!~/^$/ { print hold } # If held line is non-empty, print it
{ hold= $0 } # Hold line (thus, last line is not printed)
‘ inputfile > outputfile

By: Alan Skorkin

Alan Skorkin — Mon, 08 Mar 2010 13:43:22 +0000

I just love the names that unix and linux people come up with for their utilities :). Apparently “buthead” is a Program to copy all but the first N lines of standard input to standard output.

By: Barak A. Pearlmutter

Barak A. Pearlmutter — Mon, 08 Mar 2010 13:39:25 +0000

apt-get install buthead

By: Dew Drop – March 7, 2010 | Alvin Ashcraft's Morning Dew

Dew Drop – March 7, 2010 | Alvin Ashcraft's Morning Dew — Sun, 07 Mar 2010 14:24:13 +0000

[...] The Most Common Things You Do To A Large Data File With Bash (Alan Skorkin) [...]

By: Alan Skorkin

Alan Skorkin — Sun, 07 Mar 2010 11:09:12 +0000

Hi Peter,

You’re right they are programming languages in their own right, but you can easily pipe inputs to them and pipe outputs out of them and in that way they are very much like shell tools (i.e. head, tail etc.).

So while we’re using sed and awk we’re not writing full fledged scripts in them but instead are piping data through them and allowing them to perform little bits of functionality all on the command line, as per the unix philosophy.

By: Peter Cable

Peter Cable — Sun, 07 Mar 2010 08:33:23 +0000

I’m unclear why you even mention bash here. Sed, awk and ruby are interpreters for their respective programming languages and they are doing the actual work here, not bash.

You could accomplish all these tasks in bash, but I think it would be messy.

By: Alan Skorkin

Alan Skorkin — Sun, 07 Mar 2010 07:23:05 +0000

I didn’t even consider that you can pipe stuff to ruby, although i am not sure why I didn’t consider it since it makes perfect sense :). And having said that the example you give looks very similar to how you can pipe stuff to perl on the command line, but I never tend to use it since my perl skills are abysmal. But as you say, for simpler things I prefer to use the simple tools that are built into the shell and only take it up a notch (with perl or ruby) for more complex stuff.

By: Korny

Korny — Sun, 07 Mar 2010 05:38:57 +0000

I always use ‘head’ and ‘tail’ to trim lines, and ‘grep’ to do things like remove internal blank lines.
For more complex things I have used ‘sed’, but I always have to look it up again when I need it – which is a bit of a showstopper.
Instead, I try to re-use the tool I’m most familiar with – ruby. Consider:
cat my_data | ruby -pe ‘next if $_ == “\n” ‘
the ‘-p’ means ‘loop and print every line’, and the call to “next” skips out of the loop before the printing. Alternatively, you can use ‘-n’ which loops without printing.
Or you can just roll your own loop:
cat my_data | ruby -e ‘$stdin.each {|l| puts l unless l == “\n”}’
Or to truncate your first and last lines: (unfortunately loading everything into memory)
cat my_data | ruby -e ‘$stdin.to_a[1...-1].each {|l| puts l unless l == “\n”}’
… though realistically, for such a simple example I’d probably use head, tail, and grep!

By: Alan Skorkin

Alan Skorkin — Sun, 07 Mar 2010 04:45:50 +0000

Hi Evan,

It certainly looks like it would be, can you tell that my sed skills are sub-par :). Thanks.

By: Evan

Evan — Sun, 07 Mar 2010 04:43:28 +0000

Wouldn’t “sed -e 1d -e ‘$d’ -e ‘/^$/d’ input_file > output_file” be a bit more efficient. Save that large file having to to through two pipes and two extra processes!