Crazy good awk Stuff
--------------------

To remove everything but the first column before the space in the a
file
cat file | awk '/ / {print $1}' > output.txt

To remove everything but the first column before the :
cat file | awk -F: '{print $1}' > output.txt


From http://www.acck.edu/~zjerodp/spellbook.html 
This is really good so I hope they don't mind I'm borrowing it.

Unix command line tricks
This page is always being modified. This is a listing of short
programs to accomplish small tasks. These were written for the csh on
HP-UX 11.0 but should be applicable to any BSD derived system. You
will find awk, sed, perl, and vi tricks here. Frequently I work with
files containing data that require preparation to use somewhere else.

A good way to get familiar is to create some test files filled with
data. Here is a sample for you to copy and paste in your text
editor. For this page, this file will be called myfile.


Ashbourne   14567   Crowley, Tom	crowley@fire
Belfast     65420   Stewart, Bob	stewart@electric
Chelmsford  34591   Campbell, Lisa	campbell@kinetic
Dungannon   11278   Morse, John		morse@gravity
Edinburgh    2290   Fox, Paul		fox@water
Fareham     34531   Biggs, Douglass	biggs@metal
Glasgow     98231   Anderson, Aubrey	anderson@air
Hastings     4500   Baldauf, Jack	baldauf@snuke
Ipswich     32578   Brooks, David	brooks@mineral
Jesmond     44498   Fryxell, Greta	fryxell@wood
Kensington  90124   North, Gerald	north@wnuke
London      55812   Pinckney, Jay	pinckney@shadow
Manchester  98703   Slowey, Niall	slowey@plasma
Newcastle   89239   Wormuth, John	wormuth@earth


Display file without the blank lines:

awk 'NF > 0' myfile
 awk '/./' myfile
 sed -e '/^$/d' myfile
 perl -ne 'print unless /^ *$/' myfile

You can delete the blank lines in vim by entering this command while
the cursor is at the top of the file.


:g/^$/d

Delete blanks (whitespace) at start/end of line:

In vim, to remove leading blanks   :%s/^ *//
  In vim, to remove trailing blanks   :%s/ *$//
    This will not remove blanks between words.
  awk '{sub(/^[ \t]+/, ""); print}' myfile 	# from start of line
  awk '{sub(/[ \t]+$/, "");print}' myfile 	# from end of line
  sed 's/\ \ *$//' myfile		# remove blank space at end of line

Insert blanks at start of line.

perl -p -i -e "s/$/ /g;" myfile	# insert blank space at end of line
  perl -p -i -e "s/^/ /g;" myfile	# insert blank space at start of line

Remove first 10 lines in a file:

sed '1,10d' myfile

Remove the last character in a line in vim:

:1,$s/.$//

Place a comma at the end of each line:

sed 's/$/\,/g' myfile
 perl -p -i -e "s/\n/\,\n/g;" myfile

Insert blank lines in a file:

sed G myfile               # single blank line between each line
 sed "G;G" myfile           # double blank lines between each line
 sed "n;n;n;n;G;" myfile    # insert blank line after 5 lines, 10 lines, etc.

 awk '1;{print ""}' myfile  # single blank line between each line
 awk 'NF{print $0 "\n"}'    # double blank lines between each line
 awk '1;{print "\n"}'	    # triple blank lines


Place single quotes around line:

sed "s/\(.*\)/'\1'/" myfile

Enumerate non-blank lines at the beginning:

sed '/./=' myfile | sed '/./N; s/\n/ /'

Reverse the order of a file:

sed '1!G;h;$!d' myfile    # this works in sh not csh
 perl -e 'print reverse <>' myfile
   in vim, enter this command:    :g/^/m0

Left justify each line:

awk '{sub(/^[ \t]+/, ""); print}' myfile

Add/remove leading angle bracket at start of line:

sed 's/^/> /' myfile
 sed 's/^> //' myfile

Print file without HTML tags (may not be 100% effective):

sed -e :a -e 's/<[^>]*>//g;/</N;//ba' myfile

Capitalize the first letter of each line:

awk '{ sub(/^./,toupper(substr($0,1,1))); print; }' myfile

Print even-numbered lines:

awk 'NR % 2' myfile
 awk 'NR % 2 == 0' myfile     # for odd-numbered lines

Print certain lines:

sed -n '5,8p' myfile	     # print lines 5,6,7 and 8 of myfile

Beginning at line 2, print every 4th line:

sed -n '2,${p;n;n;n;}' myfile

Place colon between fields:
You need to know the number of characters in each line. In our example
file myfile, think of a number line running across the file:

---------1---------2---------3---------4---------5---------6
Ashbourne   14567   Crowley, Tom	crowley@fire
Belfast     65420   Stewart, Bob	stewart@electric
Chelmsford  34591   Campbell, Lisa	campbell@kinetic

And determine the best character number to place a colon, then use the
substr function of awk:


awk '{print
substr($0,1,12),":",substr($0,12,7),":",substr($0,19,20),":",substr($0,39,20)}
myfile

will result in:

Ashbourne    :  14567  :   Crowley, Tom       :   crowley@fire
Belfast      :  65420  :   Stewart, Bob       :   stewart@electric
Chelmsford   :  34591  :   Campbell, Lisa     :   campbell@kinetic
Dungannon    :  11278  :   Morse, John        :   morse@gravity
Edinburgh    :   2290  :   Fox, Paul          :   fox@water
Fareham      :  34531  :   Biggs, Douglass    :   biggs@metal
Glasgow      :  98231  :   Anderson, Aubrey   :   anderson@air
Hastings     :   4500  :   Baldauf, Jack      :   baldauf@snuke
Ipswich      :  32578  :   Brooks, David      :   brooks@mineral
Jesmond      :  44498  :   Fryxell, Greta     :   fryxell@wood
Kensington   :  90124  :   North, Gerald      :   north@wnuke
London       :  55812  :   Pinckney, Jay      :   pinckney@shadow
Manchester   :  98703  :   Slowey, Niall      :   slowey@plasma
Newcastle    :  89239  :   Wormuth, John      :   wormuth@earth

Notice that awk automatically places a space flanking the colon. Find
& replace methods:


Change words to "ether"
 sed "s/mineral/ether/" myfile           # replaces 1st instance on each line
 sed "s/water/ether/4" myfile          # replaces 4th instance on each line
 sed "s/metal/ether/g" myfile          # replaces ALL instances
 sed "/metal/s/metal/ether/g" myfile     # same but executes more quickly
 sed "/North/s/nuke/ether/g" myfile     # ONLY for lines which contain "North"
 sed "/North/!s/nuke/ether/g" myfile    # EXCEPT for lines which
 contain "North"


 perl -p -e 's/fire/ether/g;' myfile    #show ether instead of fire
 perl -p -i -e 's/fire/ether/g;' myfile #actually changes the contents
 of myfile

 perl -pi -e'substr($_,6,3)="---"' myfile  # change the 7th, 8th, and
 9th characters of all lines to ---


 awk '{sub(/gravity/,"ether");print}' myfile # replaces all gravity with ether
 awk '/baz/{sub(/plasma/, "ether")};{print}' myfile # replaces plasma
 with ether only on lines with baz
 awk '{sub(/fire|electric|kinetic/, "ether"); print}' myfile  # change
 fire,electric,kinetic to ether


 in vim:
   :%s/shadow/light/g			# replaces ALL instances of
   shadow with light in file



Change the 7th character of each line to the letter j

sed 's/./j/7' myfile
 perl -pe '7 < length and substr($_,6,1)="j"' myfile

 In the vim editor:   :%s/^\(.\{7\}\)./\1j/


List directory files with a certain datestamp

ls -l | awk '{if($6=="May" && $7=="16") print $NF }'

Will list the names of files in the current directory dated May
16. Find the file 'printtest' in subdirectories, starting with the
current directory, but don't bother looking in the 'docs' subdirectory


find . ! -path ./docs -only -name printtest  # yes, I am aware that
this is not using vi, perl, sed or awk


Get the lines flanking lines with search term

awk '/Manchester/{print x};{x=$0}' myfile; awk
'/Manchester/{getline;print}' myfile


will get you the lines before and after lines with 'Manchester':

London      55812   Pinckney, Jay       pinckney@shadow
Newcastle   89239   Wormuth, John       wormuth@earth

Print the binary equivalent of the word 'fire'

perl -e 'print unpack "B*","fire"' ; perl -e 'print "\n"'

Print the hex equivalent of the word 'fire'

perl -e 'print unpack "H*","fire"' ; perl -e 'print "\n"'

To reverse the translated string back, use pack instead of unpack