by

@pkrumins Peteris Krumins [email protected] http://www.catonmat.net good coders code, great reuse

Preview copy (first 13 pages) Get full e-book at www.catonmat.net/blog/perl-book/

Contents
Contents Preface 1 Introduction 1.1 Perl One-Liners . . . . . . . . . . . . . . . . . . . . . . . . . 2 Spacing 2.1 Double space a le . . . . . . . . . . . . . . . . . . . . . 2.2 Double space a le, except the blank lines . . . . . . . . 2.3 Triple space a le . . . . . . . . . . . . . . . . . . . . . . 2.4 N-space a le . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Add a blank line before every line . . . . . . . . . . . . . 2.6 Remove all blank lines . . . . . . . . . . . . . . . . . . . 2.7 Remove all consecutive blank lines, leaving just one . . . 2.8 Compress/expand all blank lines into N consecutive ones 2.9 Double space all words . . . . . . . . . . . . . . . . . . . 2.10 Remove all spacing between words . . . . . . . . . . . . . 2.11 Change all spacing between words to one space . . . . . 2.12 Insert a space between each character . . . . . . . . . . . 3 Numbering 3.1 Number all lines in a le . . . . . . . . . . . . . 3.2 Number only non-empty lines in a le . . . . . . 3.3 Number and print only non-empty lines in a le 3.4 Number all lines but print line numbers for only lines . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Number only lines that match a pattern, print modied . . . . . . . . . . . . . . . . . . . . . . i i vi 1 1 6 6 8 9 10 10 11 12 13 13 13 14 14 15 15 15 16 16 16

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . non-empty . . . . . . . others un. . . . . . .

Preview copy (first 13 pages) Get full e-book at www.catonmat.net/blog/perl-book/

CONTENTS 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 3.15 Number and print only lines that match a pattern . . . . . . Number all lines, but print line numbers only for lines that match a pattern . . . . . . . . . . . . . . . . . . . . . . . . . Number all lines in a le using a custom format . . . . . . . Print the total number of lines in a le (emulate wc -l) . . . Print the number of non-empty lines in a le . . . . . . . . . Print the number of empty lines in a le . . . . . . . . . . . Print the number of lines in a le that match a pattern . . . Number words across all lines . . . . . . . . . . . . . . . . . Number words on each line individually . . . . . . . . . . . . Replace all words with their numeric positions . . . . . . . .

ii 17 17 17 18 19 20 21 21 21 22 23 23 24 25 25 27 27 28 28 29 29 29 30 30 31 31 31 32 32 34 34 35 35 36 37

4 Calculations 4.1 Check if a number is a prime . . . . . . . . . . . . . . . . . . 4.2 Print the sum of all the elds on a line . . . . . . . . . . . . 4.3 Print the sum of all the elds on all lines . . . . . . . . . . . 4.4 Shue all elds of every line . . . . . . . . . . . . . . . . . . 4.5 Find the minimum element on every line . . . . . . . . . . . 4.6 Find the minimum element over all the lines . . . . . . . . . 4.7 Find the maximum element on a line . . . . . . . . . . . . . 4.8 Find the maximum element over all the lines . . . . . . . . . 4.9 Replace each eld with its absolute value . . . . . . . . . . . 4.10 Find the total number of elds (words) on each line . . . . . 4.11 Print the total number of elds (words) on each line followed by the line . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.12 Find the total number of elds (words) on all lines . . . . . . 4.13 Print the total number of elds that match a pattern . . . . 4.14 Print the total number of lines that match a pattern . . . . 4.15 Print the number PI to n decimal places . . . . . . . . . . . 4.16 Print the number E to n decimal places . . . . . . . . . . . . 4.17 Print UNIX time (seconds since Jan 1, 1970, 00:00:00 UTC) 4.18 Print GMT (Greenwich Mean Time) and local computer time 4.19 Print yesterdays date . . . . . . . . . . . . . . . . . . . . . 4.20 Print date 14 months, 9 days and 7 seconds ago . . . . . . . 4.21 Calculate factorial . . . . . . . . . . . . . . . . . . . . . . . 4.22 Calculate greatest common divisor . . . . . . . . . . . . . . 4.23 Calculate least common multiple . . . . . . . . . . . . . . . 4.24 Generate 10 random numbers between 5 and 15 (excluding 15)

Preview copy (first 13 pages) Get full e-book at www.catonmat.net/blog/perl-book/

CONTENTS 4.25 4.26 4.27 4.28 Find and print all permutations of a list . . . Generate the power set . . . . . . . . . . . . . Convert an IP address to unsigned integer . . Convert an unsigned integer to an IP address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

iii 37 38 39 40 42 42 43 43 44 45 45 46 46 47 47 48 48 48 49 49 50 51 51 51 52 52 52 53 53 53 54 55 55 56 56 56

5 String Creation and Array Creation 5.1 Generate and print the alphabet . . . . . . . . . . . . 5.2 Generate and print all the strings from "a" to "zz" . 5.3 Create a hex lookup table . . . . . . . . . . . . . . . 5.4 Generate a random 8 character password . . . . . . . 5.5 Create a string of specic length . . . . . . . . . . . . 5.6 Create an array from a string . . . . . . . . . . . . . 5.7 Create a string from an array . . . . . . . . . . . . . 5.8 Find the numeric values for characters in a string . . 5.9 Convert a list of numeric ASCII values into a string . 5.10 Generate an array with odd numbers from 1 to 100 . 5.11 Generate an array with even numbers from 1 to 100. 5.12 Find the length of the string . . . . . . . . . . . . . . 5.13 Find the number of elements in an array . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

6 Text Conversion and Substitution 6.1 ROT13 a string. . . . . . . . . . . . . . . . . . . . . . . . 6.2 Base64 encode a string . . . . . . . . . . . . . . . . . . . 6.3 Base64 decode a string . . . . . . . . . . . . . . . . . . . 6.4 URL-escape a string . . . . . . . . . . . . . . . . . . . . 6.5 URL-unescape a string . . . . . . . . . . . . . . . . . . . 6.6 HTML-encode a string . . . . . . . . . . . . . . . . . . . 6.7 HTML-decode a string . . . . . . . . . . . . . . . . . . . 6.8 Convert all text to uppercase . . . . . . . . . . . . . . . 6.9 Convert all text to lowercase . . . . . . . . . . . . . . . . 6.10 Uppercase only the rst word of each line . . . . . . . . . 6.11 Invert the letter case . . . . . . . . . . . . . . . . . . . . 6.12 Camel case each line . . . . . . . . . . . . . . . . . . . . 6.13 Strip leading whitespace from the beginning of each line 6.14 Strip trailing whitespace from the end of each line . . . . 6.15 Strip whitespace from the beginning and end of each line 6.16 Convert UNIX newlines to DOS/Windows newlines . . . 6.17 Convert DOS/Windows newlines to UNIX newlines . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

Preview copy (first 13 pages) Get full e-book at www.catonmat.net/blog/perl-book/

CONTENTS 6.18 6.19 6.20 6.21 6.22 Convert UNIX newlines to Mac newlines . . . . . . . . . . Substitute (nd and replace) "foo" with "bar" on each line Substitute "foo" with "bar" on lines that match "baz" . . Reverse paragraphs in a le . . . . . . . . . . . . . . . . . Reverse all lines . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

iv 57 57 57 58 58 60 60 60 61 62 62 63 63 64 64 64 65 65 65 66 66 66 67 67 67 68 68 68 69 69 69 70 70

7 Selective Printing and Deleting of Lines 7.1 Print the rst line of a le (emulate head -1) . . . . . . . . . 7.2 Print the rst 10 lines of a le (emulate head -10) . . . . . . 7.3 Print the last line of a le (emulate tail -1) . . . . . . . . . . 7.4 Print the last 10 lines of a le (emulate tail -10) . . . . . . . 7.5 Print only lines that match a regular expression . . . . . . . 7.6 Print only lines that do not match a regular expression . . . 7.7 Print the line before a line that matches a regular expression 7.8 Print the line after a line that matches a regular expression . 7.9 Print lines that match regex AAA and regex BBB in any order 7.10 Print lines that dont match match regexes AAA and BBB . 7.11 Print lines that match regex AAA followed by BBB and CCC 7.12 Print lines that are 80 chars or longer . . . . . . . . . . . . . 7.13 Print lines that are less than 80 chars in length . . . . . . . 7.14 Print only line 13 . . . . . . . . . . . . . . . . . . . . . . . . 7.15 Print all lines except line 27 . . . . . . . . . . . . . . . . . . 7.16 Print only lines 13, 19 and 67 . . . . . . . . . . . . . . . . . 7.17 Print all lines between two regexes . . . . . . . . . . . . . . 7.18 Print all lines from line 17 to line 30 . . . . . . . . . . . . . 7.19 Print the longest line . . . . . . . . . . . . . . . . . . . . . . 7.20 Print the shortest line . . . . . . . . . . . . . . . . . . . . . 7.21 Print all lines that contain a number . . . . . . . . . . . . . 7.22 Find all lines that contain only a number . . . . . . . . . . . 7.23 Print all lines that contain only characters . . . . . . . . . . 7.24 Print every second line . . . . . . . . . . . . . . . . . . . . . 7.25 Print every second line, starting the second line. . . . . . . . 7.26 Print all lines that repeat . . . . . . . . . . . . . . . . . . . 7.27 Print all unique lines . . . . . . . . . . . . . . . . . . . . . .

8 Handy Regular Expressions 71 8.1 Match something that looks like an IP address . . . . . . . . 71 8.2 Test if a number is in range 0-255 . . . . . . . . . . . . . . . 71

Preview copy (first 13 pages) Get full e-book at www.catonmat.net/blog/perl-book/

CONTENTS 8.3 8.4 8.5 8.6 8.7 8.8 8.9 8.10 8.11 8.12 Match an IP address . . . . . . . . . . . . . . . . . . . . . Check if the string looks like an email address . . . . . . . Check if the string is a decimal number . . . . . . . . . . . Check if a word appears twice in the string . . . . . . . . . Increase all numbers by one in the string . . . . . . . . . . Extract HTTP User-Agent string from the HTTP headers Match printable ASCII characters . . . . . . . . . . . . . . Match text between two HTML tags . . . . . . . . . . . . Replace all <b> tags with <strong> . . . . . . . . . . . . Extract all matches from a regular expression . . . . . . . . . . . . . . . . .

v 72 72 73 75 75 75 76 76 77 77 78 93 93 96 97 98 99 99 100 100 101 102 103

9 perl1line.txt A Perls Special Variables A.1 Variable $_ . . . . . . A.2 Variable $. . . . . . . . A.3 Variables $1, $2, $3, ... A.4 Variable $/ . . . . . . A.5 Variable $\ . . . . . . A.6 Variable $, . . . . . . . A.7 Variable $" . . . . . . A.8 Variable @F . . . . . . A.9 Variable @ARGV . . . A.10 Variable %ENV . . . . Index

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

Preview copy (first 13 pages) Get full e-book at www.catonmat.net/blog/perl-book/

Preface
Thanks!
Thank you for buying my "Perl One-Liners Explained" e-book! This is my third e-book and I based it on the "Famous Perl One-Liners Explained" article series that I wrote on my www.catonmat.net blog. I went through all the one-liners in the articles, improved explanations, xed a lot of mistakes and typos, added a bunch of new one-liners, added an Introduction to Perl one-liners and a new chapter on Perl special variables. You might wonder why I called the article series that I based the book on "famous"? Its because I based this article series on two similar article series about awk and sed that I had written in the past. These article series are "Famous Awk One-Liners Explained" and "Famous Sed OneLiners Explained". Now why did I call these series "famous"? Well, because I based the articles on the popular awk1line.txt and sed1line.txt les by Eric Pement. These les have been circulating around Unix newsgroups and forums for years and theyre very popular among Unix programmers and sysadmins. Thats how I actually really learned awk and sed myself. I went through all the one-liners in those les, tried them out and endeavored to understand exactly how they work. Then I decided that it would be a good idea to explain them on my blog, which I did, and after that I thought, why not create my own perl1line.txt le. So I did. I also included this le in the e-book as Chapter 9. As I mentioned, this is my third e-book. My rst e-book is "Awk OneLiners Explained" and my second e-book is "Sed One-Liners Explained." All these e-books are written in the same style, so if you enjoy this e-book, youll also enjoy the other two! I have planned writing many more e-books. The next few are going to be the catonmat book, a practical guide to vim, and a practical guide to anonymity. If youre interested, subscribe to my blog, follow me on Twitter or follow me on Google+. That way youll be the rst to know when I publish it! Enjoy! vi

Preview copy (first 13 pages) Get full e-book at www.catonmat.net/blog/perl-book/

One Introduction
1.1 Perl One-Liners

Perl one-liners are small and awesome Perl programs that t in a single line of code and they do one thing really well. These things include changing line spacing, numbering lines, doing calculations, converting and substituting text, deleting and printing certain lines, parsing logs, editing les in-place, doing statistics, carrying out system administration tasks, updating a bunch of les at once, and many more. Perl one-liners will make you the shell warrior. Anything that took you minutes to solve, will now take you seconds! Lets look at several examples to get more familiar with one-liners. Here is one:
perl -pi -e 's/you/me/g' file

This one-liner replaces all occurrences of the text you with me in the le Very useful if you ask me. Imagine you are on a remote server and have this le and you need to do the replacement. You can either open it in text editor and execute nd-replace or just do it through command line and, bam, be done with it. The -e argument is the best argument. It allows you to specify the Perl code to be executed right on the command line. In this one-liner the code says, do the substitution (s/.../.../ command) and replace you with me globally (/g ag). The -p argument makes sure the code gets executed on every line, and that the line gets printed out after that. The -i argument makes sure that file gets edited in-place, meaning Perl opens the le, executes the substitution for each line, and puts it back in the le. Dont worry too much about the command line arguments right now, youll learn all about them as you work through the book! How about doing the same replacement in multiple les? Just specify them on the command line!
file.

Preview copy (first 13 pages) Get full e-book at www.catonmat.net/blog/perl-book/

CHAPTER 1. INTRODUCTION

perl -pi -e 's/you/me/g' file1 file2 file3

Now lets do the same replacement only on lines that match we? Its as simple as this:
perl -pi -e 's/you/me/g if /we/' file

Here we use the conditional if /we/. This makes sure that s/you/me/g gets executed only on lines that match the regular expression /we/. The regular expression here can be anything. Lets say you want to execute the substitution only on lines that have digits on them. You can then use the /\d/ regular expression that matches numbers:
perl -pi -e 's/you/me/g if /\d/' file

Now how about nding all repeated lines in a le?

This one-liner records the lines seen so far in the %a hash and keeps the counter of how many times it has seen the lines. The $a{$_}++ creates elements in the %a hash automagically. When it sees a repeated line, the value of that hash element is greater than one, and if $a{$_} is true, so it prints the line. This one-liner also uses the -n command line argument that loops over the input but unlike -p doesnt print the lines automatically. Youll nd much more detailed description of -n in the rst chapter of the book. How about numbering lines? Super simple! Perl has the $. special variable that maintains the current line number. You can just print it out together with the line:
perl -ne 'print "$. $_"'

$_

You can also achieve the same by using -p argument and modifying the variable:

perl -pe '$_ = "$. $_"'

Preview copy (first 13 pages) Get full e-book at www.catonmat.net/blog/perl-book/

CHAPTER 1. INTRODUCTION

Here each line gets replaced by the string "$. $_", which is the current line number followed by the line itself. See 3.1 for a full explanation of this one-liner. How about we combine the previous two one-liners and create one that numbers repeated lines? Here we go:
perl -ne 'print "$. $_" if $a{$_}++'

Now lets do something dierent. Lets sum up all the numbers in each line. Well use the sum function from the List::Util CPAN module. You can install it as easily as running perl -MCPAN -einstall List::Util.
perl -MList::Util=sum -alne 'print sum @F'

The -MList::Util command line argument imports the List::Util module, and the =sum part of it imports the sum function from it. Next -a enables automatic splitting of elds into the @F array. The -l argument makes sure that print outputs a newline at the end of each line. Finally the sum @F sums up all the elements in the @F list and print prints it out, followed by a newline (that was added by the -l argument). This one-liner is explained in more details in section 4.2. How about nding the date 1299 days ago? Thats also solvable by a simple one-liner:
perl -MPOSIX -le ' @now = localtime; $now[3] -= 1299; print scalar localtime mktime @now '

This one-liner didnt quite t in one line, but thats just because this book has large margins. I explain this example in great detail in oneliner 4.19. Basically we modify the 4th element of the structure returned by localtime, which happens to be days. So we just subtract 1299 days from the current day. Then we reassembles it into a new time through localtime mktime @now and print it in scalar context that prints human readable time. This one-liner is explained in more details in sections 4.18, 4.19 and 4.20.

Preview copy (first 13 pages) Get full e-book at www.catonmat.net/blog/perl-book/

CHAPTER 1. INTRODUCTION

How about generating an 8 letter password? Again, solvable elegantly with a one-liner:
perl -le 'print map { (a..z)[rand 26] } 1..8'

The a..z generates a list of letters from a to z (total 26). Then we randomly choose one of them, and we repeat it 8 times! This example is explained in much more detail in one-liner 5.4. Here is another one. Suppose you want to quickly nd the decimal number that corresponds to an IP address. You can use the unpack function and nd it really quickly:
perl -le 'print unpack("N", 127.0.0.1)'

This one-liner uses a vstring, which is a version literal. The IP address 127.0.0.1 is treated as a vstring, which is basically the numbers 127, 0, 0, 1 concatenated together. Next the unpack function unpacks them to a single decimal number. A much more detailed explanation is given in one-liner 4.27. Now how about calculations? Lets nd the sum of the numbers in the rst column:
perl -lane '$sum += $F[0]; END { print $sum }'

Here the lines automatically get split up into elds through the -a argument. The elds can be now accessed through the @F array. The rst element of the array, $F[0], is the rst column. So all we have to do is sum all the columns up through $sum += $F[0]. When the Perl program ends its job, it executes any code in the special END block. In this case we print out the total sum there. Really easy! Now lets nd out how many packets have gone through all the iptables rules. Its this simple:
iptables -L -nvx | perl -lane ' $_ = $F[0]; $pkts += $_ if /^\d/; END { print $pkts } '

Preview copy (first 13 pages) Get full e-book at www.catonmat.net/blog/perl-book/

CHAPTER 1. INTRODUCTION

The iptables program outputs the packets as the rst eld. All we do is check if the rst eld is numeric (because it also outputs label header), and if so, sum the packets up, just like in the previous one-liner. How about getting a list of the names of all users on the system?
perl -a -F: -lne 'print $F[4]' /etc/passwd

Combining -a with -F argument allows you to specify the character that lines should be split on. In this case its the column, which is the record separator of /etc/passwd. Next we just print the 5th eld $F[4], which is the real name of the user. Really quick and easy. As you can see, knowing Perl one-liners lets you accomplish many tasks quickly. Overall this e-book has 130 unique one-liners, which will let you become the shell wizard. Many one-liners are presented in several dierent ways so the total number of one-liners in this book is over 200. Enjoy!

Preview copy (first 13 pages) Get full e-book at www.catonmat.net/blog/perl-book/

Two Spacing
2.1 Double space a le

perl -pe '$\="\n"'

This one-liner double spaces a le. There are three things to explain in this one-liner. The -p and -e command line options, and the $\ variable. First lets start with the -e option. The -e option can be used to enter a Perl program directly in the command line. Typically you dont want to create source les for every small program. With -e you can easily write the program right in the command line as a one-liner. Next the -p switch. Specifying -p to a Perl program causes it to assume the following loop around your program:
while (<>) { # your program goes here (specified by -e) } continue { print or die "-p failed: $!\n"; }

Broadly speaking, this construct loops over all the input, executes your code and prints the value of $_. This way you can quickly modify all or some lines of the input. The $_ variable can be explained as an anonymous variable that gets lled with the current line of text. (Youll see later that it can be lled with other stu as well.). However, its important to understand what is going on in this loop in more details. First the while (<>) loop takes each line from the standard input and puts it in the $_ variable. Next the code specied by -e gets executed. Then the print or die part gets executed. The continue statement executes the print or die statement after each line. The print or die tries to print the contents of the $_ variable. If it fails (for example, the 6

Preview copy (first 13 pages) Get full e-book at www.catonmat.net/blog/perl-book/

This is a preview copy (first 13 pages)

Get the full e-book at:

www.catonmat.net/blog/perl-book/