2.1 The Unix World in 1991 ¶

Undoubtedly, many readers of this paper were not using computers in 1991, so this section provides the context in which Henry’s paper was written. In March of 1991:

• Commercial Unix systems were the norm, with offerings from AT&T, Digital Equipment Corporation, Hewlett Packard, IBM, Sun Microsystems, and many others, all vying for market share. Microsoft Windows existed, but was primarily a layer on top of MS-DOS and was not taken seriously.
• Very few sites still ran the original Bell Labs or direct-from-UCB variants of Unix; those did not keep up with the available hardware and AT&T was itself trying to succeed in the Unix hardware market.
• GNU/Linux did not exist! Some unencumbered BSD variants were available, but they were still under the cloud of the AT&T/UCB law suit.²
• So-called “new” awk was about 2.5 years old. The book by Aho, Weinberger and Kernighan was published in October of 1987, so most people knew about new awk, but they just couldn’t get it.

  Who could? New awk was available to educational institutions from the Bell Labs research group, and to those who had Unix source licenses for System V Releases 3.1, 3.2, and 4. By this time, source licensees were an extremely rare breed, since the cost for commercial licenses had skyrocketed, and even for educational licensees it had increased greatly.³ If I recall correctly, an educational license cost around US $1,000, considerably more than the earlier Unix licenses.

• PERL⁴ existed and was starting to gain in popularity. In 1991, “PERL” most likely meant PERL 3 or a very early version of PERL 4. The World Wide Web, which was one of the major reasons for PERL’s growth in popularity, had not yet really taken off.
• Other implementations of new awk were available:
  • MKS Awk for PC systems (MS-DOS).
  • GNU Awk was available and relatively stable, but could not be called “solid.”

  The problem with the first of these is that source code was not available. And the latter came with (to quote Henry) “troublesome licenses.” (Actually, Henry no longer remembers whether his statement about “troublesome licenses” referred to the GPL, or to the Bell Labs source licenses.)

• Michael Brennan’s mawk (also GPL’ed) was not yet available. Version 1.0 was accepted for posting in comp.sources.reviewed on September 30, 1991, half a year after Henry’s paper was published.

2.2 What Awk Lacked In 1991 ¶

Here is a summary of what was wrong with the awk picture in 1991. These are in the same order as presented Henry’s paper. We qualify each issue in order to later discuss how it has been addressed over time.

1. New awk was not widely available. Most Unix vendors still shipped only old awk. (Here is where he mentions that “the independently-available implementations either cost substantial amounts of money or come with troublesome [sic] licenses.”) His point then was that for portability, awk programs had to be restricted to old awk.

   This could be considered a quality of implementation issue, although it’s really a “lack of available implementation” issue.

2. There is no way to tell awk to start matching all its patterns over again against the existing $0. This is a language design issue.
3. There is no array assignment. (Language design issue.)
4. Getting an error message out to standard error is difficult. (Implementation issue.)
5. There is no precise language specification for awk. This leads to gratuitous portability problems. This too is thus a quality of implementation issue, in that without a specification, it’s difficult to produce uniform, high quality implementations.
6. The existing widely available implementation is slow; a much faster implementation is needed and the best thing of all would be an optimizing compiler. (Implementation issue.)
7. There is no awk-level debugger. (Support tool or quality of implementation issue.)
8. There is no awk-level profiler. (Support tool or quality of implementation issue.)

In private email, Henry added the following items, saying “there are a couple more things I’d add now, in hindsight.” These are direct quotes:

9. [I can’t believe I didn’t discuss this in the paper, because I was certainly aware of it then!] Lack of any convenient mechanism for adding libraries. When awk is being invoked from a shell file, the shell file can do substitutions or use multiple -f options, but those are mechanisms outside the language, and not very convenient ones. What’s really wanted is something like you get in Python etc., where one little statement up near the top says “arrange for this program to have the xyz library available when it runs.”
10. I think it was Rob Pike who later said (roughly): “It says something bad about Awk that in a language with integrated regular expressions, you end up using substr() so often.” My paper did allude to the difficulty of finding out where something matched in old-awk programs, but even in new awk, what you get is a number that you then have to feed to substr(). The language could really use some more convenient way of dissecting a string using regexp matching. [Caveat: I have not looked lately at Gawk to see if it has one.]

The first of these is somewhere between a language design and a language implementation issue. The latter is a language design issue.

3.1 What Awk Has Today ¶

The state of the awk world is much better now. In the same order:

1. New awk is the standard version of awk today on GNU/Linux, BSD, and commercial Unix systems. The one notable exception is Solaris, where /usr/bin/awk is still the old one; on all other systems, plain awk is some version of new awk.
2. There remains no way to tell awk to start matching all its patterns over again against the existing $0. Furthermore, this is a feature that has not been called for by the awk community, except in Henry’s paper. (We do acknowledge that this might be a useful feature.)
3. There continues to be no array assignment. However, this function in gawk, which has arrays of arrays, can do the trick nicely. It is also efficient, since gawk uses reference counted strings internally:

   function copy_array(dest, source,   i, count)
   {
       delete dest
   
       for (i in source) {
           if (typeof(source[i]) == "array")
               count += copy_array(dest[i], source[i])
           else {
               dest[i] = source[i]
               count++
           }
       }
   
       return count
   }
   

4. Getting error messages out is easier. All modern systems have a /dev/stderr special file to which error messages may be sent directly. gawk, mawk and Brian Kernighan’s awk all have "/dev/stderr" built in for I/O redirections, so even on systems without a real /dev/stderr special file, you can still send error messages to standard error.
5. Perhaps most important of all, with the POSIX standard, there is a formal standard specification for awk. As with all formal standards, it isn’t perfect. But it provides an excellent starting point, as well as chapter and verse to cite when explaining the behavior of a standards-compliant version of awk.

   Additionally, the second edition of The AWK Progamming Language is now available.

6. There are a number of freely available implementations, with different licenses, such that everyone ought to be able to find a suitable one:
   • Brian Kernighan’s awk is the direct lineal descendant of Unix awk. He calls it the “One True Awk” (sic). It is available from Github:

     $ git clone git://github.com/onetrueawk/awk bwkawk
     

   • GNU Awk, gawk, is available from the Free Software Foundation. You may use an HTTPS downloader: https://ftp.gnu.org/gnu/gawk/gawk-5.3.0.tar.gz is the current version. There may be a newer one.
   • Michael Brennan’s awk, known as mawk. In 2009, Thomas Dickey took on mawk maintenance. Basic information is available on the project’s web page. The download URL is https://invisible-island.net/datafiles/release/mawk.tar.gz.

     In 2017 Michael published a beta of mawk 2.0. It’s available from the project’s GitHub page.

   • MKS Awk was used for Solaris’s /usr/xpg4/bin/awk, which is their standards-compliant version of new awk. For a while it was available as part of Open Solaris, but is no longer so. Some years ago, we were able to make this version compile and run on GNU/Linux after just a few hours work.

     Although Open Solaris is now history, the Illumos project does make the MKS Awk available. You can view the files one at a time from https://github.com/joyent/illumos-joyent/blob/master/usr/src/cmd/awk_xpg4.

   • Other, more esoteric versions as well. See the Wikipedia article, and also the gawk documentation.

3.2 And What GNU Awk Has Today ¶

The more difficult of the quality of implementation issues are addressed by gawk. In particular:

7. Beginning with version 4.0 in 2011, gawk provides an awk-level debugger which is modeled after GDB. This is a full debugger, with breakpoints, watchpoints, single statement stepping and expression evaluation capabilities. (Older versions had a separate executable named dgawk. Today it’s built into regular gawk.)
8. gawk has provided an awk-level statement profiler for many years (pgawk). Although there is no direct correlation with CPU time used, the statement level profiler remains a powerful tool for understanding program behavior.
9. Since version 4.0, gawk has had an ‘@include’ facility whereby gawk goes and finds the named awk source progrm. For much longer it has searched for files specified with -f along the path named by the AWKPATH environment variable. The ‘@include’ mechanism also uses AWKPATH.
10. In terms of getting at the pieces of text matched by a regular expression, gawk provides an optional third argument to the match() function. This argument is an array which gawk fills in with both the matched text for the full regexp and subexpressions, and index and length information for use with substr(). gawk also provides the gensub() general substitution function, an enhanced version of the split() function, and the patsplit() function for specifying contents instead of separators using a regexp.

While gawk has almost always been faster than Brian Kernighan’s awk, performance improvements bring it closer to mawk’s performance level (a byte-code based execution engine and internal improvements in array indexing).

And gawk clearly has the most features of any version, many of which considerably increase the power of the language.

3.3 So Where Does Awk Stand? ¶

Despite all of the above, gawk is not as popular as other scripting languages. Since 1991, we can point to four major scripting languages which have enjoyed, or currently enjoy, differing levels of popularity: PERL, tcl/tk, Python, and Ruby. We think it is fair to say that Python is the most popular scripting languages in the third decade of the 21st century.

Is awk, as we’ve described it up to this point, now ready to compete with the other languages? Not quite yet.

5.1 API Overview ¶

The API makes it possible to write functions in C or C++ that are callable from an awk program as if the function were written in awk. The most straightforward way to think of these functions is as user-defined functions that happen to be implemented in a different language.

The API provides the following facilities:

• Structures that map awk string, numeric, and undefined values into C types that can be worked with.
• Management of function parameters, including the ability to convert a parameter whose original type is undefined, into an array. That is, there is full call-by-reference for arrays. Scalars are passed by value, of course.
• Access to the symbol table. Extension functions can read all awk variables, and create and update new variables. As an initial, relatively arbitrary design decision, extensions cannot update special variables such as NR or NF, with the single exception of PROCINFO.
• Full array management, including the ability to create arrays, and arrays of arrays, and the ability to add and delete elements from an array. It is also possible to “flatten” an array into a data structure that makes it simple for C code to loop over all the elements of an array.
• The ability to run a procedure when gawk exits. This is conceptually the same as the C atexit() function.
• Hooks into the built-in I/O redirection mechanisms in gawk. In particular, there are separate facilities for input redirections with getline and ‘<’, output redirections with print or printf and ‘>’ or ‘>>’, and two-way pipelines with gawk’s ‘|&’ operator.

5.2 Discussion ¶

Considerable thought went into the design of the API. The gawk documentation provides a full description of the API itself, with examples (over 50 pages worth!), as well as some discussion of the goals and design decisions behind the API (in an appendix). The development was done over the course of about a year and a half, together with the developers of xgawk, a fork of gawk that added features that made using extensions easier, and included an extension for processing XML files in a way that fit naturally with the pattern–action paradigm. While it may not be perfect, the gawk developers feel that it is a good start.

FIXME: Henry Spencer suggests adding more info on the API and on the design decisions. I think this paper is long enough, and the full doc is quite big. It’d be hard to pull API doc into this paper in a reasonable fashion, although it would be possible to review some of the design decisions. Comments?

The major xgawk additions to the C code base have been merged into gawk, and the extensions from that project have been rewritten to use the new API. As a result, the xgawk project developers renamed their project gawkextlib, and the project now provides only extensions.⁵

It is notable that functions written in awk can do a number of things that extension functions cannot, such as modify any variables, do I/O, call awk built-in functions, and call other user-defined functions.

While it would certainly be possible to provide APIs for all of these features for extension functions, this seemed to be overkill. Instead, the gawk developers took the view that extension functions should provide access to external facilities, and provide communication to the awk level via function parameters and/or global variables, including associative arrays, which are the only real data structure.

Consider a simple example. The standard du program can recursively walk one or more arbitrary file hierarchies, call stat() to retrieve file information, and then sum up the blocks used. In the process, du must track hard links, so that no file is accounted for or reported more than once.

The ‘filefuncs’ extension shipped with gawk provides a stat() function that takes a pathname and fills in an associative array with the information retrieved from stat(). The array elements have names like "size", "mtime" and so on, with corresponding appropriate values. (Compare this to PERL’s stat() function that returns a linearly-indexed array!)

The fts() function in the ‘filefuncs’ extension builds on stat() to create a multidimensional array of arrays that describes the requested file hierarchies, with each element being an array filled in by stat(). Directories are arrays containing elements for each directory entry, with an element named "." for the array itself.

Given that fts() does the heavy lifting, du can be written quite nicely, and quite portably⁶, in awk. See Awk Code For du, for the code, which weighs in at under 250 lines. Much of this is comments and argument parsing.

5.3 Future Work ¶

The extension facility is relatively new, and undoubtedly has introduced new “dark corners” into gawk. These remain to be uncovered and any new bugs need to be shaken out and removed.

Some issues are known and may not be resolvable. For example, 64-bit integer values such as the timestamps in stat() data on modern systems don’t fit into awk’s 64-bit double-precision numbers which only have 53 bits of significand. This is also a problem for the bit-manipulation functions.

With respect to namespaces, in 2017 I (finally) figured out how namespaces in awk ought to work to provide the needed functionality while retaining backwards compatibility. The was released with gawk 5.0.

One or two of the sample extensions shipped with gawk and in gawkextlib have been modified to take advantage of namespaces.