1.1 Goals

Here are my personal goals for bringing new maintainers on board. Over the next few years, I want to:

• Do a thorough review of the manual, and get it published again.
• Bring you two up to speed to the point that each of you would be able to:
  • — Fix bugs as reported on the bug list
  • — Make releases
  • — Make improvements in the code base
• This will allow me to become “Maintainer Emeritus”. In that position I hope that I would still be contributing, but no longer bearing the main responsibility.

This won’t happen overnight. The code base is large and there’s a lot to come up to speed on, and I don’t expect miracles, we all have lives and day jobs. “Slow but steady” will be the watch word.

1.2 Some History

The manual states:

Paul Rubin wrote gawk in 1986. Jay Fenlason completed it, with advice from Richard Stallman. John Woods contributed parts of the code as well. In 1988 and 1989, David Trueman, with help from me, thoroughly reworked gawk for compatibility with the newer awk. Circa 1994, I became the primary maintainer. Current development focuses on bug fixes, performance improvements, standards compliance, and, occasionally, new features.

In October of 1987, I picked up a copy of the Aho, Kernighan and Weinberger book on awk. Up until then, the documentation on awk was so skimpy that I’d stayed away from it; the book made things much clearer, and also more exciting! I wanted to give (new) awk a try.

New awk wasn’t easily available at the time, so I went to see what the GNU project had, and lo and behold, there was indeed a GNU awk. This very first version was a (buggy!) clone of the original Unix awk. Having an interest in compiler and interpreter technology, I volunteered to work on gawk and help bring it into compatibility with the new version. (I was single and owned my own Unix system...)

David Trueman had already volunteered to do that, so I joined his effort. We set up UUCP links and exchanged diffs via email; he was the primary maintainer.

At some point, I asked if there was a manual; a 90 page Texinfo draft existed (written by Diane Barlow Close) and I started adding to and improving it.

David and I met a time or two in person at USENIX conferences; I may even still have some notes on our feature plans, most of which eventually made their way into gawk.

Circa 1994 (I don’t remember if it was ’93 or ’94) David had to pull out of the project, and I became the sole maintainer. We had a port to MS-DOS and also to VMS around then, and mawk had also been written and released. There were also various ports to different kinds of then-common Unix systems.

In the mid-90s gawk got some heavy-duty use (Rick Adams as UUNET started to use it for his accounting because of the “no arbitrary limits” principle), but people were still using the Unix version mostly.

As Linux systems started to grow in popularity, gawk became the installed version of awk, and it really began to stabilize and get many bugs worked out. When it was just “also” available, it wasn’t as heavily stressed as when it was the awk. So, this was good.

The manual has a section listing in detail the order in which features were added to gawk over time. That list is very long, so there’s no need to repeat it here.

The major versions of gawk through 3.x all built a parse tree representing the program, and then recursively evaluated that tree in order to execute the program.

At version 4.0, that changed, when an outside developer (John Haque) changed gawk to generate and interpret “byte codes” representing the program. As part of that work, John also added an awk-level debugger, which is really useful. This structure remains in place today.

John Haque left the project in December of 2012 over a disagreement with me as to how the project should be run. He wanted to see things being done in a certain way, which I did not have the cycles to do, but he himself did not want to help make those changes. (I had offered to promote him to co-maintainer, but he declined). Effectively, “here is what you should do, but I’m not going to help.” Since that didn’t suit me, he left in a bit of huff, but that’s life.

Today, I feel like I have stretched the awk language as much as it should be (more really). I think future facilities should be added via the extension mechanism and not via language changes.

There remains plenty of interesting work to do in terms of the code structure, data structures, and performance, which I no longer have the motivation to really tackle; I’m hoping new blood will be able to do some of these things. :-)

For what it’s worth, as many versions as I could find are in the git repository and tagged appropriately, so it’s possible to go back and see the earlier versions.

1.3 First Steps

Here are some first steps for every new co-maintainer candidate:

1. Start the process to sign paperwork to assign copyright in your changes to gawk and its documentation to the Free Software Foundation. Please send an email to copyright-clerk@fsf.org asking for paperwork to assign copyright in all current and future changes to gawk and its documentation to the FSF. You will have to do this before any patches can be integrated into the code base.
2. Check out the gawk documentation online at the GNU project. In particular, look at the gawk workflow document for how things will start out for your contributions.
3. Also, if you haven’t read the gawk manual, please do. You will probably know a lot of what’s there, but maybe not everything. There’s a lot there. The section on contributing code is most relevant in terms of how patches need to be done, but you’ll need to learn the language itself as well.

2.1 Workflow and Tooling Outline

This section provides some background introductory material and pointers.

• GNU Coding Standards
• maintain.texi
• Savannah
• Mailing list or bug tracker
• Release management

git clone https://git.savannah.gnu.org/git/gnulib.git
cd gnulib/doc
texi2pdf standards.texi

cd gnulib/doc
texi2pdf maintain.texi

2.2 Technical Skills

Here are some the technical skills you’ll need as a maintainer. These all also apply pretty much across the board, not just to gawk.

• Git skills
• C (and maybe also C++)
• Make
• Autoconf, Automake, libtool, GNU m4
• Texinfo
• troff -man — semi-optional
• GDB, valgrind, and gprof
• The GCC Compile Farm

3.1 Compiler / Interpreter Skills

You should be generally familiar with the terminology and components of compilers and interpreters. Things like: parsing, scanning, and symbol tables. In particular, gawk uses a hand-written scanner and a Bison parser (LALR, based on YACC). The Flex and Bison book by John Levine is probably a good resource if you have no background in this.

As gawk parses a program, it builds up a linked list of so-called “byte codes,” representing the program to be run. This is “compiling” and “code generation” of a sort, but the instructions generated are simply structs representing operations defined by gawk itself, not code for a real machine or for any standard virtual machine such as the Java JVM or the .Net CLR.

Once the progam is built, gawk then interprets the byte codes, by way of a loop around a gigantic switch statement on each type of instruction.

The bytes codes are the basis also for the debugger and the pretty printer. The pretty printer interprets the byte codes to reconstitute awk source code. The pretty printer also handles line count profiling.

The debugger remains something of a black box to me; I’m not sure I can explain how it works off the top of my head. (This is sad.)

Future notes will talk about some of these things in more detail.

3.2 Learn The Language

To maintain gawk, you have to know awk, as well as what gawk provides in addition to standard awk. There are three major resources for learning the language and for knowing how it’s supposed to work.

1. The AWK Programming Language by Aho, Kernighan and Weinberger. The first edition of this book is the classic reference for awk, despite it being 35 years old. The second edition came out in 2023, and it has been completely rewritten to reflect the changes in BWK awk. It is compact, but very worthwhile.
2. The manual, GAWK: Effective AWK Programming, by yours truly. The manual has grown over the years, as gawk has grown. I’ve done my best to make it both tutorial and reference for both standard awk and for gawk’s capabilities. If you want to print it out, here are the steps:

   cd /path/to/gawk/repo
   ./bootstrap.sh && ./configure && make
   cd doc
   make pdf
   

   The gawk.pdf file will be the manual. You’ll need all the right TeX and Texinfo bits installed for this. Alternatively, you can find it online in various formats at https://www.gnu.org/software/gawk/manual/.

   Eventually y’all will be maintaining this manual, so you should become familiar with it in any case.

   You should read the other Texinfo files too, at your leisure: gawkinet.pdf for networking with gawk, and gawkworkflow.pdf. The workflow document is most relevant for getting started working with the team.

3. The POSIX standard for awk. This can be found online at: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html.

   It is the official standard for awk. It’s written in more formal prose than the AK&W book or the gawk manual. It’s handy to have a copy of this to refer to, to know what POSIX specifies and what it doesn’t. In particular, gawk --posix is supposed to be (mostly) POSIX compliant, meaning that all gawk extensions are disabled.

   There are a few (minor) areas where gawk --posix doesn’t follow the standard, since doing so makes it too painful to use. See the POSIX.STD file in the distribution (or Git).

3.3 Test Suite Structure

The test suite is fairly formalized. For each test foo, there is at least a foo.awk and a foo.ok file. Optionally, there may be a foo.in file, which is input data for foo.awk to process. foo.ok is the combined standard output and standard error from running gawk on foo.awk (with foo.in, if relevant).

Adding new tests generally does not require actually writing the test stanzas. Instead the Makefile process takes care of this, using an awk script that reads Makefile.am and generates the test stanzas for any tests that have not been written manually.

To add a new test, first be sure that configure and make have already been run. You should also have the “Autotools” installed, using the same versions as I do. In that case, the steps to add a test involve editing Makefile.am:

1. In the list of files for EXTRA_DIST, add foo.awk and foo.ok. Add foo.in also if relevant. Keep this list sorted alphabetically.
2. Add the new test name in the relevant list:

   ARRAYDEBUG_TESTS	Array debugging tests.
   BASIC_TESTS	Tests for standard awk functionality.
   EXTRA_TESTS	Tests we don’t usually run. (I should rethink these, maybe.)
   GAWK_EXT_TESTS	Tests of gawk-specific extensions.
   INET_TESTS	Networking facility tests.
   LOCALE_CHARSET_TESTS	Tests whose results depend on different locales.
   MACHINE_TESTS	Tests that may vary based on machine architecture.
   SHLIB_TESTS	Tests of the extension facility.
   UNIX_TESTS	Tests that require *nix.

3. In addition, there are a number of NEED_xxx groups listing tests that need additional gawk options, such as --lint or --debug. The Makefile doesn’t use them directly. Instead, the test/Gentests script, which generates the test stanzas, reads Makefile.am and processes these lists in order to generate the test stanzas. I recommend reviewing the test/Gentests script as well.
4. After saving Makefile.am, just run make, and everything will be regenerated. This should be done in the top level directory. This will also update pc/Makefile.tst, which is used for running the tests on MinGW.

3.4 Documentation

The doc directory contains all of gawk’s documentation. This includes manuals in Texinfo as well as man pages, and a four-color reference card, written in tbl and troff.

The various files are:

ChangeLog*

ChangeLog files tracking changes.

Makefile.am

Makefile.in

Makefile stuff via the Autotools.

README.card

ad.block

awkcard.in

cardfonts

colors

macros

no.colors

setter.outline

The files for making the reference card.

api-figure*.fig

api-figure*.*

array-elements.fig

array-elements.*

general-program.fig

general-program.*

lflashlight-small.xpic

lflashlight.*

process-flow.fig

process-flow.*

rflashlight-small.xpic

rflashlight.*

statist.*

Image files used in the manual, both the original from the drawing programs (.fig and .xpic) and the generated files.

awkforai.txt

A draft of an article for ACM SIGPLAN Notices on why awk is good for AI.

bc_notes

implementation-notes.txt

Some implementation notes. (I should review these.)

gawk.1

gawkbug.1

igawk.1

pm-gawk.1

Manual pages. igawk.1 is no longer installed.

gawkinet.texi

gawk.texi

gawkworkflow.texi

pm-gawk.texi

texinfo.tex

The Texinfo manuals.

it/*

The Italian translation of the manual and of the man pages¹.

wordlist*

Lists of valid words that my spell(1) doesn’t think are valid. Used for spell checking the various documentation files.

3.5 Portability: POSIX and Non-POSIX

There are places in gawk where it needs to find out OS-specific information or do an OS-specific task. These are abstracted into the following routines:

os_arg_fixup(int *argcp, char ***argvp)
os_devopen(const char *name, int flag)
os_close_on_exec(int fd, const char *name, const char *what, const char *dir)
os_isdir(int fd)
os_isreadable(const awk_input_buf_t *iobuf, bool *isdir)
os_is_setuid()
os_setbinmode(int fd, int mode)
os_restore_mode(int fd)
os_isatty(int fd)
os_maybe_set_errno(void)

There are three different implementations of these routines, one for OpenVMS, one for Windows with MinGW, and one for everything else (POSIX).

In the top level directory, the gawkmisc.c file includes a subsidiary file for each environment, depending upon the system where gawk is being compiled.

On POSIX systems, this is posix/gawkmisc.c. On OpenVMS, this is vms/gawkmisc.vms, and on Windows with MinGW, this is pc/gawkmisc.pc. Each of these files are C code that define the above routines. On some of the systems, these routines return default or error values, as appropriate.

In this way, the mainline gawk code avoids lots of #ifdef statements, yet the ‘os_’ prefix provides an indication that the operation being performed is OS-specific.

If you look at the files, you will see that they are very similar to each other.

3.6 GNULIB

In the support directory are files that gawk uses as essentially “library” or support routines. For example, the regex and dfa matchers. (Why gawk has two regular expression matchers is another story.) Almost all these files come from GNULIB.

Unlike most other projects that take files from GNULIB, all I do is take the source files. I don’t use the GNULIB infrastructure (gnulibtoolize, or whatever it’s called) to bring in all their m4 files. The reason is that they tend to pull in the transitive closure of everything any particular item depends upon. Thus the supporting infrastructure ends up overwhelming the actual code in the project. That’s both silly and wasteful of CPU time, person time, and disk space.

The script helpers/update-support.sh can be used to update the support directory’s files, although it assumes that the local git clone of the GNULIB repository lives in a certain directory.

I am the only one who runs it, although we can look into making it more general going forward.