\input texinfo

@c TODO:  Checklist for release:
@c    revise all   U P D A T E   items as appropriate
@c    check all to-do notes
@c    remove most comments
@c    spell check (last 2am 15 Aug 2022)

@c verbatim limits:  47 rows x 75 cols, smallformat 58 x 90

@macro gwk {}
@command{gawk}
@end macro

@macro pmg {}
pm-@gwk{}
@end macro

@set TYTL Persistent-Memory @gwk{} User Manual

@setfilename pm-gawk.info
@settitle @value{TYTL}

@dircategory Text creation and manipulation
@direntry
* pm-gawk: (pm-gawk).    Persistent memory version of gawk.
@end direntry

@fonttextsize 11

@c it seems to do no harm and possibly some good if color
@c distinguishes internal links from URLs to outside web
@tex
\gdef\linkcolor{0.12 0.09 .5} % TK's attempt at subdued blue
%\gdef\linkcolor{0.5 0.09 0.12} % Dark Red
\gdef\urlcolor{0.5 0.09 0.12} % Dark Red
\global\urefurlonlylinktrue
@end tex

@setchapternewpage off

@copying
@noindent
@c UPDATE copyright info below
Copyright @copyright{} 2022 Terence Kelly @*
@ifnottex
@noindent
@email{tpkelly@@eecs.umich.edu} @*
@email{tpkelly@@cs.princeton.edu} @*
@email{tpkelly@@acm.org} @*
@url{http://web.eecs.umich.edu/~tpkelly/pma/} @*
@url{https://dl.acm.org/profile/81100523747}
@end ifnottex

@noindent
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3
or any later version published by the Free Software Foundation;
with the Invariant Sections being ``Introduction'' and ``History'',
no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is available at @*
@url{https://www.gnu.org/licenses/fdl-1.3.html}
@end copying

@titlepage
@title @value{TYTL}
@c UPDATE date below
@subtitle 16 August 2022
@subtitle @gwk{} version 5.2
@subtitle @pmg{} version 2022.08Aug.03.1659520468 (Avon 7)
@author Terence Kelly
@author @email{tpkelly@@eecs.umich.edu}
@author @email{tpkelly@@cs.princeton.edu}
@author @email{tpkelly@@acm.org}
@author @url{http://web.eecs.umich.edu/~tpkelly/pma/}
@author @url{https://dl.acm.org/profile/81100523747}
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@headings off

@c @contents  @c no need for this in a short document

@node Top
@ifnottex
@ifnotxml
@ifnotdocbook
@top  General Introduction
@gwk{} 5.2 introduces a @emph{persistent memory} feature that can
``remember'' script-defined variables and functions across executions;
pass variables between unrelated scripts without serializing/parsing
text files; and handle data sets larger than available memory plus
swap.  This supplementary manual provides an in-depth look at
persistent-memory @gwk{}.

@insertcopying
@end ifnotdocbook
@end ifnotxml
@end ifnottex

@menu
* Introduction::
* Quick Start::
* Examples::
* Performance::
* Data Integrity::
* Acknowledgments::
* Installation::
* Debugging::
* History::
@end menu

@c ==================================================================
@node    Introduction
@chapter Introduction

@sp 1

@c UPDATE below after official release
GNU AWK (@gwk{}) 5.2, expected in September 2022, introduces a new
@emph{persistent memory} feature that makes AWK scripting easier and
sometimes improves performance.  The new feature, called ``@pmg{},''
can ``remember'' script-defined variables and functions across
executions and can pass variables and functions between unrelated
scripts without serializing/parsing text files---all with near-zero
fuss.  @pmg{} does @emph{not} require non-volatile memory hardware nor
any other exotic infrastructure; it runs on the ordinary conventional
computers and operating systems that most of us have been using for
decades.

@sp 1

@c TODO:  ADR:  hyperlinks to info page below

@noindent
The main @gwk{} documentation@footnote{See
@url{https://www.gnu.org/software/gawk/manual/} @w{ } and @w{ }
@code{man gawk} @w{ } and @w{ } @code{info gawk}.}  covers the basics
of the new persistence feature.  This supplementary manual provides
additional detail, tutorial examples, and a peek under the hood of
@pmg{}.  If you're familiar with @gwk{} and Unix-like environments,
dive straight in: @*

@itemize @c @w{}
@item @ref{Quick Start} hits the ground running with a few keystrokes.
@item @ref{Examples} shows how @pmg{} streamlines typical AWK scripting.
@item @ref{Performance} covers asymptotic efficiency, OS tuning, and more.
@item @ref{Data Integrity} explains how to protect data from mishaps.
@item @ref{Acknowledgments} thanks those who made @pmg{} happen.
@item @ref{Installation} shows where obtain @pmg{}.
@item @ref{Debugging} explains how to handle suspected bugs.
@item @ref{History} traces @pmg{}'s persistence technology.
@end itemize

@c UPDATE:  revise above when content finalized

@sp 1

@noindent
You can find the latest version of this manual, and also the
``director's cut,'' at the web site for the persistent memory
allocator used in @pmg{}: @*
@center @url{http://web.eecs.umich.edu/~tpkelly/pma/}

@sp 1

@noindent
Two publications describe the persistent memory allocator and early
experiences with a @pmg{} prototype based on a fork of the official
@gwk{} sources:
@itemize
@item @url{https://queue.acm.org/detail.cfm?id=3534855}
@item @url{http://nvmw.ucsd.edu/nvmw2022-program/nvmw2022-data/nvmw2022-paper35-final_version_your_extended_abstract.pdf}
@end itemize

@sp 1

@noindent
Feel free to send me questions, suggestions, and experiences: @*

@noindent
@w{ } @w{ } @w{ } @w{ } @w{ } @w{ } @w{ } @w{ }  @email{tpkelly@@eecs.umich.edu} @w{ } (preferred) @*
@w{ } @w{ } @w{ } @w{ } @w{ } @w{ } @w{ } @w{ }  @email{tpkelly@@cs.princeton.edu} @*
@w{ } @w{ } @w{ } @w{ } @w{ } @w{ } @w{ } @w{ }  @email{tpkelly@@acm.org}

@page
@c ==================================================================
@node    Quick Start
@chapter Quick Start

@c example heaps are larger than strictly necessary so that readers
@c who use them more extensively are less likely to exhaust memory

Here's @pmg{} in action at the @command{bash} shell prompt (@samp{$}):
@verbatim
        $ truncate -s 4096000 heap.pma
        $ export GAWK_PERSIST_FILE=heap.pma
        $ gawk 'BEGIN{myvar = 47}'
        $ gawk 'BEGIN{myvar += 7; print myvar}'
        54
@end verbatim
@noindent
First, @command{truncate} creates an empty (all-zero-bytes) @dfn{heap
file} where @pmg{} will store script variables; its size is a multiple
of the system page size (4@tie{}KiB).  Next, @command{export} sets an
environment variable that enables @pmg{} to find the heap file; if
@gwk{} does @emph{not} see this envar, persistence is not activated.
The third command runs a one-line AWK script that initializes variable
@code{myvar}, which will reside in the heap file and thereby outlive
the interpreter process that initialized it.  Finally, the fourth
command invokes @pmg{} on a @emph{different} one-line script that
increments and prints @code{myvar}.  The output shows that @pmg{} has
indeed ``remembered'' @code{myvar} across executions of unrelated
scripts.  (If the @gwk{} executable in your search @env{$PATH} lacks
the persistence feature, the output in the above example will be
@samp{7} instead of @samp{54}. @xref{Installation}.)  To disable
persistence until you want it again, prevent @gwk{} from finding the
heap file via @command{unset GAWK_PERSIST_FILE}.  To permanently
``forget'' script variables, delete the heap file.

@sp 2

Toggling persistence by @command{export}-ing and @command{unset}-ing
``ambient'' envars requires care: Forgetting to @command{unset} when
you no longer want persistence can cause confusing bugs.  Fortunately,
@command{bash} allows you to pass envars more deliberately, on a
per-command basis:
@verbatim
        $ rm heap.pma                    # start fresh
        $ unset GAWK_PERSIST_FILE        # eliminate ambient envar
        $ truncate -s 4096000 heap.pma   # create new heap file

        $ GAWK_PERSIST_FILE=heap.pma gawk 'BEGIN{myvar = 47}'
        $ gawk 'BEGIN{myvar += 7; print myvar}'
        7
        $ GAWK_PERSIST_FILE=heap.pma gawk 'BEGIN{myvar += 7; print myvar}'
        54
@end verbatim
@noindent
The first @gwk{} invocation sees the special envar prepended on the
command line, so it activates @pmg{}.  The second @gwk{} invocation,
however, does @emph{not} see the envar and therefore does not access
the script variable stored in the heap file.  The third @gwk{}
invocation does see the special envar and therefore uses the script
variable from the heap file.

While sometimes less error prone than ambient envars, per-command
envar passing as shown above is verbose and shouty.  A shell alias
saves keystrokes and reduces visual clutter:
@verbatim
        $ alias pm='GAWK_PERSIST_FILE=heap.pma'
        $ pm gawk 'BEGIN{print ++myvar}'
        55
        $ pm gawk 'BEGIN{print ++myvar}'
        56
@end verbatim

@page
@c ==================================================================
@node    Examples
@chapter Examples

Our first example uses @pmg{} to streamline analysis of a prose
corpus, Mark Twain's @cite{Tom Sawyer} and @cite{Huckleberry Finn}
from
@c
@url{https://gutenberg.org/files/74/74-0.txt}
@c
and
@c
@url{https://gutenberg.org/files/76/76-0.txt}.
@c
We first convert non-alphabetic characters to newlines (so each line
has at most one word) and convert to lowercase:
@verbatim
        $ tr -c a-zA-Z '\n' < 74-0.txt | tr A-Z a-z > sawyer.txt
        $ tr -c a-zA-Z '\n' < 76-0.txt | tr A-Z a-z > finn.txt
@end verbatim

It's easy to count word frequencies with AWK's associative arrays.
@pmg{} makes these arrays persistent, so we need not re-ingest the
entire corpus every time we ask a new question (``read once, analyze
happily ever after''):
@verbatim
        $ truncate -s 100M twain.pma
        $ export GAWK_PERSIST_FILE=twain.pma
        $ gawk '{ts[$1]++}' sawyer.txt                    # ingest
        $ gawk 'BEGIN{print ts["work"], ts["play"]}'      # query
        92 11
        $ gawk 'BEGIN{print ts["necktie"], ts["knife"]}'  # query
        2 27
@end verbatim
@noindent
The @command{truncate} command above creates a heap file large enough
to store all of the data it must eventually contain, with plenty of
room to spare.  (As we'll see in @ref{Sparse Heap Files}, this isn't
wasteful.)  The @command{export} command ensures that subsequent
@gwk{} invocations activate @pmg{}.  The first @pmg{} command stores
@cite{Tom Sawyer}'s word frequencies in associative array @code{ts[]}.
Because this array is persistent, subsequent @pmg{} commands can
access it without having to parse the input file again.

Expanding our analysis to encompass a second book is easy.  Let's
populate a new associative array @code{hf[]} with the word frequencies
in @cite{Huckleberry Finn}:
@verbatim
        $ gawk '{hf[$1]++}' finn.txt
@end verbatim
@noindent
Now we can freely intermix accesses to both books' data conveniently
and efficiently, without the overhead and coding fuss of repeated
input parsing:
@verbatim
        $ gawk 'BEGIN{print ts["river"], hf["river"]}'
        26 142
@end verbatim

By making AWK more interactive, @pmg{} invites casual conversations
with data.  If we're curious what words in @cite{Finn} are absent from
@cite{Sawyer}, answers (including ``flapdoodle,'' ``yellocution,'' and
``sockdolager'') are easy to find:
@verbatim
        $ gawk 'BEGIN{for(w in hf) if (!(w in ts)) print w}'
@end verbatim
@c also: doxolojer meedyevil ridicklous dingnation gumption cavortings
@c       phrenology [words about slavery] shakespeare camelopard ope
@c       mesmerism sapheads disremember consekens prevarication
@c       missionaryin cannibal nebokoodneezer sentimentering palavering

Rumors of Twain's death may be exaggerated.  If he publishes new books
in the future, it will be easy to incorporate them into our analysis
incrementally.  The performance benefits of incremental processing for
common AWK chores such as log file analysis are discussed in
@url{https://queue.acm.org/detail.cfm?id=3534855} and the companion
paper cited therein, and below in @ref{Performance}.
@c @url{http://nvmw.ucsd.edu/nvmw2022-program/nvmw2022-data/nvmw2022-paper35-final_version_your_extended_abstract.pdf}.

Exercise: The ``Markov'' AWK script on page 79 of Kernighan & Pike's
@cite{The Practice of Programming} generates random text reminiscent
of a given corpus using a simple statistical modeling technique.  This
script consists of a ``learning'' or ``training'' phase followed by an
output-generation phase.  Use @pmg{} to de-couple the two phases and
to allow the statistical model to incrementally ingest additions to
the input corpus.

@page
@c  = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

Our second example considers another domain that plays to AWK's
strengths, data analysis.  For simplicity we'll create two small input
files of numeric data.
@verbatim
        $ printf '1\n2\n3\n4\n5\n' > A.dat
        $ printf '5\n6\n7\n8\n9\n' > B.dat
@end verbatim
@noindent
A conventional @emph{non}-persistent AWK script can compute basic
summary statistics:
@verbatim
        $ cat summary_conventional.awk
            1 == NR  { min = max = $1 }
            min > $1 { min  = $1 }
            max < $1 { max  = $1 }
                     { sum += $1 }
            END { print "min: " min " max: " max " mean: " sum/NR }

        $ gawk -f summary_conventional.awk A.dat B.dat
        min: 1 max: 9 mean: 5
@end verbatim

To use @pmg{} for the same purpose, we first create a heap file for
our AWK script variables and tell @pmg{} where to find it via the
usual environment variable:
@verbatim
        $ truncate -s 10M stats.pma
        $ export GAWK_PERSIST_FILE=stats.pma
@end verbatim
@noindent
@pmg{} requires changing the above script to ensure that @code{min}
and @code{max} are initialized exactly once, when the heap file is
first used, and @emph{not} every time the script runs.  Furthermore,
whereas script-defined variables such as @code{min} retain their
values across @pmg{} executions, built-in AWK variables such as
@code{NR} are reset to zero every time @pmg{} runs, so we can't use
them in the same way.  Here's a modified script for @pmg{}:
@verbatim
        $ cat summary_persistent.awk
            ! init   { min = max = $1; init = 1 }
            min > $1 { min = $1 }
            max < $1 { max = $1 }
                     { sum += $1; ++n }
            END { print "min: " min " max: " max " mean: " sum/n }
@end verbatim
@noindent
Note the different pattern on the first line and the introduction of
@code{n} to supplant @code{NR}.  When used with @pmg{}, this new
initialization logic supports the same kind of cumulative processing
that we saw in the text-analysis scenario.  For example, we can ingest
our input files separately:
@verbatim
        $ gawk -f summary_persistent.awk A.dat
        min: 1 max: 5 mean: 3

        $ gawk -f summary_persistent.awk B.dat
        min: 1 max: 9 mean: 5
@end verbatim
@noindent
As expected, after the second @pmg{} invocation consumes the
second input file, the output matches that of the non-persistent
script that read both files at once.

Exercise: Amend the AWK scripts above to compute the median and
mode(s) using both conventional @gwk{} and @pmg{}.  (The median is the
number in the middle of a sorted list; if the length of the list is
even, average the two numbers at the middle.  The modes are the values
that occur most frequently.)

@c heaps not portable across machines, use only with same gawk executable (?)
@c    refer to gawk docs for portability constraints on heaps
@c can use only one heap at a time

@page
@c  = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

Our third and final set of examples shows that @pmg{} allows us to
bundle both script-defined data and also user-defined @emph{functions}
in a persistent heap that may be passed freely between unrelated AWK
scripts.

@c ADR doesn't like return in count() below
@c TK:  it was put there for a reason:
@c $ truncate -s 10M funcs.pma
@c $ export GAWK_PERSIST_FILE=funcs.pma
@c $ gawk 'function count(A,t) {for(i in A)t++; return t}'
@c $ gawk 'BEGIN { a["x"] = 4; a["y"] = 5; a["z"] = 6 }'
@c $ gawk 'BEGIN { print count(a) }'
@c 3
@c $ gawk 'BEGIN { delete a }'
@c $ gawk 'BEGIN { print count(a) }'
@c [!!blank line, not zero!!]
@c $

The following shell transcript repeatedly invokes @pmg{} to create and
then employ a user-defined function.  These separate invocations
involve several different AWK scripts that communicate via the heap
file.  Each invocation can add user-defined functions and add or
remove data from the heap that subsequent invocations will access.
@smallformat
@verbatim
        $ truncate -s 10M funcs.pma
        $ export GAWK_PERSIST_FILE=funcs.pma
        $ gawk 'function count(A,t) {for(i in A)t++; return ""==t?0:t}'
        $ gawk 'BEGIN { a["x"] = 4; a["y"] = 5; a["z"] = 6 }'
        $ gawk 'BEGIN { print count(a) }'
        3
        $ gawk 'BEGIN { delete a["x"] }'
        $ gawk 'BEGIN { print count(a) }'
        2
        $ gawk 'BEGIN { delete a }'
        $ gawk 'BEGIN { print count(a) }'
        0
        $ gawk 'BEGIN { for (i=0; i<47; i++) a[i]=i }'
        $ gawk 'BEGIN { print count(a) }'
        47
@end verbatim
@end smallformat
@noindent
The first @pmg{} command creates user-defined function @code{count()},
which returns the number of entries in a given associative array; note
that variable @code{t} is local to @code{count()}, not global.  The
next @pmg{} command populates a persistent associative array with
three entries; not surprisingly, the @code{count()} call in the
following @pmg{} command finds these three entries.  The next two
@pmg{} commands respectively delete an array entry and print the
reduced count, 2.  The two commands after that delete the entire array
and print a count of zero.  Finally, the last two @pmg{} commands
populate the array with 47 entries and count them.

@c I could be persuaded to leave the polynomial example as an
@c exercise, offering to send my answer to readers upon request.

The following shell script invokes @pmg{} repeatedly to create a
collection of user-defined functions that perform basic operations on
quadratic polynomials: evaluation at a given point, computing the
discriminant, and using the quadratic formula to find the roots.  It
then factorizes @math{x^2 + x - 12} into @math{(x - 3)(x + 4)}.
@smallformat
@verbatim
        #!/bin/sh
        rm -f                    poly.pma
        truncate -s 10M          poly.pma
        export GAWK_PERSIST_FILE=poly.pma
        gawk 'function q(x) { return a*x^2 + b*x + c }'
        gawk 'function p(x) { return "q(" x ") = " q(x) }'
        gawk 'BEGIN { print p(2) }'           # evaluate & print
        gawk 'BEGIN{ a = 1; b = 1; c = -12 }' # new coefficients
        gawk 'BEGIN { print p(2) }'           # eval/print again
        gawk 'function d(s) { return s * sqrt(b^2 - 4*a*c)}'
        gawk 'BEGIN{ print "discriminant (must be >=0): " d(1)}'
        gawk 'function r(s) { return (-b + d(s))/(2*a)}'
        gawk 'BEGIN{ print "root: " r( 1) "   " p(r( 1)) }'
        gawk 'BEGIN{ print "root: " r(-1) "   " p(r(-1)) }'
        gawk 'function abs(n) { return n >= 0 ? n : -n }'
        gawk 'function sgn(x) { return x >= 0 ? "- " : "+ " } '
        gawk 'function f(s) { return "(x " sgn(r(s)) abs(r(s))}'
        gawk 'BEGIN{ print "factor: " f( 1) ")" }'
        gawk 'BEGIN{ print "factor: " f(-1) ")" }'
        rm -f poly.pma
@end verbatim
@end smallformat
@noindent

@page
@c ==================================================================
@node    Performance
@chapter Performance

This chapter explains several performance advantages that result from
the implementation of persistent memory in @pmg{}, shows how tuning
the underlying operating system sometimes improves performance, and
presents experimental performance measurements.  To make the
discussion concrete, we use examples from a GNU/Linux system---GNU
utilities atop the Linux OS---but the principles apply to other modern
operating systems.

@c  = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@node    Constant-Time Array Access
@section Constant-Time Array Access

@pmg{} preserves the efficiency of data access when data structures
are created by one process and later re-used by a different process.

Consider the associative arrays used to analyze Mark Twain's books in
@ref{Examples}.  We created arrays @code{ts[]} and @code{hf[]} by
reading files @file{sawyer.txt} and @file{finn.txt}.  If @i{N} denotes
the total volume of data in these files, building the associative
arrays typically requires time proportional to @i{N}, or ``@i{O(N)}
expected time'' in the lingo of asymptotic analysis.  If @i{W} is the
number of unique words in the input files, the size of the associative
arrays will be proportional to @i{W}, or @i{O(W)}.  Accessing
individual array elements requires only @emph{constant} or @i{O(1)}
expected time, not @i{O(N)} or @i{O(W)} time, because @gwk{}
implements arrays as hash tables.

@c how much larger is N than W for the Twain texts?
@c % wc -w sawyer.txt finn.txt
@c   77523 sawyer.txt
@c  120864 finn.txt
@c  198387 total
@c % cat sawyer.txt finn.txt | sort | uniq | wc -w
@c 10447
@c
@c #words is 19x larger than #uniquewords
@c
@c Note that the total number of English words in existence is fixed,
@c so as the size of a corpus increases without bound, the ratio of
@c vocabulary size to corpus size tends toward zero.

The performance advantage of @pmg{} arises when different processes
create and access associative arrays.  Accessing an element of a
persistent array created by a previous @pmg{} process, as we did
earlier in
@c
@verb{|BEGIN{print ts["river"], hf["river"]}|},
@c
still requires only @i{O(1)} time, which is asymptotically far
superior to the alternatives.  Na@"{@dotless{i}}vely reconstructing
arrays by re-ingesting all raw inputs in every @gwk{} process that
accesses the arrays would of course require @i{O(N)} time---a
profligate cost if the text corpus is large.  Dumping arrays to files
and re-loading them as needed would reduce the preparation time for
access to @i{O(W)}.  That can be a substantial improvement in
practice; @i{N} is roughly 19 times larger than @i{W} in our Twain
corpus.  Nonetheless @i{O(W)} remains far slower than @pmg{}'s
@i{O(1)}.  As we'll see in @ref{Results}, the difference is not merely
theoretical.

The persistent memory implementation beneath @pmg{} enables it to
avoid work proportional to @i{N} or @i{W} when accessing an element of
a persistent associative array.  Under the hood, @pmg{} stores
script-defined AWK variables such as associative arrays in a
persistent heap laid out in a memory-mapped file (the heap file).
When an AWK script accesses an element of an associative array, @pmg{}
performs a lookup on the corresponding hash table, which in turn
accesses memory on the persistent heap.  Modern operating systems
implement memory-mapped files in such a way that these memory accesses
trigger the bare minimum of data movement required: Only those parts
of the heap file containing needed data are ``paged in'' to the memory
of the @pmg{} process.  In the worst case, the heap file is not in the
file system's in-memory cache, so the required pages must be faulted
into memory from storage.  Our asymptotic analysis of efficiency
applies regardless of whether the heap file is cached or not.  The
entire heap file is @emph{not} accessed merely to access an element of
a persistent associative array.

Persistent memory thus enables @pmg{} to offer the flexibility of
de-coupling data ingestion from analytic queries without the fuss and
overhead of serializing and loading data structures and without
sacrificing constant-time access to the associative arrays that make
AWK scripting convenient and productive.

@c Further details on @pmg{}'s persistent heap are available in
@c @url{https://queue.acm.org/detail.cfm?id=3534855}
@c
@c and [excessively long NVMW URL below]
@c
@c @url{http://nvmw.ucsd.edu/nvmw2022-program/nvmw2022-data/nvmw2022-paper35-final_version_your_extended_abstract.pdf}.

@page
@c  = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@node    Virtual Memory and Big Data
@section Virtual Memory and Big Data

Small data sets seldom spoil the delights of AWK by causing
performance troubles, with or without persistence.  As the size of the
@gwk{} interpreter's internal data structures approaches the capacity
of physical memory, however, acceptable performance requires
understanding modern operating systems and sometimes tuning them.
Fortunately @pmg{} offers new degrees of control for
performance-conscious users tackling large data sets.  A terse
mnemonic captures the basic principle: Precluding paging promotes peak
performance and prevents perplexity.

Modern operating systems feature @dfn{virtual memory} that strives to
appear both larger than installed DRAM (which is small) and faster
than installed storage devices (which are slow).  As a program's
memory footprint approaches the capacity of DRAM, the virtual memory
system transparently @dfn{pages} (moves) the program's data between
DRAM and a @dfn{swap area} on a storage device.  Paging can degrade
performance mildly or severely, depending on the program's memory
access patterns.  Random accesses to large data structures can trigger
excessive paging and dramatic slowdown.  Unfortunately, the hash
tables beneath AWK's signature associative arrays inherently require
random memory accesses, so large associative arrays can be
problematic.

@c ADR comments regarding below, "SSDs alleviate much of the
@c performance problem of hard disks vs. RAM disks."
@c
@c TK replies:  When a significant amount of paging to *any*
@c conventional block storage device starts, speed plummets by orders
@c of magnitude.  I'd wager that the difference between paging vs. not
@c is larger than the difference between paging to SSD vs. HDD.  So
@c while SSDs are faster than HDDs, when paging begins they won't
@c usually make the difference between acceptable vs. unacceptable
@c performance.
@c
@c If you decide to try to find out for yourself, note that on many
@c Linux systems the notorious OOM killer terminates a process well
@c before its anonymous-memory footprint reaches the capacity of DRAM,
@c so it's difficult even to provoke the phenomenon of interest.  And
@c paging anonymous memory to swap can be managed differently than
@c paging involving a file-backed memory mapping.
@c
@c Bottom line:  For the large majority of purposes, gawk users
@c would be wise to avoid paging entirely, regardless of whether
@c the pm-gawk feature is used.

Persistence changes the rules in our favor: The OS pages data to
@pmg{}'s @emph{heap file} instead of the swap area.  This won't help
performance much if the heap file resides in a conventional
storage-backed file system.  On Unix-like systems, however, we may
place the heap file in a DRAM-backed file system such as
@file{/dev/shm/}, which entirely prevents paging to slow storage
devices.  Temporarily placing the heap file in such a file system is a
reasonable expedient, with two caveats: First, keep in mind that
DRAM-backed file systems perish when the machine reboots or crashes,
so you must copy the heap file to a conventional storage-backed file
system when your computation is done.  Second, @pmg{}'s memory
footprint can't exceed available DRAM if you place the heap file in a
DRAM-backed file system.

Tuning OS paging parameters may improve performance if you decide to
run @pmg{} with a heap file in a conventional storage-backed file
system.  Some OSes have unhelpful default habits regarding modified
(``dirty'') memory backed by files.  For example, the OS may write
dirty memory pages to the heap file periodically and/or when the OS
believes that ``too much'' memory is dirty.  Such ``eager'' writeback
can degrade performance noticeably and brings no benefit to @pmg{}.
Fortunately some OSes allow paging defaults to be over-ridden so that
writeback is ``lazy'' rather than eager.  For Linux see the discussion
of the @code{dirty_*} parameters at
@url{https://www.kernel.org/doc/html/latest/admin-guide/sysctl/vm.html}.
Changing these parameters can prevent wasteful eager
paging:@footnote{The @command{tee} rigmarole is explained at
@url{https://askubuntu.com/questions/1098059/which-is-the-right-way-to-drop-caches-in-lubuntu}.}
@verbatim
        $ echo 100    | sudo tee /proc/sys/vm/dirty_background_ratio
        $ echo 100    | sudo tee /proc/sys/vm/dirty_ratio
        $ echo 300000 | sudo tee /proc/sys/vm/dirty_expire_centisecs
        $ echo 50000  | sudo tee /proc/sys/vm/dirty_writeback_centisecs
@end verbatim
@noindent
Tuning paging parameters can help non-persistent @gwk{} as well as
@pmg{}.  [Disclaimer: OS tuning is an occult art, and your mileage may
vary.]

@c sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
@c
@c sudo doesn't convey root privileges to the redirection '>' when calling from cmd line

@page
@c  = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@node    Sparse Heap Files
@section Sparse Heap Files

To be frugal with storage resources, @pmg{}'s heap file should be
created as a @dfn{sparse file}: a file whose logical size is larger
than its storage resource footprint.  Modern file systems support
sparse files, which are easy to create using the @command{truncate}
tool shown in our examples.

Let's first create a conventional @emph{non}-sparse file using
@command{echo}:
@verbatim
        $ echo hi > dense
        $ ls -l dense
        -rw-rw-r--. 1 me me 3 Aug  5 23:08 dense
        $ du -h dense
        4.0K    dense
@end verbatim
@noindent
The @command{ls} utility reports that file @file{dense} is three bytes
long (two for the letters in ``hi'' plus one for the newline).  The
@command{du} utility reports that this file consumes 4@tie{}KiB of
storage---one block of disk, as small as a non-sparse file's storage
footprint can be.  Now let's use @command{truncate} to create a
logically enormous sparse file and check its physical size:
@verbatim
        $ truncate -s 1T sparse
        $ ls -l sparse
        -rw-rw-r--. 1 me me 1099511627776 Aug  5 22:33 sparse
        $ du -h sparse
        0       sparse
@end verbatim
@noindent
Whereas @command{ls} reports the logical file size that we expect (one
TiB or 2 raised to the power 40 bytes), @command{du} reveals that the
file occupies no storage whatsoever.  The file system will allocate
physical storage resources beneath this file as data is written to it;
reading unwritten regions of the file yields zeros.

The ``pay as you go'' storage cost of sparse files offers both
convenience and control for @pmg{} users.  If your file system
supports sparse files, go ahead and create lavishly capacious heap
files for @pmg{}.  Their logical size costs nothing and persistent
memory allocation within @pmg{} won't fail until physical storage
resources beneath the file system are exhausted.  But if instead you
want to @emph{prevent} a heap file from consuming too much storage,
simply set its initial size to whatever bound you wish to enforce; it
won't eat more disk than that.  Copying sparse files with GNU
@command{cp} creates sparse copies by default.

File-system encryption can preclude sparse files: If the cleartext of
a byte offset range within a file is all zero bytes, the corresponding
ciphertext probably shouldn't be all zeros!  Encrypting at the storage
layer instead of the file system layer may offer acceptable security
while still permitting file systems to implement sparse files.

Sometimes you might prefer a dense heap file backed by pre-allocated
storage resources, for example to increase the likelihood that
@pmg{}'s internal memory allocation will succeed until the persistent
heap occupies the entire heap file.  The @command{fallocate} utility
will do the trick:
@verbatim
        $ fallocate -l 1M mibi
        $ ls -l mibi
        -rw-rw-r--. 1 me me 1048576 Aug  5 23:18 mibi
        $ du -h mibi
        1.0M    mibi
@end verbatim
@noindent
We get the MiB we asked for, both logically and physically.

@c UPDATE:  search for username in "ls" examples

@page
@c  = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@node    Persistence versus Durability
@section Persistence versus Durability

Arguably the most important general guideline for good performance in
computer systems is, ``pay only for what you
need.''@footnote{Remarkably, this guideline is widely ignored in
surprising ways.  Certain well-known textbook algorithms continue to
grind away fruitlessly long after having computed all of their
output. @*
@c
See @url{https://queue.acm.org/detail.cfm?id=3424304}.}
@c
To apply this maxim to @pmg{} we must distinguish two concepts that
are frequently conflated: persistence and durability.@footnote{In
recent years the term ``persistent memory'' has sometimes been used to
denote novel byte-addressable non-volatile memory hardware---an
unfortunate practice that contradicts sensible long-standing
convention and causes needless confusion.  NVM provides durability.
Persistent memory is a software abstraction that doesn't require NVM.
See @url{https://queue.acm.org/detail.cfm?id=3358957}.} (A third
logically distinct concept is the subject of @ref{Data Integrity}.)

@dfn{Persistent} data outlive the processes that access them, but
don't necessarily last forever.  For example, as explained in
@command{man mq_overview}, message queues are persistent because they
exist until the system shuts down.  @dfn{Durable} data reside on a
physical medium that retains its contents even without continuously
supplied power.  For example, hard disk drives and solid state drives
store durable data.  Confusion arises because persistence and
durability are often correlated: Data in ordinary file systems backed
by HDDs or SSDs are typically both persistent and durable.
Familiarity with @code{fsync()} and @code{msync()} might lead us to
believe that durability is a subset of persistence, but in fact the
two characteristics are orthogonal: Data in the swap area are durable
but not persistent; data in DRAM-backed file systems such as
@file{/dev/shm/} are persistent but not durable.

Durability often costs more than persistence, so performance-conscious
@pmg{} users pay the added premium for durability only when
persistence alone is not sufficient.  Two ways to avoid unwanted
durability overheads were discussed in @ref{Virtual Memory and Big
Data}: Place @pmg{}'s heap file in a DRAM-backed file system, or
disable eager writeback to the heap file.  Expedients such as these
enable you to remove durability overheads from the critical path of
multi-stage data analyses even when you want heap files to eventually
be durable: Allow @pmg{} to run at full speed with persistence alone;
force the heap file to durability (using the @command{cp} and
@command{sync} utilities as necessary) after output has been emitted
to the next stage of the analysis and the @pmg{} process using the
heap has terminated.

Experimenting with synthetic data builds intuition for how persistence
and durability affect performance.  You can write a little AWK or C
program to generate a stream of random text, or just cobble together a
quick and dirty generator on the command line:
@verbatim
        $ openssl rand --base64 1000000 | tr -c a-zA-Z '\n' > random.dat
@end verbatim
@noindent
Varying the size of random inputs can, for example, find where
performance ``falls off the cliff'' as @pmg{}'s memory footprint
exceeds the capacity of DRAM and paging begins.

@c TODO:
@c virtual *machines* / cloud machines can make performance hard to measure repeatably
@c    here we assume good old fashioned OS install directly on "bare metal"

Experiments require careful methodology, especially when the heap file
is in a storage-backed file system.  Overlooking the file system's
DRAM cache can easily misguide interpretation of results and foil
repeatability.  Fortunately Linux allows us to invalidate the file
system cache and thus mimic a ``cold start'' condition resembling the
immediate aftermath of a machine reboot.  Accesses to ordinary files
on durable storage will then be served from the storage devices, not
from cache.  Read about @command{sync} and
@file{/proc/sys/vm/drop_caches} at @*
@c
@url{https://www.kernel.org/doc/html/latest/admin-guide/sysctl/vm.html}.

@page
@c  = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@node    Experiments
@section Experiments

The C-shell (@command{csh}) script listed below illustrates concepts
and implements tips presented in this chapter.  It produced the
results discussed in @ref{Results} in roughly 20 minutes on an aging
laptop.  You can cut and paste the code listing below into a file, or
download it from @url{http://web.eecs.umich.edu/~tpkelly/pma/}.

@c TODO: post script to Web site when finalized  

The script measures the performance of four different ways to support
word frequency queries over a text corpus: The na@"{@dotless{i}}ve
approach of reading the corpus into an associative array for every
query; manually dumping a text representation of the word-frequency
table and manually loading it prior to a query; using @gwk{}'s
@code{rwarray} extension to dump and load an associative array; and
using @pmg{} to maintain a persistent associative array.

Comments at the top explain prerequisites.  Lines 8--10 set input
parameters: the directory where tests are run and where files
including the heap file are held, the off-the-shelf timer used to
measure run times and other performance characteristics such as peak
memory usage, and the size of the input.  The default input size
results in @pmg{} memory footprints under 3 GiB, which is large enough
for interesting results and small enough to fit in DRAM and avoid
paging on today's computers.  Lines 13--14 define a homebrew timer.

Two sections of the script are relevant if the default run directory
is changed from @file{/dev/shm/} to a directory in a conventional
storage-backed file system: Lines 15--17 define the mechanism for
clearing file data cached in DRAM; lines 23--30 set Linux kernel
parameters to discourage eager paging.

Lines 37--70 spit out, compile, and run a little C program to generate
a random text corpus.  This program is fast, flexible, and
deterministic, generating the same random output given the same
parameters.

Lines 71--100 run the four different AWK approaches on the same random
input, reporting separately the time to build and to query the
associative array containing word frequencies.

@c ADR suggests making shell script available on web site
@c TK will do eventually

@sp 1

@c first line of C-shell script can't contain line-number comment

@smallformat
@verbatim
#!/bin/csh -f
# Set PMG envar to path of pm-gawk executable and AWKLIBPATH                        #   2
#    to find rwarray.so                                                             #   3
# Requires "sudo" to work; consider this for /etc/sudoers file:                     #   4
#    Defaults:youruserid    !authenticate                                           #   5
echo 'begin:      ' `date` `date +%s`                                               #   6
unsetenv GAWK_PERSIST_FILE  # disable persistence until wanted                      #   7
set dir = '/dev/shm'        # where heap file et al. will live                      #   8
set tmr = '/usr/bin/time'   # can also use shell built-in "time"                    #   9
set isz = 1073741824        # input size; 1 GiB                                     #  10
# set isz = 100000000       # small input for quick testing                         #  11
cd $dir   # tick/tock/tyme below are homebrew timer, good within ~2ms               #  12
alias tick 'set t1 = `date +%s.%N`' ; alias tock 'set t2 = `date +%s.%N`'           #  13
alias tyme '$PMG -v t1=$t1 -v t2=$t2 "BEGIN{print t2-t1}"'                          #  14
alias tsync 'tick ; sync ; tock ; echo "sync time:  " `tyme`'                       #  15
alias drop_caches 'echo 3 | sudo tee /proc/sys/vm/drop_caches'                      #  16
alias snd 'tsync; drop_caches'                                                      #  17
echo "pm-gawk:     $PMG" ; echo 'std gawk:   ' `which gawk`                         #  18
echo "run dir:     $dir" ; echo 'pwd:        ' `pwd`                                #  19
echo 'dir content:'      ; ls -l $dir |& $PMG '{print "             " $0}'          #  20
echo 'timer:      ' $tmr ; echo 'AWKLIBPATH: ' $AWKLIBPATH                          #  21
@end verbatim
@page  @c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@verbatim
echo 'OS params:' ; set vm = '/proc/sys/vm/dirty'                                   #  22
sudo sh -c "echo 100     > ${vm}_background_ratio"     # restore these              #  23
sudo sh -c "echo 100     > ${vm}_ratio"                # paging params              #  24
sudo sh -c "echo 1080000 > ${vm}_expire_centisecs"     # to defaults                #  25
sudo sh -c "echo 1080000 > ${vm}_writeback_centisecs"  # if necessary               #  26
foreach d ( ${vm}_background_ratio  ${vm}_ratio \                                   #  27
            ${vm}_expire_centisecs  ${vm}_writeback_centisecs )                     #  28
    printf "             %-38s %7d\n" $d `cat $d`                                   #  29
end                                                                                 #  30
tick ; tock ; echo 'timr ovrhd: ' `tyme` 's (around 2ms for TK)'                    #  31
tick ; $PMG 'BEGIN{print "pm-gawk?     yes"}'                                       #  32
tock ; echo 'pmg ovrhd:  ' `tyme` 's (around 4-5 ms for TK)'                        #  33
set inp = 'input.dat'                                                               #  34
echo 'input size  ' $isz                                                            #  35
echo "input file:  $inp"                                                            #  36
set rg = rgen  # spit out and compile C program to generate random inputs           #  37
rm -f $inp  $rg.c $rg                                                               #  38
cat <<EOF > $rg.c                                                                   #  39
// generate N random words, one per line, no blank lines                            #  40
// charset is e.g. 'abcdefg@' where '@' becomes newline                             #  41
#include <stdio.h>                                                                  #  42
#include <stdlib.h>                                                                 #  43
#include <string.h>                                                                 #  44
#define RCH     c = a[rand() % L];                                                  #  45
#define PICK    do { RCH } while (0)                                                #  46
#define PICKCH  do { RCH } while (c == '@')                                         #  47
#define FP(...) fprintf(stderr, __VA_ARGS__)                                        #  48
int main(int argc, char *argv[]) {                                                  #  49
  if (4 != argc) {                                                                  #  50
    FP("usage:  %s  charset  N  seed\n",                                            #  51
                    argv[0]);   return 1; }                                         #  52
  char c, *a =      argv[1];    size_t L = strlen(a);                               #  53
  long int N = atol(argv[2]);                                                       #  54
  srand(       atol(argv[3]));                                                      #  55
  if (2 > N) { FP("N == %ld < 2\n", N); return 2; }                                 #  56
  PICKCH;                                                                           #  57
  for (;;) {                                                                        #  58
    if (2 == N) { PICKCH; putchar(c); putchar('\n'); break; }                       #  59
    if ('@' == c) { putchar('\n'); PICKCH; }                                        #  60
    else          { putchar( c  ); PICK;   }                                        #  61
    if (0 >= --N) break;                                                            #  62
  }                                                                                 #  63
}                                                                                   #  64
EOF                                                                                 #  65
gcc -std=c11 -Wall -Wextra -O3 -o $rg $rg.c                                         #  66
set t = '@@@@@@@' ; set c = "abcdefghijklmnopqrstuvwxyz$t$t$t$t$t$t"                #  67
tick ; ./$rg "$c" $isz 47 > $inp ; tock ; echo 'gen time:   ' `tyme`                #  68
echo "input file:  $inp"                                                            #  69
echo 'input wc:   ' `wc < $inp` ; echo 'input uniq: ' `sort -u $inp | wc`           #  70
@end verbatim
@page  @c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@verbatim
snd ############################################################################    #  71
tick ; $tmr $PMG '{n[$1]++}END{print "output: " n["foo"]}' $inp                     #  72
tock ; echo 'T naive O(N):          ' `tyme` ; echo ''                              #  73
rm -f rwa                                                                           #  74
snd ############################################################################    #  75
echo ''                                                                             #  76
tick ; $tmr $PMG -l rwarray '{n[$1]++}END{print "writea",writea("rwa",n)}' $inp     #  77
tock ; echo 'T rwarray build O(N):  ' `tyme` ; echo ''                              #  78
snd  # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #    #  79
tick ; $tmr $PMG -l rwarray 'BEGIN{print "reada",reada("rwa",n); \                  #  80
                                   print "output: " n["foo"]}'                      #  81
tock ; echo 'T rwarray query O(W):   ' `tyme` ; echo ''                             #  82
rm -f ft                                                                            #  83
snd ############################################################################    #  84
tick ; $tmr $PMG '{n[$1]++}END{for(w in n)print n[w], w}' $inp > ft                 #  85
tock ; echo 'T freqtbl build O(N):  ' `tyme` ; echo ''                              #  86
snd  # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #    #  87
tick ; $tmr $PMG '{n[$2] = $1}END{print "output: " n["foo"]}' ft                    #  88
tock ; echo 'T freqtbl query O(W):   ' `tyme` ; echo ''                             #  89
rm -f heap.pma                                                                      #  90
snd ############################################################################    #  91
truncate -s 3G heap.pma  # enlarge if needed                                        #  92
setenv GAWK_PERSIST_FILE heap.pma                                                   #  93
tick ; $tmr $PMG '{n[$1]++}' $inp                                                   #  94
tock ; echo 'T pm-gawk build O(N):  ' `tyme` ; echo ''                              #  95
snd  # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #    #  96
tick ; $tmr $PMG 'BEGIN{print "output: " n["foo"]}'                                 #  97
tock ; echo 'T pm-gawk query O(1):    ' `tyme` ; echo ''                            #  98
unsetenv GAWK_PERSIST_FILE                                                          #  99
snd ############################################################################    # 100
echo 'Note:  all output lines above should be identical' ; echo ''                  # 101
echo 'dir content:' ; ls -l $dir |& $PMG '{print "              " $0}'              # 102
echo '' ; echo 'storage footprints:'                                                # 103
foreach f ( rwa ft heap.pma )  # compression is very slow, so we comment it out     # 104
    echo "    $f " `du -BK $dir/$f` # `xz --best < $dir/$f | wc -c` 'bytes xz'      # 105
end                                                                                 # 106
echo '' ; echo 'end:        ' `date` `date +%s` ; echo ''                           # 107
@end verbatim
@end smallformat

@page
@c  = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@node    Results
@section Results

Running the script of @ref{Experiments} with default parameters on an
aging laptop yielded the results summarized in the table below.  More
extensive experiments, not reported here, yield qualitatively similar
results.  Keep in mind that performance measurements are often
sensitive to seemingly irrelevant factors.  For example, the program
that runs first may have the advantage of a cooler CPU; later
contestants may start with a hot CPU and consequent clock throttling
by a modern processor's thermal regulation apparatus.  Very generally,
performance measurement is a notoriously tricky business.  For
scripting, whose main motive is convenience rather than speed, the
proper role for performance measurements is to qualitatively test
hypotheses such as those that follow from asymptotic analyses and to
provide a rough idea of when various approaches are practical.

@sp 1

@verbatim
                            run time      peak memory    intermediate
        AWK script            (sec)      footprint (K)    storage (K)

        naive         O(N)   242.132       2,081,360           n/a
        rwarray build O(N)   250.288       2,846,868         156,832
        rwarray query O(W)    11.653       2,081,444
        freqtbl build O(N)   288.408       2,400,120          69,112
        freqtbl query O(W)    11.624       2,336,616
        pm-gawk build O(N)   251.946       2,079,520       2,076,608
        pm-gawk query O(1)     0.026           3,252
@end verbatim

@sp 1

The results are consistent with the asymptotic analysis of
@ref{Constant-Time Array Access}.  All four approaches require roughly
four minutes to read the synthetic input data.  The
na@"{@dotless{i}}ve approach must do this every time it performs a
query, but the other three build an associative array to support
queries and separately serve such queries.  The @code{freqtbl} and
@code{rwarray} approaches build an associative array of word
frequencies, serialize it to an intermediate file, and then read the
entire intermediate file prior to serving queries; the former does
this manually and the latter uses a @gwk{} extension.  Both can serve
queries in roughly ten seconds, not four minutes.  As we'd expect from
the asymptotic analysis, performing work proportional to the number of
words is preferable to work proportional to the size of the raw input
corpus: @i{O(W)} time is faster than @i{O(N)}.  And as we'd expect,
@pmg{}'s constant-time queries are faster still, by roughly two orders
of magnitude.  For the computations considered here, @pmg{} makes the
difference between blink-of-an-eye interactive queries and response
times long enough for the user's mind to wander.

Whereas @code{freqtbl} and @code{rwarray} reconstruct an associative
array prior to accessing an individual element, @pmg{} stores a
ready-made associative array in persistent memory.  That's why its
intermediate file (the heap file) is much larger than the other two
intermediate files, why the heap file is nearly as large as @pmg{}'s
peak memory footprint while building the persistent array, and why its
memory footprint is very small while serving a query that accesses a
single array element.  The upside of the large heap file is @i{O(1)}
access instead of @i{O(W)}---a classic time-space tradeoff.  If
storage is a scarce resource, all three intermediate files can be
compressed, @code{freqtbl} by a factor of roughly 2.7, @code{rwarray}
by roughly 5.6x, and @pmg{} by roughly 11x using @command{xz}.
Compression is CPU-intensive and slow, another time-space tradeoff.

@page
@c ==================================================================
@node    Data Integrity
@chapter Data Integrity

Mishaps including power outages, OS kernel panics, scripting bugs, and
command-line typos can harm your data, but precautions can mitigate
these risks.  In scripting scenarios it usually suffices to create
safe backups of important files at appropriate times.  As simple as
this sounds, care is needed to achieve genuine protection and to
reduce the costs of backups.  Here's a prudent yet frugal way to back
up a heap file between uses:
@verbatim
        $ backup_base=heap_bk_`date +%s`
        $ cp --reflink=always heap.pma $backup_base.pma
        $ chmod a-w $backup_base.pma
        $ sync
        $ touch $backup_base.done
        $ chmod a-w $backup_base.done
        $ sync
        $ ls -l heap*
        -rw-rw-r--. 1 me me 4096000 Aug  6 15:53 heap.pma
        -r--r--r--. 1 me me       0 Aug  6 16:16 heap_bk_1659827771.done
        -r--r--r--. 1 me me 4096000 Aug  6 16:16 heap_bk_1659827771.pma
@end verbatim
@noindent
Timestamps in backup filenames make it easy to find the most recent
copy if the heap file is damaged, even if last-mod metadata are
inadvertently altered.

@c TODO:  sync individual files above instead of globally (?)
@c        First carefully check what sync does in both cases
@c        using strace, verify that "sync [file]" is correct.
@c        Also check whether non-GNU/Linux offers fine-grained
@c        sync command.  Cygwin?  Solaris?

The @command{cp} command's @command{--reflink} option reduces both the
storage footprint of the copy and the time required to make it.  Just
as sparse files provide ``pay as you go'' storage footprints, reflink
copying offers ``pay as you @emph{change}'' storage
costs.@footnote{The system call that implements reflink copying is
described in @command{man ioctl_ficlone}.}  A reflink copy shares
storage with the original file.  The file system ensures that
subsequent changes to either file don't affect the other.  Reflink
copying is not available on all file systems; XFS, BtrFS, and OCFS2
currently support it.@footnote{The @command{--reflink} option creates
copies as sparse as the original.  If reflink copying is not
available, @command{--sparse=always} should be used.}  Fortunately you
can install, say, an XFS file system @emph{inside an ordinary file} on
some other file system, such as @code{ext4}.@footnote{See
@url{https://www.usenix.org/system/files/login/articles/login_winter19_08_kelly.pdf}.}

@c The @command{filefrag} utility reveals how the storage allocated to
@c the two files changes if they diverge.

After creating a backup copy of the heap file we use @command{sync} to
force it down to durable media.  Otherwise the copy may reside only in
volatile DRAM memory---the file system's cache---where an OS crash or
power failure could corrupt it.@footnote{On some OSes @command{sync}
provides very weak guarantees, but on Linux @command{sync} returns
only after all file system data are flushed down to durable storage.
If your @command{sync} is unreliable, write a little C program that
calls @code{fsync()} to flush a file.  To be safe, also call
@code{fsync()} on every enclosing directory on the file's
@code{realpath()} up to the root.}  After @command{sync}-ing the
backup we create and @command{sync} a ``success indicator'' file with
extension @file{.done} to address a nasty corner case: Power may fail
@emph{while} a backup is being copied from the primary heap file,
leaving either file, or both, corrupt on storage---a particularly
worrisome possibility for jobs that run unattended.  Upon reboot, each
@file{.done} file attests that the corresponding backup succeeded,
making it easy to identify the most recent successful backup.

@c TODO:  ".done" -> ".ready" so ls alphabetizes nicely (?)

Finally, if you're serious about tolerating failures you must ``train
as you would fight'' by testing your hardware/software stack against
realistic failures.  For realistic power-failure testing, see
@c @url{https://cacm.acm.org/magazines/2020/9/246938-is-persistent-memory-persistent/fulltext}
@c
@c and
@url{https://queue.acm.org/detail.cfm?id=3400902}.

@page
@c ==================================================================
@node    Acknowledgments
@chapter Acknowledgments

@c UPDATE:  make sure nobody is overlooked

Haris Volos, Zi Fan Tan, and Jianan Li developed a persistent @gwk{}
prototype based on a fork of the @gwk{} source.  Advice from @gwk{}
maintainer Arnold Robbins to me, which I forwarded to them, proved
very helpful.  Robbins moreover implemented, documented, and tested
@pmg{} for the official version of @gwk{}; along the way he suggested
numerous improvements for the @code{pma} memory allocator beneath
@pmg{}.  Corinna Vinschen suggested other improvements to @code{pma}
and tested @pmg{} on Cygwin.  Nelson H.@: F.@: Beebe provided access
to Solaris machines for testing.  Robbins, Volos, Li, Tan, Jon
Bentley, and Hans Boehm reviewed drafts of this user manual and
provided useful feedback.  Bentley suggested the min/max/mean example
in @ref{Examples}, and also the exercise of making Kernighan & Pike's
``Markov'' script persistent.  Volos provided and tested the advice on
tuning OS parameters in @ref{Virtual Memory and Big Data}.  Stan Park
provided insights about virtual memory, file systems, and utilities.

@c ==================================================================
@c ==================================================================
@c ==================================================================

@node     Installation
@appendix Installation

@c UPDATE below or remove this section if it's obsolete

@gwk{} 5.2 featuring persistent memory is expected to be released in
September 2022; look for it at @url{http://ftp.gnu.org/gnu/gawk/}.  If
5.2 is not released yet, the master git branch is available at
@c
@url{http://git.savannah.gnu.org/cgit/gawk.git/snapshot/gawk-master.tar.gz}.
@c
Unpack the tarball, run @command{./bootstrap.sh},
@command{./configure}, @command{make}, and @command{make check}, then
try some of the examples presented earlier.  In the normal course of
events, 5.2 and later @gwk{} releases featuring @pmg{} will appear in
the software package management systems of major GNU/Linux distros.
Eventually @pmg{} will be available in the default @gwk{} on such
systems.

@c ADR comments on above, "run ./bootstrap.sh, ./configure ..."
@c TK replies: I haven't been doing this.  Neither the README nor the
@c    INSTALL in the gawk tarball mention bootstrap.sh.  If it's
@c    important, shouldn't they?  The top of bootstrap.sh says its
@c    purpose is "to avoid out-of-date issues in Git sandboxes."
@c    When a neurodivergent source code control system requires us to
@c    write shell scripts to work around the problems that it creates
@c    gratuitously, the universe is trying to tell us something about
@c    it.

@c official gawk:
@c http://ftp.gnu.org/gnu/gawk/                                               [where to look for 5.2 after release]
@c https://www.skeeve.com/gawk/gawk-5.1.62.tar.gz                             [doesn't support persistent functions]
@c http://git.savannah.gnu.org/cgit/gawk.git/snapshot/gawk-master.tar.gz      [if 5.2 isn't released yet]
@c http://git.savannah.gnu.org/cgit/gawk.git                                  [ongoing development]

@c ==================================================================
@node     Debugging
@appendix Debugging

@c TODO:  ADR:  @cite -> @ref to info file below

For bugs unrelated to persistence, see the @gwk{} documentation,
e.g., @cite{GAWK: Effective AWK Programming},
available at @url{https://www.gnu.org/software/gawk/manual/}.

If @pmg{} doesn't behave as you expect, first consider whether you're
using the heap file that you intend; using the wrong heap file is a
common mistake.  Other fertile sources of bugs for newcomers are the
fact that a @code{BEGIN} block is executed every time @pmg{} runs,
which isn't always what you really want, and the fact that built-in
AWK variables such as @code{NR} are always reset to zero every time
the interpreter runs.  See the discussion of initialization
surrounding the min/max/mean script in @ref{Examples}.

If you suspect a persistence-related bug in @pmg{}, you can set
an environment variable that will cause its persistent heap module,
@code{pma}, to emit more verbose error messages; for details see the
main @gwk{} documentation.
@c or the @code{pma} documentation at
@c @url{http://web.eecs.umich.edu/~tpkelly/pma/}.

Programmers: You can re-compile @gwk{} with assertions enabled, which
will trigger extensive integrity checks within @code{pma}.  Ensure
that @file{pma.c} is compiled @emph{without} the @code{-DNDEBUG} flag
when @command{make} builds @gwk{}.  Run the resulting executable on small
inputs, because the integrity checks can be very slow.  If assertions
fail, that likely indicates bugs somewhere in @pmg{}.  Report such
bugs to me (Terence Kelly) and also following the procedures in the
main @gwk{} documentation.  Specify what version of @gwk{} you're
using, and try to provide a small and simple script that reliably
reproduces the bug.

@page
@c ==================================================================
@node     History
@appendix History

The @pmg{} persistence feature is based on a new persistent memory
allocator, @code{pma}, whose design is described in
@url{https://queue.acm.org/detail.cfm?id=3534855}.  It is instructive
to trace the evolutionary paths that led to @code{pma} and @pmg{}.

I wrote many AWK scripts during my dissertation research on Web
caching twenty years ago, most of which processed log files from Web
servers and Web caches.  Persistent @gwk{} would have made these
scripts smaller, faster, and easier to write, but at the time I was
unable even to imagine that @pmg{} is possible.  So I wrote a lot of
bothersome, inefficient code that manually dumped and re-loaded AWK
script variables to and from text files.  A decade would pass before
my colleagues and I began to connect the dots that make persistent
scripting possible, and a further decade would pass before @pmg{} came
together.

Circa 2011 while working at HP Labs I developed a fault-tolerant
distributed computing platform called ``Ken,'' which contained a
persistent memory allocator that resembles a simplified @code{pma}: It
presented a @code{malloc()}-like C interface and it allocated memory
from a file-backed memory mapping.  Experience with Ken convinced me
that the software abstraction of persistent memory offers important
attractions compared with the alternatives for managing persistent
data (e.g., relational databases and key-value stores).
Unfortunately, Ken's allocator is so deeply intertwined with the rest
of Ken that it's essentially inseparable; to enjoy the benefits of
Ken's persistent memory, one must ``buy in'' to a larger and more
complicated value proposition.  Whatever its other virtues might be,
Ken isn't ideal for showcasing the benefits of persistent memory in
isolation.

Another entangled aspect of Ken was a crash-tolerance mechanism that,
in retrospect, can be viewed as a user-space implementation of
failure-atomic @code{msync()}.  The first post-Ken disentanglement
effort isolated the crash-tolerance mechanism and implemented it in
the Linux kernel, calling the result ``failure-atomic @code{msync()}''
(FAMS).  FAMS strengthens the semantics of ordinary standard
@code{msync()} by guaranteeing that the durable state of a
memory-mapped file always reflects the most recent successful
@code{msync()} call, even in the presence of failures such as power
outages and OS or application crashes.  The original Linux kernel FAMS
prototype is described in a paper by Park et al. in EuroSys 2013.  My
colleagues and I subsequently implemented FAMS in several different
ways including in file systems (FAST 2015) and user-space libraries.
My most recent FAMS implementation, which leverages the reflink
copying feature described elsewhere in this manual, is now the
foundation of a new crash-tolerance feature in the venerable and
ubiquitous GNU @command{dbm} (@command{gdbm}) database
(@url{https://queue.acm.org/detail.cfm?id=3487353}).

In recent years my attention has returned to the advantages of
persistent memory programming, lately a hot topic thanks to the
commercial availability of byte-addressable non-volatile memory
hardware (which, confusingly, is nowadays marketed as ``persistent
memory'').  The software abstraction of persistent memory and the
corresponding programming style, however, are perfectly compatible
with @emph{conventional} computers---machines with neither
non-volatile memory nor any other special hardware or software.  I
wrote a few papers making this point, for example
@url{https://queue.acm.org/detail.cfm?id=3358957}.

In early 2022 I wrote a new stand-alone persistent memory allocator,
@code{pma}, to make persistent memory programming easy on conventional
hardware.  The @code{pma} interface is compatible with @code{malloc()}
and, unlike Ken's allocator, @code{pma} is not coupled to a particular
crash-tolerance mechanism.  Using @code{pma} is easy and, at least to
some, enjoyable.

Ken had been integrated into prototype forks of both the V8 JavaScript
interpreter and a Scheme interpreter, so it was natural to consider
whether @code{pma} might similarly enhance an interpreted scripting
language.  GNU AWK was a natural choice because the source code is
orderly and because @gwk{} has a single primary maintainer with an
open mind regarding new features.

Jianan Li, Zi Fan Tan, Haris Volos, and I began considering
persistence for @gwk{} in late 2021.  While I was writing @code{pma},
they prototyped @pmg{} in a fork of the @gwk{} source.  Experience
with the prototype confirmed the expected convenience and efficiency
benefits of @pmg{}, and by spring 2022 Arnold Robbins was implementing
persistence in the official version of @gwk{}.  The persistence
feature in official @gwk{} differs slightly from the prototype: The
former uses an environment variable to pass the heap file name to the
interpreter whereas the latter uses a mandatory command-line option.
In many respects, however, the two implementations are similar.  A
description of the prototype, including performance measurements, is
available at
@url{http://nvmw.ucsd.edu/nvmw2022-program/nvmw2022-data/nvmw2022-paper35-final_version_your_extended_abstract.pdf}.

@c lessons learned  [these are smallish ideas]
@c    compatibility with malloc
@c    make programmer do *nothing*
@c    components (pma) are easier to sell than monoliths (Ken)
@c    open source offers more impact than research
@c    work with colleagues who Think Different from one another

@sp 2

I enjoy several aspects of @pmg{}.  It's unobtrusive; as you gain
familiarity and experience, it fades into the background of your
scripting.  It's simple in both concept and implementation, and more
importantly it simplifies your scripts; much of its value is measured
not in the code it enables you to write but rather in the code it lets
you discard.  It's all that I needed for my dissertation research
twenty years ago, and more.  Anecdotally, it appears to inspire
creativity in early adopters, who have devised uses that @pmg{}'s
designers never anticipated.  I'm curious to see what new purposes
you find for it.

@c ==================================================================
@c ==================================================================
@c ==================================================================

@bye

@c ==================================================================
@c ==================================================================
@c ==================================================================