Using \toks registers

Question

I was reading this tutorial for TeX programming:

http://pgfplots.sourceforge.net/TeX-programming-notes.pdf

recommended in the 2-nd answer here: Where do I start LaTeX programming?

It says:

\toks<number> There are also 255 token registers which can be thought of as special string variables. Of course, every macro assignment \def\macro{ content } is also some kind of string variable, but token registers are special: their contents won’t be expanded when used with \the\toks number . This can be used for fine grained expansion control, see Section 2.3 below.

But no examples of using \toks registers. What does he mean: "special string variables"? For 2 previous kinds of registers I could assign the value:

\count0=12
\dimen0=1.102pt

And print it:

Value1: \the\count0
Value2: \the\dimen0

But I don't understand, how to use \toks.

Answer 1

A token register is very similar to a macro with no arguments, but there are differences in syntax and expansion rules.

Using the plain and latex definition of \toks@ for \toks 0 you can store the tokens abc via

\toks@{abc}

or

\def\tmp{abc}

Note that a macro has a specific command to set the macro (\def or wrappers around that such as \newcommand) but token registers just use the register name followed by a balanced text group.

Conversely unlike macros which expand just by referencing their name, token registers, like other registers, are not expandable and their contents are accessed by expanding \the.

So

\the\toks@

and

\tmp

each expand to abc.

Apart from these syntactic differences the expansion of token registers differs from macros in two important ways.

1. expansion in \edef and other expansion only contexts is limited to one level:

   \def\tmp{aaa}
   \def\tmpb{xx \tmp}
   \toks@{xx \tmp}
   \edef\A{\tmpb}
   \edef\B{\the\toks@}
   

   Now \tmpb and \toks@ contain the same token list xx \tmp but \A is defined by fully expanding expandable tokens and so has defnition xx aaa but the tokens returned by \the are not further expanded in the \edef and so \B has definition xx \tmp

2. The second difference is that as token registers do not have arguments # does not need to be (and is not) special when defining or expanding the register.

   \toks@{#1}
   

   is just a token register consisting of a list of two tokens # and 1.

This is put to use in LaTeX's \g@addto@macro macro.

\g@addto@macro@\foo{abc}

is supposed to add abc to the end of the current definition of \foo.

A simple (and if I recall correctly original) definition could be

\makeatletter
\gdef\foo{123}
\def\gaA#1#2{%
  \expandafter\gdef\expandafter#1\expandafter{#1#2}}
\gaA\foo{abc}
\show\foo

That works fine and shows the definition as 123abc. However try

\gaA\foo{#}

and you get

! Illegal parameter number in definition of \foo.

However we can use the fact that # isn't special in a toks register and that the register contents are only expanded once

\long\def\g@addto@macro#1#2{%
  \begingroup
    \toks@\expandafter{#1#2}%
    \xdef#1{\the\toks@}%
  \endgroup}

First the \toks@ register is defined to contain the expansion of the macro passed in the first argument, followed by the contents of the second argument. # is safe to use in a toks assignment. Then the macro is globally defined to be the expansion of \thetoks which is exactly the contents of \toks@ with no further expansion, even if that contains # tokens:

\makeatletter
\gdef\foo{123}
\g@addto@macro\foo{abc}
\show\foo
\g@addto@macro\foo{#}
\show\foo

produces

> \foo=macro:
->123abc.
l.7 \show\foo
? 
> \foo=macro:
->123abc##.
l.11 \show\foo

where # has been added as intended (only one # has been added, the doubled ## is an artefact of using \show).