Allow macro to end without new line

Question

The following macro, which a full MWE can be found at Unmodified Source Code ,

\def\SourceCode{%
  \begingroup%
  \endlinechar`\^^J%
  \catcode`\\=12\catcode`\^^M=12\catcode`\#=12\catcode`\~=12\catcode`\%=12\catcode`\^=12\catcode`\_=12\catcode`\@=12\catcode`\ =12\catcode`\|=12%      
  \SourceCodeAux}%
\def\SourceCodeAux#1#2{\endgroup\directlua{print("\luaescapestring{#1}")}}%

has the issue that the last terminating } cannot have anything else after it, on it's line:

\begin{itemize}
\item 2
\SourceCode{
{this
is
a
test}
}\end{itemize} % < This line is the problem, } \end{itemize} works but produces protected space in pdf.

will not work unless \end{itemize} is moved to the next line or a space is put between the } \end{itemize} (which then puts a protected space in the pdf)

The issue seems to be how the macro handles the last character but I don't quite understand what is going on. Is there any way to get the }\end{itemize} case to work?

I'm a bit confused on what the second argument for \def\SourceCodeAux#1#2 is doing. It is required but I'm not sure how it actually works. I've replaced it with other stuff and can get }\end{itemize} to work but end up with protected spaces or other characters showing up in the pdf. My thinking is if I can insert a new-line into the stream right after the final } then TeX will see }\^^M\end{itemize} and allow it to work. Anyways, maybe someone can enlighten me on what exactly is going on.

here is how I see the macro:

1. Start new group to locally define catcodes(so the catcode modifications do not effect anything but what is in the macro)

2. Modify endlinechar and catcodes so that tokens after the macro call are not "Special"

3. Call an auxiliary macro that accepts two arguments. The first being will end up being the token stream(the source code argument "passed" to the original macro call). The second argument is then going to be the last token in the source code. (In the problem case happens to be a \ which gets picked up "accidently" and breaks the \end{itemize} (tex see's it as end{itemize})

4. endgroup is called to return the catcodes back to normal then processing is done on the first "argument". (I'm not quite sure how the macro knows when to stop processing tokens though but I guess it has something to do with some trick using the #2 which is also what is causing the problem)

Here is a full MWE:

    \documentclass{book}
    \usepackage{luatex}
    \directlua{tex.enableprimitives('',tex.extraprimitives())}
    \begin{document}

        \def\SourceCode{%
          \begingroup%
          \endlinechar`\^^J%
          \catcode`\\=12\catcode`\^^M=12\catcode`\#=12\catcode`\~=12\catcode`\%=12\catcode`\^=12\catcode`\_=12\catcode`\@=12\catcode`\ =12\catcode`\|=12%
          \SourceCodeAux}%
        \def\SourceCodeAux#1#2{\endgroup\directlua{print("\luaescapestring{#1}")}}%


    \begin{itemize}
    \item 1
    \SourceCode{
    {this
    is
    a sucessful
    test}
    }
    \end{itemize}

\iftrue
    \begin{itemize}
    \item 2
    \SourceCode{
    {this
    is
    a failed
    test}
    }\end{itemize}
\fi
    \end{document}

Answer 1

The problem is that TeX adds the End-Of-Line (EOL, \endlinechar) character at the end of the line when it reads the full line, not when it processes the line end. This means the EOL of the last line with any code after the actual \SourceCode content is still a ^^J (prints as an uppercase Ohm). The #2 is included to gobble a following EOL, which is assumed to follow. If not, the next verbatim character is gobbled instead and the ^^J EOL is still there.

A solution is to use a cleanup macro which reads the rest of the line and removes the ^^J while reinserting the other code of the source code line. Note that the normal EOL would insert a space, which is included in the cleanup macro, but might not always behave exactly the same in all cases. Also this will break with an TeX error if there is a comment in that line which is masking the EOL character.

In general it is a bad idea to mix the first and last lines of a multi-line verbatim environment or macro with normal material, so having a line break after the } is actually a very good idea.

\documentclass{book}
\usepackage{luatex}
\directlua{tex.enableprimitives('',tex.extraprimitives())}
\begin{document}

\def\SourceCode{%
    \begingroup%
    \endlinechar`\^^J%
    \catcode`\\=12\catcode`\^^M=12\catcode`\#=12\catcode`\~=12\catcode`\%=12\catcode`\^=12\catcode`\_=12\catcode`\@=12\catcode`\ =12\catcode`\|=12%
    \SourceCodeAux}%
\def\SourceCodeAux#1{\endgroup\directlua{print("\luaescapestring{#1}")}\SourceCodeCleanup}%

\def\SourceCodeCleanup#1^^J{#1 }%


\begin{itemize}
\item 1
\SourceCode{
{this
is
a successful
test}
}
\end{itemize}

\iftrue
\begin{itemize}
\item 2
\SourceCode{
{this
is
now also a successful
test}
}\end{itemize}
\fi
\end{document}

Answer 2

I know from the 'thread' of questions that xparse might not be what you want, but there is no problem using a verbatim argument here to get the output you indicate

\documentclass{article}
\usepackage{luatex}
\directlua{tex.enableprimitives('',tex.extraprimitives())}
\usepackage{xparse}
\NewDocumentCommand{\SourceCode}{+v}{\directlua{print("\luaescapestring{#1}")}}
\begin{document}
\begin{itemize}
\item 2
\SourceCode{
{this
is
a
test}
}\end{itemize}
\end{document}

A demonstration that this can be used to handle ^^M (end-of-line) within TeX (the above demo prints the output at the Lua level):

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{xparse}
\ExplSyntaxOn
\char_set_catcode_other:n { `\^^M }
\NewDocumentCommand { \SourceCode } { +v }
  {
    \tl_set:Nn \l_tmpa_tl {#1}
    \tl_replace_all:Nnn \l_tmpa_tl { ^^M } { \par }
    \ttfamily { \l_tmpa_tl }
  }
\char_set_catcode_end_line:n { `\^^M }
\ExplSyntaxOff
\begin{document}
\begin{itemize}
\item 2
\SourceCode{
{this is
is
a
test}
}\end{itemize}

\end{document}

This is needed because a truly verbatim grab of each line includes ^^M at the end of each line. You can see that by showing an analysis of the grabbed tokens:

\documentclass{article}
\usepackage{xparse}
\ExplSyntaxOn
\NewDocumentCommand { \SourceCode } { +v }
  { \tl_show_analysis:n {#1} }
\ExplSyntaxOff
\begin{document}
\begin{itemize}
\item 2
\SourceCode{
{this is
is
a
test}
}\end{itemize}

\end{document}

(For releases before TeX Live 2017, you will need \usepackage{l3tl-analysis} in addition to \usepackage{xparse} for this to work.)