Devanagari UTF-8 text extraction code doesn't produce lua error while using fontspec's Node renderer, but still doesn't produce correct result due to a possible bug in Node renderer. I have filed that as a separate question: LuaTeX: Devanagari glyph order is reversed in tex's internal nodelist, how to recover the correct order while traversing glyph nodes?
While experimenting with different techniques for UTF-8 text extraction from TeX boxes, I encountered that the two techniques that do not produce any lua error for extraction of Devanagari text with fontspec's Node renderer, both produce lua error while using fontspec's HarfBuzz renderers (Renderer = Harfbuzz, Renderer=OpenType).
The two techniques are detailed here: technique-1 (using micahl-h21's get_unicode function) and here: technique-2 (just applying unicode.utf8.char on constituent components of complex glyphs). I tried multiple Devanagari fonts, all result in same behavior.
Complete test code for both techniques, and their respective error signatures are listed one after the other in the blocks below. For my example, I used freely available Noto Sans Devanagari (regular weight) available here: link to google fonts GitHub for Noto Sans Devanagari
Technique-1 with Devanagari and HarfBuzz (no lua error if compiled with Node renderer):
\documentclass{article}
\usepackage[lmargin=0.5in,tmargin=0.5in,rmargin=0.5in,bmargin=0.5in]{geometry}
\usepackage{fontspec}
\usepackage{microtype}
%\newfontscript{Devanagari}{deva,dev2}
\newfontfamily{\devanagarifam}{Noto Sans Devanagari}[Script=Devanagari, Scale=1, Renderer=HarfBuzz]
\begin{document}
% Devanagari text is at the right end of following line
% of code, you might have to scroll right to read it
\setbox0=\hbox{Příliš žluťoučký \textit{kůň} úpěl \hbox{ďábelské} ódy difference diffierence. \devanagarifam एक गांव -- में मोहन नाम का लड़का रहता था। उसके पिताजी एक मामूली मजदूर थे।}
\directlua{
% local fontstyles = require "l4fontstyles"
local char = unicode.utf8.char
local glyph_id = node.id("glyph")
local glue_id = node.id("glue")
local hlist_id = node.id("hlist")
local vlist_id = node.id("vlist")
local disc_id = node.id("disc")
local minglue = tex.sp("0.2em")
local usedcharacters = {}
local identifiers = fonts.hashes.identifiers
local function get_unicode(xchar,font_id)
local current = {}
local uchar = identifiers[font_id].characters[xchar].tounicode
for i= 1, string.len(uchar), 4 do
local cchar = string.sub(uchar, i, i + 3)
print(xchar,uchar,cchar, font_id, i)
table.insert(current,char(tonumber(cchar,16)))
end
return current
end
local function nodeText(n)
local t = {}
for x in node.traverse(n) do
% glyph node
if x.id == glyph_id then
% local currentchar = fonts.hashes.identifiers[x.font].characters[x.char].tounicode
local chars = get_unicode(x.char,x.font)
for _, current_char in ipairs(chars) do
table.insert(t,current_char)
end
% glue node
elseif x.id == glue_id and node.getglue(x) > minglue then
table.insert(t," ")
% discretionaries
elseif x.id == disc_id then
table.insert(t, nodeText(x.replace))
% recursivelly process hlist and vlist nodes
elseif x.id == hlist_id or x.id == vlist_id then
table.insert(t,nodeText(x.head))
end
end
return table.concat(t)
end
local n = tex.getbox(0)
print(nodeText(n.head))
local f = io.open("hello.txt","w")
f:write(nodeText(n.head))
f:close()
}
\box0
\end{document}
Error signature for Technique-1 (HarfBuzz renderer):
[\directlua]:1: bad argument #1 to 'len' (string expected, got nil)
stack traceback:
[C]: in function 'string.len'
[\directlua]:1: in upvalue 'get_unicode'
[\directlua]:1: in local 'nodeText'
[\directlua]:1: in main chunk.
l.62 }
Technique-2 with Devanagari and HarfBuzz (no lua error if compiled with Node renderer):
\documentclass{article}
\usepackage[lmargin=0.5in,tmargin=0.5in,rmargin=0.5in,bmargin=0.5in]{geometry}
\usepackage{fontspec}
\usepackage{microtype}
%\newfontscript{Devanagari}{deva,dev2}
\newfontfamily{\devanagarifam}{Noto Sans Devanagari}[Script=Devanagari, Scale=1, Renderer=HarfBuzz]
\begin{document}
% Devanagari text is at the right end of following line
% of code, you might have to scroll right to read it
\setbox0=\hbox{Příliš žluťoučký \textit{kůň} úpěl \hbox{ďábelské} ódy difference diffierence. \devanagarifam एक गांव -- में मोहन नाम का लड़का रहता था। उसके पिताजी एक मामूली मजदूर थे।}
\directlua{
local glyph_id = node.id("glyph")
local disc_id = node.id("disc")
local glue_id = node.id("glue")
local hlist_id = node.id("hlist")
local vlist_id = node.id("vlist")
local minglue = tex.sp("0.2em")
local function nodeText(n)
local t = {}
for x in node.traverse(n) do
% glyph node
if x.id == glyph_id then
if bit32.band(x.subtype,2) \csstring~=0 and unicode.utf8.char(x.char) \csstring~="“" and unicode.utf8.char(x.char) \csstring~="”" then %
for g in node.traverse_id(glyph_id,x.components) do
if bit32.band(g.subtype, 2) \csstring~=0 then
for gc in node.traverse_id(glyph_id,g.components) do
table.insert(t,unicode.utf8.char(gc.char))
end
else
table.insert(t,unicode.utf8.char(g.char))
end
end
else
table.insert(t,unicode.utf8.char(x.char))
end
% disc node
elseif x.id == disc_id then
for g in node.traverse_id(glyph_id,x.replace) do
if bit32.band(g.subtype, 2) \csstring~=0 then
for gc in node.traverse_id(glyph_id,g.components) do
table.insert(t,unicode.utf8.char(gc.char))
end
else
table.insert(t,unicode.utf8.char(g.char))
end
end
% glue node
elseif x.id == glue_id and node.getglue(x) > minglue then
table.insert(t," ")
elseif x.id == hlist_id or x.id == vlist_id then
table.insert(t,nodeText(x.head))
end
end
return table.concat(t)
end
local n = tex.getbox(0)
print(nodeText(n.head))
local f = io.open("hello.txt","w")
f:write(nodeText(n.head))
f:close()
}
\box0
\end{document}
Error signature for Technique-2 (HarfBuzz renderer):
[\directlua]:1: bad argument #1 to 'char' (invalid value)
stack traceback:
[C]: in field 'char'
[\directlua]:1: in local 'nodeText'
[\directlua]:1: in main chunk.
l.64 }
\documentclass{article} \usepackage{fontspec} \newfontfamily\fordinary{Noto Sans}[Renderer=HarfBuzz] \begin{document} \fordinary \setbox0=\hbox{abc}...), as if there is no string anymore. – Cicada Sep 23 '20 at 09:50\box0 \end{document}to the tail end your code, it does compile successfully and produce a correct pdf. Am using Texlive 2020 with packages last updated a week or so ago. – codepoet Sep 25 '20 at 10:03\documentclass{article} \usepackage{fontspec} \newfontfamily\fordinary{Noto Sans}[Renderer=HarfBuzz] \begin{document} \fordinary \setbox0=\hbox{abc} \directlua{ .... } \box0 \end{document}. – Cicada Sep 26 '20 at 03:14