% language=us runpath=texruns:manuals/luametatex

\environment luametatex-style

\startdocument[title=principles]

\startsection[title={Introduction}]

This is a bit odd manual but needed anyway. In the process of adding features to
\LUAMETATEX\ and adapting \CONTEXT\ accordingly some decisions were made. On the
one hand generic flexibility is a criterion used when the extending engine, on
the other hand practical usability in \CONTEXT\ is used to decide where to draw
a line or make some choices. It makes no sense to complicate the already complex
engine even more, or cripple \CONTEXT\ when cleaner (low level) solutions are
possible.

Here I will collect some of the considerations and mention the choices made.
These are mostly mine but some result from discussions and experiments. This
overview is not complete, new primitives are discussed elsewhere and the
\CONTEXT\ low level manuals explain how to use these. Consider this to be a
teaser.

{\em This summary is work in progress.}

\stopsection

\startsection[title=Text fonts]

Plenty has been written about fonts in \TEX, so here I will only mention a few
aspects. Traditionally the \TEX\ engines works with copies of fonts at given
sizes. For large fonts that is kind of inefficient. This is why in \LUAMETATEX\
we can scale a font on|-|the|-|fly using \typ {\glyphscale}, \typ {\glyphxscale}
and \typ {\glyphyscale}. This feature is also used to implement a more efficient
(although not 100\percent\ metric compatible) compact font mode. It works okay in
text as well as math although it comes at a price: many more calculations are
needed at the engine end.

One way to get an expanded, squeezed, emboldened or slanted font in \CONTEXT\ is
to use the effects mechanism. It is quite flexible but again comes at a price
because the backend has to do more work which is measurable, especially because
effects can apply to the font or individual glyphs. However, the advantages
out|-|weight the disadvantages. At the cost of yet a bit more performance a more
native variant is also available using \typ {\glyphslant} and \typ
{\glyphweight}.

\definefontfeature[extendtest] [default][extend=1.25]
\definefontfeature[squeezetest][default][squeeze=1.25]
\definefontfeature[slanttest]  [default][slant=0.5]
\definefontfeature[weighttesta][default][weight=0.2] % 0.4pt is bad for the g in lm
\definefontfeature[weighttestb][default][effect={weight=0.2,auto=yes}]

\starttexdefinition demo #1
    \startcombination[nx=6,ny=1,distance=12.5mm]
      {\scale[scale=8000]{\definedfont[Serif*default     @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {}
      {\scale[scale=8000]{\definedfont[Serif*extendtest  @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {}
      {\scale[scale=8000]{\definedfont[Serif*squeezetest @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {}
      {\scale[scale=8000]{\definedfont[Serif*slanttest   @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {}
      {\scale[scale=8000]{\definedfont[Serif*weighttesta @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {}
      {\scale[scale=8000]{\definedfont[Serif*weighttestb @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {}
    \stopcombination
\stoptexdefinition

\startlinecorrection \demo{a} \stoplinecorrection
\startlinecorrection \demo{x} \stoplinecorrection
\startlinecorrection \demo{p} \stoplinecorrection
%startlinecorrection \demo{(} \stoplinecorrection
\startlinecorrection \demo{g} \stoplinecorrection
\startlinecorrection \demo{-} \stoplinecorrection

Extending, as seen in the second renderings, scales the shapes horizontally,
while squeezing, in the third renderings, does it in the vertical direction. In
both cases the dimensions have to be adapted. This is not the case when we slant.
The last two samples in a row have an increased weight, and these are the more
tricky cases because here one can argue how to scale and reposition a shape. When
a shape is above the baseline we increase the height, and when it goes below we
increase the depth. The engine is capable to increase the width, height and depth
and shift the shape a little. It only makes sense to adapt the height and depth
when they are non|-|zero. It will never be perfect, but this feature is not
perfect anyway.

The way fonts are set up in a \TEX\ macro package often originates in the past,
if only because it came with fonts. The Computer Modern fonts are among the few
that have multiple design sizes. However, the collection is pretty much based on
a ten point design. For math there are seven and five point variants for the
script sizes, for footnotes an eight point makes sense and section heads can use
the larger twelve point plus the few larger sizes. Setting up a twelve point
body font environment, as we have in \CONTEXT, is quite doable with the fonts but
for an eleven point body font more compromised have to be made.

One can wonder why in \CONTEXT\ the ten point math setup of 10/7/5 became 12/9/7
instead if 12/8.4/6 and the reason is just that when there were still bitmap
fonts one didn't want too many (intermediate) sizes. Anyway, we're sort of stuck
with this default setup now, but nothing prevents users to redefine a body font
environment.

Another speciality of \TEX\ (fonts) is that they have italic correction, something
that lacks in \OPENTYPE\ fonts (apart from math btu there it serves a different
purpose). We can however emulate it, and in \CONTEXT\ that is an option. Given that
we have to make choices it is clear that the engine can only be supportive here,
especially when we use the \typ {\glyphslant} method.

A curious case is the following: in Computer Modern we find italic correction in
the upright fonts, for instance between an \quote {f} and \quote {h}. Dealing with
this automatically is impossible because italic correction is not to be applied
between glyph runs of the same font.

% quite some italics with not angle and uprights with angles
% why not proper boundingbox (right side bearing) with (now also small) width
% instead of unpredictable either or not urx

\stopsection

\startsection[title=Math fonts]

Support for math in an \UNICODE\ aware engine is also driven by the repertoire of
characters and their organization in \UNICODE, as well as by \OPENTYPE\ math as
cooked up by \MICROSOFT\ with a bit of input from \TEX\ folk.

The engine is agnostic when it comes to \UNICODE: there are no character codes
interpreted in special ways. There are math alphabets but these are not special:
in a traditional eight bit engine we have families to deal with them, in a
\UNICODE\ aware engine there are several solutions. The most important character
property that has some consequence is the math class but for dealing with that
we're on our own anyway. Everything \UNICODE\ related is up to \CONTEXT\ to deal
with, and it is the macro package that drives the engine, using the constructs
that are available, like atoms with specific classes, fractions, accents,
delimiters, fences, radicals, operators etc.

When it comes to fonts it is more complex. The \OPENTYPE\ math standard is driven
by the fact that \MSWORD\ provides a math editor and therefore needs a font. That
font is Cambria and it is (at the time of writing this) the only font that comes
from the origin. It has not been extended, nor fixed so basically what is in
there kind of has become the standard. The other \OPENTYPE\ math fonts are a
curious mix of old and new technology and again not much has happened there.

Now, when it comes to choices here, a few can be made based on conclusions drawn
during decades of dealing with these fonts and the assumed technology.

\startitemize

\startitem
    There has be no real developments so we can just assume that what we got is
    what we will have forever. Cambria is and remains the standard, quite some
    fonts shipped with \TEX\ have issues that will stay, and new fonts,
    especially when developed outside \TEX's scope likely also have issues,
    because, after all, what is used for testing them?
\stopitem

\startitem
    Only a few renders support the new technology. It is unlikely that \MSWORD\
    will \ change and basically \XETEX\ and \LUATEX\ are also frozen. On the web
    old school fonts are used, at least till 2023. Plenty of time went by since
    the beginning of the century and nothing improved.
\stopitem

\startitem
    The most important font properties that play a role are parameters, italic
    correction, variants and extensibles, anchors for accents, stylistic
    alternates, script alternates and staircase kerns. There are some rules of
    how to apply italic correction, but many fonts make them unapplicable. The
    same is true for anchors and kerns. There are only top kerns.
\stopitem

\startitem
    Italic correction is a flawed concept and we decided to just ignore them:
    when specified we add it to the width and discard them afterwards. The value
    is translated into a bottom right corner kern. For large operators we
    translate them to top and bottom accents.
\stopitem

\startitem
    Top accents can be flawed so in many cases we can just ignore them. They
    only make sense for italic shapes anyway.
\stopitem

\startitem
    Staircase kerns are a nice idea but make no sense. First of all they concern
    two characters, nucleus and script, but we can also have accents, fraction,
    fenced stuff and other constructs in scripts so instead we prefer a system of
    corner kerns. Also, we noticed that staircase kerns are often implemented
    partially and even then not that well, probably because there was no way to
    test them. Even worse is that when they are inconsistent formulas can look
    rather inconsistent. So, we translate staircase kerns into corner kerns and
    add and|/| overload them by corner kerns. These kerns can then be applied for
    any reasonable combination.
\stopitem

\startitem
    Extensible are mixed breed. Rules should be extensibles but aren't. Some snippets
    have \UNICODE\ points so they can be used to construct missing glyphs but the
    repertoire is inconsistent. Because we don't expect \UNICODE\ to adapt we
    therefore provide alternative solutions.
\stopitem

\startitem
    The repertoire of math parameters is on the one hand incomplete and on the
    other hand less dependent on the font and more on intended usage. So, apart
    from a few, we end up with adapting to our needs. It is part of the more granular
    control that we wish.
\stopitem

\startitem
    Gaps in alphabet vectors are a pain but the engine is agnostic of them. For some
    reason the \TEX\ community let itself down on this so it has to cope at the macro
    level. It is by now an old problem.
\stopitem

\stopitemize

So, to summarize the font part, an alternative standard could discard the concept
of italic correction and go for proper widths, a simplified corner kern model,
provide top and bottom accents, prescribe a repertoire of extensibles and
snippets and at least fill the gaps in alphabets instead of relying on shared
glyphs. It won't happen any time soon, but still we do follow that approach and
have the engine ready for it. Because we adapt the fonts runtime to this, we can
eventually remove all the code related to italic correction and staircase kerns,
simply because it is not used.

\stopsection

\startsection[title=Rules]

The original \TEX\ engine actually has only two graphical elements: glyphs and
rules. These have a width, height and depth and when decisions are made, for
instance when deciding where to break a line, or when boxes are constructed these
dimensions have to be known. Actually, \TEX\ doesn't really care what these
elements are, it's the dimensions that matter most. Graphics for instance can be
abstract objects, traditionally injected via so called specials wrapped into a
box of given dimensions. The \PDFTEX\ and later engines added a native
representation but basically it acted like a box (or rule if you like). It's the
backend that turns glyphs, rules and these special boxes into something that one
can see and print.

Rules have the three dimensions we mentioned. There are horizontal and vertical rules,
but only at the primives level. Once you specified an \type {\hrule} or \type {\vrule}
it became a generic rule with the main difference being the default dimensions. A rule
initializes with so called running dimensions, think of signals that the final dimension
comes from the encapsulating box.

Here we have a vertical rule: {\darkblue \vrule width 3cm height 5mm depth 2mm}
with width 3cm, height 5mm and depth 2mm. If we don't specify a width we get the
default thickness of 0.4pt, as in {\darkblue \vrule height 5mm depth 2mm} and when we
prefix it with \type {\leaders} and let it follow by a \type {\hfill} we get this:
{\darkblue \leaders \vrule height 5mm depth 2mm\hfill}.

When we put on an \type {\hrule} on an empty line the running width kicks in:
\crlf {\darkblue \hrule height 5mm depth 2mm}\crlf which is a feature that one
can use in for instance tables. However the fact that we only talk rectangles
means that there is only a limited repertoire of applications. In order to frame
some text you need four (disconnected) rules, For a background fill you can use a
single rule. There is also an application for rules that have height and depth
but no width: these so called struts that can enforce vertical spacing and
dimensions.

So what does \LUAMETATEX\ bring to the rules? Because the engine itself is only
interested in dimensions it's more about passing information to the backend. For
this we have a few more fields in the rule nodes that can be set from \LUA. This
permits for instance to hook in \METAPOST\ graphics that adapt like rules. There
are a few more primitives, one for making struts: they can take their dimensions
from a character. In math mode they're invisible and don't influence inter-atom
spacing but still take their role in determining dimensions. Then there are the virtual
rules that have dimensions (to be used in the backend) but don't contribute in the
frontend. The \type {\novrule} and \type {\nohrule} do contribute but are ignores in the
backend so they are cheap alternatives for empty boxes with specific dimensions set.

Some rule subtypes are set by the engine, for instance the math engine marks
over, under, fraction and radical rules. In \LUA\ one can mark outline, user, box
and image rules so that node list processors can take their properties into
accounT when needed, the frontend is only interested in the dimensions and sees
them as normal rules.

\startbuffer
\hrule height \strutht depth \strutdp on 0.04tw off 0.01tw \relax
\stopbuffer

\bgroup \nowhitespace
\strut\darkblue \getbuffer
\egroup

Here we have the following call:

\typebuffer

The \type {on} and \type {off} are among the new keys and they do nothing at the
\TEX\ end. It is the backend that will create the dash pattern. You can achieve
the same effect with leaders but while here we have a single rule, for a leader
the engine will make as many rules as are needed for this dash pattern. This is a
good example of adding little to the fontend in order to make the backend do the
job. In a similar fashion outlines are delegated. Other tricks involve offsets
and there is room for some additional features but for now they are on the
\quotation {Only when I need it.} list, after all we need something to wish for.

\stopsection

\startsection[title=Paragraphs]

A lot can be said about paragraphs but we keep it short here. Much more can be
found in for instance the articles that we wrote on the subject. When you enter
(or generate) text it will be added to a list (of nodes). That list can become a
horizontal box, vertical box, or end up in the main vertical list. When we go
vertical the list will be split in lines and the process is called line breaking.
Between the lines we can get penalties that tell the machinery how a paragraph of
lines can be split over page boundaries.

When breaking the engine can use up to three passes: a first pass that uses \type
{\pretolerance} as criterion, a tolerant pass with hyphenation enabled using
\type {\tolerance} and an emergency pass that kicks in \type {\emergencystretch}
when set. In \LUAMETATEX\ we can have additional passes that come online
depending on criteria and|/|or thresholds; search for \type {\parpasses} to learn
more about this.

The par builder in \LUAMETATEX\ has more features that users can control and also
normalized the resulting lines so that later on from the \LUA\ end they can be
manipulated easier. There are also ways to let embedded inserts, marks and (v)adjusts
migrate to the outer level. All this takes more runtime than in original \TEX\ but
in practice one won't really notice this because we gain in other places.

Most or what is new is available as features in \CONTEXT, most noticeably in
extra keys to \type {\setupalign}. It is also good to know that we have ways to
hook specific features in what is called \quote {wrapping up paragraph}. Also,
contrary to traditional \TEX\ we configured \CONTEXT\ to use the mechanism that
freezes paragraph specific parameters with the current paragraph so that there is
no (or at least less) interference with grouping.

\stopsection

\startsection[title=Pages]

{\em todo}

\stopsection

\startsection[title=Alignments]

{\em todo}

\stopsection

\startsection[title=Adjusts]

You can put stuff before and after lines using \type {\vadjust} and at the edges
using \type {\localleftbox} and alike. Both are seen in the par builder, where the
boxes contribute to the dimensions and the adjusted material is inserted when the
paragraph is wrapped up and contributed to the current list. In \LUAMETATEX\ these
mechanism have been extended so that we can actually uses them in am meaningful
way.

\stopsection


\startsection[title=Marks]

These signals in the text are used for managing (for instance) running headers
and a few extra features have been added, like migration to an outer level and
resets. In \MKIV\ we handled marks in \LUA\ but with \LUAMETATEX\ it makes sense
to use the engine.

\stopsection

\startsection[title=Inserts]

Inserts are signals that end up in lines and migrate to the outer level, that is
the main vertical list. An example of usage is footnotes. In the main vertical
list they are bound to the line they relate to so that the page builder can make
sure that they end up on the same page. In \LUAMETATEX\ they can bubble up from
deeply nested boxes. Contrary to the traditional binding of an insert class to
various registers in \LUAMETATEX\ they can be managed independently which means
that they have more properties.

\stopsection

\startsection[title=Boxes]

{\em todo}

\stopsection

\startsection[title=Language]
\stopsection

\startsection[title=Math]

Plenty has been written about the multi-year project of opening up and extending
the math engine. Opening up and providing full control is part of supporting and
experimenting with \OPENTYPE\ math fonts but we already discussed this in a
previous section. Another aspect of opening up is making hard coded properties
configureable, even if that feature will hardly be used, simply because the
built-in defaults make sense. Then there is all kind of control over rendering
that can be controlled by keywords to the math specific elements like atoms,
fractions, operators, accents, radicals and fences.

Because traditional fonts are phased out in favour of (often flawed) \OPENTYPE\
variants much of what is new is also controlled by fonts, be it that we have our
own extensions. In \CONTEXT\ mathfonts are tweaked to fit our model. Inter atom
spacing, penalties, discretionaries, continuation scripts (think multiscripts,
pre and post), additional classes, dictionaries, linebreaks, carrying properties
over math groups, are all features that make it possible to renderer more precise
math without the need for manual intervention. It often looks, for instance from
posts on support platforms, that the more or less standard math has to come with
tweaking your source; it has become an accepted practice. In \CONTEXT\ we always
had structure and we added some more of that and because the math engine carries
more information around we could eventually simplify some code otherwise done in
\LUA.

By looking at what \CONTEXT\ actually needs, we could decide to strip down the
math engine (old as well as new features). We can also decide to eventually just
assume wide fonts to be used and drop old font support. After all, because one
has to load the fonts with \LUA, it's not hard to map traditional fonts to
(extended) \OPENTYPE\ alternatives, which is actually what we do anyway with for
instance Antykwa.

\stopsection

\startsection[title=Programming]

{\em todo}

\stopsection

\startsection[title=Protection]

The idea behind \TEX\ is that users define macros. However, when they do so in
the perspective of a macro package there is the danger that core functionality
can be overwritten. Now, one can of course make all primitives less accessible,
for instance by some prefix. But that makes no real sense for features that
belong to the language. When users use CamelCase for their names they're unlikely
to run into issues, so while internal macros are actually prefixed, we don't
do that with the primitives, so you can write code that looks \TEX.

Over time \CONTEXT\ has been ridiculed by non users for prefixing with \type
{\do} or \type {\dodo} but that's by folk who love long (cryptic) names with many
underscores and other inaccessible characters. The way we protect users from
accidental overloading is by using the \LUAMETATEX\ overload protection system.
Macros (and primitives) can be tagged in way so that the engine can issue warning
or even error in case of an undesirable definition.

There is of course some overhead involved in for instance every \type {\def} or
\type {\let} but it is little and the engine is fast anyway.

\stopsection

\startsection[title=Optimization]

There are many places where the engine could be optimized without getting
obscure. One reason is that the memory layout is somewhat different because we
snap to 8, 16, 32 or 64 bits and the engine being a \UNICODE\ capable one already
has more memory available in some places than what was needed. Also, knowing
usage patterns, it was possible to identify possible bottlenecks and widen the
necks.

Furthermore, it was possible to improve input handling, logging, save stack
usage, keyword parsing, expressions, and much more. On the other hand nodes
became larger so there we loose some. The \LUAMETATEX\ engine is faster than
\LUATEX, although some of the gain is lost on the fact that one needs to use
\LUA\ backend.

\stopsection

\startsection[title=Input]

The input can come from files, token lists, macros and \LUA\ which means many
places. When it comes from \LUA\ it can be tokens, nodes, string, and each has
its special way of handling and the engine has to keep track of this when it
accumulated the input that pops up after a \LUA\ call. This is done as efficient
as possible without sacrificing performance. The fact that we have \UTF\ should
not have too much impact.

\stopsection

\startsection[title=Nesting]

When you enter a group a stack boundary is set and when some value changes the
original value is pushed on the stack. After leaving the group values are
restored. The engine tries to avoid redundant actions which improves memory usage
and runtime.

Every macro expansion, opened file, expanded token list, etc.\ pushes the input
stack and that comes with overhead. Again we have tried to minimize the impact
and thereby gain a bit over \LUATEX.

Other stacks like those used by math, alignment, conditionals, expressions etc.\
have also been improved some. On the other hand, by unweaving some shared code
there can be a price to pay, but as with everything usage patterns indicate no
penalty here.

\stopsection

\startsection[title=conditions]

We already had more conditionals in \LUATEX\ but again the repertoire of
conditionals has been extended. This permits us to remove some middle-layer
helpers and stay closer to the core language. It also helps to improve
performance.

Another important addition has been \type {\orelse} than permits us to write test
in a way similar to what other language provide with for instance \type {elseif}
or \type {else if}.

\stopsection

\startsection[title=macros]

Expanding macros happens a lot especially in a more complex macro package. This
means that adding features in that area can have a large impact on runtime.
Nevertheless the argument parser now provides a few handfuls of variants in
picking up arguments with out noticeable degradation, especially because these
new features can gain performance.

At the same time there have been some optimizations in storing macro related
states, checking and accessing parameters. There are additional (internal)
classes of macros that make for a more natural implementation; for instance
\type {\protected} macros are now first class commands.

\stopsection

\startsection[title=Keywords]

Some primitives accept one or more keywords and \LUAMETATEX\ adds some more. In
order to deal with this efficiently the keyword scanner has been optimized, where
even the context was taken into account. As a result the scanner was quite a bit
faster. This kind of optimization was a graduate process the eventually ended up
in what we have now. In traditional \TEX\ (and also \LUATEX) the order of
keywords is sometimes mixed and sometimes prescribed. In most cases only one
occurrence is permitted. So, for instance, this is valid in \LUATEX:

\starttyping
\hbox attr 123 456 attr 123 456 spread 10cm { }
\hrule width 10cm depth 3mm
\hskip 3pt plus 2pt minus 1pt
\stoptyping

The \type {attr} comes before the \type {spread}, rules can have multiple mixed
dimension specifiers, and in glue the optional \type {minus} part always comes
last. The last two commands are famous for look ahead side effects which is why
macro packages will end them with something not keyword, like \type {\relax},
when needed.

In \LUAMETATEX\ the following is okay. Watch the few more keywords in box and
rule specifications.

\starttyping
\hbox reverse to 10cm attr 123 456 orientation 4 xoffset 10pt spread 10cm { }
\hrule xoffset 10pt width 10cm depth 3mm
\hskip 3pt minus 1pt plus 2pt
\stoptyping

Here the order is not prescribed and, as demonstrated with the box specifier, for
instance dimensions (specified by \type {to} or \type {spread} can be overloaded
by later settings. In case you wonder if that breaks compatibility: in some way
it does but bad or sloppy keyword usage breaks a run anyway. For instance \type
{minuscule} results in \type {minus} with no dimension being seen. So, in the end
the user should not noticed it and when a user does, the macro package already
had an issue that had to be fixed.

\stopsection

\startsection[title=Directions]

The directional model in \LUAMETATEX\ is a simplified version the the model used
in \LUATEX. In fact, not much is happening at all: we only register a change in
direction. The approach is that we try to make node lists balanced but also try
to avoid some side effects. What happens is quite intuitive if we forget about
spaces (turned into glue) but even there what happens makes sense if you look at
it in detail. However that logic makes in|-|group switching kind of useless when
no properly nested grouping is used: switching from right to left several times
nested, results in spacing ending up after each other due to nested mirroring. Of
course a sane macro package will manage this for the user but here we are
discussing the low level injection of directional information.

This is what happens:

\starttyping
\textdirection 1 nur {\textdirection 0 run \textdirection 1 NUR} nur
\stoptyping

This becomes stepwise:

\startnarrower
\starttyping
injected: [push 1]nur {[push 0]run [push 1]NUR} nur
balanced: [push 1]nur {[push 0]run [pop 0][push 1]NUR[pop 1]} nur[pop 0]
result  : run {RUNrun } run
\stoptyping
\stopnarrower

And this:

\starttyping
\textdirection 1 nur {nur \textdirection 0 run \textdirection 1 NUR} nur
\stoptyping

becomes:

\startnarrower
\starttyping
injected: [+TRT]nur {nur [+TLT]run [+TRT]NUR} nur
balanced: [+TRT]nur {nur [+TLT]run [-TLT][+TRT]NUR[-TRT]} nur[-TRT]
result  : run {run RUNrun } run
\stoptyping
\stopnarrower

Now, in the following examples watch where we put the braces:

\startbuffer
\textdirection 1 nur {{\textdirection 0 run} {\textdirection 1 NUR}} nur
\stopbuffer

\typebuffer

This becomes:

\startnarrower
\getbuffer
\stopnarrower

Compare this to:

\startbuffer
\textdirection 1 nur {{\textdirection 0 run }{\textdirection 1 NUR}} nur
\stopbuffer

\typebuffer

Which renders as:

\startnarrower
\getbuffer
\stopnarrower

So how do we deal with the next?

\startbuffer
\def\ltr{\textdirection 0\relax}
\def\rtl{\textdirection 1\relax}

run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur}
run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run}
\stopbuffer

\typebuffer

It gets typeset as:

\startnarrower
\startlines
\getbuffer
\stoplines
\stopnarrower

We could define the two helpers to look back, pick up a skip, remove it and
inject it after the dir node. But that way we loose the subtype information that
for some applications can be handy to be kept as|-|is. This is why we now have a
variant of \prm {textdirection} which injects the balanced node before the skip.
Instead of the previous definition we can use:

\startbuffer[def]
\def\ltr{\linedirection 0\relax}
\def\rtl{\linedirection 1\relax}
\stopbuffer

\typebuffer[def]

and this time:

\startbuffer[txt]
run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur}
run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run}
\stopbuffer

\typebuffer[txt]

comes out as a properly spaced:

\startnarrower
\startlines
\getbuffer[def,txt]
\stoplines
\stopnarrower

Anything more complex that this, like combination of skips and penalties, or
kerns, should be handled in the input or macro package because there is no way we
can predict the expected behavior. In fact, the \prm {linedirection} is just a
convenience extra which could also have been implemented using node list parsing.

Directions are complicated by the fact that they often need to work over groups
so a separate grouping related stack is used. A side effect is that there can be
paragraphs with only a local par node followed by direction synchronization
nodes. Paragraphs like that are seen as empty paragraphs and therefore ignored.
Because \prm {noindent} doesn't inject anything but a \prm {indent} injects
an box, paragraphs with only an indent and directions are handles and paragraphs
with content. When indentation is normalized a paragraph with an indentation
skip is seen as content.

\stopsection

\startsection[title=Hooks]

{\em todo}

\stopsection

\startsection[title=Units]

The familiar \TEX\ units like \type {pt} and \type {cm} are supported but since
the 2021 \CONTEXT\ meeting we also support the Knuthian Potrzebie, cf.\ \typ
{en.wikipedia.org/wiki/Potrzebie}. The two character acronym is \type {dk}. One
\type {dk} is 6.43985pt. This unit is particularly suited for offsets in framed
examples.

In 2023 we added the Edith (\type {es}) and Tove (\type {ts}) as metric
replacements for the inch (\type {in}). As with the \type {dk} more background
information can be found in documents that come with \CONTEXT\ and user group
journals. The \type {eu} unit starts out as one \type {es} but can be scaled with
\prm {eufactor}.

\startbuffer
\localcontrolledloop -5 55 5 {
    \eufactor=\currentloopiterator
    \dontleavehmode\strut
    \vrule height .1es depth .25ts width 1dk\relax\quad
    \vrule height .1es depth .25ts width 1eu\relax\quad
    \the\currentloopiterator
    \par
}
\stopbuffer

\typebuffer

This example code shows all four new units. Watch how \prm {eufactor} is clipped
to a value in the range $1-50$. The default factor of $10$ makes the European
Unit equivalent to ten Toves or one Edith.

\startpacked
\startcolor[darkgray]
\getbuffer
\stopcolor
\stoppacked

In addition to these there can be user units but because these are macro package
dependent they are not discussed here.

\stopsection

\startsection[title=Local control]

There are a few new primitives that permit what we call local controlled
expansion. This permits for instance expanding non expandable macros and even
typesetting inside an expansion context like \type {\edef}. Regular \TEX has a
main loop to where it returns after every primitive action, but local control let
the engine go into a nested main loop.

\stopsection

\startsection[title=Overload protection]

Protection is achieved via prefixes. Depending on the value of the \prm
{overloadmode} variable warnings or errors will be triggered. Examples of usage
can be found in some documents that come with \CONTEXT, so here we just stick to
the basics.

\starttyping
\mutable  \def\foo{...}
\immutable\def\foo{...}
\permanent\def\foo{...}
\frozen   \def\foo{...}
\aliased  \def\foo{...}
\stoptyping

A \prm {mutable} macro can always be changed contrary to an \prm {immutable} one.
For instance a macro that acts as a variable is normally \prm {mutable}, while a
constant can best be immutable. It makes sense to define a public core macro as
\prm {permanent}. Primitives start out a \prm {permanent} ones but with a primitive
property instead.

\startbuffer
          \let\relaxone  \relax 1: \meaningfull\relaxone
\aliased  \let\relaxtwo  \relax 2: \meaningfull\relaxtwo
\permanent\let\relaxthree\relax 3: \meaningfull\relaxthree
\stopbuffer

\typebuffer

The \prm {meaningfull} primitive is like \prm {meaning} but report the
properties too. The \prm {meaningless} companion reports the body of a macro.
Anyway, this typesets:

\startlines \tttf \getbuffer \stoplines

So, the \prm {aliased} prefix copies the properties. Keep in mind that a macro
package can redefine primitives, but \prm {relax} is an unlikely candidate.

There is an extra prefix \prm {noaligned} that flags a macro as being valid
for \prm {noalign} compatible usage (which means that the body must contain that
one. The idea is that we then can do this:

\starttyping
\permanent\protected\noaligned\def\foo{\noalign{...}} % \foo is unexpandable
\stoptyping

that is: we can have protected macros that don't trigger an error in the parser
where there is a look ahead for \prm {noalign} which is why normally protection
doesn't work well. So: we have macro flagged as permanent (overload protection),
being protected (that is, not expandable by default) and a valid equivalent of
the noalign primitive. Of course we can also apply the \prm {global} and \prm
{tolerant} prefixes here. The complete repertoire of extra prefixes is:

\starttabulate
\FL
\NC \type {frozen}     \NC a macro that has to be redefined in a managed way \NC \NR
\NC \type {permanent}  \NC a macro that had better not be redefined \NC \NR
\NC \type {primitive}  \NC a primitive that normally will not be adapted \NC \NR
\NC \type {immutable}  \NC a macro or quantity that cannot be changed, it is a constant \NC \NR
\NC \type {mutable}    \NC a macro that can be changed no matter how well protected it is \NC \NR
\ML
\NC \type {instance}   \NC a macro marked as (for instance) be generated by an interface \NC \NR
\ML
\NC \type {noaligned}  \NC the macro becomes acceptable as \type {\noalign} alias \NC \NR
\ML
\NC \type {overloaded} \NC when permitted the flags will be adapted \NC \NR
\NC \type {enforced}   \NC all is permitted (but only in zero mode or ini mode) \NC \NR
\NC \type {aliased}    \NC the macro gets the same flags as the original \NC \NR
\ML
\NC \type {untraced}   \NC the macro gets a different treatment in tracing \NC \NR
\LL
\stoptabulate

The not yet discussed \prm {instance} is just a flag with no special meaning
which can be used as classifier. The \prm {frozen} also protects against overload
which brings amount of blockers to four.

To what extent the engine will complain when a property is changed in a way that
violates the flags depends on the parameter \prm {overloadmode}. When this
parameter is set to zero no checking takes place. More interesting are values
larger than zero. If that is the case, when a control sequence is flagged as
mutable, it is always permitted to change. When it is set to immutable one can
never change it. The other flags determine the kind of checking done. Currently
the following overload values are used:

\starttabulate[|l|l|c|c|c|c|c|]
    \NC   \NC         \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR
    \NC 1 \NC warning \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
    \NC 2 \NC error   \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
    \NC 3 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
    \NC 4 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
    \NC 5 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
    \NC 6 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
\stoptabulate

The even values (except zero) will abort the run. A value of 255 will freeze this
parameter. At level five and above the \prm {instance} flag is also checked but
no drastic action takes place. We use this to signal to the user that a specific
instance is redefined (of course the definition macros can check for that too).

The \prm {overloaded} prefix can be used to overload a frozen macro. The \prm
{enforced} is more powerful and forces an overload but that prefix is only
effective in ini mode or when it's embedded in the body of a macro or token list
at ini time unless of course at runtime the mode is zero.

So far for a short explanation. More details can be found in the \CONTEXT\
documentation where we can discuss it in a more relevant perspective. It must be
noted that this feature only makes sense a controlled situation, that is: user
modules or macros of unpredictable origin will probably suffer from warnings and
errors when de mode is set to non zero. In \CONTEXT\ we're okay unless of course
users redefine instances but there a warning or error is kind of welcome.

There is an extra prefix \prm {untraced} that will suppress the meaning when
tracing so that the macro looks more like a primitive. It is still somewhat
experimental so what gets displayed might change.

The \prm {letfrozen}, \prm {unletfrozen}, \prm {letprotected} and \prm
{unletprotected} primitives do as their names advertise. Of course the \prm
{overloadmode} must be set so that it is permitted.

\stopsection

\startsection[title=Tracing]

There is are more tracing options, like in math, alignments and inserts, and
tracing can be more detailed. This is partly a aide effect of the need for
exploring new features. Tracing is not always compatible, if only because there
are more possibilities, for instance in the way macros are defined and can handle
arguments.

\stopsection

\stopdocument

% \setupbodyfont[pagella]
% \starttext
% \definefont[TestA][SerifBold*default,oldstyle]
% \def\TestB  {{\infofont[\the\glyphdatafield]}}
% \def\TestC#1{{\infofont(#1)}\csname#1\endcsname}
% \bgroup
%     \crlf abc 123\crlf
%     \bgroup
%         \TestA
%         \TestB
%         \crlf \strut \kern10pt abc 123 \crlf
%     \egroup
%     \TestB
%     \TestC{smallcaps}%
%     \TestB
%     \crlf abc 123 \crlf
%     \bgroup
%         \TestA
%         \TestB
%         \crlf \strut \kern10pt abc 123 \crlf
%     \egroup
%     \TestB
%     \bgroup
%         \TestC{oldstyle}%
%         \TestB
%         \crlf \strut \kern10pt abc 123 \crlf
%     \egroup
%     \TestB
%     \crlf abc 123 \crlf
% \egroup
% \stoptext
