There are a number of discussions going on elsewhere, and I want to make sure the discussion is going to be here. So for a start, I just took Matthias' version D proposal.

Ideas discussed:

• Bikeshedding the names again (we already did that).
• Revert the postpone action to version A
• Introduce RECOGNIZER-SEQUENCE: word, which creates a new recognizer that actually consists of a sequence of those.

Please feel free to introduce these ideas here.

The "RECOGNIZER-SEQUENCE:" proposal is now online: Nestable Recognizer Sequences.

There are a number of discussions going on elsewhere, and I want to make sure the discussion is going to be here.

There are too many questions to discuss all of them on a single page.

I opened several issues on the subject of review API v4. Please check.

Some notable of them:

• side effects are not acceptable (Issue #7)
• arguments against VALUE in API (Issue #5)
• accessors are not needed (Issue #6)
• new arguments concerning "unrecognized vs zero" (Issue #4)
• choosing better names (Issue #3)

I suggest the following road map:

1. Terms definitions and data types
2. A minimal essential part
3. Choose among the different approaches (e.g., I suggest to discard both postponing and reproducing actions in the public API).
4. Choose solutions for some problems
5. Choose names

Take a note on Comparison of terminology to some past proposals.

For example, it shows that "to translate" is important and very useful term.

I adapted the suggested term changes of wrt to the Recognizer-A-Proposal to the latest D-Proposal:

| Term in proposal D        | Term suggested                     | comment                               |
| ------------------------- | ---------------------------------- | ------------------------------------- |
| recognizer stack          | recognizer-order                   | similar to search-order               |
| data type id              | recognizer information token (rit) | explicit and consistent               |
| RECOGNIZE                 | RECOGNIZE                          |                                       |
| RECTYPE:                  | RECOGNIZER                         | similar to WORDLIST, no defining word |
| RECTYPE-NULL              | UNRECOGNIZED                       | better english                        |
| REC-SOMETYPE              | recognize-xxx                      | better english                        |
| RECTYPE-SOMETYPE          | xxx-recognized                     | better english                        |

The items to be discussed mentioned in the preamble at the beginning of Recognizer RfD rephrase 2020 remain valid.

The output of the scanner is often called "token class":

From Wikipedia (Lexical Analysis)

Lexers and parsers are most often used for compilers, but can be used for other computer language tools, such as prettyprinters or linters. Lexing can be divided into two stages: the scanning, which segments the input string into syntactic units called lexemes and categorizes these into token classes; and the evaluating, which converts lexemes into processed values.

Also see "Modern Compiler Design" Second Edition, Dick Grune • Kees van Reeuwijk • Henri E. Bal • Ceriel J.H. Jacobs • Koen Langendoen they use the same term.

So it might be reasonable to call RECTYPE: just TOKEN-CLASS.

or maybe TOKENCLASS ?

TL;DR: token class from lexers is a wrong association.

The "token" term in lexical analysis

1. in computer science, the token term is used in various meanings.

2. In lexical analysis (and compilers theory), token is actually a shorthand for lexical token. (It's the same as in the Forth topic, the "definition" term is a shorthand for "Forth definition").

3. A lexical token is a tuple of a lexeme and the kind of this lexeme

(it's my rewording from Wikipedia, and also: "The lexeme's type combined with its value is what properly constitutes a token").

A lexical token usually doesn't bear any semantic information, it bears only lexical kind — it it an identifier, a number, string literal, or a particular key word.

Lexical tokens are not used in Forth since Forth doesn't distinguish lexemes by the different kinds.

4. In Forth, the token term is not used in the sense of lexical token. As we can conclude from the "execution token" and "name token" terms, in the Forth standard a token is just a kind of identifier, symbol, or something that represents something another (in general case; but numbers can represent themselves).

Lexical token class

The output of the scanner is often called "token class"

It's not quite correct. Output of the scanner (i.e. a lexer) is a sequence of lexical tokens.

Concerning "token class" — it is a shorthand for "lexical token class".

And actually, token class, token type, token category, token name (in the famous Dragon book) — all of them refers to the same thing, that I have called lexeme kind above.

Qualification of the tokens in Forth

We need to name the entity that qualifies a token.

We can call it "token class" (but without referring to "token class" in lexers). Initially I called this entity "token type". But then I realized that "token descriptor" is far better.

This entity can be created in run time, and it describes how to translate a token, — it is not an abstraction, it is an actual object (and the identifier of this object). So "create descriptor" shorthand sounds better than "create class" or "create type" (that look as more abstract things).

Only when we find names for the terms (in the human language, in the language of the standard ), we can find good names for words.

In Forth we have a number of tokens already: execution token, name token. Therefore, the terminology of a book about compilers would be misleading in Forth (cf. the meaning of "word" in a computer architecture book).

Moreover, different recognizers often produce the same rectype. E.g., in Gforth, rectype-num is returned by at least rec-num, rec-tick, and rec-body. So the role of a rectype is different from that of a token class in Grune's book.

One advantage that the rectype-* names have over tokenclass-* is that the association with recognizers is more obvious.

Finally, the committee already had a discussion on the names, and agreed on the rectype-* names. As far as I am concerned, that is good enough. And in particular, I think that the incremental benefits of better names (even if one could be agreed upon, which is doubtful) are not worth the costs of trying to find it, finding consensus on it, changing the existing code and documentation, and making existing discussions about this concept harder to understand.

One advantage that the rectype-* names have over tokenclass-* is that the association with recognizers is more obvious.

But it should have association with tokens, not with recognizers!

What is etymology of "rectype" ?

I see the following disadvantages of "rectype":

1. It's not an English word; it's an abbreviation, and it isn't explained.
2. "rec" makes the first association with "record" that nothing to do with recognizers.
3. "rectype" makes the first association that it's a type of record, that is wrong. The second association is that it's a type of a recognizer — that is also wrong.
4. "rectype" describes a token, not a recognizer, but it refers to a recognizer.

E.g., why does the entity that describes "execution token" is called "rectype of execution token"? (My suggestion is: "descriptor of execution token").

not worth the costs of trying to find it, finding consensus on it, changing the existing code and documentation,

When we make a mistake, we pay for this mistake. "rectype" is a mistake in a name choice, — it seems, the most of us (who works on recognizers proposals) understand it, but didn't want to find the better name in an earlier stage, and postponed this choice to a later stage. And now you say this is not worth to change this name.

So we should have make this correction earlier. And now, I believe, we should pay the price, and this price worth this mistake.

Actually, the cost of changing the code is a weak argument. The internal code is not required to be updated (it's enough to make synonyms), and it also can be updated via auto-replacing.

The cost of changing the documentation is even a more weak argument. It can be updated via auto-replacing. But its terminology is wrong in any case, and this terminology should be fixed manually in the far more places.

Concerning "trying to find it, finding consensus on it" — we didn't even try it: we have only two alternative suggestions (perhaps, only one already). And nobody said that "rectype" by itself is better than "token descriptor" for our purpose.

A proof-of-concept dot-parser recognizer

There have been questions about writing a dot-parser as a recognizer, in particular wrt. POSTPONE and the way that works in the present proposal. Here I present a proof-of-concept of such a recognizer.

A proof-of-concept ClassVFX subset and its dot-parser

A dot-parser is used in the context of a way to control name spaces, as is typical for an object-oriented package. So first we implement a subset of ClassVFX (which has a dot-parser), without implementing the ClassVFX syntax exactly. Here's an example of it's usage:

type: point:
  int: x
  int: y
end-type

type: line:
  point: start
  point: end
end-type

instance: point: p1
instance: point: p2
instance: line: l1

This is essentially a structure package. The fields x and y are in a wordlist private to point:; likewise, start and end are private to line:. So how do we access these fields? We access them by writing point:.x and line:.start. The dot-parser is responsible for recognizing these "words".

In order to avoid having to write line:.start @ point:.x, you can combine these into line:.start.x (for arbitrarily long sequences).

In this proof-of-concept, the start and end fields in line: contain the address of a point: instance, not the point: itself. For the recognizer, this is the harder problem, because the dot-expression corresponds to a sequence like offset1 + @ offset2 + @ offset3 + (with the length of the sequence depending on the number of dots), while with the other variant a word equivalent to offset + would be sufficient.

You can find the complete implementation with examples (tested on gforth 0.7.9_20200917) in dot-parser.fs. Below you find the implementation of the recognizer.

Dot-parser implementation

The questions were about writing the dot-parser recognizer, so here I'll explain the code in more detail.

A central point is how to represent the dot-parsed "word". I represent it as a sequence of xts plus an integer sprecifying the number of xts in the sequence. E.g., line:.start.x is represented by the xts of start @ x (where start is in the line:-private wordlist, and x in the point:-private wordlist) followed by 3 (the count).

The recognizer itself has the stack effect

rec-dot-parser ( c-addr u -- xt1 ... xtn n rectype-dp | rectype-null )

For the rectype-dp we need three actions. The first is the action for interpreting the "word"; it just executes the xts, starting with xt1:

: dp-int ( ... xt1 .. xtn n -- ... )
    \ remove xt1 .. xtn n from the data stack, then execute xt1 .. xtn.
    dup if
    swap >r 1- recurse r> execute exit then
    drop ;

The action for compiling the "word" compiles these xts into the current definition:

: dp-comp ( xt1 .. xtn n -- )
    \ compile, xt1 .. xtn, in this order
    dup if
    swap >r 1- recurse r> compile, exit then
    drop ;

POSTPONE compiles dp-comp into the current definition (rather than executeing it). But because this compiled dp-comp needs the xts at a different time than when it is available from the recognizer, we need a literal-like action to get the xts from the recognizer time to the time when this dp-comp finally runs. This literal-like action is called the "postpone action" in the proposal. Anyway, its implementation is similar to that of dp-int and dp-comp:

: dp-lit1 ( x1 .. xn n -- )
    \ compile x1 .. xn as literals, in this order
    dup if
    swap >r 1- recurse r> postpone literal exit then
    drop ;

t{ :noname [ 2 3 4 3 dp-lit1 ] ; execute -> 2 3 4 }t

: dp-lit ( xt1 .. xtn n -- )
    \ compile xt1 .. xtn n as literals
    dup >r dp-lit1 r> postpone literal ;

Once all these actions exist, we can define rectype-dp:

' dp-int ' dp-comp ' dp-lit rectype: rectype-dp

The recognizer itself is a more complex piece of code, but not particularly important for understanding how postponeing a dot-parsed piece of code works, so I'll not explain it in detail:

: split ( c-addr1 u1 c-addr2 u2 -- c-addr3 u3 c-addr4 u4 true | c-addr1 u1 false )
    \ If c-addr2 u2 is found in c-addr1 u1, return true, and c-addr3
    \ u3 and caddr4 u4 are the parts to the left and the right of the
    \ found string.  If not return c-addr1 u1 and false
    2over 2>r dup >r search if
    over swap r> /string 2r> 2swap 2>r drop tuck - 2r> true
    else
    r> drop 2r> 2drop false
    then ;

: rec-dot-parser ( c-addr u -- xt1 ... xtn n rectype-dp | rectype-null )
    \ this leaves out the handling of a number of cases resulting in
    \ rectype-null in the interest of showing the successful case more clearly
    s" ." split 0= if
    2drop rectype-null exit then
    2swap find-name \ !! deal with not-found and not-\<typename\>
    name>interpret >body @ >r 1 -rot begin ( xt1 .. xtn n c-addr1 u1 r:wid )
    s" ." split while
        2swap r> find-name-in \ !! deal with not-found and not-\<fieldname\>
        name>interpret dup >body cell+ @ @ >r
        -rot 2>r ['] @ rot 2 + 2r>
    repeat
    r> find-name-in \ !! deal with not-found and not-\<fieldname\>
    name>interpret swap rectype-dp ;

Note that this recognizer does not properly handle the cases where the string contains a dot, but should not be recognized by the dot-parser (it's a proof-of-concept).

In gforth 0.7.9_20200917, this recognizer is searched last with

' rec-dot-parser get-recognizers 1+ set-recognizers

Usage examples

Using the definitions of point:, line:, their fields, and instances l1, p1, p2, you can do:

\ interpretive uses:
p1 l1 line:.start !
8  l1 line:.start.y !

\ compiled use:
: foo line:.start.y @ ;

\ postpone use:
: bar postpone line:.start.x ; immediate
: flip bar ;

Gforth's see decompiles foo, bar, and flip as:

: foo  
  start @ y @ ;
: bar  ['] start ['] @ ['] x 3 
  dp-comp ; immediate
: flip  
  start @ x ;

Just for comparison, in the minimalistic API, instead of dp-int, dp-comp, dp-lit1, dp-lit and rectype-dp words (five words in total) we should define only one general purpose word tt-nxt:

: tt-nxt ( ... xt1 .. xtn n -- ... )
    \ remove xt1 .. xtn n from the data stack, then translate xt1 .. xtn one by one in this order.
    dup if
    swap >r 1- recurse r> tt-xt exit then
    drop ;

Changes in rec-dot-parser is that rectype-dp is replaced by ['] tt-nxt.

The Rationale for this proposal was split away by Matthias Trute to satisfy complaints about the length of the proposal, but can be useful in understanding the design decisions.