[435] 2026-07-14 15:31:42 ruv wrote:

requestClarification - The data type of `rec-name` output is too wide

Anton wrote on 2026-07-05:

The details of the translations produced by rec-name and rec-float are specified in prose.

I don't see it in the prose. For rec-name the prose says (a fragment):

• rec-name ( c-addr u -- translation )
  If c-addr u is the name of a visible named word, translation represents the text-interpretation semantics (interpreting, compiling, postponing) of that word, and has the translation token translate-name.

From this specification, even the size of the output data object on the stack remains unclear.

If you prefer to put these details in the stack comment, for rec-name that would be rec-name ( c-addr u -- local-sys translate-local | nt translate-name | translate-none )

This is better, but then translate-name (and others) should be specified as a data type symbol.

[r1685] 2026-07-05 10:38:59 AntonErtl replies:

proposal - Recognizer committee proposal 2025-09-11

The details of the translations produced by rec-name and rec-float are specified in prose. If you prefer to put these details in the stack comment, for rec-name that would be

rec-name ( c-addr u -- local-sys translate-local | nt translate-name | translate-none )

I have considered both ways when writing the proposal (and IIRC also wrote it down with the detailed stack comment), and decided to propose the abstract stack comment. One benefit is that it becomes obvious that all recognizers have the same stack effect (at that abstraction level), which is very important for using them in contexts where you do not dissect the translations, but just plug a recognizer in a recognizer sequence, or pass the translation to a consumer like interpreting. In any case, the concrete implementation for the translations for the proposed translation tokens is specified in the proposal (except that local-sys is opaque, so not very concrete).

[r1686] 2026-07-05 11:07:32 AntonErtl replies:

proposal - Recognizer committee proposal 2025-09-11

Recognizer committee proposal 2025-09-11

The committee has found consensus on the words in this proposal. I was asked to write it up.

Author:

M. Anton Ertl (based on previous work by Matthias Trute, Bernd Paysan, and others, and the input of the standardization committee).

Change Log:

• 2026-02-20 Changes to the Rationale (non-normative): Expanded (and renamed) the discussion on (now) "Dropping a translation and filtering", and added an example of that kind of usage to the section "Typical use".

• 2026-02-15 Non-substantive changes based on [r1616], [r1620], [r1621]. Added links to the tests and reference implementation. Added translate-local and locals discussion. Improved "Typical use"

• 2026-02-09 Specify the translation tokens of the rec-... words. Also provide ( -- translation-token ) stack effects for translate-... words.

• 2026-02-08 [r1614] Fleshed out proposal; worked in feedback up to now.

• 2025-09-12 [r1535] Some fixes

• 2025-09-12 [412] Initial version

Problem:

The classical text interpreter is inflexible: E.g., adding floating-point recognizers requires hardcoding the change; several systems include system-specific hooks (sometimes more than one) for plugging in functionality at various places in the text interpreter.

The difficulty of adding to the text interpreter may also have led to missed opportunities: E.g., for string literals the standard did not task the text interpreter with recognizing them, but instead introduced S" and S\" (and their complicated definition with interpretation and compilation semantics).

Solution:

The recognizer proposal is a factorization of the central part of the text interpreter.

As before the text interpreter parses a white-space-delimited string. Unlike before, the string is now passed to the recognizers in the default recognizer sequence rec-forth, one recognizer after another, until one matches. The result of the matching recognizer is a translation, an on-stack representation of the word or literal; the kind of translation is identified by a translation token, which is a part of the translation. The translation is then processed according to the needs of the text-interpreter (interpreting, compiling) or postpone.

There are five usage levels of recognizers and related recognizer words:

1. Programs that use the default recognizers. This is the common case, and is essentially like using the traditional hard-coded Forth text interpreter. You do not need to use recognizer words for this level, but you can inform yourself about the recognizers in the current default recognizer sequence with recs. The default recognizer sequence contains at least rec-name and rec-number, and, depending on the wordsets present, rec-float and rec-local. Postponeing numbers and other reconized things is a feature that a system that implements this proposal provides; this feature is also at this usage level; e.g., postpone 1 is equivalent to 1 postpone literal.

2. Programs that change which of the existing recognizers are used and in what order.

   The default recognizer sequence is rec-forth, a deferred word.

   You can also create a recognizer sequence with rec-sequence:. A recognizer sequence is a recognizer itself, and can be used everwhere where a recognizer can be used, including in a recognizer sequence. One can change rec-forth to call such a sequence.

   You can get the recognizers contained in a recognizer sequence with get-recs and set them with set-recs (including the recognizer sequence in rec-forth). The committee makes no recommendation on how to change rec-forth, as there are different preferences among committee members.

   This proposal contains pre-defined recognizers rec-local rec-name rec-number rec-float, which can be used with set-recs or for defining a recognizer sequence.

3. Programs that define new recognizers that use existing translation tokens. New recognizers are usually colon definitions. Proposed-standard translation tokens are translate-none translate-cell translate-dcell translate-float translate-local translate-name. There is also rec-none, which is sometimes a useful factor when defining recognizers.

4. Programs that define new translation tokens. New translation tokens are defined with translate:.

5. Programs that define text interpreters and programming tools that have to deal with recognizers. Words for achieving that are not defined in this proposal, but discussed in the rationale.

See the rationale for more detail and answers to specific questions.

Reference implementation:

Latest version

This may not be the version corresponding to this proposal, so you may want to look at the history.

Testing:

Latest version

This may not be the version corresponding to this proposal, so you may want to look at the history.

Proposal:

Usage requirements:

Data Types

translation: The result of a recognizer; the input of interpreting, compiling, and postponing; it's a type that consists of a translation token at the top of the data stack and additional data on various stacks; which stack items are required on the data and floating-point stack depends on the translation token.

translation token: (This has formerly been called a rectype.) Single-cell item that identifies a certain kind of translation.

local-sys: A local-sys contains information about a specific local necessary to translate it. The local-sys becomes invalid once the local becomes invisible by going out of scope. Local-sys is a system-compilation type with an implementation-dependent number of stack items.

Translations and text-interpretation

A recognizer pushes a translation on the stack(s). The text interpreter (and other users, such as postpone) removes the translation from the stack(s), and then either performs the interpreting run-time, compiling run-time, or postponing run-time.

All the proposed-standard translate-... words and the words defined with translate: have the stack effect ( -- translation-token ), and each of these words pushes the same translation token every time it is invoked, so you can compare the result of a recognizer against the result of a translate-... word or a word defined with translate:.

In addition the definitions of the translate-... words also show a "Stack effect to produce a translation"; this stack effect points out which additional stack items need to be pushed before the translation token in order to produce a translation. E.g., for translate-dcell this stack effect is ( xd -- translation ), which means that the translation with this particular translation token consists of ( xd translation-token ).

Compiling and postponing run-time

Unless otherwise specified, the compiling run-time compiles the interpreting run-time. The postponing run-time compiles the compiling run-time.

Exceptions

Add the following exception to table 9.1:

-80 too many recognizers

Words

rec-name ( c-addr u -- translation )

(formerly rec-nt)

If c-addr u is the name of a visible local, translation represents the text-interpretation semantics (interpreting, compiling, postponing) of that word, and has the translation token translate-local.

If c-addr u is the name of a visible named word, translation represents the text-interpretation semantics (interpreting, compiling, postponing) of that word, and has the translation token translate-name.

Otherwise, translation is translate-none.

rec-number ( c-addr u -- translation )

(formerly rec-num) If c-addr u is a single-cell or double-cell number (without or with prefix), or a character, all as described in section 3.4.1.3 (Text interpreter input number conversion), translation represents pushing that number at run-time. If a single-cell number is recognized, the translation token of translation is translate-cell, for a double cell translate-dcell. If neither is recognized, translation is translate-none.

rec-float ( c-addr u -- translation )

If c-addr u is a floating-point number, as described in section 12.3.7 (Text interpreter input number conversion), rec-float recognizes it as floating-point number r. If c-addr u has the syntax of a double number without prefix according to section 8.3.1 (Text interpreter input number conversion), and it corresponds to the floating-point number r according to section 12.6.1.0558 (>float), rec-float may (but is not required to) recognize it as a floating-point number. If rec-float recognized c-addr u as floating-point number, translation represents pushing that number at run-time, and the translation token is translate-float. If c-addr u is not recognized as a floating-point number, translation is translate-none.

rec-none ( c-addr u -- translation )

This word does not recognize anything. Its translation and translation token is translate-none.

recs ( -- )

(formerly .recognizers) Print the recognizers in the recognizer sequence in rec-forth, the first searched recognizer leftmost.

rec-forth ( c-addr u -- translation )

(formerly forth-recognize) This is a deferred word that contains the recognizer (sequence) that is used by the Forth text interpreter.

rec-sequence: ( xtu .. xt1 u "name" -- )

Define a recognizer sequence "name" containing u recognizers represented by their xts. If set-recs is implemented, the sequence must be able to accomodate at least 16 recognizers.

name execution: ( c-addr u -- translation )

Execute xt1; if the resulting translation is the result of translate-none, restore the data stack to ( c-addr u -- ) and try the next xt. If there is no next xt, remove ( c-addr u -- ) and perform translate-none.

get-recs ( xt -- xt_u ... xt_1 u )

xt is the execution token of a recognizer sequence. xt_1 is the first recognizer searched by this sequence, xt_u is the last one.

set-recs ( xt_u ... xt_1 u xt -- )

xt is the execution token of a recognizer sequence. Replace the contents of this sequence with xt_u ... xt_1, where xt_1 is searched first, and xt_u is searched last. Throw -80 (too many recognizers) if u exceeds the number of elements supported by the recognizer sequence.

translate-none ( -- translation-token )

(formerly r:fail or notfound)

Stack effect to produce a translation: ( -- translation )

translation interpreting run-time: ( ... -- )

-13 throw

translation compiling run-time: ( ... -- )

-13 throw

translation postponing run-time: ( ... -- )

-13 throw

translate-cell ( -- translation-token )

(formerly translate-num)

Stack effect to produce a translation: ( x -- translation )

translation interpreting run-time: ( -- x )

translate-dcell ( -- translation-token )

(formerly translate-dnum)

Stack effect to produce a translation: ( xd -- translation )

translation interpreting run-time: ( -- xd )

translate-float ( -- translation-token )

Stack effect to produce a translation: ( r -- translation )

translation interpreting run-time: ( -- r )

translate-local ( -- translation-token )

Stack effect to produce a translation: ( local-sys -- translation )

translation interpreting run-time: ( ... -- ... )

-14 throw

translation compiling run-time: ( ... -- ... )

Append the "name Execution" semantics (see 13.6.2.2550 {:) for the local name corresponding to local-sys to the current definition.

translation postponing run-time ( -- )

Perform the translation compiling run-time. Append the compilation semantics of literal to the current definition.

translate-name ( -- translation-token )

(formerly translate-nt)

Stack effect to produce a translation: ( nt -- translation )

translation interpreting run-time: ( ... -- ... )

Perform the interpretation semantics of nt.

translation compiling run-time: ( ... -- ... )

Perform the compilation semantics of nt.

translate: ( xt-int xt-comp xt-post "name" -- )

(formerly rectype:)

Define "name"

"name" exection: ( -- translation-token )

Stack effect to produce a translation: ( i*x -- translation )

"name" interpreting action: ( ... translation -- ... )

Remove the top of stack (the translation token) and execute xt-int.

"name" compiling action: ( ... translation -- ... )

Remove the top of stack (the translation token) and execute xt-comp.

"name" postponing action: ( translation -- )

Remove the top of stack (the translation token) and execute xt-post.

postpone

Interpretation:

Interpretation semantics for this word are undefined.

Compilation: ( "<spaces>name" -- )

Skip leading space delimiters. Parse name delimited by a space. Use rec-forth to recognize name, resulting in translation with translation-token. For a system-defined translation token, first consume the translation, then compile the 'compiling' run-time. For a user-defined translation token, remove it from the stack and execute its post-xt.

Rationale

Names

The names of terms and the proposed Forth words in this proposal have been arrived at after several lengthy discussions in the committee. Experience tells me that many readers (including from the committee) will take issue with one or the other name, but any suggestion for changing names will be ignored by the me. If you want them changed, petition the committee (but I hope they will be as weary of renamings as I am).

In particular, I suggested to use "recognized" instead of "translation", and IIRC also to rename the translate-... words accordingly, but the committee eventually decided to stay with translation and translate-....

Face it: The names are good enough. Any renaming, even if it results in a better name, increases the confusion more than it helps: even committee members (culprits in the renaming game themselves) have complained about being confused by the new, possibly better names for concepts and words that have already been present in Matthias Trute's proposal.

If you want to improve the proposal, please read it, play with the words in Gforth, read the reference implementation and the tests when they arrive, and point out any mistake or lack of clarity.

Translation tokens and translate-... words

[r1541] points out interesting uses of knowledge about translation tokens, and, conflictingly, potential implementation variations. This proposal decides against the implementation variations and for the uses by specifying in the Usage Requirements that a translate-... word just pushes a translation token, and it always pushes the same one.

Moreover, this proposal specifies the translation tokens that the proposed-standard recognizers produce. This is useful in various contexts where recognizers are not used directly in rec-forth, and it also makes it possible to write tests for the recognizers.

Dropping a translation and filtering

[r1541] also asks for a way to drop (discard) a translation. This need has also come up in some recognizers implemented in Gforth (e.g., rec-tick).

In particular, one technique used for building recognizers is to use rec-forth on a substring, but only use the result if it produces a translation with a specific translation token. In the other cases the translation has to be dropped. See the second implementation of rec-tick in Section "Typical Use". The advantage of using rec-forth and, e.g., filtering for translate-name over using rec-name is that when you add other ways that produce translate-name to rec-forth (e.g., rec-scope in Gforth), they also work in the superstring recognizers.

To drop a translation, Gforth uses (non-standard) words like sp@ and sp! for that. Standard options would be to wrap the word that pushes a translation into catch and discard the stacks with a non-zero throw, or to use depth and fdepth in combination with loops of drop and fdrop; both ways are cumbersome, but viable. At the 2026-02-13 meeting the committee decided not to standardize support for dropping translations for now.

Locals

translate-local

Locals need a separate translation token translate-local, because the postponing action differs from that of name tokens for permanent words. Also, there is much variation in how systems represent locals internally during compilation, and many system implementors prefer not to have to change their system to work with nts.

rec-local ?

One contentious issue has been how to recognize locals: Should it happen in rec-name or in a separate rec-local? This reflects the existing divergence among systems, some of which find or find-name locals and some of which use a mechanism outside these words. I think that both options are relatively easy to implement for all systems, and that there is currently no legacy based on having a separate rec-local or not. OTOH, leaving this decision undecided is going to significantly reduce the usefulness of this proposal: Already usage level 2 (Programs that change which of the existing recognizers are used and in what order) would become much more cumbersome for many uses, and other uses may become completely unusable. For now I have kept recognizing locals in rec-name, but changing the proposal to have a separate locals recognizer would just mean removing the local recognition paragraph from rec-name and adding:

rec-local ( c-addr u -- translation )

If c-addr u is the name of a visible local, translation represents the text-interpretation semantics (interpreting, compiling, postponing) of that word, and has the translation token translate-local. If not, translation is translate-none.

Postponeing a local

The postponing action for a local x is specified such that writing POSTPONE x is equivalent to x POSTPONE LITERAL. This is the current behaviour of Gforth. It is easy to implement and has proven to be useful ("notation matters"). However, probably no other Forth system implements this behaviour (yet), so an alternative is to specify that the postponing action throws or performs an implementation-defined action, and I am prepared for changing the proposal accordingly.

Consumers of translations (Usage level 5)

The committee has decided not to standardize words that consume translations for now. Such words would be useful for defining a user-defined text interpreter, but the experience with recognizers has shown that a recognizer-using text interpreter is flexible enough that it is no longer necessary to write such text interpreters, so such words are only used internally in the text interpreter, eliminating the need to standardize them.

However, to give an idea how all this works together, here's the words that Gforth provides for that purpose:

interpreting ( ... translation -- ... )

For a system-defined translation token, first remove the translation from the stack(s), then perform the interpreting run-time specified for the translation token. For a user-defined translation token, remove it from the stack and execute its int-xt.

compiling ( ... translation -- ... )

For a system-defined translation token, first remove the translation from the stacks, then perform the compiling run-time specified for the translation token, or, if none is specified, compile the 'interpreting' run-time. For a user-defined translation token, remove it from the stack and execute its comp-xt.

postponing ( ... translation -- )

For a system-defined translation token, first consume the translation, then compile the 'compiling' run-time. For a user-defined translation token, remove it from the stack and execute its post-xt.

Typical use:

\ Usage level 1

\ Just program in Forth

\ You want to see what recognizers are in rec-forth:
recs

\ And here's an example of using POSTPONE in a new way:
: [foo] postpone #123 ; immediate
: foo [foo] ;
foo . \ prints 123

\ Usage level 2

\ let's check for a float before an integer
action-of rec-forth constant rec-forth-default
' rec-number ' rec-float ' rec-name 3 rec-sequence: rec-nfn
' rec-nfn is rec-forth
\ whether the following are recognizes as integers or floats depends
\ on your rec-float
123 .   \ in Gforth rec-float does not recognize that
123. f. \ in Gforth rec-float recognizes that
\ restore the old world order
rec-forth-default is rec-forth

\ you can also check a particular string with a particular recognizer:
s" 123" rec-float \ in Gforth, returns the same as TRANSLATE-NONE

\ Usage level 3

\ A recognizer that recognizes `\<name\> and produces the xt of \<name\>
: umin ( u1 u2 -- u )
  2dup u< if drop else nip then ;

: string-prefix? ( c-addr1 u1 c-addr2 u2 -- f )
    tuck 2>r umin 2r> compare 0= ;

\ two implementations
\ Implementation 1 based on find-name
: rec-tick1 ( addr u -- translation )
    2dup "`" string-prefix? if
        1 /string find-name dup if
            name>interpret translate-cell
        else
            drop translate-none then
        exit then
    \ this recognizer did not recognize anything, therefore:
    rec-none ;

\ Implementation 2 based on filtering the output of rec-forth
: filter-rec ( c-addr u xt-rec xt-filter -- translation f )
    \ apply xt-rec ( c-addr u -- translation ) to c-addr u, then run
    \ xt-filter ( translation -- translation f ) on the resulting
    \ *translation*; if *f* is true, return *translation f*, otherwise
    \ return *translate-none f*.
    depth 4 - >r fdepth >r >r execute r> execute dup if
        2r> 2drop
    else
        fdepth r> ?do fdrop loop
        depth  r> ?do  drop loop
        translate-none false
    then ;

: name-filter ( translation -- translation f )
  dup translate-name = ;

: rec-tick2 ( addr u -- translation )
    2dup "`" string-prefix? if
        1 /string ['] rec-forth ['] name-filter filter-rec if
            drop name>interpret translate-cell then
        exit then
    \ this recognizer did not recognize anything, therefore:
    rec-none ;

\ And now install one of them last in rec-nfn
\ ' rec-tick1 ' rec-nfn get-recs 1+ ' rec-nfn set-recs
' rec-tick2 ' rec-nfn get-recs 1+ ' rec-nfn set-recs

\ now use it:
' rec-nfn is rec-forth
5 `. execute \ prints 5
\ restore the default recognizer sequence
`rec-forth-default is rec-forth

\ Usage level 4

\ Here's how you could define the proposed-standard translate-cell yourself
: lit, postpone literal ;

' noop                       ( x -- x )                             \ int-xt
' lit,                       ( compilation: x -- ; run-time: -- x ) \ comp-xt
:noname lit, postpone lit, ; ( postponing: x -- ;  run-time: -- x ) \ post-xt
translate: translate-cell