I think this work on rephrasing, making better terminology and wording, and even reforming the conceptions, — is very important.

My thoughts regarding the terminology are the following.

3. The word "RECOGNIZER".

It seems the most bright conflict in the terminology lays between RECOGNIZER and GET-RECONIZERS words. The former returns rit, the latter returns rec. Such conflict is inadmissible in the Standard. Another term (and another name for the word) should be found instead of "recognizer".

Anyway, holding for this word a kind of semantic similarity to the WORDLIST word looks like a good choice (if any).

5. A name for triple of interpret/compile/postpone xts

This item is closely connected to the above one (3).

It should be taken into account that we already have execution token xt (that identifies execution semantics) and name token nt (that identifies a named definition). I.e., the specification applies the term "token" to an attribute itself (a value only), without the corresponding information about its type (or class, or kind). Hence, the information about the corresponding type should be called "token type" (an identifier of a token type).

Under the hood, this token type identifier should be associated with handlers: how to execute (interpret) the corresponding token, how to compile the corresponding token, etc.

See also my approach in comp.lang.forth post in 2018 (news:pngvcc$pta$1@gioia.aioe.org, copy).

4. Changes in the specifications of other words

Yes, the specification for MARKER should be updated.

But there's no need to mention changes in the behavior of the words that:

• a) do interpretation (like EVALUATE, INCLUDE-FILE, INCLUDED), — it is enough to change 3.4 The Forth text interpreter.

• b) do finding according to the current search order (like ', ['], POSTPONE, [COMPILE], [DEFINED], [UNDEFINED]), — it is enough to change 3.4.2 Finding definition names (for example, see 16.3.3 Finding definition names).

The items (1) and (2) are more about API than about terminology.

2. rit structure accessors

I think, suggesting programs to use N CELL+ @ to access xt — is a bad choice. Even for the mentioned counted strings we have the COUNT word, i.e. an accessor. Therefore, even for a transparent structure, if we provide an access, we should provide the access words. (But I think, we don't need access to xts, see 1.ii below)

1. interpret/compile/postpone xts

"Programs [...] most likely need to use the interpret/compile/postpone xts of the returned recognizer information token. [...] Without these access words standardizing the word RECOGNIZE is doubtful."

We have the following issues with this:

i. Nobody provides a use-case

Nobody provides a use-case for the scenario when a program needs access to these xts.

OTOH we have a one important principle: a user-defined text interpreter should be implementable in a standard way (in this case, without re-implementing recognizers). NB: POSTPONE-action is not needed for this text interpreter.

ii. Better to avoid successors

Perhaps another way is better. Instead of using the corresponding xts, use the words that do the corresponding actions:

INTERPRET-TOKEN ( i*x token{k*x} token-type -- j*x )
COMPILE-TOKEN ( i*x token{k*x} token-type -- j*x )
POSTPONE-TOKEN ( token{k*x} token-type -- )

I.e., in place of

( ... rit ) _R>COMP EXECUTE

you just do

( ... rit ) COMPILE-TOKEN

Rationale: in the most cases a user/program needs just to perform these actions. Getting an xt and then executing it has an excessive step without any profit.

Other issues regarding the API and examples.

6. Naming in A.XY.2

find-name was proposed to be a standard word that returns a name token nt.

It is better to use another name for your word ( addr len -- xt +/-1 | 0 ).

I can suggest

find-word-in ( c-addr u  wid -- c-addr u 0 | xt immediate-flag true )
find-word ( c-addr u -- c-addr u 0 | xt immediate-flag true )

Rationale: 1) the FIND word returns c-addr on fail 2) when implementing a text interpreter, on fail you need ( c-addr u ) to convert it into the number; 3) in some cases, always the same number of the result items is an advantage for optimization.

7. Action of postponing isn't essential

i. Why does a user (a program) need to use the POSTPONE-action?

The only known use case is to implement ]] ... [[ construct. But this construct, when implemented via the POSTPONE-action, have a set of known flaws: it doesn't follow copy-pastebility design principle, and (as the result) it doesn't handle the immediate parsing words in a convenient way (including comments, and [IF] ... [THEN]).

Actually, a user doesn't need to use the POSTPONE-action, but he just needs to postpone fragments of code! Therefore, it is better to provide something like c{ ... }c construct (see my s-state PoC) that provides full copy-patebility.

ii. Why does a user (a program) have to specify POSTPONE-action for a new token-type?

The only reason is to make a Forth system aware of how to apply POSTPONE to a user-defined literal. But a user doesn't need to apply POSTPONE to the literals if a Forth system provides a way to postpone any fragments of code.

Well, perhaps a Forth system cannot postpone a user-defined literal (or even a parsing word) as part of a fragment of code, if a user doesn't provide POSTPONE-action? But it is wrong. Since any COMPILE-action is defined via the standard words (and the words defined via standard words), then a Forth system is able to postpone any tokens, having a definition of COMPILE-action for them (see the same c-state PoC)

Yes, it is not quite easy in implementation, but it is very convenient in using!

Another open question

8. Dependency on STATE

Obviously, the results of RECOGNIZE may depend on search order and BASE. Also, a user-defined recognizer may depend on user-defined states.

But what about STATE-dependency for initial recognizer? May the results of RECOGNIZE depend on STATE?

E.g. recognize-word from example in A.XY.2 is based on FIND, and hence it may depend on STATE. And hence in some cases the result is not allowed to be performed in the different STATE (see also my proposal for FIND clarification).

Yes - this text has the potential of starting a bikeshedding discussion but as the recognizer concepts seem to be stable over the last couple of years it is about time to agree on appropriate names and notions.

It seems to me that the seeming stability is treacherous. Stephen Pelc wants to go back to an earlier version of the proposal, and Alex McDonald wants to revise it. We have had renamings already, and as a result, it is not as easy to compare versions as it could be.

However, we have also had a new concept that got the name of an old concept that it replaced: the postpone action was replaced by a time-shifting action (from which the postpone action can be built), but the time-shifting action was still called "postpone action". I am not sure if changing the name now would be helpful, or if it's too late.

I apologise for my delay in responding to Ulli's document.

Overall I think that it's a really good first step, and Ruv's comments are also good.

I do not want to go back to an older proposal in particular, I want a proposal that ordinary mortals can understand. I just want clarity.

Naming

I don't much care whether the recogniser triple is called a rid or a rit. The more neutral term seems to be rid, but rit is now in use, so let's keep to it. Either can be pronounced clearly in discussions.

In normal Forth usage the word that lays the implementation- dependent data would be called RECOGNIZER, (with a comma). It should return a rit. What's the point of having an identifier if we don't use it? If we use this terminology, then the obvious way to refer to return values is RIT-NUM, RIT-FLOAT and so on.

Accessors

Do these ever get used outside the internals of other words? If not, a standard team has no business prescribing these. How many other implementations of a rit exist (in the wild) apart from the xt triple? Ruv's point about a use-case is well taken here.

The POSTPONE action xt is needed for two reasons:

1. POSTPONE needs it
2. Not all parsers are for literals, e.g. OOP parsers.

We cannot predict how recognisers will be used, so attempting to automate the POSTPONE actions is doomed to failure. OOP is not a hand-waving prediction, VFX's CIAO and ClassVFX packages both use recognisers.

STATE dependency

Having RECOGNISE be dependent of STATE is horrible.

Regarding time-shifting action

I am not sure if changing the name now would be helpful, or if it's too late.

I think correcting the name (and corresponding terminology) will be helpful. Since we should not call something "postpone action" if it isn't actually "postpone action" (well, perhaps it is "postponing" in some sense, but not in the sense of POSTPONE word, so it is confusing). Moreover, we still need to refer in discussions the both conceptions: full postpone action and "time-shifting" action.

OTOH, it seems "time-shifting" is a sub-optimal term.

What the corresponding Forth definition should do? It should take the token from the stacks (the data stack, floating-point stack, or something else) and compile code that when executed will place the token on the stacks. In other words it compiles the token as numbers (i.e. literally). So it is distinct from the "compile action" that performs the compilation semantics for the source lexeme.

I suggest the term reproducing. So we can have: interpreting action, reproducing action, compiling action, postponing action (if any).

to reproduce a token: to take the token and compile code that when executed will place the token on the stacks

Solution for items 3 and 5.

I also have found a solution for a "RECOGNIZER" mess and a "triple" of xts.

I suggest the term token descriptor (or just descriptor).

token descriptor: an implementation dependent data object (a set of information) that describes translating a token.

Also I have one more idea to discuss how to avoid providing all these (one, or two, or three, or even four) actions when you create a token descriptor.

So far so good.

Further development

I created SpfDev team in ForthHub, and fep-recognizer private repository to design the specification for Recognizer (fep from Forth Enhancement Proposal, after PEP). I included some people that I have found in ForthHub and who made proposals here. Write to me here or in a distinct issue if you are interested to be included.

I published a draft for terms definitions and data types, and an issue for feedback. I created this draft since I see too more issues in the current proposals from the formal point of view, and no answers. See also news:rb43tl$elj$1@dont-email.me (copy). Now I'm looking forward to your thoughts.

Please let me know if you think it is worth to make fep-recognizer repository public (since Gerald Wodni said "normally proposals are developed in smaller groups and only presented to the public once they are pretty solid"). I hope the GitHub tools help us to better organize collaboration on this work.

Naming of return values

There's a proposal that we should standardise the values returned by RECOGNIZE (RECTYPE-xxx, rit-xxx ...). After a while debugging two new FP packages on the same host, and even loading one after the other, I believe that this proposal is doomed to fail.

If two float packs return the same value, they are impossible to distinguish and hence to debug. If we return rit-SSE64 and rit-NDPfloat then they can be separated. The source code becomes impenetrable without separate names.

We should also acknowledge that parsers may return one or more rits on success, e.g R:SNUM and R:DNUM . There's no point saying "don't do that"; such systems exist in the wild. They are a natural consequence of what Ruvim calls compound recognisers.

Standardizing token descriptors

I think, a set of the basic descriptors should be standardized. We need the descriptors at least for the following tokens: xt, nt, x, xd, f, c-addr u (a string), and also an implementation dependent token for the result of FIND (for back compatibility) — seven descriptors in total.

Should the values be standardized (as in the case of throw codes), or the names? The values are required for binary interoperability — not our case at the moment; also, they allow to reduce the number of names. But in any case, it is better to have names instead of just magic numbers in source code.

I would suggest to form these names using a mnemonical prefix TD- (after "Token Descriptor"). E.g. TD-XT, TD-NT, TD-LIT, TD-2LIT, TD-FLIT, TD-SLIT, TD-WORD.

The names TD-LIT, TD-2LIT, TD-FLIT, TD-SLIT are after the corresponding standard words LITERAL, 2LITERAL, FLITERAL, SLITERAL.

Other variants for them: TD-NUM, TD-2NUM, TD-FLOAT,TD-STRING — are not well mapped into the corresponding words for compilation the tokens.

An FP package, as well as any other, may provide its own token descriptor, if it is reasonable.

Why do we need standard token descriptors?

In many cases a user-defined recognizer returns either a token of these basic data types: xt, nt, x, xd, f, (c-addr u), or a tuple of such tokens. Also, the most standard recognizers (i.e. that can be standardized) return some of these tokens too. So, one argument is that the corresponding descriptors have very high factor of reusing.

The second argument is that they are needed to analyze a result of the standard compound recognizers (if any).

The third argument is that in some cases their can help to create a new user-defined descriptor.

One approach to create descriptors

If a user-defined recognizer returns a token that is just a tuple of other known tokens, a descriptor for this token can be created from the tuple of the corresponding descriptors. I.e., there is no need in this case to specify interpreting action, compiling action, reproducing action, etc.

For example, in my comparison of Recognizer and Resolver APIs a token is represented as ( d-numerator u-denominator ) pair. And the descriptor TD-3LIT for this token can be defined as:

TD-2LIT TD-LIT 2  DESCRIPTOR-COMPOSITE{ TD-3LIT }

Or, if a user needs a distinct descriptor for an already known token:

TD-FLIT 1 DESCRIPTOR-COMPOSITE{ TD-FLIT-SPECIAL }

OTOH, at the moment I don't sure that such a method of creating a descriptor is worth to be standardized.

I personally prefer a more easy way, that is based not on the token descriptors but on the token translators. For token descriptors, not every user-defined token descriptor can be created from other standard descriptors. But any user-defined token translator can be always created from other standard token translators, so we don't need to specifying the particular actions at all.

Terms definitions

to create a token descriptor (informally): to create an implementation dependent token descriptor object producing its identifier or producing a named definition that returns this identifier. (see other terms at GitHub/ForthHub/fep-recognizer)

An approach to create any token descriptor

If we will add one special predefined descriptor, we will be able to express any possible semantics for the token descriptors, without providing a reproducing or a postponing action.

This descriptor should describe a token that is a pair of xt: ( xt-interpret xt-compile ). Let's call it TD-DUAL for a while.

And let's consider an example: the string literals (see the parse-slit-end definition there). The old recognizer: rec-sliteral ( c-addr1 u1 -- c-addr2 u2 rectype-slit|rectype-slit-parsing | 0 )

The new one:

\ Create the token descriptor for a token ( c-addr u  xt-interp xt-compil )
td-slit td-dual 2 descriptor constant td-slit-parsing

\ A helper that returns the second part of the fully qualified token
: token-slit-parsing ( -- xt-interp xt-compil td-slit-parsing ) ['] parse-slit-end  [: parse-slit-end slit, ;]   td-slit-parsing ;

\ The recognizer for a string literal: a lexeme that starts with '"' (quote)
: recognize-sliteral ( c-addr1 u1 -- c-addr2 u2 td-slit | c-addr2 u2 xt-interp xt-compil td-slit-parsing | 0 )
  dup 0= if nip exit then
  over c@ '"' <> if 2drop 0 exit then
  1 /string dup 0= if token-slit-parsing exit then
  \ todo: fail if '"' is found in the middle of the string (c-addr1 u1)
  2dup + char- c@ '"' = if char- td-slit exit then
  token-slit-parsing
;

NB: we have to provide neither a reproducing action nor a postponing action. POSTPONE "abc" works as expected.

POSTPONE "x can work in the same way as POSTPONE S" — i.e., to append the compilation semantics for the corresponding lexeme. And the compilation semantics for "x is to parse characters up to ", prepend the result string with "x " and compile this united string. Although, I don't sure it has a practical application. If supported, it should work in this way from conceptual point of view, but it is not obligated to be supported.

Automation of POSTPONE action

StephenPelc wrote:

We cannot predict how recognisers will be used, so attempting to automate the POSTPONE actions is doomed to failure.

I know only one case when POSTPONE action cannot be automated: a special behavior that Anton suggested when POSTPONE is applied to a local variable, namely that POSTPONE local-x should be equivalent to local-x POSTPONE LITERAL, that is local-x LIT,. Actually, this behavior violates the specification for POSTPONE, that should append the compilation semantics for local-x in this case. And the only reason for this violation is to seamlessly use of local variables inside ]] ... [[ construct (in the simplest implementation).

Could somebody provide a correct POSTPONE action that cannot be automated?

Yes; for example a JIT compiler that might want a set of actions where the intent is not to immediately interpret, compile or postpone but to provide input for a later compilation pass.

Just to close this question concerning a JIT compiler:

for example a JIT compiler that might want a set of actions where the intent is not to immediately interpret, compile or postpone but to provide input for a later compilation pass.

If the action for compilation (and reproduction) is defined in such a way that it provides input for a later compilation pass, then the automated action for POSTPONE will generate the input for a later compilation pass too, automatically. Since the action for POSTPONE is just expressed via other actions.

So this example of a JIT compiler doesn't show that POSTPONE action cannot be automated.

To be more concrete, please provide a particular code example when POSTPONE action cannot be automated, by your view.

Could somebody provide a correct POSTPONE action that cannot be automated?

A postpone action is automated via 1) a reproduce action, 2) a compile action, and 3) the system's compile, word.

In some edge cases, these components are not consistent with each other, namely a compile action may generate code that is not compatible with what the system's compile, generates. In such cases a correct postpone action cannot be automatically generated (see also my post Against a reproducer in a token descriptor).

So between a reproduce (partial postpone) action and a full postpone action the latter one should be chosen.

Personally, I prefer not he descriptor-based approach, but the translator-based approach, which allows to implement a postponing mode. This mode works transparently and automatically everywhere. And it is a far more convenient means than postpone for user-defined literals.