proposal - New words: latest-name and latest-name-in

Author

Ruv

Change Log

• 2023-10-22 Initial revision

Problem

In some applications, mainly in libraries and extensions, the capability to obtain the most recently added definition is very useful and demanded.

For example, if we are creating a library for decoration, tracing, support for OOP, simple DSLs (e.g., to describe Final State Machines), etc — it is always useful to have an accessor to the recent definition, instead of redefining a lot of words to define such an access method yourself, or juggling with the input buffer and search.

However, many Forth systems have such internal methods to access the recently added word. Among them: latest ( -- nt|0 ), last @ ( -- nt|0 ), latestxt ( -- xt|0 ), etc.

And additionally, there has been much discussions regarding standardization of such a method in recent decades. For example, Elizabeth D. Rather wrote on 2011-12-09 in comp.lang.forth:

AFAIK most if not all Forths have some method for knowing the latest definition, it's kinda necessary. The problem is, that they all do it differently (at different times, in different forms, etc.), which is why it hasn't been possible to standardize it.

Although it's a system necessity, I haven't found this of much value in application programming.

Elizabeth D. Rather

It's true: depending on the system, an internal method can return the recent word regardless of the compilation word list, or depending on the compilation word list, a completed definition, or not yet completed definition, also unnamed definition, or only named definition, etc.

Nevertheless, the author convinced that we can standardize a consistent method that solves the problem.

Solution

Let's introduce the following words:

• LATEST-NAME-IN ( wid -- nt|0 )
• LATEST-NAME ( -- nt )

The former word returns the name token for the definition name that was placed most recently into given word list, or zero if this word list is empty.

The latter word returns the name token for the definition name that was placed most recently into the compilation word list, or throws an exception if such a definition is absent.

It seems, the best place for these words is the section 15.6.2 Programming-Tools extension words), where TRAVERSE-WORDLIST is also placed.

Rationale

Connection with word lists

By considering definitions in the frame of a word list only, we solve several problems, namely:

1. A word list contains only completed definitions (see the accepted proposal #153 Traverse-wordlist does not find unnamed/unfinished definitions). This eliminates the question of whether the word of returned nt is finished — yes, it is always finished (completed).

2. Unnamed definitions are not considered since they are not placed into the compilation word list (regardless of whether the system creates a name token for them, or places them into an internal system-specific word list).

3. An extension or library can create definitions in its internal word list for internal purposes. And it will not affect the compilation word list or other user-defined word lists. Thus, the user of such library always gets the expected result from latest-name (regardless of what words are created by this library for internal purposes on the fly).

Return values

As a matter of practice, almost all the use cases for the word LATEST-NAME imply that the requested definition exists, and if it doesn't exist, only an error can be reported. So the option to return 0 by this word only burdens users with having to analyze this zero, or redefine this word as:

: latest-name ( -- nt ) latest-name dup 0= -80 and throw ;

If the user needs to handle the case where the compilation word list is empty, they can use the word latest-name-in as:

get-current latest-name-in dup if ( nt ) ... else ( 0 ) drop ... then

Implementation options

In some plausible Forth systems, the word list structure doesn't contain any information about the definition that was placed into this word list most recently. Such systems might not provide the proposed words, or they are changed to keep the mentioned information in the word list structure. It seems, in most systems the word list structure contains this information.

If a system does not implement The optional Search-Order word set, it might not provide the word LATEST-NAME-IN.

Naming

The names for these words LATEST-NAME-IN and LATEST-NAME are similar to FIND-NAME-IN and FIND-NAME. Ditto for stack effects.

Typical use

: STRUCT: ( "name" -- wid.current.old u.offset )
  GET-CURRENT  VOCABULARY
  ALSO  LATEST-NAME NAME> EXECUTE  DEFINITIONS
  0
;

Proposal

Add the following line into the Table 9.1: THROW code assignments:

-80 compilation word list is empty

Add the following sections into 15.6.2 Programming-Tools extension words:

15.6.2.2541 LATEST-NAME-IN

( wid -- nt|0 )
Remove the word list identifier wid from the stack. If this word list is empty, then return 0, otherwise return the name token nt for the definition that was placed most recently into this word list.

15.6.2.2542 LATEST-NAME

( -- nt )
Return the name token nt for the definition that was placed most recently into the compilation word list, if such definition exists. Otherwise throw exception code -80.

Reference implementation

In this implementation we assume that wid is an address that contains nt of the most recently placed definition name into the word list wid.

: LATEST-NAME-IN ( wid -- nt|0 ) @ ;

: LATEST-NAME ( -- nt )
  GET-CURRENT LATEST-NAME-IN  DUP IF EXIT THEN  -80 THROW
;

testcase - TRAVERSE-WORDLIST must not expose the current definition

TRAVERSE-WORDLIST is not allowed to visit the current definition, unfinished or hidden definitions (see also the accepted proposal #153 Traverse-wordlist does not find unnamed/unfinished definitions).

Some systems place the current definition into the compilation word list and change its header in such a way that this definition cannot be found via FIND-NAME-IN or SEARCH-WORDLIST. By a mistake, nt for this definition can be exposed by TRAVERSE-WORDLIST.

This test ensured that the current definition is not exposed, and that the correct number of names is placed into the compilation word list.

: WORDLIST-LENGTH ( wid -- u )
  >R 0 [: DROP 1+ TRUE ;] R> TRAVERSE-WORDLIST
;
: L ( -- u ) GET-CURRENT WORDLIST-LENGTH ;

T{ L :NONAME [ L ] LITERAL ;  EXECUTE  L =  = -> TRUE }T

T{ L : TW1 [ L ] LITERAL ;  TW1 OVER = SWAP 1+ L = AND  -> TRUE }T

proposal - NAME>INTERPRET wording

Author

Ruv

Change Log

• 2020-02-20 Initial comment for NAME>INTERPRET
• 2023-09-14 Make this proposal more formal
• 2023-09-19 Add rationale, better wording, fix some typos
• 2023-09-21 Add rationale re system-defined semantics, and declare an ambiguous condition
• 2023-09-22 Require xt if interpretation semantics for the word are defined by the standard
• 2023-09-28 Add a clause re STATE-dependent execution semantics in Problem, add a section re execution of xt meaning in Rationale, better wording and formatting in some places

Problem

1. It's currently unclear from the specification for name>interpret how to perform the interpretation semantics for the word identified by nt — namely, how to perform the behavior that a Forth system performs when the Forth text interpreter, while in interpretation state, encounters the name of a word identified by this nt (see also Rationale below).

2. The specification for name>interpret says that returned "xt represents the interpretation semantics of the word nt". But actually, in some cases a Forth system cannot provide an xt that performs the interpretation semantics for the corresponding word regardless of STATE.

   Particularly, when nt is for a word whose execution semantics depends on STATE, or for a word like s" or to, if it is implemented as an immediate word. Technically, it is possible to return a correct xt according to the current specification (e.g. via generation of the corresponding definition on the fly), but it can be too burden.

3. A minor problem is that it's not clear what the term represent means. According to the language of the standard, xt identifies some semantics.

Rationale

How to perform interpretation semantics

It should be clear from the specification how, having an nt, to perform the very behavior that a Forth system performs when the Forth text interpreter, while in interpretation state, encounters the name of a word identified by this nt.

Namely, in the general case, to perform this behavior, must the xt returned from name>interpret be executed only in interpretation state, or it may be also executed in compilation state.

Execution of xt

In this proposal, the author assumes that compile, is equivalent to lit, ['] execute compile,, and therefore "xt is executed" means "the execution semantics identified by xt are performed" — no matter how — via execute directly, or as a part of execution semantics of another definition, in any nesting, when this part is appended by applying compile, to this xt.

NB: one consequence from this equivalence is that the words that operate on the return stack cannot be defined as ordinary words (see a corresponding discussion).

System-defined semantics

If the standard does not define interpretation semantics for a word, a Forth system may provide system-defined interpretation semantics for the word (see A.3.4.3.2).

The same is true for execution semantics — if the standard does not define them for a word, a Forth system may provide system-defined execution semantics for the word. But, due to 6.1.0070, performing these execution semantics in interpretation state must always be equivalent to performing the interpretation semantics for the word (regardless whether they are standard-defined or system-defined).

Connection with Tick

It we want ticking any word for which interpretation semantics are defined by the standard, name>interpret cannot return 0 for these words.

Forth systems in which name>interpret returns 0 for such a word are unknown to the author.

Solution

The specification for name>interpret can be adjusted to solve the mentioned problem. There are two options:

1. Allow to return 0 if the system cannot return xt that identifies the interpretation semantics for the word identified by nt (see also the clarification re execution tokens).

   • Probably, in this case we have to introduce a word like name> (experimental in Forth-83), which returns xt that identifies the execution semantics of the word identified by nt. Since otherwise a user-defined Forth text interpreter is impossible without correct find.

2. Allow to return state-dependent xt, which performs the interpretation semantics for the word in interpretation state only.

In this proposal I stick to the second option.

If anyone prefers the first option or has other objections, please feel free to share your ideas in a comment.

Proposal

Replace the following paragraph in the section 15.6.2.1909.20 NAME>INTERPRET:

xt represents the interpretation semantics of the word nt. If nt has no interpretation semantics, NAME>INTERPRET returns 0.

by the following paragraphs:

xt identifies the execution semantics of the word identified by nt. When this xt is executed in interpretation state, the interpretation semantics for the word are performed.

If and only if interpretation semantics for the word are not defined by this standard and the Forth system does not provide the execution token for the word, NAME>INTERPRET returns 0.

An ambiguous condition exists in any of the following conditions:

• interpretation semantics for the word are not defined by this standard and xt is executed;
• execution semantics of the word are not defined by this standard and xt is executed in compilation state;

proposal - NAME>INTERPRET wording

In the ways of execution an xt the word catch should be mentioned.

The word find (according to the proposed clarification) allows the program to check whether a word is ordinary (i.e., for which default interpretation semantics and default compilation semantics are defined): if and only if find, in interpretation state, returns -1 at the top, then the word is ordinary.

At the moment, the set of words find-name, name>interpret and name>compile does not provide a ways to check whether a word is ordinary.

Perhaps name>compile could be tightened up so that analysis its result and the results of name>interpret allows the program to conclude whether the word is ordinary.

proposal - minimalistic core API for recognizers

FORTH-RECOGNIZER ( -- xt ) RECOGNIZER EXT
Obtain the recognizer xt that is assigned to FORTH-RECOGNIZE.

Rationale:
FORTH-RECOGNIZE is likely a deferred word, but systems that implement it otherwise, can use this word to change the behavior instead of using ACTION-OF FORTH-RECOGNIZE. The old API has this function under the name FORTH-RECOGNIZER (as a value) and this name is reused. Systems that want to continue to support the old API can support TO FORTH-RECOGNIZER, too.

There is no way for a program to check whether it can apply TO to FORTH-RECOGNIZER, or FORTH-RECOGNIZE, or RECOGNIZE-FORTH-LEXEME, etc. Thus, TO cannot be optional. And it cannot be mandatory too. Thus, TO cannot be a part of the API at all — neither RECOGNIZER, nor RECOGNIZER EXT.

Then the getter and setter should be a mandatory part of the API.

proposal - minimalistic core API for recognizers

In continuation to the message:

Returning something half-done isn't a good idea and makes maintaining this code difficult, as all other possibles results (like ints, floats or such) are fully converted into something useful at this stage.

This would be a valid argument if it were possible to return something useful from a recognizer in all use cases

Another example of an unuseful token is the result of the mentioned recognizer REC-TO, which recognizes a syntax like ->foo.

It's too restrictive to require this token be ( xt n tt ), since in some systems it can be just ( xt.set-value tt ), in other — ( addr.data-field xt.store tt ). This means that this token is not something useful to a program at all (apart of translation).

comment - A better approach for SYNONYM wording

With the catch-all phrase for ambiguous conditions of name.old and name.new being identical, you can drop all those child from executing one or another defining word: TO/IS/ACTION-OF name.new, etc. are identical to the same construct with name.old. That way, these combinations have to work even for non-standard extensions, like Gforth's, where you can define your own TO/IS semantics with SET-TO, regardless of the original definer.

proposal - F>R and FR> to support dynamically-scoped floating point variables

These words cannot specify how many cells on the return stack are taken (it's similar to N>R in this regard). Also, they should not be defined via execution semantics, but via compilation semantics and Run-time semantics.

• F>R Run-time: ( F: r -- ) ( R: -- i*x )

  Remove the top item from the floating-point stack and store it as zero or more items on the return stack for later retrieval by FR>.
  Data placed on the return stack before performing these semantics are inaccessible for a program till performing the corresponding FR> Run-time semantics.

• FR> Run-time: ( F: -- r ) ( R: i*x -- )

  Remove from the return stack the items previously stored by F>R, and place the corresponding item on the floating-point stack.

example - Digests and Meta discussion

Persistent Login

You can now try out the new "Remember Me" checkbox at the login. It should automatically log you in if you stay active for less than 42 days. After that the persistent login expires. Please let me know if any issues occur.

New Digest.

If all works, you should get notified about this via a newly designed HTML and new TXT digest. Fingers crossed ;)

proposal - Include a revised 79-STANDARD Specification for "><" To "Core Ext"

< fooo

proposal - NAME>INTERPRET wording

name>interpret is allowed to return 0 if interpretation semantics for the word are not defined by the standard.

But in fact, the result of 0 does not give programs any additional information, since for the same word the result can be zero in one system and non-zero in another system.

So, programs won't loose anything, and will only become simpler, if name>interpret never returns zero (i.e., if it's required to return some xt in all cases) — since programs will not need to analyze the result (e.g. before compile or execute it). Well, in some rare cases an error report shown by programs will be carried over from compile time to runtime in some systems due to this change, but anyway this report is generated at compile time only in some systems — i.e., this is a system dependent behavior that programs cannot rely on.

Suppose we changed the specification for name>interpret so that this word will require to return a system-dependent xt if interpretation semantics for the word are not defined by the standard. This change does not affect programs at all. But it affects systems — they need to be updated to comply with the new specification and new programs (which relies on the new specification).

The question is: is it acceptable to change name>interpret specification in this way, or is it better to introduce a new word for this?

The new word can be defined as follows:

: name> ( nt -- xt )
  name>interpret dup if exit then drop [: -14 throw ;]
;

Or name>interpret can be redefined in this way in a polyfill.

referenceImplementation - Imlementation seems wrong for the "-2" case

If the top of the stack is -2 , and there is no handler, then THROW is supposed to display a message.
I can't see how the reference implementation does that.

A reference implementation is just an example, it is not required to cover all the cases.

The reference implementation for THROW is preceded by the note: "This is the counter part to E.9.6.1.0875 CATCH", if we follow the link, we in turn can read in the note: "This sample implementation does not explicitly handle the case in which CATCH has never been called (i.e., the ABORT behavior)".

So, the implementation explicitly says that the case of no handler is not covered.

referenceImplementation - Possible Reference Implementation

Both variants are correct and viable. For reference implementations, the shorter and simpler the better.

And anyway, stack diagrams are required.

example -

variable counts N allot
counts N erase

This is incorrect, see 3.3.3.3 Variables. And for example, take a look at a possible implement for variable.

Correct variants are as follows:

create counts N allot
counts N erase

or

align here N allot constant counts
counts N erase

example -

there is no similar extension to \, which continues to discard the entire rest of the buffer.

According to the word included specification, it fills the input buffer with one line only. It means, the input buffer in this case contains not more than a single line, and the word 6.2.2535 \ "backslash" (CORE EXT) works correctly (there is no need for update this word in the FILE word set).

when the editor evaluates its buffer then \ comments are broken, but if it separates and evaluates each line one-at-a-time then ( comments are broken.

A possible solution of this problem is to redefine "\" and evaluate the whole buffer in the editor. The new behavior for the word "\" is to skip the characters up to the next line terminator when the source-id is -1 and blk is 0 (otherwise no changes in behavior). This behavior is slightly not compliant, but in practice it does not cause problems in programs.

Cross-reference also ruv's post about exactly this problem.

Yes, my post's title is Portable line-oriented parsing, and the mentioned solution is provided in the "Comments in evaluated strings" section:

: is-input-string ( -- flag )
  \ Return a flag: is the input source a string (being evaluated).
  [defined] blk [if] blk @ 0<> if false exit then [then]
  source-id -1 =
;
: source-following ( -- sd )
  \ Return the parse area (a string).
  \ NB: the returned string may contain a line-terminator sequence in any position.
  source >in @ /string
;
: skip-source-line ( -- )
  \ Discard a part of the parse area that belongs to the current line.
  is-input-string 0= if ['] \ execute exit then
  source-following  over >r  s\" \n"  dup >r  search  if drop r@ then  +  rdrop
  r> -  >in +!
;
: \ ( -- )
  \ This Backslash works as expected in evaluated strings too
  skip-source-line
; immediate

testcase - TRAVERSE-WORDLIST must not expose the current definition

TRAVERSE-WORDLIST is not allowed to visit the current definition, unfinished or hidden definitions (see also the accepted proposal #153 Traverse-wordlist does not find unnamed/unfinished definitions).

Some systems place the current definition into the compilation word list and change its header in such a way that this definition cannot be found via FIND-NAME-IN or SEARCH-WORDLIST. By a mistake, nt for this definition can be exposed by TRAVERSE-WORDLIST.

This test ensured that the current definition is not exposed, and that the correct number of names is placed into the compilation word list.

: WORDLIST-LENGTH ( wid -- u )
  >R 0 [: DROP 1+ TRUE ;] R> TRAVERSE-WORDLIST
;
: L ( -- u ) GET-CURRENT WORDLIST-LENGTH ;

T{ L :NONAME [ L ] LITERAL ;  EXECUTE  OVER = SWAP L = AND  -> TRUE }T

T{ L : TW1 [ L ] LITERAL ;  TW1 OVER = SWAP 1+ L = AND  -> TRUE }T

proposal - New words: latest-name and latest-name-in

Author

Ruv

Change Log

• 2023-10-22 Initial revision
• 2023-10-23 Add testing, examples, a question to discuss, change the throw code description

Problem

In some applications, mainly in libraries and extensions, the capability to obtain the most recently added definition is very useful and demanded.

For example, if we are creating a library for decoration, tracing, support for OOP, simple DSLs (e.g., to describe Final State Machines), etc — it is always useful to have an accessor to the recent definition, instead of redefining a lot of words to define such an access method yourself, or juggling with the input buffer and search.

However, many Forth systems have such internal methods to access the recently added word. Among them: latest ( -- nt|0 ), last @ ( -- nt|0 ), latestxt ( -- xt|0 ), etc.

And additionally, there has been much discussions regarding standardization of such a method in recent decades. For example, Elizabeth D. Rather wrote on 2011-12-09 in comp.lang.forth:

AFAIK most if not all Forths have some method for knowing the latest definition, it's kinda necessary. The problem is, that they all do it differently (at different times, in different forms, etc.), which is why it hasn't been possible to standardize it.

Although it's a system necessity, I haven't found this of much value in application programming.

Elizabeth D. Rather

It's true: depending on the system, an internal method can return the recent word regardless of the compilation word list, or depending on the compilation word list, a completed definition, or not yet completed definition, also unnamed definition, or only named definition, etc.

Nevertheless, the author convinced that we can standardize a consistent method that solves the problem.

Solution

Let's introduce the following words:

• LATEST-NAME-IN ( wid -- nt|0 )
• LATEST-NAME ( -- nt )

The former word returns the name token for the definition name that was placed most recently into given word list, or zero if this word list is empty.

The latter word returns the name token for the definition name that was placed most recently into the compilation word list, or throws an exception if such a definition is absent.

It seems, the best place for these words is the section 15.6.2 Programming-Tools extension words), where TRAVERSE-WORDLIST is also placed.

Rationale

Connection with word lists

By considering definitions in the frame of a word list only, we solve several problems, namely:

1. A word list contains only completed definitions (see the accepted proposal #153 Traverse-wordlist does not find unnamed/unfinished definitions). This eliminates the question of whether the word of returned nt is finished — yes, it is always finished (completed).

2. Unnamed definitions are not considered since they are not placed into the compilation word list (regardless of whether the system creates a name token for them, or places them into an internal system-specific word list).

3. An extension or library can create definitions in its internal word list for internal purposes. And it will not affect the compilation word list or other user-defined word lists. Thus, the user of such library always gets the expected result from latest-name (regardless of what words are created by this library for internal purposes on the fly).

Return values

As a matter of practice, almost all the use cases for the word LATEST-NAME imply that the requested definition exists, and if it doesn't exist, only an error can be reported. So the option to return 0 by this word only burdens users with having to analyze this zero, or redefine this word as:

: latest-name ( -- nt ) latest-name dup 0= -80 and throw ;

If the user needs to handle the case where the compilation word list is empty, they can use the word latest-name-in as:

get-current latest-name-in dup if ( nt ) ... else ( 0 ) drop ... then

Implementation options

In some plausible Forth systems, the word list structure doesn't contain any information about the definition that was placed into this word list most recently. Such systems might not provide the proposed words, or they are changed to keep the mentioned information in the word list structure. It seems, in most systems the word list structure contains this information.

If a system does not implement The optional Search-Order word set, it might not provide the word LATEST-NAME-IN.

Naming

The names for these words LATEST-NAME-IN and LATEST-NAME are similar to FIND-NAME-IN and FIND-NAME. Ditto for stack effects.

Things to discuss

Is it worth introducing the word LATEST-NAME-XT ( -- xt )?

If name>interpret never returns 0 (see my comment), this word can be implemented as:

: latest-name-xt ( -- xt ) latest-name name>interpret ;

The desired (and much discussed) pattern is:

defer bar

: foo ... ; latest-name-xt is bar

Sometimes the name "it" has been suggested for this word, but this name is too short and has more chance for conflicts. Guido Draheim wrote in comp.lang.forth on 2003-03-16:

I think that everyone has been thinking of using IT for something really clever, it's a nice short word - and I'd say that we should leave it for application usage.

I want to support that argument also with real life experience in the telco world where there are a whole lot of abbreviations for various services, signals, connectors around. All too often now I see people making a SYNONYM at the file-start to get a second name for an ANS forth word that is needed in the implemenation but coincides with a common term of the application.

This seems convincing to me.

Typical use

: STRUCT: ( "name" -- wid.current.old u.offset )
  GET-CURRENT  VOCABULARY
  ALSO  LATEST-NAME NAME> EXECUTE  DEFINITIONS
  0
;

  \ In the application's vocabulary
  : IT ( -- xt ) LATEST-NAME NAME>INTERPRET ;

  DEFER FOO

  : BAR ... ; IT IS FOO

Proposal

Add the following line into the Table 9.1: THROW code assignments:

-80 latest name is missing

Add the following sections into 15.6.2 Programming-Tools extension words:

15.6.2.2541 LATEST-NAME-IN

( wid -- nt|0 )
Remove the word list identifier wid from the stack. If this word list is empty, then return 0, otherwise return the name token nt for the definition that was placed most recently into this word list.

15.6.2.2542 LATEST-NAME

( -- nt )
Return the name token nt for the definition that was placed most recently into the compilation word list, if such definition exists. Otherwise throw exception code -80.

Reference implementation

In this implementation we assume that wid is an address that contains nt of the most recently placed definition name into the word list wid.

: LATEST-NAME-IN ( wid -- nt|0 ) @ ;

: LATEST-NAME ( -- nt )
  GET-CURRENT LATEST-NAME-IN  DUP IF EXIT THEN  -80 THROW
;

Testing

: IT ( -- xt ) LATEST-NAME NAME>INTERPRET ;

T{ : LN1 ; IT  ' LN1 =  -> TRUE }T
T{ :NONAME [ IT ] LITERAL ; EXECUTE  ' LN1 =  -> TRUE }T
T{ : LN2 [ IT ] LITERAL ; LN2  ' LN1 =  -> TRUE }T