Digest #40 2018-08-18

Contributions

[65] 2018-08-17 16:27:53 UlrichHoffmann wrote:

proposal - S( "Request for Discussion" (revised 2018-08-16)

S( "Request for Discussion"

Change History

2018-08-16 Improve with additional explanation, rewording
2018-07-09 First version

Problem

There have been extensive discussions about correct implementations (cf. [Ertl98], [Pelc17], [clf18]) of Forth words that have both

  • an interpretation semantics and
  • a compilation semantics that is different from adding the word's execution semantics to the current definition (sometimes called non-default compilation semantics, NDCS).

The Forth-94 word S" is an example for this as it has a defined compilation semantics (and an explicitly undefined interpretation semantics) in the CORE word set and a defined interpretation semantics in the FILE word set. (cf. [Forth94])

Words like this can have the so called copy&paste property: Program phrases that have been tested in interpretation mode can be copied unchanged into definitions and continue to work there in a seemingly identical way. This is attractive to quite some developers.

The desire to have words with diverging compilation semantics and interpretation semantics lead many system implementors to state-smart immediate words (which inspect the variable STATE to distinguish between compilation and interpretation) but these have the drawback to fail unexpectedly in corner cases [Ertl98].

To identify whether or not a word is state-smart typically requires studying its implementation or documentation. Problems arise when these state-smart words are used as buildung blocks for more sophisticated words, e.g. by means of POSTPONE-ing them. The distinction between interpret and compile state is then also postponed and happens at inapropriate times.

There are ways to structure the outer interpreter so that words with non-default compilation semantics can be implemented without failures in corner cases. Systems such as gforth development version and VFX version 5 deal with this issue (cf. [Pelc17], [clf18]).

Looking at the Forth-94 standard only few words actually have non default compilation semantics, namely the already mentioned S" and TO. Forth-2012 adds others.

In general Forth-94 and Forth-2012 define execution semantics for normal words. When the word is interpreted this execution semantics is performed, when it is compiled its execution semantics is appended to the current definition (the standard compilation semantics). For most compiling words the standards explicitly leave the interpretation semantics undefined and define only a compilation semantics. This compilation semantics might include appending an additionally defined runtime semantics to the current definition. Colloquially these words are called compile-only. The standards also explicitly define immediate words: Words that have identical interpretation and compilation semantics: both perform the word's execution semantics. Finally there are special words with non-default compilation semantics (NDCS) where interpretation semantics and compilation semantics diverge.

The following table summarizes this situation:


word kind interpretation semantics compilation semantics example comment
normal perform execution semantics compile execution semantics DUP normal :-definitions
immediate perform execution semantics perform execution semantics .( IMMEDIATE definitions
compile-only undefined perform execution semantics IF interpretation semantics undefined
NDCS perform execution semantics for interpretation perform execution semantics for compilation S" divergent interpretation, compilation

S" and TO seem to be very special: For other words such as ' (tick) or CHAR, or .( (dot-paren) the standards define similar compilation words namely ['](bracket-tick), [CHAR] (bracket-char), or ." (dot-quote) for use in compiling mode inside definitions.

Some argue that complicating the outer interpreter or dictionary design is not beneficial especially in memory restricted small system and either live with the failures of state-smart words in corner cases or avoid implementing an interpretive S" to stay standard compliant.

Solution

For small systems it might be reasonable and simpler to not implement an S" with non default compilation semantics as defined in the FILE word set but to define two diffently named words that captures the compilation semantics and the interpretation semantics of FILE S" respectively.

Possible naming choices could be

  • S" (s-quote) for interpretation and
    [S"] (bracket-s-quote-bracket) for compilation
    used in the form S" hello" and : xxx ... [S"] hello" ... ; or

  • S" (s-quote) for interpretation and
    [S" (bracket-s-quote) for compilation
    used in the form S" hello" and : xxx ... [S" hello"] ... ;

neither of which is really appealing.

Instead --- similar to ." for string output in definitions and. .( for string output outside definitions --- we propose to keep CORE S" with its behaviour and to standardize S( with interpretation semantics of FILE S" but using ) (right parenthesis) as delimiter.


Proposal

Add the word S( to the CORE Extension Word Set (CORE EXT):


S( "s-paren" CORE EXT

  • Interpretation: Perform the execution semantics given below.

  • Compilation: Perform the execution semantics given below.

  • Execution: ( "ccc\<paren\>" -- c-addr u )
    Parse ccc delimited by ) (right parenthesis). Store the resulting string in a transient buffer c-addr u.

    S( is an immediate word.

See: Parsing, S"


Typical Use

Typical use of S( would be interactively when temporary strings are required for example for use with INCLUDED:

S( s-paren.fs) INCLUDED

Remarks

As S( is proposed to be standardized in the CORE extension word set no standard system is required to provide S(. However if a system chooses to implement the S" compilation and interpretation semantics with two separately named words, it could choose the standard name S" and the (not yet standardized) name S( for this.

Defining the interpretation semantics explicitly in the glossary entry above is not strictly necessary as both Forth-94 and Forth-2012 state:

Unless otherwise specified in an “Interpretation:” section of the glossary entry, the interpretation semantics of a Forth definition are its execution semantics.

Reference implementation

With only a single string buffer, a minimal S(-implementation could look like this:

CREATE buf DECIMAL 80 CHARS ALLOT

: S( ( "ccc\<paren\>"-- c-addr len )
    [CHAR] ) PARSE  80 MIN >R  buf R@ MOVE  buf R> ;

A more elabortated implementation using mutliple string buffers in a circular fashion is:

DECIMAL 80 CHARS CONSTANT |buf|
               4 CONSTANT #bufs

CREATE bufs  |buf| #bufs * ALLOT
VARIABLE buf#  0 buf# !

: buf ( -- c-addr )
   bufs  buf# @ |buf| * + ;

: bump ( -- )
   buf# @  1+  #bufs MOD  buf# ! ;

: str ( char "ccc\<char\>" -- c-addr u )
   bump   buf SWAP PARSE  |buf| MIN >R  OVER R@ MOVE  R> ;

: S( ( "ccc\<paren\>"-- c-addr len )
   [CHAR] ) str ;

Testing

The following tests assure that S( pushes the desired c-addr u

CREATE s   3 c, CHAR a c, CHAR b c, CHAR c c, 

t{ 99 S( abc) SWAP DROP -> 99 3 }t
t{ 99 S( abc) s COUNT COMPARE -> 99 0 }t

Experience

S( is not yet defined in any of the contemporary systems such as

  • gForth
  • VFX
  • PFE
  • DXForth
  • FLT
  • SwiftForth
  • Win32Forth
  • noForth
  • amForth
  • camelForth
  • ciForth
  • mecrisp

so it has no common use. However the name S( seems to be available in all these systems.

Discussion

The proposal avoids issues with the NDCS word S" by providing S( as an alternative notation for an interpretive S".

It is intended for ressource restricted standard systems that want to support interpretive strings but which are not able to provide the FILE word set S".

S( is very simple to implement so this proposal is rather about standarizing the name S( with the intended functionality than a sophisticated feature.

One can argue to remove S" from the FILE word set, however this is not proposed here. Forth systems that provide the FILE word set are hopefully capable of providing a complete and correct S" implementation.

References

[Forth-94]: "American National Standard for Information Systems — Programming Languages — Forth", ANSI X3.215-1994

[clf18]: discussion about special words in comp.lang.forth, https://groups.google.com/forum/#!topic/comp.lang.forth/Gb9Hvj3Wm_Y%5B1-25%5D

[Ertl98]: "State-smartness - Why it is Evil and How to Exorcise it", Anton Ertl, euroForth 1998

[Pelc17]: "Special Words in Forth", Stephen Pelc, euroForth 2017

Author

Ulrich Hoffmann uho@xlerb.de

Replies

[r169] 2018-07-11 08:51:48 MarkWills replies:

proposal - Let us adopt the Gerry Jackson test suite as part of Forth 200x

Sounds like a sensible idea to me!


[r170] 2018-07-13 16:25:47 alextangent replies:

proposal - Let us adopt the Gerry Jackson test suite as part of Forth 200x

Great idea!


[r171] 2018-07-14 06:32:34 AntonErtl replies:

proposal - Let us adopt the Gerry Jackson test suite as part of Forth 200x

The proposal is short on details. What do you mean by "adopt"?

Should the standard document include the tests in the test suite, and be updated whenever the test suite is? That would cause a lot of work for the editor; not sure if the benefit is worth it.

Or should the standard document replace its tests with a reference to the test suite? That would make it harder to see both at once, which can be useful for understanding the normative part (by someone who is not familiar with the described word), and also useful for checking the tests (for someone who is familiar with the described word).

I guess the ideal approach would be that the tests contain information that states the section of the standard that the tests test, and that forth-standard.org automatically puts the tests into the appropriate sections. Then we don't need the tests in the latex version and the PDF (where I don't find them useful). But this would need work by both Gerry Jackson and Gerald Wodni, so if we want to go there, we should ask them if they are willing to do that.


[r172] 2018-07-16 22:45:40 StephenPelc replies:

proposal - Let us adopt the Gerry Jackson test suite as part of Forth 200x

By adopt, I mean that we say that the test suite maintained by Gerry, the version currently at some location is the official test suite for Forth200x. This does not mean that it is the only test suite, but that we the committee recognise it as a quality test suite. The details of what we do and do not want or say can be thrashed out at the next meeting. Since this is a non-normative section, we can even vote if we can find a suitable form of words.

Ideally, the current document's tests and Gerry's should be merged - a source file is needed for people who wish to test their systems. I agree that the tests can be valuable for understanding and I have encourage Gerry (and potential assistants) to include more documentation.

Your point about taking workload off the editor is valuable.

I will put the topic on the meeting agenda.


[r173] 2018-08-15 10:12:05 AntonErtl replies:

proposal - EXCEPTION LOCALs

There is a lot of STATE that can be changed in a word called through CATCH, and that we may want to restore when an exception is THROWn, not just BASE and STATE. Conversely, there are many words that are called through CATCH that do not change BASE and STATE, so always saving and restoring BASE and STATE would be wasteful. My solution for this is cleanup wrappers, such as

: base-execute ( xt u -- )
  base @ >r base ! catch r> base ! throw ;

And while the standard describes the input source specification as a kind of exception locals, at least Gforth uses a cleanup wrapper for restoring it. You can read more about cleanup in my EuroForth'08 paper.

Anyway, I don't think we need exception locals.

Another issue is that some people fail to use such wrappers (after all Thinking Forth does not advocate it), and then a THROW returns to a different CATCH or the Forth command line without restoring the state. For STATE, the standard specifies that it is set to interpreting when there is no user-supplied exception handler, for BASE the user has to fend for himself (as advocated in Thinking Forth). It is a good idea if the system warns of non-decimal BASE, e.g., by mentioning it after "ok".


[r174] 2018-08-15 10:38:39 AntonErtl replies:

proposal - find-name

ChangeLog

2018-08-15 Reference implementation now provides FIND-NAME-IN; more information on other systems; added NAME-PRINTER example

Problem

FIND has several problems:

  1. It takes a counted string.

  2. Its interface is designed for single-xt+immediate-flag systems (which cannot implement FILE S" correctly in all cases), and there is allowance for a STATE-dependent result to support other systems.

  3. As currently specified (there is a proposal for fixing that), it fails to meet the goal of guaranteeing that the user-defined text-interpreter (outlined in the rationale of COMPILE,).

As a consequence of problem 2, the following implementation of ' is wrong:

: ' bl word find 0= -13 and throw ;

This ' produces the wrong result in the following case on Gforth:

: x ' ; immediate
] x s" [

And that's because Gforth implements S" correctly, and FIND such that the user-defined text interpreter works. The following definition would fix this problem on Gforth:

: ' bl word state @ >r postpone [ find r> if ] then 0= -13 and throw ;

but actually the specification of FIND currently is sufficiently unclear that we cannot be sure that that's guaranteed to work, either.

SEARCH-WORDLIST has a related, but different set of problems:

  1. It produces results with a varying number of stack items. In some situations that makes it more cumbersome to use.

  2. Its interface is designed for single-xt+immediate-flag systems, but this time without allowing a STATE-dependent result. As a consequence, Gforth produces the xt representing the interpretation semantics of the word, independent of STATE, with no way to get the compilation semantics of the word.

  3. In the general case, given that the xt part of the result does not represent a part of the compilation semantics, the 1/-1 part of the result is pointless.

Solution

Introduce FIND-NAME (for the search order) and FIND-NAME-IN (for specific wordlists). Both take strings in c-addr u form, and both return the name token of the found word (or 0 if not found). We can then go from the name token to the interpretation semantics with NAME>INTERPRET, and to the compilation semantics with NAME>COMPILE.

Typical use

: ' parse-name find-name dup 0= -13 and throw name>interpret ;

: postpone
  parse-name find-name dup 0= -13 and throw
  name>compile swap postpone literal compile, ; immediate

\ user-defined text interpreter
: interpret-word
  parse-name 2dup find-name if
     nip nip state @ if name>compile else name>interpret then execute
  else
     ... \ process numbers
  then ;

\ alternative:
defer name>statetoken ( nt -- ... xt )
: [ ['] name>interpret is name>statetoken false state ! ;
[ \ initialize STATE and NAME>STATETOKEN
: ] ['] name>compile is name>statetoken true state ! ;

: interpret-word
  parse-name 2dup find-name if
     nip nip state name>statetoken execute
  else
     ... \ process numbers
  then ;

\ a defining word for words that print their name
: name-printer ( "name" -- )
  >in @ create >in ! parse-name get-current find-name-in ,
does> ( -- )
  @ name>string type ;

Remarks

FIND-NAME and FIND-NAME-IN are natural factors of all words that look up words in the dictionary, such as FIND, ', POSTPONE, the text interpreter, and SEARCH-WORDLIST. So implementing them does not cost additional code in the Forth system, only some refactoring effort.

This approach is not compatible with the cmForth approach and Mark Humphries' approach, because they both use one word header each for interpretation and compilation semantics. This problem already exists for the other words that deal with name tokens, but the present proposal would make name tokens more important, and systems that do not support them less viable. However, both approaches have been known for at least two decades, and have seen little to no uptake in standard systems. And we have good approaches for implementing systems with name tokens, so excluding these approaches is not a significant loss.

Proposal

Add the following words:

FIND-NAME ( c-addr u -- nt | 0 )

Find the definition identified by the string c-addr u in the current search order. Return its name token nt, if found, otherwise 0.

FIND-NAME-IN ( c-addr u wid -- nt | 0 )

Find the definition identified by the string c-addr u in the wordlist wid. Return its name token nt, if found, otherwise 0.

Reference implementation

Reference Implementation

Testing

Testcases

Experience

Gforth has implemented FIND-NAME and (under the name (search-wordlist)) FIND-NAME-IN since 1996. No problems were reported or found internally.

amForth has FIND-NAME "since ever" (first release 2006), and FIND-NAME-IN under the name SEARCH-NAME.

Other systems have been reported to implement FIND-NAME under this or other names (e.g., FOUND in ciforth).


[r175] 2018-08-16 11:10:39 ruv replies:

comment - NAME>COMPILE result

The returned xt has the stack effect ( i*x x -- j*x )

Where should these i*x come from?

Could somebody provide an example when i or j is greater than 0?


[r176] 2018-08-16 11:23:47 ruv replies:

comment - proposal: recurse' to complement recurse

In the light of recognizers it is better to leave the special characters for various literals and the like.


[r177] 2018-08-16 11:32:01 ruv replies:

comment - proposal: recurse' to complement recurse

Also, itself is not directly connected with recursion — it can be returned as result or compiled into another definition.

: tt  ... if  ... lit, itself compile, ... then ... ;

It is one real usage case.


[r178] 2018-08-17 07:52:57 JennyBrien replies:

comment - NAME>COMPILE result

Control words such as IF where the control stack is the data stack.