Digest #40 2018-08-18
Contributions
proposal - S( "Request for Discussion" (revised 2018-08-16)
S( "Request for Discussion"
Change History
2018-08-16 Improve with additional explanation, rewording
2018-07-09 First version
Problem
There have been extensive discussions about correct implementations (cf. [Ertl98], [Pelc17], [clf18]) of Forth words that have both
- an interpretation semantics and
- a compilation semantics that is different from adding the word's execution semantics to the current definition (sometimes called non-default compilation semantics, NDCS).
The Forth-94 word S" is an example for this as it has a defined compilation semantics
(and an explicitly undefined interpretation semantics) in the CORE word set
and a defined interpretation semantics in the FILE word set. (cf. [Forth94])
Words like this can have the so called copy&paste property: Program phrases that have been tested in interpretation mode can be copied unchanged into definitions and continue to work there in a seemingly identical way. This is attractive to quite some developers.
The desire to have words with diverging compilation semantics and interpretation semantics lead many system implementors to state-smart immediate words (which inspect the variable STATE to distinguish between compilation and interpretation) but these have the drawback to fail unexpectedly in corner cases [Ertl98].
To identify whether or not a word is state-smart typically requires studying its implementation or documentation. Problems arise when these state-smart words are used as buildung blocks for more sophisticated words, e.g. by means of POSTPONE-ing them. The distinction between interpret and compile state is then also postponed and happens at inapropriate times.
There are ways to structure the outer interpreter so that words with non-default compilation semantics can be implemented without failures in corner cases. Systems such as gforth development version and VFX version 5 deal with this issue (cf. [Pelc17], [clf18]).
Looking at the Forth-94 standard only few words actually have non default compilation semantics, namely the already mentioned S" and TO. Forth-2012 adds others.
In general Forth-94 and Forth-2012 define execution semantics for normal words. When the word is interpreted this execution semantics is performed, when it is compiled its execution semantics is appended to the current definition (the standard compilation semantics). For most compiling words the standards explicitly leave the interpretation semantics undefined and define only a compilation semantics. This compilation semantics might include appending an additionally defined runtime semantics to the current definition. Colloquially these words are called compile-only. The standards also explicitly define immediate words: Words that have identical interpretation and compilation semantics: both perform the word's execution semantics. Finally there are special words with non-default compilation semantics (NDCS) where interpretation semantics and compilation semantics diverge.
The following table summarizes this situation:
| word kind | interpretation semantics | compilation semantics | example | comment |
|---|---|---|---|---|
| normal | perform execution semantics | compile execution semantics | DUP | normal :-definitions |
| immediate | perform execution semantics | perform execution semantics | .( | IMMEDIATE definitions |
| compile-only | undefined | perform execution semantics | IF | interpretation semantics undefined |
| NDCS | perform execution semantics for interpretation | perform execution semantics for compilation | S" | divergent interpretation, compilation |
S" and TO seem to be very special: For other words such as ' (tick) or CHAR, or .( (dot-paren)
the standards define similar compilation words namely ['](bracket-tick), [CHAR] (bracket-char), or ." (dot-quote)
for use in compiling mode inside definitions.
Some argue that complicating the outer interpreter or dictionary design is not beneficial especially
in memory restricted small system and either live with the failures of state-smart words
in corner cases or avoid implementing an interpretive S" to stay standard compliant.
Solution
For small systems it might be reasonable and simpler to not implement an S" with non default compilation semantics
as defined in the FILE word set but to define two diffently named words
that captures the compilation semantics and the interpretation semantics of FILE S" respectively.
Possible naming choices could be
S"(s-quote) for interpretation and[S"](bracket-s-quote-bracket) for compilation
used in the formS" hello"and: xxx ... [S"] hello" ... ;orS"(s-quote) for interpretation and
[S"(bracket-s-quote) for compilation
used in the formS" hello"and: xxx ... [S" hello"] ... ;
neither of which is really appealing.
Instead --- similar to ." for string output in definitions and. .( for
string output outside definitions --- we propose to keep CORE S" with its behaviour and to standardize
S( with interpretation semantics of FILE S" but using ) (right parenthesis) as delimiter.
Proposal
Add the word S( to the CORE Extension Word Set (CORE EXT):
S( "s-paren" CORE EXT
Interpretation: Perform the execution semantics given below.
Compilation: Perform the execution semantics given below.
Execution:
( "ccc\<paren\>" -- c-addr u )
Parse ccc delimited by ) (right parenthesis). Store the resulting string in a transient buffer c-addr u.S(is an immediate word.
See: Parsing, S"
Typical Use
Typical use of S( would be interactively when temporary strings are required
for example for use with INCLUDED:
S( s-paren.fs) INCLUDED
Remarks
As S( is proposed to be standardized in the CORE extension word set no standard
system is required to provide S(. However if a system chooses to implement the S" compilation
and interpretation semantics with two separately named words, it could choose the standard name
S" and the (not yet standardized) name S( for this.
Defining the interpretation semantics explicitly in the glossary entry above is not strictly necessary as both Forth-94 and Forth-2012 state:
Unless otherwise specified in an “Interpretation:” section of the glossary entry, the interpretation semantics of a Forth definition are its execution semantics.
Reference implementation
With only a single string buffer, a minimal S(-implementation could
look like this:
CREATE buf DECIMAL 80 CHARS ALLOT
: S( ( "ccc\<paren\>"-- c-addr len )
[CHAR] ) PARSE 80 MIN >R buf R@ MOVE buf R> ;
A more elabortated implementation using mutliple string buffers in a circular fashion is:
DECIMAL 80 CHARS CONSTANT |buf|
4 CONSTANT #bufs
CREATE bufs |buf| #bufs * ALLOT
VARIABLE buf# 0 buf# !
: buf ( -- c-addr )
bufs buf# @ |buf| * + ;
: bump ( -- )
buf# @ 1+ #bufs MOD buf# ! ;
: str ( char "ccc\<char\>" -- c-addr u )
bump buf SWAP PARSE |buf| MIN >R OVER R@ MOVE R> ;
: S( ( "ccc\<paren\>"-- c-addr len )
[CHAR] ) str ;
Testing
The following tests assure that S( pushes the desired c-addr u
CREATE s 3 c, CHAR a c, CHAR b c, CHAR c c,
t{ 99 S( abc) SWAP DROP -> 99 3 }t
t{ 99 S( abc) s COUNT COMPARE -> 99 0 }t
Experience
S( is not yet defined in any of the contemporary systems such as
- gForth
- VFX
- PFE
- DXForth
- FLT
- SwiftForth
- Win32Forth
- noForth
- amForth
- camelForth
- ciForth
- mecrisp
so it has no common use. However the name S( seems to be available
in all these systems.
Discussion
The proposal avoids issues with the NDCS word S" by providing S( as an alternative notation for an interpretive S".
It is intended for ressource restricted standard systems that want to support interpretive strings but which
are not able to provide the FILE word set S".
S( is very simple to implement so this proposal is rather about standarizing the name S( with the intended
functionality than a sophisticated feature.
One can argue to remove S" from the FILE word set, however this is not proposed here. Forth systems that provide
the FILE word set are hopefully capable of providing a complete and correct S" implementation.
References
[Forth-94]: "American National Standard for Information Systems — Programming Languages — Forth", ANSI X3.215-1994
[clf18]: discussion about special words in comp.lang.forth, https://groups.google.com/forum/#!topic/comp.lang.forth/Gb9Hvj3Wm_Y%5B1-25%5D
[Ertl98]: "State-smartness - Why it is Evil and How to Exorcise it", Anton Ertl, euroForth 1998
[Pelc17]: "Special Words in Forth", Stephen Pelc, euroForth 2017
Author
Ulrich Hoffmann uho@xlerb.de
Replies
proposal - Let us adopt the Gerry Jackson test suite as part of Forth 200x
Sounds like a sensible idea to me!
proposal - Let us adopt the Gerry Jackson test suite as part of Forth 200x
Great idea!
proposal - Let us adopt the Gerry Jackson test suite as part of Forth 200x
The proposal is short on details. What do you mean by "adopt"?
Should the standard document include the tests in the test suite, and be updated whenever the test suite is? That would cause a lot of work for the editor; not sure if the benefit is worth it.
Or should the standard document replace its tests with a reference to the test suite? That would make it harder to see both at once, which can be useful for understanding the normative part (by someone who is not familiar with the described word), and also useful for checking the tests (for someone who is familiar with the described word).
I guess the ideal approach would be that the tests contain information that states the section of the standard that the tests test, and that forth-standard.org automatically puts the tests into the appropriate sections. Then we don't need the tests in the latex version and the PDF (where I don't find them useful). But this would need work by both Gerry Jackson and Gerald Wodni, so if we want to go there, we should ask them if they are willing to do that.
proposal - Let us adopt the Gerry Jackson test suite as part of Forth 200x
By adopt, I mean that we say that the test suite maintained by Gerry, the version currently at some location is the official test suite for Forth200x. This does not mean that it is the only test suite, but that we the committee recognise it as a quality test suite. The details of what we do and do not want or say can be thrashed out at the next meeting. Since this is a non-normative section, we can even vote if we can find a suitable form of words.
Ideally, the current document's tests and Gerry's should be merged - a source file is needed for people who wish to test their systems. I agree that the tests can be valuable for understanding and I have encourage Gerry (and potential assistants) to include more documentation.
Your point about taking workload off the editor is valuable.
I will put the topic on the meeting agenda.
There is a lot of STATE that can be changed in a word called through CATCH, and that we may want to restore when an exception is THROWn, not just BASE and STATE. Conversely, there are many words that are called through CATCH that do not change BASE and STATE, so always saving and restoring BASE and STATE would be wasteful. My solution for this is cleanup wrappers, such as
: base-execute ( xt u -- )
base @ >r base ! catch r> base ! throw ;
And while the standard describes the input source specification as a kind of exception locals, at least Gforth uses a cleanup wrapper for restoring it. You can read more about cleanup in my EuroForth'08 paper.
Anyway, I don't think we need exception locals.
Another issue is that some people fail to use such wrappers (after all Thinking Forth does not advocate it), and then a THROW returns to a different CATCH or the Forth command line without restoring the state. For STATE, the standard specifies that it is set to interpreting when there is no user-supplied exception handler, for BASE the user has to fend for himself (as advocated in Thinking Forth). It is a good idea if the system warns of non-decimal BASE, e.g., by mentioning it after "ok".
ChangeLog
2018-08-15 Reference implementation now provides FIND-NAME-IN; more information on other systems; added NAME-PRINTER example
Problem
FIND has several problems:
It takes a counted string.
Its interface is designed for single-xt+immediate-flag systems (which cannot implement FILE S" correctly in all cases), and there is allowance for a STATE-dependent result to support other systems.
As currently specified (there is a proposal for fixing that), it fails to meet the goal of guaranteeing that the user-defined text-interpreter (outlined in the rationale of COMPILE,).
As a consequence of problem 2, the following implementation of ' is wrong:
: ' bl word find 0= -13 and throw ;
This ' produces the wrong result in the following case on Gforth:
: x ' ; immediate
] x s" [
And that's because Gforth implements S" correctly, and FIND such that the user-defined text interpreter works. The following definition would fix this problem on Gforth:
: ' bl word state @ >r postpone [ find r> if ] then 0= -13 and throw ;
but actually the specification of FIND currently is sufficiently unclear that we cannot be sure that that's guaranteed to work, either.
SEARCH-WORDLIST has a related, but different set of problems:
It produces results with a varying number of stack items. In some situations that makes it more cumbersome to use.
Its interface is designed for single-xt+immediate-flag systems, but this time without allowing a STATE-dependent result. As a consequence, Gforth produces the xt representing the interpretation semantics of the word, independent of STATE, with no way to get the compilation semantics of the word.
In the general case, given that the xt part of the result does not represent a part of the compilation semantics, the 1/-1 part of the result is pointless.
Solution
Introduce FIND-NAME (for the search order) and FIND-NAME-IN (for
specific wordlists). Both take strings in c-addr u form, and both
return the name token of the found word (or 0 if not found). We can
then go from the name token to the interpretation semantics with
NAME>INTERPRET,
and to the compilation semantics with
NAME>COMPILE.
Typical use
: ' parse-name find-name dup 0= -13 and throw name>interpret ;
: postpone
parse-name find-name dup 0= -13 and throw
name>compile swap postpone literal compile, ; immediate
\ user-defined text interpreter
: interpret-word
parse-name 2dup find-name if
nip nip state @ if name>compile else name>interpret then execute
else
... \ process numbers
then ;
\ alternative:
defer name>statetoken ( nt -- ... xt )
: [ ['] name>interpret is name>statetoken false state ! ;
[ \ initialize STATE and NAME>STATETOKEN
: ] ['] name>compile is name>statetoken true state ! ;
: interpret-word
parse-name 2dup find-name if
nip nip state name>statetoken execute
else
... \ process numbers
then ;
\ a defining word for words that print their name
: name-printer ( "name" -- )
>in @ create >in ! parse-name get-current find-name-in ,
does> ( -- )
@ name>string type ;
Remarks
FIND-NAME and FIND-NAME-IN are natural factors of all words that
look up words in the dictionary, such as FIND, ', POSTPONE, the text
interpreter, and SEARCH-WORDLIST. So implementing them does not cost
additional code in the Forth system, only some refactoring effort.
This approach is not compatible with the cmForth approach and Mark Humphries' approach, because they both use one word header each for interpretation and compilation semantics. This problem already exists for the other words that deal with name tokens, but the present proposal would make name tokens more important, and systems that do not support them less viable. However, both approaches have been known for at least two decades, and have seen little to no uptake in standard systems. And we have good approaches for implementing systems with name tokens, so excluding these approaches is not a significant loss.
Proposal
Add the following words:
FIND-NAME ( c-addr u -- nt | 0 )
Find the definition identified by the string c-addr u in the current search order. Return its name token nt, if found, otherwise 0.
FIND-NAME-IN ( c-addr u wid -- nt | 0 )
Find the definition identified by the string c-addr u in the wordlist wid. Return its name token nt, if found, otherwise 0.
Reference implementation
Testing
Experience
Gforth has implemented FIND-NAME and (under the name
(search-wordlist)) FIND-NAME-IN since
1996.
No problems were reported or found internally.
amForth has FIND-NAME "since ever" (first release 2006), and
FIND-NAME-IN under the name SEARCH-NAME.
Other systems have been reported to implement FIND-NAME under this
or other names (e.g., FOUND in ciforth).
The returned xt has the stack effect
( i*x x -- j*x )
Where should these i*x come from?
Could somebody provide an example when i or j is greater than 0?
In the light of recognizers it is better to leave the special characters for various literals and the like.
Also, itself is not directly connected with recursion — it can be returned as result or compiled into another definition.
: tt ... if ... lit, itself compile, ... then ... ;
It is one real usage case.
Control words such as IF where the control stack is the data stack.