,---------------. | Contributions | `---------------´ ,------------------------------------------ | 2018-08-17 16:27:53 UlrichHoffmann wrote: | proposal - S( "Request for Discussion" (revised 2018-08-16) | see: https://forth-standard.org/proposals/s-request-for-discussion-revised-2018-08-16-#contribution-65 `------------------------------------------ S( "Request for Discussion" =========================== Change History -------------- 2018-08-16 Improve with additional explanation, rewording 2018-07-09 First version Problem ------- There have been extensive discussions about correct implementations (cf. [Ertl98], [Pelc17], [clf18]) of Forth words that have both - an interpretation semantics and - a compilation semantics that is different from adding the word's execution semantics to the current definition (sometimes called non-default compilation semantics, NDCS). The Forth-94 word `S"` is an example for this as it has a defined compilation semantics (and an explicitly undefined interpretation semantics) in the CORE word set and a defined interpretation semantics in the FILE word set. (cf. [Forth94]) Words like this can have the so called *copy&paste* property: Program phrases that have been tested in interpretation mode can be copied unchanged into definitions and continue to work there in a seemingly identical way. This is attractive to quite some developers. The desire to have words with diverging compilation semantics and interpretation semantics lead many system implementors to *state-smart* immediate words (which inspect the variable STATE to distinguish between compilation and interpretation) but these have the drawback to fail unexpectedly in corner cases [Ertl98]. To identify whether or not a word is state-smart typically requires studying its implementation or documentation. Problems arise when these state-smart words are used as buildung blocks for more sophisticated words, e.g. by means of POSTPONE-ing them. The distinction between interpret and compile state is then also postponed and happens at inapropriate times. There are ways to structure the outer interpreter so that words with non-default compilation semantics can be implemented without failures in corner cases. Systems such as gforth development version and VFX version 5 deal with this issue (cf. [Pelc17], [clf18]). Looking at the Forth-94 standard only few words actually have non default compilation semantics, namely the already mentioned S" and TO. Forth-2012 adds others. In general Forth-94 and Forth-2012 define execution semantics for *normal words*. When the word is interpreted this execution semantics is performed, when it is compiled its execution semantics is appended to the current definition (the *standard compilation semantics*). For most *compiling words* the standards explicitly leave the interpretation semantics undefined and define only a compilation semantics. This compilation semantics might include appending an additionally defined runtime semantics to the current definition. Colloquially these words are called *compile-only*. The standards also explicitly define *immediate* words: Words that have identical interpretation and compilation semantics: both perform the word's execution semantics. Finally there are special words with non-default compilation semantics (NDCS) where interpretation semantics and compilation semantics diverge. The following table summarizes this situation: --- | word kind | interpretation semantics | compilation semantics | example | comment |--------------|------------------------------------------------|---------------------------------------------|---------|-------------------------------------- | *normal* | perform execution semantics | compile execution semantics | DUP | normal :-definitions | immediate | perform execution semantics | perform execution semantics | .( | IMMEDIATE definitions | compile-only | undefined | perform execution semantics | IF | interpretation semantics undefined | NDCS | perform execution semantics for interpretation | perform execution semantics for compilation | S" | divergent interpretation, compilation --- S" and TO seem to be very special: For other words such as `'` (tick) or `CHAR`, or `.(` (dot-paren) the standards define similar compilation words namely `[']`(bracket-tick), `[CHAR]` (bracket-char), or `."` (dot-quote) for use in compiling mode inside definitions. Some argue that complicating the outer interpreter or dictionary design is not beneficial especially in memory restricted small system and either live with the failures of state-smart words in corner cases or avoid implementing an interpretive `S"` to stay standard compliant. Solution -------- For small systems it might be reasonable and simpler to **not implement** an `S"` with non default compilation semantics as defined in the FILE word set but to define two diffently named words that captures the compilation semantics and the interpretation semantics of FILE ``S"`` respectively. Possible naming choices could be - `S"` (s-quote) for interpretation and `[S"]` (bracket-s-quote-bracket) for compilation used in the form `S" hello"` and `: xxx ... [S"] hello" ... ;` or - `S"` (s-quote) for interpretation and `[S"` (bracket-s-quote) for compilation used in the form `S" hello"` and `: xxx ... [S" hello"] ... ;` neither of which is really appealing. Instead --- similar to `."` for string output in definitions and. `.(` for string output outside definitions --- we propose to keep CORE `S"` with its behaviour and to standardize `S(` with interpretation semantics of FILE `S"` but using `)` (right parenthesis) as delimiter. --- Proposal -------- Add the word `S(` to the CORE Extension Word Set (CORE EXT): --- `S(` "s-paren" CORE EXT - Interpretation: Perform the execution semantics given below. - Compilation: Perform the execution semantics given below. - Execution: `( "ccc" -- c-addr u )` Parse ccc delimited by ) (right parenthesis). Store the resulting string in a transient buffer c-addr u. `S(` is an immediate word. See: Parsing, `S"` --- Typical Use ----------- Typical use of `S(` would be interactively when temporary strings are required for example for use with `INCLUDED`: S( s-paren.fs) INCLUDED Remarks ------- As `S(` is proposed to be standardized in the CORE extension word set no standard system is required to provide `S(`. However if a system chooses to implement the `S"` compilation and interpretation semantics with two separately named words, it could choose the standard name `S"` and the (not yet standardized) name `S(` for this. Defining the interpretation semantics explicitly in the glossary entry above is not strictly necessary as both Forth-94 and Forth-2012 state: > Unless otherwise specified in an “Interpretation:” section of the > glossary entry, the interpretation semantics of a Forth definition > are its execution semantics. Reference implementation ------------------------ With only a single string buffer, a minimal `S(`-implementation could look like this: CREATE buf DECIMAL 80 CHARS ALLOT : S( ( "ccc"-- c-addr len ) [CHAR] ) PARSE 80 MIN >R buf R@ MOVE buf R> ; A more elabortated implementation using mutliple string buffers in a circular fashion is: DECIMAL 80 CHARS CONSTANT |buf| 4 CONSTANT #bufs CREATE bufs |buf| #bufs * ALLOT VARIABLE buf# 0 buf# ! : buf ( -- c-addr ) bufs buf# @ |buf| * + ; : bump ( -- ) buf# @ 1+ #bufs MOD buf# ! ; : str ( char "ccc" -- c-addr u ) bump buf SWAP PARSE |buf| MIN >R OVER R@ MOVE R> ; : S( ( "ccc"-- c-addr len ) [CHAR] ) str ; Testing ------- The following tests assure that `S(` pushes the desired c-addr u CREATE s 3 c, CHAR a c, CHAR b c, CHAR c c, t{ 99 S( abc) SWAP DROP -> 99 3 }t t{ 99 S( abc) s COUNT COMPARE -> 99 0 }t Experience ---------- `S(` is not yet defined in any of the contemporary systems such as - gForth - VFX - PFE - DXForth - FLT - SwiftForth - Win32Forth - noForth - amForth - camelForth - ciForth - mecrisp so it has no common use. However the name `S(` seems to be available in all these systems. Discussion ---------- The proposal avoids issues with the NDCS word `S"` by providing `S(` as an alternative notation for an interpretive `S"`. It is intended for ressource restricted standard systems that want to support interpretive strings but which are not able to provide the FILE word set `S"`. `S(` is very simple to implement so this proposal is rather about standarizing the name `S(` with the intended functionality than a sophisticated feature. One can argue to remove `S"` from the FILE word set, **however this is not proposed here**. Forth systems that provide the FILE word set are hopefully capable of providing a complete and correct `S"` implementation. References ---------- [Forth-94]: "American National Standard for Information Systems — Programming Languages — Forth", ANSI X3.215-1994 [clf18]: discussion about *special words* in comp.lang.forth, https://groups.google.com/forum/#!topic/comp.lang.forth/Gb9Hvj3Wm_Y%5B1-25%5D [Ertl98]: "State-smartness - Why it is Evil and How to Exorcise it", Anton Ertl, euroForth 1998 [Pelc17]: "Special Words in Forth", Stephen Pelc, euroForth 2017 Author ------ Ulrich Hoffmann uho@xlerb.de ,---------. | Replies | `---------´ ,------------------------------------------ | 2018-07-11 08:51:48 MarkWills replies: | proposal - Let us adopt the Gerry Jackson test suite as part of Forth 200x | see: https://forth-standard.org/proposals/let-us-adopt-the-gerry-jackson-test-suite-as-part-of-forth-200x#reply-169 `------------------------------------------ Sounds like a sensible idea to me! ,------------------------------------------ | 2018-07-13 16:25:47 alextangent replies: | proposal - Let us adopt the Gerry Jackson test suite as part of Forth 200x | see: https://forth-standard.org/proposals/let-us-adopt-the-gerry-jackson-test-suite-as-part-of-forth-200x#reply-170 `------------------------------------------ Great idea! ,------------------------------------------ | 2018-07-14 06:32:34 AntonErtl replies: | proposal - Let us adopt the Gerry Jackson test suite as part of Forth 200x | see: https://forth-standard.org/proposals/let-us-adopt-the-gerry-jackson-test-suite-as-part-of-forth-200x#reply-171 `------------------------------------------ The proposal is short on details. What do you mean by "adopt"? Should the standard document include the tests in the test suite, and be updated whenever the test suite is? That would cause a lot of work for the editor; not sure if the benefit is worth it. Or should the standard document replace its tests with a reference to the test suite? That would make it harder to see both at once, which can be useful for understanding the normative part (by someone who is not familiar with the described word), and also useful for checking the tests (for someone who is familiar with the described word). I guess the ideal approach would be that the tests contain information that states the section of the standard that the tests test, and that forth-standard.org automatically puts the tests into the appropriate sections. Then we don't need the tests in the latex version and the PDF (where I don't find them useful). But this would need work by both Gerry Jackson and Gerald Wodni, so if we want to go there, we should ask them if they are willing to do that. ,------------------------------------------ | 2018-07-16 22:45:40 StephenPelc replies: | proposal - Let us adopt the Gerry Jackson test suite as part of Forth 200x | see: https://forth-standard.org/proposals/let-us-adopt-the-gerry-jackson-test-suite-as-part-of-forth-200x#reply-172 `------------------------------------------ By adopt, I mean that we say that the test suite maintained by Gerry, the version currently at some location is the official test suite for Forth200x. This does not mean that it is the only test suite, but that we the committee recognise it as a quality test suite. The details of what we do and do not want or say can be thrashed out at the next meeting. Since this is a non-normative section, we can even vote if we can find a suitable form of words. Ideally, the current document's tests and Gerry's should be merged - a source file is needed for people who wish to test their systems. I agree that the tests can be valuable for understanding and I have encourage Gerry (and potential assistants) to include more documentation. Your point about taking workload off the editor is valuable. I will put the topic on the meeting agenda. ,------------------------------------------ | 2018-08-15 10:12:05 AntonErtl replies: | proposal - EXCEPTION LOCALs | see: https://forth-standard.org/proposals/exception-locals#reply-173 `------------------------------------------ There is a lot of STATE that can be changed in a word called through CATCH, and that we may want to restore when an exception is THROWn, not just BASE and STATE. Conversely, there are many words that are called through CATCH that do not change BASE and STATE, so always saving and restoring BASE and STATE would be wasteful. My solution for this is cleanup wrappers, such as ``` : base-execute ( xt u -- ) base @ >r base ! catch r> base ! throw ; ``` And while the standard describes the input source specification as a kind of exception locals, at least Gforth uses a cleanup wrapper for restoring it. You can read more about cleanup in [my EuroForth'08 paper](http://www.complang.tuwien.ac.at/anton/euroforth/ef08/papers/ertl.pdf). Anyway, I don't think we need exception locals. Another issue is that some people fail to use such wrappers (after all Thinking Forth does not advocate it), and then a THROW returns to a different CATCH or the Forth command line without restoring the state. For STATE, the standard specifies that it is set to interpreting when there is no user-supplied exception handler, for BASE the user has to fend for himself (as advocated in Thinking Forth). It is a good idea if the system warns of non-decimal BASE, e.g., by mentioning it after "ok". ,------------------------------------------ | 2018-08-15 10:38:39 AntonErtl replies: | proposal - find-name | see: https://forth-standard.org/proposals/find-name#reply-174 `------------------------------------------ ## ChangeLog 2018-08-15 Reference implementation now provides FIND-NAME-IN; more information on other systems; added NAME-PRINTER example ## Problem `FIND` has several problems: 1. It takes a counted string. 2. Its interface is designed for single-xt+immediate-flag systems (which cannot implement FILE S" correctly in all cases), and there is allowance for a STATE-dependent result to support other systems. 3. As currently specified (there is a proposal for fixing that), it fails to meet the goal of guaranteeing that the user-defined text-interpreter (outlined in the [rationale of COMPILE,](http://forth-standard.org/standard/rationale#rat:core:COMPILE,)). As a consequence of problem 2, the following implementation of ' is wrong: ``` : ' bl word find 0= -13 and throw ; ``` This ' produces the wrong result in the following case on Gforth: ``` : x ' ; immediate ] x s" [ ``` And that's because Gforth implements S" correctly, and FIND such that the user-defined text interpreter works. The following definition would fix this problem on Gforth: ``` : ' bl word state @ >r postpone [ find r> if ] then 0= -13 and throw ; ``` but actually the specification of FIND currently is sufficiently unclear that we cannot be sure that that's guaranteed to work, either. `SEARCH-WORDLIST` has a related, but different set of problems: 1. It produces results with a varying number of stack items. In some situations that makes it more cumbersome to use. 2. Its interface is designed for single-xt+immediate-flag systems, but this time without allowing a STATE-dependent result. As a consequence, Gforth produces the xt representing the interpretation semantics of the word, independent of STATE, with no way to get the compilation semantics of the word. 3. In the general case, given that the xt part of the result does not represent a part of the compilation semantics, the 1/-1 part of the result is pointless. ## Solution Introduce `FIND-NAME` (for the search order) and `FIND-NAME-IN` (for specific wordlists). Both take strings in c-addr u form, and both return the name token of the found word (or 0 if not found). We can then go from the name token to the interpretation semantics with [NAME>INTERPRET](http://forth-standard.org/standard/tools/NAMEtoINTERPRET), and to the compilation semantics with [NAME>COMPILE](http://forth-standard.org/standard/tools/NAMEtoCOMPILE). ## Typical use ``` : ' parse-name find-name dup 0= -13 and throw name>interpret ; : postpone parse-name find-name dup 0= -13 and throw name>compile swap postpone literal compile, ; immediate \ user-defined text interpreter : interpret-word parse-name 2dup find-name if nip nip state @ if name>compile else name>interpret then execute else ... \ process numbers then ; \ alternative: defer name>statetoken ( nt -- ... xt ) : [ ['] name>interpret is name>statetoken false state ! ; [ \ initialize STATE and NAME>STATETOKEN : ] ['] name>compile is name>statetoken true state ! ; : interpret-word parse-name 2dup find-name if nip nip state name>statetoken execute else ... \ process numbers then ; \ a defining word for words that print their name : name-printer ( "name" -- ) >in @ create >in ! parse-name get-current find-name-in , does> ( -- ) @ name>string type ; ``` ## Remarks `FIND-NAME` and `FIND-NAME-IN` are natural factors of all words that look up words in the dictionary, such as FIND, ', POSTPONE, the text interpreter, and SEARCH-WORDLIST. So implementing them does not cost additional code in the Forth system, only some refactoring effort. This approach is not compatible with the cmForth approach and Mark Humphries' approach, because they both use one word header each for interpretation and compilation semantics. This problem already exists for the other words that deal with name tokens, but the present proposal would make name tokens more important, and systems that do not support them less viable. However, both approaches have been known for at least two decades, and have seen little to no uptake in standard systems. And we have good approaches for implementing systems with name tokens, so excluding these approaches is not a significant loss. ## Proposal Add the following words: `FIND-NAME ( c-addr u -- nt | 0 )` Find the definition identified by the string c-addr u in the current search order. Return its name token nt, if found, otherwise 0. `FIND-NAME-IN ( c-addr u wid -- nt | 0 )` Find the definition identified by the string c-addr u in the wordlist wid. Return its name token nt, if found, otherwise 0. ## Reference implementation [Reference Implementation](http://www.forth200x.org/reference-implementations/find-name.fs) ## Testing [Testcases](http://www.forth200x.org/tests/find-name.fs) ## Experience Gforth has implemented `FIND-NAME` and (under the name `(search-wordlist)`) `FIND-NAME-IN` since [1996](http://git.savannah.gnu.org/cgit/gforth.git/commit/?id=3955fa40889bd5e49e995e08e0606c7c6905028e). No problems were reported or found internally. amForth has `FIND-NAME` "since ever" (first release 2006), and `FIND-NAME-IN` under the name `SEARCH-NAME`. Other systems have been reported to implement `FIND-NAME` under this or other names (e.g., FOUND in ciforth). ,------------------------------------------ | 2018-08-16 11:10:39 ruv replies: | comment - NAME>COMPILE result | see: https://forth-standard.org/standard/tools/NAMEtoCOMPILE#reply-175 `------------------------------------------ > The returned xt has the stack effect `( i*x x -- j*x )` Where should these `i*x` come from? Could somebody provide an example when `i` or `j` is greater than 0? ,------------------------------------------ | 2018-08-16 11:23:47 ruv replies: | comment - proposal: recurse' to complement recurse | see: https://forth-standard.org/standard/core/RECURSE#reply-176 `------------------------------------------ In the light of recognizers it is better to leave the special characters for various literals and the like. ,------------------------------------------ | 2018-08-16 11:32:01 ruv replies: | comment - proposal: recurse' to complement recurse | see: https://forth-standard.org/standard/core/RECURSE#reply-177 `------------------------------------------ Also, `itself` is not directly connected with recursion — it can be returned as result or compiled into another definition. ``` : tt ... if ... lit, itself compile, ... then ... ; ``` It is one real usage case. ,------------------------------------------ | 2018-08-17 07:52:57 JennyBrien replies: | comment - NAME>COMPILE result | see: https://forth-standard.org/standard/tools/NAMEtoCOMPILE#reply-178 `------------------------------------------ Control words such as IF where the control stack is the data stack.