Digest #230 2023-09-11
Contributions
proposal - 2023 Standards meeting agenda (2023-09-13 to 2023-09-15)
2023 Forth Standards Meeting Agenda
13 Sept 2023 12:00-19:00 MEST
14 Sept 2023 09:00-19:00 MEST
15 Sept 2023 09:00-13:00 MEST
On site at Hotel Villa Aricia, Rome, Italy
For latest details see chat.forth-standard.org
See also: euro.theforth.net, forth-standard.org
Schedule
Wednesday, 13th September 2023 (MEST)
12:00 Get together - Setup your gear and smalltalk
12:15 Call to order - get ready
12:30 Session 1
14:30 Bio Break
15:00 Session 2
17:00 Bio Break
17:15 Session 3
19:00 End of main sessions
Workshops
Thursday, 14th September 2023 (MEST)
09:00 Get together - Setup your gear and smalltalk
09:15 Call to order - get ready
09:30 Session 4
11:30 Lunch Break
12:30 Session 5
14:30 Bio Break
15:00 Session 6
17:00 Bio Break
17:15 Session 7
19:00 End of main sessions
Workshops
Friday, 15th September 2023 (MEST)
09:00 Get together - Setup your gear and smalltalk
09:15 Call to order - get ready (please be online by now)
09:30 Session 8
11:15 Bio Break
11:30 Session 9
13:00 End of Standards Meeting
afternoon euroForth conference
Agenda
Participants
- Welcome
- Determine the persons present
- Meeting transcript
Review of Procedures
- How we organize this meeting
- Progress of current work
Draft Document update (last draft is from 2019)
Are we ready for a new Standard snapshop (Forth2023)?
How can we speed up our work?
How we can better serve the Forth community?
How can we encourage Forthers to submit proposals?
What are addtional topics for future standardisation?
Reports
- Chair
- Editor
- Technical Officer
- Treasurer
Election/Confirmation of officers
If you would like to stand for election, please suggest your name and please shortly introduce yourself.
We might have to elect (by secret ballot) or confirm a Chair, Editor, Technical Officer and a Treasurer.
- Chair (currently Ulrich Hoffman)
- Editor (currently Peter Knaggs)
- Technical (currently Gerald Wodni)
- Treasurer (currently Bernd Paysan)
Review of Proposals/Contributions
Proposals can be retrieved on the fly from forth-standard.org/proposals
Please follow the links below for up to date lists of proposals and contribuions.
- Proposals in the state formal
- Proposals in the state informal
- Contributions on forth-standard.org
This includes also the above proposals. We will skip them on the fly.
Revision of contributions since the interim meeting in March 2023
Workshop Topics
Workshops are topics for discussion outside the formal meeting.
We will collect topics on the fly during the meeting's discussions.
Consideration of proposals + CfV votes
Which proposals should go for vote?
Any topics for proposal in the pipeline?
Workshop reports
Let's collect the results of our workshops.
Matters arising
Whats up?
Any other business
Something else?
Adjourn
Date of next meeting
When shall we three meet again? In thunder, lightning, or in rain?
Replies
proposal - Include a revised 79-STANDARD Specification for "><" To "Core Ext"
Have you looked at other current implementations to see how they solve this problem?
proposal - Include a revised 79-STANDARD Specification for "><" To "Core Ext"
Do you want a word equivalent to htonl, htons, ntohl and ntohs in C as included in arpa/inet.h or netinet/in.h?
Or do you want a word to swap bytes of cells?
The former has to take the type of processor into account whereas the latter does not.
If the former then I don't think it belongs in CORE EXT.
If the latter then why? I'm a strong believer in less is more.
And I think Forth-83 had that word too.
proposal - Include a revised 79-STANDARD Specification for "><" To "Core Ext"
As the author of Open Firmware, a Forth-based system that became a national standard (IEEE 1275-1994) . Open Firmware was used on millions of computers, of different byte orders and word sizes and alignment restrictions, produced by many different companies, Open Firmware supported portable device drives that would work unmodified across the full range of systems. Thus I can claim relevant experience on this topic
The typical use of byte-swizzling is to prepare or decode in-memory data structures that need to be passed to and from another device, whether another computer across a network or an I/O device that is controlled by memory-based descriptors. The data structure will have a defined byte order that might or might not match the host computer's byte order. Furthermore, the host might impose alignment restrictions different from the data structure. If you want to write portable code, then you cannot use a byte-swizzling primitive directly. You need a set of access wrappers akin to the "hton"/"ntoh" suite. Open Firmware drivers typically combine the host-dependent swizzling with unaligned-access, resulting in a set of works like "be-w@", "be-w!", "le-w@", "le-w!", and similar words for 32-bit and perhaps 64-bit access. It is a lot of words, but it actually solves the problem. Just having one swizzler primitive is "minimal", but doesn't actually get the job done. It can be useful as an implementation factor, but it shouldn't appear directly in portable code.
Before proposing something, it would be helpful to study modern Forth implementations like VFX Forth, Swift Forth, gforth, etc, to see how they solve this common problem. A proposal that conforms to a modern system's usage is much more likely to succeed than something modeled on a 40+ year old standard that never achieved much commercial traction.
Author:
Bernd Paysan
Change Log:
- 2020-09-06 initial version
- 2020-09-08 taking ruv's approach and vocabulary at translators
- 2020-09-08 replace the remaining rectypes with translators
- 2022-09-08 add the requested extensions, integrate results of bikeshedding discussion
- 2022-09-08 adjust reference implementation to results of last bikeshedding discussion
- 2022-09-09 Take comments from ruv into account, remove specifying STATE involvement
- 2022-09-10 More complete reference implementation
- 2022-09-10 Add use of extended words in reference implementation
- 2022-09-10 Typo fixed
- 2022-09-12 Fix for search order reference implementation
- 2022-09-15 Revert to Trute's table approach to call specific modes deliberately
- 2023-08-08 Remove names for table access words; there's no usage outside POSTPONE seen; POSTPONE can do that without a standardized way.
Problem:
The current recognizer proposal has received a number of critics. One is that its API is too big. So this proposal tries to create a very minimalistic API for a core recognizer, and allows to implement more fancy stuff as extensions. The problem this proposal tries to solve is the same as with the original recognizer proposal, this proposal is therefore not a full proposal, but sketches down some changes to the original proposal.
Solution:
Define the essentials of the recognizer in a RECOGNIZER word set, and allow building upon that. Common extensions go to the RECOGNIZER EXT wordset.
Important changes to the original proposal:
- Make the recognizer types executable to dispatch the methods (interpret, compile, postpone) themselves
- Make the recognizer sequence executable with the same effect as a recognizer
- Make sure the API is not mandating a special implementation
This replaces one poor man's method dispatch with another poor man's method dispatch, which is maybe less daunting and more flexible.
The core principle is still that the recognizer is not aware of state, and the returned translator is. If you have for some reason legacy code that looks like
: rec-xt ( addr u -- translator )
here place here find dup IF
0< state @ and IF compile, ELSE execute THEN ['] drop
ELSE drop ['] notfound THEN ;
then you should factor the part starting with state @ out and return it as translator:
: translate-xt ( xt flag -- )
0< state @ and IF compile, ELSE execute THEN ;
: rec-xt ( addr u -- ... translator )
here place here find dup IF ['] translate-xt
ELSE drop ['] notfound THEN ;
The standard interpreter loop should look like this:
: interpret ( i*x -- j*x )
BEGIN parse-name dup WHILE forth-recognize execute REPEAT
2drop ;
with the usual additions to check e.g. for empty stacks and such.
Typical use
TBD
Proposal:
XY. The optional Recognizer Wordset
A recognizer takes the string of a lexeme and returns a translator xt and additional data on the stack (no additional data for NOTFOUND):
REC-SOMETYPE ( addr len -- i*x translate-xt | NOTFOUND )
XY.3 Additional usage requirements
XY.3.1 Translator
translator: subtype of xt, and executes with the following stack effect:
TRANSLATE-THING ( j*x i*x -- k*x )
A translator xt that interprets, compiles or postpones the action of the thing according to what the state the system is in.
i*x is the additional information provided by the recognizer, j*x and k*x are the stack inputs and outputs of interpreting/compiling or postponing the thing.
XY.6 Glossary
XY.6.1 Recognizer Words
FORTH-RECOGNIZE ( addr len -- i*x translator-xt | NOTFOUND-xt ) RECOGNIZER
Takes a string and tries to recognize it, returning the translator xt and additional information if successful, or NOTFOUND if not.
NOTFOUND ( -- ) RECOGNIZER
Performs -13 THROW. If the exception word set is not present, the system shall use a best effort approach to display an adequate error message.
TRANSLATE: ( xt-int xt-comp xt-post "name" -- ) RECOGNIZER EXT
Create a translator word under the name "name". This word is the only standard way to define a user-defined translator from scratch.
"name:" ( jx ix -- k*x ) performs xt-int in interpretation, xt-comp in compilation and xt-post in postpone state using a system-specific way to determine the current mode.
XY.6.2 Recognizer Extension Words
SET-FORTH-RECOGNIZE ( xt -- ) RECOGNIZER EXT
Assign the recognizer xt to FORTH-RECOGNIZE.
Rationale:
FORTH-RECOGNIZE is likely a deferred word, but systems that implement it otherwise can use this word to change the behavior instead of using IS FORTH-RECOGNIZE.
FORTH-RECOGNIZER ( -- xt ) RECOGNIZER EXT
Obtain the recognizer xt that is assigned to FORTH-RECOGNIZE.
Rationale:
FORTH-RECOGNIZE is likely a deferred word, but systems that implement it otherwise, can use this word to change the behavior instead of using ACTION-OF FORTH-RECOGNIZE. The old API has this function under the name FORTH-RECOGNIZER (as a value) and this name is reused. Systems that want to continue to support the old API can support TO FORTH-RECOGNIZER, too.
RECOGNIZER-SEQUENCE: ( xt1 .. xtn n "name" -- ) RECOGNIZER EXT
Create a named recognizer sequence under the name "name", which, when executed, tries to recognize strings starting with xtn and proceeding towards xt1 until successful.
SET-RECOGNIZER-SEQUENCE ( xt1 .. xtn n xt-seq -- ) RECOGNIZER EXT
Set the recognizer sequence of xt-seq to xt1 .. xtn.
GET-RECOGNIZER-SEQUENCE ( xt-seq -- xt1 .. xtn n ) RECOGNIZER EXT
Obtain the recognizer sequence xt-seq as xt1 .. xtn n.
TANSLATE-NT ( jx nt -- kx ) RECOGNIZER EXT
Translates a name token; system component you can use to construct other translators of.
TRANSLATE-NUM ( jx x -- kx ) RECOGNIZER EXT
Translates a number; system component you can use to construct other translators of.
Reference implementation:
This is a minimalistic core implementation for a recognizer-enabled system, that handles only words and single numbers without base prefix. This implementation does only take interpret and compile state into account, and uses the STATE variable to distinguish.
Defer forth-recognize ( addr u -- i*x translator-xt / notfound )
: interpret ( i*x -- j*x )
BEGIN
?stack parse-name dup WHILE
forth-recognize execute
REPEAT ;
: lit, ( n -- ) postpone literal ;
: notfound ( state -- ) -13 throw ;
: translate: ( xt-interpret xt-compile xt-postpone "name" -- )
create , , ,
does> state @ 2 + cells + @ execute ;
:noname name>interpret execute ;
:noname name>compile execute ;
:noname name>compile swap lit, compile, ;
translate: translate-nt ( nt -- )
' noop
' lit,
:noname lit, postpone lit, ;
translate: translate-num ( n -- )
: rec-nt ( addr u -- nt nt-translator / notfound )
forth-wordlist find-name-in dup IF ['] translate-nt ELSE drop ['] notfound THEN ;
: rec-num ( addr u -- n num-translator / notfound )
0. 2swap >number 0= IF 2drop ['] translate-num ELSE 2drop drop ['] notfound THEN ;
: minimal-recognize ( addr u -- nt nt-translator / n num-translator / notfound )
2>r 2r@ rec-nt dup ['] notfound = IF drop 2r@ rec-num THEN 2rdrop ;
' minimal-recognizer is forth-recognize
Extensions reference implementation:
: set-forth-recognize ( xt -- )
is forth-recognize ;
: forth-recognizer ( -- xt )
action-of forth-recognize ;
Stack library
: STACK: ( size "name" -- )
CREATE 0 , CELLS ALLOT ;
: SET-STACK ( item-n .. item-1 n stack-id -- )
2DUP ! CELL+ SWAP CELLS BOUNDS
?DO I ! CELL +LOOP ;
: GET-STACK ( stack-id -- item-n .. item-1 n )
DUP @ >R R@ CELLS + R@ BEGIN
?DUP
WHILE
1- OVER @ ROT CELL - ROT
REPEAT
DROP R> ;
Recognizer sequences
: recognize ( addr len rec-seq-id -- i*x translator-xt | NOTFOUND )
DUP >R @
BEGIN
DUP
WHILE
DUP CELLS R@ + @
2OVER 2>R SWAP 1- >R
EXECUTE DUP ['] NOTFOUND <> IF
2R> 2DROP 2R> 2DROP EXIT
THEN
DROP R> 2R> ROT
REPEAT
DROP 2DROP R> DROP ['] NOTFOUND
;
#10 Constant min-sequence#
: recognizer-sequence: ( rec1 .. recn n "name" -- )
min-sequence# stack: min-sequence# 1+ cells negate here + set-stack
DOES> recognize ;
: ?defer@ ( xt1 -- xt2 )
BEGIN dup is-defer? WHILE defer@ REPEAT ;
: set-recognizer-sequence ( rec1 .. recn n rec-seq-xt -- ) ?defer@ >body set-stack ;
: get-recognizer-sequence ( rec-seq-xt -- rec1 .. recn n ) ?defer@ >body get-stack ;
Once you have recognizer sequences, you shall define
' rec-num ' rec-nt 2 recognizer-sequence: default-recognize
' default-recognize is forth-recognize
The recognizer stack looks surprisingly similar to the search order stack, and Gforth uses a recognizer stack to implement the search order. In order to do so, you define wordlists in a way that a wid is an execution token which searches the wordlist and returns the appropriate translator.
: find-name-in ( addr u wid -- nt / 0 )
execute ['] notfound = IF 0 THEN ;
root-wordlist forth-wordlist dup 3 recognizer-sequence: search-order
: find-name ( addr u -- nt / 0 )
['] search-order find-name-in ;
: get-order ( -- wid1 .. widn n )
['] search-order get-recognizer-sequence ;
: set-order ( wid1 .. widn n -- )
['] search-order set-recognizer-sequence ;
Testing
TBD