6 Glossary

6.2.1177 DEFER@ defer-fetch CORE EXT

( xt₁ -- xt₂ )

xt₂ is the execution token xt₁ is set to execute. An ambiguous condition exists if xt₁ is not the execution token of a word defined by DEFER, or if xt₁ has not been set to execute an xt.

See:

6.2.0698 ACTION-OF, 6.2.1173 DEFER, 6.2.1175 DEFER!, 6.2.1725 IS.

Implementation:

: DEFER@ ( xt1 -- xt2 )
>BODY @ ;

Testing:

T{ DEFER defer4 -> }T

T{ ' * ' defer4 DEFER! -> }T
T{ 2 3 defer4 -> 6 }T
T{ ' defer4 DEFER@ -> ' * }T

T{ ' + IS defer4 -> }T
T{ 1 2 defer4 -> 3 }T
T{ ' defer4 DEFER@ -> ' + }T

ContributeContributions

EricBlake [388] Must the xt returned by DEFER@ have the same value as the one passed to DEFER!, or merely the same behavior?Request for clarification2025-07-29 20:21:10

The test implies that the identity of the xt passed to DEFER! (or IS) should be preserved:

T{ DEFER defer4 -> }T
T{ ' + IS defer4 -> }T
T{ ' defer4 DEFER@ -> ' + }T

But elsewhere, in 2.1, we read

execution token:
    A value that identifies the execution semantics of a definition.

and 3.1 is clear that an xt is 1 cell. However, I'm working on an implementation of Forth where the implied requirement of identical tokens is overly restrictive. In my implementation, I can dispatch to words faster if every execution token is treated as the address to a pair of cells: one cell holding a pointer to the code handler to execute, and the other cell holding a parameter to be (optionally) used by that handler. For example, implementing a CONSTANT creates a pair "do-lit, value" where do-lit is a handler that knows how to push value to the stack (but where COMPILE, can bypass calling the do-lit handler and just compile code that directly pushes value to the stack); a VALUE creates a pair "do-val, addr" where addr is a one-cell location reserved at the time word was defined, and where the do-val handler performs (or COMPILE, inlines) "addr @", and so forth.

The interesting aspect of this is that for most other handlers, copying the two-cell contents to any other address still has the same semantics as the two cells in their original location (it doesn't matter whether I use a pointer to the two cells "do-lit, 5" that were compiled into 5 CONSTANT five, or a pointer to the two cells "do-lit, 5" that were compiled as part of the body of : doit 5 ; - any time my execution engine sees that two-cell sequence, it has the same semantics of pushing 5 to the stack). Note that in my scheme, VALUEs store an address in the parameter field of its 2-cell representation (where that address is basically ALIGN HERE 1 CELLS ALLOT at the time VALUE was run), and not the current value set by the most recent TO; that's because I have planned for the contents of an xt to be copied around, while still preserving the semantics regardless of the address where that copy of the 2 cell xt contents lives. Put another way, the compilation of 5 VALUE v : getv v ; must not hard-code a 5 as the value that v happened to have when getv was compiled, but rather must compile code that looks up the current value that v has at the time getv is executed; but it is more efficient for the compilation body of getv to have the two-cell sequence "do-val, addr" where do-val does "addr @" than it is to have the compilation body of getv have the two-cell sequence "do-call, ' v".

As fallout of that design, in my system, two distinct single-cell values can both be considered equivalent xts if the two cells they each point to have the same contents:

: xt= ( xt2 xt1 -- flag ) \ determine if xt1 and xt2 have the same execution semantics. False negatives are possible, but not false positives
  2@ ROT 2@ D= ;  \ implementation-dependent

With that background, my implementation of DEFER could be as simple as:

: DEFER ( "name" -- )
  [: abort" defer not assigned yet" ;] 2@ 2VALUE  \ share the same dictionary implementation as 2VALUE...
  do-defer latest !  \ ...except that I swap the handler from do-val2 to do-defer
  \ where do-val2 performs "addr 2@", do-defer performs the equivalent of "addr execute"
  \ in this implementation, "' name >BODY" gives addr
  ;
: DEFER! ( xt2 xt1 -- )
  >R 2@ R> >BODY 2! ;
: DEFER@ ( xt1 -- xt2 )
  >BODY ;

However, that implementation fails the testsuite as written: the two cells residing in the body of a DEFERred word have the same contents and thus the same semantics as the xt that was passed to DEFER!, but live at a different address (although the test passed ' + to IS defer4, DEFER@ gives back ' defer4 >BODY). Observe that my ' + ' defer4 DEFER@ xt= predicate correctly reports a true flag, but that xt= predicate is not portable to other implementations, and thus is not viable for the testsuite.

I can argue that section 2.1 merely requires that an xt be "A value that identifies the execution semantics of a definition.", and not "The unique value..."; thus, I see no compelling reason that consecutive calls to ' word must return the same immutable value for that word. In fact, I could envision a Forth system that provides an extension to optimize existing words in the dictionary, which recompiles them to better code and changes the xt that future ' word will produce even while preserving execution semantics. And it's also not hard to argue that with word-lists and the use of SYNONYM to copy a definition from one list to another while keeping the name, that ' word may produce different results based on the current wordlist order even when those various xts still resolve to the same execution semantics of the original word that all the other wordlists copied from.

In the case of my system, I believe that as long as I have the same two-cell contents passed to the execution engine, then the address of those two cells forms an xt of name even if that address differs from the one that ' name returns. If I'm right, my implementation complies with the standard but fails the testsuite, meaning the testsuite is too strict; in which case any use of -> ' in the testsuite is non-portable, and the most it can portably do is assert things like T{ 1 2 ' defer4 DEFER@ -> 3 }T after defer4 has been directed to ' +. But if I'm wrong, I could change my implementation to instead share the implementation of DEFER with VALUE (only 1 CELLS ALLOT instead of 2), at the expense of now every time my execution encounters the cell pair "do-defer addr", it must execute the slower sequence "addr @ execute" (an extra indirection from addr to xt, compared to my earlier implementation "addr execute" treating addr as the xt to dereference).

So, I'm asking clarification on whether the xt value passed to DEFER! must be preserved verbatim to that given by DEFER@, or whether the standard permits any other xt value so long as its execution semantics are the same.

EricBlakeNew Version: Must the xt returned by DEFER@ have the same value as the one passed to DEFER!, or merely the same behavior? [r1458] 2025-07-30 03:29:04

Hide differences

The test implies that the identity of the xt passed to DEFER! (or IS) should be preserved:

T{ DEFER defer4 -> }T
T{ ' + IS defer4 -> }T
T{ ' defer4 DEFER@ -> ' + }T

But elsewhere, in 2.1, we read

execution token:
    A value that identifies the execution semantics of a definition.

As fallout of that design, in my system, two distinct single-cell values can both be considered equivalent xts if the two cells they each point to have the same contents:

: xt= ( xt2 xt1 -- flag ) \ determine if xt1 and xt2 have the same execution semantics. False negatives are possible, but not false positives
  2@ ROT 2@ D= ;  \ implementation-dependent

With that background, my implementation of DEFER could be as simple as:

: DEFER ( "name" -- )
  [: abort" defer not assigned yet" ;] 2@ 2VALUE  \ share the same dictionary implementation as 2VALUE...
  do-defer latest !  \ ...except that I swap the handler from do-val2 to do-defer
  \ where do-val2 performs "addr 2@", do-defer performs the equivalent of "addr execute"
  \ in this implementation, "' name >BODY" gives addr
  ;
: DEFER! ( xt2 xt1 -- )
  >R 2@ R> >BODY 2! ;
: DEFER@ ( xt1 -- xt2 )
  >BODY ;

In the case of my system, I believe that as long as I have the same two-cell contents passed to the execution engine, then the address of those two cells forms an xt of name even if that address differs from the one that ' name returns. If I'm right, my implementation complies with the standard but fails the testsuite, meaning the testsuite is too strict; in which case any use of -> ' in the testsuite is non-portable, and the most it can portably do is assert things like T{ 1 2 ' defer4 DEFER@ EXECUTE -> 3 }T after defer4 has been directed to ' +. But if I'm wrong, I could change my implementation to instead share the implementation of DEFER with VALUE (only 1 CELLS ALLOT instead of 2), at the expense of now every time my execution encounters the cell pair "do-defer addr", it must execute the slower sequence "addr @ execute" (an extra indirection from addr to xt, compared to my earlier implementation "addr execute" treating addr as the xt to dereference).

AntonErtl [r1459] 2025-07-30 08:39:25

Before answering your question:

Your implementation of defer@ is incorrect even if there is only a requirement for EXECUTE equivalence, but no requirement for xt equality. Consider

T{ 1 2 ' defer4 DEFER@ ' - ' defer4 DEFER! EXECUTE -> 3 }T

Changing defer4 after the defer@ should not change the behaviour of the xt produced by defer@, which should represent +. But in your implementation, it does. Note that in your implementation, you would always get the same xt for a given deferred word, so it would represent the behaviour of the deferred word, not the behaviour that the deferred word is set to.

In general, creating a new xt from a pair of code and data (each requiring one cell) requires two cells of memory every time you do it, so even if there is no requirement for defer@ to produce the same xt that was given to defer!, I don't see a useful way that does not satisfy this requirement: Neither defer! nor defer@ allocates memory (unless the Forth system employs garbage collection for xts), so the obvious way for implementing defer@ is to produce the same xt.

I am trying to imagine an implementation where you work foremost with code/data pairs, and in most cases you just provide a pointer to that pair, but for defer@ you have a lookup table and it gets the xt from there. Now if you have two words with the same code/data pairs but a different xt (at the moment I have trouble thinking up an example; maybe synonyms, but why not have the same xt for them?), the lookup would only produce one of these xts, and defer@ could produce an xt that defer! did not store there. I am not aware of such an implementation, or anybody wanting to implement such an implementation; it also does not have any merits compared to existing implementation strategies that would make me want to hope that it complies with the standard.

An alternative implementation of DEFER etc. for your implementation strategy is to have three cells: the two cells for code and data pointer (used for, e.g., execute) and a cell for the original xt (used for defer@).

In general, we have had to deal with the problem of how to represent code and data in a single cell from the very earliest Forth implementations, and I have published a paper about this problem at EuroForth 2024.

Onwards to your question:

For defer! the specification says:

Set the word xt1 to execute xt2.

For defer@ the specification says:

xt2 is the execution token xt1 is set to execute.

This sounds to me that the xt2 produced by defer@ should be the same as the one that was last stored with defer! or is. That certainly was my thinking when I wrote the proposal, and the wording also points in that direction: "the execution token ...", not "an execution token equivalent to the one that ...".

Relevance: Are there any programs that rely on xt equality? While I don't remember writing code that checks the result of defer@ or action-of for equality, I can imagine that some people might do it for good reasons. There is no standard xt= that would allow such code to be easily written without xt equality, so one would have to replace it with code that is significantly more cumbersome to write.

EricBlake [r1461] 2025-07-30 12:56:14

Your implementation of defer@ is incorrect even if there is only a requirement for EXECUTE equivalence, but no requirement for xt equality. Consider

T{ 1 2 ' defer4 DEFER@ ' - ' defer4 DEFER! EXECUTE -> 3 }T

Thanks - that's a useful test that should be added, and indeed, is reason enough for me to add in the extra indirection.

I am trying to imagine an implementation where you work foremost with code/data pairs, and in most cases you just provide a pointer to that pair, but for defer@ you have a lookup table and it gets the xt from there. Now if you have two words with the same code/data pairs but a different xt (at the moment I have trouble thinking up an example; maybe synonyms, but why not have the same xt for them?), the lookup would only produce one of these xts, and defer@ could produce an xt that defer! did not store there. I am not aware of such an implementation, or anybody wanting to implement such an implementation; it also does not have any merits compared to existing implementation strategies that would make me want to hope that it complies with the standard.

Implementing a (useful!) SEE requires the ability to look up a word's name from either an xt or a code/data pair - SEE does not have to be fast, but I certainly find it more useful if it can disassemble code back into guesses for which words were compiled in the first place, rather than just outputting raw instruction values (at least the standard was wise enough to say that SEE produces implementation-defined output, so it is a quality-of-implementation rather than a conformance issue if SEE maps a code/data pair in compiled code to the wrong word out of multiple words that happen to share the same code/data implementation). But that is a lookup from xt -> nt (and could be done with a TRAVERSE-WORDLIST that checks if a given xt or code/data matches the code/data of each nt in succession), not a lookup from xt -> xt, and is not relevant to the speed of EXECUTE.

An alternative implementation of DEFER etc. for your implementation strategy is to have three cells: the two cells for code and data pointer (used for, e.g., execute) and a cell for the original xt (used for defer@).

Indeed, tracking 3 or more cells (one for DEFER@, and two or more as a trampoline for less indirection during EXECUTE) appears to be a viable strategy for optimization, in the vein of trading increased memory usage for runtime speed. Off-hand, I'm guessing that most uses of DEFER have more instances of execution (whether compiled or through EXECUTE) than DEFER@ or DEFER!, so the extra time taken in DEFER! to update more than one cell in order to make execution faster is worthwhile in a smart COMPILE,. But my takeaway is also that whatever COMPILE, and EXECUTE do with a defer word, the implementation must ensure that DEFER@ produces a point-in-time snapshot (this is the xt at the time you queried, and the semantics of that xt no longer depend on the future of the defer) while execution remains dynamic (the effects of running the defer must correspond to the xt that was most-recently installed, even if the installation occurred after the point where the deferred word was compiled). At any rate, in the short term I've gone with just one cell (the xt) and the extra indirection, as that was the simplest approach that meets the intended semantics, at which point DEFER@ returns a bit-identical copy of ' + if that is the xt originally stored by DEFER!.

In general, we have had to deal with the problem of how to represent code and data in a single cell from the very earliest Forth implementations, and I have published a paper about this problem at EuroForth 2024.

Very useful reading.

Relevance: Are there any programs that rely on xt equality? While I don't remember writing code that checks the result of defer@ or action-of for equality, I can imagine that some people might do it for good reasons. There is no standard xt= that would allow such code to be easily written without xt equality, so one would have to replace it with code that is significantly more cumbersome to write.

https://forth-standard.org/proposals/input-values-other-than-true-and-false#reply-1083 has a proposed reference implementation for [IF] that uses xt comparisons during [if]-decode. Admittedly, that is performing xt comparisons on the result of WORD and FIND, rather than on the result of DEFER@, but it goes a long ways towards demonstrating that Forth programs depend on being able to reliably compare xts for equality. And it may be possible to implement FIND in such a way that it is using DEFER@ on objects in each nt of the dictionary. It may indeed help if the standard either documents additional guarantees (once a word is defined, bitwise equality testing on the result of FIND and DEFER@ are reliable) or else standardizes XT= (which can then abstract away the implementation magic on why two equivalent xts would ever have different bit values).

One thought I had while playing with all this was whether it might be possible, on a system where a-addr is always a positive value, to encode the difference between interpretation and compilation semantics, and/or identify IMMEDIATE words, based on whether an xt is positive or negative. That is, there may be optimizations possible if FIND is allowed to report two pieces of information in one query: the xt being positive or negative conveying something orthogonal to whether the top stack cell was 1 or -1. In those scenarios, xt equivalence might be : xt= ( xt1 xt2 -- flag) abs swap abs = ;.

Closed

EricBlake [r1462] 2025-07-30 13:41:26

One thought I had while playing with all this was whether it might be possible, on a system where a-addr is always a positive value, to encode the difference between interpretation and compilation semantics, and/or identify IMMEDIATE words, based on whether an xt is positive or negative. That is, there may be optimizations possible if FIND is allowed to report two pieces of information in one query: the xt being positive or negative conveying something orthogonal to whether the top stack cell was 1 or -1. In those scenarios, xt equivalence might be : xt= ( xt1 xt2 -- flag) abs swap abs = ;.

That said, if I need a way to look up multiple pieces of information for a given parsed word, it's probably better to have my own internal functions or accessors on an nt on top of traverse-wordlist or search-wordlist (among others, taking c-addr u rather than a counted string, possibly taking a flag on whether to do case-sensitive or case-insensitive lookup), and then implement FIND as a thin shim around my internal functions. At that point, even if the internal function encodes multiple pieces of information in an internal xt (such as whether it is native code, a Forth thread with no locals, a Forth thread that needs local cleanup on exit, whether the xt has an associated nt or was created by :noname, ...), the version of the xt exposed via FIND and SEARCH-WORDLIST is scrubbed to mask out any transient bits that help the internal interpret/compile loop but do not impact how EXECUTE would use the result. There would need to be a strong reason why standardizing an XT= rather than living with existing = for xt comparison would be needed, and I don't think I've come up with any such reason.

AntonErtl [r1464] 2025-07-31 09:43:47

About "encode the difference between interpretation and compilation semantics". For immediate words, there is no difference. If a Forth system supports arbitrary combinations of interpretation and compilation semantics (e.g., with Gforth's interpet/compile:), the same xt can represent the interpretation semantics of one word and the compilation semantics of another word. E.g.:

: foo ." foo" ;
: bar ." bar" ;
' foo ' bar interpret/compile: foobar

Here the interpretation semantics of bar has the same behaviour as the compilation semantics of foobar. But actually, when you use s" foobar" find-name name>compile to access the compilation semantics of foobar, the result is the same as you get from ' bar ' execute.

So, I don't see a point in encoding "the difference between interpretation and compilation semantics". Any behaviour of a word can be both.

Reply New Version