Author:

Nathan Hesterman

Change Log:

2026-09-01 — Initial proposal.

Problem:

Forth source has traditionally been treated as an undifferentiated stream of characters. The text interpreter parses that stream, normally using whitespace, and determines the meaning of the resulting tokens from dictionary contents, interpreter state, parsing words, and other context. Traditional Forth must therefore recover semantic distinctions from an otherwise untyped character stream. These distinctions include such things as: • ordinary Forth source • compile semantics • interpretation semantics • postpone semantics • definitions • data • text • comments

This results in mechanisms such as parsing words, delimiters, escaping conventions, STATE, immediate words, and words such as POSTPONE.

For example, text requires some delimiter: S" This is text" The delimiter consequently cannot occur freely in the payload without an escaping or doubling convention.

Similarly: POSTPONE foo uses one source word to specify how another source word is to be treated. The semantic information is not attached to foo in the source; it must be established by the operation of POSTPONE.

colorForth demonstrated an alternative source model in which semantic information can be part of the source representation itself. Color was one means of presenting that information to the programmer. It was not necessary to the underlying concept.

There is also a broader purpose behind this proposal.

Forth has continued to evolve outside the family of systems normally described as ANS Forth. Systems such as colorForth, etherForth, r3, and other descendants should be regarded as experiments in Forth language design, not merely as incompatible dialects outside the scope of standardization. The continued existence of such systems gives the Forth community something valuable: decades of experience with alternative solutions to problems that traditional Forth solves differently.

A future Forth standard should consider that experience.

This does not mean incorporating another Forth wholesale, nor does it mean abandoning compatibility with existing standard programs. Instead, later Forths should be examined for abstractions that have proven simpler or more expressive than mechanisms inherited by standard Forth. Where such an abstraction can coexist with existing Forth, the standard should consider providing a path toward it. Semantic source is one example.

Traditional Forth reconstructs much of a program's intended meaning from an undifferentiated character stream. colorForth demonstrated that some of this information can instead accompany the source. etherForth retained the principle using a different encoding, and r3 demonstrates another way of expressing semantic distinctions in source. The particular representation is less important than the common lesson.

Backward compatibility and evolution need not be opposing goals. An existing Forth source file containing no semantic information can continue to mean exactly what it means today. New source can make use of richer semantics when an implementation supports them. Over time, experience can determine whether those semantics provide better abstractions for facilities presently expressed through parsing words, delimiters, compiler state, or other mechanisms. The standard need not freeze Forth at any particular historical implementation model in order to preserve programs written to that model.

A standard can preserve the past while still providing a path toward a smaller, simpler Forth.

Solution:

Provide a minimal, backward-compatible mechanism by which Forth source can carry semantic information explicitly.

A NUL byte (00) in the source indicates that the following byte identifies a semantic source mode: 00 <mode> The selected semantic mode persists until another semantic mode is encountered.

All source containing no semantic mode markers remains conventional Forth source and is processed exactly as before. The intent is not to standardize colors, an editor, or a particular source-code presentation. It is to standardize a means by which source can say what subsequent source means.

For example, instead of requiring a textual delimiter: S" This is text"

semantic source could express: <text mode> This is arbitrary text containing " ; or any other Forth characters <ordinary mode>

There is no textual terminator to escape. The change of semantic mode terminates the text.

Similarly: POSTPONE foo

could conceptually be represented by selecting postpone semantics and supplying foo. The semantic information belongs to the source item rather than being supplied by another word that parses it.

The semantic mode persists until another mode marker occurs. It is therefore unnecessary to attach a tag to every word or character. This retains the compactness that motivated tagged source systems such as colorForth while allowing conventional source and semantic source to coexist.

Backward compatibility

Backward compatibility is a primary objective. Existing Forth source contains no semantic mode markers and requires no modification. A semantic-source-aware system encountering no NUL bytes behaves as an ordinary Forth system. Thus semantic source is an extension of the existing source model rather than a replacement for traditional Forth source. UTF-8 is also unaffected because NUL does not occur as part of the UTF-8 representation of any character other than U+0000 itself. If literal NUL is required in semantic payload, a semantic mode or escape convention can be defined for that purpose.

Presentation is not semantics

This proposal deliberately does not specify how semantic modes are displayed or entered.

An editor might represent semantic modes using color, bold, italic, underline, strike-through, font differences, textual prefixes, symbols, auditory cues, or any other suitable presentation. A programmer unable to distinguish colors must not lose semantic information. Likewise, an editor might assign function keys or other input methods to semantic modes. These are editor concerns and should not be part of the language standard.

The source contains the semantic identifier. Its presentation is implementation-defined.

Relationship to colorForth and etherForth

colorForth and later etherForth provide prior art for semantically tagged Forth source. The important idea being adopted here is not literal color. It is that source elements can carry semantic information which would otherwise have to be reconstructed by the interpreter. This proposal attempts to separate that abstraction from the particular editors, visual representations, character encodings, and hardware environments in which those systems implemented it.

Relationship to Recognizers

Recognizer proposals improve the ability of a Forth system to determine the semantics of parsed input.

Semantic source addresses an earlier stage of the problem. A recognizer begins with source text and attempts to determine what it means. Semantic source permits the producer of the source to state what it means. These mechanisms need not conflict.

Traditional source can continue to be parsed and recognized in the usual manner, while semantic source can supply information directly. This suggests a possible longer-term abstraction in which conventional text parsing, recognizers, colorForth-like source, and other source representations all produce semantic source items for the interpreter/compiler.

Initial semantic modes

This proposal does not attempt to assign a final set of standardized semantic mode numbers. That should follow agreement on the source mechanism itself.

Candidate modes include: ordinary interpret compile postpone text comment define data

Experience with colorForth, etherForth, r3, and existing standard Forth should be examined before selecting the smallest useful standardized set. Implementations should also have room for implementation-defined modes.

Questions remaining for discussion include:

1 Is a persistent semantic source mode an appropriate abstraction, or should semantic tags apply to individual source items?

2 Is NUL followed by a mode byte an appropriate backward-compatible encoding?

3 Which semantic distinctions should initially be standardized?

4 Should ordinary mode explicitly invoke the existing standard text interpreter?

5 How should literal NUL be represented when required?

6 Can the existing recognizer work provide the interpreter/compiler interface to which semantic source modes are ultimately translated?

7 Should the standard define semantic source independently of its serialized representation, with NUL mode being only one standard interchange encoding?

Typical use: (Optional)

A conventional Forth source file remains unchanged: : foo 1 2 + ;

A semantic-source-aware system processes this exactly as existing Forth source because no semantic mode markers occur. A source stream can enter a semantic mode by including: NUL <semantic-mode>

and remains in that mode until another: NUL <semantic-mode>

is encountered.

Conceptually, a mixture of conventional and semantic source might be: ordinary Forth source NUL <text> arbitrary text including " ; and other Forth characters NUL <ordinary> ordinary Forth source

The visual representation of <text> and <ordinary> is not prescribed. An editor may represent them as colors, font attributes, symbols, or other cues. The stored semantic information, rather than its presentation, determines the meaning.

Proposal:

Add the concept of a semantic source mode to Forth source processing.

1 A NUL byte encountered in semantic-source-aware input indicates that the following byte is a semantic source-mode identifier.

2 A semantic source mode remains in effect until another semantic source-mode identifier is encountered.

3 Source containing no semantic source-mode identifiers retains existing Forth source semantics.

4 The visual, textual, auditory, or other presentation of semantic source modes is implementation-defined and has no effect on program semantics.

5 The initial standardized set and numerical assignment of semantic modes remain to be determined. Candidate modes include ordinary, interpret, compile, postpone, text, comment, define, and data.

6 Implementations may provide implementation-defined semantic modes in addition to standardized modes.

7 A means shall be provided to represent a literal NUL when required within semantic payload.

The proposed mechanism can therefore be summarized as: NUL <semantic-mode>

which changes the semantic interpretation of subsequent source. Without this sequence, Forth source behaves exactly as it does today. The proposal does not standardize color. It proposes standardizing the idea demonstrated by colorForth: semantics can be part of the source.

Reference implementation:

No reference implementation is provided at this stage.

Implementation requires access to the mechanism by which an implementation obtains and interprets source input. This mechanism is system-dependent in existing Forth systems. A reference implementation should be developed after the semantic-source interface and initial semantic modes have been agreed upon.

A prototype implementation should demonstrate at minimum: • conventional Forth source executing without modification; • recognition of NUL <mode>; • persistent switching between ordinary and semantic source modes; • semantic text containing characters that would otherwise serve as delimiters; • restoration of ordinary Forth source processing; • at least one semantic mode affecting interpretation or compilation semantics.

Such an implementation would also provide experience for determining whether the proposed encoding belongs in the standard itself or should instead be one serialization of a more abstract semantic-source interface.

Testing: (Optional)

Tests should establish that:

1 Existing Forth source containing no NUL bytes retains its existing behavior.

2 A NUL followed by a valid semantic mode changes the current semantic source mode.

3 The selected semantic mode persists across subsequent source until another mode marker is encountered.

4 Returning to ordinary mode restores conventional Forth source processing.

5 Text or other payload may contain characters normally used as Forth delimiters without those characters terminating the semantic region.

6 UTF-8 input is not confused with semantic mode markers.

7 The defined literal-NUL mechanism permits U+0000 or a NUL payload byte to be represented when required.

8 Invalid or unsupported semantic mode identifiers produce defined implementation behavior.

Detailed Appendix F test cases should be supplied after the standardized semantic modes and their required behavior have been determined.