Execution Flow

Below is a high-level walkthrough of how CrossBasic’s VM goes from your source text all the way to executing cross-platform compiled bytecode. Use this as a guide to understand each major phase, the key data structures, and how control actually flows through the system.

---
config:
    theme: dark
    layout: elk
---
flowchart TD
    %% Developer
    Developer["Developer\n(IDE Browser & CLI Shell)"]:::actor

    %% IDE Subsystem
    subgraph "IDE Subsystem"
        UI["Web UI - (www)"]:::ide
        Server["IDE Server - (server.cpp)"]:::ide
        PerfTest["Performance Tests - (ServerPerformanceTest)"]:::ide
        %%BuildIDE["IDE Build Scripts (build-64.bat/.sh)"]:::scripts
    end

    %% Core Engine
    subgraph "Core Engine"
        CBC["crossbasic CLI"]:::core
        VM["VM & Bytecode Compiler"]:::core
        XComp["xcompile CLI"]:::core
        %%BuildCB["build_crossbasic.bat/.sh"]:::scripts
        %%BuildXC["build_xcompile.bat/.sh"]:::scripts
    end

    %% Plugin Subsystem
    subgraph "Plugin Subsystem"
        Loader["Plugin Loader"]:::core
        Plugins["Plugins"]:::plugin
    end

    %% Interop Layer
    subgraph "Interop Layer"
        FFI["libffi"]:::interop
        CURL["libcurl"]:::interop
    end

    %% Build & Deploy
    subgraph "Build & Deploy"
        Scripts["Build System & Scripts - (build_plugins.bat/.sh, runallscripts.bat/.sh)"]:::scripts
    end

    %% Interoperability Templates
    subgraph "Interoperability Templates"
        Templates["Plugin Language Templates - (Interoperability/Plugin Language Templates)"]:::interop
    end

    %% Xojo Integration
    subgraph "3rd Party Integration - ie) Xojo, Python, NodeJS, etc."
        XojoRoot["Xojo Integration Demo"]:::ide
    end

    %% External Services
    subgraph "External Services"
        LLM["LLM APIs"]:::external
        FS["Filesystem"]:::external
        NET["Network"]:::external
    end

    %% Relationships
    Developer -->|HTTP Requests| Server
    Server -->|serves| UI
    Server -->|invokes compile| CBC
    Developer -->|CLI Commands| CBC
    Developer -->|CLI Commands| XComp

    CBC -->|compiles bytecode| VM
    CBC -->|loads via libffi| Loader
    Loader -->|discovers| Plugins
    Plugins -->|uses FFI| FFI
    Plugins -->|calls external APIs| CURL
    Plugins -->|HTTP calls| NET
    Plugins -->|uses external| LLM
    VM -->|I/O| FS

    CBC -->|native handoff| XComp

    Scripts -->|build core| CBC
    Scripts -->|build plugins| Plugins

    Templates -->|generate bindings for| Plugins

    XojoRoot -->|embeds library| CBC

    %% Click Events
    click UI "https://github.com/simulanics/crossbasic/tree/main/CrossBasic-IDE/www/"
    click Server "https://github.com/simulanics/crossbasic/blob/main/CrossBasic-IDE/server.cpp"
    click PerfTest "https://github.com/simulanics/crossbasic/tree/main/CrossBasic-IDE/ServerPerformanceTest/"
    click BuildIDE "https://github.com/simulanics/crossbasic/blob/main/CrossBasic-IDE/build-64.bat"
    click CBC "https://github.com/simulanics/crossbasic/blob/main/crossbasic.cpp"
    click CBC "https://github.com/simulanics/crossbasic/blob/main/crossbasic.rc"
    click BuildCB "https://github.com/simulanics/crossbasic/blob/main/build_crossbasic.bat"
    click BuildCB "https://github.com/simulanics/crossbasic/blob/main/build_crossbasic.sh"
    click XComp "https://github.com/simulanics/crossbasic/blob/main/xcompile.cpp"
    click XComp "https://github.com/simulanics/crossbasic/blob/main/xcompile.rc"
    click BuildXC "https://github.com/simulanics/crossbasic/blob/main/build_xcompile.bat"
    click BuildXC "https://github.com/simulanics/crossbasic/blob/main/build_xcompile.sh"
    click Scripts "https://github.com/simulanics/crossbasic/blob/main/build_plugins.bat"
    click Scripts "https://github.com/simulanics/crossbasic/blob/main/build_plugins.sh"
    click Scripts "https://github.com/simulanics/crossbasic/blob/main/runallscripts.bat"
    click Scripts "https://github.com/simulanics/crossbasic/blob/main/runallscripts.sh"
    click Plugins "https://github.com/simulanics/crossbasic/tree/main/Plugins/"
    click Templates "https://github.com/simulanics/crossbasic/tree/main/Interoperability/Plugin Language Templates/"
    click XojoRoot "https://github.com/simulanics/crossbasic/tree/main/Xojo Integration/"

    %% Styles
    classDef actor fill:#292929,stroke:#333,stroke-width:1px
    classDef ide fill:#571f1f,stroke:#f66,stroke-width:1px
    classDef core fill:#393961,stroke:#66f,stroke-width:1px
    classDef plugin fill:#224222,stroke:#6c6,stroke-width:1px
    classDef interop fill:#572057,stroke:#c6c,stroke-width:1px
    classDef scripts fill:#6e5d5d,stroke:#999,stroke-width:1px
    classDef external fill:#005f6b,stroke:#cc0,stroke-width:1px

1. Program Startup¶

1. main()

2. Records a start time (startTime) for built-ins like ticks and microseconds.

3. Attempts to embed or retrieve pre-compiled bytecode via retrieveData().

4. If no embedded bytecode is found, reads your .xs source file.

5. Environment Initialization

InitializeEnvironment(vm);

• Creates the global Environment (a case-insensitive map of names → Value).
• Defines all built-in constants (pi, eol,…) and built-in functions (print, sleep, array, math, string, array utilities, plugin loaders, etc.).
• Loads any plugin DLLs/shared-objects from libs/, registering their functions and classes into vm.environment.

2. Front-End: From Text to AST¶

1. Preprocessing

source = preprocessSource(rawText);

• Strips comments (//, '), handles line-continuations (_), and preserves string literals.

• Lexing

Lexer lexer(source);
auto tokens = lexer.scanTokens();

• Scans character by character, producing a vector<Token> with types like NUMBER, IDENTIFIER, PLUS, FUNCTION, ENUM, etc.

• Parsing

Parser parser(tokens);
auto ast = parser.parse();  // vector<shared_ptr<Stmt>>

• Builds an AST of Stmt nodes (functions, classes, if/while/for, enums, modules, Goto/Label, etc.) and Expr nodes (binary, call, property, literal, new, etc.).
• Supports Select Case, Declare for external APIs, Extends for module-level extensions, ASSIGNS-style calls, labels & gotos, circular references, etc.

3. Compilation: AST → Bytecode¶

1. Compiler Setup

Compiler compiler(vm);
compiler.compile(ast);

• All code emits into vm.mainChunk: CodeChunk { code: vector<int>, constants: vector<Value> }.

• Emitting Instructions

• Constants (literals, compiled ObjFunction closures, compiled ObjModule, ObjEnum) are collected into chunk.constants.

• OpCodes and their operands (e.g. OP_CONSTANT <idx>, OP_ADD, OP_CALL <arity>, OP_JUMP <offset>, …) are emitted into chunk.code.

• Label & Goto Fix-ups

• Forward GotoStmt emit a placeholder OP_JUMP 0.

• After all statements are compiled, the compiler patches each placeholder with the final target offset.

• Extension & Module Exports

• For Extends and module-scoped Public members, the compiler also registers wrappers in vm.extensionMethods[type][methodName] or in a per-module publicMembers map.

4. Execution: The Fetch–Decode–Execute Loop¶

All execution happens in:

Value runVM(VM& vm, const ObjFunction::CodeChunk& chunk);

4.1 Setup¶

• int ip = 0; — instruction pointer into chunk.code.
• vm.stack: vector<Value> — the single evaluation stack.
• vm.environment points at the current Environment (starts at vm.globals).

4.2 Loop¶

while (ip < chunk.code.size()) {
    processPendingCallbacks();        // handle any queued plugin/event callbacks

    int instruction = chunk.code[ip++];
    switch (instruction) {
      case OP_CONSTANT:
        // read index, push chunk.constants[index]
      case OP_ADD:
        // pop two values, add (int/double/string), push result
      case OP_CALL:
        // pop <arity> args, pop callee Value, dispatch:
        //  • BuiltinFn → invoke directly  
        //  • ObjFunction → new Environment, bind params, recursive runVM  
        //  • overload set → resolve by arg count  
        //  • ObjBoundMethod → handle methods, plugin or scripted  
        //  • ObjArray → get/set index  
        //  • string literal → built-ins like print/val/str/ticks  
      case OP_OPTIONAL_CALL:
        // like OP_CALL but treats NIL callee as a no-op
      case OP_RETURN:
        // pop return value and return from runVM
      case OP_NEW:
        // pop class, instantiate ObjInstance (plugin or builtin), push it
      case OP_GET_PROPERTY:
      case OP_SET_PROPERTY:
        // dynamic property lookup or setter on instances, modules, enums, extensions
      case OP_JUMP_IF_FALSE:
      case OP_JUMP:
        // conditional or unconditional branch
      … etc …
    }
    // Optional debug log prints stack contents after each instruction
}

• processPendingCallbacks() drains a thread-safe queue of CallbackRequest and calls back into script via invokeScriptCallback(), ensuring plugin events run on the VM thread with proper locking.

4.3 Stack & Environments¶

• Arguments for calls are pushed before invoking OP_CALL.

• On scripted calls, we:

• Save vm.environment and stack depth.

• Create a new child Environment (parameters + locals).
• runVM(...) recursively on the function’s bytecode.
• Restore the previous environment and trim the stack back to its saved depth, then push the single return Value.

5. Memory Management & ARC¶

• Every heap-allocated object (ObjFunction, ObjClass, ObjInstance, ObjArray, ObjModule, ObjEnum) lives inside a std::shared_ptr<…> held in a Value.
• ARC (via shared_ptr) increments on push/assignment, decrements on pop/reassignment, and destroys objects as soon as their count reaches zero—except in circular graphs, where you must use weak references or manual breaks.

Key Takeaways¶

• Single Stack VM: All intermediate values live on one vector stack.
• Recursive C++ Calls: Each scripted function invokes runVM(...) in a new C++ call frame plus a new Environment.
• Deterministic ARC: No GC pauses—objects free immediately when unreferenced.
• Extensible: Built-in, user-scripted, and plugin code all interoperate via BuiltinFn lambdas and libffi callbacks (AddressOf → scriptCallbackTrampoline).
• Modular: You can Module … End Module to scope definitions and expose only Public members as a proper ObjModule.

With this flow, CrossBasic turns your .xs text into fast, safe, and extensible bytecode running on a minimal C++ stack-machine core.