Skip to content

Commit 456458c

Browse files
authored
fixes (#14 and more) & compile-time literal casting & improved error reporting, robustness & test coverage (#12)
* 1 * wip * Update README.RU.md * update v version * update version command * Update README.RU.md * Update CHANGELOG.md * Fix None literal in object variable | update v version * Update V setup action to v1.4 in workflows Changed the V setup GitHub Action from vlang/setup-v@v1 to vlang/setup-v@v1.4 in both CI and manual release workflows for improved compatibility and stability. * add sexlab sources * small rename file iequip_test -> proj_iequip_test * add test project sexlab * Move iEquip scripts to iEquipSources directory Renamed all iEquip related script and README files from modules/tests/iEquip to modules/tests/iEquipSources for improved organization and clarity. * Add MantellaSpellSources scripts and assets Introduces the MantellaSpellSources module with Papyrus scripts for conversation, actions, constants, equipment description, dialogue handling, and related functionality. Includes MIT license and README for documentation. * Add OStim Papyrus source scripts and license Added OStimSources directory containing Papyrus scripts and supporting files for OStim, including OAiScript.psc, OBarsScript.psc, and others. Also included the GNU GPL v3 license and a README for documentation. * Add PapyrusUtil source scripts for testing Added PapyrusUtil, JsonUtil, MiscUtil, ObjectUtil, StorageUtil, and ActorUtil Papyrus scripts, along with a README referencing the source. These files provide utility functions for Skyrim mod development and are intended for use in the test modules. * Add ModEvent and NetImmerse script dependencies Introduces ModEvent.psc and NetImmerse.psc to the psc_deps test module. These scripts provide extended mod event handling and node manipulation functions for testing and integration purposes. * Rename and update iEquip test file Renamed proj_iequip_test.v to project_iequip_test.v and updated the test function and input directory to reflect new naming and source location. * Rename and update SexLab test function Renamed test file and function from 'proj_sexlab_test' to 'project_sexlab_test' for improved clarity and consistency. * Add Mantella spell project test Introduces a test for compiling the Mantella spell project, verifying input and header directories, and ensuring output directory creation. Uses builder.compile with specific preferences for compilation. * Add test for OStim project compilation Introduces a new test in project_ostim_test.v to verify compilation of OStim sources with required dependencies and output directory setup. * Add UIExtensionsSources scripts for testing Added multiple Papyrus script files and a README to modules/tests/UIExtensionsSources, sourced from https://www.nexusmods.com/skyrim/mods/57046. These scripts provide various UI extension menu implementations for Skyrim mod testing. * Add RaceMenu Papyrus source scripts for testing Added RaceMenu Papyrus scripts and README to modules/tests/RaceMenuSources for testing purposes. These files are sourced from https://www.nexusmods.com/skyrim/mods/29624. * Add Papyrus script stubs for SKSE dependencies Added new Papyrus script files for Potion, SKSE, Scene, SoundDescriptor, TopicInfo, Weather, and WorldSpace to support SKSE-dependent features in tests. Also added GetEquippedArmorInSlot to Actor.psc for armor slot queries. * Add NiOverride and ArmorAddon Papyrus scripts Introduces NiOverride.psc and ArmorAddon.psc to the NiOverrideSources test module. These scripts provide native Papyrus interfaces for manipulating armor, weapon, skin, and node overrides, as well as body morphs and dye functions for Skyrim modding. * Add MCM Helper source scripts for testing Introduces MCM Helper Papyrus scripts and related files under modules/tests/MCMHelperSources for test purposes. Includes LICENSE, README, and all core scripts from the MCM Helper project. * Add MasterOfDisguise source scripts and docs Added LICENSE, README, and Papyrus source scripts for MasterOfDisguise from https://github.com/fireundubh/MasterOfDisguiseSSE to modules/tests/MasterOfDisguiseSources for testing and reference. * Add LibTurtleClub source files and license Added LICENSE, LibTurtleClub.psc script, and README.md to modules/tests/LibTurtleClubSources. These files provide the LibTurtleClub Papyrus script and documentation, licensed under MIT. * Add LibFireSources test module Introduces the LibFireSources test module with LICENSE, Papyrus script (LibFire.psc), and README. These files provide utility functions and documentation for testing, sourced from https://github.com/fireundubh/LibFire. * Add LibMathf.psc with math utility functions Introduces LibMathf.psc script containing various global native math functions such as Abs, Clamp, Lerp, Pow, and trigonometric operations. These utilities provide common mathematical operations for use in other scripts. * Add JContainers Papyrus API source files Added initial Papyrus script sources for JContainers API, including JArray, JAtomic, JContainers, JDB, JFormDB, JFormMap, JIntMap, JLua, JMap, JString, and JValue. These files provide native function definitions and documentation for container, atomic, database, map, string, and value management in Skyrim mod scripting. * Remove iEquipSources test scripts Deleted LibTurtleClub.psc, SKI_ConfigBase.psc, SKI_ConfigManager.psc, and SKI_QuestBase.psc from modules/tests/iEquipSources. These files are no longer needed for the test suite. * Add ConsoleUtil sources for testing Added ConsoleUtil.psc script, license, and README to modules/tests/ConsoleUtilSources. These files are sourced from https://github.com/darkconsole/ConsoleUtilSSE and will be used for testing purposes. * Consolidate project test files into projects_test.v Removed individual test files for iEquip, MantellaSpell, OStim, and SexLab, and merged their test logic into a single projects_test.v file. This centralizes all project test cases for easier maintenance and extensibility. * Add debug print for compile command execution Prints the command being executed in compile_original for easier debugging and visibility into the build process. * wip * wip * wip * Add OSA Papyrus source scripts for testing Added multiple Papyrus script files and a README under modules/tests/OSASources, sourced from the OSA Skyrim SE project. These scripts provide utility functions and core framework logic for OSA, including actor utilities, console interaction, codepage conversion, and scene management. * Remove unused SexLab and iEquip test sources Deleted SexLabSources and iEquipSources Papyrus script files from the test modules, as well as their references. Added a new test for the OSA project and updated the OStim test to include the OSA sources. This streamlines the test suite and removes dependencies on unused scripts. * Add OStimNG Papyrus source scripts for testing Introduces the OStimNGSources directory containing Papyrus source scripts and a GPLv3 license for testing purposes. These scripts provide utility functions and APIs for actor management, animation selection, and scene handling in the OStimNG framework. * Refactor SkyUI SDK sources by version and add v5.2 Removed old SkyuiSDKSources directories and README files, and reorganized source scripts into versioned folders: SkyuiSDKSources_v5.1 and SkyuiSDKSources_v5.2. Added new scripts and README for v5.2, including SKI_ConfigBase.psc, SKI_QuestBase.psc, SKI_WidgetBase.psc, and SKI_WidgetManager.psc. This improves clarity and maintainability by separating SDK sources by version. * Add Camera.psc and update Game.psc for camera functions Introduces a new Camera.psc script providing camera state and FOV functions. Updates Game.psc to use Camera.psc for camera-related functions, adds light mod and plugin functions, and comments out the deprecated GetNthModDependency. * Update test projects for multiple SkyUI SDK versions Split SkyUI SDK tests into v5.1 and v5.2, updating test_project_skyui_sdk and related test cases to use version-specific source directories. Added new dependencies (LibMathfSources, LibFireSources, NiOverrideSources, UIExtensionsSources) to relevant tests and adjusted header_dirs accordingly. Refactored OStim test to use OStimNGSources and updated dependency checks and comments for clarity. * Remove debug output from ast_test.v Deleted unnecessary eprintln(errs) statements from test_error_state_fn_with_default_arg and test_error_msg functions to clean up test output. * Print header dirs only outside test builds Wrapped the println statement for header directories with a compile-time check to prevent output during test builds. * Update V compiler version and improve temp var handling Bump V compiler version to weekly.2025.48 in CI, release workflow, and documentation. Refactor gen_call in gen_expr.v to defer freeing temporary variables until after use, improving memory management. * Refactor test project setup with helper functions Introduces get_source_dir and get_output_dir helper functions to reduce code duplication and improve readability in test project setup. All test functions now use these helpers for directory and file validation, making the code more maintainable. * Improve test output handling and directory management Replaced os.real_path with os.abs_path and switched to os.mkdir_all for more robust directory creation in test helpers. Set output_mode to .silent in all test project configurations to suppress unnecessary output. Updated builder to use b.print for header dir logging and improved timer output formatting. * Set output_mode to silent in test preferences Updated test preference configurations in multiple test files to set output_mode to .silent, ensuring test output is suppressed for cleaner test runs. * Refactor test project setup with get_prefs helper Introduced a get_prefs helper function to reduce duplication in test project setup. All test functions now use get_prefs to construct pref.Preferences, improving readability and maintainability. * Refactor test project source path usage Introduced constants for source file paths to reduce repetition and improve readability in test functions. All test functions now use these constants instead of repeated get_source_dir calls. * Replace println with b.print and improve error output Updated modules/builder/original.v to use b.print instead of println for consistent logging. Enhanced error reporting in modules/builder/pex.v by printing errors and backtrace in test mode. Minor error message formatting changes in compiler.v for clarity. * Update output directory names in test functions Changed output directory arguments in test_project_skyrim_deps_sources, test_project_skyui_sdk_51, and test_project_skyui_sdk_52 to use more specific names, improving clarity and consistency in test output locations. * Improve type validation and error handling in checker Added checks for undefined types in function return types and return statements. Updated compile-time casting for literals in properties and default parameters. Enhanced error messages in tests and added missing script dependency for tests. Commented out placeholder asserts in autocast logic for better stability. * Add README for Original Compiler directory Introduces a README.md file to document the purpose of the 'Original Compiler' directory, clarifying that it contains the original Papyrus compiler files from the Creation Kit. * Fix indentation in ast_test.v preferences constant Corrected the indentation of the prefs constant initialization for improved code readability and consistency. * Prefix output dir with __project_test_ in get_output_dir Updated get_output_dir to prefix the compiled directory name with '__project_test_' to avoid potential naming conflicts or improve clarity in test output directories. * Update CHANGELOG Added details about fixes for default parameter checks, new tests with third-party script compilation, and clarified previous entries. * Validate script name matches file and uniqueness Add checks in script_decl to ensure a .psc file's script name matches the file base name (case-insensitive) and to error if a script with the same name is already defined. Uses os.base/all_before_last for filename extraction and reports errors with node.name_pos. * Add AGENTS.md system prompt Add AGENTS.md: a comprehensive system prompt and contributor guide for the Papyrus Compiler project. Documents project layout, compilation pipeline, key data structures, V language conventions, build/run/test commands, testing and mandatory rules, module dependency graph, implementation details, and common how-to tasks — intended to guide contributors and AI assistants working on the compiler. * Improve cast assertion message Enhance the assertion in modules/papyrus/checker/checker.v to include a descriptive error message when a cast is invalid. The message prints the source and target type names via get_type_name to make debugging invalid cast attempts easier. * add RequiemSources * add CampfireSources * Add try_cast_to_type and use safe casts Introduce Checker.try_cast_to_type to validate casts (using can_cast/can_autocast) and report a clear error instead of unconditionally performing casts. Replace direct cast_to_type calls in infix expression checking with try_cast_to_type so casts are only applied on success. Also translate a couple of comments to English and add a test (none + 1) that expects a cast error from None to Int. * Update CHANGELOG with new features and fixes Document new features and fixes: add `version` command; allow string literals as default values for typed properties and parameters (auto-cast to declared type). Add compiler validations and error reporting for missing scripts (extends/variable types), duplicate script names across source folders, mismatched Scriptname vs filename, and type-compatibility of default parameters. Fix handling of `None` as default values and crashes when `None` appears in expressions; validate calls with default parameters inside State blocks. Cleanup and reorganize changelog entries. * Add changelog rules for user-visible changes Add guideline #8 to AGENTS.md requiring updates to the CHANGELOG.md 'Next Release' section for any user-visible behavior, feature, or bugfix. Entries must be written in English, aimed at compiler users (what they will observe), and must not mention internal implementation details, module/function names, or CI/build-only changes. Includes examples of good and bad changelog entries. * Prevent placeholder type crashes; support comments Add guards in the type checker to treat placeholder/undefined types as non-castable and emit undefined-type errors instead of crashing. Update try_cast_to_type to report undefined types for both source and target. Add Parser.skip_comments and call it in expr, fn arg parsing and call argument handling so comments inside parenthesized expressions and argument lists are accepted. Update tests and CHANGELOG to reflect the fixes and parser behavior improvements. * move papyrus sources to separate repo Add .gitmodules to configure repository submodules and remove a large number of files under modules/tests (CampfireSources, ConsoleUtilSources, FNISSources, JContainersSources, etc.). These deletions move bulky test/source modules out of the main repo and into submodules (or external management) to reduce repository size and centralize versioning of those component modules. * Update checkout action to v4; enable submodules Bump actions/checkout to v4 and add `with: submodules: recursive` in CI and manual-release workflows to ensure repository submodules are checked out. Changes applied to .github/workflows/ci.yml and .github/workflows/manual-release.yml. * Update header_dir path and translate comments Point header_dir to the relocated psc_deps under modules/tests/sources and clean up the inline comments in the test. The Russian comments were translated to English and simplified (clarifying that some types may be placeholders but aren't required), ensuring the test uses the new directory layout and has clearer documentation. * Update test source path for selective headers Adjust test_selective_headers_loading to use the relocated PSC source file under modules/tests/sources/psc/TestSelectiveLoading.psc (was modules/tests/psc/TestSelectiveLoading.psc). This aligns the test with the repository restructure of PSC sources so the test loads the correct file path. * Fix path to PSC test file in selective headers test Update test src_file to point to modules/tests/psc/TestSelectiveLoading.psc instead of the incorrect modules/tests/sources/psc/... path. This corrects the file location used by the test so the PSC source is found during test runs. * Remove Campfire project and update Requiem deps Delete the unused Campfire source reference and remove the commented-out Campfire test. Update the Requiem test prefs to include the USSEP and SkyUI SDK 5.1 dependencies so Requiem builds with those integrations. * Handle alternate test source/header paths Make tests resilient to differing repo layouts by adding fallback path checks. In projects_test.v, get_source_dir now checks the new modules/tests/sources/<dir> layout, special-cases psc_deps, and falls back to modules/tests/<dir> before failing. In selective_headers_loading_test.v, header_dir resolution tries modules/tests/sources/psc_deps first then modules/tests/psc_deps, validating that Form.psc exists. These changes improve backward compatibility with submodule or layout changes. * update test sources submodule * header image to READMEs Move the Discord invite badge to the top of both README.md and README.RU.md, add a project header image (docs/image.png), and remove the duplicate badge placement. This improves visibility and adds a visual header for the project. * Enable debug (-g) in V build step Add the -g flag to the V compiler invocation in .github/workflows/manual-release.yml so the built binary includes debug information. This updates the Build Project step only; output path (bin/papyrus) and other flags remain unchanged. * Add v0.0.4 changelog and improve errors Insert a ## V 0.0.4 section into CHANGELOG.md and update release notes. Documented improvements to error handling: clearer and more consistent error messages (e.g., 'undefined identifier') and structured internal compiler error diagnostics that include version info, a stack trace, and reporting instructions. Also cleaned up placeholder ellipses in the file. * Add -g flag to build command in READMEs Update README.md and README.RU.md to include the -g flag in the v compiler invocation (v -o "bin\papyrus.exe" -prod -g -gc none compiler.v). This ensures debug information is generated during compilation.
1 parent dcae26e commit 456458c

123 files changed

Lines changed: 2197 additions & 10731 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/ci.yml

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,12 +14,14 @@ jobs:
1414
runs-on: ${{ matrix.os }}
1515
steps:
1616
- name: Install V
17-
uses: vlang/setup-v@v1
17+
uses: vlang/setup-v@v1.4
1818
with:
19-
version: weekly.2025.09
19+
version: weekly.2025.48
2020

2121
- name: Checkout ${{ github.event.repository.name }}
22-
uses: actions/checkout@v2
22+
uses: actions/checkout@v4
23+
with:
24+
submodules: recursive
2325

2426
- name: Build ${{ github.event.repository.name }}
2527
run: v -prod -gc none compiler.v

.github/workflows/manual-release.yml

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -41,15 +41,17 @@ jobs:
4141

4242
steps:
4343
- name: Checkout Code
44-
uses: actions/checkout@v2
44+
uses: actions/checkout@v4
45+
with:
46+
submodules: recursive
4547

4648
- name: Install V
47-
uses: vlang/setup-v@v1
49+
uses: vlang/setup-v@v1.4
4850
with:
49-
version: weekly.2025.09
51+
version: weekly.2025.48
5052

5153
- name: Build Project
52-
run: v -prod -gc none -o "bin/papyrus" compiler.v
54+
run: v -prod -g -gc none -o "bin/papyrus" compiler.v
5355

5456
- name: Get the date
5557
id: date

.gitmodules

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[submodule "modules/tests/sources"]
2+
path = modules/tests/sources
3+
url = https://github.com/russo-2025/papyrus-compiler-test-sources

AGENTS.md

Lines changed: 271 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,271 @@
1+
# AGENTS.md — System Prompt for AI Assistants
2+
3+
You are working on **Papyrus Compiler** — an open-source compiler for the Papyrus scripting language (Skyrim SE/AE). The compiler is written in the **V programming language** (vlang.io). It compiles `.psc` (Papyrus Source Code) files into `.pex` (Papyrus Executable) binary files — the same format as Bethesda's official Creation Kit compiler.
4+
5+
---
6+
7+
## Project Structure
8+
9+
```
10+
compiler.v — Entry point: CLI dispatch (compile, read, disassembly, dump, help, version)
11+
fast.v — Benchmarking harness for performance tracking across commits
12+
v.mod — V module metadata (version, dependencies)
13+
run.bat — Quick-launch script for compilation
14+
15+
modules/
16+
├── pref/ — CLI argument parsing, Preferences struct
17+
├── papyrus/
18+
│ ├── token/ — Token kinds (enum Kind), Position struct
19+
│ ├── scanner/ — Lexer: .psc text → token stream
20+
│ ├── ast/ — AST nodes, type system, symbol table (Table), scopes
21+
│ ├── parser/ — Recursive-descent parser: tokens → AST
22+
│ ├── checker/ — Semantic analysis: type checking, cast validation, scope resolution
23+
│ ├── errors/ — Error/Warning structs, predefined error message constants
24+
│ └── util/ — Helpers: BOM handling, char classification, error formatting
25+
├── pex/ — PEX binary format: data structures, reader, writer, opcodes, dump
26+
├── gen/
27+
│ ├── gen_pex/ — Code generator: AST → PEX bytecode instructions
28+
│ └── ts_binding/ — TypeScript binding generator (secondary feature)
29+
├── builder/ — Orchestrator: drives the full compile pipeline, caching, stats
30+
└── tests/ — Test suite (V's built-in test framework)
31+
```
32+
33+
### Directories that should NOT be modified:
34+
- `modules/tests/*Sources/` — Third-party Skyrim mod source files used as test fixtures. Do not edit.
35+
- `modules/tests/psc_deps/` — 83 Skyrim base class header stubs (Actor.psc, Form.psc, etc.) used as dependencies for tests. Do not edit unless specifically adding new base class stubs.
36+
- `test-files/` — Output directory for compiled test artifacts (.pex files). Not source code.
37+
- `bin/` — Build output directory for the compiler binary.
38+
39+
---
40+
41+
## Compilation Pipeline
42+
43+
The compiler processes files through these stages in order:
44+
45+
```
46+
.psc source → Scanner → Parser → Checker → Gen (gen_pex) → PEX Writer → .pex binary
47+
```
48+
49+
1. **Scanning** (`modules/papyrus/scanner/`) — Character-by-character lexer. Handles `;` line comments, `{block}` comments, `;/ multi-line /;` comments. Case-insensitive keywords. Line continuation with `\`.
50+
2. **Parsing** (`modules/papyrus/parser/`) — Recursive-descent parser producing `[]&ast.File`. Split into: `parser.v` (main + statements), `expr.v` (expressions with precedence climbing), `fn.v` (function/event declarations), `type.v` (type parsing). **Selective header loading**: unknown types are pushed onto `table.deps` stack, resolved by the builder iteratively.
51+
3. **Checking** (`modules/papyrus/checker/`) — Type checking, autocast validation, scope resolution, method resolution via inheritance chains. Split into: `checker.v` (core), `checker_stmt.v` (statements), `checker_expr.v` (expressions).
52+
4. **Code Generation** (`modules/gen/gen_pex/`) — Generates PEX bytecode from AST. Manages temp variables (`::temp0`, `::temp1`, ...), string interning, control flow jump patching. Split into: `gen.v` (main), `gen_stmt.v` (statements), `gen_expr.v` (expressions).
53+
5. **PEX Writing** (`modules/pex/writer.v`) — Big-endian binary serialization. Generic `write[T]()` with compile-time type dispatch.
54+
55+
---
56+
57+
## Key Data Structures
58+
59+
### Type System (`modules/papyrus/ast/types.v`)
60+
- `Type = int` — Index into `Table.types[]`
61+
- Built-in type indices: 1=None, 2=Int, 3=Float, 4=String, 5=Bool, 6=Array, 7-10=typed arrays (String[], Int[], Float[], Bool[])
62+
- `TypeSymbol` — Holds: kind, parent_idx, methods, props, states, vars
63+
- Placeholder types used for forward references, resolved during dependency loading
64+
65+
### Symbol Table (`modules/papyrus/ast/table.v`)
66+
- `Table` — Central registry: `types[]TypeSymbol`, `type_idxs map[string]int`, `fns map[string]Fn`, `deps Stack[string]`
67+
- **All lookups are case-insensitive** (`.to_lower()` on keys)
68+
- Functions keyed as `"objname.fnname"`
69+
70+
### AST Nodes (`modules/papyrus/ast/ast.v`, `expr.v`)
71+
- Top-level: `TopStmt = ScriptDecl | FnDecl | Comment | PropertyDecl | VarDecl | StateDecl`
72+
- Statements: `Stmt = Return | If | While | ExprStmt | AssignStmt | VarDecl | Comment`
73+
- Expressions: `Expr = InfixExpr | IntegerLiteral | FloatLiteral | BoolLiteral | StringLiteral | Ident | CallExpr | SelectorExpr | IndexExpr | ParExpr | PrefixExpr | EmptyExpr | ArrayInit | NoneLiteral | CastExpr`
74+
75+
---
76+
77+
## V Language Conventions Used in This Project
78+
79+
### Naming
80+
- Structs: `PascalCase`
81+
- Functions/methods: `snake_case`
82+
- Enum variants: `snake_case` with `.` prefix (e.g., `.key_if`)
83+
- Constants: `snake_case`
84+
- Module names: lowercase
85+
86+
### V-specific patterns used
87+
- Sum types for AST: `type Expr = InfixExpr | IntegerLiteral | ...`
88+
- Compile-time generics: `$if T is u8 { ... }` in binary read/write
89+
- Performance attributes: `@[inline]`, `@[direct_array_access]`, `@[heap]`
90+
- Result type: `!` operator for error-returning functions
91+
- `mut` receivers for mutable method calls
92+
- `spawn` for parallel code generation (up to 8 threads)
93+
94+
### Comments
95+
- Comments may contain Russian (Cyrillic) text. When you encounter Russian comments during your work, replace them with an English equivalent.
96+
97+
---
98+
99+
## Build & Run Commands
100+
101+
### Build
102+
```bash
103+
# Debug build
104+
v -o "bin\papyrus.exe" compiler.v
105+
106+
# Production build (optimized, no GC)
107+
v -o "bin\papyrus.exe" -prod -gc none compiler.v
108+
109+
# Debug with symbols
110+
v -g -gc none -o "bin\papyrus.exe" compiler.v
111+
```
112+
113+
### Run Tests
114+
```bash
115+
# Run all tests with stats
116+
v -stats test modules
117+
118+
# Run specific test file
119+
v test modules/tests/ast_test.v
120+
```
121+
122+
---
123+
124+
## Testing Conventions
125+
126+
Tests are in `modules/tests/` and use V's built-in test framework (functions prefixed with `fn test_*`).
127+
128+
### Test Categories
129+
130+
| Category | File | Purpose |
131+
|----------|------|---------|
132+
| AST shape | `ast_test.v` | Verify parsed AST node types and properties |
133+
| Error messages | `errors_test.v` | Verify compiler produces correct errors |
134+
| Type casting | `checker_cast_test.v` | Exhaustive autocast/explicit cast matrix |
135+
| PEX codegen | `pex_stmt_test.v` | End-to-end: source → PEX instructions |
136+
| PEX binary | `binary_read_write_test.v`, `pex_read_write_test.v` | Serialize/deserialize roundtrip |
137+
| Project integration | `projects_test.v` | Compile entire real Skyrim mod sources |
138+
| Selective loading | `selective_headers_loading_test.v` | Verify only needed headers are parsed |
139+
| PEX enum ordinals | `pex_test.v` | Validate opcode/value enum values |
140+
141+
### Test Helpers Pattern
142+
Each test file defines its own `compile()` helper tailored to its needs:
143+
144+
- **`ast_test.v`**: `compile(src) → (&ast.File, &ast.Table, []errors.Error)` plus `compile_stmts()`, `compile_stmt()`, `compile_expr()` wrappers for convenience.
145+
- **`errors_test.v`**: Same helpers but return only `[]errors.Error`.
146+
- **`pex_stmt_test.v`**: `compile(src) → &pex.PexFile` (full pipeline round-trip) plus `get_instructions()`.
147+
148+
All test files define `const prefs` with `no_cache: true, output_mode: .silent` and provide fixture scripts as module-level string constants (`src_template`, `other_src`, `parent_src`).
149+
150+
### Test Fixtures
151+
- `modules/tests/psc/` — Small custom `.psc` files for specific tests
152+
- `modules/tests/psc_deps/` — Skyrim base script headers (dependencies for all tests)
153+
- `modules/tests/*Sources/` — Real Skyrim mod source trees for integration tests
154+
155+
---
156+
157+
## Mandatory Rules
158+
159+
### 1. Bug fixes MUST include a test
160+
When fixing a bug, **always** create a test case that covers the bug scenario. Choose the appropriate test file:
161+
- Parser/AST bug → add test in `ast_test.v`
162+
- Type checking / semantic error → add test in `errors_test.v` or `checker_cast_test.v`
163+
- Code generation bug → add test in `pex_stmt_test.v`
164+
- Binary format bug → add test in `binary_read_write_test.v` or `pex_read_write_test.v`
165+
166+
### 2. Run tests after changes
167+
After any code change, run `v -stats test modules` to verify nothing is broken. All existing tests must pass.
168+
169+
### 3. Preserve case-insensitivity
170+
Papyrus is a **case-insensitive language**. All identifier lookups, type resolution, and keyword matching must use `.to_lower()`. Never add case-sensitive comparisons for Papyrus identifiers.
171+
172+
### 4. Error messages must be clear and user-facing
173+
Error messages shown to the user should be clear, lowercase, and descriptive. Avoid technical jargon. Include context like expected vs actual values. Follow the existing style in `errors_test.v` — e.g., `'function takes 1 parameters not 0'`.
174+
175+
### 5. Maintain pipeline separation
176+
Keep the compiler stages cleanly separated:
177+
- **Scanner** should only tokenize, never interpret semantics
178+
- **Parser** should only build AST, never type-check
179+
- **Checker** should only validate, never generate code
180+
- **Gen** should only emit bytecode, never modify AST
181+
182+
### 6. PEX format compliance
183+
Generated `.pex` files must be binary-compatible with Bethesda's Papyrus VM. The format is big-endian. Do not change the opcode enum values or binary layout without verifying against the [Compiled Script File Format](https://en.uesp.net/wiki/Skyrim_Mod:Compiled_Script_File_Format) specification.
184+
185+
### 7. Do not modify third-party test sources
186+
Files in `modules/tests/*Sources/` directories are real-world Skyrim mod scripts. They must not be modified — they serve as regression test fixtures.
187+
188+
### 8. Update changelog for user-visible changes
189+
When you modify behavior, add features, or fix bugs, update the `Next Release` section in `CHANGELOG.md` in the same change. Use clear, user-facing bullet points that state what was changed or added. Changelog entries in this section must be written in English; if Russian text is present, translate it to English.
190+
191+
Changelog entries must be written for **users of the compiler** (people writing Papyrus scripts), not for compiler developers. Follow these rules:
192+
- Describe what the user would observe: what input previously failed or behaved incorrectly, and what happens now.
193+
- Do **not** mention internal implementation details: no function names (`try_cast_to_type`, `cast_to_type`), no module names (`checker`, `gen_pex`), no compiler-internal terms (`AST`, `infix expression`, `autocast`, `opcode`, `assert`).
194+
- Do **not** include entries about internal tests, CI, or build system changes unless they directly affect the user.
195+
- Good example: `Fixed a compiler crash when using None in arithmetic expressions (e.g., None + 1). A proper error is now shown.`
196+
- Bad example: `Fixed a checker assert on invalid autocasts in infix expressions: added try_cast_to_type.`
197+
198+
---
199+
200+
## How To: Common Tasks
201+
202+
### Add support for a new language feature
203+
1. Add token(s) in `modules/papyrus/token/` if needed
204+
2. Update scanner in `modules/papyrus/scanner/` to recognize new tokens
205+
3. Add AST node(s) in `modules/papyrus/ast/`
206+
4. Update parser in `modules/papyrus/parser/` to produce new AST
207+
5. Add semantic checks in `modules/papyrus/checker/`
208+
6. Add code generation in `modules/gen/gen_pex/`
209+
7. Write tests at each level (AST, errors, PEX output)
210+
211+
### Add a new error message
212+
1. If it's a CLI-level error, add a `pub const msg_*` in `modules/papyrus/errors/errors.v`
213+
2. If it's a checker/parser error, use inline string in the `error()` call
214+
3. Add a test in `errors_test.v` that triggers the error and asserts the exact message
215+
216+
### Add a new integration test project
217+
1. Place source files in a new directory `modules/tests/<ProjectName>Sources/`
218+
2. In `projects_test.v`, add a `const` using `get_source_dir('<ProjectName>Sources', '<required_file.psc>')`
219+
3. Add `fn test_project_<name>()` calling `get_prefs()` + `builder.compile()`
220+
4. The test uses `backend: .check` (type-check only, no PEX output)
221+
222+
### Fix a bug in code generation
223+
1. Write a minimal `.psc` snippet that reproduces the issue
224+
2. Add test in `pex_stmt_test.v` using `compile()` + `get_instructions()`
225+
3. Fix the code in `modules/gen/gen_pex/`
226+
4. Assert on expected PEX opcodes/operands in the test
227+
228+
---
229+
230+
## Module Dependency Graph
231+
232+
```
233+
compiler.v → pref, builder, pex, papyrus.util
234+
builder → pref, papyrus.{ast, parser, checker, util}, gen.gen_pex, pex
235+
gen_pex → papyrus.{ast, token, util}, pex, pref
236+
checker → papyrus.{ast, token, errors, util}, pex, pref
237+
parser → papyrus.{ast, scanner, token, errors, util}, pex, pref
238+
scanner → papyrus.{token, util, errors}, pref
239+
ast → papyrus.{token, util}
240+
pex → papyrus.util, encoding.binary
241+
pref → papyrus.errors
242+
```
243+
244+
**Rule**: Do not introduce circular dependencies between modules. The dependency flow is: `compiler.v → builder → {checker, parser, gen_pex} → {ast, scanner, pex} → {token, util, errors}`.
245+
246+
---
247+
248+
## Notable Implementation Details
249+
250+
- **Selective header loading**: Only headers referenced by `table.deps` stack are parsed — not the entire Skyrim script library. This is a key performance optimization.
251+
- **Caching**: File modification times are cached in `.papyrus/*.obj` files. Use `-nocache` to bypass.
252+
- **Parallel codegen**: Enabled with `-use-threads`, divides work across up to 8 OS threads. Parse and check remain sequential.
253+
- **String interning**: PEX uses a shared string table. All strings are interned via `gen_string_ref()` during code generation.
254+
- **Temp variables**: Code generator manages a pool of `::temp0`, `::temp1`, etc. with free/reuse tracking.
255+
- **Built-in methods**: `GetState`, `GotoState`, `onBeginState`, `onEndState` are auto-added to every object. Arrays have `Find`, `RFind`, `Length`.
256+
257+
---
258+
259+
## Papyrus Language Quick Reference
260+
261+
Papyrus is a **case-insensitive**, statically-typed, object-oriented scripting language:
262+
- Types: `Int`, `Float`, `Bool`, `String`, `None`, script objects, arrays (`Type[]`)
263+
- Inheritance: `Scriptname X extends Y`
264+
- Properties: `Auto`, `AutoReadOnly`, full (with get/set)
265+
- States: `State`, `Auto State`
266+
- Events and Functions (can be `native`, `global`)
267+
- Comments: `; line`, `{block}`, `;/ multi-line /;`
268+
- Operators: arithmetic (`+`, `-`, `*`, `/`, `%`), comparison, logical (`&&`, `||`, `!`), string concatenation (`+`), compound assignment (`+=`, `-=`, etc.)
269+
- `as` keyword for explicit type casting
270+
- `new` keyword for array initialization: `new Int[10]`
271+
- `self` refers to the current script object, `parent` calls parent's version of a function

CHANGELOG.md

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,38 @@
1-
## Next release
1+
## Next Release
2+
23
...
34

5+
## V 0.0.4
6+
7+
### New Features
8+
9+
- Added `version` command — run `papyrus version` to display the current compiler version.
10+
- String literals are now accepted as default values for typed properties and function parameters and are automatically converted to the declared type.
11+
```
12+
int Property MyProp = "123" Auto ; now valid — "123" is converted to 123
13+
```
14+
```
15+
Int Function MyFunc(int n1, int n2 = "12") ; "12" is converted to 12
16+
EndFunction
17+
```
18+
19+
### Improvements
20+
21+
- The compiler now reports an error when a script referenced in `extends` or a variable type cannot be found, instead of failing silently.
22+
- The compiler now reports an error when two scripts with the same name are found in different source folders.
23+
- Added a check that the script name declared in `Scriptname` matches the source file name.
24+
- Default parameter values are now validated to be type-compatible with the declared parameter type.
25+
- Improved error messages to be clearer and more consistent (e.g., "undefined identifier" instead of "variable declaration not found").
26+
- Internal compiler errors now display a structured diagnostic message with version info, a stack trace, and instructions for reporting the issue, instead of crashing with an unhelpful message.
27+
28+
### Fixes
29+
30+
- Fixed incorrect handling of `None` as a default value in properties and function parameters.
31+
- Fixed a compiler crash when `None` was used in arithmetic or logical expressions (e.g., `None + 1`). A proper error message is now shown instead.
32+
- Fixed a compiler crash when an undefined script type was used in expressions that require conversion (for example, `value && true` where `value` has an unknown type). The compiler now reports an undefined type error instead of crashing.
33+
- Fixed an issue where calling a function with default parameters inside a `State` block was not validated correctly (#14).
34+
- Fixed parsing of comments inside parenthesized expressions and call argument lists (for example, `if !(PlayerRef ;/comment/;)`). The compiler now accepts these scripts instead of failing with a parser error.
35+
436
## V 0.0.3
537

638
### Fixes

0 commit comments

Comments
 (0)