Skip to content

Conversation

pyramation
Copy link
Collaborator

@pyramation pyramation commented Aug 8, 2025

Add WebAssembly memory configuration for large SQL files

Summary

This PR adds Emscripten memory configuration flags to prevent "memory access out of bounds" errors when parsing large SQL files with the WebAssembly build of libpg_query. The changes set a larger initial heap (256MB) and higher maximum memory limit (1GB) while preserving dynamic memory growth and all existing module exports.

Key Changes:

  • Added -sINITIAL_MEMORY=268435456 (256MB initial heap)
  • Added -sMAXIMUM_MEMORY=1073741824 (1GB maximum memory)
  • Kept existing -sALLOW_MEMORY_GROWTH=1 for dynamic expansion
  • Preserved -sMODULARIZE=1 and all existing exports
  • Updated both template system and standalone build configurations

Review & Testing Checklist for Human

  • Build WASM modules - Verify that all PostgreSQL versions (13-17) and the full parser compile successfully with the new memory flags
  • Test large SQL files - Confirm that previously failing large SQL inputs no longer crash with "memory access out of bounds" errors
  • Verify existing functionality - Test that small/normal SQL parsing still works correctly in both Node.js and browser environments
  • Check all build configurations - Ensure that the multi-version parser (@pgsql/parser) and full parser (@libpg-query/parser) properly inherit the memory settings
  • Performance testing - Verify that the larger initial memory doesn't negatively impact startup time or memory usage for typical workloads

Diagram

%%{ init : { "theme" : "default" }}%%
graph TD
    subgraph "Build System"
        Template["templates/Makefile.template"]:::major-edit
        CopyScript["scripts/copy-templates.js"]:::context
    end
    
    subgraph "Version Builds"
        V13["versions/13/Makefile"]:::major-edit
        V14["versions/14/Makefile"]:::major-edit
        V15["versions/15/Makefile"]:::major-edit
        V16["versions/16/Makefile"]:::major-edit
        V17["versions/17/Makefile"]:::major-edit
    end
    
    subgraph "Standalone Builds"
        Full["full/Makefile"]:::major-edit
    end
    
    Template -->|"copy:templates"| V13
    Template -->|"copy:templates"| V14
    Template -->|"copy:templates"| V15
    Template -->|"copy:templates"| V16
    Template -->|"copy:templates"| V17
    
    V13 -->|"emscripten build"| WASM13["libpg-query.wasm (v13)"]:::context
    V17 -->|"emscripten build"| WASM17["libpg-query.wasm (v17)"]:::context
    Full -->|"emscripten build"| WASMFull["libpg-query.wasm (full)"]:::context
    
    subgraph Legend
        L1["Major Edit"]:::major-edit
        L2["Minor Edit"]:::minor-edit
        L3["Context/No Edit"]:::context
    end
    
    classDef major-edit fill:#90EE90
    classDef minor-edit fill:#87CEEB
    classDef context fill:#FFFFFF
Loading

Notes

  • Session Info: Requested by Dan Lynch (@pyramation) - Devin Session
  • Memory Values: 256MB initial (268435456 bytes) and 1GB max (1073741824 bytes) were specifically requested by the user
  • Template System: Changes were made to the template and propagated using the existing copy:templates script to maintain consistency across all PostgreSQL versions
  • Untested: The WASM modules were not built or tested during development due to Docker/Emscripten setup requirements - this is the highest risk area for the review

- Set INITIAL_MEMORY to 268435456 (256MB) to prevent memory access out of bounds errors
- Set MAXIMUM_MEMORY to 1073741824 (1GB) to allow handling of very large SQL files
- Keep existing ALLOW_MEMORY_GROWTH=1 for dynamic memory expansion
- Preserve MODULARIZE=1 and all existing exports
- Updated template system and full/Makefile to ensure consistency across all PostgreSQL versions

Co-Authored-By: Dan Lynch <[email protected]>
Copy link

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

- Restore LIBPG_QUERY_REPO to launchql/libpg_query.git
- Restore LIBPG_QUERY_TAG to fix/negative-int-pg15
- Keep memory configuration flags as intended

Co-Authored-By: Dan Lynch <[email protected]>
@pyramation pyramation closed this Aug 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant