WASM_X64: Support Data Segment #1585

ubaidsk · 2023-03-16T18:44:42Z

towards #1539

ubaidsk · 2023-03-23T04:55:07Z

Example:

(lp) lpython$ cat integration_tests/test_complex_01.py 
from ltypes import i32, i64, f32, f64, c32, c64

def test_real_imag():
    x: c64
    x = c64(2) + 3j
    a: f64
    b: f64
    eps: f64
    eps = 1e-12
    a = x.real
    b = x.imag
    assert abs(a - 2.0) <= eps
    assert abs(b - 3.0) <= eps

    print(x)

def test_complex():
    x: c64
    x = complex(4.5, 6.7)
    eps: f64
    eps = 1e-12
    assert abs(x.real - 4.5) <= eps
    assert abs(x.imag - 6.7) <= eps

    x = complex(-4, 2)
    assert abs(x.real - (-4.0)) <= eps
    assert abs(x.imag - 2.0) <= eps

    x = complex(4, 7.89)
    assert abs(x.real - 4.0) <= eps
    assert abs(x.imag - 7.89) <= eps

    x = complex(5.6, 0)
    assert abs(x.real - 5.6) <= eps
    assert abs(x.imag - 0.0) <= eps

    a: f64
    a = 534.6
    x = complex(a, -a) # (f64, f64)

    assert abs(x.real - 534.60000000000002274) <= eps
    assert abs(x.imag - (-534.60000000000002274)) <= eps

    a2: f32
    a2 = -f32(423.5430806348152437)
    a3: f32
    a3 = f32(34.5)
    x2: c32
    x2 = c32(complex(a2, a3)) # (f32, f32)

    assert f64(abs(x2.imag - f32(34.5))) <= eps

    i1: i32
    i1 = -5
    i2: i64
    i2 = -i64(6)

    x = complex(a3, a) # (f32, f64)
    x = complex(a, a3) # (f64, f32)
    x = complex(i1, i2) # (i32, i64)
    x = complex(i1, -i1) # (i32, i32)
    x = complex(-i2, -i2) # (i64, i64)
    x = complex(i2, -i1) # (i64, i32)

    print(x)

def test_complex_unary_minus():
    c: c32
    c = c32(complex(3, 4.5))
    _c: c32
    _c = -c
    assert abs(f64(_c.real) - (-3.0)) <= 1e-12
    assert abs(f64(_c.imag) - (-4.5)) <= 1e-12
    _c = c32(complex(5, -78))
    _c = -_c
    assert abs(f64(_c.real) - (-5.0)) <= 1e-12
    assert abs(f64(_c.imag) - 78.0) <= 1e-12
    c2: c64
    c2 = complex(-4.5, -7.8)
    c2 = -c2
    assert abs(c2.real - 4.5) <= 1e-12
    assert abs(c2.imag - 7.8) <= 1e-12
    c2 = c64(3) + 4j
    c2 = -c2
    assert abs(c2.real - (-3.0)) <= 1e-12
    assert abs(c2.imag - (-4.0)) <= 1e-12

    print(c, _c, c2)

def test_complex_not():
    c: c32
    c = c32(complex(4, 5))
    b: bool
    b = not c
    assert not b

    c2: c64
    c2 = complex(0, 0)
    b = not c2
    assert b

    print(c,c2, b)

def check():
    test_real_imag()
    test_complex()
    test_complex_unary_minus()
    test_complex_not()

check()

ASM Binary Format:

(lp) lpython$ lpython integration_tests/test_complex_01.py --backend wasm_x64 -o tmp > main.asm
(lp) lpython$ ./tmp
(2.000000000,3.000000000)
(-6.000000000,5.000000000)
(3.000000000,4.50000000) (-5.000000000,78.000000000) (-3.000000000,-4.000000000)
(4.000000000,5.000000000) (0.000000000,0.000000000) 1
(lp) lpython$ readelf -h tmp
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x804e2ea
  Start of program headers:          64 (bytes into file)
  Start of section headers:          0 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         3
  Size of section headers:           0 (bytes)
  Number of section headers:         0
  Section header string table index: 0
(lp) lpython$ readelf -l tmp

Elf file type is EXEC (Executable file)
Entry point 0x804e2ea
There are 3 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000008048000 0x0000000008048000
                 0x0000000000001000 0x0000000000001000  R      0x1000
  LOAD           0x0000000000001000 0x0000000008049000 0x0000000008049000
                 0x0000000000006425 0x0000000000006425  R E    0x1000
  LOAD           0x0000000000008000 0x0000000008050000 0x0000000008050000
                 0x0000000000000100 0x0000000000000100  RW     0x1000
(lp) lpython$

ASM Text Format:

(wasm_asm) lpython$ nasm -fbin main.asm && chmod +x main
(wasm_asm) lpython$ ./main
(2.000000000,3.000000000)
(-6.000000000,5.000000000)
(3.000000000,4.50000000) (-5.000000000,78.000000000) (-3.000000000,-4.000000000)
(4.000000000,5.000000000) (0.000000000,0.000000000) 1
(wasm_asm) lpython$ readelf -h main
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x804cba9
  Start of program headers:          64 (bytes into file)
  Start of section headers:          0 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         3
  Size of section headers:           0 (bytes)
  Number of section headers:         0
  Section header string table index: 0
(wasm_asm) lpython$ readelf -l main

Elf file type is EXEC (Executable file)
Entry point 0x804cba9
There are 3 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000008048000 0x0000000008048000
                 0x0000000000001000 0x0000000000001000  R      0x1000
  LOAD           0x0000000000001000 0x0000000008049000 0x0000000008049000
                 0x00000000000047c9 0x00000000000047c9  R E    0x1000
  LOAD           0x00000000000063a4 0x000000000804e3a4 0x000000000804e3a4
                 0x0000000000000100 0x0000000000000100  RW     0x1000
(wasm_asm) lpython$

ubaidsk · 2023-03-23T05:40:15Z

src/libasr/codegen/x86_assembler.h

+        for (size_t i = 0; i < padding_size; i++) {
+            m_code.push_back(m_al, 0);
+        }
+        EMIT("times " + std::to_string(padding_size) + " db 0");


Line 473: I think the padding_size computed here is valid only for the binary that we generate. For assembly text format, while encoding the text_segment, since nasm can encode instructions using different number of bytes than our encoding, the padding size required by nasm binary could be different from that of our binary.

I am looking into the possibilities of computing the padding size dynamically in assembly text format.

For NASM use the $ and $$ trick that we use to compute the size of the binary for the header.

ubaidsk · 2023-03-24T06:43:09Z

This is ready. Please review and share feedback.

certik

I think this looks great!

Should we add some tests for this new feature?

ubaidsk · 2023-03-24T18:26:46Z

Should we add some tests for this new feature?

Sure, we can add tests. I think the test integration_tests/test_complex_01.py tests reading as well as writing data to our new data segment (It reads and writes to global variables which in assembly are located/allocated in the new data segment).

(lp) lpython$ lpython integration_tests/test_complex_01.py --backend wasm_x64 -o tmp > main.asm
(lp) lpython$ ./tmp
(2.000000000,3.000000000)
(-6.000000000,5.000000000)
(3.000000000,4.50000000) (-5.000000000,78.000000000) (-3.000000000,-4.000000000)
(4.000000000,5.000000000) (0.000000000,0.000000000) 1
(lp) lpython$ 
(wasm_asm) lpython$ nasm -fbin main.asm && chmod +x main
(wasm_asm) lpython$ ./main
(2.000000000,3.000000000)
(-6.000000000,5.000000000)
(3.000000000,4.50000000) (-5.000000000,78.000000000) (-3.000000000,-4.000000000)
(4.000000000,5.000000000) (0.000000000,0.000000000) 1
(wasm_asm) lpython$

(In the above example, I added some print statements to the test integration_tests/test_complex_01.py so that we have some output and can verify that the assembly binary and text generated actually works.)

PS: The integration_tests/test_complex_01.py is already enabled for wasm_x64 backend.

ubaidsk · 2023-03-24T18:28:12Z

As this PR seems approved, I am merging this. I will work on further changes/suggestions (if any) in another PR.

certik · 2023-03-24T18:39:45Z

I see, perfect!

I don't see any global variables in https://github.com/lcompilers/lpython/blob/1dc74c6200cac7c983f3ae15d3dce25f37c55f3f/integration_tests/test_complex_01.py, which line is a global variable?

What capability does this new RW data segment allows us to do? One is that the code segment becomes read-only, which is good. The other might be that now we should be able to resize it, thus allow arbitrary length heap allocation. Anything else?

ubaidsk · 2023-03-24T19:16:08Z

lpython/src/libasr/codegen/asr_to_wasm.cpp

Lines 74 to 79 in 1dc74c6

    
           enum GLOBAL_VAR { 
        
               cur_mem_loc = 0, 
        
               tmp_reg_f32 = 1, 
        
               tmp_reg_f64 = 2, 
        
               global_vars_cnt = 3 
        
           };

We currently use global variable for temporary operations like stack manipulation (for example: to swap the top and top-second stack elements). (The cur_mem_loc global variable is not used yet I think.)

For example, the global get/set operations are used here

lpython/src/libasr/codegen/asr_to_wasm.cpp

Lines 1699 to 1710 in 1dc74c6

    
           if (f->m_kind == 4) { 
        
               this->visit_expr(*x.m_arg); 
        
               wasm::emit_f32_neg(m_code_section, m_al); 
        
               wasm::emit_set_global(m_code_section, m_al, m_global_var_name_idx_map[tmp_reg_f32]); 
        
               wasm::emit_f32_neg(m_code_section, m_al); 
        
               wasm::emit_get_global(m_code_section, m_al, m_global_var_name_idx_map[tmp_reg_f32]); 
        
           } else if (f->m_kind == 8) { 
        
               this->visit_expr(*x.m_arg); 
        
               wasm::emit_f64_neg(m_code_section, m_al); 
        
               wasm::emit_set_global(m_code_section, m_al, m_global_var_name_idx_map[tmp_reg_f64]); 
        
               wasm::emit_f64_neg(m_code_section, m_al); 
        
               wasm::emit_get_global(m_code_section, m_al, m_global_var_name_idx_map[tmp_reg_f64]);

and here

lpython/src/libasr/codegen/asr_to_wasm.cpp

Lines 2615 to 2621 in 1dc74c6

    
           wasm::emit_set_global(m_code_section, m_al, 
        
           (a_kind == 4) ? m_global_var_name_idx_map[tmp_reg_f32] 
        
               : m_global_var_name_idx_map[tmp_reg_f64]); 
        
           wasm::emit_drop(m_code_section, m_al); 
        
           wasm::emit_get_global(m_code_section, m_al, 
        
           (a_kind == 4) ? m_global_var_name_idx_map[tmp_reg_f32] 
        
               : m_global_var_name_idx_map[tmp_reg_f64]);

The integration_tests/test_complex_01.py utilizes the above code lines. Thus, it tests getting and setting globals.

ubaidsk · 2023-03-24T19:28:40Z

The other might be that now we should be able to resize it, thus allow arbitrary length heap allocation

I think it can allow us compile-time memory allocation (For example: we can have a label and can emit 0 bytes for as much space we require. This way I think we have a zero initialised memory of the required size. But this could increase the file size.)

Using the above approach, I think we can now support arrays by initializing memory for them in the data segment. For dynamic/runtime memory allocation (which could be needed for allocatable arrays), I think we should try to see/use system calls for memory allocation.

certik · 2023-03-24T20:30:00Z

I see, perfect, thanks for the explanation!

ubaidsk force-pushed the x64_data_section branch from c51cbb5 to acf01fb Compare March 23, 2023 04:51

ubaidsk force-pushed the x64_data_section branch from acf01fb to f22efa5 Compare March 23, 2023 04:59

ubaidsk commented Mar 23, 2023

View reviewed changes

ubaidsk added 6 commits March 24, 2023 11:48

X86Assembler: Fix elf comment

7a6849e

WASM_X64: Support text and data segments

da04529

X86Assembler: Update add_var() and add_var64() to use labels

cd769ef

WASM_X64: Align segments by alignment byte

7e48c49

WASM_X64: Remove use of add_var_size()

aa6c67f

X86Assembler: Remove unused parameter

ba1da0f

ubaidsk force-pushed the x64_data_section branch from f22efa5 to ab72766 Compare March 24, 2023 06:19

ubaidsk added 2 commits March 24, 2023 12:02

X86Assembler: Use align macro for assembly text format

7357202

CTEST: Update reference test for x86 asm

1b62e6b

ubaidsk force-pushed the x64_data_section branch from ab72766 to 1b62e6b Compare March 24, 2023 06:41

ubaidsk marked this pull request as ready for review March 24, 2023 06:42

ubaidsk requested a review from certik March 24, 2023 06:42

czgdp1807 approved these changes Mar 24, 2023

View reviewed changes

certik approved these changes Mar 24, 2023

View reviewed changes

ubaidsk merged commit 1dc74c6 into lcompilers:main Mar 24, 2023

ubaidsk deleted the x64_data_section branch March 24, 2023 18:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WASM_X64: Support Data Segment #1585

WASM_X64: Support Data Segment #1585

Uh oh!

ubaidsk commented Mar 16, 2023 •

edited

Loading

Uh oh!

ubaidsk commented Mar 23, 2023

Uh oh!

ubaidsk Mar 23, 2023

Uh oh!

certik Mar 23, 2023

Uh oh!

ubaidsk commented Mar 24, 2023

Uh oh!

certik left a comment

Uh oh!

ubaidsk commented Mar 24, 2023 •

edited

Loading

Uh oh!

ubaidsk commented Mar 24, 2023

Uh oh!

certik commented Mar 24, 2023

Uh oh!

ubaidsk commented Mar 24, 2023

Uh oh!

ubaidsk commented Mar 24, 2023

Uh oh!

certik commented Mar 24, 2023

Uh oh!

Uh oh!

WASM_X64: Support Data Segment #1585

WASM_X64: Support Data Segment #1585

Uh oh!

Conversation

ubaidsk commented Mar 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ubaidsk commented Mar 23, 2023

Uh oh!

ubaidsk Mar 23, 2023

Choose a reason for hiding this comment

Uh oh!

certik Mar 23, 2023

Choose a reason for hiding this comment

Uh oh!

ubaidsk commented Mar 24, 2023

Uh oh!

certik left a comment

Choose a reason for hiding this comment

Uh oh!

ubaidsk commented Mar 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ubaidsk commented Mar 24, 2023

Uh oh!

certik commented Mar 24, 2023

Uh oh!

ubaidsk commented Mar 24, 2023

Uh oh!

ubaidsk commented Mar 24, 2023

Uh oh!

certik commented Mar 24, 2023

Uh oh!

Uh oh!

ubaidsk commented Mar 16, 2023 •

edited

Loading

ubaidsk commented Mar 24, 2023 •

edited

Loading