Initial support for global Lists #1384

Thirumalai-Shaktivel · 2022-12-25T12:22:19Z

Related: #1376

Thirumalai-Shaktivel · 2022-12-25T12:24:16Z

Here I'm trying to move all the symbols in the global to the _lpython_main_program scope.

x: list[i32] = [1,2,3,4]
print(x[0])

Here is the current ASR output:

(TranslationUnit 
    (SymbolTable
        1
        {
            _lpython_main_program:
                (Function 
                    (SymbolTable
                        3
                        {
                            x:
                                (Variable 
                                    3 
                                    x 
                                    [] 
                                    Local 
                                    (ListConstant 
                                        [(IntegerConstant 1 (Integer 4 []))
                                        (IntegerConstant 2 (Integer 4 []))
                                        (IntegerConstant 3 (Integer 4 []))
                                        (IntegerConstant 4 (Integer 4 []))] 
                                        (List 
                                            (Integer 4 [])
                                        )
                                    ) 
                                    () 
                                    Default 
                                    (List 
                                        (Integer 4 [])
                                    ) 
                                    Source 
                                    Public 
                                    Required 
                                    .false.
                                )
                            
                        }) 
                    _lpython_main_program 
                    [] 
                    [] 
                    [(Print 
                        () 
                        [(ListItem 
                            (Var 3 x) 
                            (IntegerConstant 0 (Integer 4 [])) 
                            (Integer 4 []) 
                            ()
                        )] 
                        () 
                        ()
                    )] 
                    () 
                    Source 
                    Public 
                    Implementation 
                    () 
                    .false. 
                    .false. 
                    .false. 
                    .false. 
                    .false. 
                    [] 
                    [] 
                    .false.
                ), 
            main_program:
                (Program 
                    (SymbolTable
                        2
                        {
                            
                        }) 
                    main_program 
                    [] 
                    [(SubroutineCall 
                        1 _lpython_main_program 
                        () 
                        [] 
                        ()
                    )]
                )
            
        }) 
    []
)

Am I doing this correct?
What are your thoughts? @certik @czgdp1807 @Smit-create

certik · 2022-12-26T06:47:51Z

Something like this. I think for global statements we have a pass.

For global variables it gets tricky, if other functions access them. Not sure what the best solution there is.

Thirumalai-Shaktivel · 2022-12-31T09:00:47Z

+1, This seems to be tricky to handle.
We already know that all the global statements will be included inside the _lpython_main_program (By this PR), so while looking into the global scope, we will look into the _lpython_main_program scope as well.

@certik @czgdp1807 @Smit-create
Can you please share your thoughts as well? Like Different possible ways to make LPython supports global scope or how to move forward with this PR.

certik · 2022-12-31T17:22:51Z

We might need to discuss this over video. I think using TranslationUnit symbol table is fine. I think LLVM supports global variables. But we need to figure out all the details and dependencies.

certik · 2023-01-03T17:12:28Z

Here is an example to consider:

x: list[i32] = [1,2,3,4]
def f():
    print(x[1])
print(x[0])
f()

The first step is to turn all global statements and expressions into a function. We already have an existing global_stmt pass for that and it works well. By calling it, it should create an equivalent of:

x: list[i32] = [1,2,3,4]
def f():
    print(x[1])
def _global_statements():
    print(x[0])
    f()

There is one issue: it's not clear from ASR that the new generated function _global_statements should be called. So we might need some solution for that. For now let's assume it is "implicit", in other words, that this function will get called implicitly. We can add a member "identifier main" into TranslationUnit to designate a function that's the "main" function, similar to the main function in C. Either way, this is a minor issue to fix.

The major issue is how to implement the above global variables. A full robust solution requires a solid support for closures and lambda functions with capture and we are working on that. Until then, let's create a temporary solution.

The temporary solution will assume that the global variables are only accessed from the global statements, and not from any other function. That is a very common case and for this case, all we need to do is to improve the global_stmt pass to also optionally include all the global variables into this function. (If any other function is accessing them, we would get a verify() error, which is a good design.)

certik · 2023-01-03T17:22:32Z

src/libasr/pass/global_stmts.cpp

+        unit.m_global_scope->move_symbols_from_global_scope(fn_scope, symbols);
+        for (auto &a: symbols) {
+            unit.m_global_scope->erase_symbol(a);
+        }


I would not do this every time. Either as an option (via an argument), or even better, refactor this into a function and let the frontend call it. Add verify() in this refactored function.

You can add this function into global_stmts.h, so there will be two separate functions: one moves the global statements/expressions, the other moves the global variables.

This function is just a temporary solution, as later once we have solid closure support, we don't need to call this, as the global variables would be handled like any other parent scope variables.

Thirumalai-Shaktivel · 2023-01-12T15:49:25Z

The following example seems to work:

x: list[i32]
x = [1,2,3]
print(x, x[0])

But, I got some doubts:
We already support global declaration for int, float,... But the global declaration fails for the list (or maybe tuple, dict...). So, I was thought to leave int, float, ... declarations and move only those that fail in the global scope (like list).

@czgdp1807 @certik thoughts?

Thirumalai-Shaktivel · 2023-01-13T14:52:46Z

We need to discuss more on this for further implementations.
Currently, we move the variables like list, dict, tuples into the function. After moving, we create an External symbol for that moved item and erase it.

I have a doubt, can we have an External symbol in TranslationUnit sym_table?
Does this violate the design?

I think the ASR seems correct, but the LLVM fails:

x: list[i32]
x = [1,2,3]
print(x, x[0])

def main0():
    print(x[0])

ASR:

(TranslationUnit
    (SymbolTable
        1
        {
            _lpython_main_program:
                (Function
                    (SymbolTable
                        4
                        {
                            x:
                                (Variable
                                    4
                                    x
                                    []
                                    Local
                                    ()
                                    ()
                                    Default
                                    (List
                                        (Integer 4 [])
                                    )
                                    Source
                                    Public
                                    Required
                                    .false.
                                )
                        })
                    _lpython_main_program
                    [main0]
                    []
                    [(=
                        (Var 1 x)
                        (ListConstant
                            [(IntegerConstant 1 (Integer 4 []))
                            (IntegerConstant 2 (Integer 4 []))]
                            (List
                                (Integer 4 [])
                            )
                        )
                        ()
                    )
                    (Print
                        ()
                        [(ListItem
                            (Var 1 x)
                            (IntegerConstant 1 (Integer 4 []))
                            (Integer 4 [])
                            ()
                        )]
                        ()
                        ()
                    )
                    (SubroutineCall
                        1 main0
                        ()
                        []
                        ()
                    )]
                    ()
                    Source
                    Public
                    Implementation
                    ()
                    .false.
                    .false.
                    .false.
                    .false.
                    .false.
                    []
                    []
                    .false.
                ),
            main0:
                (Function
                    (SymbolTable
                        2
                        {
                            
                        })
                    main0
                    []
                    []
                    [(Print
                        ()
                        [(ListItem
                            (Var 1 x)
                            (IntegerConstant 1 (Integer 4 []))
                            (Integer 4 [])
                            ()
                        )]
                        ()
                        ()
                    )]
                    ()
                    Source
                    Public
                    Implementation
                    ()
                    .false.
                    .false.
                    .false.
                    .false.
                    .false.
                    []
                    []
                    .false.
                ),
            main_program:
                (Program
                    (SymbolTable
                        3
                        {
                            
                        })
                    main_program
                    []
                    [(SubroutineCall
                        1 _lpython_main_program
                        ()
                        []
                        ()
                    )]
                ),
            x:
                (ExternalSymbol
                    1
                    x
                    4 x
                    _lpython_main_program
                    []
                    x
                    Public
                )
        })
    _lpython_main_program
    []
)

LLVM:

code generation error: asr_to_llvm: module failed verification. Error:
Instruction does not dominate all uses!
  %x = alloca %list, align 8
  %0 = getelementptr %list, %list* %x, i32 0, i32 0
Instruction does not dominate all uses!
  %x = alloca %list, align 8
  %4 = getelementptr %list, %list* %x, i32 0, i32 2

certik · 2023-01-17T19:23:13Z

Related: lfortran/lfortran#1059

certik · 2023-01-17T19:27:31Z

Let's create two passes:

One pass transforms the executable code into a function in the same translation unit, but leaves global variables in translation unit. After this pass, there are no global statements or expressions, only global variables (in translation unit) and functions that access those global variables. We already have a pass for this, so perhaps it just needs to be improved to handle all cases. This pass does not rewrite any existing function.
Second pass transforms the global variables into a module. We have to support module variables anyway in all backends, so this might make it easy to support global variables. This pass requires to modify all functions that access the global variable to import the variable from the module. This pass in a way could simply be a transformation of everything inside the TranslationUnit's symbol table into a module, promoting the translationunit to a module. The old translation unit contains a main program that calls a global function. The new translation unit would contain a main program and a module, and the main program would call the init function from the module.

The second bullet point seems like a clean approach: each module in ASR is self contained, it cannot access anything from global scope (=translation unit's symbol table), it accesses everything via ExternalSymbol. The main program currently can call functions from global scope. Finally, the only place that can call global variables in global scope (translation unit) are other functions from global scope (translation unit). So if we promote translation unit to a module, the new translation unit will only have a main program and other modules, and only the main program has to be modified to use ExternalSymbol from this new module.

Thirumalai-Shaktivel · 2023-01-18T09:09:11Z

x: list[i32]
x = [1,2,3]
print(x, x[0])

def main0():
    print(x[0])

ASR output with current changes:

(TranslationUnit
    (SymbolTable
        1
        {
            _global_symbols:
                (Module
                    (SymbolTable
                        5
                        {
                            _lpython_main_program:
                                (Function
                                    (SymbolTable
                                        4
                                        {
                                            x:
                                                (ExternalSymbol
                                                    4
                                                    x
                                                    5 x
                                                    _global_symbols
                                                    []
                                                    x
                                                    Public
                                                )
                                        })
                                    _lpython_main_program
                                    []
                                    []
                                    [(=
                                        (Var 4 x)
                                        (ListConstant
                                            [(IntegerConstant 1 (Integer 4 []))
                                            (IntegerConstant 2 (Integer 4 []))
                                            (IntegerConstant 3 (Integer 4 []))]
                                            (List
                                                (Integer 4 [])
                                            )
                                        )
                                        ()
                                    )
                                    (Print
                                        ()
                                        [(Var 5 x)
                                        (ListItem
                                            (Var 5 x)
                                            (IntegerConstant 0 (Integer 4 []))
                                            (Integer 4 [])
                                            ()
                                        )]
                                        ()
                                        ()
                                    )]
                                    ()
                                    Source
                                    Public
                                    Implementation
                                    ()
                                    .false.
                                    .false.
                                    .false.
                                    .false.
                                    .false.
                                    []
                                    []
                                    .false.
                                ),
                            main0:
                                (Function
                                    (SymbolTable
                                        2
                                        {
                                            x:
                                                (ExternalSymbol
                                                    2
                                                    x
                                                    5 x
                                                    _global_symbols
                                                    []
                                                    x
                                                    Public
                                                )
                                        })
                                    main0
                                    []
                                    []
                                    [(Print
                                        ()
                                        [(ListItem
                                            (Var 2 x)
                                            (IntegerConstant 0 (Integer 4 []))
                                            (Integer 4 [])
                                            ()
                                        )]
                                        ()
                                        ()
                                    )]
                                    ()
                                    Source
                                    Public
                                    Implementation
                                    ()
                                    .false.
                                    .false.
                                    .false.
                                    .false.
                                    .false.
                                    []
                                    []
                                    .false.
                                ),
                            x:
                                (Variable
                                    5
                                    x
                                    []
                                    Local
                                    ()
                                    ()
                                    Default
                                    (List
                                        (Integer 4 [])
                                    )
                                    Source
                                    Public
                                    Required
                                    .false.
                                )
                        })
                    _global_symbols
                    []
                    .false.
                    .false.
                ),
            main_program:
                (Program
                    (SymbolTable
                        3
                        {
                            _lpython_main_program:
                                (ExternalSymbol
                                    3
                                    _lpython_main_program
                                    5 _lpython_main_program
                                    _global_symbols
                                    []
                                    _lpython_main_program
                                    Public
                                )
                        })
                    main_program
                    []
                    [(SubroutineCall
                        3 _lpython_main_program
                        ()
                        []
                        ()
                    )]
                )
        })
    []
)

Thirumalai-Shaktivel · 2023-01-18T09:16:33Z

After everything works (List in the LLVM), then I will create a separate pass for pass_wrap_symbols_into_module

Thirumalai-Shaktivel · 2023-01-27T07:17:00Z

lpython/src/libasr/codegen/asr_to_llvm.cpp

Line 2855 in 2c5034a

llvm::AllocaInst *ptr = builder->CreateAlloca(type, nullptr, v->m_name);

This AllocaInst is in the decl_vars function to declare variables, I think. My doubt is, here ptr is declared but where does it get loaded or stored for Lists in LLVM?
@czgdp1807

examples/expr2.py

certik · 2023-02-01T20:58:33Z

In here I would simply focus on creating a pass that transform the global statements and variables from TranslationUnit into a module. And you can create integration_test, and test it using ASR tests in tests/. Let's not worry if things do not work with LLVM.

Then in a separate PR let's test that module variables (of all types, including lists) work.

Thirumalai-Shaktivel · 2023-03-07T05:24:59Z

I followed the first approach from here: #1491 (comment)
examples/expr2.py seems to work.

@czgdp1807 What do you think about this?

certik · 2023-03-08T19:33:12Z

Looks good so far. You can add a test, get everything passing. Then you can figure out how to change void* into "struct" as needed. Regarding LLVM pointers, use ASR to figure out all the types, not LLVM.

czgdp1807

LGTM.

Thirumalai-Shaktivel · 2023-03-11T04:41:19Z

Then you can figure out how to change void* into "struct" as needed.

Should I try this in this PR? @certik
After getting this done, I will work on importing the global variables.

Thirumalai-Shaktivel · 2023-03-11T04:58:28Z

Actually, it was easy to figure out.

certik · 2023-03-11T05:44:05Z

There are some test failures.

czgdp1807 · 2023-03-15T15:06:51Z

src/libasr/codegen/asr_to_llvm.cpp

@@ -2296,6 +2297,14 @@ class ASRToLLVMVisitor : public ASR::BaseVisitor<ASRToLLVMVisitor>
                }
            }
            llvm_symtab[h] = ptr;
+        } else if (x.m_type->type == ASR::ttypeType::List) {
+            llvm::StructType* list_type = (llvm::StructType *)


Is this C style cast needed? I would recommend to go for,

Suggested change

llvm::StructType* list_type = (llvm::StructType *)

llvm::StructType* list_type = static_cast<llvm::StructType*>(

czgdp1807 · 2023-03-15T15:07:01Z

src/libasr/codegen/asr_to_llvm.cpp

@@ -2296,6 +2297,14 @@ class ASRToLLVMVisitor : public ASR::BaseVisitor<ASRToLLVMVisitor>
                }
            }
            llvm_symtab[h] = ptr;
+        } else if (x.m_type->type == ASR::ttypeType::List) {
+            llvm::StructType* list_type = (llvm::StructType *)
+                get_type_from_ttype_t_util(x.m_type);


Suggested change

get_type_from_ttype_t_util(x.m_type);

get_type_from_ttype_t_util(x.m_type));

czgdp1807 · 2023-03-15T15:08:06Z

src/lpython/semantics/python_ast_to_asr.cpp

+
+    void visit_Assert(const AST::Assert_t &/*x*/) {
+        // We skip this in the SymbolTable visitor, but visit it in the BodyVisitor
+    }


Probably we should generate this for all the statement types. In a different PR though after this goes in.

czgdp1807 · 2023-03-15T15:10:32Z

Actually, it was easy to figure out.

CreateBitCast I guess?

Thirumalai-Shaktivel · 2023-03-15T15:19:20Z

CreateBitCast I guess?

The thing is I switched from the void* type to the list type, so there is no need to do BitCast, I think.

; ModuleID = 'LFortran'
source_filename = "LFortran"

%list = type { i32, i32, i32* }

@i = global i32 0
@x = global %list zeroinitializer     <-----
...

Thirumalai-Shaktivel · 2023-03-15T15:23:29Z

src/libasr/codegen/asr_to_llvm.cpp

+            llvm::StructType* list_type = static_cast<llvm::StructType*>(
+                get_type_from_ttype_t_util(x.m_type));
+            llvm::Constant *ptr = module->getOrInsertGlobal(x.m_name, list_type);
+            module->getNamedGlobal(x.m_name)->setInitializer(
+                llvm::ConstantStruct::get(list_type,
+                llvm::Constant::getNullValue(list_type)));
+            llvm_symtab[h] = ptr;


Then you can figure out how to change void* into "struct" as needed.
#1384 (comment)

Sorry, I think I understood this wrong!
@czgdp1807 is the approach used here correct?

Using global variable is better. It stays in scope and is independent of all LLVM functions. I checked the generated LLVM code and it looks fine to me.

certik · 2023-03-15T17:10:05Z

tests/reference/llvm-print_04-443a8d8.stdout

+@u = global i64 0
+@x = global i32 0
+@y = global i16 0
+@z = global i8 0


I think I would keep these initializations here. This is relatively minor, so we can do it in the next PR.

I created an issue for this at #1582.

certik reviewed Jan 3, 2023

View reviewed changes

Thirumalai-Shaktivel force-pushed the global_stmts branch from f9d7013 to 12de970 Compare January 12, 2023 15:42

Thirumalai-Shaktivel force-pushed the global_stmts branch from 9090781 to 47a1d0c Compare January 13, 2023 14:24

Thirumalai-Shaktivel force-pushed the global_stmts branch from 47a1d0c to 3c8d70c Compare January 18, 2023 09:07

Thirumalai-Shaktivel force-pushed the global_stmts branch from 3c8d70c to 104f5f9 Compare January 28, 2023 04:24

Thirumalai-Shaktivel commented Jan 28, 2023

View reviewed changes

examples/expr2.py Outdated Show resolved Hide resolved

Thirumalai-Shaktivel force-pushed the global_stmts branch from 104f5f9 to e530245 Compare January 29, 2023 16:34

Thirumalai-Shaktivel mentioned this pull request Feb 18, 2023

[ASRPass] Wrap all the global symbols into a module #1491

Merged

Thirumalai-Shaktivel force-pushed the global_stmts branch 3 times, most recently from 3635e32 to 468379f Compare March 7, 2023 05:22

Thirumalai-Shaktivel force-pushed the global_stmts branch from 468379f to 322f9c3 Compare March 10, 2023 06:04

Thirumalai-Shaktivel changed the title ~~Move all the symbols from global scope into a function~~ Initial support for global Lists Mar 10, 2023

Thirumalai-Shaktivel marked this pull request as ready for review March 10, 2023 07:13

czgdp1807 approved these changes Mar 10, 2023

View reviewed changes

Thirumalai-Shaktivel force-pushed the global_stmts branch from 73d75d1 to 935c560 Compare March 10, 2023 16:13

Thirumalai-Shaktivel requested a review from certik March 11, 2023 04:39

Thirumalai-Shaktivel force-pushed the global_stmts branch from 935c560 to 2ec8413 Compare March 11, 2023 04:57

Thirumalai-Shaktivel force-pushed the global_stmts branch from 2264b79 to 06e58ed Compare March 11, 2023 05:19

Thirumalai-Shaktivel marked this pull request as draft March 11, 2023 05:49

Thirumalai-Shaktivel force-pushed the global_stmts branch from 06e58ed to 0a6e983 Compare March 15, 2023 09:38

Thirumalai-Shaktivel marked this pull request as ready for review March 15, 2023 10:16

czgdp1807 approved these changes Mar 15, 2023

View reviewed changes

Thirumalai-Shaktivel force-pushed the global_stmts branch from 0a6e983 to f4ced93 Compare March 15, 2023 15:12

Thirumalai-Shaktivel added 4 commits March 15, 2023 20:45

Initial support for global Lists

0baea95

Allow assert statement in global scope

bd48497

Support Annotated assignment in the global_scope

4a91d33

Add a test and register in CMakeLists

f86901d

Thirumalai-Shaktivel force-pushed the global_stmts branch from f4ced93 to f86901d Compare March 15, 2023 15:16

Thirumalai-Shaktivel commented Mar 15, 2023

View reviewed changes

certik reviewed Mar 15, 2023

View reviewed changes

certik approved these changes Mar 15, 2023

View reviewed changes

certik merged commit 78c8347 into lcompilers:main Mar 15, 2023

certik mentioned this pull request Mar 15, 2023

Keep initializers for global integers/floats/logical/complex #1582

Closed

Thirumalai-Shaktivel deleted the global_stmts branch March 16, 2023 04:11

Thirumalai-Shaktivel mentioned this pull request Mar 16, 2023

Implementing Lists as global variables #1464

Closed

Pranavchiku mentioned this pull request Jun 4, 2023

Assignment to ArrayItem from module throws LCOMPILERS_ASSERT failed lfortran/lfortran#1735

Open

	llvm::StructType* list_type = (llvm::StructType *)
	llvm::StructType* list_type = static_cast<llvm::StructType*>(

	get_type_from_ttype_t_util(x.m_type);
	get_type_from_ttype_t_util(x.m_type));

Initial support for global Lists #1384

Initial support for global Lists #1384

Uh oh!

Conversation

Thirumalai-Shaktivel commented Dec 25, 2022

Uh oh!

Thirumalai-Shaktivel commented Dec 25, 2022

Uh oh!

certik commented Dec 26, 2022

Uh oh!

Thirumalai-Shaktivel commented Dec 31, 2022

Uh oh!

certik commented Dec 31, 2022

Uh oh!

certik commented Jan 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

certik Jan 3, 2023

Choose a reason for hiding this comment

Uh oh!

certik Jan 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Thirumalai-Shaktivel commented Jan 12, 2023

Uh oh!

Thirumalai-Shaktivel commented Jan 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

certik commented Jan 17, 2023

Uh oh!

certik commented Jan 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thirumalai-Shaktivel commented Jan 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thirumalai-Shaktivel commented Jan 18, 2023

Uh oh!

Thirumalai-Shaktivel commented Jan 27, 2023

Uh oh!

Uh oh!

certik commented Feb 1, 2023

Uh oh!

Thirumalai-Shaktivel commented Mar 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

certik commented Mar 8, 2023

Uh oh!

czgdp1807 left a comment

Choose a reason for hiding this comment

Uh oh!

Thirumalai-Shaktivel commented Mar 11, 2023

Uh oh!

Thirumalai-Shaktivel commented Mar 11, 2023

Uh oh!

certik commented Mar 11, 2023

Uh oh!

czgdp1807 Mar 15, 2023

Choose a reason for hiding this comment

Uh oh!

czgdp1807 Mar 15, 2023

Choose a reason for hiding this comment

Uh oh!

czgdp1807 Mar 15, 2023

Choose a reason for hiding this comment

Uh oh!

czgdp1807 commented Mar 15, 2023

Uh oh!

Thirumalai-Shaktivel commented Mar 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thirumalai-Shaktivel Mar 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

czgdp1807 Mar 15, 2023

Choose a reason for hiding this comment

Uh oh!

Thirumalai-Shaktivel Mar 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

certik commented Jan 3, 2023 •

edited

Loading

certik Jan 3, 2023 •

edited

Loading

Thirumalai-Shaktivel commented Jan 13, 2023 •

edited

Loading

certik commented Jan 17, 2023 •

edited

Loading

Thirumalai-Shaktivel commented Jan 18, 2023 •

edited

Loading

Thirumalai-Shaktivel commented Mar 7, 2023 •

edited

Loading

Thirumalai-Shaktivel commented Mar 15, 2023 •

edited

Loading

Thirumalai-Shaktivel Mar 15, 2023 •

edited

Loading

Thirumalai-Shaktivel Mar 15, 2023 •

edited

Loading