Skip to content

Initial support for global Lists #1384

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 15, 2023

Conversation

Thirumalai-Shaktivel
Copy link
Collaborator

Related: #1376

@Thirumalai-Shaktivel
Copy link
Collaborator Author

Here I'm trying to move all the symbols in the global to the _lpython_main_program scope.

x: list[i32] = [1,2,3,4]
print(x[0])

Here is the current ASR output:

(TranslationUnit 
    (SymbolTable
        1
        {
            _lpython_main_program:
                (Function 
                    (SymbolTable
                        3
                        {
                            x:
                                (Variable 
                                    3 
                                    x 
                                    [] 
                                    Local 
                                    (ListConstant 
                                        [(IntegerConstant 1 (Integer 4 []))
                                        (IntegerConstant 2 (Integer 4 []))
                                        (IntegerConstant 3 (Integer 4 []))
                                        (IntegerConstant 4 (Integer 4 []))] 
                                        (List 
                                            (Integer 4 [])
                                        )
                                    ) 
                                    () 
                                    Default 
                                    (List 
                                        (Integer 4 [])
                                    ) 
                                    Source 
                                    Public 
                                    Required 
                                    .false.
                                )
                            
                        }) 
                    _lpython_main_program 
                    [] 
                    [] 
                    [(Print 
                        () 
                        [(ListItem 
                            (Var 3 x) 
                            (IntegerConstant 0 (Integer 4 [])) 
                            (Integer 4 []) 
                            ()
                        )] 
                        () 
                        ()
                    )] 
                    () 
                    Source 
                    Public 
                    Implementation 
                    () 
                    .false. 
                    .false. 
                    .false. 
                    .false. 
                    .false. 
                    [] 
                    [] 
                    .false.
                ), 
            main_program:
                (Program 
                    (SymbolTable
                        2
                        {
                            
                        }) 
                    main_program 
                    [] 
                    [(SubroutineCall 
                        1 _lpython_main_program 
                        () 
                        [] 
                        ()
                    )]
                )
            
        }) 
    []
)

Am I doing this correct?
What are your thoughts? @certik @czgdp1807 @Smit-create

@certik
Copy link
Contributor

certik commented Dec 26, 2022

Something like this. I think for global statements we have a pass.

For global variables it gets tricky, if other functions access them. Not sure what the best solution there is.

@Thirumalai-Shaktivel
Copy link
Collaborator Author

+1, This seems to be tricky to handle.
We already know that all the global statements will be included inside the _lpython_main_program (By this PR), so while looking into the global scope, we will look into the _lpython_main_program scope as well.

@certik @czgdp1807 @Smit-create
Can you please share your thoughts as well? Like Different possible ways to make LPython supports global scope or how to move forward with this PR.

@certik
Copy link
Contributor

certik commented Dec 31, 2022

We might need to discuss this over video. I think using TranslationUnit symbol table is fine. I think LLVM supports global variables. But we need to figure out all the details and dependencies.

@certik
Copy link
Contributor

certik commented Jan 3, 2023

Here is an example to consider:

x: list[i32] = [1,2,3,4]
def f():
    print(x[1])
print(x[0])
f()

The first step is to turn all global statements and expressions into a function. We already have an existing global_stmt pass for that and it works well. By calling it, it should create an equivalent of:

x: list[i32] = [1,2,3,4]
def f():
    print(x[1])
def _global_statements():
    print(x[0])
    f()

There is one issue: it's not clear from ASR that the new generated function _global_statements should be called. So we might need some solution for that. For now let's assume it is "implicit", in other words, that this function will get called implicitly. We can add a member "identifier main" into TranslationUnit to designate a function that's the "main" function, similar to the main function in C. Either way, this is a minor issue to fix.

The major issue is how to implement the above global variables. A full robust solution requires a solid support for closures and lambda functions with capture and we are working on that. Until then, let's create a temporary solution.

The temporary solution will assume that the global variables are only accessed from the global statements, and not from any other function. That is a very common case and for this case, all we need to do is to improve the global_stmt pass to also optionally include all the global variables into this function. (If any other function is accessing them, we would get a verify() error, which is a good design.)

unit.m_global_scope->move_symbols_from_global_scope(fn_scope, symbols);
for (auto &a: symbols) {
unit.m_global_scope->erase_symbol(a);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not do this every time. Either as an option (via an argument), or even better, refactor this into a function and let the frontend call it. Add verify() in this refactored function.

Copy link
Contributor

@certik certik Jan 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add this function into global_stmts.h, so there will be two separate functions: one moves the global statements/expressions, the other moves the global variables.

This function is just a temporary solution, as later once we have solid closure support, we don't need to call this, as the global variables would be handled like any other parent scope variables.

@Thirumalai-Shaktivel
Copy link
Collaborator Author

The following example seems to work:

x: list[i32]
x = [1,2,3]
print(x, x[0])

But, I got some doubts:
We already support global declaration for int, float,... But the global declaration fails for the list (or maybe tuple, dict...). So, I was thought to leave int, float, ... declarations and move only those that fail in the global scope (like list).

@czgdp1807 @certik thoughts?

@Thirumalai-Shaktivel
Copy link
Collaborator Author

Thirumalai-Shaktivel commented Jan 13, 2023

We need to discuss more on this for further implementations.
Currently, we move the variables like list, dict, tuples into the function. After moving, we create an External symbol for that moved item and erase it.

I have a doubt, can we have an External symbol in TranslationUnit sym_table?
Does this violate the design?

I think the ASR seems correct, but the LLVM fails:

x: list[i32]
x = [1,2,3]
print(x, x[0])

def main0():
    print(x[0])

ASR:

(TranslationUnit
    (SymbolTable
        1
        {
            _lpython_main_program:
                (Function
                    (SymbolTable
                        4
                        {
                            x:
                                (Variable
                                    4
                                    x
                                    []
                                    Local
                                    ()
                                    ()
                                    Default
                                    (List
                                        (Integer 4 [])
                                    )
                                    Source
                                    Public
                                    Required
                                    .false.
                                )
                        })
                    _lpython_main_program
                    [main0]
                    []
                    [(=
                        (Var 1 x)
                        (ListConstant
                            [(IntegerConstant 1 (Integer 4 []))
                            (IntegerConstant 2 (Integer 4 []))]
                            (List
                                (Integer 4 [])
                            )
                        )
                        ()
                    )
                    (Print
                        ()
                        [(ListItem
                            (Var 1 x)
                            (IntegerConstant 1 (Integer 4 []))
                            (Integer 4 [])
                            ()
                        )]
                        ()
                        ()
                    )
                    (SubroutineCall
                        1 main0
                        ()
                        []
                        ()
                    )]
                    ()
                    Source
                    Public
                    Implementation
                    ()
                    .false.
                    .false.
                    .false.
                    .false.
                    .false.
                    []
                    []
                    .false.
                ),
            main0:
                (Function
                    (SymbolTable
                        2
                        {
                            
                        })
                    main0
                    []
                    []
                    [(Print
                        ()
                        [(ListItem
                            (Var 1 x)
                            (IntegerConstant 1 (Integer 4 []))
                            (Integer 4 [])
                            ()
                        )]
                        ()
                        ()
                    )]
                    ()
                    Source
                    Public
                    Implementation
                    ()
                    .false.
                    .false.
                    .false.
                    .false.
                    .false.
                    []
                    []
                    .false.
                ),
            main_program:
                (Program
                    (SymbolTable
                        3
                        {
                            
                        })
                    main_program
                    []
                    [(SubroutineCall
                        1 _lpython_main_program
                        ()
                        []
                        ()
                    )]
                ),
            x:
                (ExternalSymbol
                    1
                    x
                    4 x
                    _lpython_main_program
                    []
                    x
                    Public
                )
        })
    _lpython_main_program
    []
)

LLVM:

code generation error: asr_to_llvm: module failed verification. Error:
Instruction does not dominate all uses!
  %x = alloca %list, align 8
  %0 = getelementptr %list, %list* %x, i32 0, i32 0
Instruction does not dominate all uses!
  %x = alloca %list, align 8
  %4 = getelementptr %list, %list* %x, i32 0, i32 2

@certik
Copy link
Contributor

certik commented Jan 17, 2023

Related: lfortran/lfortran#1059

@certik
Copy link
Contributor

certik commented Jan 17, 2023

Let's create two passes:

  • One pass transforms the executable code into a function in the same translation unit, but leaves global variables in translation unit. After this pass, there are no global statements or expressions, only global variables (in translation unit) and functions that access those global variables. We already have a pass for this, so perhaps it just needs to be improved to handle all cases. This pass does not rewrite any existing function.

  • Second pass transforms the global variables into a module. We have to support module variables anyway in all backends, so this might make it easy to support global variables. This pass requires to modify all functions that access the global variable to import the variable from the module. This pass in a way could simply be a transformation of everything inside the TranslationUnit's symbol table into a module, promoting the translationunit to a module. The old translation unit contains a main program that calls a global function. The new translation unit would contain a main program and a module, and the main program would call the init function from the module.

The second bullet point seems like a clean approach: each module in ASR is self contained, it cannot access anything from global scope (=translation unit's symbol table), it accesses everything via ExternalSymbol. The main program currently can call functions from global scope. Finally, the only place that can call global variables in global scope (translation unit) are other functions from global scope (translation unit). So if we promote translation unit to a module, the new translation unit will only have a main program and other modules, and only the main program has to be modified to use ExternalSymbol from this new module.

@Thirumalai-Shaktivel
Copy link
Collaborator Author

Thirumalai-Shaktivel commented Jan 18, 2023

x: list[i32]
x = [1,2,3]
print(x, x[0])

def main0():
    print(x[0])

ASR output with current changes:

(TranslationUnit
    (SymbolTable
        1
        {
            _global_symbols:
                (Module
                    (SymbolTable
                        5
                        {
                            _lpython_main_program:
                                (Function
                                    (SymbolTable
                                        4
                                        {
                                            x:
                                                (ExternalSymbol
                                                    4
                                                    x
                                                    5 x
                                                    _global_symbols
                                                    []
                                                    x
                                                    Public
                                                )
                                        })
                                    _lpython_main_program
                                    []
                                    []
                                    [(=
                                        (Var 4 x)
                                        (ListConstant
                                            [(IntegerConstant 1 (Integer 4 []))
                                            (IntegerConstant 2 (Integer 4 []))
                                            (IntegerConstant 3 (Integer 4 []))]
                                            (List
                                                (Integer 4 [])
                                            )
                                        )
                                        ()
                                    )
                                    (Print
                                        ()
                                        [(Var 5 x)
                                        (ListItem
                                            (Var 5 x)
                                            (IntegerConstant 0 (Integer 4 []))
                                            (Integer 4 [])
                                            ()
                                        )]
                                        ()
                                        ()
                                    )]
                                    ()
                                    Source
                                    Public
                                    Implementation
                                    ()
                                    .false.
                                    .false.
                                    .false.
                                    .false.
                                    .false.
                                    []
                                    []
                                    .false.
                                ),
                            main0:
                                (Function
                                    (SymbolTable
                                        2
                                        {
                                            x:
                                                (ExternalSymbol
                                                    2
                                                    x
                                                    5 x
                                                    _global_symbols
                                                    []
                                                    x
                                                    Public
                                                )
                                        })
                                    main0
                                    []
                                    []
                                    [(Print
                                        ()
                                        [(ListItem
                                            (Var 2 x)
                                            (IntegerConstant 0 (Integer 4 []))
                                            (Integer 4 [])
                                            ()
                                        )]
                                        ()
                                        ()
                                    )]
                                    ()
                                    Source
                                    Public
                                    Implementation
                                    ()
                                    .false.
                                    .false.
                                    .false.
                                    .false.
                                    .false.
                                    []
                                    []
                                    .false.
                                ),
                            x:
                                (Variable
                                    5
                                    x
                                    []
                                    Local
                                    ()
                                    ()
                                    Default
                                    (List
                                        (Integer 4 [])
                                    )
                                    Source
                                    Public
                                    Required
                                    .false.
                                )
                        })
                    _global_symbols
                    []
                    .false.
                    .false.
                ),
            main_program:
                (Program
                    (SymbolTable
                        3
                        {
                            _lpython_main_program:
                                (ExternalSymbol
                                    3
                                    _lpython_main_program
                                    5 _lpython_main_program
                                    _global_symbols
                                    []
                                    _lpython_main_program
                                    Public
                                )
                        })
                    main_program
                    []
                    [(SubroutineCall
                        3 _lpython_main_program
                        ()
                        []
                        ()
                    )]
                )
        })
    []
)

@Thirumalai-Shaktivel
Copy link
Collaborator Author

After everything works (List in the LLVM), then I will create a separate pass for pass_wrap_symbols_into_module

@Thirumalai-Shaktivel
Copy link
Collaborator Author

llvm::AllocaInst *ptr = builder->CreateAlloca(type, nullptr, v->m_name);

This AllocaInst is in the decl_vars function to declare variables, I think. My doubt is, here ptr is declared but where does it get loaded or stored for Lists in LLVM?
@czgdp1807

@certik
Copy link
Contributor

certik commented Feb 1, 2023

In here I would simply focus on creating a pass that transform the global statements and variables from TranslationUnit into a module. And you can create integration_test, and test it using ASR tests in tests/. Let's not worry if things do not work with LLVM.

Then in a separate PR let's test that module variables (of all types, including lists) work.

@Thirumalai-Shaktivel
Copy link
Collaborator Author

Thirumalai-Shaktivel commented Mar 7, 2023

I followed the first approach from here: #1491 (comment)
examples/expr2.py seems to work.

@czgdp1807 What do you think about this?

@certik
Copy link
Contributor

certik commented Mar 8, 2023

Looks good so far. You can add a test, get everything passing. Then you can figure out how to change void* into "struct" as needed. Regarding LLVM pointers, use ASR to figure out all the types, not LLVM.

@Thirumalai-Shaktivel Thirumalai-Shaktivel changed the title Move all the symbols from global scope into a function Initial support for global Lists Mar 10, 2023
@Thirumalai-Shaktivel Thirumalai-Shaktivel marked this pull request as ready for review March 10, 2023 07:13
Copy link
Collaborator

@czgdp1807 czgdp1807 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@Thirumalai-Shaktivel
Copy link
Collaborator Author

Then you can figure out how to change void* into "struct" as needed.

Should I try this in this PR? @certik
After getting this done, I will work on importing the global variables.

@Thirumalai-Shaktivel
Copy link
Collaborator Author

Actually, it was easy to figure out.

@certik
Copy link
Contributor

certik commented Mar 11, 2023

There are some test failures.

@Thirumalai-Shaktivel Thirumalai-Shaktivel marked this pull request as draft March 11, 2023 05:49
@Thirumalai-Shaktivel Thirumalai-Shaktivel marked this pull request as ready for review March 15, 2023 10:16
@@ -2296,6 +2297,14 @@ class ASRToLLVMVisitor : public ASR::BaseVisitor<ASRToLLVMVisitor>
}
}
llvm_symtab[h] = ptr;
} else if (x.m_type->type == ASR::ttypeType::List) {
llvm::StructType* list_type = (llvm::StructType *)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this C style cast needed? I would recommend to go for,

Suggested change
llvm::StructType* list_type = (llvm::StructType *)
llvm::StructType* list_type = static_cast<llvm::StructType*>(

@@ -2296,6 +2297,14 @@ class ASRToLLVMVisitor : public ASR::BaseVisitor<ASRToLLVMVisitor>
}
}
llvm_symtab[h] = ptr;
} else if (x.m_type->type == ASR::ttypeType::List) {
llvm::StructType* list_type = (llvm::StructType *)
get_type_from_ttype_t_util(x.m_type);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
get_type_from_ttype_t_util(x.m_type);
get_type_from_ttype_t_util(x.m_type));

Comment on lines +3803 to +3806

void visit_Assert(const AST::Assert_t &/*x*/) {
// We skip this in the SymbolTable visitor, but visit it in the BodyVisitor
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably we should generate this for all the statement types. In a different PR though after this goes in.

@czgdp1807
Copy link
Collaborator

Actually, it was easy to figure out.

CreateBitCast I guess?

@Thirumalai-Shaktivel
Copy link
Collaborator Author

Thirumalai-Shaktivel commented Mar 15, 2023

CreateBitCast I guess?

The thing is I switched from the void* type to the list type, so there is no need to do BitCast, I think.

; ModuleID = 'LFortran'
source_filename = "LFortran"

%list = type { i32, i32, i32* }

@i = global i32 0
@x = global %list zeroinitializer     <-----
...

Comment on lines +2300 to +2306
llvm::StructType* list_type = static_cast<llvm::StructType*>(
get_type_from_ttype_t_util(x.m_type));
llvm::Constant *ptr = module->getOrInsertGlobal(x.m_name, list_type);
module->getNamedGlobal(x.m_name)->setInitializer(
llvm::ConstantStruct::get(list_type,
llvm::Constant::getNullValue(list_type)));
llvm_symtab[h] = ptr;
Copy link
Collaborator Author

@Thirumalai-Shaktivel Thirumalai-Shaktivel Mar 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then you can figure out how to change void* into "struct" as needed.
#1384 (comment)

Sorry, I think I understood this wrong!
@czgdp1807 is the approach used here correct?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using global variable is better. It stays in scope and is independent of all LLVM functions. I checked the generated LLVM code and it looks fine to me.

Copy link
Collaborator Author

@Thirumalai-Shaktivel Thirumalai-Shaktivel Mar 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect!

@u = global i64 0
@x = global i32 0
@y = global i16 0
@z = global i8 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I would keep these initializations here. This is relatively minor, so we can do it in the next PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created an issue for this at #1582.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants