proposal: go/types: lazy resolution of imported objects

[mdempsky]

cmd/compile uses an indexed export format that allows lazily resolving imported objects so that only the objects that are actually used by user code need to be loaded into memory. However, the go/types API doesn't currently support lazy import resolution, so the gcexportdata importer still needs to eagerly load all objects. (It's able to use the index to skip loading of objects that were already loaded by another importer though.)

This proposal is to add a new function to facilitate lazy import resolution:

func (s *Scope) InsertLazy(name string, resolve func() Object) Object

Semantically from the end user's point of view, this function behaves like:

func (s *Scope) InsertLazy(name string, resolve func() Object) Object {
    obj := resolve()
    if name != obj.Name() {
        panic("invalid resolved object name")
    }
    return s.Insert(obj)
}

except it's unspecified when/whether resolve is actually called.

Importer implementations can then use this function to create lazy objects to add to the Package.Scope without having to fully load all details right away. go/types then assumes responsibility for noticing when it has a lazy object and calling resolve to get the actual object.

For example, a partial sketch of how this might really work:

type lazyObject func() Object

// Unused methods to implement Object.
func (*lazyObject) Pkg() *Package { panic("unresolved object") }
// ...

func (s *Scope) InsertLazy(name string, resolve func() Object) Object {
    if alt := s.elems[name]; alt != nil {
        return alt
    }
    if s.elems == nil {
        s.elems = make(map[string]Object)
    }
    s.elems[name] = lazyObject(resolve)
    return nil
}

func (s *Scope) Lookup(name string) Object {
    obj := s.elems[name]
    if resolve, ok := obj.(lazyObject); ok {
        obj = resolve()
        if name != obj.Name() {
            panic("invalid resolved object name")
        }
        s.elems[name] = obj
        if obj.Parent() == nil {
            obj.setParent(s)
        }
    }
    return obj
}

Other internal code that directly accesses Scope.elems would need to be similarly updated to handle lazyObjects. User code should be unaffected though: the public APIs will always return fully resolved objects.

One possible concern with lazy resolution though is the resolve function for any unresolved objects will need to hang onto the importer resources for constructing them. Which in turn means the Scope, Package, and any other Objects from that same package will also keep those resources live. I would expect this is likely still an improvement over the status quo (i.e., holding onto a single flat byte slice, rather than the complete object/type graph).

Another possible future extension would be for the Scope to only keep a weak reference to the resolved object, and then re-resolve it on demand as needed. I think this would be transparent both to users and importer implementations (as long as the resolve method is safe to invoke multiple times).

/cc @griesemer @findleyr

[mdempsky]

@dominikh points out that this has implications for concurrent calls to Scope.Lookup, which I hadn't considered.

But I think we have a bunch of options on how to keep that working. E.g., either synchronize on the Scope (with a fast path for scopes without any lazy objects) or turn lazyObject into a separate struct with its own internal synchronization.

We can also add a flag to either turn on/off lazy resolution (depending on whether we make lazy importing the default or not), though I think it's probably preferable if we can just make lazy importing always work.

[gopherbot]

Change https://golang.org/cl/323569 mentions this issue: [dev.typeparams] cmd/compile: lazy import resolution for types2

[mdempsky]

CL 323569 has a working example of using this extension for types2. I expect the same design would map easily to go/types.

Two observations from updating the types2 importer to use InsertLazy:

1. Initially, I thought I could just add lazy objects for the directly imported package's symbols, and then lazily add indirect imports as needed. But that caused this test to fail, because it imports "net/http" and then expects sync.Mutex to be Lookup'able:

go/src/cmd/compile/internal/importer/gcimporter_test.go

Lines 380 to 386 in 79bda65

	imports := make(map[string]*types2.Package)
	_, err := Import(imports, "net/http", ".", nil)
	if err != nil {
	t.Fatal(err)
	}
	mutex := imports["sync"].Scope().Lookup("Mutex").(*types2.TypeName).Type()

This wasn't hard to fix, and fixing it actually made the importer somewhat simpler (able to get rid of some extra maps), and is probably necessary anyway to make concurrent Lookups safe. But it surprised me, so I thought I'd note it.

2. Previously the gcexportdata importer broke type cycles by declaring the TypeName early so it would be visible later via Scope.Lookup when importing the underlying type and any associated methods. However, the proposed API doesn't support this, because the object won't be inserted until the resolve function returns.

   Instead I changed to to keep a separate "predeclared" map that holds TypeNames until they've been declared properly. This is a little clumsy, but it turns out to also be useful for indicating when it's safe to Complete any interfaces. And handling recursive defined types is always a little clumsy because of cycles. So I don't think this was too bad of a solution.

[gopherbot]

Change https://golang.org/cl/323629 mentions this issue: [dev.typeparams] cmd/compile: lazy type definitions for types2

[mdempsky]

CL 323629 further adds another API to types2, conceptually equivalent to:

func NewTypeNameLazy(pos token.Pos, pkg *Package, name string, resolve func(named *Named) (tparams []*TypeParam, underlying Type, methods []*Func)) *TypeName {
    obj := NewTypeName(pos, pkg, name, nil)
    named := NewNamed(obj, nil, nil)

    // [This interface for `resolve` is just what was easiest to prototype.
    // It's definitely clumsy though, and I think ideally we'd eliminate
    // either the `named` parameter or the results.]
    tparams, underlying, methods := resolve(named)

    named.SetTParams(named)
    named.SetUnderlying(underlying)
    for _, method := range methods {
        named.AddMethod(method)
    }

    return obj
}

except that again resolve may be called later (or never).

Not only does this allow avoiding resolving defined types whose definitions go unused, but it also completely eliminates the complexity of handling recursive types within the importer, instead letting go/types manage these details. I've also wanted to add lazy type loading to cmd/compile for a while, but it hasn't been practical.

Finally, after prototyping it, I realized that it actually seems somewhat similar to the current instance stuff, so maybe they can actually share a common implementation.