TypeGuard applied outside of it's scope

[ikamensh]

Hello, below is a suspected bug / inconsistent behavior I couldn't understand.

In the code below, a value is passed through to a submethod with same typing annotation, but an error is generated.

Code:

    def __init__(
        self,
        low: Union[SupportsFloat, np.ndarray],
       ...
    ):
        if shape is not None:
            shape = tuple(shape)
        elif not np.isscalar(low):
            shape = low.shape  # type: ignore
        elif not np.isscalar(high):
            shape = high.shape  # type: ignore
        else:
            raise ValueError(
                "shape must be provided or inferred from the shapes of low or high"
            )
        low = _broadcast(low, dtype, shape, inf_sign="-")  # error


def _broadcast(
    value: Union[SupportsFloat, np.ndarray],
    ...
) -> np.ndarray:
        ...

Error:

/home/runner/work/gym/gym/gym/spaces/box.py
  /home/runner/work/gym/gym/gym/spaces/box.py:53:32 - error: Argument of type "SupportsFloat | ndarray[Unknown, Unknown] | generic | bool | int | float | complex | str | bytes | memoryview" cannot be assigned to parameter "value" of type "SupportsFloat | ndarray[Unknown, Unknown]" in function "_broadcast"
    Type "SupportsFloat | ndarray[Unknown, Unknown] | generic | bool | int | float | complex | str | bytes | memoryview" cannot be assigned to type "SupportsFloat | ndarray[Unknown, Unknown]"
      Type "generic" cannot be assigned to type "SupportsFloat | ndarray[Unknown, Unknown]"
        "generic" is incompatible with protocol "SupportsFloat"
          "__float__" is not present
        "generic" is incompatible with "ndarray[Unknown, Unknown]"
      Type "complex" cannot be assigned to type "SupportsFloat | ndarray[Unknown, Unknown]"
        "complex" is incompatible with protocol "SupportsFloat"
          "__float__" is not present (reportGeneralTypeIssues)
1 error, 0 warnings, 0 infos 

CI:

Run jakebailey/pyright-action@v1
  with:
    version: 1.1.204
    python-platform: Linux
    python-version: 3.7
    no-comments: true
    lib: false
    warnings: false
  env:
    PYRIGHT_VERSION: 1.1.204
    pythonLocation: /opt/hostedtoolcache/Python/3.10.1/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.1/x64/lib

Complete source code: https://github.com/boschresearch/gym/blob/df82bdc27bc877bcaf98aecb203bc07fbea19ceb/gym/spaces/box.py#L53

Full CI log:
https://github.com/openai/gym/runs/4789425067?check_suite_focus=true

[JelleZijlstra]

Does np.isscalar return a TypeGuard? I suspect that's affecting the type afterwards.

[ikamensh]

I've looked it up, you are right isscalar in the latest numpy is using TypeGuard. https://github.com/numpy/numpy/blob/4adc87dff15a247e417d50f10cc4def8e1c17a03/numpy/core/numeric.pyi#L575

Yet the line where the problem is reported is outside of any if / else conditioned on a typeguard.

[ikamensh]

Update: this error only occurs when following code is present before the statement where error occurs:

if shape is not None:
  shape = tuple(shape)
elif not np.isscalar(low):
  shape = low.shape  # type: ignore
elif not np.isscalar(high):
  shape = high.shape  # type: ignore
else:
  raise ValueError(
      "shape must be provided or inferred from the shapes of low or high"
  )

I've updated the initial issue description to include this code.

[erictraut]

Thanks for the bug report. When reporting problems like this, it's really helpful if you can produce a minimal, self-contained code sample.

In this particular case, the problem comes down to this:

from typing import SupportsFloat, Union
import numpy as np

def func(val: np.generic):
    x: Union[SupportsFloat, np.ndarray] = val

The issue is that np.generic isn't compatible with either the type SupportsFloat or np.ndarray.

I think pyright is correct in generating an error here.

If you think this should not generate a type error, please consider reporting it to the numpy maintainers.

[ikamensh]

I disagree, because I don't see how in my case low becomes of type np.generic. It's passed as SupportsFloat | np.ndarray and is interpreted as possibly np.generic only after following block of code:

        if shape is not None:
            shape = tuple(shape)
        elif not np.isscalar(low):
            shape = low.shape  # type: ignore
        elif not np.isscalar(high):
            shape = high.shape  # type: ignore
        else:
            raise ValueError(
                "shape must be provided or inferred from the shapes of low or high"
            )

Removing following lines, which deal with another variable alltogether, removes the error:

        elif not np.isscalar(high):
            shape = high.shape  # type: ignore

I will try to produce more minimal example, but I don't think the example you have provided captures this problem.

[erictraut]

The np.isscalar function is defined as a user-defined type guard that uses the TypeGuard mechanism defined in PEP 647. (I'm the author of that PEP, BTW.)

def isscalar(element: object) -> TypeGuard[
    generic | bool | int | float | complex | str | bytes | memoryview
]: ...

This is telling the type checker that the value passed in should be "narrowed" to type generic | bool | int | float | complex | str | bytes | memoryview if the function returns True. Note that generic is included in this union.

[JelleZijlstra]

Yes, this is a case where we'd want TypeGuard to narrow the Union instead of setting the type to exactly the TypeGuard argument.

[ikamensh]

Thank you for this PEP, it's very much appreciated. I think I have decent grasp of what TypeGuard currently does.

Here is a minimal example, having nothing to do with numpy, that demonstrates the issue:

from typing import TypeGuard


def tg(x) -> TypeGuard[str | int]:
    return isinstance(x, (str, int))


def foo(y: str) -> None:

    if 1 % 2:
        pass
    elif tg(y):
        print("CASE 1")  # Type is narrowed here, but it's not important for this example
    else:
        raise Exception("Invalid case")

    # Since we always take 1%2 branch, y is still a `str`
    z: str = y  # bla.py:19:14 - error: Expression of type "str | int" cannot be assigned to declared type "str"

[erictraut]

Yeah, the example above is an incorrect usage of TypeGuard for the reason you pointed out.