Yield from

[rockneurotiko]

This is a text to explain how yield from works, and why I took some decisions :-)
(In the commits, are so much "white lines" changed, that's because in the original source code are lines with just spaces, or spaces at the end of the line, and my editor (Subl3) remove that)

Yield From

yield from can be applied to Iterables and Futures (A kind of Iterable).

And can be applied in two ways:

• Statement:

def h():
    yield from x()

• Expression:

def h():
    y = yield from x()

About Iterables:

Statement:

Check if the return type of the function is an iterable and the infered of the iterable applied is compatible with the return type.
Ex:

def h() -> Iterator[int]:
    yield from [1, 2, 3]   # Here, List is an Iterable, and have ints inside.

x = h() # the type will be Iterator[int]
for i in x:
    print(i)

In the example above, the function return a Iterator of ints, that is a supertype of Iterable.
Here another example with a function Iterator:

def h() -> Iterator[int]:
    yield from h2()
def h2() -> Iterator[int]:
    yield 1
    yield 2
    yield 3
x = h() # the type will be Iterator[int]
for i in x:
    print(i)

Expression:

Right now just checks if the applied is an Iterable.
In the future, when other things are added to mypy, here should check the
return type, and check the types (just like in statement).
The things that mypy needs to add this to the expression type check:

• Function with yields have a return.

def h() -> Iterator[int]:
    yield 1
    return 2 # Here now fail, but this is a valid syntax

def h2() -> Iterator[int]:
    x = yield from h() # Here when iterate will send the h() iterator
                       # and when the iterator is finished,
                       # will get the return value in x

• yield statement can be an expression assignment.

def h() -> Iterator[int]:
    y = yield 1

This two things have interaction with the new send(), close() and throw() method of the iterables (this should have added too)

About Futures.

I said that Futures are a "kind" of iterables, because it have an iter function but we never iterate over there (we just can do one next(), if try to do any more raises an error). The main loop will catch the futures that had been "yielded from" or that are inserted directly (call_soon, run_until_complete, ...)

So, the idea that I had to write the Future type, and that I've implemented is that one:

• Future type is writed like Iterator: 'Future[Some]'
• But also can be a instance, but have to say the type inside:

x = Future()  # type: Future[int]

• If the function have a Future type as type returned, that function can be "yielded from".
• If a function have the @coroutine decorator, or "yield from" a Future inside, then, that function can be "yielded from", and the type must be 'Future[Some]'. You can always return 'Some', and will be wrapped inside a Future type. (Examples after)
• When yield from a 'Future[Some]', the type returned to a variable assignment will be 'Some', and not the 'Future[Some]'. Right here an example, using asyncio.sleep(float, T), that have this return type: 'Future[T]':

x = asyncio.sleep(100, '1') # the function will return the type:'Future[str]'
                            # (and x will be of that type)
                            # won't sleep (just called, not "yielded from",
                            # but we doesn't care about that)
y = yield from x   # 'x' is a 'Future[str]',and we do a "yield from" to it
                   # so, 'y' will be 'str' type (take out the Future)
                   # (When executed the code, will sleep here, because we
                   # give the future to the main loop and execute it)

So, if "yield from" is applied to a Future, we won't get it (the loop will).
But if we wan't to do a function that can be "yielded from" (this is called coroutine), then the function need to have the return type as Future[Some], and the @asyncio.coroutine decorator (this is explained after)

Statement:

We just don't care about it, we won't get the Future, the loop will, so we don't need to check anything.
Ex:

@asyncio.coroutine
def h() -> 'Future[int]':
    yield from asyncio.sleep(2)
    return 1

We set the return type as Future[int] because we want to let know that this is a coroutine and can be yielded from but, also, that will return an int.
If you do: "y = yield from h()", the program will sleep 2 seconds, and then y will be 1 (the return value).

Actually, if you just call the function, and not "yield from" it, you will have a generator (but can't iterate over there, that raises an exception)

y = yield from h()  # type: int # this is infered
z = h()             # type: Future[int] # this is infered too
# now you can "yield from z" because it's a Future
for i in z:         # This raises an exception, we can't iterate
    print(i)

Expression:

When we check the expression yield from, if we see that the type applied is a Future[Some], then, the type to return (to assignment_stmt) won't be Future[Some], instead of that, we'll return the type Some, because, as said, the Future is just for know where are a coroutine, or where can be "yielded from", but in the assignment expression, the variable get the type returned, not the Future.
Ex:

@asyncio.coroutine        # say that is a coroutine
def h() -> 'Future[str]': # this will be a future, so must be yielded from
                          # and is a coroutine too
    return 'hello!'       # Here we return a str, but see that
                          # is a coroutine, and will wrap the
                          # str in a Future[str]

@asyncio.coroutine
def h2() -> 'Future[None]':
    x = yield from h()    # Here, we can yield from because h is a Future,
                          # but 'x' won't be 'Future[str]', it will
                          # be str
    print(x)

Other things of Futures

• Can I "yield from" a func?.
  • If the function returns Future[Some], yes, you can.
• How wrap the return type into a Future in a function.
  • The @asyncio.coroutine decorator or yield from a Future is needed, when some of that happens, the function is setted as coroutine, and the return type will try to wrap it in a Future.
  • Count how many "Futures" have the function return type against the type returned, and if the difference is up to one, will wrap the type returned.
    Ex:

def h() -> Future[int]:
    return 1  # Here, the difference of "Futures" is one, so the
              # int returned will be wrapped
              # inside a Future as Future[int]

def h() -> Future[int]:
    return asyncio.Future() # type: Future[int]
    # Here nothing is wrapped because the difference is 0
    # but the type check will fail, because the first
    # Future[] of the return type is removed when "yielded from"
    # so, that function would need another Future[] in the function type

def h() -> Future[int]:
    a = asyncio.Future()  # type: Future[Future[int]]
    return a
    # Here nothing is wrapped, but the type check will fail
    # because a Future with an int was expected
    # and a Future with a Future was given.

• The Future type used comes from asyncio.futures.Future, and the type definition is in "stubs/3.4/asyncio/futures.py" so, you will need to do "from asyncio import Future", or use "asyncio.Future" instead of "Future".
  For that reason, the type have to be writted as string ( 'Future[int]' ), if it's not writted in that way, the script will fail when try to run it, because the real Future type is not subscritable.
  I think that in the future, can be a type in "typing", called Futur or some like that, to can write the type without the string.
  Because of that (Future comes from asyncio), now the tests to Futures can't be written (fails trying to import all things)
  Anyway, in stubs/3.4/asyncio I added a directory called "examples" where are so much examples about Futures.
• What if I want a Future that have a Future inside, that have a Future inside, ...?
  You have to do it like if you have a Iterator that return an Iterator, ...:

from typing import Iterator

def h4(x: int) -> Iterator[int]:
    yield x
    yield x+1

def h3(x: int) -> Iterator[Iterator[int]]:
    yield h4(x)
    yield h4(x+2)

def h() -> Iterator[Iterator[Iterator[int]]]:
    yield h3(1)
    yield h3(5)

x = h()
for i in x:   # First Iterator
    for j in i:  # Second Iterator
        for k in j:  # Third Iterator
            print(k)  # Here get the ints

But with Futures!

import typing
import asyncio
from asyncio import Future

@asyncio.coroutine
def h4() -> 'Future[Future[int]]':
    """
    Return a Future[int], and have one more Future anotation saying
    that can be "yielded from".
    """
    yield from asyncio.sleep(1)  # Just to wait a little bit :)
    f = asyncio.Future() #type: Future[int]
    return f   # Here Future[int] is wrapped in Future[Future[int]]

@asyncio.coroutine
def h3() -> 'Future[Future[Future[int]]]':
    """
    Same as h4() but with one more Future :P
    """
    x = yield from h4()    # Get the Future[int] of h4()
    x.set_result(42)       # set 42 as result
    f = asyncio.Future() #type: Future[Future[int]]  # Create a new Future
    f.set_result(x)      # set the result as the Future taked from h4
    return f            # return it

@asyncio.coroutine
def h() -> 'Future[None]':
    print("Before")
    x = yield from h3()   # Get the Future[Future[int]] from h3()
    y = yield from x      # Get the Future[int] from the Future[Future[int]]
    z = yield from y      # Get the int from the Future[int]
    print(z)              # Prints 42
    print(y)              # Prints Future<result=42>
    print(x)              # Prints Future<result=Future<result=42>>

loop = asyncio.get_event_loop()
loop.run_until_complete(h())
loop.close()

Summary

To say that a function can be "yielded from", the return type need to have a Future[Some]

The first Future of the type, is just to indicate that can be yielded from, so, if you
are going to return an int, the type of the function is Future[int] and if you are going to return a Future
that have inside an int, the type of the function is: Future[Future[int]]

The type in the function have to be writted in string.

    def h() -> 'Future[None]':
        pass

See all the examples in "stubs/3.4/asyncio/examples"

[JukkaL]

Thanks for the pull request! I'll give it a spin as soon as I have some time.

[rockneurotiko]

Thank you for this awesome project :)

No problem, is a long PR with so many changes, I tried to explain all in the text, I hope that I did well, my english is not awesome.

The essential commit are 873b1e5, the 5 before this is because I didn't know how to merge all commits in one u.U

[JukkaL]

Sorry @rockneurotiko , I've been busy responding to all the issues and finishing the corrections to my PhD dissertation (and also recovering from a cold) so I haven't been able to review your PR.

[rockneurotiko]

I understand @JukkaL , I'm seeing all the mails in python-ideas and the new issues & PR.

With all that pep-8 merges, this can't be merged "cleanly".

If you want, when have time, check the theory (the text of the PR) and the commit 873b1e5 and if you give me the approval to the idea and/or code (or some changes), I clean the PR merging from the master and do a new clean PR (or maybe add a commit to that one).

I don't know if I explain myself, I don't have so good english XD

Good luck with your PhD and cold :)

[JukkaL]

I went through the new commits. Looks pretty good! There were a bunch of mostly minor things -- see above for detailed comments.

[JukkaL]

@rockneurotiko Are you interested in continuing work on this PR? It's almost there and yield from support would be great to have.

[rockneurotiko]

@JukkaL Yeah, I'm interested in continuing this PR, I'm sorry for the lack of news, I'm in a new work and I have some familiar problems.

I couldn't work in the code, but I readed all the comments and thought about it, the most I think is the way I tried to do the Futures, I will write and ask you about it today or tomorrow, and answer the comments that need it.

Do I write the ask and idea here or I write an email?

P.S: I'm sorry again.

[JukkaL]

No problem, don't push yourself too hard :-)

Feel free to write here or send me an email, either way works for me.

[rockneurotiko]

I write here because I writed it in warkdown.

(Again, sorry for my bad english, feel free to ask anything)

Here is what I thought about the Futures this days.

How is now

I started to think and write the add of "yield from" to mypy thinking primary in asyncio support, and the "Futures/Promises" use that this module make it easy to use.

It's needed to say that I'm learning some other languages for fun, languages like like Scala or Haskell, and when I thinked how to use the Futures in the type definition, the Scala's Futures or the Haskell's Promises ways comes to my mind (More the Scala's way, because the syntax is similar to mypy's syntax)

With that, here is an example that code that I thinked (and with the actual implementation works):

import asyncio
from asyncio import Future

@asyncio.coroutine
def compute(x: int, y: int) -> 'Future[int]':
    """
    That function will return a int, but can be "yielded from", so
    the type is Future[int]
    The return type (int) will be wrapped into a Future.
    """
    print("Compute %s + %s ..." % (x, y))
    yield from asyncio.sleep(1.0)
    return x + y   # Here the int is wrapped in Future[int]

@asyncio.coroutine
def print_sum(x: int, y: int) -> 'Future[None]':
    """
    Don't return nothing, but can be "yielded from", so is a Future.
    """
    result = yield from compute(x, y)  # The type of result will be int (is extracted from Future[int]
    print("%s + %s = %s" % (x, y, result))

loop = asyncio.get_event_loop()
loop.run_until_complete(print_sum(1, 2))
loop.close()

Here is some notes about the code and implementation:

1. The @asyncio.coroutine is optional
2. The return signature of compute is Future[int] but when assigned to "result" in print_sum, the type of result is int.
3. The firs Future[] in the return type definition is just to let know mypy that this function can be used in a yield from
4. Thanks to that syntax, we can set the type as a Future[int] and do a return with an int (Actually, the Iterators functions can do the same in python, but not now in mypy)
5. Also, we can use the functions that accept a function future-like (in-code is determined for have the coroutine decorator or yield from something), like loop.run_until_complete() without do any "type-magic" (just accept a Future[] and the function returns a Future[]).

But this comes with some things that I had to fix:

• Future is actualy a class that can be instanced and used (and yielded from, too), because of that, mypy have to count the Futures setted in the function return type signature and the Futures in the return statement. When the ability to do the same with Iterator[], that will be needed too! Code like that can be used (see that the signature have one more Future):

import typing
import asyncio
from asyncio import Future

@asyncio.coroutine
def h4() -> 'Future[Future[int]]':
    yield from asyncio.sleep(1)
    f = asyncio.Future() #type: Future[int]
    return f

@asyncio.coroutine
def h3() -> 'Future[Future[Future[int]]]':
    x = yield from h4()
    x.set_result(42)
    f = asyncio.Future() #type: Future[Future[int]]
    f.set_result(x)
    return f

@asyncio.coroutine
def h() -> 'Future[None]':
    print("Before")
    x = yield from h3()
    y = yield from x
    z = yield from y
    print(z)
    print(y)
    print(x)

loop = asyncio.get_event_loop()
loop.run_until_complete(h())
loop.close()

• In all the expressions of yield from that add the result type to a variable, the type of that variable will be the type inside the first indicator (Future[] or Iterator[])

x = yield from some_future  # if some_future is Future[int], x will be int
y = yield from some_iter  # if some_iter is Iterator[str], y will be str

What to do?

• Use Iterable word instead of Future word.
• Use some type in typing, some like "Futur" or "Promise"
• Some other that I didn't think.
• Be like it's now.

Personally, I like how is now, seems intuitive to me, for iterators and for futures.

[JukkaL]

Thanks for the update. I've been busy the past couple of days, but I should have time to respond later this week.

[JukkaL]

Sorry for the slow progress. I should have more time this weekend.

[JukkaL]

Looks pretty good! Again, apologies for the slow progress. The last month has been one of the busiest I've ever had...

Some comments below.

• I'm getting a ton of merge conflicts when trying to merge to the latest default. I wonder if you could merge the latest master (or rebase)?
• The examples (under stubs) look more like test cases to me. I recommend rewriting (some of) them as 'evaluation' test cases that use the full stubs. Add a file such as mypy/test/data/pythoneval-asyncio.test (this should be similar to mypy/test/data/pythoneval.test) with your examples as test cases and add the test data file name to mypy/test/testpythoneval.py so that it will be executed. You can run the test cases using python3 -m mypy.testpythoneval. For examples that use sleep, use short sleep times so that the test cases don't run very slowly (e.g., 0.2 seconds rather 2 seconds).
• It would also be nice to have a few simple but realistic asyncio examples under samples/asyncio. As asyncio is mostly useful for networking, it seems natural to have two examples, a server and a client, that can talk to each other. You can also pick some of your existing examples, but I think having just one or two examples that don't involve networking is enough. Having examples under stubs is not great because they will be difficult to find and could be confused with stubs.

[rockneurotiko]

Hi @JukkaL !

I'm sorry for late reply, I'm so busy too. I think that next week I will can do this, after the PyCON Spain :-)

About the comments:

• Yeah, some merges after the last commit :S I will merge (I prefer merge than rebase, but if you like more rebase I do ti).
• Actually, are tests, but when I did, didn't pass the tests because when importing asincio crashes (I don't remember exactly why, I will run again and see if still happend, and comment if the problem still exist).
• Nice idea, I'll pick some of my server/client and change for the samples.

[rockneurotiko]

Hi @JukkaL

After a long day, I've made the merge with the actual master, created full tests for asyncio (only run in python >= 3.4) and removed the examples from the stubs.

I have to write the general samples, but right now can be merged and works all :-)

[JukkaL]

Cool, thanks for making the updates! Now it's looking much better :-)