CS2 Discussion: Output: for-in/for-of

[coffeescriptbot]

From @rattrayalex on July 23, 2016 2:59

CoffeeScript and ES6 have very different takes on these features. CoffeeScript's version is arguably more useful, but also more complicated – and non-standard.

What should we do?

Copied from original issue: coffeescript6/discuss#11

[coffeescriptbot]

From @DomVinyard on July 25, 2016 12:11

Non-standard yes, not sure I agree that it's more complicated. I'm not convinced that there is any compelling need to change these.

[coffeescriptbot]

From @carlsmith on July 28, 2016 18:34

Personally, I think CoffeeScript made a mess of this. I would have opted for doing this like Python does. It's less complex, and more powerful.

[coffeescriptbot]

From @kirly-af on July 28, 2016 18:55

Can you elaborate on how you would expect it to work ?

What do you mean Python loops are "more powerful" ? ?
Furthermore, this complexity (I guess you refer to the different for statements in CS) is the result of JS bad design.

The real issue is #3832.

[coffeescriptbot]

From @carlsmith on July 28, 2016 20:3

So, in Python, you only have one kind of for-loop, and it always iterates over a sequence. You can use it on strings, lists and sets.

$ for n in [1, 2, 3]: print n
1
2
3

You can also use it on a dictionary, where it will iterate over the keys.

$ for key in {"foo": 1, "bar": 2}: print key
foo
bar

If you want to iterate over the values, just call values to get a list of them:

$ for value in {"foo": 1, "bar": 2}.values(): print value
1
2

Python also supports unpacking sequences in assignment statements. It's a lot like CoffeeScript's version, but doesn't require any brackets.

$ x, y, z = [1, 2, 3]
$ print x + y + z
6

Python combines the two ideas, and supports unpacking a sequence inside of a for-loop.

$ sequence_of_tuples = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
$ for x, y, z in sequence_of_tuples: print x + y + z
6
15
24

While this doesn't seem that helpful at first, it is very powerful once you have some functions and methods that feed into it. For example, CoffeeScript has its magical fudge for getting the index:

$ (console.log index, char) for char, index in "abc"
0 a
1 b
2 c

In Python, there's no magic involved. You just use the enumerate function that takes a sequence, and returns an enumeration of the sequence, which is essentially a list of index-item tuples.

$ print list(enumerate("abc"))
[(0, 'a'), (1, 'b'), (2, 'c')]

So we can then do this to get the same effect as CoffeeScript, without any magic:

$ for index, character in enumerate("abc"): print index, character
0 a
1 b
2 c

The main advantage of not using magic is that we can use the same feature to iterate over anything we like, not just index-item tuples.

$ for key, value in {"foo": 1, "bar": 2}.items(): print key, value
foo 1
bar 2

$ colors = [(225, 0, 60), (0, 255, 100)]
$ for r, g, b in colors: use(r, g, b)

And so on. There's lots of ways to break a structure into a sequence of pairs or triples or n-length tuples.

Python doesn't need to distinguish between for-in and for-of. A for-loop is a for-loop. It also doesn't need magic to get the index. Python just combines its concept of a basic for-loop and its concept of unpacking a sequence, and everything else flows naturally from there.

Python actually takes iteration much further, but it's all possible because they got their for-loops right, and could build on them.

[coffeescriptbot]

From @carlsmith on July 28, 2016 20:27

Doing things Pythonically would allow us to get rid of for-of loops altogether, and then just create methods or functions for turning whatever we have into whatever we need. In CoffeeScript, that'd look like this:

for key in hash.ownKeys()
for key in hash.allKeys()
for index in array.indexes()
for [item, index] in enumerate(array)
for [r, g, b] in elements.map (element) -> element.backgroundColorTuple()

Due to JavaScript's issues with defining methods on builtins, we'd probably use functions instead (they can be contained in a module these days), but you get the idea. You convert whatever structure you have into a sequence of tuples, and iterate over them.

You can use the same logic in list, dictionary and set comprehensions too.

foo = {
  key.toLowercase(): value * 2
  for key, value in someHash.items()
}

And generator expressions... Well, you get the point. Python nailed iteration.

If we supported unpacking a sequence inside a for-loop when the names are wrapped in brackets (for [key, value] in hash.items()), then we would have the option of using Python style for-loops without breaking backwards compatibility, assuming we want to keep something like the old behaviour as well. The square brackets are also consistent with CoffeeScript assignment syntax ([a, b, c] = [1, 2, 3]).

[coffeescriptbot]

From @JimPanic on July 29, 2016 5:57

This is nice and a lot more intuitive. I support this a lot!

[coffeescriptbot]

From @kirly-af on August 4, 2016 14:46

@carlsmith: I like the idea of unpacking sequences inside for loops.

Though I completely disagree we should introduce new functions and/or methods in the language like the enumerate() you suggest. I personally think CoffeeScript success is (partially) due its idiom It's just JavaScript.

Also, I don't consider the way CoffeeScript provides the index in for-in loops like magical.
The way I see it is that CoffeeScript treats Objects like Maps: a collection of sub-objects, iterable.

In the end, the only real difference we have with array-like values that are intended to work with for-in loops (ie. arrays and strings) is the key type (string vs number).

So to me the actual issue is the inconsistency we have between for-of and for-in:

(console.log key, val) for key, val of {'foo': 'bar'}
(console.log idx, val) for val, idx in ['foo', 'bar']

What would make the more sense to me would be to have only one for (I would choose for-in over for-of but this is not our concern here), and keep being consistent with the way we treat the index/key.

(console.log key, val) for key, val in {'foo': 'bar'}
(console.log idx, val) for idx, val in ['foo', 'bar']
(console.log idx, char) for idx, char in 'foobar'
(console.log count, item) for count, item in someIterrableObject # generator yield, es6 map, es6 set, etc.

Also, it would be easy to assume only value is required if we provide only one parameter to for (apologize I know parameter is not the proper term), we can assume only the value is requested.

(console.log key, val) for key, val in {'foo': 'bar'}
(console.log val) for val in {'foo': 'bar'}
(console.log idx, char) for idx, char in 'foobar'
(console.log char) for char in 'foobar'
# ... and so on

Now we have the main structure of our for loop: for <optionalKey>, value in someCollection.
Then, we could integrate the destructuring on both key and value:

(console.log foo, bar) for {foo: bar} in [{a: 1},{b: 2},{c: 3}]
(console.log thisIsA, mapKey) for {thisIsA: mapKey}, {andA: mapValue} in new Map(...) # maps keys can be any kind of values

This would require more efforts but in a perfect world, this is the way I would expect iteration to be.

[coffeescriptbot]

From @carlsmith on August 5, 2016 2:8

Of course, there are different systems, and they each have there own merits, but Python is known for being especially good at iteration, so in an ideal world, I'd always opt for that approach. It has implications we didn't get into here as well. Python does a lot with iteration.

You don't have to add new functions like enum to the language. You just allow unpacking when more than one name inside brackets is used in a for-loop for [x, y] in coords and then people can add the rest of the functionality with library code and their own stuff. It's just sugar for for coord in coords then [x, y] = coord.

[coffeescriptbot]

From @kirly-af on August 5, 2016 12:40

@carlsmith

Due to JavaScript's issues with defining methods on builtins, we'd probably use functions instead (they can be contained in a module these days), but you get the idea. You convert whatever structure you have into a sequence of tuples, and iterate over them.

That's the part of your post that confused me concerning the possible introduction of new functions.
I got your point about unpacking sequences and I completely agree this would be a great benefit to be able to use it in for loops as you demonstrated.

Though, my point was more about how to unify the different kind of CoffeeScript for loops, and so the inconsistency between for-in and for-of (ie. for val, index in vs for key, val of). Furthermore, we haven't talked yet about ES6 iterables. AFAIK those need to be integrated as neither for-in nor for-of loops can handle it.

Thus, a for-from form has been suggested to handle iterables in jashkenas/coffeescript#3832 (the whole issue is worth reading btw).
Should we introduce a new form ? Or should we unify it through a unique single for-in ?
I'd vote for the second option. The only issue is that it would involve if statements in the transpiled code, like suggested in that comment. I personally this would be better option.

Having 3 different kinds of for loops sounds crazy to me, there should be only one.

[coffeescriptbot]

From @DomVinyard on August 8, 2016 9:13

I really like @kirly-af's example

(console.log key, val) for key, val in {'foo': 'bar'}
(console.log idx, val) for idx, val in ['foo', 'bar']

[coffeescriptbot]

From @carlsmith on August 8, 2016 14:55

What would it compile to? We have to assume we don't know the type of the iterable, as it will often be an expression that evaluates to something that's defined elsewhere.

[coffeescriptbot]

From @carlsmith on August 8, 2016 15:15

Having 3 different kinds of for loops sounds crazy to me, there should be only one.

Adding unpacking to loops doesn't create a new type of loop. It just makes the assignment in for-loops consistent with normal assignments. We could do what you're suggesting, and still allow unpacking in for-loops. We could have any kind or any number of for-loops, and still allow assignments to be unpacked within them.

I agree we should only have one kind of for-loop, but don't think it should have pseudo-unpacking, with only two names, and hardcode what gets unpacked based on the iterable's type. If we ever did create a dialect of CoffeeScript with one kind of for-loop, I'd much prefer Python's iteration, which handles all iterables, including instances of user-defined classes, and is far more flexible.

[coffeescriptbot]

From @carlsmith on August 8, 2016 16:16

Sorry if that came across badly. I didn't mean to be rude. I appreciate you're trying to come up with something as close to CoffeeScript as possible, but sane as well, and in that context, what you have looks good.

I'm not certain, but I think CoffeeScript is inconsistent because it doesn't know the types, so it's difficult or impossible to cover every case correctly in the compiled code without injecting a function call for the type check, and they don't like to do that because of performance. Injecting a type check invocation wouldn't be too expensive as the iterable expression is only evaluated once (not per iteration, like while and until), so you probably could do what you're suggesting without any serious performance issues. Still, if you're changing the language enough to only have one kind of for-loop, why not just redo for-loops altogether?

[coffeescriptbot]

From @kirly-af on August 8, 2016 17:8

Thanks @carlsmith for sharing your opinion, no worries I've taken no offense ! Sorry if that comment is going
to be long but I have many comments to answer.

Adding unpacking to loops doesn't create a new type of loop. It just makes the assignment in for-loops consistent with normal assignments.

Just to avoid misunderstandings, I confirm you I got your point here. Unpacking is a good thing with or without the new loop form/behavior.

What would it compile to?

Mostly inspired from this comment, I imagine something like:

foreach key, item in collection

would compile to:

for (var _iterator = collection,
  _i = 0,
  _it, _keys,
  _isArrayLike = Array.isArray(_iterator) || typeof _iterator === 'string',
  _isIterrable = !_isArrayLike && typeof collection[Symbol.iterator] === 'function',
  _iterator = _isIterrable ? _iterator[Symbol.iterator]() : _iterator;;
  ++_i)  {
  var _ref, _ref2;

  if (_isArrayLike) { // collection is array or string
    if (_i >= _iterator.length) break;
    _ref2 = _i;
    _ref = _iterator[_i];
  } else if (_isIterrable) {
    _it = _iterator.next();
    if (_it.done) break;
    _ref2 = _i;
    _ref = _it.value;
  } else { // object
    _keys = _keys || Object.keys(_iterator);
    if (_i >= _keys.length) break;
    _ref2 = _keys[_i];
    _ref = _iterator[_ref2];
  }

  var key = _ref2;
  var item = _ref;

  // loop body
}

This works for strings, arrays, objects and iterables.

Injecting a type check invocation wouldn't be too expensive as the iterable expression is only evaluated once (not per iteration, like while and until), so you probably could do what you're suggesting without any serious performance issues.

Indeed, here I keep the if statements inside the loop, and I don't even see any overhead on 100,000 items collections (every type of collections). It could be preferable to copy the loop in 3 different blocks but, referring to that comment:

Copying the entire loop body is not at all practical.

I don't know how reliable this comment is (though still babel author words).

We have to assume we don't know the type of the iterable, as it will often be an expression that evaluates to something that's defined elsewhere.

I miss your point. Could you provide an example of such case ?

I agree we should only have one kind of for-loop, but don't think it should have pseudo-unpacking, with only two names, and hardcode what gets unpacked based on the iterable's type.

I admit unpacking the index (I wouldn't use this term in this case) is questionable. Especially if we consider ES6 (Weak)Map, as the proposed foreach would require unpacking anyway to use it properly:
e.g.:

map = new Map [
  ['foo', 'bar']
  ['bar', ['foo']
]
(console.log count + ':', item) foreach count, item in map
# ouput: count: [key, value]
# 0: ['foo', 'bar']
# 1: ['bar', 'foo']

This is manageable and ideally should behave exactly like Objects.

If we ever did create a dialect of CoffeeScript with one kind of for-loop, I'd much prefer Python's iteration, which handles all iterables, including instances of user-defined classes, and is far more flexible.

As long as we have a way to handle iterables, we can handle user-defined classes (or more exactly objects). AFAIK, an iterable is just an object wich Symbol.iterator property yields iterators (ie. a generator).

Still, if you're changing the language enough to only have one kind of for-loop, why not just redo for-loops altogether

As I explain in coffeescript6/discuss#16, I changed my mind a little bit and I am now in favor of a new form. Of course the only reason to introduce a new kind of loop is to preserve the old behavior, without any possible overhead for those who would hate the feature (you know there would be MANY). I dislike for-from, and would suggest foreach-in.