Built-in filtering and sorting for iterative sections? - RFC 0.10

[evs-chris]

Iteration is one of the most common constructs in view libraries, and iteration often needs filtering and/or sorting applied to it. You can currently achieve this to some extent with an expression in ractive, but there are some strong disadvantages, as the expression breaks the connection between the source array and the items in the template e.g. {{#each articles}} -> {{#each filtered(articles)}} puts the iterations in a computed context. 0.9 has support for bindings in computed contexts, but they're not cheap for non-trivial arrays or expressions.

To address those issues, I'd propose that we adjust the iteration syntax a little and introduce special aliases to handle filter, sort, limit, and offset. With the special aliases applied, the iterative section would adjust the array before it rendered/updated any of the iterations. It would also subscribe to each of the special aliases, so that any updates to them or their deps would cause the section to update as well. Here's roughly what the render cycle would look like:

Render process

1. Get start array.
2. Apply filter resulting in a mapping of index to index or -1.
3. Apply sort resulting in a further mapping of index to target position.
4. Scope the mapping using offset and limit.
5. Unrender any iterations that should no longer be visible and render those that should based on the mapping.
   • Optionally, render transitions could wait on unrender transitions to complete, which would make items appear to leave the list and re-enter at a new position. That could also be done for shuffled sets.

Syntax change

The index and/or key for an iterative section can currently be aliased by using a : after the expression and providing names separated by commas. That doesn't really work with additional things, and it's extra code for a function that's already handled by the normal aliasing that came along later. To accommodate additional special aliases, the index refs would be dropped and replaced with aliases:

{{#each items: i}} ... {{/each}}
{{#each items as item: i}} ... {{/each}}
{{#items: i}} ... {{/}}
{{#each obj: k, i}} ... {{/each}}
{{#each obj as val: k, i}} ... {{/each}}
{{#obj: k}} ... {{/}}
would become
{{#each items with @index as i}} ... {{/each}}
{{#each items as item, @index as i}} ... {{/each}}
{{#items with @index as i}} ... {{/}}
{{#each obj with @key as k, @index as i}} ... {{/each}}
{{#each obj as val, @key as k, @index as i}} ... {{/each}}
{{#obj with @key as k}} ... {{/}}

Everything gets a little longer, but it is also considerably more clear. The additional special aliases could appear anywhere in the alias list e.g.

{{#each items as item, someFilter as filter, 10 as limit, page * 10 as offset, @index as i}} ... {{/each}}

Arrows

There are a number of helper libraries that would greatly benefit from some sort of limited function syntax supported by the expression parsing available to templates. Filtering and sorting would also benefit tremendously, so I'd propose that short-form arrows that compile to plain functions be added to the expression syntax e.g.

() => num * 20; // function() { return num * 20; }, which expressions out to function() { return _0 * 20; } with num as a dep subbed in at _0
x => x * x; // function(x) { return x * x; }
(a, c) => c + (a * num); // function(a, c) { return c + (a * _0); }
item => regex.test(item.prop) || regex.test(item.otherProp); // function(item) { return _0.test(item.prop) || _0.test(item.otherProp); }
(l, r) => l.name.localeCompare(r.name); // function(l, r) { l.name.localeCompare(r.name); }

I haven't fully thought through the implications of inlining functions into the existing expression constructs, but from a first pass point of view, it looks like it would work nicely. Limiting it to the simple arrow syntax also significantly limits the hairier issues that come with function support, and I'd basically ignore binding, as this references get subbed out to context either way.

New special ref

Since @index is relative to the source array, we'd probably need a new special ref that represents the index of the iteration in the final collection after filter/sort is applied. I'm not sure what that would be called at this point.

Thoughts?

Opinions? Observations?

[evs-chris]

Another, much less involved option that I've considered but not really come up with a good way to do is have some sort of pragma that tells an iterative section to look up computed items in a source array in order to map to the original path rather than the computed path. Something like:

{{#each compute(list)}} {{!pragma 'mapToSource' list}} ... {{/each}}
or maybe
{{#each compute(list): i,k <- list}} ... {{/each}}

where the actual contexts within the iterations would be list.${@index} models rather than compute(list).${@index} models.

[fskreuz]

• Arrow functions inside expressions? Without thinking it through as well, yes please 😄

• I suggest creating another section type. This way, we avoid overloading {{#each }} and keep it simple. It also eliminates the pragma option which I find strange. Mapping to source would then be the default behavior. For this I suggest the following

  // PHP-like foreach
  {{#foreach items as item }} ... {{/foreach}}
  
  // Python-like list comprehension
  {{#for item in items }} ... {{/for}}
  

  I'm leaning towards for to clearly distinguish it from each (foreach is a bit too close).

• Allow some form of destructuring?

  // [{ prop1: '...', prop2: '...'}, ...] or { key1: { prop1: '...', prop2: '...'}, ... }
  
  {{#for (prop1, prop2) in items  }} ... {{/for}}
  

• Maybe drop as in the sort and friends portion? One less term to remember. Suggesting filterBy, orderBy, offsetBy and limitBy followed by their arguments.

                       .............              .............
  {{#for item in items filterBy expr orderBy expr offsetBy expr limitBy expr }}
                                     ''''''''''''               ''''''''''''
  

• Can we also drop punctuations? Here's @Rich-Harris 's ideas regarding Angular's filter syntax.

• For the index reference inside this special section... @offset?

[andreujuanc]

• Just comparing:
  {{#each items as item, someFilter as filter, 10 as limit, page * 10 as offset, @index as i}} ... {{/each}}
  and
  {{#for item in items filterBy expr orderBy expr offsetBy expr limitBy expr }}
  Made me think of joining both: (prefer to keep just one section type for iteration) I do like the cleaner syntax that @fskreuz proposed.
  {{#each items as item filterBy expr orderBy expr offsetBy expr limitBy expr }} ... {{/each}}

I would separate the expressions from aliases. Aliases can just be on pragma? since they are less of a functional thing. Maybe if the original names are good enough, there will be less need for aliases.

This is a really good addition, but once it's implemented documentation should be created for it.! 💃

• Arrows: (points)=> ++points

• Destructuring is nice but I I can't think of any use for it as for my projects as of today. But you know, it's like a life insurance :) it's better to have it.

• New special ref:
  Lets say I have a 50 items list. 10 as limit, 10 as current offset. it's rendering from index 10 to 19. I think there are tree concepts that are important: "page index", "computed index" and "source index" and they should have totally different but easy to remember names.

@index by default should stay as source index.

@computedIndex is nice but maybe too long?

@offset sounds more like the current page offset, more than the @computedIndex maybe @viewIndex or @renderIndex I think the word index is good 🐱

Looking forward to help in these features.

[dagnelies]

Hi,

seems you are in the mood to do some stuff @evs-chris 😉

I'll split the RFC into two parts.

arrows

to make it short: 👎
my reasoning is simple: IMHO, declaring functions should not be in templates.
I consider it bad practice to do so, never had the need to, and is a source of complexity.

filtering/sorting/range

Funnily enough, this reminds me of my dataset plugin:
http://dagnelies.github.io/ractive-dataset/
where you declare:

<dataset data="{{users}}" filter="{{filter}}" sortby="{{sortby}}" from="{{from}}" to="{{to}}">...whatever you want with the resulting user subset...

...and I consider it a relatively usual subject people encounter.

However, the proposal how to solve this leaves me luke warm. It introduces lots of keywords, is verbose, and not very flexible.

Alternative

As alternative, I'd suggest pipes/filters/transforms (|) so that it would look like this:

{{#each items | filterBy expr | orderBy expr | range from to }}

However, it would also differ from angular filters or ractive functions/expressions because these pipes have a more specific meaning. Their "contract" is: the items contained in the result are a subset of the input. They can be re-anranged, filtered, etc, but the items must remain the same, so that the data-binding can still be safely ensured.

In practice, a ractive instance could look like this:

var r = new Ractive({
...
pipes/filters/transforms: {
    customTransform: function(list, arg1, arg2) {
        // returns a subset list of the indexes/keys, possibly reordered
        // so that ractive knows what to iterate over while keeping the binding
    }
}

Unlike normal functions, they're just here to filter/reorder the lists, so that the bindings can be kept. They are also flexible enough to allow customized filtering/sorting rules.

Side notes

I kind of like @fskreuz's suggestion about {{#for item in items}}. It's kind of nice/familiar. 😉

[kouts]

@evs-chris I really like the idea of filtering and sorting but won't this add even more codebase?
I'm feeling Ractive's size is getting bigger and bigger, maybe this sort of things can be handled via plugins
as the one posted from @dagnelies

Btw @dagnelies nice plugin!

[evs-chris]

@kouts it would add more to the codebase, but it would also replace some redundancy and refactor one of the larger, hairier chunks of code (RepeatedFragment). It might end up a net loss. Unfortunately, this is a really close to the metal change, so a plugin couldn't handle it.

Weight Gain

On that note, I think the planned adaptor and transition refactors will also shoot for adding some modularity to be able to leave features out if they're not needed. Transitions alone account for nigh 10% of the codebase, and maybe a good bit more if you include supporting code like that in the runloop. I think stripping that down to bare hooks and a simpler limited builtin would let more flexible transitions take over as plugins (notably using animations rather than transitions).

Syntax

As far as syntax goes, I was trying to stick with the existing syntax for aliases. I don't think pipes will happen because they are legit operators, if rarely used outside of bitmasking flaggy code. I would like some sort of delimiter in there because {{#each items filterBy x => /joe/i.test(x.name) sortBy compare('name') limit 10}} runs together a bit much for me. Aliasing was inspired by SQL syntax, in which the as is optional, so we could probably eliminate that as a requirement for this and aliasing in general. That would remove a little bit of the extra noise. Alias benefits:

• There's already parser code that handles them
• They already get along with references and expressions
• They don't require a special order and any can be optional
• They have a good spot in the template structures, meaning no extra props have to be added to the section struct

The pragma was an alternate solution with a keyword/operator alternate itself 😄. It would basically let you do whatever computation you wanted to on an array and then look each item up in the given source to use that as context. The downside there is that shuffling can't work and twiddling keypaths will get really weird downstream from the iterations. It would be much less code though, and the weirdness could probably be mitigated, leaving only issues with shuffling, which computed iterations already have.

[petterek]

I do not get the idea here...
When we want to do such action as described here we splice our array to a new one with the data we want to show.
Altering that array is simple and can be done when ever something changes.

[ceremcem]

+1 for the pipe syntax | (even |> might be considered, since it's also experimental in Javascript, is used in Livescript and is now used in Coffeescript, is already used in Elixir, F# and so on).

+1 for automatic destructing data, as we are familiar with Livescript:

a = 
  * x: 1
    y: 2
  * x: 5
    y: 8
  * x: 4
    y: 14

for obj in a 
  console.log "x: #{obj.x}, y: #{obj.y}"

for i, {x,y} of a
  console.log "index: #{i}, x: #{x}, y: #{y}"

Here we can see there is both in and of operators are used for similar purposes.

Putting all of them together, here is a valid Livescript code (run here) which would have very much similarities with above:

a = 
  {x: 1, y: 2}
  {x: 5, y: 8}
  {x: 4, y: 14}

for i, {x,y} of a |> filter (.y < 9) |> reverse
  "index: #{i}, x: #{x}, y: #{y}"

# => ["index: 0, x: 5, y: 8","index: 1, x: 1, y: 2"]

...where above loop will be presented in the template:

{{#each i, {x, y} of a |> filter myFilterHelper |> reverse}}
  <p>index: {{i}}, x: {{x}}, y: {{y}}</p>
{{/each}}

to use special references and very long chains:

{{#each ...}}
  {{#each {x, y, k: @key, i: @index} in a 
      |> filter myFilterHelper 
      |> group-by something
      |> sort-by someOther
      |> drop 1
      |> reverse
      |> take myPageSize}}
    <p>outer index: {{i}}, outer key: {{k}}, this index: {{@index}}, x: {{x}}, y: {{y}}</p>
  {{/each}}
{{/each}}

(which reminds me #2226, btw :D)

[dagnelies]

Although all these alternatives to JS are nice, they have the downside of being too many of them and have an uncertain future. Although if I had to pick one, I would take typescript rather than livescript because of its vast ecosystem.

That said, I have a slight preference for | but of course the |> is ok too:

1. {{#each items | filterBy expr | orderBy expr | range from to }}
2. {{#each items | filterBy(expr) | orderBy(expr) | range(from, to) }}
3. {{#each items |> filterBy expr |> orderBy expr |> range from to }}

...they are quite similar in the end.

PS: related to a past remark, despite | is actually a bitwise operator in JS, I think it's so rarely used that it could actually used for that purpose in templates.

Note also that IMHO these pipes have a more specific meaning. Their "contract" is: the items contained in the result are a subset of the input. They can be re-anranged, filtered, etc, but the items must remain the same, so that the data-binding can still be safely ensured.

[andreujuanc]

I'm restarting a new project that's gonna take advantage of this. In some tests i've done looks like computing the array is very expensive, and if this solution gonna help, even just a bit , it will be very welcome.

Regarding to syntax, i think i prefer @evs-chris 's comma separated expressions proposal, since it's what Ractive already does with events: on-click="a, b, c, d", this way syntax stay similar all around, and not commas here, pipes and arrows over there.

EDIT: Forgot to say that |> looks very sexy tho.

[dagnelies]

In some tests i've done looks like computing the array is very expensive

No idea what you are talking about :P

Ractive already does with events: on-click="a, b, c, d", this way syntax stay similar all around, and not commas here, pipes and arrows over there.

Just to avoid confusion: whether arrow functions are supported or not is completely independent from using as or | pipes. ...and what's on-click="a,b,c,d" ?!?

---

Lastly, and most importantly, the as in the original proposal is IMHO very misleading.

In the first case @index as i, it's aliasing/relabelling. In the second case someFilter as filter it's specifying a filter called someFilter. It's the same syntax, used in the same mustache, for two completely different things.