Stack `ParseQuery.aggregate()` on top of a constructed Query

[RaschidJFR]

Is your feature request related to a problem? Please describe.
Currently ParseQuery.aggregate() ignores all the conditions of a query created with the Query functions: equalTo(), matchesQuery(), matchesKeyInQuery, or event ParseRelation.query()...

So it becomes extremely difficult to aggregate complex because aggregate() relies only on the pipeline argument.

Example:

return Parse.Query
      .or(
        this.relation('characters').query(),
        new Parse.Query('Character')
          .equalTo('objectId', this.get('character').id),
        new Parse.Query('Character')
          .matchesKeyInQuery('objectId', 'character.objectId',
            this.relation('clues').query())
      )
      .aggregate(pipeline);  // Using aggregate will ignore the previous conditions

Describe the solution you'd like
ParseQuery.aggregate() should merge the previous conditions with the ones in the pipeline.

Describe alternatives you've considered
Right now, most of the requirements can be achieved by adding the conditions in a $match statement at the beginning of the pipeline:

const q = Parse.Query
      .or(
        this.relation('characters').query(),
        new Parse.Query('Character')
          .equalTo('objectId', this.get('character').id),
        new Parse.Query('Character')
          .matchesKeyInQuery('objectId', 'character.objectId',
            this.relation('clues').query())
      );

const pipeline = [
  { match: q.toJSON().where },
  { ... }
];

return new Parse.Query('Character').aggregate(pipeline);

Additional context
Other constraints like include(), limit(), etc will not be passed in where, but those can be added easily at the end of the pipeline if required.

[mtrezza]

Thanks for opening the issue for your PR.

ParseQuery.aggregate() should merge the previous conditions with the ones in the pipeline.

What do you mean by merging, can you give some examples?

Other constraints like include(), limit(), etc will not be passed in where, but those can be added easily at the end of the pipeline if required.

Which conditions would be translated into the pipeline?
How do you handle conditions that are not translated into the pipeline?

So it becomes extremely difficult to aggregate complex because aggregate() relies only on the pipeline argument.

That is what the pipeline is for - it allows to describe queries that may be too complex for traditional operators. In my view a query becomes more opaque when it is partly declared in Parse conditions and partly in a pipeline. In addition, some conditions may be discarded because they are not translated into the pipeline, which makes it even harder to debug.

Currently ParseQuery.aggregate() ignores all the conditions of a query created with the Query functions

Which makes this a breaking change; not a stopper but to consider.

General concerns:

• Aggregation is an alternative to traditional MongoDB operators, that is why it's not possible to db.collection.find(...).aggregate(...), so this would be counter-intuitive to the MondoDB syntax.
• Aggregation has different mongoDB-node resource implications than an operator, because the aggregation framework is executed differently than an operator. In other words, find({key:"value"}) is not the same as aggregate([{$match:{key:"value"}}]). It makes sense to transparently separate traditional vs. aggregate query for the developer and to avoid mixing them syntactically.

[RaschidJFR]

What do you mean by merging, can you give some examples?

Same example: this query should aggregate the results of the original OR query, but it does not:

return Parse.Query
      .or(
        this.relation('characters').query(),
        new Parse.Query('Character')
          .equalTo('objectId', this.get('character').id),
        new Parse.Query('Character')
          .matchesKeyInQuery('objectId', 'character.objectId',
            this.relation('clues').query())
      )
      .aggregate(pipeline);  // Using aggregate will ignore the previous conditions

Which conditions would be translated into the pipeline?
How do you handle conditions that are not translated into the pipeline?

Ideally, all of them. As a developer you would like to aggregate the whole query you've constructed so you can leverage the friendly SDK constraints, instead of manually building their equivalent into a mongodb pipeline. I think the less pipeline you have to write, the better.

I'm not proposing to replace find({key:"value"}) with aggregate([{$match:{key:"value"}}]) neither to use find().aggregate(), but to smoothly take the most we can of the query constraints (equalTo, or, and, matchesKeyInQuery, etc) to work automatically with aggregate, so you can happily do:

ParseQuery.equalTo('key', value).aggregate({ group: {...} })

The SDK's ParseQuery is mature and doing very well for retrieving segmented data easily, but aggregate() does not leverage it because you have to build the whole pipeline from scratch. A normalized database using all the Parse SDK features becomes extremely difficult to aggregate. For instance, how could we translate the sample query into a mongo pipeline? I wouldn't even know where to start!

So I think this would count towards making real-world production-grade apps easy for developer. That's what this SDK is about, right?

[RaschidJFR]

So, any thoughts?

[mtrezza]

For instance, how could we translate the sample query into a mongo pipeline? I wouldn't even know where to start!

Well, this sums up my point of critique pretty well. The pipeline command is an alternative for when the existing command set is insufficient to articulate the complexity of a query. The pipeline tool allows to write more complex queries, so to look for a way to avoid diving into how to properly write a pipeline query seems contradictory. 🙂

The main issue I see is incompatibility. Not every find command has an equivalent aggregate command and vice-versa. Or sometimes there is an equivalent but it has more / less / different parameters. And aggregate is an ordered pipeline, but find is without order. Many reasons why the two are strictly separated in mongodb.

Imagine this example:

query.equalTo("fieldA", "1");
query.limit(10);

Would that convert into:

[{ $match: { fieldA: 1 }}, { $limit: 10}]

or

[{ $limit: 10}, {$match: { fieldA: 1 }}]

Now you may decide for one of these options, but as soon as someone needs the other option, the conversion feature becomes partly useless. And that is true for every query with more than 1 command, because order matters.

The command order and stage structure are crucial aspects when writing an efficient query. The reason is that the mongodb pipeline optimizer refactors certain commands for efficient execution. You also don't want to carry unnecessary data into the next stage. So writing a pipeline really only makes sense when you control the whole pipeline.

I think that the alternative you described in #1171 (comment) makes more sense. Converting the query the other way around gives you control over command order in the pipeline. You can also create 2 queries and put each one into the pipeline at the stage you want:

const q1 = Parse.Query(...);
const q2 = Parse.Query(...);

const pipeline = [
  { match: q1.toJSON().where },
  { match: q2.toJSON().where },
];

But that example really only works for $match, it can get complicated with other commands. How do you convert a $near or $geoWithin command into a $geoNear stage that has an additional $query parameter for example?

What do you think?

[mtrezza]

For reference, something similar already exists for mongo -> parse, but excluding aggregation: https://github.com/yog27ray/mongoToParse