Invalid reference '_id'! Error In spring data mongodb [DATAMONGO-2644]

[spring-projects-issues]

Deepak Patankar opened DATAMONGO-2644 and commented

I have written my aggregation query in spring data mongodb. In projection step I want to use the "_id" field, but it gives error.I have written my aggregation query in spring data mongodb. In projection step I want to use the "_id" field, but it gives error.
My Projection Stage:

private ProjectionOperation getProjectionForStartAndEndTime(long timeUnit) {
  return Aggregation.project()
      .and(Fields.UNDERSCORE_ID)
      .as("startTime")
      .and(Add.valueOf(Fields.UNDERSCORE_ID).add(timeUnit))
      .as("endTime").andInclude("successfulActivitiesCount");
}

Before this I had a bucket stage, I suspect whether that could be the reason ?

private BucketOperation getBucketOperation(TimeGroupType timeGroupType, long startTime, long endTime) {
  List<Long> timeIntervalList = getBoundariesForAggregating(timeGroupType, startTime, endTime); // The time list is [1603115723209, 1603202123209, 1603288523209,1603374923209,1603461323209,1603547723209]
  return bucket(ActivityHistoryEntityKeys.activityTime)
      .withBoundaries(timeIntervalList.toArray())
      .andOutput(isSuccessfulActivityField).sum()
      .as("successfulActivitiesCount");
}

The error is :

java.lang.IllegalArgumentException: Invalid reference '_id'! at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:114) at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:77) at org.springframework.data.mongodb.core.aggregation.ProjectionOperation$ProjectionOperationBuilder$FieldProjection.renderFieldValue(ProjectionOperation.java:1445) at org.springframework.data.mongodb.core.aggregation.ProjectionOperation$ProjectionOperationBuilder$FieldProjection.toDocument(ProjectionOperation.java:1432)

The _id field will always be present in the document, then why does it give an invalid reference. 

I think that there was a similar bug in spring data MongoDB, but that issue fixed in 1.4.1 and I am using 2.2.7.

Sorry, I could not generate the exact aggregation query as the code itself is failing. 

I tried this project step after bucket on my MongoDB shell and it passed. So to me, it looks like an issue in the spring-data code I have written or its a bug.

Please suggest me how I can solve this issue.

My records looks like :

/* 1 */
{
    "_id" : ObjectId("5f92caaab3823704a346766b"),
    "accountIdentifier" : "accountIdentifier",
    "type" : "ENTITY_USAGE", //other value is `CONNECTIVITY_CHECK`
    "activityStatus" : "SUCCESS",
    "activityTime": 1603115723209
}

The monogodb query which works:

db.getCollection('entityActivity').aggregate([
     { 
         "$bucket" : 
         { 
             "boundaries" : [1603115723209, 1603202123209, 1603288523209,1603374923209,1603461323209,1603547723209], 
             "default":  0,
             "groupBy" : "$activityTime", 
             "output" : { 
                           "successfulActivitiesCount" : { "$sum" : "$isSuccessfulActivity"},
                         }
         }                  
     },
     { 
       "$project":
         {
           "startTime": "$_id",
            "successfulActivitiesCount": 1 
         }
     }   
])

---

Referenced from: pull request #890

Backported to: 3.1.1 (2020.0.1), 3.0.6 (Neumann SR6)

1 votes, 2 watchers

[spring-projects-issues]

Christoph Strobl commented

Do you use a typed or untyped aggregation? The latter does not not try to map fields against the domain type (see: reference documentation).

A minimal reproducible example would help us identify the issue?

[spring-projects-issues]

Deepak Patankar commented

Hey!!

Thanks for replying, I was using untyped aggregation, then I switched to typed one. But it is still giving the same error. 

Sorry my previous code was long, here I am attaching a minimal version.

The Aggregation Code.

public Page<OutputClass> getBucketsByDay(long start, long end) {
  BucketOperation bucketOperation = getBucketOperation(start, end);
  ProjectionOperation projectionStep = getProjectionForStart();
  TypedAggregation<Entity> aggregation =
      TypedAggregation.newAggregation(Entity.class, bucketOperation, projectionStep);
  List<OutputClass> outputs =
      mongoTemplate.aggregate(aggregation, OutputClass.class).getMappedResults();
  return new PageImpl<>(outputs);
}

private BucketOperation getBucketOperation(long startTime, long endTime) {
  List<Long> timeIntervalList = getBoundariesForAggregating(startTime, endTime);
  return bucket("time").withBoundaries(timeIntervalList.toArray());
}

private List<Long> getBoundariesForAggregating(long startTime, long endTime) {
  long timeUnit = 24 * 60 * 60 * 1000;
  List<Long> timeIntervalList = new ArrayList<>();
  for (long time = startTime; time <= endTime; time += timeUnit) {
    timeIntervalList.add(time);
  }
  return timeIntervalList;
}

private ProjectionOperation getProjectionForStart() {
  return Aggregation.project().and(UNDERSCORE_ID).as("startTime");
}

Output Class

@Data
@Builder
public class OutputClass {
  long startTime;
}

Entity Class

@Data
@SuperBuilder
@NoArgsConstructor
@Document("entity")
public class Entity implements PersistentEntity {
  @Id @org.mongodb.morphia.annotations.Id String id;
  @NotNull long time;
}

 

[spring-projects-issues]

Deepak Patankar commented

Stack Trace: 

Invalid reference '_id'!Invalid reference '_id'!java.lang.IllegalArgumentException:  at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:114) at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:77) at org.springframework.data.mongodb.core.aggregation.ProjectionOperation$ProjectionOperationBuilder$FieldProjection.renderFieldValue(ProjectionOperation.java:1445) at org.springframework.data.mongodb.core.aggregation.ProjectionOperation$ProjectionOperationBuilder$FieldProjection.toDocument(ProjectionOperation.java:1432) at org.springframework.data.mongodb.core.aggregation.ProjectionOperation.toDocument(ProjectionOperation.java:261) at org.springframework.data.mongodb.core.aggregation.AggregationOperation.toPipelineStages(AggregationOperation.java:55) at org.springframework.data.mongodb.core.aggregation.AggregationOperationRenderer.toDocument(AggregationOperationRenderer.java:56) at org.springframework.data.mongodb.core.aggregation.Aggregation.toPipeline(Aggregation.java:658) at org.springframework.data.mongodb.core.AggregationUtil.createPipeline(AggregationUtil.java:92) at org.springframework.data.mongodb.core.MongoTemplate.doAggregate(MongoTemplate.java:2166) at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:2141) at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:2022) at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:2008)

[spring-projects-issues]

Deepak Patankar commented

Hey!!. Can you please let me know whether the example I have provided is good ?

[spring-projects-issues]

Christoph Strobl commented

thanks for the sample! Please be patient, we're working on some changes (a more relaxed field name check) that might also solve this one