Need to filter subresources with original Api Platform filters

[renta]

I have a task to filter entities by their subresources. Original Api Platform filters (SearchFilter with exact strategy) work as follow: they are filter a main entity by the value of the subresource property, and then push a list of the main entity ids to the IN condition of a main doctrine query. This behaviour produce logic: you will get list of filtered main entities with all their subresources. And this is a logically right behaviour (we filter the MAIN entity by the SUBRESOURCE property).

However, seems to be logically right this filter behaviour produce a huge problem (especially, for collection of entities with several nested subresource levels): we show list of unfiltered subresources and to show them in a filtered state we need to give up an ability to retrieve subresources with one request and make additional requests for all of them. In our case it could lead to tens or even thousands requests in the case where we could be satisfied with a single one.

I've also found a class, that seems to create this behaviour and linked Github issues for it and my case.
See class

https://github.com/api-platform/core/blob/master/src/Bridge/Doctrine/Orm/Extension/FilterEagerLoadingExtension.php

and issues:

#944
api-platform/api-platform#424

I think that filtering of subresources should be an optional ability for Api Platform filters. It will be a really performance friendly feature.

[ymarillet]

Hi there,

I am the one who opened #944

I'm currently having the same use case as you (and surely many others). I'd also like to have this feature natively, however it could get very difficult to handle all use cases (=> you would need to configure filters several times depending on the behaviour you would like to apply, filtering subresources or not, maybe not all subfilters should filter subresources, etc., not very handy)

As @soyuka says (api-platform/api-platform#424 (comment)), you can actually achieve the behaviour with having a single sql statement by adding Doctrine filters https://api-platform.com/docs/core/filters#using-doctrine-filters . Up to you to determine if the filters should be enabled by default, then (dis|en)abled/configured on demand (by query string, specific custom action/event listener, etc.).

Maybe it's mostly a documentation problem, describing this common use case ?

[renta]

Thank you for the answer @ymarillet. I've tried to solve this task with a custom filter and it would work for very simple use cases. But when I wanted to achieve the same behaviour as in SearchFilter for example, I had to copy-paste all its code, altered some blocks of Doctrine query builder, but my changes went not in the needed part of the result sql-query. Also, this way is related with copy-pasting of the original APIP filters (because I want SearchFilter "on steroids")...

Now I'm trying to deal with this task with a custom extension. Despite of custom filter, where you need to reinvent original APIP filters, extensions query builder in the applyToCollection method already has all the filter stuff - I just need to "monkey patch" it (copy filter condition to the part external part of the query). But it's also seems to be an ugly quirk (because I need to hardcode a filter value processing code there).

It would be ideal to have an ability of switching "on" and "off" FilterEagerLoadingExtension. Maybe there is a way to do it with service configuration or on events level?

Maybe it's mostly a documentation problem, describing this common use case ?

I'll be immensely glad to see an example of this problem workaround.

[ymarillet]

I'll be immensely glad to see an example of this problem workaround.

I unfortunatly don't have a clean way to do that for now, I find my solution pretty ugly atm, I'd need to think more about it to expose clean code covering most common use cases. My business use cases are: filtering subresource from an active/inactive status and/or validity dates (date start, date end).

For now, I have a class implementing ApiPlatform\Core\Bridge\Doctrine\Orm\Extension\ContextAwareQueryCollectionExtensionInterface and ApiPlatform\Core\Bridge\Doctrine\Orm\Extension\QueryItemExtensionInterface, introspecting into $context['filters'] to determine which Doctrine filters should be enabled.

[marcgraub]

Hi!
I'm trying to solve the same problem here.

I'm new in api-platform and I'm using it for a GraphQL API, I think a good aproach would be to apply filter dependant of the nest level where we apply the filter.
I'm going to try my best to explain it.

This example shows a list of products that only have stores with storeId = 3:

query test {
  products(productStore_storeId: 3) {
    edges {
      node {
        customCode
        description
        
        productStore {
          edges {
            node {
              storeId
              productId
              state
            }
          }
        }
      }
    }
  }
}

This example shows a list of products and her stores, and filter the list by storeId = 1

query test {
  products {
    edges {
      node {
        customCode
        description
        
        productStore(productStore_storeId: 1) {
          edges {
            node {
              storeId
              productId
              state
            }
          }
        }
      }
    }
  }
}

I think this aproach isn't viable for a REST API, but makes sense on a GraphQL API.
To implement this aproach I don't know where to start, I have questions like, where are the filter parameters recollected.

At this moment I'm implementing ContextAwareQueryCollectionExtensionInterface

[elboletaire]

@ymarillet is there any way you could share here, not the file, but at least some tips about how to do it? I already know how to filter resources, but I've no idea about how to filter nested resources from a parent resource without creating a custom controller with custom methods. I'd like to use the api-platform REST structure and add my filters there and avoid creating methods for specific uses that should be easily done using the available framework utilities.

So.. If anyone can tell me some clues about how to filter subresources... I'll owe him/her a beer (or a coffee, I don't drink btw, but most developers love beer x_D)

[marcgraub]

Hi! I have a quick and dirty fix for this, you need to implement a custom search filter and overwrite the "filterProperty" function, here's my code, I think it's all to change:

protected function filterProperty(string $property, $value, QueryBuilder $queryBuilder, QueryNameGeneratorInterface $queryNameGenerator, string $resourceClass, string $operationName = null, $context = [])
    {
        if (
            null === $value ||
            !$this->isPropertyEnabled($property, $resourceClass) ||
            !$this->isPropertyMapped($property, $resourceClass, true)
        ) {
            // If the property exist in the current entity, add it to the where, with this, we can filter lists on nest level
            if(!$this->existPropertyInProperties($property, $resourceClass)) {
                return;
            }
        }

        $alias = $queryBuilder->getRootAliases()[0];
        $field = $property;

        if ($this->isPropertyNested($property, $resourceClass)) {
            list($alias, $field, $associations) = $this->addJoinsForNestedProperty($property, $alias, $queryBuilder, $queryNameGenerator, $resourceClass);
            $metadata = $this->getNestedMetadata($resourceClass, $associations);
        } else {
            $metadata = $this->getClassMetadata($resourceClass);
        }

        $values = $this->normalizeValues((array) $value);

        if (empty($values)) {
            $this->logger->notice('Invalid filter ignored', [
                'exception' => new InvalidArgumentException(sprintf('At least one value is required, multiple values should be in "%1$s[]=firstvalue&%1$s[]=secondvalue" format', $property)),
            ]);

            return;
        }

        $caseSensitive = true;

        // If the property exist in the current entity, add it to the where, with this, we can filter lists on nest level
        if (($pos = strpos($property, ".")) !== FALSE) {
            $extractedProperty = substr($property, strpos($property, ".") + 1);

            if(array_key_exists($extractedProperty, $this->properties)) {
                $property = $extractedProperty;
                $field = $extractedProperty;
            }
        }

        if ($metadata->hasField($field)) {
            if ('id' === $field) {
                $values = array_map([$this, 'getIdFromValue'], $values);
            }

            if (!$this->hasValidValues($values, $this->getDoctrineFieldType($property, $resourceClass))) {
                $this->logger->notice('Invalid filter ignored', [
                    'exception' => new InvalidArgumentException(sprintf('Values for field "%s" are not valid according to the doctrine type.', $field)),
                ]);

                return;
            }

            $strategy = $this->properties[$property] ?? self::STRATEGY_EXACT;

            // prefixing the strategy with i makes it case insensitive
            if (0 === strpos($strategy, 'i')) {
                $strategy = substr($strategy, 1);
                $caseSensitive = false;
            }

            if (1 === \count($values)) {
                $this->addWhereByStrategy($strategy, $queryBuilder, $queryNameGenerator, $alias, $field, $values[0], $caseSensitive);
                return;
            }

            if (self::STRATEGY_EXACT !== $strategy) {
                $this->logger->notice('Invalid filter ignored', [
                    'exception' => new InvalidArgumentException(sprintf('"%s" strategy selected for "%s" property, but only "%s" strategy supports multiple values', $strategy, $property, self::STRATEGY_EXACT)),
                ]);

                return;
            }

            $wrapCase = $this->createWrapCase($caseSensitive);
            $valueParameter = $queryNameGenerator->generateParameterName($field);

            $queryBuilder
                ->andWhere(sprintf($wrapCase('%s.%s').' IN (:%s)', $alias, $field, $valueParameter))
                ->setParameter($valueParameter, $caseSensitive ? $values : array_map('strtolower', $values));
        }

I check if the filter property exist in the related entity and add a where to the query

[renta]

I could provide my solution for this problem, if anyone interested. It helps to filter subresources with GET parameter, but it require patch of the Api Platform core file (I did it with post-install script in composer.json - so it could be broken one day). I'm not really proud of my solution but it works as expected with minimum code changes (that's why I'm for the same-way functionality in the core framework).

[ymarillet]

@elboletaire : my solution involves doctrine filters. Make sure you understand what this means if you want to use mine.

The example down there is very specific and pretty dirty (up to you to modify depending on the complexity/genericity needed on your project).

Say you have two entities (configure Apiplatform and ORM annotations where needed...):

class Foo
{
    private $id;

    /** @var Bar[] **/
    private $bars;
}

class Bar
{
    private $id;
    
    /** @var string */
    private $status;
}

You will be able to filter Bar entities embedded in Foo entities through the query string.
i.e. :
/api/foos?bars.status=active : means you want all Foo having at least one active Bar (which is the default meaning of such an ApiPlatform query string)
/api/foos?bars.status=active&filter_associations=true : same as above but you only want active Bars in the result (ApiPlatform would return all Bars, active or not, without the doctrine filter)

Declare the doctrine filter config/packages/doctrine.yaml :

doctrine:
  orm:
    entity_managers:
      default:
        filters:
          exposable:
            class: 'App\Bridge\Doctrine\Filter\ExposableFilter'
            enabled: false # will be enabled per use case

Code it:

<?php

namespace App\Bridge\Doctrine\Filter;

use Doctrine\ORM\EntityManagerInterface;
use Doctrine\ORM\Mapping\ClassMetadata;
use Doctrine\ORM\Query\Filter\SQLFilter;

final class ExposableFilter extends SQLFilter
{
    /**
     * @var EntityManagerInterface
     */
    private $entityManager;

    /**
     * @param ClassMetadata $targetEntity
     * @param string        $targetTableAlias
     *
     * @return string
     *
     * @throws \Exception
     */
    public function addFilterConstraint(ClassMetadata $targetEntity, $targetTableAlias): string
    {
        $em = $this->getEntityManager();
        if (!$em->getFilters()->isEnabled('exposable')) {
            return '';
        }

        // this is a example of how you can select the entities affected by this filter
        if (Bar::class !== $targetEntity->reflClass->getName()) {
            return '';
        }

        if (!($value = $this->getParameter('status'))) {
            return '';
        }

        return sprintf('%s.status = %s', $targetTableAlias, $this->getParameter('status'));
    }

    /**
     * @return EntityManagerInterface
     *
     * @throws \ReflectionException
     */
    private function getEntityManager(): EntityManagerInterface
    {
        if (null === $this->entityManager) {
            $refl = new \ReflectionProperty(SQLFilter::class, 'em');
            $refl->setAccessible(true);
            $this->entityManager = $refl->getValue($this);
        }

        return $this->entityManager;
    }
}

Code ApiPlatform extension:

<?php

namespace App\Bridge\Doctrine\ORM\Extension;

use ApiPlatform\Core\Bridge\Doctrine\Orm\Extension\ContextAwareQueryCollectionExtensionInterface;
use ApiPlatform\Core\Bridge\Doctrine\Orm\Extension\QueryItemExtensionInterface;
use ApiPlatform\Core\Bridge\Doctrine\Orm\Util\QueryNameGeneratorInterface;
use Doctrine\Bundle\DoctrineBundle\Registry;
use Doctrine\ORM\EntityManagerInterface;
use Doctrine\ORM\QueryBuilder;

class ExposableFilterExtension implements ContextAwareQueryCollectionExtensionInterface, QueryItemExtensionInterface
{
    /**
     * @var Registry
     */
    private $doctrine;

    /**
     * @param Registry $doctrine
     */
    public function __construct(Registry $doctrine)
    {
        $this->doctrine = $doctrine;
    }

    /**
     * @param QueryBuilder                $queryBuilder
     * @param QueryNameGeneratorInterface $queryNameGenerator
     * @param string                      $resourceClass
     * @param string|null                 $operationName
     * @param array                       $context
     */
    public function applyToCollection(
        QueryBuilder $queryBuilder,
        QueryNameGeneratorInterface $queryNameGenerator,
        string $resourceClass,
        string $operationName = null,
        array $context = []
    ) {
        $this->enableFilters($context);
    }

    /**
     * @param QueryBuilder                $queryBuilder
     * @param QueryNameGeneratorInterface $queryNameGenerator
     * @param string                      $resourceClass
     * @param array                       $identifiers
     * @param string|null                 $operationName
     * @param array                       $context
     */
    public function applyToItem(
        QueryBuilder $queryBuilder,
        QueryNameGeneratorInterface $queryNameGenerator,
        string $resourceClass,
        array $identifiers,
        string $operationName = null,
        array $context = []
    ) {
        $this->enableFilters($context);
    }

    /**
     * @param array       $context
     */
    private function enableFilters(array $context): void
    {
        $filters = $context['filters'] ?? [];

        if (empty($filters['filter_associations'])) {
            return;
        }

        /** @var EntityManagerInterface $em */
        $em = $this->doctrine->getManager();

        // hardcoded !! change to your needs.
        $status = $filters['bars.status'] ?? null;

        if (empty($status)) {
            return;
        }

        if ($em->getFilters()->isEnabled('exposable')) {
            return;
        }

        $em->getFilters()->enable('exposable');
        $em->getFilters()->getFilter('exposable')->setParameter('status', $status);
    }
}

Configure the service:

services:
    App\Bridge\Doctrine\ORM\Extension\ExposableFilterExtension:
        tags:
        - { name: 'api_platform.doctrine.orm.query_extension.collection', priority: 128 }
        - { name: 'api_platform.doctrine.orm.query_extension.item', priority: 128 }

[elboletaire]

Ok, now that I'm filtering nested results... doctrine throws an EntityNotFoundException because it can't find the just filtered entity... sorry, what? 😂

[ymarillet]

That's probably because of doctrine/orm#4543

I workarounded that with a normalizer

<?php
namespace App\Serializer\Normalizer;

use Doctrine\Common\Persistence\Proxy;
use Doctrine\ORM\EntityNotFoundException;
use Symfony\Component\Serializer\Normalizer\NormalizerAwareTrait;
use Symfony\Component\Serializer\Normalizer\NormalizerInterface;

class ExposableNormalizer implements NormalizerInterface
{
    use NormalizerAwareTrait;

    /**
     * @param Proxy $object
     * @param string|null $format
     * @param array $context
     *
     * @return mixed
     */
    public function normalize($object, $format = null, array $context = [])
    {
        try {
            return $this->normalizer->normalize($object, $format, $context);
        } catch (EntityNotFoundException $e) {
            return null;
        }
    }

    /**
     * @param mixed $data
     * @param string|null $format
     * @param array $context
     *
     * @return bool
     */
    public function supportsNormalization($data, $format = null, array $context = [])
    {
        return $data instanceof Bar && $data instanceof Proxy;
    }
}