Type of `request.data` inconsistent (`dict` / `QueryDict`) depending on request content-type

[ge0rg]

Checklist

• I have verified that that issue exists against the master branch of Django REST framework.
• I have searched for similar issues in both open and closed tickets and cannot find a duplicate. --> Aspects of this issue are documented in app/json VS querystring request leads to different data representations #1593 and Unit tests using APIClient request.DATA do not use a QueryDict. #3347 but I wanted to provide a proper root cause analysis
• This is not a usage question. (Those should be directed to the discussion group instead.)
• This cannot be dealt with as a third party library. (We prefer new functionality to be in the form of third party libraries where possible.)
• I have reduced the issue to the simplest possible case.
• I have included a failing test as a pull request. (If you are unable to do so we can still accept the issue.)

Steps to reproduce

• Have an API view that you can POST a list of elements to, e.g. with a list of file attachments or maybe a multi-selection box in a ModelViewSet.perform_create
• access the items via self.request.data.getlist('items', [])

Expected behavior

A list of items or an empty list should be returned, regardless of the Content-type used to call the API.

Actual behavior

The view only works on form-data (x-www-form-urlencoded) uploads, JSON uploads lead to an exception:

Traceback (most recent call last):
  File "/srv/myapp-dev/lib/python3.7/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/myapp-dev/lib/python3.7/site-packages/django/core/handlers/base.py", line 179, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/srv/myapp-dev/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/srv/myapp-dev/lib/python3.7/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/srv/myapp-dev/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/srv/myapp-dev/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/srv/myapp-dev/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/srv/myapp-dev/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/srv/myapp-dev/lib/python3.7/site-packages/rest_framework/mixins.py", line 19, in create
    self.perform_create(serializer)
  File "/srv/myapp-dev/app/app/apis.py", line 597, in perform_create
    items = self.request.data.getlist('items', [])
AttributeError: 'dict' object has no attribute 'getlist'

As unit tests use form-data by default, and most modern apps use JSON, this leads to a bug that won't be detected until rolled out to a live server.

Parser Inconsistency

The reason is that FormParser returns an (immutable) QueryDict whereas JSONParser returns the dict response obtained from json.loads().

The most reasonable way to achieve consistency would be to make JSONParser also return a QueryDict, e.g. using this conversion:

parsed_dict = json.load(decoded_stream, parse_constant=parse_constant)
query_dict = QueryDict('', mutable=True)
query_dict.update(parsed_dict)
query_dict._mutable = False # set to immutable to match FormParser
return query_dict

[xordoquy]

This inconsistence is due to the totally different nature of HTML forms vs JSON.
It is expected in parts of the code to handle some weird tricks from HTML forms specifications (booleans fields, impossibility to know whether it's a list or a single item).
Beside, the documentation make it clear that request.data relates to parsers and that only FormParser and MultipartParser returns a QueryDict.

[ge0rg]

I can understand that the difference and its implications are well-understood by DRF developers, much more so than by DRF users (as documented here and in the linked issues).

The documentation paragraph that you linked to contains the following statement, which contradicts the findings outlined above:

For example you can handle incoming JSON data in the same way that you handle incoming form data.

The linked parsers page is a long-ish page about how DRF determines the right parser and how to configure them, after which comes a list of the specific parser types. The JSONParser description doesn't mention the return type at all.

Making the parsed data type consistent is surely a significant API change that would have to go into the next major version of DRF, and I can understand that you are not very keen to do it. Still, it would improve usability of DRF for developers.

I'd be willing to spend the time needed to create a PR for the API change if you are not categorically opposed to integrating it.

Otherwise, it would be great to improve the documentation in a fashion that makes the differences explicit, e.g. by documenting the return type of JSONParser and mentioning the difference in the request.data section you linked. I can create a PR for that as well.

[xordoquy]

Good point, the quote is - in my opinion - not correct although I'm sure that was the initial intent.

That would be a good thing to fix that sentence and possibly state explicit that JSONParser will return a dict.

As for unifying request.data, I don't think that's technically possible without the serialization layer - and note that once the serialization is applied, you'd want to go with validated_data. With HTML forms themselves, you have no way to know whether an item is just a single item or a list with only one item - not even speaking about that weird boolean field.

[ge0rg]

Well, maybe I only ran into the issue because I was using the serialization layer incorrectly. My object model has a many-to-many relationship to file attachments, and my serializer uses a SerializerMethodField to return a list of absolute URLs of all attachments for an object, so I resorted to manually extracting the files on upload. The docs for FileField are rather short, and it's hard to find good examples for multiple files. But this is probably something that I need to take to the forum.

That said, I'll prepare a PR to the documentation. Thanks for your feedback :)