pagination example in hyper-schema

[dlax]

In #448, a paginated collection example got added, with pagination handled using hyperschema links. This makes example fairly complicated, especially (I think) because the "thing-collection" schema is an object with metadata to hold pagination information instead of a plain array.

For some context on the initial discussion with @handrews and @philsturgeon, see #448 (comment).

Quoting @handrews #448 (comment)

Collections that are filterable or paginated/scrolling are unmanageable under hyper-schema without meta-data, and showing a "collection" that can't be filtered or paginated is not realistic.

I would don't see a compelling reason to use hyper-schema to handle pagination. In the "thing-collection" is a plain array, prev/next links could be HTTP Link headers I think. Maybe this design is less sophisticated, but I think it does the job.

[handrews]

@dlax I really don't understand why you want this to be a plain array. I just haven't seen that as best practices for collections for a long time. I would consider it unrealistic.

Yes, you can use HTTP Link headers for this, but this is the Hyper-Schema spec so the point is to show how to do things with Hyper-Schema. There's no reason to not use Hyper-Schema here. Having your links split up among Hyper-Schema and Link headers would be excessively confusing.

Duplicating certain links, particularly easily recognizable ones like pagination, in both places would make sense. And the Overview notes that you can use Hyper-Schema on the server side to generate Link headers. But I don't see one person's inclination to not use Hyper-Schema for the most common pattern in REST to be a reason to keep it out of the spec.

For me, the entire point of a hypermedia format is to do your hypermedia in the hypermedia format. So I'm really not finding any aspect of this argument- the array part or the Link header part- to be convincing. I strongly dispute that there needs to be any sort of reason, compelling or otherwise, on a link-by-link basis to use Hyper-Schema rather than Link headers. You choose your hypermedia approach and apply it consistently.

I'm also literally on my way out the door for vacation, so I'll have to get back to this next week when I return.

[philsturgeon]

Hey! I think we are having two conversations in one, and one of them is shrouding the other.

1. Should APIs return a collection as an array or an object
2. What should the JSON HyperSchema example show

API's returning a bare array for collections is something I've been wondering about a lot myself recently. With The security vuln I posted about in #448 being kinda irrelevant these days it certainly seems like it could, but as @handrews points out that's definitely going to cause confusion when your links are split between hyperschema and Link headers.

@dlax Regardless of how you or I might feel about collections being an object or an array, my experience (anecdotal as it is), is for objects to be used.

If arrays to you are the common practice, that would explain why you're pushing for arrays.

The goal here is to provide examples based off of common practice, and not a place to try and push for change in those common practices. In the future we might see bare arrays being used more, but at the time of writing, it really does feel like objects are in 99% of arrays I work with (rightly or wrongly).

[handrews]

@dlax @philsturgeon one thing that I believe strongly about hyper-schema systems is that you must always be able to fulfill the "self" link. So if your representation is paginated through the URI (as it should be if it's paginated at all), then your "self" link needs to be of the current paginated (and/or filtered) page.

That typically means that you must have a meta section of some sort, or a pagination section and a filters or query section. There are so many reasons to carry additional fields in a collection representation- aggregate values across the set being another.

Aside from the fulfill-the-self-link rule, which I do feel pretty strongly about, I don't even think the array vs object thing is about which practice is more common. It's about which better illustrates Hyper-Schema. An array collection that relies exclusively on HTTP Link headers isn't wrong, but it also doesn't illustrate Hyper-Schema well.

[handrews]

Personally, I would recommend that next/prev/first/last get serialized as HTTP headers as well so that this particularly common pattern can be easily recognized and used by non-hyper-schema-aware implementations. But that probably goes somewhere else, either the appendix on APIs or on the web site.

[dlax]

My primary concern is making the collection+item understandable and following the comments I made in #448 and #464, my suggestion was to leave pagination aside until this happen and then go with plain arrays to simplify all the things.

So that's about:

2. What should the JSON HyperSchema example show

This is absolutely not motivated by some kind of "best practice". So while I don't quite buy your arguments on this to reject this idea, I'm very open to any suggestion to make the "collection+item" example understandable.

[handrews]

I guess to me, since there are many use cases that require an object-based collection, and nearly all REST APIs use collections, if I, as a reader, can't understand how to do an object-based collection in hyper-schema then I am not going to use the format.

This was a huge debate point when I tried to sell co-workers on draft-04. How do collections work? We all agreed that we needed collection meta-data next to the collection contents. And then what? A lot of the confusion then had to do with the way method was defined in draft-04, so that's not a concern, but recognizing how to use all of the link types correctly within hyper-schema, not HTTP, is still a concern.

[handrews]

@dlax @philsturgeon any further thoughts here? To summarize my position:

• The example should show using link types within hyper-schema. We can add a note about also using HTTP Link headers (if it's not already there), but this is not a spec about implementing collections via HTTP headers.
• I consider pagination (or filtering, but pagination is more well-defined) to be an essential part of illustrating how to build collections in hyper-schema. In my experience, failing to explain a real-world collection in the spec makes it very likely for the spec to be rejected by potential users. Simple examples are great until you try to do something complex. One reason for such a long and extensive example section is specifically to build up something at a real-world complexity level.
• We should stay with an object because it is increasingly common to see objects with meta-data for collections, and because there's no good way to implement the links with an API representation.

So what needs to be done at this point? Are any of the reasons I list above things to which you object?

[dlax]

* The example should show using link types within hyper-schema. We can add a note about also using HTTP |Link| headers (if it's not already there), but this is not a spec about implementing collections via HTTP headers.

I don't think it's need to add such a note. I suggested using HTTP Link header in reply to a question about how I would implement the pagination pattern, not as a proposal to write this in the specification.

* I consider pagination (or filtering, but pagination is more well-defined) to be an essential part of illustrating how to build collections in hyper-schema. In my experience, failing to explain a real-world collection in the spec makes it very likely for the spec to be rejected by potential users. Simple examples are great until you try to do something complex. One reason for such a long and extensive example section is specifically to build up something at a real-world complexity level.

I agree with that but on the other hand if we fail to provide a comprehensive and real-world example (whatever that means), we are not doing a good job either.

* We should stay with an object because it is increasingly common to see objects with meta-data for collections, and because there's no good way to implement the links with an API representation.

I don't buy this argument, but don't object to it either :) Anyways, I reread the "Collections" and "Pagination" example sections and I still find them hard to follow. There's the last paragraph in "Collections" ("But what if we do not have any "thing"s yet?...") which makes the reader feel that there's something "blurry", not to mention the CREF. The "Pagination" section also has a bunch of CREF that gives the same impression. But that's only my feeling, so if anybody else find these sections understandable enough to get the draft released, that's fine by me.

[philsturgeon]

I dont have anything much new to add. :)

[handrews]

I agree with that but on the other hand if we fail to provide a
comprehensive and real-world example (whatever that means), we are not
doing a good job either.

I'm not sure to interpret this. My point is that I am trying to provide that example in the most convincing way possible. This issue is about whether and how the example series should include pagination. I'm saying yes it should, and I have not heard a viable alternative (since I don't consider an array approach to be viable, at least not in terms of using JSON Hyper-Schema to construct the links).

which makes the reader feel that there's something "blurry"

Concerns over the proper scope of the examples (whether to repeat schemas previously seen, or include more technically not necessary but possibly helpful fields is being tracked in #478, which I assume applies to both parts of the now-split collection example).

not to mention the CREF

Is there something specific you want with the CREF? This is a challenging example to implement despite being the single most common usage pattern in REST (collection + item in general, not necessarily this exact approach). The point of the CREFs is to acknowledge that and hopefully get some discussion going, since we haven't made any progress on issue #421 which is where we should be discussing how the spec seems to fall short here.

[dlax]

I'm not sure to interpret this.

This indeed needs some details. I provided some in my previous comment but here are more: the example is hard to follow, especially at the end of the section because of the discussion about how to create and add an item to the collection that gives the impression that there's no satisfactory solution. Which leads to the CREF and the unresolved issue #421; perhaps this is fine to keep things in this state. Admittedly I forgot about the issue while reading, and the reader will not even be aware of it. I don't have any suggestion to improve the situation.

This issue is about whether and how the example series should include pagination.

Yes, we have somehow diverged, part of this is due to the discussion being fragmented between PR reviews and a bunch of issues. It's really not comfortable for me either but I'm making reasonable efforts to follow the process. Acknowledging that the discussion is not making progress, I'm going to close this issue.

I'm saying yes it should

So do I. Let's keep the example as is and gather more feedback after publication.

[handrews]

because of the discussion about how to create and add an item to the collection that gives the impression that there's no satisfactory solution

That's exactly what it's supposed to do. AFAIK there is no satisfactory solution. If you are aware of one, please comment on #421, but it sounds like you don't have an answer either :-)

Part of the goal of drafts is to highlight areas that need further work. This should be a big glaring problem that is apparent to anyone reading the spec. If there's a better way to show that that is the intent, I would be more than happy to improve the language. Maybe we can put a link in the cref to the issue? I'm not sure whether that's problematic in some way or not.

And yeah, I agree the discussion got fragmented and me going on vacation in the middle didn't help as I lost track of stuff. Hopefully we can avoid similar problems in the future.