Skip to content

Conversation

@mbhaskar
Copy link
Member

@mbhaskar mbhaskar commented Oct 8, 2021

This PR adds pagination support to the queries.

Schema design

We use connection objects to achieve pagination along with first and after keywords
For example, A list query today looks like below

type Query{
  planets : [Planet]
}

To paginate instead of fetching all the results like the above query, we use a page size and a connection object which would return the nodes as result list and a continuation which would help us obtain next page.
We use first as a keyword for page size and after as a keyword for specifying continuation. Here is how the schema should look like

type Query{
 planets(first: Int, after: String): PlanetConnection
}

type PlanetConnection{
 nodes: [Planet]
 endCursor: String
 hasNextPage: Boolean
}

A null continuation in the result would mean there are no more results.
A base64 encoded continuation token looks like this: W3sidG9rZW4iOiItUklEOn5saHQxQVBKZFpPMERBQUFBQUFBQUFBPT0jUlQ6MiNUUkM6MyNJU1Y6MiNJRU86NjU1NjcjUUNGOjQiLCJyYW5nZSI6eyJtaW4iOiIiLCJtYXgiOiJGRiJ9fV0=

Resolver design:
Resolver should now have a "isPaginated" property set to true to identify a query as a paginated query
ex:
{
"id": "planets",
"isPaginated": true,
"databaseName": "graphqldb",
"containerName": "planet",
"parametrizedQuery": "select * FROM c"
}

This PR

** This PR contains some double deserialization serialization which happens because of the conversions between Netwonsoft and System.Text.Json. Since System.Text.Json seems to have limited capabilities in json writing/creating, I will be removing this and move the code to use only Newtonsoft which will also avoid the problem of mutliple serializations. Created an issue for the same #60

corresponding RFC: https://github.com/Azure/project-hawaii/blob/main/rfcs/cursor-connection-pagination.md

References:
Pagination in cosmosdb: https://docs.microsoft.com/en-us/azure/cosmos-db/sql/sql-query-pagination

@jarupatj
Copy link
Contributor

Can we please add tests.

Copy link
Contributor

@jarupatj jarupatj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🕐

@jarupatj
Copy link
Contributor

We need a spec on the other repository.

@jarupatj
Copy link
Contributor

what is the plan for adding support for SQL and PG?

@mbhaskar
Copy link
Member Author

mbhaskar commented Oct 13, 2021

what is the plan for adding support for SQL and PG?

In this PR, we are modifying the query interface to indicate if this is a continuation query, and also pass the first and after values. Adding detailed implementation is something we have to work with respective service implementation @Aniruddh25 lets syncup?

@JelteF
Copy link
Contributor

JelteF commented Nov 5, 2021

@mbhaskar Can you update this according to the modified RFC.

Merged RFC PR: https://github.com/Azure/project-hawaii/pull/20
RFC: https://github.com/Azure/project-hawaii/blob/main/rfcs/cursor-connection-pagination.md

@JelteF
Copy link
Contributor

JelteF commented Nov 5, 2021

Also, I think it's good to have this PR only implement the Cosmos side and do PG and SQL afterwards. That way there's a base for PG and SQL to build on.

@mbhaskar mbhaskar force-pushed the mbhaskar/pagination-support branch from 8e8ffc3 to 4de5f8d Compare November 15, 2021 17:47
@mbhaskar mbhaskar requested a review from Mathos1432 as a code owner November 15, 2021 17:47
Copy link
Contributor

@JelteF JelteF left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apart from the comments I left, there's some merge conflicts again now that my SQL query generation PR got merged.

Part of my comments is advice on how to make the tests more readable and maintainable. Feel free to ignore this for now, it can be addressed in a future PR dedicated to it as well if you prefer.

@seantleonard
Copy link
Contributor

Thanks for queueing up these changes! Would like to see additional elaboration in some sections RE: comments I've left. Also, in your PR description could you include a sample query or two that can be copy pasted into a request body to quickly test your functionality?

Copy link
Contributor

@JelteF JelteF left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for addressing the comments :shipit:

@mbhaskar mbhaskar enabled auto-merge (squash) December 10, 2021 17:18
Copy link
Contributor

@jarupatj jarupatj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@mbhaskar mbhaskar disabled auto-merge December 10, 2021 18:45
@mbhaskar mbhaskar merged commit 40196e9 into main Dec 10, 2021
@mbhaskar mbhaskar deleted the mbhaskar/pagination-support branch December 10, 2021 20:34
abhishekkumams added a commit that referenced this pull request Jul 25, 2022
updating commands from --permission to --permissions
abhishekkumams added a commit that referenced this pull request Aug 3, 2022
* Initial commit

* added first design proposal

* CODE_OF_CONDUCT.md committed

* LICENSE committed

* README.md updated to template

* SECURITY.md committed

* SUPPORT.md committed

* new console app

* initial config generator

* updated implementation proposal

* Update README.md

* adding .gitignore

* updating gitignore

* removing compiled files

* added support for command-add

* updating gitignore

* cleaning files and adding todos

* Update README.md

* updated file path for config generation

* using classes of hawaii-gql

* adding design_doc and added update command

* removing temp file

* updating readme

* updating readme

* removing temp files

* removing temp files

* updating README

* removing temp files

* fixing the update command

* removing temp files

* Added relationship compatibility

* updated ReadMe

* removing temp files

* updating gitignore

* update README.md (#5)

* Update README.md (#8)

* Update README.md

* Update README.md

* Fix for connection-string issue (#9)

* fix connection-string issue

* removing temp files

* removing resolver config as it is specific to cosmosDB

* updating .gitignore

* updating .gitignore

* Dev/abhishekkuma/code refactoring (#16)

* fix connection-string issue

* removing temp files

* Use Action class from Runtime Config

* removing temp files

* minor fix

* code refactoring

* added summary for all the methods

* making mappings.field optional (#13)

* Update default host global settings (#19)

* updated default host global settings

* updating default value of Cors.Origin

* fixing spelling errors

* Add support for linking object in relationship (#22)

* adding support for linking object in relationship

* updating validation checks

* fixing grammatical errors

* Add CODEOWNERS file (#24)

* added CODEOWNERS file

* updating names

* adding Davide as code owner

* Update version to 0.0.2 (#25)

* updating version to 0.0.2

* fix indentation

* Add commands in help window (#27)

* added commands in help window

* updated help screen

* updated help screen

* fixing typo

* Update README.md (#26)

* Update README.md

Updating Install Information in Readme.

* Update README.md

Co-authored-by: Aniruddh Munde <[email protected]>

Co-authored-by: Aniruddh Munde <[email protected]>

* Updated Typos (#30)

* Quick fix for typo (#31)

* Add editorconfig to auto format code (#35)

* Add editorconfig

* Fix formatting warning

* Fix more warning

* use is null

* Update CommandLineOptions property to start with capital letter.

* Add skeleton tests project (#36)

* Add tests project

* Update gitignore

* fixing update command for permissions (#34)

* Refactoring commandline handling options. (#37)

**What are the changes**
This change refactors the code to use Verb attribute https://github.com/commandlineparser/commandline/wiki/Verbs
to link command line options to command line operation.

The main change are in CommandLineOptions.cs. With this change, we can use the default help print from Commandline library. We do not need to write our own help text.
Remove Operations.cs and Commandlinehelp.cs. I fold the functionality to program.cs and configgenerator.cs.

Make options class immutable. https://github.com/commandlineparser/commandline/wiki/Immutable-Options-Type

Make database type and host option explicit type from Enum instead of string.

**Test**
Manual testing. I don't change how the config is generated. Same variables are passed to config generator.

* Update cosmosDB options (#29)

* updated cosmosDB options

* fix formatting

* making code more modular

* Update src/Models/CommandLineOptions.cs

Co-authored-by: Aniruddh Munde <[email protected]>

* updating test name

* refactoring code

* refactoring code

Co-authored-by: Aniruddh Munde <[email protected]>

* Move relationship related options to update. (#46)

* Update README.md (#51)

* Refactor config generator for Init and Add command (#41)

* Add tests for initConfig and addEntity. Refactor code a bit to make it more readable.

* Add comments.

* More verbose with arg passing.

* Format code.

* Add more tests for AddEntity.

* resolving comments

* removing temp changes

* small fix

Co-authored-by: Abhishek Kumar <[email protected]>

* fixed tests (#55)

* Update README.md (#59)

updating commands from --permission to --permissions

* Simplify Update Entity logic (#52)

* replace name to config

* symplifying update logic

* adding validations

* added more tests

* resolved comments

* adding test description

* updating file names for Add and Update EntityTests

* adding comments and tests

* adding class comments

* adding class comments

* adding more tests and comments

* updated accessebiity of some methods

* updating tests

* adding TODO for policy support

* updating few comments

* updated function summary

* minor change

* fixed comments

* refactoring changes

* using csproj instead of dll files (#62)

* Create CI pipeline (#56)

* Create main.yml

* Update main.yml

* adding Datagateway.Config.csproj

* fixing path

* fixing tests

* add tests validation in pipeline

* adding formatting check

* updating pipeline

* removing temp changes

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* updating pipeline

* updating pipeline files

* updating pipeline files

* updating pipeline

* updating pipeline

* updating pipeline

* updating pipeline

* fix pipeline

* fix pipeline

* fix pipeline

* fix pipeline

* fix pipeline

* updating runner

* updating runner

* adding pipeline for ubuntu

* fixing build job name

* using matrix for running multiple os

* updating .editorconfig

* publishing code coverage

* updating coverage directory

* getting dir info

* fixing pipeline

* updating test script

* fixing code coverage path

* fixing code coverage path

* fixing code coverage issue

* fixing code coverage issue

* fixing code coverage issue

* adding deploy steps for the nuget package

* adding deploy steps for the nuget package

* add source path for nuget package

* add source path for nuget package

* fix source path for nuget package

* fix test

* updating format check

* updating format check

* updating format check

* updating format check

* updating format check

* updating format check

* updating format check

* updating format check

* testing submodules

* fixing yaml

* fixing yaml

* fixing yaml

* fixing yaml

* updating .gitmodules

* updating .gitmodules

* updating .gitmodules

* checking submodules

* checking submodules

* checking submodules

* checking submodules

* checking submodules

* adding submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* testing submodules

* removing temp files

* updating yaml

* updating yaml

* updating tests

* updating tests

* testing workflow_dispatch

* testing workflow_dispatch

* refactoring changes

* testing coverage report to pr

* testing coverage report to pr

* testing coverage report to pr

* testing coverage report to pr

* testing coverage report to pr

* testing coverage report to pr

* testing coverage report to pr

* fix coverage report to pr

* fix coverage report to pr

* fix coverage report to pr

* testing different test reported

* testing different test reported

* testing coverage summary

* fix pipeline

* updating comments

* fixing coverage report

* removing temp files

* removing submodules coverage

* removing unused import

* fixing pipeline

* cleaning comments

* Add support for custom graphql singular plural types (#65)

* adding support for custom graphql singular plural types

* fixed formatting

* fixing pipeline

* updating comments

* removing redundant steps in pipeline

* Adding test for Case Sensitive Entity Name (#64)

* Adds test for entity creation

* Adds change to create the options differently

* Adding change to summary and comments

* Removing jwt from the config json strings

* Changing the default rest api endpoint

* Builds config differently to avoid duplication

* Fixing formatting issues

* Preventing users from creation Permissions with Invalid Actions (#70)

* adding tests

* simplyfying code

* added a new test

* simplifying code

* Changing true/false to be treated as a Boolean when used with --rest and --graphql options  (#74)

* Making true/false bool in --rest/graphql options

* addressing review comments

* checking in latest hawaii-engine commits

* addressing review comments

* adding a check for serialization of entity.graphql value

* Add support for Policy in Permission Actions (#71)

* added support for policy

* fix formatting

* added tests

* fix build failure

* fix formatting

* adding comments

* removed redundant code

* updating tests

* updating authentication provider

* updating tests

* updating editorconfig

* updating submodules

* updating submodules

* fixing some dependencies

* Add support for mappings (#76)

* adding test

* fix formatting

* fix error

* fix formatting

* updating option description

* updating comment

* fixing build

* Add support for cors origin (#77)

* added support for cors-origin

* fix formatting

* fix test

* fix formatting

* Prevent user from adding relationship in cosmos db (#79)

* preventing user from creating relationship in cosmosdb

* updating test

* Using IEnumerables for Options (#80)

* adding changes for using ienumerables for options

* removes redundant conversion

* Adding checks for empty items

* fixing console output messages

* adding empty checks for ienumerable options in addition to null

* checking in hawaii-engine sub-module

* Checking in changes from engine

* fixes formatting

* adding test for update command with multiple mapping fields

* fix formatting

* removing extra spaces

* incorporating PR review suggestions

* checnking-in latest changes from hawaii engine

* adding changes to remove empty mappings property in config json

* minor correction in the test command

* adding a test to validate the config json generated

* minor correction in the test

* New Release 0.0.4 (#84)

* Fix serialization issue with special character (#87)

* fixed serialization issue with special charachter

* updating test

* fix formatting

* removing temp changes

* fix test

* updating test

* updating test

* updating test summary

* fix pipeline

* Renaming the option from --mapping.fields to --relationship.fields (#86)

* renaming command option from mapping.fields to relationship.fields

* changing sample commadns in README fiel

* adding latest changes from hawaii engine

* adding latest changes from engine

* updating tests to use relationship.fields

* merge cli into engine

* updating csproj file

* removing cli sln file

* removing duplicate files

* fix formatting

* fix formatting

* adding cli tests into build pipeline

* fix format

* updating folder name for cli-tests

* updating csproj filename for hawaii-cli

* updating csproj filename for hawaii-cli

* fixing build

* fix format

* updating pipeline to create nuget package

* publish nuget packe to artifact

* fix build pipeline

* testing

* testing

* testing

* fixing path for exe

* fixing path for exe

* updating feed for nuget package

* commenting nuget push

* using dependency version from Directory.Build.Props

* testing nuget push

* fixing publish of nuget package

* Update CLI to use Operation Enum.

* Fix build break in tests.

* Verify that operation is valid for action.

* Remove unnessary using.

* updating pipeline

* updating nuget command in pipeline

* updating nuget command in pipeline

* updating nuget command in pipeline

* updating nuget command in pipeline

* testing nupkg creation

* testing nupkg creation

* testing nupkg creation

* testing nupkg creation

* testing nupkg creation

* testing nupkg creation

* testing nupkg creation

* testing nupkg creation

* testing nupkg creation

* testing nupkg publishing

* updating pipeline for nuget push

* fixing pipeline

* fixing pipeline

* fixing pipeline

* fixing pipeline

* fixing pipeline

* fixing pipeline

* fixing pipeline

* fixing pipeline

* fix formatting

* fix formatting

Co-authored-by: Microsoft Open Source <[email protected]>
Co-authored-by: Sajeetharan <[email protected]>
Co-authored-by: Aniruddh Munde <[email protected]>
Co-authored-by: Chris LaFreniere <[email protected]>
Co-authored-by: Jarupat Jisarojito <[email protected]>
Co-authored-by: Shyam Sundar J <[email protected]>
Co-authored-by: Shyam Sundar J <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cosmos DB: Paginated Query

7 participants