Composing public GraphQL APIs

[KyleAMathews]

A more relevant question now as of yesterday ;-)

So with Github or any other of what I'm assuming is the coming avalanche of public GraphQL APIs, it'd often be really handy if you could compose all or parts of a public schema with your private one.

For example, what if all of your users had a connected Github account and you could write queries like:

{
  viewer {
    githubViewer {
      repositories { edges { node { name, id } } }

  }
}

Or if there was a Wikipedia GraphQL API (PLEEAASSEEE someone build this!):

{
  viewer {
    hometown { // Jump into Wikipedia here
      population
    }
  }
}

Ideally there'd be some helper code that'd consume the introspection query and auto-construct the schema locally + provide helpers for bridging between schemas. Then there'd need to be some sort of AST slicer to pull out the part of the query that's for the remote API and then send it there to be resolved.

Thoughts? Doable? Someone already explored this further than I have? NPM package I missed ;-)?

/cc @taion who helped develop some of these ideas.

[taion]

It does seem like this could live in userspace. Something like

gitHubUser: {
  type: remoteType(gitHubSchema, 'User'),
  resolve: (obj, args, context, info) => (
    resolveRemote(
      GITHUB_API_URL,
      fragmentName => `user($user: ${obj.gitHubId}) { ${fragment} }`,
      info,
    )
  ),
}

Perhaps?

[stubailo]

I'd love to work on something like this. My first thought would be:

1. Import schema with introspection
2. Namespace all types, for example by prepending GitHub_ and Wikipedia_
3. Write some code or a DSL that adds some fields/resolvers to existing types, to wire together the schema; for example:

extend GitHub_Repository {
  wikipediaPage: Wikipedia_Page
}

resolver: (root, args, context, info) => {
  return <something goes here>
}

I think the main help here would be a way to get the relevant fragment to send to the Wikipedia API from somewhere like info.

[taion]

For introspection, I think it'd be important to do that as a build step rather than a runtime step. You don't want to be firing off an introspection query at runtime.

[stubailo]

Agreed!

[nybble73]

I've been messing around with extendSchema - but as with a few other ideas I've played with, it falls apart on interface naming (i.e. the two services both are relay compliant so implement the node interface, but extendSchema of the external services graphql and the actual services node aren't the same so it conflicts - even though the shape of them is identical).

[theorygeek]

One important consideration would be how you secure it? That is, if the query against GitHub runs with elevated privileges, you probably care about the contents of the fragment you're sending to the remote server.

Maybe you only import part of the remote schema, and exclude parts that shouldn't be accessible? Or do you look at the AST of the inner fragment and apply security before you execute the remote query or something?

Just don't want something like this:

{
  viewer {
    someRemoteApi {
      node(id: "some-object-i-shouldn't-be-able-to-access") {
        ... on SomeType { sensitiveField }
      }
    }
  }
}

[pluma]

This gets a huge 👍 from me because it ties in with my work on GraphQL at ArangoDB. Basically ArangoDB allows developers to build data-centric microservices in JS that run directly in the database and expose their own HTTP APIs. We've supported GraphQL using graphql-sync (which is a wrapper around graphql-js without promises) since ArangoDB 2.8. The upcoming release (thanks to the great example of express-graphql) will make it even easier to expose GraphQL APIs from those microservices.

With REST APIs these services would typically be put behind a public-facing proxy that takes care of authentication, CORS and so on. There are even dedicated SaaS solutions for this like Kong.

With GraphQL there's currently no way to tie multiple GraphQL APIs together like this. IMO this currently doesn't make for a compelling story for GraphQL as a replacement for REST APIs in microservices.

The security concerns when composing GraphQL APIs would be almost exactly the same as when providing a credentialed proxy to a REST API. A simple whitelist or blacklist would suffice IMO (though even that seems non-trivial to implement).

[taion]

I don't think exposing a number of GraphQL APIs internally is an effective way to build a unified public-facing GraphQL API when using microservices internally.

The missing piece conceptually is that, in an effective GraphQL API, it's useful for types to form a graph. Suppose you were building a simple blog-like structure with a "users" service and a "posts" service. It's not just the case that these can trivially live in a flat URL-like space – the User type should have a posts: [Post] field, but the Post type should have an author: User field.

Unlike with a REST API where perhaps the post service can expose an author_id field, an effective GraphQL schema should actually expose relationships – in which case you need some way to express what they are, and then you're halfway toward actually building out a GraphQL schema.

I think the good pattern here is connecting to federated services in a limited way – having a nice way to link to the GitHub data for a user is great.

The idea of a GraphQL reverse proxy like you'd use for REST is IMO not a good pattern, though. Just build an actual schema – the richness available there is one of the main benefits of using GraphQL.

[nybble73]

We're moving toward a federated graphql microservices system as well. I agree that this is not a trivial undertaking. But if our internal structure has specific guidelines on naming and global ids, it should be possible to federate to each service requests that they provide and have the central dispatching service understand how to hydrate the links between them.

[KyleAMathews]

Drupal's graphql support seems to be getting really good https://youtu.be/yhyoQwuSUWo

[nodkz]

@KyleAMathews I'm trying implement what you want in https://github.com/nodkz/graphql-compose
graphql-compose is a tool for building graphql types. Still in progress (polish API). So I'm not ready to write docs for it right now.

On top of graphql-compose, I builded this plugins:

• graphql-compose-mongoose which generate types from mongoose models.
• graphql-compose-relay which reassemble provided type with resolvers and make them Relay compatible
• graphql-compose-connection which build connection resolver for type, if type has findMany and count resolvers.
• next will be plugins for elasticsearch and dataLoader.

All these tools I'm using for our internal project. So for OSS I'm moving not so fast as I want. And compose plugins for 3rd party GraphQL Services and REST API planned at the beginning of next year.

With graphql-compose and its plugins, the building of a graphql schema on the server is not pain anymore. For me ;) Ridiculously redeem from ctrl+с, ctrl+v. Just see on example app for northwind data, where ~700LoC converted 8 mongoose models to schema with 123 graphql types [live demo] [source code]. Relay app example which works with this northwind graphql server.

---

PS. Some funny slides about the graphql difference between frontend and backend developers:

[linonetwo]

hahahah that's why we sometimes needs to use client side graphql server @nodkz

[linonetwo]

Any progress on this?

[linonetwo]

@stubailo I have an idea: Using JSON-LD 's concepts.

The JSON-LD Expansion is for transforming local types to global types in schema.org, to make a single source of types (think about the single source of truth). And namespacing a remote schema is about doing something like JSON-LD Expansion.

And JSON-LD Compaction is for using types in schema.org. It will transform long URL-like schema.org types to a short type name. (See https://www.youtube.com/watch?v=Tm3fD89dqRE if you aren't clear about it) . In GraphQL, we can just use introspection to do that, which is costly. But in JSON-LD way, what we need is to expose a 「@content」 field. And using a remote type means using something from a type source likes schema.org. we can do costless cascading by this, since cascading GraphQL may need to use tons of remote types.

Just an idea inspired by #271

[linonetwo]

This project seems to be using a remote schema https://github.com/leancloud/leancloud-graphql/blob/master/schema.js#L44

[loganpowell]

I'm from the Census Bureau. This would be huge for open data.

[linonetwo]

Now there are DBs using GraphQL in their API…How to deal with it？

[kevinsimper]

I think you would want to publish the schema as npm package that would contain the GraphQL schema for wikipedia or github and then just mount that in and write your custom resolver that was needed already :)

[linonetwo]

cnschema is considering using json-ld to compose multiple source of truth cnschema/cnSchema#8

[linonetwo]

A real use case: Neo4j now can export GraphQL API from Neo4j-GraphQL Extension, If there is a PostgreSQL exports another GraphQL API, composing them into a single source of GraphQL becomes a problem.

There is a discussion:
neo4j-graphql/neo4j-graphql#54

[linonetwo]

Things seem emerging!

ardatan/graphql-tools#382
https://github.com/stubailo/schema-stitching-demo
https://github.com/AEB-labs/graphql-weaver

[loganpowell]

Whooohooo!!!! Schema Stitching!!!
https://www.youtube.com/watch?v=DRdnGtW5AcM
https://github.com/stubailo/schema-stitching-demo

[loganpowell]

@KyleAMathews I'm also getting started with Gatsby! Cool project... Windows 10 is giving me some trouble tho (side-note). Love what you guys are doing!

[IvanGoncharov]

I don't see any actionable item for graphql-js in this issue + discussion is stale so I'm closing it.

[CreatCodeBuild]

Not sure if this is still a relevant topic, but I recently implemented a very small library that does the job.

https://github.com/CreatCodeBuild/deno-graphql/tree/master/remote-graph

The story is that in my company, we have been running GraphQL in production at a large scale for over a year. Recently we have the need to merge multiple schemas together. Since Apollo Federation is out, we deployed that. However, Federation requires changes of downstream schema if it wants to be composed in upstream schemas and it also requires a centralized gateway. This introduced a 3-way coupling among the data provider, the data consumer, and the gateway. The gateway also made our (distributed) monitoring, tracing much harder. Not saying Apollo isn't good. Apollo is amazing and we use it in production all the time.

Therefore, I created this lib out of my own frustration. This lib has zero-modification principle. That is, you don't need to change anything in the data provider(downstream schema) and as a by-product, you can

1. compose public GraphQL API which you don't own into your own schema.
2. compose schema implemented in other languages.

Here is a short example. Assuming you are writing a User service.

# User Service
type Query {
  user: User
}
type User {
  name: String
  products: [Product]  # Assuming Product is defined in another server.
}
# You only need to define the parts of Product you want to use in this service.
# Maybe Product has 10 fields, but you only need 2.
type Product {
  id: ID
  price: Float
}

Assume Product service has this schema

# Product Service
type Query {
  getProducts: [Product]
}
type Product {
  id: ID
  price: Float
  ... other fields ....
}

Here is how you define the resovler

// User Service
let resolvers = {
        user: {
                 name: ()=>{...}
                 products:  await RemoteType(
                      HTTP('product.your-domain.com'),
                      `query`,
                      `getProducts`
                );
        },
};

Done. You don't need to change how you write queries. It just works.

HTTP and RemoteType are the only 2 helper functions you need to use. What RemoteType (maybe I should name it RemoteField) does is to introspect the schema HTTP points to, and check if local type matches remote type. If so, it creates a resolver that is intelligent enough to parse info object and generate the correct queries to fetch data from downstream GraphQL services.

It also supports:

1. Authentication by http headers
2. Custom transport. HTTP is just a Transport interface. You can implement different transports. In-memory for example.
3. Batching. Yes, it works with dataloader. Just swap RemoteType with BatchedRemoteType
4. Type renaming. You can modify the type name. For example, you can define ProductOfUser instead of Product in User service.
5. Field modification. You can add extra arguments on a remote field in your local schema definition. It just works.

Todo:

1. I want to support a nice directive syntax. For example

type Query {
    me: User
}
type User @remote(url: "github.com/graphql-api", entry: "Query.viewer") {
    login:  String
    age:    Int @local  # should allow local schema to expand a remote type.
}

2. Subscription
3. Remote schema auto-generation. Instead of copy & paste remote schema into your local. Maybe I should write a tool to generate a schema language source from the introspected schema.

That been said, I need your feedback.

Full example here: https://github.com/CreatCodeBuild/deno-graphql/blob/master/remote-graph/integration-tests/example.ts