Merging schemas

[xpepermint]

I work on a huge project. To ensue project's long-term maintainability we decided to split the application into 3 standalone pieces thus we now have 3 specialized teams. Each app exposes its own GraphQL schema. At the end we plan to create the 4th app which will merge all 3 pieces together into a gateway application using the express-graphql package.

What is the best way to merge schemas? What we would need is a mechanism similar to Express Router. Our current plan is to make all 3 apps available as private packages from which we can build a new schema for the main (4th) app. Well, we could make each app act as a standalone private graphql server and then the main application would just send requests to sub-apps but I believe it could potentially represent a huge overhead. Am I wrong?

Are there any plans for supporting this? What do you suggest? How do you handle this at Facebook?

[KyleAMathews]

Just have each project export its types and combine those types together
into a schema.

On Sun, Nov 8, 2015 at 3:43 PM Kristijan Sedlak [email protected]
wrote:

I work on a huge project. To ensue project's long-term maintainability we
decided to split the application into 3 standalone pieces thus we now have
3 specialized teams. Each app exposes its own GraphQL schema. At the end we
plan to create the 4th app which will merge all 3 pieces together into a
gateway application using the express-graphql package.

What is the best way to merge schemas? What we would need is a
mechanism similar to Express Router
http://expressjs.com/guide/routing.html. Our current plan is to make
all 3 apps available as private packages from which we can build a new
schema for the main (4th) app. Well, we could make each app act as a
standalone private graphql server and then the main application would just
send requests to sub-apps but I believe it could potentially represent a
huge overhead. Am I wrong?

Are there any plans for supporting this? What do you suggest? How do you
handle this at Facebook?

—
Reply to this email directly or view it on GitHub
#223.

[xpepermint]

@KyleAMathews Thanks, that's what we plan to do. I just want to make sure that our thinking is correct.

[leebyron]

We maintain one schema at Facebook. Different teams are responsible for maintaining the types that are specific to their products, but there is only one final schema that combines all the types together into one GraphQL service available to our client applications.

[xpepermint]

@leebyron Thanks!

[worldsayshi]

@leebyron How is this done when using schema lang? If I run buildSchema it seems that I must first combine all the schema lang code into one string because I can't combine schemas once they are compiled as far as I can see..

[leebyron]

You're correct. If you're using the schema language to build a schema than you need to first combine it into a single string before calling buildSchema. That typically is a nice developer experience for small to medium sized schema, though for more complex schema we recommend building the schema programmatically. Even in this case you still need to collect all the types to assemble the schema. A similar operation to collecting all the strings before calling buildSchema

[worldsayshi]

Hmm, it seems that I don't strictly need to combine it to a string but a list of strings. There seems to be an util for combining schemas however, extendSchema. But it takes a GraphQLSchema and a Document. Not sure how to produce a Document.

[worldsayshi]

Oh! http://graphql.org/graphql-js/language/#parse

And: http://graphql.org/graphql-js/language/#source

[liamcurry]

For anyone else looking for a way to merge .graphql files, I've started working on some helper scripts in this repo: https://github.com/liamcurry/gql

You can use this to merge .graphql files into a single file. For example, if you have these files:

# schema1.graphql
type Query {
    foo: String
}

type Mutation {
    bar(input: String): String
}

# schema2.graphql
type Query {
    hello: String
}

type Mutation {
    world(input: String): String
}

and you ran this:

npm install -g gql-cli
gql merge **/*.graphql

You'd end up with this output:

type Query {
    foo: String
    hello: String
}

type Mutation {
    world(input: String): String
    bar(input: String): String
}

It should work with other type definitions too (enums, interfaces, inputs, etc).

[stubailo]

@liamcurry wow, this would be a really awesome addition to graphql-tools, since it would make it easy to split up stuff like the root Query type: https://github.com/apollostack/graphql-tools

Is it available as a library without the CLI?

[liamcurry]

@stubailo Yes it is. Check out the gql-merge npm package here: https://www.npmjs.com/package/gql-merge

The source for that is all in this file: https://github.com/liamcurry/gql/blob/master/packages/gql-merge/src/index.js

Here's an example (untested):

import {mergeStrings} from 'gql-merge'

const testSchema1 =
`type Query {
    hello: String
}`

const testSchema2 =
`type Query {
    world: String
}`

const resultSchema = mergeStrings([testSchema1, testSchema2])

console.log(resultSchema)
/* Should print this to the console:
type Query {
    hello: String
    world: String
}
*/

[stubailo]

Nice, we should definitely use that in graphql-tools - right now we just concatenate the strings but merging would probably be better!

[akeelnazir]

I having issues where two merged schemas have a name-space collision. Say "Reservation" is defined in both schemas and when I try to merge them into a Root_Schema it is throwing an error. Are there any best practices that I should follow to avoid such problems?

[leebyron]

Hard to say without knowing why you're merging schema in the first place.

If you're merging two schema that come from two unrelated sources (perhaps two different companies) - then you're treading into fairly new GraphQL territory! I've seen people attempting this sort of thing work around this by adding a pseudo namespace to the foreign source. For example if you were merging Github's GraphQL schema into yours, you might replace every type in Github's schema with "Github" + typename such that Repository becomes GithubRepository. This way you can still have control over the names of all types in your schema.

If you're merging two schema that you control, then you likely need some sort of internal tool or integration test which blocks commits which introduce these kinds of issues. While we don't merge schemas anywhere at Facebook, we have a similar situation where someone might inadvertently create a new type which is named the same as an existing type in our schema, resulting in two types of the same name. We run a simple integration test before commit which asserts that this doesn't occur.

[smolinari]

I know what I am about to write goes against the single root node concept of GraphQL. But to me, this could be another case of splitting the root to a bunch of uniquely identifiable resources. With this split there is no need to mix schema at all. This would only have the requirement of each resource needing its own unique name. So, for the reservations example mentioned, you'd need to have "/reservationsA" and "/reservationsB".

Sure, it means extra calls to the server to get the data, but it isn't going to REST kinds of extremes. So, is calling on the "main branches" of the graph really all that bad? With a big app, this "one level down from the root" would offer a slight avoidance of too much coupling, which obviously a singular schema is causing (like with this issue and others).

I'd love to understand why this is a stupid idea. I'm sure there is a reason for wanting to always have just the one "graph" and not request only the main branches and only when they are needed.

Scott

[ariesshrimp]

@leebyron

Hard to say without knowing why you're merging schema in the first place.

One use case is just project file organization. This proposal is intuitive to me: ardatan/graphql-tools#186

Spreading out query/type/mutation definitions across a directory structure organized by entities.

- API
    |_ users
        |_ types.js or types.graphql whatever
        |_ queries.js
        |_ resolvers.js
        |_ mutations.js
    |_ authors
        ...
    |_ books
        ...
    |_ collections
        ...
    ...
    Schema.js <- import everything and put it together in here for your /graphql endpoint to consume

Without some pattern for dividing things up, how do you avoid enormous Query definitions at Facebook? Consider this file defining a single entity's query: https://github.com/joefraley/humane-society-api/blob/master/api/animals/queries.js

It already seems long for my taste. I can only imagine that things are more complicated at Facebook.

[ariesshrimp]

@liamcurry graphql-tools defines a function makeExecutableSchema that accepts all your Schema's type definitions. Something like your graphql-merge is exactly what I imagine happening under the hood there.

[jamesgorman2]

@smolinari namespace collision when naively merging means you are have distinct two Bounded Contexts. The recommendation from Domain Driven Design would be to either have two graphql resources/domains (ie two completely seperate graphs) with shared references or produce a new Bounded Context for your UI domain that is designed specifically to support you UI domain.[1]

Why is this necessary? The longer story is about impedance mismatch between your layers,[2] but, in short, if you reservationA and reservationB and I'm building the UI, which one do I use? Worse yet if I'm updating things do I use updateReservationA or updateResrvationB. While you can get away with just documenting which one to use when, a better long term solution is to deliberately provide a new type reservation that is the union/concatenation/etc of reservationA and reservationB. It will now be transparent to the UI builder which reservation to use. If the UI needs to know that specific data comes from a or b you can put that clearly as fields in the base reservation type, otherwise you can have a completely transparent API.

@akeelnazir beyond this I've been putting together graphql-type-repository which will check the merging of schema fragments and report where conflicts (and other errors) are happening. It should be in beta by the end of the year.

[1] I use UI domain here since graphql's main aim is to support more efficient communications between UI code and server code.
[2] in DDD, as distinct from Object Relational impedance mismatch, this speaks to the ability of your team to talk about their work with users - ideally all terms in the users vocabulary match to the UI and to the services and data storage layer. When you need to merge domains you should have clear points of inflection so it is obvious two domains are merging.

[smolinari]

@jamesgorman2 - right. I was thinking the whole time the reservationA and reservationB example is a poor one for splitting up the graph branches into their own resources/ APIs. There should only be a single "reservation" resource.

@leebyron Lee said,

Different teams are responsible for maintaining the types that are specific to their products.

And I was contemplating some about that. How do teams know they aren't duplicating types? If the merge tool mentioned above works, that is great and there are no worries. But, doesn't duplicating types unknowingly mean wasted developer effort, at least theoretically? Does this possible duplication happen so rarely, it doesn't matter? Or maybe, to avoid duplication, the front-end devs are told to first search the Schema with a tool like GraphiQL to see what is available first, before requesting that any new Types be added? I'd really love to learn the Facebook workflow dealing with Type creation better. It would help me, and I am certain others too, understand possible best practices, when working with a large schema. A couple questions that burn in my mind are:

How often does the Facebook schema get updated/ changed?
Is the back-end creation of types (and even resolvers) automated in any way? If not, why not?

Scott

[turadg]

While we don't merge schemas anywhere at Facebook, we have a similar situation where someone might inadvertently create a new type which is named the same as an existing type in our schema, resulting in two types of the same name. We run a simple integration test before commit which asserts that this doesn't occur.

Maybe I'm missing something, but is #744 a viable solution to preventing duplicate type definitions?

[abhiaiyer91]

Hey there! Sorry to comment on a closed issue, but I'm currently trying to merge schemas and going with the namespace approach @leebyron mentioned above.

Does anyone have a straight forward approach to this? Currently I'm trying to muck around with the schema ast using visit and hit success renaming types until I hit the ofType key, which has its own nested structure.

Is there a better way to merge the schemas?

What id like to do is just namespace all types in a schema with some prefix. The end goal being is showing the types from many GraphQL servers in one GraphiQL. I'm trying to build a GraphQL proxy server!

[leebyron]

I recommend looking at extendSchema.js in this repo. You can likely borrow a lot from how it is implemented for this purpose

[RodMachado]

Hey @abhiaiyer91

Take a look at the merge-graphql-schemas package. Maybe your use case is a little different from what the package is trying to solve, but we could make some adjustments if it's something that might benefit other people as well. 😄

Cheers!

[jasonbahl]

@abhiaiyer91 I also have the need to setup a proxy server. We have ~10 different systems with nearly identical schemas (but potential to be different) and another system with a completely different schema. We want to be able to query data from a single endpoint and have that server handle passing the request to the appropriate other GraphQL server for resolving.

I'd be curious to see what you come up with.

[abhiaiyer91]

Oh man! I have a dope solution to this problem I'll open source with a few more prod level tests! Free to use!!

[jasonbahl]

@abhiaiyer91 Can't wait to see it!

[jrop]

@abhiaiyer91 I am curious to see the solution you developed to this problem.

[abhiaiyer91]

@jrop I worked with the apollo team to get this done!

https://github.com/Workpop/roxy is sort of an outdated version of this but the source code lives in

https://github.com/apollographql/graphql-tools

[lastmjs]

@abhiaiyer91 Where is it in graphql-tools?

[tomyitav]

This post demonstrates how to modularize your graphql schemas. Hope it helps! :)