Skip to content

Conversation

@ciaranschutte
Copy link
Contributor

@ciaranschutte ciaranschutte commented Jun 5, 2024

  • retrieval of remote schemas
  • responses build to full schema using gql tools (means we can use the well tested inbuilt methods without manually parsing a gql request as intermediary objects)
  • output is array of full schemas as GraphQlObjects

Please note this is part of the bigger schema federation work merging into feat/featured_search
The broken build issues are for some missing TS types, implicit anys
These will be addressed before merging full feature into develop branch

Ciaran Schutte and others added 30 commits June 4, 2024 22:33
Co-authored-by: Anders Richardsson <[email protected]>
// type cast because rejected errors are handled
const networkQueries = (await Promise.allSettled(
networkQueryPromises,
)) as PromiseFulfilledResult<FetchRemoteSchemaResult>[];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should not use this type assertion. Promise.allSettled returns a PromiseSettledResult<T> where T is your type.

This is important to use as any rejected response will NOT have a .value property, we need to always check for the .status in each element of network queries array.

See type information for PromiseSettledResult<T>: https://github.com/microsoft/TypeScript/blob/main/src/lib/es2020.promise.d.ts#L11


// build schema
const schemaResults = networkQueries.map((networkResult) => {
const { config, introspectionResult } = networkResult.value;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.value won't exist on rejected promises. We can filter those out if we only care about succeses:

const schemaResults = networkQueries
		.filter(
			(result): result is PromiseFulfilledResult<FetchRemoteSchemaResult> =>
				result.status === 'fulfilled',
		)
		.map((networkResult) => {
			const { config, introspectionResult } = networkResult.value;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's how the code is currently handling

@ciaranschutte
Copy link
Contributor Author

@joneubank addressed feedback. TS building. please take another look when you have some time.

@ciaranschutte ciaranschutte requested a review from joneubank June 21, 2024 17:52
Copy link
Contributor

@joneubank joneubank left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking strong throughout.

The axios request failure handling I think is not what you would expect to see so consider that comment and if there are changes needed.

Comment on lines +4 to +7
/**
* Creates a graphql query string with variables for use in a POST request
* @returns string
*/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙏

Comment on lines 13 to 16
/**
* TODO: Add typing for variables. Type will be dependant on gql endpoint being queried.
*/
variables: unknown;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at how this is used, we don't really care about the specific contents of the variables. This function is doing no work to ensure that the variables expected in the query are included in variables (or vice versa). The only thing we know is we want to be able to run JSON.stringify(variables) which takes any. In practice, we expect this parameter to be an object with some variable values which themselves could be anything, and we probably don't want it to be string or number etc. So in this case I might reccomend:

variables: Record<string, any>

* Creates a graphql query string with variables for use in a POST request
* @returns string
*/
export const createGqlQuery = ({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noticed this function isn't actually used in your code here, is that expected?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

expected. #877 (comment)

};

export const fetchGql = ({ url, gqlRequest }) => {
export const fetchGql = ({ url, gqlRequest }: { url: string; gqlRequest: { query: string } }) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TSDoc will be wanted before feature branch is complete.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think your TSDoc is incorrect, instead of returning an axios instance that you can use to send future requests, this sends your defined request with axios and returns the AxiosResponse (or throws error, stupid axios).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great feedback, useful insight... but let's try keeping unhelpful personal opinions to a minimum.

type FetchRemoteSchemaResult = {
config: NetworkAggregationConfigInput;
introspectionResult: IntrospectionQuery | null;
introspectionResult: IntrospectionQuery | undefined;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not just:

introspectionResult?: IntrospectionQuery

Comment on lines 57 to 58
throw new NetworkAggregationError(
`Unexpected data in response object. Please verify the endpoint at ${graphqlUrl} is returning a valid GQL Schema.`,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Comment on lines 67 to 71
} catch (error) {
console.log(`failed to retrieve schema from url: ${config.graphqlUrl}`);
console.error(error);
return { config, introspectionResult: undefined };
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Axios throws errors on status 4xx and 5xx. That means that the else block you have above is behaviour that you are likely expecting to be handled here as well. You should try checking if the error here is an instance of AxiosError, and if so you can handle these error statuses in the catch block. I imagine you will want to throw your NetworkAggregationError.

Comment on lines 86 to 89
.filter(
(result): result is PromiseFulfilledResult<FetchRemoteSchemaResult> =>
result.status === 'fulfilled',
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With TS 5.5 we no longer require the result is return type :D

} catch (error) {
console.log(`failed to retrieve schema from url: ${config.graphqlUrl}`);
console.error(error);
return { config, introspectionResult: undefined };
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PS. don't need to do introsepectionResult: undefined if you have the type defined as introspectionResult?: IntrospectionQuery

@ciaranschutte ciaranschutte requested a review from joneubank June 25, 2024 20:36
};

export const fetchGql = ({ url, gqlRequest }) => {
export const fetchGql = ({ url, gqlRequest }: { url: string; gqlRequest: { query: string } }) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think your TSDoc is incorrect, instead of returning an axios instance that you can use to send future requests, this sends your defined request with axios and returns the AxiosResponse (or throws error, stupid axios).

Comment on lines +3 to +8
// environment config
export type NetworkAggregationConfigInput = {
graphqlUrl: string;
documentType: string;
displayName: string;
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment here doesn't tell me anything new. You can use TSDoc on types as well to leave a description of the type in intellisense for future devs that use it.

Comment on lines +10 to +13
// additional properties after initialising network
export type NetworkAggregationConfig = NetworkAggregationConfigInput & {
schema?: GraphQLSchema;
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a helpful comment but it wasn't easily understandable. I had to read the type information to understand what you mean. Maybe something like:

Suggested change
// additional properties after initialising network
export type NetworkAggregationConfig = NetworkAggregationConfigInput & {
schema?: GraphQLSchema;
};
/**
* Complete aggregation config that defines the network search setup. This includes the original
* config information plus computed fields that are generated in the network search initialization process.
*/
export type NetworkAggregationConfig = NetworkAggregationConfigInput & {
schema?: GraphQLSchema;
};

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a nice improvement, for sure!
Naive question: am I the only one who gets bugged by the line length inconsistency?
personally, I'd chop it like this

* Complete aggregation config that defines the network search setup.
* This includes the original config information plus computed fields
* that are generated in the network search initialization process.

* @returns
*/
const isGqlIntrospectionQuery = (input: unknown): input is IntrospectionQuery =>
!!input && typeof input === 'object' && '__schema' in input && typeof input.__schema === 'object';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wouldn't typeof input === 'object' inherently cover the !!?
(an object cannot be falsy)

Suggested change
!!input && typeof input === 'object' && '__schema' in input && typeof input.__schema === 'object';
typeof input === 'object' && '__schema' in input && typeof input.__schema === 'object';

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does it matter that this function would allow input to be an array customised to include __schema?

Copy link
Contributor Author

@ciaranschutte ciaranschutte Jun 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1st point:
typeof null is object - <3 js (so '__schema' in input would break)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2nd point:
I don't think that's a concern. I'm using the getIntrospectionQuery which is part of the graphql library (conforming to spec) and this check is only happening to check response data from a GQL server.

It queries an automatically generated field in a graphql server, not a user created field.
For an array to occur the graphql spec would need to be changed.

Copy link
Member

@justincorrigible justincorrigible Jun 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a null.__schema? otherwise, !! is still redundant

introspectionSchema !== undefined ? buildClientSchema(introspectionSchema) : undefined;
return { ...config, schema };
} catch (error) {
console.error('build schema error', error);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if these messages were more descriptive. The best error messages let the reader know what was being attempted, why it failed, and (if possible) what action should be taken.

It is likely that this is about on par with other error messaging in this repo, but working to improve these as we go would be a huge help.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for sure - this is just the "retrieve schema" step
buildClientSchema is not implemented
will be keeping this in mind working on the "build schema" ticket

@ciaranschutte ciaranschutte merged commit 4f62745 into feature/federated_search Jun 26, 2024
@ciaranschutte ciaranschutte deleted the feat/#873_retrieve_remote_schemas branch June 26, 2024 15:14
ciaranschutte added a commit that referenced this pull request Mar 14, 2025
* initial config

* add network to config template. add default values to base.json

* add schema merge placeholders

* add conditional to graphql route gen for env flag

* throw error on invalid json parse eg. empty file

* rename type

* add vscode debug file to gitignore

* add gql client lib

* change gql lib

* Update modules/server/README.md

Co-authored-by: Anders Richardsson <[email protected]>

* add fetch field

* rename field to documentType

* Update modules/server/configTemplates/base.json

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/config/utils/getConfigFromFiles.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update types.ts

* Update README.md

* add ignore to gitignore

* rename fields, add nesting

* remove env config to enable federated search

* remove env config to enable federated search

* remove flag from env schema

* correct schema retrieval

* fix GQL obj wrapping on schema construction

* example

* cleanup

* cleanup gql file

* cleanup packages

* fix import

* add introspection query type predicate

* add tsdoc

* remove local folder ignore from project.gitignore

* remove unused util

* renaming

* break up query promise and results

* Update .gitignore

Co-authored-by: Anders Richardsson <[email protected]>

* use axios

* axios network check, doesn't throw like fetch

* change fetch config to axios config

* Update modules/server/src/network/index.ts

Co-authored-by: Anders Richardsson <[email protected]>

* add logs to thrown errors

* small feedback fixes

* add typings + additional info  in comments

* add better errors

* remove unused func. renaming

* add return type

* remove assertion. add TS filter

* change nulls to undefined

* change explicit undefined to optional ts

* add tsdoc comment

* change var type to loose record instead of uknown

* if else TS

* if else TS

* change to nested promise return

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>
ciaranschutte added a commit that referenced this pull request Mar 27, 2025
* initial config

* add network to config template. add default values to base.json

* add schema merge placeholders

* add conditional to graphql route gen for env flag

* throw error on invalid json parse eg. empty file

* rename type

* add vscode debug file to gitignore

* add gql client lib

* change gql lib

* Update modules/server/README.md

Co-authored-by: Anders Richardsson <[email protected]>

* add fetch field

* rename field to documentType

* Update modules/server/configTemplates/base.json

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/config/utils/getConfigFromFiles.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update types.ts

* Update README.md

* add ignore to gitignore

* rename fields, add nesting

* remove env config to enable federated search

* remove env config to enable federated search

* remove flag from env schema

* correct schema retrieval

* fix GQL obj wrapping on schema construction

* example

* cleanup

* cleanup gql file

* cleanup packages

* fix import

* add introspection query type predicate

* add tsdoc

* remove local folder ignore from project.gitignore

* remove unused util

* renaming

* break up query promise and results

* Update .gitignore

Co-authored-by: Anders Richardsson <[email protected]>

* use axios

* axios network check, doesn't throw like fetch

* change fetch config to axios config

* Update modules/server/src/network/index.ts

Co-authored-by: Anders Richardsson <[email protected]>

* add logs to thrown errors

* small feedback fixes

* add typings + additional info  in comments

* add better errors

* remove unused func. renaming

* add return type

* remove assertion. add TS filter

* change nulls to undefined

* change explicit undefined to optional ts

* add tsdoc comment

* change var type to loose record instead of uknown

* if else TS

* if else TS

* change to nested promise return

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>
ciaranschutte added a commit that referenced this pull request Apr 22, 2025
* Network aggregation search - Initial Config (#876)

* initial config

* add network to config template. add default values to base.json

* add schema merge placeholders

* add conditional to graphql route gen for env flag

* throw error on invalid json parse eg. empty file

* rename type

* add vscode debug file to gitignore

* Update modules/server/README.md

Co-authored-by: Anders Richardsson <[email protected]>

* rename field to documentType

* Update modules/server/configTemplates/base.json

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/config/utils/getConfigFromFiles.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update types.ts

* Update README.md

* rename fields, add nesting

* remove env config to enable federated search

* remove env config to enable federated search

* remove flag from env schema

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Feat/#873 retrieve remote schemas (#877)

* initial config

* add network to config template. add default values to base.json

* add schema merge placeholders

* add conditional to graphql route gen for env flag

* throw error on invalid json parse eg. empty file

* rename type

* add vscode debug file to gitignore

* add gql client lib

* change gql lib

* Update modules/server/README.md

Co-authored-by: Anders Richardsson <[email protected]>

* add fetch field

* rename field to documentType

* Update modules/server/configTemplates/base.json

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/config/utils/getConfigFromFiles.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update types.ts

* Update README.md

* add ignore to gitignore

* rename fields, add nesting

* remove env config to enable federated search

* remove env config to enable federated search

* remove flag from env schema

* correct schema retrieval

* fix GQL obj wrapping on schema construction

* example

* cleanup

* cleanup gql file

* cleanup packages

* fix import

* add introspection query type predicate

* add tsdoc

* remove local folder ignore from project.gitignore

* remove unused util

* renaming

* break up query promise and results

* Update .gitignore

Co-authored-by: Anders Richardsson <[email protected]>

* use axios

* axios network check, doesn't throw like fetch

* change fetch config to axios config

* Update modules/server/src/network/index.ts

Co-authored-by: Anders Richardsson <[email protected]>

* add logs to thrown errors

* small feedback fixes

* add typings + additional info  in comments

* add better errors

* remove unused func. renaming

* add return type

* remove assertion. add TS filter

* change nulls to undefined

* change explicit undefined to optional ts

* add tsdoc comment

* change var type to loose record instead of uknown

* if else TS

* if else TS

* change to nested promise return

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Pr feedback - doc updates (#879)

* update tsdoc

* update tsdoc

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/#874 merge schemas (#878)

* initial config

* add network to config template. add default values to base.json

* add schema merge placeholders

* add conditional to graphql route gen for env flag

* throw error on invalid json parse eg. empty file

* rename type

* add vscode debug file to gitignore

* add gql client lib

* change gql lib

* Update modules/server/README.md

Co-authored-by: Anders Richardsson <[email protected]>

* add fetch field

* rename field to documentType

* Update modules/server/configTemplates/base.json

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/config/utils/getConfigFromFiles.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update types.ts

* Update README.md

* add ignore to gitignore

* rename fields, add nesting

* remove env config to enable federated search

* remove env config to enable federated search

* remove flag from env schema

* correct schema retrieval

* fix GQL obj wrapping on schema construction

* example

* cleanup

* cleanup gql file

* cleanup packages

* fix import

* basic merge schema

* cleanup comment

* exclusuions

* add introspection query type predicate

* add tsdoc

* remove local folder ignore from project.gitignore

* remove unused util

* renaming

* break up query promise and results

* Update .gitignore

Co-authored-by: Anders Richardsson <[email protected]>

* use axios

* axios network check, doesn't throw like fetch

* update packages for typedefs

* change fetch config to axios config

* Update modules/server/src/network/index.ts

Co-authored-by: Anders Richardsson <[email protected]>

* add logs to thrown errors

* fix merge conflict

* add test watch command with basic test

* tests

* Fix type. narrow scope. hoist filtering

* remove debug code. add comment

* update comments + add type assertion

* add comments

* remove redundant options arg for schema merginging

* seperate test utils and fixture from main file

* seperate test utils and fixture from main file

* tests, cleanup, extra tests, split responsibility

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Add core functionality for remote connection resolvers (#880)

* add remote connection resolver func

* adds resolver functionality for remote connection details, including health check to gql server

* add types, improve comment, move into remote connection resolver into distinct file

* remove unused imports

* add utility type. use utility objectValues type for type values from const

* fix type lookup in gql object

* add comments

* PR feedback: comment blocks

* additional remote connection node data resolver response data

* create typeDefs file for remote connection data

* move typeDefs for aggregations

* add resolver map, rename remote connection resolver file

* cleanup imports and creation of schema, imorts from distinct resolver and typedef files

* rename and fix type on Remote Connection gql resp

* remove ping healthcheck for nodes on resolver

* pass correct object to get available types

* add temp gql server for network aggs

* add status check

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/agg resolver (#881)

* add remote connection resolver func

* adds resolver functionality for remote connection details, including health check to gql server

* add types, improve comment, move into remote connection resolver into distinct file

* remove unused imports

* add utility type. use utility objectValues type for type values from const

* fix type lookup in gql object

* add comments

* PR feedback: comment blocks

* additional remote connection node data resolver response data

* create typeDefs file for remote connection data

* move typeDefs for aggregations

* add resolver map, rename remote connection resolver file

* cleanup imports and creation of schema, imorts from distinct resolver and typedef files

* rename and fix type on Remote Connection gql resp

* remove ping healthcheck for nodes on resolver

* pass correct object to get available types

* add temp gql server for network aggs

* add status check

* fetchGQL helper func working with axios

* rework dynamic network search types based on remote connections types

* use __type query instead of full introspectionQuery

* remove old code

* add single typeDef export, responsible for merging remote connection and aggregation types

* correct structure for merging typedefs with other typedefs, merge on Query

* get all types from remote connections once

* agg resolver structuring

* add aggregations resolvers

* add agg resolver network calls

* rename and narrow NetworkType type across files

* add common file

* add network queries file

* rename var

* query remote connections

* move query

* add comments

* add supported and unsupported aggregations with test

* ts-jest disable diagnostics to run tests without all types passing

* rework connection status property to do ping healthcehck

* types cleanup across files

* improved typing for reducer in field types, covering supported and unsupported aggregates

* tighten and rename types

* type cleanup and comments

* add explicit name mapping

* rename type correctly

* Cleanup query lookup for aggregations for remote connections

* move TSdoc comment

* move types

* add util func to convert gql AST into string

* move aggregations resolver code into a module

* simplify query to be composable, better naming for agg query map

* aggregation logic placeholder

* cleanup

* rework logic to use single request and mapping fields, instead of slicing central query

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/resolve stats (#883)

* add resolve aggregation function with tests

* add basic aggregation TS types

* cleanup

* improve comments

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/resolver responses (#884)

* add documentName to config and use in remote connection gql queries

* move fulfilled promise type guard to util and make it a generic

* resolve aggregations from network query results'

* return resolved response object

* PR feedback

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fix network agg resolution (#885)

* add Bucket gql object type

* text change, move func outside of nested func

* fix resolved aggregations overriding

* Update modules/server/src/network/aggregations/index.ts

Co-authored-by: Anders Richardsson <[email protected]>

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Response data & batching requests (#887)

* move response object creation into module

* change type structure for gql server

* change resolver code based on new gql type

* adjust agg type gql resolved response

* add http resp helpers

* change resolveAggregation code to batch processing

* add nodeInfo to resolver

* change resolver method to reducer with mutations

* add type

* fix typedef

* format response object

* comments and cleanup

* remove log

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fed Search request timeouts (#889)

* move response object creation into module

* change type structure for gql server

* change resolver code based on new gql type

* adjust agg type gql resolved response

* add http resp helpers

* change resolveAggregation code to batch processing

* add nodeInfo to resolver

* change resolver method to reducer with mutations

* add type

* fix typedef

* format response object

* comments and cleanup

* add GQL request timeout with config option and default value

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Reorg connection (#891)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Accumulator pipeline (#893)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed total hits (#895)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

* add total hits to query string

* cleanup field, param order

* add test

* rename RemoteAggregations type

* add hits TS type

* fix env toggle for network fed search

* renaming, update types

* documentName and documentType are the same for functionality

* return unique fields

* fix accumulator resolve for new input shape

* tighten types. add Hits resolution

* rename count to hits

* fix logging

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* feat/agg acc tests (#897)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

* add total hits to query string

* cleanup field, param order

* add test

* rename RemoteAggregations type

* add hits TS type

* fix env toggle for network fed search

* renaming, update types

* documentName and documentType are the same for functionality

* return unique fields

* fix accumulator resolve for new input shape

* tighten types. add Hits resolution

* rename count to hits

* add first test for requested fields

* tighten types and error checking

* fix up types and comments, resolvers code fix

* update aggregation test

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed numeric agg resolution (#898)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

* add total hits to query string

* cleanup field, param order

* add test

* rename RemoteAggregations type

* add hits TS type

* fix env toggle for network fed search

* renaming, update types

* documentName and documentType are the same for functionality

* return unique fields

* fix accumulator resolve for new input shape

* tighten types. add Hits resolution

* rename count to hits

* add first test for requested fields

* tighten types and error checking

* fix up types and comments, resolvers code fix

* update aggregation test

* add test + expand types

* rework resolution iteration and add numericAggregations resolver

* add typename to types

* cleanup

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* use Math methods (#900)

Co-authored-by: Ciaran Schutte <[email protected]>

* Individual node config query fields (#899)

* use more detailed NodeConfig object to store available agg types

* filter requested aggs with available aggs

* add available aggs to node connection resolver type defs

* cleanup

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/aggregate not available (#903)

* add TSdoc example

* add TSdoc comment

* pass more data into Aggs accumulator

* add schema map for query time

* rework agg accumulator to add empty agg to bucket if aggregation not available for node

* fix test

* clean up logic to account for only Aggregations type

* account for empty aggs field in node query

* cleanup

* disabled network Agg tests

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/filters (#904)

* add TSdoc example

* add TSdoc comment

* pass more data into Aggs accumulator

* add schema map for query time

* rework agg accumulator to add empty agg to bucket if aggregation not available for node

* fix test

* clean up logic to account for only Aggregations type

* account for empty aggs field in node query

* cleanup

* disabled network Agg tests

* add args to Aggregations gql typedef

* add args to network field

* pass gql resolvers args to query creation

* remove console log

* add include missing:

* add overture sqon builder dep

* move up type

* add SQON check and args type

* rename file

* check for SQON correctness but pass ordinary object through on success

* add comment

* Update modules/server/src/network/utils/sqon.ts

Co-authored-by: Anders Richardsson <[email protected]>

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Minor comment updates (#905)

* improve comment

* improve comment

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fed search cleanup typings (#906)

* remove unsued type defs

* add type watch script

* fix import

* fix import

* type an ANY type

* fix agg resolver typing

* ts no-check for test file

* fix Agg Accumulator types

* type response formatter

* fix test file type

* address feedback

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fix broken build (#907)

* remove unsued type defs

* add type watch script

* fix import

* fix import

* type an ANY type

* fix agg resolver typing

* ts no-check for test file

* fix Agg Accumulator types

* type response formatter

* fix test file type

* fix gql query creation test

* query back to normal test

* disables tests, adds comment

* fix merged file

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* add agg mode env config (#908)

* add agg mode env config

* env config to strip out section of gql schema, typedef + resolver

* prettier

* prettier

* Update modules/server/.env.schema

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/createConnectionResolvers.ts

Co-authored-by: Anders Richardsson <[email protected]>

* alphabetize

* move up import

* change AGG_ONLY_MODE to enabled DOCUMENT_HITS

* add empty line at EOF

* add comment

* fix typo

* fix inverted logic

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Feature/threshold calc (#909)

* add conditional prop to schema

* conditional schema field

* conditional add dataMasking schema fields

* add data masking logic

* add agg only conditional to resolvers

* clarify todo

* get optional aggregations field and values for hits resolution

* add mask func stub

* fix missing prop from spread

* masking logic

* conditional resolve hits

* move thresholding higher on resolver chain

* add falsey belowThreshold value instead of null

* data mask threshold env var

* update wrong process.env path

* seperate resolver creation from routing

* check for undefiend value in lookup

* clarify comment

* add threshold value explanation

* move types to common file

* typing

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* multiple node with data thresholding (#912)

* fix build error

* single endpoint

* fix GQL type name collisions

* move data mask threshold env. ensure data masking applies even if hits.total is not queried

* change belowThreshold to relation object

* surface api errors to axios responses

* add relation logic and agg merging for network search

* read data mask threshold min from config

* check for valid env val

* remove unused func

* fix TS build errors

* fix schema merge error

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed search cleanup (#920)

* add placeholder

* Adjust config template example and file

* remove unused arranger constructor param

* reorder constants file, parse env var

* update arranger network config type

* type, add gql server types

* graphqlRoutes, remove unused import, export func

* resolver creation, improve TS usage, split functionality into distinct files, improve usage'

* add surrounding brackets for func

* cleanup schema resolvers

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed search networked cleanup (#921)

* cleanup utils

* reorg types, decompose file, improve comments

* clean up types

* cleanup types, redundant code and fix a couple of bad merge files

* fix enum

* cleanup test types

* type fix

* explicit type import, use path alias

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Update modules/server/src/app.js

Co-authored-by: Anders Richardsson <[email protected]>

* make horrible ajax.ts console logging somewhat useful

* improve release process

* fix paths and imports for new build config

* update tests to todos

* fix some imports

* fix build issues with lodash and missing config flag

* fix paths

* comment out test files, broken with tsx vs jest

* fix path

* fix path

* improve release process

* allow toolbar options customization

* fix setsIndex conflict resolutioon misses

* default for ENABLE_DOCUMENT_HITS env flag is true

* default enable document hits to true

* add DATA_MASK_MIN_THRESHOLD env var as pass through var

* debug

* debug

* debug

* add default param value

* debug

* string util important fix

* fix for tests

* Add comment re data masking

* remove debug log

* Update modules/server/src/mapping/createConnectionTypeDefs.js

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/createConnectionTypeDefs.js

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/masking.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/resolveHitsFromAggs.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/resolvers.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/resolvers.ts

Co-authored-by: Anders Richardsson <[email protected]>

* remove specific NetworkAggError class

* fix error response type narrow

* missing param

* add explanation comment for aggs resolver, gql info lookup

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>
justincorrigible added a commit that referenced this pull request May 5, 2025
* Network aggregation search - Initial Config (#876)

* initial config

* add network to config template. add default values to base.json

* add schema merge placeholders

* add conditional to graphql route gen for env flag

* throw error on invalid json parse eg. empty file

* rename type

* add vscode debug file to gitignore

* Update modules/server/README.md

Co-authored-by: Anders Richardsson <[email protected]>

* rename field to documentType

* Update modules/server/configTemplates/base.json

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/config/utils/getConfigFromFiles.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update types.ts

* Update README.md

* rename fields, add nesting

* remove env config to enable federated search

* remove env config to enable federated search

* remove flag from env schema

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Feat/#873 retrieve remote schemas (#877)

* initial config

* add network to config template. add default values to base.json

* add schema merge placeholders

* add conditional to graphql route gen for env flag

* throw error on invalid json parse eg. empty file

* rename type

* add vscode debug file to gitignore

* add gql client lib

* change gql lib

* Update modules/server/README.md

Co-authored-by: Anders Richardsson <[email protected]>

* add fetch field

* rename field to documentType

* Update modules/server/configTemplates/base.json

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/config/utils/getConfigFromFiles.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update types.ts

* Update README.md

* add ignore to gitignore

* rename fields, add nesting

* remove env config to enable federated search

* remove env config to enable federated search

* remove flag from env schema

* correct schema retrieval

* fix GQL obj wrapping on schema construction

* example

* cleanup

* cleanup gql file

* cleanup packages

* fix import

* add introspection query type predicate

* add tsdoc

* remove local folder ignore from project.gitignore

* remove unused util

* renaming

* break up query promise and results

* Update .gitignore

Co-authored-by: Anders Richardsson <[email protected]>

* use axios

* axios network check, doesn't throw like fetch

* change fetch config to axios config

* Update modules/server/src/network/index.ts

Co-authored-by: Anders Richardsson <[email protected]>

* add logs to thrown errors

* small feedback fixes

* add typings + additional info  in comments

* add better errors

* remove unused func. renaming

* add return type

* remove assertion. add TS filter

* change nulls to undefined

* change explicit undefined to optional ts

* add tsdoc comment

* change var type to loose record instead of uknown

* if else TS

* if else TS

* change to nested promise return

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Pr feedback - doc updates (#879)

* update tsdoc

* update tsdoc

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/#874 merge schemas (#878)

* initial config

* add network to config template. add default values to base.json

* add schema merge placeholders

* add conditional to graphql route gen for env flag

* throw error on invalid json parse eg. empty file

* rename type

* add vscode debug file to gitignore

* add gql client lib

* change gql lib

* Update modules/server/README.md

Co-authored-by: Anders Richardsson <[email protected]>

* add fetch field

* rename field to documentType

* Update modules/server/configTemplates/base.json

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/config/utils/getConfigFromFiles.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update types.ts

* Update README.md

* add ignore to gitignore

* rename fields, add nesting

* remove env config to enable federated search

* remove env config to enable federated search

* remove flag from env schema

* correct schema retrieval

* fix GQL obj wrapping on schema construction

* example

* cleanup

* cleanup gql file

* cleanup packages

* fix import

* basic merge schema

* cleanup comment

* exclusuions

* add introspection query type predicate

* add tsdoc

* remove local folder ignore from project.gitignore

* remove unused util

* renaming

* break up query promise and results

* Update .gitignore

Co-authored-by: Anders Richardsson <[email protected]>

* use axios

* axios network check, doesn't throw like fetch

* update packages for typedefs

* change fetch config to axios config

* Update modules/server/src/network/index.ts

Co-authored-by: Anders Richardsson <[email protected]>

* add logs to thrown errors

* fix merge conflict

* add test watch command with basic test

* tests

* Fix type. narrow scope. hoist filtering

* remove debug code. add comment

* update comments + add type assertion

* add comments

* remove redundant options arg for schema merginging

* seperate test utils and fixture from main file

* seperate test utils and fixture from main file

* tests, cleanup, extra tests, split responsibility

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Add core functionality for remote connection resolvers (#880)

* add remote connection resolver func

* adds resolver functionality for remote connection details, including health check to gql server

* add types, improve comment, move into remote connection resolver into distinct file

* remove unused imports

* add utility type. use utility objectValues type for type values from const

* fix type lookup in gql object

* add comments

* PR feedback: comment blocks

* additional remote connection node data resolver response data

* create typeDefs file for remote connection data

* move typeDefs for aggregations

* add resolver map, rename remote connection resolver file

* cleanup imports and creation of schema, imorts from distinct resolver and typedef files

* rename and fix type on Remote Connection gql resp

* remove ping healthcheck for nodes on resolver

* pass correct object to get available types

* add temp gql server for network aggs

* add status check

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/agg resolver (#881)

* add remote connection resolver func

* adds resolver functionality for remote connection details, including health check to gql server

* add types, improve comment, move into remote connection resolver into distinct file

* remove unused imports

* add utility type. use utility objectValues type for type values from const

* fix type lookup in gql object

* add comments

* PR feedback: comment blocks

* additional remote connection node data resolver response data

* create typeDefs file for remote connection data

* move typeDefs for aggregations

* add resolver map, rename remote connection resolver file

* cleanup imports and creation of schema, imorts from distinct resolver and typedef files

* rename and fix type on Remote Connection gql resp

* remove ping healthcheck for nodes on resolver

* pass correct object to get available types

* add temp gql server for network aggs

* add status check

* fetchGQL helper func working with axios

* rework dynamic network search types based on remote connections types

* use __type query instead of full introspectionQuery

* remove old code

* add single typeDef export, responsible for merging remote connection and aggregation types

* correct structure for merging typedefs with other typedefs, merge on Query

* get all types from remote connections once

* agg resolver structuring

* add aggregations resolvers

* add agg resolver network calls

* rename and narrow NetworkType type across files

* add common file

* add network queries file

* rename var

* query remote connections

* move query

* add comments

* add supported and unsupported aggregations with test

* ts-jest disable diagnostics to run tests without all types passing

* rework connection status property to do ping healthcehck

* types cleanup across files

* improved typing for reducer in field types, covering supported and unsupported aggregates

* tighten and rename types

* type cleanup and comments

* add explicit name mapping

* rename type correctly

* Cleanup query lookup for aggregations for remote connections

* move TSdoc comment

* move types

* add util func to convert gql AST into string

* move aggregations resolver code into a module

* simplify query to be composable, better naming for agg query map

* aggregation logic placeholder

* cleanup

* rework logic to use single request and mapping fields, instead of slicing central query

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/resolve stats (#883)

* add resolve aggregation function with tests

* add basic aggregation TS types

* cleanup

* improve comments

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/resolver responses (#884)

* add documentName to config and use in remote connection gql queries

* move fulfilled promise type guard to util and make it a generic

* resolve aggregations from network query results'

* return resolved response object

* PR feedback

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fix network agg resolution (#885)

* add Bucket gql object type

* text change, move func outside of nested func

* fix resolved aggregations overriding

* Update modules/server/src/network/aggregations/index.ts

Co-authored-by: Anders Richardsson <[email protected]>

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Response data & batching requests (#887)

* move response object creation into module

* change type structure for gql server

* change resolver code based on new gql type

* adjust agg type gql resolved response

* add http resp helpers

* change resolveAggregation code to batch processing

* add nodeInfo to resolver

* change resolver method to reducer with mutations

* add type

* fix typedef

* format response object

* comments and cleanup

* remove log

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fed Search request timeouts (#889)

* move response object creation into module

* change type structure for gql server

* change resolver code based on new gql type

* adjust agg type gql resolved response

* add http resp helpers

* change resolveAggregation code to batch processing

* add nodeInfo to resolver

* change resolver method to reducer with mutations

* add type

* fix typedef

* format response object

* comments and cleanup

* add GQL request timeout with config option and default value

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Reorg connection (#891)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Accumulator pipeline (#893)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed total hits (#895)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

* add total hits to query string

* cleanup field, param order

* add test

* rename RemoteAggregations type

* add hits TS type

* fix env toggle for network fed search

* renaming, update types

* documentName and documentType are the same for functionality

* return unique fields

* fix accumulator resolve for new input shape

* tighten types. add Hits resolution

* rename count to hits

* fix logging

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* feat/agg acc tests (#897)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

* add total hits to query string

* cleanup field, param order

* add test

* rename RemoteAggregations type

* add hits TS type

* fix env toggle for network fed search

* renaming, update types

* documentName and documentType are the same for functionality

* return unique fields

* fix accumulator resolve for new input shape

* tighten types. add Hits resolution

* rename count to hits

* add first test for requested fields

* tighten types and error checking

* fix up types and comments, resolvers code fix

* update aggregation test

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed numeric agg resolution (#898)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

* add total hits to query string

* cleanup field, param order

* add test

* rename RemoteAggregations type

* add hits TS type

* fix env toggle for network fed search

* renaming, update types

* documentName and documentType are the same for functionality

* return unique fields

* fix accumulator resolve for new input shape

* tighten types. add Hits resolution

* rename count to hits

* add first test for requested fields

* tighten types and error checking

* fix up types and comments, resolvers code fix

* update aggregation test

* add test + expand types

* rework resolution iteration and add numericAggregations resolver

* add typename to types

* cleanup

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* use Math methods (#900)

Co-authored-by: Ciaran Schutte <[email protected]>

* Individual node config query fields (#899)

* use more detailed NodeConfig object to store available agg types

* filter requested aggs with available aggs

* add available aggs to node connection resolver type defs

* cleanup

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/aggregate not available (#903)

* add TSdoc example

* add TSdoc comment

* pass more data into Aggs accumulator

* add schema map for query time

* rework agg accumulator to add empty agg to bucket if aggregation not available for node

* fix test

* clean up logic to account for only Aggregations type

* account for empty aggs field in node query

* cleanup

* disabled network Agg tests

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/filters (#904)

* add TSdoc example

* add TSdoc comment

* pass more data into Aggs accumulator

* add schema map for query time

* rework agg accumulator to add empty agg to bucket if aggregation not available for node

* fix test

* clean up logic to account for only Aggregations type

* account for empty aggs field in node query

* cleanup

* disabled network Agg tests

* add args to Aggregations gql typedef

* add args to network field

* pass gql resolvers args to query creation

* remove console log

* add include missing:

* add overture sqon builder dep

* move up type

* add SQON check and args type

* rename file

* check for SQON correctness but pass ordinary object through on success

* add comment

* Update modules/server/src/network/utils/sqon.ts

Co-authored-by: Anders Richardsson <[email protected]>

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Minor comment updates (#905)

* improve comment

* improve comment

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fed search cleanup typings (#906)

* remove unsued type defs

* add type watch script

* fix import

* fix import

* type an ANY type

* fix agg resolver typing

* ts no-check for test file

* fix Agg Accumulator types

* type response formatter

* fix test file type

* address feedback

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fix broken build (#907)

* remove unsued type defs

* add type watch script

* fix import

* fix import

* type an ANY type

* fix agg resolver typing

* ts no-check for test file

* fix Agg Accumulator types

* type response formatter

* fix test file type

* fix gql query creation test

* query back to normal test

* disables tests, adds comment

* fix merged file

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* add agg mode env config (#908)

* add agg mode env config

* env config to strip out section of gql schema, typedef + resolver

* prettier

* prettier

* Update modules/server/.env.schema

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/createConnectionResolvers.ts

Co-authored-by: Anders Richardsson <[email protected]>

* alphabetize

* move up import

* change AGG_ONLY_MODE to enabled DOCUMENT_HITS

* add empty line at EOF

* add comment

* fix typo

* fix inverted logic

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Feature/threshold calc (#909)

* add conditional prop to schema

* conditional schema field

* conditional add dataMasking schema fields

* add data masking logic

* add agg only conditional to resolvers

* clarify todo

* get optional aggregations field and values for hits resolution

* add mask func stub

* fix missing prop from spread

* masking logic

* conditional resolve hits

* move thresholding higher on resolver chain

* add falsey belowThreshold value instead of null

* data mask threshold env var

* update wrong process.env path

* seperate resolver creation from routing

* check for undefiend value in lookup

* clarify comment

* add threshold value explanation

* move types to common file

* typing

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* multiple node with data thresholding (#912)

* fix build error

* single endpoint

* fix GQL type name collisions

* move data mask threshold env. ensure data masking applies even if hits.total is not queried

* change belowThreshold to relation object

* surface api errors to axios responses

* add relation logic and agg merging for network search

* read data mask threshold min from config

* check for valid env val

* remove unused func

* fix TS build errors

* fix schema merge error

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed search cleanup (#920)

* add placeholder

* Adjust config template example and file

* remove unused arranger constructor param

* reorder constants file, parse env var

* update arranger network config type

* type, add gql server types

* graphqlRoutes, remove unused import, export func

* resolver creation, improve TS usage, split functionality into distinct files, improve usage'

* add surrounding brackets for func

* cleanup schema resolvers

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed search networked cleanup (#921)

* cleanup utils

* reorg types, decompose file, improve comments

* clean up types

* cleanup types, redundant code and fix a couple of bad merge files

* fix enum

* cleanup test types

* type fix

* explicit type import, use path alias

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Update modules/server/src/app.js

Co-authored-by: Anders Richardsson <[email protected]>

* make horrible ajax.ts console logging somewhat useful

* improve release process

* fix paths and imports for new build config

* update tests to todos

* fix some imports

* fix build issues with lodash and missing config flag

* fix paths

* comment out test files, broken with tsx vs jest

* fix path

* fix path

* improve release process

* allow toolbar options customization

* fix setsIndex conflict resolutioon misses

* default for ENABLE_DOCUMENT_HITS env flag is true

* default enable document hits to true

* add DATA_MASK_MIN_THRESHOLD env var as pass through var

* debug

* debug

* debug

* add default param value

* debug

* string util important fix

* fix for tests

* Add comment re data masking

* remove debug log

* Update modules/server/src/mapping/createConnectionTypeDefs.js

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/createConnectionTypeDefs.js

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/masking.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/resolveHitsFromAggs.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/resolvers.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/resolvers.ts

Co-authored-by: Anders Richardsson <[email protected]>

* remove specific NetworkAggError class

* fix error response type narrow

* missing param

* add explanation comment for aggs resolver, gql info lookup

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>
ciaranschutte added a commit that referenced this pull request May 7, 2025
* Feature/federated (#926)

* Network aggregation search - Initial Config (#876)

* initial config

* add network to config template. add default values to base.json

* add schema merge placeholders

* add conditional to graphql route gen for env flag

* throw error on invalid json parse eg. empty file

* rename type

* add vscode debug file to gitignore

* Update modules/server/README.md

Co-authored-by: Anders Richardsson <[email protected]>

* rename field to documentType

* Update modules/server/configTemplates/base.json

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/config/utils/getConfigFromFiles.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update types.ts

* Update README.md

* rename fields, add nesting

* remove env config to enable federated search

* remove env config to enable federated search

* remove flag from env schema

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Feat/#873 retrieve remote schemas (#877)

* initial config

* add network to config template. add default values to base.json

* add schema merge placeholders

* add conditional to graphql route gen for env flag

* throw error on invalid json parse eg. empty file

* rename type

* add vscode debug file to gitignore

* add gql client lib

* change gql lib

* Update modules/server/README.md

Co-authored-by: Anders Richardsson <[email protected]>

* add fetch field

* rename field to documentType

* Update modules/server/configTemplates/base.json

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/config/utils/getConfigFromFiles.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update types.ts

* Update README.md

* add ignore to gitignore

* rename fields, add nesting

* remove env config to enable federated search

* remove env config to enable federated search

* remove flag from env schema

* correct schema retrieval

* fix GQL obj wrapping on schema construction

* example

* cleanup

* cleanup gql file

* cleanup packages

* fix import

* add introspection query type predicate

* add tsdoc

* remove local folder ignore from project.gitignore

* remove unused util

* renaming

* break up query promise and results

* Update .gitignore

Co-authored-by: Anders Richardsson <[email protected]>

* use axios

* axios network check, doesn't throw like fetch

* change fetch config to axios config

* Update modules/server/src/network/index.ts

Co-authored-by: Anders Richardsson <[email protected]>

* add logs to thrown errors

* small feedback fixes

* add typings + additional info  in comments

* add better errors

* remove unused func. renaming

* add return type

* remove assertion. add TS filter

* change nulls to undefined

* change explicit undefined to optional ts

* add tsdoc comment

* change var type to loose record instead of uknown

* if else TS

* if else TS

* change to nested promise return

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Pr feedback - doc updates (#879)

* update tsdoc

* update tsdoc

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/#874 merge schemas (#878)

* initial config

* add network to config template. add default values to base.json

* add schema merge placeholders

* add conditional to graphql route gen for env flag

* throw error on invalid json parse eg. empty file

* rename type

* add vscode debug file to gitignore

* add gql client lib

* change gql lib

* Update modules/server/README.md

Co-authored-by: Anders Richardsson <[email protected]>

* add fetch field

* rename field to documentType

* Update modules/server/configTemplates/base.json

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/config/utils/getConfigFromFiles.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update types.ts

* Update README.md

* add ignore to gitignore

* rename fields, add nesting

* remove env config to enable federated search

* remove env config to enable federated search

* remove flag from env schema

* correct schema retrieval

* fix GQL obj wrapping on schema construction

* example

* cleanup

* cleanup gql file

* cleanup packages

* fix import

* basic merge schema

* cleanup comment

* exclusuions

* add introspection query type predicate

* add tsdoc

* remove local folder ignore from project.gitignore

* remove unused util

* renaming

* break up query promise and results

* Update .gitignore

Co-authored-by: Anders Richardsson <[email protected]>

* use axios

* axios network check, doesn't throw like fetch

* update packages for typedefs

* change fetch config to axios config

* Update modules/server/src/network/index.ts

Co-authored-by: Anders Richardsson <[email protected]>

* add logs to thrown errors

* fix merge conflict

* add test watch command with basic test

* tests

* Fix type. narrow scope. hoist filtering

* remove debug code. add comment

* update comments + add type assertion

* add comments

* remove redundant options arg for schema merginging

* seperate test utils and fixture from main file

* seperate test utils and fixture from main file

* tests, cleanup, extra tests, split responsibility

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Add core functionality for remote connection resolvers (#880)

* add remote connection resolver func

* adds resolver functionality for remote connection details, including health check to gql server

* add types, improve comment, move into remote connection resolver into distinct file

* remove unused imports

* add utility type. use utility objectValues type for type values from const

* fix type lookup in gql object

* add comments

* PR feedback: comment blocks

* additional remote connection node data resolver response data

* create typeDefs file for remote connection data

* move typeDefs for aggregations

* add resolver map, rename remote connection resolver file

* cleanup imports and creation of schema, imorts from distinct resolver and typedef files

* rename and fix type on Remote Connection gql resp

* remove ping healthcheck for nodes on resolver

* pass correct object to get available types

* add temp gql server for network aggs

* add status check

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/agg resolver (#881)

* add remote connection resolver func

* adds resolver functionality for remote connection details, including health check to gql server

* add types, improve comment, move into remote connection resolver into distinct file

* remove unused imports

* add utility type. use utility objectValues type for type values from const

* fix type lookup in gql object

* add comments

* PR feedback: comment blocks

* additional remote connection node data resolver response data

* create typeDefs file for remote connection data

* move typeDefs for aggregations

* add resolver map, rename remote connection resolver file

* cleanup imports and creation of schema, imorts from distinct resolver and typedef files

* rename and fix type on Remote Connection gql resp

* remove ping healthcheck for nodes on resolver

* pass correct object to get available types

* add temp gql server for network aggs

* add status check

* fetchGQL helper func working with axios

* rework dynamic network search types based on remote connections types

* use __type query instead of full introspectionQuery

* remove old code

* add single typeDef export, responsible for merging remote connection and aggregation types

* correct structure for merging typedefs with other typedefs, merge on Query

* get all types from remote connections once

* agg resolver structuring

* add aggregations resolvers

* add agg resolver network calls

* rename and narrow NetworkType type across files

* add common file

* add network queries file

* rename var

* query remote connections

* move query

* add comments

* add supported and unsupported aggregations with test

* ts-jest disable diagnostics to run tests without all types passing

* rework connection status property to do ping healthcehck

* types cleanup across files

* improved typing for reducer in field types, covering supported and unsupported aggregates

* tighten and rename types

* type cleanup and comments

* add explicit name mapping

* rename type correctly

* Cleanup query lookup for aggregations for remote connections

* move TSdoc comment

* move types

* add util func to convert gql AST into string

* move aggregations resolver code into a module

* simplify query to be composable, better naming for agg query map

* aggregation logic placeholder

* cleanup

* rework logic to use single request and mapping fields, instead of slicing central query

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/resolve stats (#883)

* add resolve aggregation function with tests

* add basic aggregation TS types

* cleanup

* improve comments

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/resolver responses (#884)

* add documentName to config and use in remote connection gql queries

* move fulfilled promise type guard to util and make it a generic

* resolve aggregations from network query results'

* return resolved response object

* PR feedback

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fix network agg resolution (#885)

* add Bucket gql object type

* text change, move func outside of nested func

* fix resolved aggregations overriding

* Update modules/server/src/network/aggregations/index.ts

Co-authored-by: Anders Richardsson <[email protected]>

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Response data & batching requests (#887)

* move response object creation into module

* change type structure for gql server

* change resolver code based on new gql type

* adjust agg type gql resolved response

* add http resp helpers

* change resolveAggregation code to batch processing

* add nodeInfo to resolver

* change resolver method to reducer with mutations

* add type

* fix typedef

* format response object

* comments and cleanup

* remove log

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fed Search request timeouts (#889)

* move response object creation into module

* change type structure for gql server

* change resolver code based on new gql type

* adjust agg type gql resolved response

* add http resp helpers

* change resolveAggregation code to batch processing

* add nodeInfo to resolver

* change resolver method to reducer with mutations

* add type

* fix typedef

* format response object

* comments and cleanup

* add GQL request timeout with config option and default value

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Reorg connection (#891)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Accumulator pipeline (#893)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed total hits (#895)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

* add total hits to query string

* cleanup field, param order

* add test

* rename RemoteAggregations type

* add hits TS type

* fix env toggle for network fed search

* renaming, update types

* documentName and documentType are the same for functionality

* return unique fields

* fix accumulator resolve for new input shape

* tighten types. add Hits resolution

* rename count to hits

* fix logging

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* feat/agg acc tests (#897)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

* add total hits to query string

* cleanup field, param order

* add test

* rename RemoteAggregations type

* add hits TS type

* fix env toggle for network fed search

* renaming, update types

* documentName and documentType are the same for functionality

* return unique fields

* fix accumulator resolve for new input shape

* tighten types. add Hits resolution

* rename count to hits

* add first test for requested fields

* tighten types and error checking

* fix up types and comments, resolvers code fix

* update aggregation test

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed numeric agg resolution (#898)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

* add total hits to query string

* cleanup field, param order

* add test

* rename RemoteAggregations type

* add hits TS type

* fix env toggle for network fed search

* renaming, update types

* documentName and documentType are the same for functionality

* return unique fields

* fix accumulator resolve for new input shape

* tighten types. add Hits resolution

* rename count to hits

* add first test for requested fields

* tighten types and error checking

* fix up types and comments, resolvers code fix

* update aggregation test

* add test + expand types

* rework resolution iteration and add numericAggregations resolver

* add typename to types

* cleanup

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* use Math methods (#900)

Co-authored-by: Ciaran Schutte <[email protected]>

* Individual node config query fields (#899)

* use more detailed NodeConfig object to store available agg types

* filter requested aggs with available aggs

* add available aggs to node connection resolver type defs

* cleanup

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/aggregate not available (#903)

* add TSdoc example

* add TSdoc comment

* pass more data into Aggs accumulator

* add schema map for query time

* rework agg accumulator to add empty agg to bucket if aggregation not available for node

* fix test

* clean up logic to account for only Aggregations type

* account for empty aggs field in node query

* cleanup

* disabled network Agg tests

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/filters (#904)

* add TSdoc example

* add TSdoc comment

* pass more data into Aggs accumulator

* add schema map for query time

* rework agg accumulator to add empty agg to bucket if aggregation not available for node

* fix test

* clean up logic to account for only Aggregations type

* account for empty aggs field in node query

* cleanup

* disabled network Agg tests

* add args to Aggregations gql typedef

* add args to network field

* pass gql resolvers args to query creation

* remove console log

* add include missing:

* add overture sqon builder dep

* move up type

* add SQON check and args type

* rename file

* check for SQON correctness but pass ordinary object through on success

* add comment

* Update modules/server/src/network/utils/sqon.ts

Co-authored-by: Anders Richardsson <[email protected]>

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Minor comment updates (#905)

* improve comment

* improve comment

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fed search cleanup typings (#906)

* remove unsued type defs

* add type watch script

* fix import

* fix import

* type an ANY type

* fix agg resolver typing

* ts no-check for test file

* fix Agg Accumulator types

* type response formatter

* fix test file type

* address feedback

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fix broken build (#907)

* remove unsued type defs

* add type watch script

* fix import

* fix import

* type an ANY type

* fix agg resolver typing

* ts no-check for test file

* fix Agg Accumulator types

* type response formatter

* fix test file type

* fix gql query creation test

* query back to normal test

* disables tests, adds comment

* fix merged file

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* add agg mode env config (#908)

* add agg mode env config

* env config to strip out section of gql schema, typedef + resolver

* prettier

* prettier

* Update modules/server/.env.schema

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/createConnectionResolvers.ts

Co-authored-by: Anders Richardsson <[email protected]>

* alphabetize

* move up import

* change AGG_ONLY_MODE to enabled DOCUMENT_HITS

* add empty line at EOF

* add comment

* fix typo

* fix inverted logic

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Feature/threshold calc (#909)

* add conditional prop to schema

* conditional schema field

* conditional add dataMasking schema fields

* add data masking logic

* add agg only conditional to resolvers

* clarify todo

* get optional aggregations field and values for hits resolution

* add mask func stub

* fix missing prop from spread

* masking logic

* conditional resolve hits

* move thresholding higher on resolver chain

* add falsey belowThreshold value instead of null

* data mask threshold env var

* update wrong process.env path

* seperate resolver creation from routing

* check for undefiend value in lookup

* clarify comment

* add threshold value explanation

* move types to common file

* typing

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* multiple node with data thresholding (#912)

* fix build error

* single endpoint

* fix GQL type name collisions

* move data mask threshold env. ensure data masking applies even if hits.total is not queried

* change belowThreshold to relation object

* surface api errors to axios responses

* add relation logic and agg merging for network search

* read data mask threshold min from config

* check for valid env val

* remove unused func

* fix TS build errors

* fix schema merge error

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed search cleanup (#920)

* add placeholder

* Adjust config template example and file

* remove unused arranger constructor param

* reorder constants file, parse env var

* update arranger network config type

* type, add gql server types

* graphqlRoutes, remove unused import, export func

* resolver creation, improve TS usage, split functionality into distinct files, improve usage'

* add surrounding brackets for func

* cleanup schema resolvers

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed search networked cleanup (#921)

* cleanup utils

* reorg types, decompose file, improve comments

* clean up types

* cleanup types, redundant code and fix a couple of bad merge files

* fix enum

* cleanup test types

* type fix

* explicit type import, use path alias

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Update modules/server/src/app.js

Co-authored-by: Anders Richardsson <[email protected]>

* make horrible ajax.ts console logging somewhat useful

* improve release process

* fix paths and imports for new build config

* update tests to todos

* fix some imports

* fix build issues with lodash and missing config flag

* fix paths

* comment out test files, broken with tsx vs jest

* fix path

* fix path

* improve release process

* allow toolbar options customization

* fix setsIndex conflict resolutioon misses

* default for ENABLE_DOCUMENT_HITS env flag is true

* default enable document hits to true

* add DATA_MASK_MIN_THRESHOLD env var as pass through var

* debug

* debug

* debug

* add default param value

* debug

* string util important fix

* fix for tests

* Add comment re data masking

* remove debug log

* Update modules/server/src/mapping/createConnectionTypeDefs.js

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/createConnectionTypeDefs.js

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/masking.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/resolveHitsFromAggs.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/resolvers.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/resolvers.ts

Co-authored-by: Anders Richardsson <[email protected]>

* remove specific NetworkAggError class

* fix error response type narrow

* missing param

* add explanation comment for aggs resolver, gql info lookup

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* initial cleanup

* remove remaining data masking config + calculations

* redo remove data masking relation ref

* remove leftover env vars

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>
ciaranschutte added a commit that referenced this pull request May 7, 2025
* Feature/federated (#926)

* Network aggregation search - Initial Config (#876)

* initial config

* add network to config template. add default values to base.json

* add schema merge placeholders

* add conditional to graphql route gen for env flag

* throw error on invalid json parse eg. empty file

* rename type

* add vscode debug file to gitignore

* Update modules/server/README.md

Co-authored-by: Anders Richardsson <[email protected]>

* rename field to documentType

* Update modules/server/configTemplates/base.json

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/config/utils/getConfigFromFiles.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update types.ts

* Update README.md

* rename fields, add nesting

* remove env config to enable federated search

* remove env config to enable federated search

* remove flag from env schema

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Feat/#873 retrieve remote schemas (#877)

* initial config

* add network to config template. add default values to base.json

* add schema merge placeholders

* add conditional to graphql route gen for env flag

* throw error on invalid json parse eg. empty file

* rename type

* add vscode debug file to gitignore

* add gql client lib

* change gql lib

* Update modules/server/README.md

Co-authored-by: Anders Richardsson <[email protected]>

* add fetch field

* rename field to documentType

* Update modules/server/configTemplates/base.json

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/config/utils/getConfigFromFiles.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update types.ts

* Update README.md

* add ignore to gitignore

* rename fields, add nesting

* remove env config to enable federated search

* remove env config to enable federated search

* remove flag from env schema

* correct schema retrieval

* fix GQL obj wrapping on schema construction

* example

* cleanup

* cleanup gql file

* cleanup packages

* fix import

* add introspection query type predicate

* add tsdoc

* remove local folder ignore from project.gitignore

* remove unused util

* renaming

* break up query promise and results

* Update .gitignore

Co-authored-by: Anders Richardsson <[email protected]>

* use axios

* axios network check, doesn't throw like fetch

* change fetch config to axios config

* Update modules/server/src/network/index.ts

Co-authored-by: Anders Richardsson <[email protected]>

* add logs to thrown errors

* small feedback fixes

* add typings + additional info  in comments

* add better errors

* remove unused func. renaming

* add return type

* remove assertion. add TS filter

* change nulls to undefined

* change explicit undefined to optional ts

* add tsdoc comment

* change var type to loose record instead of uknown

* if else TS

* if else TS

* change to nested promise return

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Pr feedback - doc updates (#879)

* update tsdoc

* update tsdoc

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/#874 merge schemas (#878)

* initial config

* add network to config template. add default values to base.json

* add schema merge placeholders

* add conditional to graphql route gen for env flag

* throw error on invalid json parse eg. empty file

* rename type

* add vscode debug file to gitignore

* add gql client lib

* change gql lib

* Update modules/server/README.md

Co-authored-by: Anders Richardsson <[email protected]>

* add fetch field

* rename field to documentType

* Update modules/server/configTemplates/base.json

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/config/utils/getConfigFromFiles.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update types.ts

* Update README.md

* add ignore to gitignore

* rename fields, add nesting

* remove env config to enable federated search

* remove env config to enable federated search

* remove flag from env schema

* correct schema retrieval

* fix GQL obj wrapping on schema construction

* example

* cleanup

* cleanup gql file

* cleanup packages

* fix import

* basic merge schema

* cleanup comment

* exclusuions

* add introspection query type predicate

* add tsdoc

* remove local folder ignore from project.gitignore

* remove unused util

* renaming

* break up query promise and results

* Update .gitignore

Co-authored-by: Anders Richardsson <[email protected]>

* use axios

* axios network check, doesn't throw like fetch

* update packages for typedefs

* change fetch config to axios config

* Update modules/server/src/network/index.ts

Co-authored-by: Anders Richardsson <[email protected]>

* add logs to thrown errors

* fix merge conflict

* add test watch command with basic test

* tests

* Fix type. narrow scope. hoist filtering

* remove debug code. add comment

* update comments + add type assertion

* add comments

* remove redundant options arg for schema merginging

* seperate test utils and fixture from main file

* seperate test utils and fixture from main file

* tests, cleanup, extra tests, split responsibility

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Add core functionality for remote connection resolvers (#880)

* add remote connection resolver func

* adds resolver functionality for remote connection details, including health check to gql server

* add types, improve comment, move into remote connection resolver into distinct file

* remove unused imports

* add utility type. use utility objectValues type for type values from const

* fix type lookup in gql object

* add comments

* PR feedback: comment blocks

* additional remote connection node data resolver response data

* create typeDefs file for remote connection data

* move typeDefs for aggregations

* add resolver map, rename remote connection resolver file

* cleanup imports and creation of schema, imorts from distinct resolver and typedef files

* rename and fix type on Remote Connection gql resp

* remove ping healthcheck for nodes on resolver

* pass correct object to get available types

* add temp gql server for network aggs

* add status check

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/agg resolver (#881)

* add remote connection resolver func

* adds resolver functionality for remote connection details, including health check to gql server

* add types, improve comment, move into remote connection resolver into distinct file

* remove unused imports

* add utility type. use utility objectValues type for type values from const

* fix type lookup in gql object

* add comments

* PR feedback: comment blocks

* additional remote connection node data resolver response data

* create typeDefs file for remote connection data

* move typeDefs for aggregations

* add resolver map, rename remote connection resolver file

* cleanup imports and creation of schema, imorts from distinct resolver and typedef files

* rename and fix type on Remote Connection gql resp

* remove ping healthcheck for nodes on resolver

* pass correct object to get available types

* add temp gql server for network aggs

* add status check

* fetchGQL helper func working with axios

* rework dynamic network search types based on remote connections types

* use __type query instead of full introspectionQuery

* remove old code

* add single typeDef export, responsible for merging remote connection and aggregation types

* correct structure for merging typedefs with other typedefs, merge on Query

* get all types from remote connections once

* agg resolver structuring

* add aggregations resolvers

* add agg resolver network calls

* rename and narrow NetworkType type across files

* add common file

* add network queries file

* rename var

* query remote connections

* move query

* add comments

* add supported and unsupported aggregations with test

* ts-jest disable diagnostics to run tests without all types passing

* rework connection status property to do ping healthcehck

* types cleanup across files

* improved typing for reducer in field types, covering supported and unsupported aggregates

* tighten and rename types

* type cleanup and comments

* add explicit name mapping

* rename type correctly

* Cleanup query lookup for aggregations for remote connections

* move TSdoc comment

* move types

* add util func to convert gql AST into string

* move aggregations resolver code into a module

* simplify query to be composable, better naming for agg query map

* aggregation logic placeholder

* cleanup

* rework logic to use single request and mapping fields, instead of slicing central query

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/resolve stats (#883)

* add resolve aggregation function with tests

* add basic aggregation TS types

* cleanup

* improve comments

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/resolver responses (#884)

* add documentName to config and use in remote connection gql queries

* move fulfilled promise type guard to util and make it a generic

* resolve aggregations from network query results'

* return resolved response object

* PR feedback

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fix network agg resolution (#885)

* add Bucket gql object type

* text change, move func outside of nested func

* fix resolved aggregations overriding

* Update modules/server/src/network/aggregations/index.ts

Co-authored-by: Anders Richardsson <[email protected]>

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Response data & batching requests (#887)

* move response object creation into module

* change type structure for gql server

* change resolver code based on new gql type

* adjust agg type gql resolved response

* add http resp helpers

* change resolveAggregation code to batch processing

* add nodeInfo to resolver

* change resolver method to reducer with mutations

* add type

* fix typedef

* format response object

* comments and cleanup

* remove log

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fed Search request timeouts (#889)

* move response object creation into module

* change type structure for gql server

* change resolver code based on new gql type

* adjust agg type gql resolved response

* add http resp helpers

* change resolveAggregation code to batch processing

* add nodeInfo to resolver

* change resolver method to reducer with mutations

* add type

* fix typedef

* format response object

* comments and cleanup

* add GQL request timeout with config option and default value

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Reorg connection (#891)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Accumulator pipeline (#893)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed total hits (#895)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

* add total hits to query string

* cleanup field, param order

* add test

* rename RemoteAggregations type

* add hits TS type

* fix env toggle for network fed search

* renaming, update types

* documentName and documentType are the same for functionality

* return unique fields

* fix accumulator resolve for new input shape

* tighten types. add Hits resolution

* rename count to hits

* fix logging

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* feat/agg acc tests (#897)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

* add total hits to query string

* cleanup field, param order

* add test

* rename RemoteAggregations type

* add hits TS type

* fix env toggle for network fed search

* renaming, update types

* documentName and documentType are the same for functionality

* return unique fields

* fix accumulator resolve for new input shape

* tighten types. add Hits resolution

* rename count to hits

* add first test for requested fields

* tighten types and error checking

* fix up types and comments, resolvers code fix

* update aggregation test

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed numeric agg resolution (#898)

* change resolver code based on new gql type

* comments and cleanup

* break up main entry point into two modules, one for querying and one for field processing

* rename network config type, remove supported aggs from type

* type cleanup

* only send network queries with original query fields

* move gql health check into module

* cleanup, renaming types, removing redundant code

* add type

* fix merged conflict

* adds full stop. fixes comment typo

* fix pipeline code, move accumulator into class

* handle unavailable node

* check undefined in Success

* add note

* rename poorly named field

* add comment:

* rename file

* http response success status as const

* renamed AggAccumulator file

* clean up types, change loose object to use record, fix any typings

* add total hits to query string

* cleanup field, param order

* add test

* rename RemoteAggregations type

* add hits TS type

* fix env toggle for network fed search

* renaming, update types

* documentName and documentType are the same for functionality

* return unique fields

* fix accumulator resolve for new input shape

* tighten types. add Hits resolution

* rename count to hits

* add first test for requested fields

* tighten types and error checking

* fix up types and comments, resolvers code fix

* update aggregation test

* add test + expand types

* rework resolution iteration and add numericAggregations resolver

* add typename to types

* cleanup

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* use Math methods (#900)

Co-authored-by: Ciaran Schutte <[email protected]>

* Individual node config query fields (#899)

* use more detailed NodeConfig object to store available agg types

* filter requested aggs with available aggs

* add available aggs to node connection resolver type defs

* cleanup

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/aggregate not available (#903)

* add TSdoc example

* add TSdoc comment

* pass more data into Aggs accumulator

* add schema map for query time

* rework agg accumulator to add empty agg to bucket if aggregation not available for node

* fix test

* clean up logic to account for only Aggregations type

* account for empty aggs field in node query

* cleanup

* disabled network Agg tests

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/filters (#904)

* add TSdoc example

* add TSdoc comment

* pass more data into Aggs accumulator

* add schema map for query time

* rework agg accumulator to add empty agg to bucket if aggregation not available for node

* fix test

* clean up logic to account for only Aggregations type

* account for empty aggs field in node query

* cleanup

* disabled network Agg tests

* add args to Aggregations gql typedef

* add args to network field

* pass gql resolvers args to query creation

* remove console log

* add include missing:

* add overture sqon builder dep

* move up type

* add SQON check and args type

* rename file

* check for SQON correctness but pass ordinary object through on success

* add comment

* Update modules/server/src/network/utils/sqon.ts

Co-authored-by: Anders Richardsson <[email protected]>

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Minor comment updates (#905)

* improve comment

* improve comment

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fed search cleanup typings (#906)

* remove unsued type defs

* add type watch script

* fix import

* fix import

* type an ANY type

* fix agg resolver typing

* ts no-check for test file

* fix Agg Accumulator types

* type response formatter

* fix test file type

* address feedback

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Fix broken build (#907)

* remove unsued type defs

* add type watch script

* fix import

* fix import

* type an ANY type

* fix agg resolver typing

* ts no-check for test file

* fix Agg Accumulator types

* type response formatter

* fix test file type

* fix gql query creation test

* query back to normal test

* disables tests, adds comment

* fix merged file

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* add agg mode env config (#908)

* add agg mode env config

* env config to strip out section of gql schema, typedef + resolver

* prettier

* prettier

* Update modules/server/.env.schema

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/createConnectionResolvers.ts

Co-authored-by: Anders Richardsson <[email protected]>

* alphabetize

* move up import

* change AGG_ONLY_MODE to enabled DOCUMENT_HITS

* add empty line at EOF

* add comment

* fix typo

* fix inverted logic

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* Feature/threshold calc (#909)

* add conditional prop to schema

* conditional schema field

* conditional add dataMasking schema fields

* add data masking logic

* add agg only conditional to resolvers

* clarify todo

* get optional aggregations field and values for hits resolution

* add mask func stub

* fix missing prop from spread

* masking logic

* conditional resolve hits

* move thresholding higher on resolver chain

* add falsey belowThreshold value instead of null

* data mask threshold env var

* update wrong process.env path

* seperate resolver creation from routing

* check for undefiend value in lookup

* clarify comment

* add threshold value explanation

* move types to common file

* typing

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* multiple node with data thresholding (#912)

* fix build error

* single endpoint

* fix GQL type name collisions

* move data mask threshold env. ensure data masking applies even if hits.total is not queried

* change belowThreshold to relation object

* surface api errors to axios responses

* add relation logic and agg merging for network search

* read data mask threshold min from config

* check for valid env val

* remove unused func

* fix TS build errors

* fix schema merge error

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed search cleanup (#920)

* add placeholder

* Adjust config template example and file

* remove unused arranger constructor param

* reorder constants file, parse env var

* update arranger network config type

* type, add gql server types

* graphqlRoutes, remove unused import, export func

* resolver creation, improve TS usage, split functionality into distinct files, improve usage'

* add surrounding brackets for func

* cleanup schema resolvers

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Feat/fed search networked cleanup (#921)

* cleanup utils

* reorg types, decompose file, improve comments

* clean up types

* cleanup types, redundant code and fix a couple of bad merge files

* fix enum

* cleanup test types

* type fix

* explicit type import, use path alias

---------

Co-authored-by: Ciaran Schutte <[email protected]>

* Update modules/server/src/app.js

Co-authored-by: Anders Richardsson <[email protected]>

* make horrible ajax.ts console logging somewhat useful

* improve release process

* fix paths and imports for new build config

* update tests to todos

* fix some imports

* fix build issues with lodash and missing config flag

* fix paths

* comment out test files, broken with tsx vs jest

* fix path

* fix path

* improve release process

* allow toolbar options customization

* fix setsIndex conflict resolutioon misses

* default for ENABLE_DOCUMENT_HITS env flag is true

* default enable document hits to true

* add DATA_MASK_MIN_THRESHOLD env var as pass through var

* debug

* debug

* debug

* add default param value

* debug

* string util important fix

* fix for tests

* Add comment re data masking

* remove debug log

* Update modules/server/src/mapping/createConnectionTypeDefs.js

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/createConnectionTypeDefs.js

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/masking.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/resolveHitsFromAggs.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/resolvers.ts

Co-authored-by: Anders Richardsson <[email protected]>

* Update modules/server/src/mapping/resolvers.ts

Co-authored-by: Anders Richardsson <[email protected]>

* remove specific NetworkAggError class

* fix error response type narrow

* missing param

* add explanation comment for aggs resolver, gql info lookup

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>

* initial cleanup

* remove remaining data masking config + calculations

* add network agg property from env

* main conflicts

* read directly from config

---------

Co-authored-by: Ciaran Schutte <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>
Co-authored-by: Anders Richardsson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants