Accumulator pipeline #893

ciaranschutte · 2024-09-24T16:19:05Z

cleans up aggregation pipeline code:

iterates over configs instead of requests, reducing # of iterations to # of configs
moves messy accumulator into a class
handles "Server not available, connection refused" state reporting node with correct error status

…for field processing

ciaranschutte · 2024-09-24T16:54:10Z

modules/server/src/network/resolvers/aggregations.ts

-			}
-		})
-		.filter(Boolean);
+	const totalAgg = new AggregationAccumulator(requestedAggregationFields);


uses a class to hold accumulator state

ciaranschutte · 2024-09-24T16:55:22Z

modules/server/src/network/resolvers/aggregations.ts

+		if (axios.isAxiosError(error)) {
+			if (error.code === 'ECONNREFUSED') {
+				console.error(`Network failure: ${url}`);
+				return failure(CONNECTION_STATUS.ERROR, `Network failure: ${url}`);
+			}
 		}


catches error is server is unavailable
nested for now - would like to check if there are other codes need to be covered

ciaranschutte · 2024-09-24T16:56:09Z

modules/server/src/network/aggregations/index.ts

+const emptyAggregation: NetworkAggregation = { bucket_count: 0, buckets: [] };
+
+export class AggregationAccumulator {
+	totalAgg: any;
+
+	constructor(requestedFields: any) {
+		/*
+		 * seed accumulator with the requested field keys
+		 * this will make it easier to add to using key lookup instead of Array.find
+		 */
+		this.totalAgg = Object.keys(requestedFields).reduce((accumulator: any, field: any) => {
+			return { ...accumulator, [field]: emptyAggregation };
+		}, {});
+	}
+
+	resolve(data, fields) {
+		resolveAggregations({
+			accumulator: this.totalAgg,
+			networkResult: data,
+			requestedAggregationFields: fields,
+		});
+	}
+
+	result() {
+		return this.totalAgg;
+	}
+}


isolates all the accumulator logic from this module cleanly into an exposed class

joneubank

Clean up the types before you commit to this approach.

joneubank · 2024-09-24T20:15:20Z

modules/server/src/network/aggregations/index.ts

+		resolveAggregations({
+			accumulator: this.totalAgg,
+			networkResult: data,
+			requestedAggregationFields: fields,
+		});


Why wrap another function here... could just move the contents of resolveAggregation to be defined inside this class...

Also, I can't approve a function without argument type declarations. Right now you could do .resolve(true, false) and TS will be happy with you.

resolveAggregation was created already and it works nice, if moved into this class it has to be tied to the class and not an isolated function because it references class properties.

tied to module scope already so I'm worried about visibility outside the class

joneubank · 2024-09-24T20:20:09Z

modules/server/src/network/httpResponses.ts

+	result?: Result<T, FailureStatus, FailureData>,
 ): result is Success<T> {
-	return result.status === 'SUCCESS';
+	return result !== undefined && result.status === 'SUCCESS';


We should probably have 'SUCCESS' as a constant somewhere.

After reviewing where this was called... We really shouldnt have the situation where we have Result<> | undefined, that goes against the entire purpose of returning that type. We probably should revert this change.

joneubank · 2024-09-24T20:23:41Z

modules/server/src/network/aggregations/index.ts

+
+const emptyAggregation: NetworkAggregation = { bucket_count: 0, buckets: [] };
+
+export class AggregationAccumulator {


May want to rename this file after this class. AggregationAccumulator.ts

ciaranschutte · 2024-09-27T01:24:33Z

@joneubank fix types for AggregationAcc

demariadaniel · 2024-09-27T20:34:29Z

modules/server/src/network/aggregations/AggregationAccumulator.ts

+		 * seed accumulator with the requested field keys
+		 * this will make it easier to add to using key lookup instead of Array.find
+		 */
+		this.totalAgg = requestedFields.reduce<RemoteAggregation>((accumulator: any, field: any) => {


Wouldn't field: string work here?

Added a generic, just forgot to remove the inline any.
The generic types these fields correctly once any is removed.

demariadaniel

One question about use of 'any' but I see other PR feedback's been updated. LGTM 🚀

* change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings --------- Co-authored-by: Ciaran Schutte <[email protected]>

* Network aggregation search - Initial Config (#876) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * Update modules/server/README.md Co-authored-by: Anders Richardsson <[email protected]> * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <[email protected]> * Update types.ts * Update README.md * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Feat/#873 retrieve remote schemas (#877) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <[email protected]> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <[email protected]> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <[email protected]> * use axios * axios network check, doesn't throw like fetch * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <[email protected]> * add logs to thrown errors * small feedback fixes * add typings + additional info in comments * add better errors * remove unused func. renaming * add return type * remove assertion. add TS filter * change nulls to undefined * change explicit undefined to optional ts * add tsdoc comment * change var type to loose record instead of uknown * if else TS * if else TS * change to nested promise return --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Pr feedback - doc updates (#879) * update tsdoc * update tsdoc --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/#874 merge schemas (#878) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <[email protected]> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <[email protected]> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * basic merge schema * cleanup comment * exclusuions * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <[email protected]> * use axios * axios network check, doesn't throw like fetch * update packages for typedefs * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <[email protected]> * add logs to thrown errors * fix merge conflict * add test watch command with basic test * tests * Fix type. narrow scope. hoist filtering * remove debug code. add comment * update comments + add type assertion * add comments * remove redundant options arg for schema merginging * seperate test utils and fixture from main file * seperate test utils and fixture from main file * tests, cleanup, extra tests, split responsibility --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Add core functionality for remote connection resolvers (#880) * add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/agg resolver (#881) * add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check * fetchGQL helper func working with axios * rework dynamic network search types based on remote connections types * use __type query instead of full introspectionQuery * remove old code * add single typeDef export, responsible for merging remote connection and aggregation types * correct structure for merging typedefs with other typedefs, merge on Query * get all types from remote connections once * agg resolver structuring * add aggregations resolvers * add agg resolver network calls * rename and narrow NetworkType type across files * add common file * add network queries file * rename var * query remote connections * move query * add comments * add supported and unsupported aggregations with test * ts-jest disable diagnostics to run tests without all types passing * rework connection status property to do ping healthcehck * types cleanup across files * improved typing for reducer in field types, covering supported and unsupported aggregates * tighten and rename types * type cleanup and comments * add explicit name mapping * rename type correctly * Cleanup query lookup for aggregations for remote connections * move TSdoc comment * move types * add util func to convert gql AST into string * move aggregations resolver code into a module * simplify query to be composable, better naming for agg query map * aggregation logic placeholder * cleanup * rework logic to use single request and mapping fields, instead of slicing central query --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/resolve stats (#883) * add resolve aggregation function with tests * add basic aggregation TS types * cleanup * improve comments --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/resolver responses (#884) * add documentName to config and use in remote connection gql queries * move fulfilled promise type guard to util and make it a generic * resolve aggregations from network query results' * return resolved response object * PR feedback --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fix network agg resolution (#885) * add Bucket gql object type * text change, move func outside of nested func * fix resolved aggregations overriding * Update modules/server/src/network/aggregations/index.ts Co-authored-by: Anders Richardsson <[email protected]> --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Response data & batching requests (#887) * move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * remove log --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fed Search request timeouts (#889) * move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * add GQL request timeout with config option and default value --------- Co-authored-by: Ciaran Schutte <[email protected]> * Reorg connection (#891) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo --------- Co-authored-by: Ciaran Schutte <[email protected]> * Accumulator pipeline (#893) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed total hits (#895) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * fix logging --------- Co-authored-by: Ciaran Schutte <[email protected]> * feat/agg acc tests (#897) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed numeric agg resolution (#898) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test * add test + expand types * rework resolution iteration and add numericAggregations resolver * add typename to types * cleanup --------- Co-authored-by: Ciaran Schutte <[email protected]> * use Math methods (#900) Co-authored-by: Ciaran Schutte <[email protected]> * Individual node config query fields (#899) * use more detailed NodeConfig object to store available agg types * filter requested aggs with available aggs * add available aggs to node connection resolver type defs * cleanup --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/aggregate not available (#903) * add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/filters (#904) * add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests * add args to Aggregations gql typedef * add args to network field * pass gql resolvers args to query creation * remove console log * add include missing: * add overture sqon builder dep * move up type * add SQON check and args type * rename file * check for SQON correctness but pass ordinary object through on success * add comment * Update modules/server/src/network/utils/sqon.ts Co-authored-by: Anders Richardsson <[email protected]> --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Minor comment updates (#905) * improve comment * improve comment --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fed search cleanup typings (#906) * remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * address feedback --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fix broken build (#907) * remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * fix gql query creation test * query back to normal test * disables tests, adds comment * fix merged file --------- Co-authored-by: Ciaran Schutte <[email protected]> * add agg mode env config (#908) * add agg mode env config * env config to strip out section of gql schema, typedef + resolver * prettier * prettier * Update modules/server/.env.schema Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/createConnectionResolvers.ts Co-authored-by: Anders Richardsson <[email protected]> * alphabetize * move up import * change AGG_ONLY_MODE to enabled DOCUMENT_HITS * add empty line at EOF * add comment * fix typo * fix inverted logic --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Feature/threshold calc (#909) * add conditional prop to schema * conditional schema field * conditional add dataMasking schema fields * add data masking logic * add agg only conditional to resolvers * clarify todo * get optional aggregations field and values for hits resolution * add mask func stub * fix missing prop from spread * masking logic * conditional resolve hits * move thresholding higher on resolver chain * add falsey belowThreshold value instead of null * data mask threshold env var * update wrong process.env path * seperate resolver creation from routing * check for undefiend value in lookup * clarify comment * add threshold value explanation * move types to common file * typing --------- Co-authored-by: Ciaran Schutte <[email protected]> * multiple node with data thresholding (#912) * fix build error * single endpoint * fix GQL type name collisions * move data mask threshold env. ensure data masking applies even if hits.total is not queried * change belowThreshold to relation object * surface api errors to axios responses * add relation logic and agg merging for network search * read data mask threshold min from config * check for valid env val * remove unused func * fix TS build errors * fix schema merge error --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed search cleanup (#920) * add placeholder * Adjust config template example and file * remove unused arranger constructor param * reorder constants file, parse env var * update arranger network config type * type, add gql server types * graphqlRoutes, remove unused import, export func * resolver creation, improve TS usage, split functionality into distinct files, improve usage' * add surrounding brackets for func * cleanup schema resolvers --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed search networked cleanup (#921) * cleanup utils * reorg types, decompose file, improve comments * clean up types * cleanup types, redundant code and fix a couple of bad merge files * fix enum * cleanup test types * type fix * explicit type import, use path alias --------- Co-authored-by: Ciaran Schutte <[email protected]> * Update modules/server/src/app.js Co-authored-by: Anders Richardsson <[email protected]> * make horrible ajax.ts console logging somewhat useful * improve release process * fix paths and imports for new build config * update tests to todos * fix some imports * fix build issues with lodash and missing config flag * fix paths * comment out test files, broken with tsx vs jest * fix path * fix path * improve release process * allow toolbar options customization * fix setsIndex conflict resolutioon misses * default for ENABLE_DOCUMENT_HITS env flag is true * default enable document hits to true * add DATA_MASK_MIN_THRESHOLD env var as pass through var * debug * debug * debug * add default param value * debug * string util important fix * fix for tests * Add comment re data masking * remove debug log * Update modules/server/src/mapping/createConnectionTypeDefs.js Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/createConnectionTypeDefs.js Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/masking.ts Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/resolveHitsFromAggs.ts Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/resolvers.ts Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/resolvers.ts Co-authored-by: Anders Richardsson <[email protected]> * remove specific NetworkAggError class * fix error response type narrow * missing param * add explanation comment for aggs resolver, gql info lookup --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> Co-authored-by: Anders Richardsson <[email protected]>

* Feature/federated (#926) * Network aggregation search - Initial Config (#876) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * Update modules/server/README.md Co-authored-by: Anders Richardsson <[email protected]> * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <[email protected]> * Update types.ts * Update README.md * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Feat/#873 retrieve remote schemas (#877) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <[email protected]> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <[email protected]> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <[email protected]> * use axios * axios network check, doesn't throw like fetch * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <[email protected]> * add logs to thrown errors * small feedback fixes * add typings + additional info in comments * add better errors * remove unused func. renaming * add return type * remove assertion. add TS filter * change nulls to undefined * change explicit undefined to optional ts * add tsdoc comment * change var type to loose record instead of uknown * if else TS * if else TS * change to nested promise return --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Pr feedback - doc updates (#879) * update tsdoc * update tsdoc --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/#874 merge schemas (#878) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <[email protected]> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <[email protected]> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * basic merge schema * cleanup comment * exclusuions * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <[email protected]> * use axios * axios network check, doesn't throw like fetch * update packages for typedefs * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <[email protected]> * add logs to thrown errors * fix merge conflict * add test watch command with basic test * tests * Fix type. narrow scope. hoist filtering * remove debug code. add comment * update comments + add type assertion * add comments * remove redundant options arg for schema merginging * seperate test utils and fixture from main file * seperate test utils and fixture from main file * tests, cleanup, extra tests, split responsibility --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Add core functionality for remote connection resolvers (#880) * add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/agg resolver (#881) * add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check * fetchGQL helper func working with axios * rework dynamic network search types based on remote connections types * use __type query instead of full introspectionQuery * remove old code * add single typeDef export, responsible for merging remote connection and aggregation types * correct structure for merging typedefs with other typedefs, merge on Query * get all types from remote connections once * agg resolver structuring * add aggregations resolvers * add agg resolver network calls * rename and narrow NetworkType type across files * add common file * add network queries file * rename var * query remote connections * move query * add comments * add supported and unsupported aggregations with test * ts-jest disable diagnostics to run tests without all types passing * rework connection status property to do ping healthcehck * types cleanup across files * improved typing for reducer in field types, covering supported and unsupported aggregates * tighten and rename types * type cleanup and comments * add explicit name mapping * rename type correctly * Cleanup query lookup for aggregations for remote connections * move TSdoc comment * move types * add util func to convert gql AST into string * move aggregations resolver code into a module * simplify query to be composable, better naming for agg query map * aggregation logic placeholder * cleanup * rework logic to use single request and mapping fields, instead of slicing central query --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/resolve stats (#883) * add resolve aggregation function with tests * add basic aggregation TS types * cleanup * improve comments --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/resolver responses (#884) * add documentName to config and use in remote connection gql queries * move fulfilled promise type guard to util and make it a generic * resolve aggregations from network query results' * return resolved response object * PR feedback --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fix network agg resolution (#885) * add Bucket gql object type * text change, move func outside of nested func * fix resolved aggregations overriding * Update modules/server/src/network/aggregations/index.ts Co-authored-by: Anders Richardsson <[email protected]> --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Response data & batching requests (#887) * move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * remove log --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fed Search request timeouts (#889) * move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * add GQL request timeout with config option and default value --------- Co-authored-by: Ciaran Schutte <[email protected]> * Reorg connection (#891) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo --------- Co-authored-by: Ciaran Schutte <[email protected]> * Accumulator pipeline (#893) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed total hits (#895) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * fix logging --------- Co-authored-by: Ciaran Schutte <[email protected]> * feat/agg acc tests (#897) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed numeric agg resolution (#898) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test * add test + expand types * rework resolution iteration and add numericAggregations resolver * add typename to types * cleanup --------- Co-authored-by: Ciaran Schutte <[email protected]> * use Math methods (#900) Co-authored-by: Ciaran Schutte <[email protected]> * Individual node config query fields (#899) * use more detailed NodeConfig object to store available agg types * filter requested aggs with available aggs * add available aggs to node connection resolver type defs * cleanup --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/aggregate not available (#903) * add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/filters (#904) * add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests * add args to Aggregations gql typedef * add args to network field * pass gql resolvers args to query creation * remove console log * add include missing: * add overture sqon builder dep * move up type * add SQON check and args type * rename file * check for SQON correctness but pass ordinary object through on success * add comment * Update modules/server/src/network/utils/sqon.ts Co-authored-by: Anders Richardsson <[email protected]> --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Minor comment updates (#905) * improve comment * improve comment --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fed search cleanup typings (#906) * remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * address feedback --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fix broken build (#907) * remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * fix gql query creation test * query back to normal test * disables tests, adds comment * fix merged file --------- Co-authored-by: Ciaran Schutte <[email protected]> * add agg mode env config (#908) * add agg mode env config * env config to strip out section of gql schema, typedef + resolver * prettier * prettier * Update modules/server/.env.schema Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/createConnectionResolvers.ts Co-authored-by: Anders Richardsson <[email protected]> * alphabetize * move up import * change AGG_ONLY_MODE to enabled DOCUMENT_HITS * add empty line at EOF * add comment * fix typo * fix inverted logic --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Feature/threshold calc (#909) * add conditional prop to schema * conditional schema field * conditional add dataMasking schema fields * add data masking logic * add agg only conditional to resolvers * clarify todo * get optional aggregations field and values for hits resolution * add mask func stub * fix missing prop from spread * masking logic * conditional resolve hits * move thresholding higher on resolver chain * add falsey belowThreshold value instead of null * data mask threshold env var * update wrong process.env path * seperate resolver creation from routing * check for undefiend value in lookup * clarify comment * add threshold value explanation * move types to common file * typing --------- Co-authored-by: Ciaran Schutte <[email protected]> * multiple node with data thresholding (#912) * fix build error * single endpoint * fix GQL type name collisions * move data mask threshold env. ensure data masking applies even if hits.total is not queried * change belowThreshold to relation object * surface api errors to axios responses * add relation logic and agg merging for network search * read data mask threshold min from config * check for valid env val * remove unused func * fix TS build errors * fix schema merge error --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed search cleanup (#920) * add placeholder * Adjust config template example and file * remove unused arranger constructor param * reorder constants file, parse env var * update arranger network config type * type, add gql server types * graphqlRoutes, remove unused import, export func * resolver creation, improve TS usage, split functionality into distinct files, improve usage' * add surrounding brackets for func * cleanup schema resolvers --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed search networked cleanup (#921) * cleanup utils * reorg types, decompose file, improve comments * clean up types * cleanup types, redundant code and fix a couple of bad merge files * fix enum * cleanup test types * type fix * explicit type import, use path alias --------- Co-authored-by: Ciaran Schutte <[email protected]> * Update modules/server/src/app.js Co-authored-by: Anders Richardsson <[email protected]> * make horrible ajax.ts console logging somewhat useful * improve release process * fix paths and imports for new build config * update tests to todos * fix some imports * fix build issues with lodash and missing config flag * fix paths * comment out test files, broken with tsx vs jest * fix path * fix path * improve release process * allow toolbar options customization * fix setsIndex conflict resolutioon misses * default for ENABLE_DOCUMENT_HITS env flag is true * default enable document hits to true * add DATA_MASK_MIN_THRESHOLD env var as pass through var * debug * debug * debug * add default param value * debug * string util important fix * fix for tests * Add comment re data masking * remove debug log * Update modules/server/src/mapping/createConnectionTypeDefs.js Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/createConnectionTypeDefs.js Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/masking.ts Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/resolveHitsFromAggs.ts Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/resolvers.ts Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/resolvers.ts Co-authored-by: Anders Richardsson <[email protected]> * remove specific NetworkAggError class * fix error response type narrow * missing param * add explanation comment for aggs resolver, gql info lookup --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * initial cleanup * remove remaining data masking config + calculations * redo remove data masking relation ref * remove leftover env vars --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> Co-authored-by: Anders Richardsson <[email protected]>

* Feature/federated (#926) * Network aggregation search - Initial Config (#876) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * Update modules/server/README.md Co-authored-by: Anders Richardsson <[email protected]> * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <[email protected]> * Update types.ts * Update README.md * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Feat/#873 retrieve remote schemas (#877) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <[email protected]> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <[email protected]> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <[email protected]> * use axios * axios network check, doesn't throw like fetch * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <[email protected]> * add logs to thrown errors * small feedback fixes * add typings + additional info in comments * add better errors * remove unused func. renaming * add return type * remove assertion. add TS filter * change nulls to undefined * change explicit undefined to optional ts * add tsdoc comment * change var type to loose record instead of uknown * if else TS * if else TS * change to nested promise return --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Pr feedback - doc updates (#879) * update tsdoc * update tsdoc --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/#874 merge schemas (#878) * initial config * add network to config template. add default values to base.json * add schema merge placeholders * add conditional to graphql route gen for env flag * throw error on invalid json parse eg. empty file * rename type * add vscode debug file to gitignore * add gql client lib * change gql lib * Update modules/server/README.md Co-authored-by: Anders Richardsson <[email protected]> * add fetch field * rename field to documentType * Update modules/server/configTemplates/base.json Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/config/utils/getConfigFromFiles.ts Co-authored-by: Anders Richardsson <[email protected]> * Update types.ts * Update README.md * add ignore to gitignore * rename fields, add nesting * remove env config to enable federated search * remove env config to enable federated search * remove flag from env schema * correct schema retrieval * fix GQL obj wrapping on schema construction * example * cleanup * cleanup gql file * cleanup packages * fix import * basic merge schema * cleanup comment * exclusuions * add introspection query type predicate * add tsdoc * remove local folder ignore from project.gitignore * remove unused util * renaming * break up query promise and results * Update .gitignore Co-authored-by: Anders Richardsson <[email protected]> * use axios * axios network check, doesn't throw like fetch * update packages for typedefs * change fetch config to axios config * Update modules/server/src/network/index.ts Co-authored-by: Anders Richardsson <[email protected]> * add logs to thrown errors * fix merge conflict * add test watch command with basic test * tests * Fix type. narrow scope. hoist filtering * remove debug code. add comment * update comments + add type assertion * add comments * remove redundant options arg for schema merginging * seperate test utils and fixture from main file * seperate test utils and fixture from main file * tests, cleanup, extra tests, split responsibility --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Add core functionality for remote connection resolvers (#880) * add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/agg resolver (#881) * add remote connection resolver func * adds resolver functionality for remote connection details, including health check to gql server * add types, improve comment, move into remote connection resolver into distinct file * remove unused imports * add utility type. use utility objectValues type for type values from const * fix type lookup in gql object * add comments * PR feedback: comment blocks * additional remote connection node data resolver response data * create typeDefs file for remote connection data * move typeDefs for aggregations * add resolver map, rename remote connection resolver file * cleanup imports and creation of schema, imorts from distinct resolver and typedef files * rename and fix type on Remote Connection gql resp * remove ping healthcheck for nodes on resolver * pass correct object to get available types * add temp gql server for network aggs * add status check * fetchGQL helper func working with axios * rework dynamic network search types based on remote connections types * use __type query instead of full introspectionQuery * remove old code * add single typeDef export, responsible for merging remote connection and aggregation types * correct structure for merging typedefs with other typedefs, merge on Query * get all types from remote connections once * agg resolver structuring * add aggregations resolvers * add agg resolver network calls * rename and narrow NetworkType type across files * add common file * add network queries file * rename var * query remote connections * move query * add comments * add supported and unsupported aggregations with test * ts-jest disable diagnostics to run tests without all types passing * rework connection status property to do ping healthcehck * types cleanup across files * improved typing for reducer in field types, covering supported and unsupported aggregates * tighten and rename types * type cleanup and comments * add explicit name mapping * rename type correctly * Cleanup query lookup for aggregations for remote connections * move TSdoc comment * move types * add util func to convert gql AST into string * move aggregations resolver code into a module * simplify query to be composable, better naming for agg query map * aggregation logic placeholder * cleanup * rework logic to use single request and mapping fields, instead of slicing central query --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/resolve stats (#883) * add resolve aggregation function with tests * add basic aggregation TS types * cleanup * improve comments --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/resolver responses (#884) * add documentName to config and use in remote connection gql queries * move fulfilled promise type guard to util and make it a generic * resolve aggregations from network query results' * return resolved response object * PR feedback --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fix network agg resolution (#885) * add Bucket gql object type * text change, move func outside of nested func * fix resolved aggregations overriding * Update modules/server/src/network/aggregations/index.ts Co-authored-by: Anders Richardsson <[email protected]> --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Response data & batching requests (#887) * move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * remove log --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fed Search request timeouts (#889) * move response object creation into module * change type structure for gql server * change resolver code based on new gql type * adjust agg type gql resolved response * add http resp helpers * change resolveAggregation code to batch processing * add nodeInfo to resolver * change resolver method to reducer with mutations * add type * fix typedef * format response object * comments and cleanup * add GQL request timeout with config option and default value --------- Co-authored-by: Ciaran Schutte <[email protected]> * Reorg connection (#891) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo --------- Co-authored-by: Ciaran Schutte <[email protected]> * Accumulator pipeline (#893) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed total hits (#895) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * fix logging --------- Co-authored-by: Ciaran Schutte <[email protected]> * feat/agg acc tests (#897) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed numeric agg resolution (#898) * change resolver code based on new gql type * comments and cleanup * break up main entry point into two modules, one for querying and one for field processing * rename network config type, remove supported aggs from type * type cleanup * only send network queries with original query fields * move gql health check into module * cleanup, renaming types, removing redundant code * add type * fix merged conflict * adds full stop. fixes comment typo * fix pipeline code, move accumulator into class * handle unavailable node * check undefined in Success * add note * rename poorly named field * add comment: * rename file * http response success status as const * renamed AggAccumulator file * clean up types, change loose object to use record, fix any typings * add total hits to query string * cleanup field, param order * add test * rename RemoteAggregations type * add hits TS type * fix env toggle for network fed search * renaming, update types * documentName and documentType are the same for functionality * return unique fields * fix accumulator resolve for new input shape * tighten types. add Hits resolution * rename count to hits * add first test for requested fields * tighten types and error checking * fix up types and comments, resolvers code fix * update aggregation test * add test + expand types * rework resolution iteration and add numericAggregations resolver * add typename to types * cleanup --------- Co-authored-by: Ciaran Schutte <[email protected]> * use Math methods (#900) Co-authored-by: Ciaran Schutte <[email protected]> * Individual node config query fields (#899) * use more detailed NodeConfig object to store available agg types * filter requested aggs with available aggs * add available aggs to node connection resolver type defs * cleanup --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/aggregate not available (#903) * add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/filters (#904) * add TSdoc example * add TSdoc comment * pass more data into Aggs accumulator * add schema map for query time * rework agg accumulator to add empty agg to bucket if aggregation not available for node * fix test * clean up logic to account for only Aggregations type * account for empty aggs field in node query * cleanup * disabled network Agg tests * add args to Aggregations gql typedef * add args to network field * pass gql resolvers args to query creation * remove console log * add include missing: * add overture sqon builder dep * move up type * add SQON check and args type * rename file * check for SQON correctness but pass ordinary object through on success * add comment * Update modules/server/src/network/utils/sqon.ts Co-authored-by: Anders Richardsson <[email protected]> --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Minor comment updates (#905) * improve comment * improve comment --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fed search cleanup typings (#906) * remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * address feedback --------- Co-authored-by: Ciaran Schutte <[email protected]> * Fix broken build (#907) * remove unsued type defs * add type watch script * fix import * fix import * type an ANY type * fix agg resolver typing * ts no-check for test file * fix Agg Accumulator types * type response formatter * fix test file type * fix gql query creation test * query back to normal test * disables tests, adds comment * fix merged file --------- Co-authored-by: Ciaran Schutte <[email protected]> * add agg mode env config (#908) * add agg mode env config * env config to strip out section of gql schema, typedef + resolver * prettier * prettier * Update modules/server/.env.schema Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/createConnectionResolvers.ts Co-authored-by: Anders Richardsson <[email protected]> * alphabetize * move up import * change AGG_ONLY_MODE to enabled DOCUMENT_HITS * add empty line at EOF * add comment * fix typo * fix inverted logic --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * Feature/threshold calc (#909) * add conditional prop to schema * conditional schema field * conditional add dataMasking schema fields * add data masking logic * add agg only conditional to resolvers * clarify todo * get optional aggregations field and values for hits resolution * add mask func stub * fix missing prop from spread * masking logic * conditional resolve hits * move thresholding higher on resolver chain * add falsey belowThreshold value instead of null * data mask threshold env var * update wrong process.env path * seperate resolver creation from routing * check for undefiend value in lookup * clarify comment * add threshold value explanation * move types to common file * typing --------- Co-authored-by: Ciaran Schutte <[email protected]> * multiple node with data thresholding (#912) * fix build error * single endpoint * fix GQL type name collisions * move data mask threshold env. ensure data masking applies even if hits.total is not queried * change belowThreshold to relation object * surface api errors to axios responses * add relation logic and agg merging for network search * read data mask threshold min from config * check for valid env val * remove unused func * fix TS build errors * fix schema merge error --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed search cleanup (#920) * add placeholder * Adjust config template example and file * remove unused arranger constructor param * reorder constants file, parse env var * update arranger network config type * type, add gql server types * graphqlRoutes, remove unused import, export func * resolver creation, improve TS usage, split functionality into distinct files, improve usage' * add surrounding brackets for func * cleanup schema resolvers --------- Co-authored-by: Ciaran Schutte <[email protected]> * Feat/fed search networked cleanup (#921) * cleanup utils * reorg types, decompose file, improve comments * clean up types * cleanup types, redundant code and fix a couple of bad merge files * fix enum * cleanup test types * type fix * explicit type import, use path alias --------- Co-authored-by: Ciaran Schutte <[email protected]> * Update modules/server/src/app.js Co-authored-by: Anders Richardsson <[email protected]> * make horrible ajax.ts console logging somewhat useful * improve release process * fix paths and imports for new build config * update tests to todos * fix some imports * fix build issues with lodash and missing config flag * fix paths * comment out test files, broken with tsx vs jest * fix path * fix path * improve release process * allow toolbar options customization * fix setsIndex conflict resolutioon misses * default for ENABLE_DOCUMENT_HITS env flag is true * default enable document hits to true * add DATA_MASK_MIN_THRESHOLD env var as pass through var * debug * debug * debug * add default param value * debug * string util important fix * fix for tests * Add comment re data masking * remove debug log * Update modules/server/src/mapping/createConnectionTypeDefs.js Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/createConnectionTypeDefs.js Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/masking.ts Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/resolveHitsFromAggs.ts Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/resolvers.ts Co-authored-by: Anders Richardsson <[email protected]> * Update modules/server/src/mapping/resolvers.ts Co-authored-by: Anders Richardsson <[email protected]> * remove specific NetworkAggError class * fix error response type narrow * missing param * add explanation comment for aggs resolver, gql info lookup --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> * initial cleanup * remove remaining data masking config + calculations * add network agg property from env * main conflicts * read directly from config --------- Co-authored-by: Ciaran Schutte <[email protected]> Co-authored-by: Anders Richardsson <[email protected]> Co-authored-by: Anders Richardsson <[email protected]>

Ciaran Schutte added 17 commits September 19, 2024 21:24

change resolver code based on new gql type

1f743da

comments and cleanup

a088522

break up main entry point into two modules, one for querying and one …

e7bc1c9

…for field processing

rename network config type, remove supported aggs from type

b4c7230

type cleanup

9412fed

only send network queries with original query fields

f764bd0

move gql health check into module

7d1a46b

cleanup, renaming types, removing redundant code

5046131

add type

c11dd43

fix merged conflict

68ab587

adds full stop. fixes comment typo

f0ddc0c

fix pipeline code, move accumulator into class

e264b12

handle unavailable node

da646e3

check undefined in Success

8160e19

add note

bba55c2

remove unused code

5105765

rename poorly named field

347b9e5

ciaranschutte commented Sep 24, 2024

View reviewed changes

add comment:

711b693

ciaranschutte marked this pull request as ready for review September 24, 2024 16:59

ciaranschutte requested a review from joneubank September 24, 2024 16:59

ciaranschutte assigned demariadaniel Sep 24, 2024

ciaranschutte requested review from demariadaniel and justincorrigible September 24, 2024 16:59

ciaranschutte unassigned demariadaniel Sep 24, 2024

joneubank requested changes Sep 24, 2024

View reviewed changes

Ciaran Schutte added 2 commits September 26, 2024 18:27

rename file

2388add

http response success status as const

6d665b7

Ciaran Schutte added 2 commits September 26, 2024 18:33

renamed AggAccumulator file

36b7fc6

clean up types, change loose object to use record, fix any typings

762beeb

ciaranschutte requested a review from joneubank September 27, 2024 01:24

demariadaniel reviewed Sep 27, 2024

View reviewed changes

demariadaniel approved these changes Sep 27, 2024

View reviewed changes

ciaranschutte merged commit 866bc0f into feature/federated_search Sep 27, 2024

ciaranschutte deleted the accumulator_pipeline branch September 27, 2024 20:49


		const emptyAggregation: NetworkAggregation = { bucket_count: 0, buckets: [] };

		export class AggregationAccumulator {

Accumulator pipeline #893

Accumulator pipeline #893

Uh oh!

Conversation

ciaranschutte commented Sep 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joneubank left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ciaranschutte commented Sep 27, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

demariadaniel left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ciaranschutte commented Sep 24, 2024 •

edited

Loading