fix: Fix sidebar when selecting JSON property #1231

pulpdrew · 2025-10-01T18:23:23Z

Closes HDX-2042
Closes HDX-2524
Closes HDX-2307
Closes #1010

Summary

This PR fixes errors that occurred when attempting to open the sidebar by clicking a log table row using a JSON logs table schema.

The error was caused by node-sql-parser throwing exceptions when parsing SQL with JSON Expressions, resulting in HyperDX being unable to extract aliases from the SQL. In the long term, we'll want to have a true ClickHouse SQL parser. In the short term, this is fixed by:

Finding and replacing all JSON expressions in the sql with placeholder tokens, prior to parsing with node-sql-parser
Parsing with node-sql-parser to find aliases correctly
Replacing the placeholder tokens with the original JSON expressions

Testing

(All of the following use a JSON schema)

Before

When selecting a JSON column with an alias

When filtering by a JSON column and using an alias on a non-JSON property

After

When selecting a JSON column with an alias

When filtering by a JSON column and using an alias on a non-JSON property

changeset-bot · 2025-10-01T18:23:26Z

🦋 Changeset detected

Latest commit: cc6d4c3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/common-utils	Patch
@hyperdx/app	Patch
@hyperdx/api	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

vercel · 2025-10-01T18:23:28Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
hyperdx-v2-oss-app	Ready	Preview	Comment	Oct 6, 2025 8:54pm

github-actions · 2025-10-01T19:09:18Z

E2E Test Results

✅ All tests passed • 25 passed • 3 skipped • 205s

Status	Count
✅ Passed	25
❌ Failed	0
⚠️ Flaky	0
⏭️ Skipped	3

View full report →

pulpdrew · 2025-10-01T19:32:58Z

packages/app/src/hooks/useRowWhere.tsx

          // Currently we can't distinguish null or 'null'
          if (value === 'null') {
-            return SqlString.format(`isNull(??)`, [column]);
+            return SqlString.format(`isNull(?)`, [SqlString.raw(valueExpr)]);


This now ensures that the original expression, rather than the alias, is used. This matches the behavior elsewhere in this function and fixes another caused of sidebar loading failures for JSON types.

wrn14897 · 2025-10-02T22:05:46Z

We can ship patches for now. However, it looks like someone has already built the ClickHouse dialect and might get merged to the upstream. I think we should give it a try. In the long term, we should focus on improving this dialect instead of relying on the postgres one.

pulpdrew · 2025-10-02T22:09:23Z

(Accidentally closed, apologies)

pulpdrew · 2025-10-02T22:26:27Z

We can ship patches for now. However, it looks like someone has already built the ClickHouse dialect and might get merged to the upstream. I think we should give it a try. In the long term, we should focus on improving this dialect instead of relying on the postgres one.

Yeah I saw that! I upvoted the comments asking the maintainer to merge the ClickHouse support 🤞 It would be nice to be able to contribute to that instead of building these workarounds.

wrn14897 · 2025-10-02T23:29:56Z

packages/common-utils/src/utils.ts

 }

+/** Returns true if the given expression is a JSON expression, eg. `col.key.nestedKey` or "json_col"."key" */
+const isJsonExpression = (expr: string) => {


Are '.' or 1.5 valid JSON? This method returns true in these cases, but I’m not sure if that affects other methods.

Thanks, updated isJsonExpression so that it does not recognize . or 1.5 as JSON expressions

Could you add a few more unit tests for this method?

Added some more unit tests 👍

wrn14897 · 2025-10-03T00:17:02Z

packages/common-utils/src/clickhouse/index.ts

+    const { sqlWithReplacements, replacements: jsonReplacementsToExpressions } =
+      replaceJsonExpressions(sql);


Is it possible to enable this only when node-sql-parser isn't able to parse the sql or if it contains JSON syntax?

Unfortunately that isn't straightforward either, because node-sql-parser doesn't error on all JSON - sometimes it just parses the JSON incorrectly as schema.table.column or table.column. And determining if it contains json syntax would probably have to make use of the same functions being used here to find and replace the JSON.

wrn14897 · 2025-10-03T00:19:55Z

packages/common-utils/src/utils.ts

+ *
+ * Note - This function does not distinguish between json references and `table.column` references - both are returned.
+ */
+export function findJsonExpressions(sql: string) {


Don’t have a lot of confidence that this will work for all edge cases, but I’m okay with shipping the patch until we fix the parser

pulpdrew · 2025-10-03T13:09:04Z

@wrn14897 I also confirmed that the forked node-sql-parser with ClickHouse support does not yet support JSON 😞 We could consider contributing that support or try to get ClickHouse Analyzer in a usable state. 100% agreed that it would be best to have a working ClickHouse SQL parser instead of a bunch of patches on node-sql-parser

…alias

pulpdrew force-pushed the drew/allow-select-json-as-alias branch from 2728847 to eb634be Compare October 1, 2025 18:26

vercel bot deployed to Preview October 1, 2025 18:38 View deployment

pulpdrew force-pushed the drew/allow-select-json-as-alias branch from eb634be to 52388fe Compare October 1, 2025 19:07

vercel bot had a problem deploying to Preview October 1, 2025 19:08 Failure

pulpdrew force-pushed the drew/allow-select-json-as-alias branch from 52388fe to 81805af Compare October 1, 2025 19:10

vercel bot deployed to Preview October 1, 2025 19:13 View deployment

pulpdrew force-pushed the drew/allow-select-json-as-alias branch from 81805af to 99c2e3f Compare October 1, 2025 19:30

vercel bot deployed to Preview October 1, 2025 19:32 View deployment

pulpdrew commented Oct 1, 2025

View reviewed changes

fix: Fix sidebar when filtering or aliasing JSON property

30dbf17

pulpdrew force-pushed the drew/allow-select-json-as-alias branch from 99c2e3f to 30dbf17 Compare October 1, 2025 19:42

vercel bot deployed to Preview October 1, 2025 19:45 View deployment

pulpdrew requested a review from wrn14897 October 1, 2025 19:57

pulpdrew closed this Oct 2, 2025

pulpdrew reopened this Oct 2, 2025

wrn14897 reviewed Oct 2, 2025

View reviewed changes

wrn14897 reviewed Oct 3, 2025

View reviewed changes

vercel bot deployed to Preview October 3, 2025 11:09 View deployment

fix: Improve isJsonExpression

8d87e0a

pulpdrew force-pushed the drew/allow-select-json-as-alias branch from 0dadd7e to 8d87e0a Compare October 3, 2025 11:19

vercel bot deployed to Preview October 3, 2025 11:22 View deployment

test: Add additional tests for isJsonExpression

45295a6

vercel bot deployed to Preview October 3, 2025 20:38 View deployment

fix: Update isJsonExpression to handle single-quoted strings

127cf87

vercel bot deployed to Preview October 3, 2025 22:27 View deployment

wrn14897 mentioned this pull request Oct 4, 2025

fix: delimited column identifier for dynamic fields #1127

Open

pulpdrew added 3 commits October 6, 2025 09:17

Merge remote-tracking branch 'origin' into drew/allow-select-json-as-…

1599d74

…alias

fix: Replace escaping of dynamic isNull()

d77dd3e

fix: Handle quotes in identifiers when escaping JSON

b719569

vercel bot deployed to Preview October 6, 2025 14:08 View deployment

pulpdrew mentioned this pull request Oct 6, 2025

Error loading row data for traces in HyperDx UI hyperdxio/helm-charts#115

Closed

vercel bot deployed to Preview October 6, 2025 14:52 View deployment

fix: Handle replacements for JSON within function calls

dacb30d

pulpdrew force-pushed the drew/allow-select-json-as-alias branch from 394636f to dacb30d Compare October 6, 2025 16:49

vercel bot deployed to Preview October 6, 2025 16:52 View deployment

wrn14897 approved these changes Oct 6, 2025

View reviewed changes

pulpdrew added the automerge label Oct 6, 2025

Merge branch 'main' into drew/allow-select-json-as-alias

cc6d4c3

kodiakhq bot merged commit b46ae2f into main Oct 6, 2025
7 of 8 checks passed

kodiakhq bot deleted the drew/allow-select-json-as-alias branch October 6, 2025 20:53

vercel bot deployed to Preview October 6, 2025 20:54 View deployment

		const { sqlWithReplacements, replacements: jsonReplacementsToExpressions } =
		replaceJsonExpressions(sql);

fix: Fix sidebar when selecting JSON property #1231

fix: Fix sidebar when selecting JSON property #1231

Uh oh!

Conversation

pulpdrew commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Before

After

Uh oh!

changeset-bot bot commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

vercel bot commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Test Results

Uh oh!

pulpdrew Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

wrn14897 commented Oct 2, 2025

Uh oh!

pulpdrew commented Oct 2, 2025

Uh oh!

pulpdrew commented Oct 2, 2025

Uh oh!

wrn14897 Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pulpdrew Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

wrn14897 Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

pulpdrew Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

wrn14897 Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

pulpdrew Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

wrn14897 Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

pulpdrew commented Oct 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pulpdrew commented Oct 1, 2025 •

edited

Loading

changeset-bot bot commented Oct 1, 2025 •

edited

Loading

vercel bot commented Oct 1, 2025 •

edited

Loading

github-actions bot commented Oct 1, 2025 •

edited

Loading

wrn14897 Oct 2, 2025 •

edited

Loading