Neo4j-Browser becomes unresponsive with large query results #1003

huboneo · 2019-12-02T13:15:59Z

This PR introduces a new configuration option in browser for truncating max field length before they are passed to rendering layers. The reason for this is that certain queries can produce a single result with a lot of values in a given field, which we had yet to account for.

Screenshot

oskarhane

Some really good stuff in here.

Some comments:

I get somewhat inconsistent results. If you have the movies database and set the "Max field items" to 1 and execute MATCH p=()-[r:ACTED_IN]->() RETURN p LIMIT 25 I see two nodes in the viz.
But switching to table view I see 25 node objects.
The limitations we already have with "Max Rows" in table view + "Initial node display" in viz, maybe those should be removed and replaced by this one (since this operates at a lower level)?
I don't see a truncation message in the viz's statusbar.
Maybe not part of this PR, but just to point it out, property size is also a performance killer (try returning a node with a property of 1MB).

oskarhane · 2019-12-20T07:27:05Z

src/browser/modules/Sidebar/Settings.jsx

      },
+      {
+        maxFieldItems: {
+          displayName: 'Max field items',


"field items" is not very clear what it means. Maybe "Item Limit" or or something to make it more clear?
"Field" isn't a word we regularly use except deep in the driver response.

oskarhane · 2019-12-20T07:30:14Z

src/browser/modules/Stream/CypherFrame/CodeView.jsx

+    const resultJson = JSON.stringify(
+      request.result.records,
+      fieldLimiterFactory(maxFieldItems),
+      2
+    )


Since this only does JSON.stringify and only executes when the user switches to this tab, can we skip the truncation here?
The purpose of the code tab/view is to give the developer a window into what's being returned from the driver without any formatting.

Unfortunately not, even JSON.stringify() gets overwhelmed with large result fields

oskarhane · 2019-12-20T07:38:19Z

src/shared/services/bolt/boltMappings.test.js

      expect(out.nodes.length).toEqual(4)
    })
-    test('should find items in paths with segments', () => {
+    test('should find items in paths with segments, and only return unique items', () => {


huboneo · 2020-01-07T11:34:41Z

@oskarhane thank you for your feedback.

Indeed there was a bug truncation.
Added a message to Vis for truncated messages
Updated setting display name
Unsure about removing max row truncation as that is on a different axis from fields. Lets leave it as is for now?

oskarhane

Thanks for the update on this.

Result bug

I still get mismatches between the different view though.
To reproduce:

Load the common movie dataset
Set the limit to 1
Run MATCH p=(n:Movie)-[]-(:Person) RETURN p LIMIT 25 and compare the views. The table view only sees one item per row while viz and text show the whole path.
See:

Phrasing

I've re-tought the phrasing as well and I think it's ok to use the word "fields" as this is the first time we describe the records width to users. So sorry about the back-and-forth on that.
I've suggest what wording to use via comments.

oskarhane · 2020-01-13T08:09:39Z

src/browser/modules/Stream/CypherFrame/TableView.jsx

+        <Ellipsis>
+          {this.state.hasTruncatedFields && (
+            <StyledTruncatedMessage>
+              <Icon name='warning sign' /> Result fields have been


"Result fields" => "Records fields"

oskarhane · 2020-01-13T08:21:10Z

src/browser/modules/Sidebar/Settings.jsx

      },
+      {
+        maxFieldItems: {
+          displayName: 'Property item limit',


"Record field limit" describes this better imo. wdyt?

oskarhane · 2020-01-13T08:22:21Z

src/browser/modules/Sidebar/Settings.jsx

+        maxFieldItems: {
+          displayName: 'Property item limit',
+          tooltip:
+            'Max amount of items shown for list properties. When reached, lists get truncated.'


"Max number of fields per record being handled" is more clear imo. wdyt?

oskarhane · 2020-01-13T08:22:57Z

src/browser/modules/D3Visualization/components/Inspector.jsx

              <StyledInspectorFooterRowListValue className="value">
+                {this.props.hasTruncatedFields && (
+                  <StyledTruncatedMessage>
+                    <Icon name="warning sign" /> Result fields have been


"Result fields" => "Records fields"

huboneo · 2020-01-13T09:50:54Z

@oskarhane thank you again for your feedback. I have updated the PR. Tests are still failing I believe

huboneo · 2020-01-13T09:52:03Z

oskarhane

Thanks for the update!

I still see different results in table vs text view. They should be the same. The viz does some magic with internal relationships, but not the different table views.

Load movie dataset
Set limit to 1
Run MATCH p=()-[r:FOLLOWS]->() RETURN p LIMIT 25 and compare table and text views.

👍 on the wording changes.

oskarhane · 2020-04-14T14:09:36Z

Stable version of react-table is now out @huboneo. When yo have time, could you update this upstream and update this PR?

- Introduced new setting "maxFieldItems" - Now truncating data passed to CypherFrames whenever a field that is too large is detected - Refactored boltMapping code to increase performance - Side effect is that we now dedupe nodes, not sure if OK

huboneo · 2020-04-29T09:46:43Z

closing in favour of #1090

huboneo added bug DONT-MERGE labels Dec 2, 2019

huboneo requested a review from oskarhane December 2, 2019 13:15

huboneo force-pushed the perf-issue-wip branch from c2f2c14 to 801d5a7 Compare December 5, 2019 08:05

huboneo removed the DONT-MERGE label Dec 12, 2019

huboneo assigned shkirando Dec 12, 2019

huboneo force-pushed the perf-issue-wip branch from b99317b to e4d3bdd Compare December 13, 2019 09:28

oskarhane reviewed Dec 20, 2019

View reviewed changes

oskarhane mentioned this pull request Jan 3, 2020

Remove duplicates from the expandedNodeMap array #1031

Merged

huboneo force-pushed the perf-issue-wip branch from e4d3bdd to 0f066d3 Compare January 7, 2020 11:32

huboneo requested a review from oskarhane January 7, 2020 11:36

oskarhane reviewed Jan 13, 2020

View reviewed changes

oskarhane added the DONT-MERGE label Jan 16, 2020

huboneo added 7 commits April 24, 2020 12:33

bugfix unit test

4f20a0b

fix performance of flattenArray

131b452

update snapshots, add warning when fields are truncated

2bc1b77

changes from review

75837fe

only truncate top level cell values in tables

269b72c

wording

1732cab

huboneo force-pushed the perf-issue-wip branch from 2bc51ab to 1732cab Compare April 24, 2020 10:33

huboneo closed this Apr 29, 2020

huboneo deleted the perf-issue-wip branch April 29, 2020 09:54

Neo4j-Browser becomes unresponsive with large query results #1003

Neo4j-Browser becomes unresponsive with large query results #1003

Uh oh!

Conversation

huboneo commented Dec 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Screenshot

Uh oh!

oskarhane left a comment

Choose a reason for hiding this comment

Uh oh!

oskarhane Dec 20, 2019

Choose a reason for hiding this comment

Uh oh!

oskarhane Dec 20, 2019

Choose a reason for hiding this comment

Uh oh!

huboneo Jan 7, 2020

Choose a reason for hiding this comment

Uh oh!

oskarhane Dec 20, 2019

Choose a reason for hiding this comment

Uh oh!

huboneo commented Jan 7, 2020

Uh oh!

oskarhane left a comment

Choose a reason for hiding this comment

Result bug

Phrasing

Uh oh!

oskarhane Jan 13, 2020

Choose a reason for hiding this comment

Uh oh!

oskarhane Jan 13, 2020

Choose a reason for hiding this comment

Uh oh!

oskarhane Jan 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oskarhane Jan 13, 2020

Choose a reason for hiding this comment

Uh oh!

huboneo commented Jan 13, 2020

Uh oh!

huboneo commented Jan 13, 2020

Uh oh!

oskarhane left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oskarhane commented Apr 14, 2020

Uh oh!

huboneo commented Apr 29, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

huboneo commented Dec 2, 2019 •

edited

Loading

oskarhane Jan 13, 2020 •

edited

Loading

oskarhane left a comment •

edited

Loading