Subgraph Composition: Reading the entities for subgraph as a datasource #5544

zorancv · 2024-07-17T15:53:25Z

Contains initial implementation of the database access functionality for fetching a single entity in a range of blocks. Will later be extended to support multiple entities.

TODOs in the code are addressed in later PRs.

zorancv · 2024-11-07T13:06:37Z

store/postgres/src/relational_queries.rs

        schema.entity_type(&self.entity).unwrap()
    }

+    pub fn deserialize_block_number<T: FromEntityData>(self) -> Result<BlockNumber, StoreError> {


This function is a temporary implementation, needed at this stage. It will be completely removed in a later PR, so it can be safely ignored here.

zorancv · 2024-11-07T13:12:08Z

store/postgres/src/relational_queries.rs

+        out.push_sql("  from ");
+        out.push_sql(self.table.qualified_name.as_str());
+        out.push_sql(" e\n where ");
+        // TODO: do we need to care about it?


Yes, we do. Enabled in a later PR.

zorancv · 2024-11-07T13:13:17Z

store/test-store/tests/postgres/writable.rs

+fn read_range_mutable() {
+    run_test(|store, writable, deployment| async move {
+        let num_entities = read_range(store, writable, deployment, true).await;
+        assert_eq!(num_entities, 6) // TODO: fix it - it should be 4 as the range is open


Will be reworked later on.

zorancv · 2024-11-07T13:13:47Z

store/postgres/src/block_range.rs

+/// Helper for generating SQL fragments for selecting entities in a specific block range
+#[derive(Debug, Clone, Copy)]
+pub enum EntityBlockRange {
+    Mutable(BlockRange), // TODO: check if this is a proper type here (maybe Range<BlockNumber>?)


Seems to be fine.

graph/src/components/store/traits.rs

store/postgres/src/block_range.rs

store/postgres/src/relational.rs

store/postgres/src/relational_queries.rs

lutter · 2024-11-12T23:44:07Z

store/postgres/src/writable.rs

+        entity_type: &EntityType,
+        block_range: Range<BlockNumber>,
+    ) -> Result<BTreeMap<BlockNumber, Vec<Entity>>, StoreError> {
+        self.store.get_range(entity_type, block_range)


This will return the wrong result if block_range covers blocks that are still in the queue - those aren't in the database yet. You'd need to use the BlockTracker to go through the queue and find entities for those blocks. You also only want to look in the store for the part of block_range that isn't in the queue.

I've removed it from the Queue and from the Writer and left if only in the WritableStore with additional comment. Would that be acceptable? Alternative is to create ReaderStore struct.

I don't think we should do that. The dependent subgraph that is only reading from a source subgraph should not have a WritableStore for the source subgraph at all, since it exposes way more functionality than we need, and it's a good idea to make it clear that we do not want to write to the source subgraph.

That will either mean that you add a new method to ReadStore and implement it for DeploymentStore, or create another trait that only has a get_range method and implement that for DeploymentStore. The SubgraphStore then needs a new method to get such a store for a given deployment.

I decided to go with a new trait since the methods of readStore would need to be implemented in WritableStore too.

Right now the WritableStore is a private member of the SourceableStore struct. It is only used for implementing the DeploymentCursorTracker methods of the SourceableStore trait, which replaces ReadStore in this context. I hope that addresses your concerns sufficiently.

Ok finally removed all traces of WritableStore from the SourceableStore. Feel free to review it with the other two PRs. Sorry for the confusion before.

lutter

Nice! I think this is almost ready to merge, but it would be good to address the redundant passing of (DeploymentHash, SourceableStore) and just use SourceableStore there.

lutter · 2024-12-06T23:57:15Z

graph/src/components/store/traits.rs

 }

+#[async_trait]
+pub trait SourceableStore: Sync + Send + 'static {


Nice! I like that this is a very small interface which makes it easy to determine what one of these things could possibly do when it is in use.

lutter · 2024-12-07T00:02:10Z

graph/src/blockchain/mod.rs

        store: impl DeploymentCursorTracker,
        start_blocks: Vec<BlockNumber>,
-        source_subgraph_stores: Vec<(DeploymentHash, Arc<dyn ReadStore>)>,
+        source_subgraph_stores: Vec<(DeploymentHash, Arc<dyn SourceableStore>)>,


You don't necessarily need the DeploymentHash here since SourceableStore.input_schema().id() will return it (and there could be a convenience method on SourceableStore for it)

Actually, not having it explicit lowers the risk that there is a mismatch between the two.

lutter · 2024-12-07T00:03:51Z

core/src/subgraph/instance_manager.rs

        }
    }

    pub async fn hashes_to_read_store<C: Blockchain>(


Maybe this should be renamed now that it returns SourceableStore not ReadStore

lutter · 2024-12-07T00:06:06Z

graph/src/util/lfu_cache.rs

    }

-    pub fn iter<'a>(&'a self) -> impl Iterator<Item = (&K, &V)> {
+    pub fn iter<'a>(&'a self) -> impl Iterator<Item = (&'a K, &'a V)> {


Did rustc start complaining about the lifetimes here?

Yes. Already merged in master as our CI updates automatically to the the latest version of rustc.

zorancv marked this pull request as draft July 17, 2024 15:53

zorancv force-pushed the zoran/subgraph-datasource branch 4 times, most recently from 7260289 to 37f8a5f Compare July 17, 2024 20:41

incrypto32 changed the base branch from master to krishna/subgraph-block-stream-exp-3 July 18, 2024 11:23

zorancv changed the base branch from krishna/subgraph-block-stream-exp-3 to master July 18, 2024 13:59

zorancv force-pushed the zoran/subgraph-datasource branch 4 times, most recently from c33222c to 41a0f8d Compare August 12, 2024 10:49

incrypto32 added this to the Subgraph Composition milestone Sep 10, 2024

incrypto32 changed the base branch from master to subgraph-composition-blockstream September 11, 2024 08:10

incrypto32 changed the base branch from subgraph-composition-blockstream to master September 11, 2024 08:10

incrypto32 changed the base branch from master to subgraph-composition-blockstream September 11, 2024 08:43

incrypto32 force-pushed the subgraph-composition-blockstream branch 3 times, most recently from d6be3d7 to dd39f66 Compare September 11, 2024 10:43

incrypto32 force-pushed the zoran/subgraph-datasource branch from 87f411b to 10725c2 Compare September 11, 2024 10:44

incrypto32 force-pushed the subgraph-composition-blockstream branch from dd39f66 to 3aba98f Compare September 12, 2024 07:26

incrypto32 force-pushed the zoran/subgraph-datasource branch from 10725c2 to 0a89a4e Compare September 12, 2024 07:26

incrypto32 changed the title ~~Reading the entities for subgraph as a datasource~~ Subgraph Composition: Reading the entities for subgraph as a datasource Sep 12, 2024

incrypto32 assigned zorancv Sep 17, 2024

incrypto32 marked this pull request as ready for review November 7, 2024 06:17

zorancv commented Nov 7, 2024

View reviewed changes

zorancv requested a review from lutter November 7, 2024 16:45

lutter requested changes Nov 12, 2024

View reviewed changes

zorancv force-pushed the zoran/subgraph-datasource branch 2 times, most recently from 8ce342f to 9acd51e Compare November 14, 2024 19:10

zorancv force-pushed the zoran/subgraph-datasource branch 6 times, most recently from 6a47d26 to 0d7e4d5 Compare November 29, 2024 14:46

zorancv marked this pull request as draft December 2, 2024 23:37

zorancv force-pushed the zoran/subgraph-datasource branch from d6ef813 to 77850be Compare December 4, 2024 12:45

zorancv marked this pull request as ready for review December 5, 2024 08:41

zorancv requested a review from lutter December 5, 2024 08:48

zorancv mentioned this pull request Dec 5, 2024

Subgraph composition sql more entities #5614

Closed

fordN added the composition label Dec 5, 2024

zorancv force-pushed the zoran/subgraph-datasource branch from 81fb399 to 70ca609 Compare December 6, 2024 14:03

lutter requested changes Dec 7, 2024

View reviewed changes

lutter approved these changes Dec 19, 2024

View reviewed changes

Subgraph composition : TriggersAdapter wrapper

383bbc9

incrypto32 force-pushed the subgraph-composition-blockstream branch from c7ca25a to 383bbc9 Compare January 31, 2025 09:06

incrypto32 force-pushed the zoran/subgraph-datasource branch from 0a4a2a6 to b5af690 Compare January 31, 2025 10:41

Subgraph Composition: Reading the entities for subgraph as a datasource

d2f6352

incrypto32 force-pushed the zoran/subgraph-datasource branch from b5af690 to d2f6352 Compare January 31, 2025 12:03

incrypto32 changed the base branch from subgraph-composition-blockstream to feat-subgraph-composition February 3, 2025 11:03

incrypto32 force-pushed the feat-subgraph-composition branch from f46fff0 to bd0ab9a Compare February 3, 2025 11:09

incrypto32 closed this Feb 3, 2025

Subgraph Composition: Reading the entities for subgraph as a datasource #5544

Subgraph Composition: Reading the entities for subgraph as a datasource #5544

Uh oh!

Conversation

zorancv commented Jul 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zorancv Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lutter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zorancv commented Jul 17, 2024 •

edited

Loading

zorancv Nov 7, 2024 •

edited

Loading