Skip to content

Conversation

@psychedelicious
Copy link
Contributor

@psychedelicious psychedelicious commented May 25, 2023

feat(nodes, ui): change how "intermediate" artefacts are handled

Requirements for Handling Intermediates

intermediates refers to node outputs that do not need to be saved after the nodes/session that used them are complete.

Currently, intermediates are an ImageType. This doesn't make sense - both results and uploads can also be intermediates. For example, the canvas' init and mask images are uploaded intermediates, and most of the images output during the overall canvas process are result intermediates.

To handle intermediates properly:

  • We need to be able to flag nodes and their outputs as intermediate
  • We need to be able to remove the outputs when they are no longer needed (else we will have tons of extra images laying around)

Additionally, we need to be able to exclude intermediate images from the gallery.

Changes in this PR

The concept intermediate now refers a node and its outputs

  • ImageType is now restricted to results and uploads.
  • Add a reserved meta field to nodes to hold the is_intermediate boolean. We can extend it in the future to support other node meta. Add a reserved is_intermediate field to nodes instead, per @Kyle0654 's suggestion.
  • Add a is_intermediate column to the images table to hold this. (When latents, conditioning etc are added to the DB, they will also have this column.)
  • All nodes default to *not* intermediate. Nodes must explicitly be marked intermediate for their outputs to be intermediate.
  • When building a graph, you can set node.meta.is_intermediate=True and it will be handled as an intermediate.

Handle marking uploads as intermediate

Session IDs are only generated when a session is created. We cannot create a session with an explicit id.

So we cannot associate an uploaded image with a session until the session has been created.

But to use an image in a graph (eg an init image on canvas), we need to know its name, which is only created when it is uploaded.

This creates a catch-22 for the canvas and likely many future features.

There are a few options, each of which I implemented and tested:

  1. Create a graph-less session (to get the session ID), then upload associated images and manually add every node and edge to the graph individually.

Very tedious, needs like a dozen network calls to build the graph

  1. Create a graph-less session, upload the images, and finally add the graph as a GraphInvocation to the session

Works but leaves the shape of the graph awkward; its nodes have a single graph node, which then has its own nodes object, which have the actual nodes to execute.

  1. Create the graph, skipping the image name and type of the nodes that need them, then upload the images, and finally update the graph's nodes to have the image name.

Works but updating the graph after the fact seems risky, because if something goes awry, the fail case is generation fails.

  1. Upload the images without associating with the session, create the graph, then afterwards update the images to associate them.

More work to set up (need to add capability to update image records), but the fail case is only that the uploads are not associated with a session - generation is not affected.

I think 4 is is the best, and lays groundwork for future capability to update image records.

So, this PR also includes these changes:

  • Add a new update() method to the ImageService, and a route to call it. Updates have a strict model, currently only session_id and image_category may be updated.
  • Add a new update() method to the ImageRecordStorageService to update the image record using the model.
  • Update the canvas graph generation to flag its uploaded init and mask images as intermediate.
  • During canvas setup, hit the update route to associate the uploaded images with the session id.

Other changes

  • Organize the socketio and RTK listener middlware better. Needed to facilitate the updated canvas logic.
  • Add a new action sessionReadyToInvoke. The sessionInvoked action is only ever run in response to this event. This lets us do whatever complicated setup (eg canvas) and explicitly invoking. Previously, invoking was tied to the socket subscribe events.
  • Some minor tidying.

@psychedelicious
Copy link
Contributor Author

fixes #3385

- `ImageType` is now restricted to `results` and `uploads`.
- Add a reserved `meta` field to nodes to hold the `is_intermediate` boolean. We can extend it in the future to support other node `meta`.
- Add a `is_intermediate` column to the `images` table to hold this. (When `latents`, `conditioning` etc are added to the DB, they will also have this column.)
- All nodes default to `*not* intermediate`. Nodes must explicitly be marked `intermediate` for their outputs to be `intermediate`.
- When building a graph, you can set `node.meta.is_intermediate=True` and it will be handled as an intermediate.
- Add a new `update()` method to the `ImageService`, and a route to call it. Updates have a strict model, currently only `session_id` and `image_category` may be updated.
- Add a new `update()` method to the `ImageRecordStorageService` to update the image record using the model.
- Update the canvas graph generation to flag its uploaded init and mask images as `intermediate`.
- During canvas setup, hit the update route to associate the uploaded images with the session id.
- Organize the socketio and RTK listener middlware better. Needed to facilitate the updated canvas logic.
- Add a new action `sessionReadyToInvoke`. The `sessionInvoked` action is *only* ever run in response to this event. This lets us do whatever complicated setup (eg canvas) and explicitly invoking. Previously, invoking was tied to the socket subscribe events.
- Some minor tidying.
@hipsterusername hipsterusername merged commit 0c3b4bb into main May 26, 2023
@hipsterusername hipsterusername deleted the feat/intermediates branch May 26, 2023 02:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants