[bug]: nodes cache broken

[psychedelicious]

Is there an existing issue for this?

• I have searched the existing issues

OS

Linux

GPU

cuda

VRAM

No response

What version did you experience this issue on?

main

What happened?

After generating for a while (hundreds of images), images now fail to save. Tried generating several times, always fails while attempting to save the image. Unfortunately I did not have the debugger running so I couldn't dig any deeper. Restarting the server resolved the issue.

Traceback:

Traceback (most recent call last):
  File "/home/bat/Documents/Code/InvokeAI/invokeai/app/services/processor.py", line 58, in __process
    outputs = invocation.invoke(
  File "/home/bat/Documents/Code/InvokeAI/invokeai/app/invocations/generate.py", line 105, in invoke
    context.services.images.save(
  File "/home/bat/Documents/Code/InvokeAI/invokeai/app/services/image_storage.py", line 236, in save
    self.__set_cache(image_path, image)
  File "/home/bat/Documents/Code/InvokeAI/invokeai/app/services/image_storage.py", line 273, in __set_cache
    del self.__cache[cache_id]
KeyError: '/home/bat/Documents/Code/outputs/images/uploads/de7c940d-9c35-4d71-9603-6c5b122ffd9f_1682844216.png'

Full invocation error:

{
    "graph_execution_state_id": "22ea0261-c3b7-49f2-99fa-06fd1f868231",
    "node": {
        "id": "99ab961c-fedb-4058-b75b-896a0a0eae56",
        "type": "txt2img",
        "prompt": "metallic alien robot cat cute chibi []",
        "seed": 846982582,
        "steps": 50,
        "width": 448,
        "height": 768,
        "cfg_scale": 7.5,
        "scheduler": "k_euler",
        "seamless": false,
        "model": "dreamlike-photoreal-2.0",
        "progress_images": true
    },
    "source_node_id": "79d3f919-5646-4961-8a4a-32c754b75f14",
    "error": "Traceback (most recent call last):\n  File \"/home/bat/Documents/Code/InvokeAI/invokeai/app/services/processor.py\", line 58, in __process\n    outputs = invocation.invoke(\n  File \"/home/bat/Documents/Code/InvokeAI/invokeai/app/invocations/generate.py\", line 105, in invoke\n    context.services.images.save(\n  File \"/home/bat/Documents/Code/InvokeAI/invokeai/app/services/image_storage.py\", line 236, in save\n    self.__set_cache(image_path, image)\n  File \"/home/bat/Documents/Code/InvokeAI/invokeai/app/services/image_storage.py\", line 273, in __set_cache\n    del self.__cache[cache_id]\nKeyError: '/home/bat/Documents/Code/outputs/images/uploads/de7c940d-9c35-4d71-9603-6c5b122ffd9f_1682844216.png'\n",
    "timestamp": 1682844757
}

Screenshots

No response

Additional context

No response

Contact Details

No response

[Kyle0654]

Interesting, looks like it's trying to delete things from the cache that aren't there. I wonder if caching is working at all. Do you have a debugger attached?

[psychedelicious]

I just happened to not have to attached when this occurred - usually while developing I do have it attached. I'm pretty sure the cache is doing something but I'll take a closer look today.

[psychedelicious]

Confirmed the cache does work generally, both setting and getting items from the cache.

The problem seems to be that the cache and cache_ids somehow got out of sync. I'm 99% sure that I didn't have two threads working at the same time when this happened, but it's possible (the threading issue was not resolved when this occurred).