background callback with MATCH is cancelled by future callbacks with different component ids

[Jonas1302]

Describe your context

dash                        2.14.1
dash-bootstrap-components   1.5.0
dash-core-components        2.0.0
dash-extensions             1.0.3
dash-html-components        2.0.0
dash-iconify                0.1.2
dash-mantine-components     0.12.1
dash-table                  5.0.0
dash-uploader               0.7.0a1

Describe the bug
If a background callback using pattern matching with MATCH is triggered twice by two different objects (and therefore different values for MATCH), the first callback will be cancelled.

This only applies to background callbacks. "Normal" callbacks work fine.

Expected behavior
Both callbacks should finish execution and return their outputs, just like non-background callbacks. (At least if their IDs are different)

MWE
Here's a small example to reproduce the problem:

import os
import time

import diskcache
import dash
from dash import Dash, html, Output, Input, MATCH, DiskcacheManager
from dash.exceptions import PreventUpdate


def layout():
    return html.Div(
        [
            *[html.Button(f"Update {i}", id=dict(type="button", index=i)) for i in range(5)],
            *[html.P(id=dict(type="text", index=i)) for i in range(5)],
        ]
    )


@app.callback(
    Output(dict(type="text", index=MATCH), "children"),
    Input(dict(type="button", index=MATCH), "n_clicks"),
    background=True,
    prevent_initial_call=True,
)
def show_text(n_clicks: int):
    if not n_clicks:
        raise PreventUpdate

    index = dash.ctx.triggered_id["index"]
    print(f"started {index}")
    time.sleep(3)

    print(f"stopped {index}")
    return str(index)


if __name__ == "__main__":
    cache = diskcache.Cache(os.path.join(os.getcwd(), ".cache"))
    app = Dash(background_callback_manager=DiskcacheManager(cache))
    app.layout = layout()
    app.run(host="0.0.0.0", port=8010)

If you click on multiple buttons (within 3 seconds after the last click), the previous execution of the callback will be canceled and only the id of the last button will be shown.

Sample output:

started 0
started 1
started 2
started 3
started 4
stopped 4

[dwmorris11]

If you are updating the same output with 2 different inputs and the second asynchronous call finishes before the first one. It updates the component. Then the first one finishes and updates the component, undoing the changes from the second input. Is that the behavior we want?

[mbschonborn]

I'm running into the same issue. The desired functionality is that all Outputs are updated the same way as if the callbacks were normal callbacks, not background callbacks.

@dwmorris11 each callback is updating a separate output. In the MWE provided there are no two callbacks changing the same output, because all the ID dictionaries for both buttons and paragraphs are different.

[alkasm]

I am also having what I think is the same issue. Here's a different minimal repro:

import time
from uuid import uuid4
from dash import MATCH, Dash, DiskcacheManager, Input, Output, State, html

app = Dash(__name__, background_callback_manager=DiskcacheManager())

N_JOBS_TO_ADD = 50

div = html.Div(children=[])
button = html.Button(children=f"Add {N_JOBS_TO_ADD} jobs")

@app.callback(
    Output(div, "children"),
    State(div, "children"),
    Input(button, "n_clicks"),
    prevent_initial_call=True,
)
def click_button(children, n_clicks):
    print(f"click_button({len(children)=}, {n_clicks=})")
    children.extend(
        [
            html.Div(["In Progress..."], id={"type": "test", "value": str(uuid4())})
            for _ in range(N_JOBS_TO_ADD)
        ]
    )
    return children

@app.callback(
    Output({"type": "test", "value": MATCH}, "children"),
    Input({"type": "test", "value": MATCH}, "id"),
    background=True,
)
def update_progress(id):
    print(f"update_progress({id=})")
    time.sleep(1)
    return ["Done"]

app.layout = html.Div([button, div])
app.run(port=8051, debug=True)

When clicking the "Add 50 Jobs" button I expect the app to create 50 divs with the text "In Progress...", kick off 50 background callbacks which take a second to run, and should replace the text in each of the divs with "Done". Instead, only a few of them update with "Done" and the rest stay at "In Progress...".

I'm not sure if the callbacks are "cancelled", but nevertheless most of the matched output components don't update. Additionally, if I try to "intercept" between this callback and the html component update, for e.g. by outputting to an intermediary store, then adding a new callback using the store to update the component, that intercepting callback similarly will only be called a few times---in the same way that only a few divs are updated.

[mbschonborn]

Alkasm's example illustrates the problem perfectly. The whole behavior switches when setting "background" between True and False. I also want to add that in my app I am using a Celery background manager, while the example above uses Diskcache manager, so it is probably unrelated to a specific manager. I also noticed that almost always six calls complete while the rest stalls (the callbacks continue to run but the result returned seems to be an empty base Object), so maybe that's exactly the first round of parallel processes being completed and from there on the rest is affected.

[mbschonborn]

This issue remains present in Dash 2.17.1 with Python 3.9 and 3.12