Struggling to Migrate Complex Time-Resource Scheduling Model to Declarative Shadow Variables

[zzk0803]

Hi Timefold team and community,

I’m reaching out for guidance on migrating a non-trivial scheduling model to the newer declarative shadow variable approach. My current implementation relies on a custom VariableListener, which works well—but I understand that VariableListener may be deprecated in future versions, and the new declarative style doesn’t allow access to ScoreDirector. I’ve tried several approaches without success and would greatly appreciate your advice.

Background & Motivation

In the legacy version, I successfully used a VariableListener on my ProducingArrangement entity to compute two shadow variables:

• producingDateTime (actual start time)
• completedDateTime (actual completion time)

These depend on two planning variables:

• factoryInstance (which production unit to assign to)
• datetimeSlot (which time granule to assign to)

The listener maintained an ordered sequence per factory, inserted the arrangement into a TreeSet, and propagated updated times across affected arrangements via ScoreDirector.getWorkingSolution().

However, the new declarative shadow variable mechanism prohibits access to ScoreDirector, making this pattern impossible to replicate directly. Given the potential deprecation of VariableListener, I’m eager to find a sustainable, forward-compatible solution.

Domain & Model Overview

My problem involves joint resource and time scheduling with sequence constraints:

• FactoryInstance: A production unit. Most have a finite queue (e.g., 6 slots) and process jobs in FIFO order; a few support parallel execution.
• DatetimeSlot: A time granule derived from a work calendar and slot size (e.g., 30min, 60min).
• Product: A problem fact containing BOM and processing duration.
• ProducingArrangement: The core planning entity with:
  • Planning variables: factoryInstance, datetimeSlot
  • Shadow variables (to be computed): producingDateTime, completedDateTime

The key challenge: the actual start time of an arrangement depends on both the factory’s queue state and the completion time of prior arrangements in the same factory, which in turn may span multiple datetimeSlots.

Why Chain/Sequence Variables Didn’t Work

I initially explored chained variables or @PlanningListVariable, but abandoned them because the logical sequence (determined by factory queue order) often diverges from the assignment order of datetimeSlot. For example, an arrangement assigned to an earlier time slot might end up later in the chain due to factory capacity constraints—breaking time consistency.

Attempts with Declarative Shadow Variables (and Why They Failed)

✅ Attempt 1: Intermediate variables + basic shadow computation

I tried computing intermediate data from planning variables and deriving shadow times from them.
Problem: When the logical order of arrangements changes (e.g., due to a different factory assignment), the underlying planning variables may not change—but the sequence does. This leads to score corruption, as shadow times aren’t updated for arrangements whose relative position changed but whose direct variables didn’t.

✅ Attempt 2: Factory holds @PlanningListVariable List<Arrangement>

Modeled like a chained list per factory.
Problem: Same as before—the list order doesn’t align with chronological datetimeSlot assignments, causing incorrect time propagation.

✅ Attempt 3: Introduce FactoryDatetimeSlot (Cartesian product)

Created a composite entity for each (factory, datetimeSlot) pair, holding a @PlanningListVariable List<Arrangement>.

• Added a shadow variable lastCompletedTime on FactoryDatetimeSlot (even though the docs discourage deriving shadow vars from mutable lists, it seemed computable).
• In the Arrangement’s shadow variable Supplier, I recursively looked up the previous FactoryDatetimeSlot’s lastCompletedTime and set:

  producingDateTime = max(previousSlot.lastCompletedTime, currentSlot.startTime)

Problem: Still results in score corruption. I suspect the dependency chain across multiple FactoryDatetimeSlot entities isn’t fully propagated during incremental score calculation.

Additional Context

• My full toy project is open-source: https://github.com/zzk0803/TownshipScheduler
• I’m happy to provide minimal reproducers, model diagrams, or test cases if helpful.

Questions for the Team

1. Is there a recommended modeling pattern for resource-constrained queues with time-slot assignments and sequence-dependent timing?
2. Can declarative shadow variables safely handle cross-entity time propagation (e.g., completion time of one arrangement affecting the start time of another in the same resource queue)?
3. If VariableListener is indeed being phased out, are there plans for alternative APIs that allow controlled access to working solution state during shadow variable updates?

Thank you very much for your time and continued work on this excellent framework. Any insights or suggestions would be deeply appreciated!

[TomCools]

Hi @zzk0803, thank you for your message and detail. I had a look at the repository you mentioned.

What confuses me a little bit is your statement on not being able to use the PlanningListVariable

the list order doesn’t align with chronological datetimeSlot assignments, causing incorrect time propagation.

I am not entirely sure I grasp your problem here. Do I understand it correctly that you want a sequence, but the increments of that sequence must match the pre-set timeslots?

As per your questions:

1. "Is there a recommended approach" depends on how you answer my question above. But typically when we talk about a "sequence" which doesn't fit in a single "datetimeslot", and you need to compute start/end times with items earlier in the sequence, that is best solved with the @PlanningListVariable

2. Declarative Shadow Variables do not really do that, they are an easy way to create Shadow Variables for 1 entity, using fields of that entity (which might be facts or variables). These limitations are added on purpose, as the current variable listeners were pretty complex to explain and use. The ScoreDirector is actually something we'd like to get rid of in our API as it just adds complexity. (people often forget to register changes). If you would be ysing the @PlanningListVariable model as suggested in the point above, you could have used the @CascadingUpdateShadowVariable to update downstream elements in a sequence, easily updating the chain without the need to hold on to a TreeMap yourself.

3. You are the 2nd person who has brought up access to the Working Solution and we indeed see some good reasons you might want access to it. You could work around that by adding the relevant parts of the working solution inside the entity that needs them to compute shadow variables.

[zzk0803]

Hi TomCools,

Thank you again for your thoughtful response — I really appreciate you taking the time to look into this!

Before diving into technical solutions, I’d like to clarify why I’m insisting on keeping datetimeGrain (previously called datetimeSlot) as a core planning variable.

This design stems from observing player behavior in simulation games like Township:
Most production tasks take 20+ minutes, and players cannot—and do not—continuously monitor each facility to “click and schedule” in real time.

A more realistic interaction pattern is:

Players open the game every hour (e.g., every 60 minutes) and batch-schedule upcoming actions, such as:

10:00: Plant wheat, corn, and potatoes (within field capacity limits);
11:00: Harvest wheat and corn (potatoes still in progress), and start producing Grain and Syrup in factories (which are queue-based: tasks must execute one after another, respecting max queue length);
12:00: Produce Eggs (its prerequisite Grain finished at 11:30);
13:00: Bake Potato Bread (all prerequisites now complete).
Thus, I divide time into fixed datetimeGrain intervals (e.g., 60 minutes each), and each ProducingArrangement must be assigned to a specific datetimeGrain, representing “the player intends to start this action at this time.”
This isn’t about optimizing utilization—it’s about modeling the player’s discrete, limited decision moments.

Here’s the key nuance:

Globally, the execution order is fixed by the natural chronological order of datetimeGrain—the solver must not reorder these, as the player has already committed to acting at specific times.
Locally, however, for each FactoryInstance (e.g., a syrup factory or a field), all assigned Arrangements must form a chain-like queue, since a single resource cannot process multiple tasks concurrently.
Additionally, there’s a hard business rule:

The actual producingDateTime must not be earlier than the start of its assigned datetimeGrain.
For example, if the player schedules syrup production at 11:00, but the factory’s previous task finishes at 11:20, the actual start time is 11:20—not 11:00.

In short:

The overall plan isn’t a single chain, but each factory internally forms a chain—and that chain’s order must strictly follow the chronological order of the datetimeGrains assigned to its arrangements.

I previously tried sorting the factory’s arrangement list by datetimeGrain inside a VariableListener before computing completion times, but this caused score corruption—because if the list is a @PlanningListVariable, its order is itself a decision variable, and forcibly reordering it in a listener breaks consistency with the solver’s state.

Therefore, datetimeGrain isn’t an optional helper—it’s the foundation of both player intent and resource sequencing, and must remain a planning variable.

This brings me to your mention of @CascadingUpdateShadowVariable, which sparked a key question:

I understand that @CascadingUpdateShadowVariable predates the newer declarative shadow variables (@ShadowVariable + @ShadowSource).
However, I’ve also noticed that the new declarative approach can, in fact, reference the previous element in a chain—for example, by using @PreviousElementShadowVariable (or a similar inverse shadow variable)—to compute something like:

producingDateTime = max(datetimeGrain.start, previous.completedDateTime)

Yet the official documentation doesn’t clearly address this pattern. Instead, your reply highlighted @CascadingUpdateShadowVariable, which made me wonder:

Does @CascadingUpdateShadowVariable offer capabilities beyond what @ShadowVariable + @PreviousElementShadowVariable can achieve (e.g., automatic cascading propagation, safer incremental updates)?
Is the newer declarative model now sufficient for chain-based time propagation, making @CascadingUpdateShadowVariable legacy?
Or do both coexist, targeting different levels of complexity?
Given the lack of documentation comparing these approaches, I’m unsure which pattern is recommended for my use case:

“Each factory must maintain a chain ordered by external datetimeGrain, with completion times cascading forward.”

If the declarative approach (@PlanningListVariable + @ShadowVariable + @PreviousElementShadowVariable) is viable, then the critical challenge becomes:
How can I ensure the factory’s internal list order always matches the natural chronological order of datetimeGrain—without letting the solver freely reorder it?

Thank you so much for your guidance. These distinctions are crucial for me to model correctly with Timefold!

Best regards,
@zzk0803

[Christopher-Chianelli]

@CascadingUpdateShadowVariable does not offer capabilities belong what @ShadowVariable + @PreviousElementShadowVariable can offer; it is basically equivalent to @ShadowSources({"all non-casading variables on entity", "previous.cascadeVariable"}). It also have a slightly different API than declarative variables, which allows one method to update several variables at the same time (since the method is responsible for updating the variable in casading, whereas you need multiple methods for declarative, since a function can only return one object).

Declarative variables are more flexible, and allow you to declare dependencies on other declarative variables, which you cannot do for @CascadingUpdateShadowVariable.

From your problem description, I would make FactoryInstance a PlanningEntity with a PlanningListVariable that contain ProducingArrangement, make factoryInstance an InverseShadowVariable, and datetimeSlot a declarative ShadowVariable. I would add a @PlanningPinToIndex fact on each FactoryInstance so the player's committed actions cannot be rearranged (https://docs.timefold.ai/timefold-solver/latest/responding-to-change/responding-to-change#partiallyPinnedPlanningListVariable). I would declare the shadow variables like this:

@ShadowSources({"producingDateTime"})
Integer datetimeSlotSupplier() {
    if (producingDateTime == null) {
        return null;
    }
    return roundToNextTimeGrain(producingDateTime, productionDuration);
}

@ShadowSources({"inverse", "previous.completedDateTime"})
Integer producingDateTimeSupplier() {
    if (inverse == null) {
        return null;
    }
    if (previous == null) {
        return FIRST_TIME_GRAIN;
    }
    return previous.completedDateTime;
}

@ShadowSources({"producingDateTime"})
Integer completedDateTimeSupplier() {
    if (producingDateTime == null) {
        return null;
    }
    return producingDateTime + productionDuration;
}

[zzk0803]

Thank you again for the detailed suggestion!
However, I realize there’s a subtle but important difference in our assumptions:

In my use case, the datetimeGrain is not a fixed commitment, but a planning variable to be optimized.
The player hasn’t decided “I will act at 11:00”; instead, the solver must choose the best time grain for each action, considering factory queues 、arrangement dependencies(product bom)、and other constrains.

Therefore, pinning the list order (via @PlanningPinToIndex) would prevent the solver from exploring better time assignments, which is the core of the optimization.

[TomCools]

I think what @Christopher-Chianelli is trying to say that if there is any cut-off, you can pin. A natural cut-off is when the time has expired (you planned something at 10am, it is now 10:01, so the previous action should not be replanned anymore as it is probably underway), or you might choose to not change anything in the plan for the next 10 minutes (you planned something at 10AM, is is now 9u51, so the planned item should stay).

Whenever you talk about queues, @PlanningListVariable is usually the correct one to use. This still optimizes the order in which you do things and you can, using shadow variables, still force it to start on the timegrains you mention.

[ShadowVariable]

Can yall stop tagging me Leeban Ahmed

…

On Mon, Sep 29, 2025 at 8:01 PM zzk0803 ***@***.***> wrote: Thank you again for the detailed suggestion! However, I realize there’s a subtle but important difference in our assumptions: In my use case, the datetimeGrain is not a fixed commitment, but a planning variable to be optimized. The player hasn’t decided “I will act at 11:00”; instead, the solver must choose the best time grain for each action, considering factory queues 、arrangement dependencies(product bom)、and other constrains. Therefore, pinning the list order (via @PlanningPinToIndex) would prevent the solver from exploring better time assignments, which is the core of the optimization. — Reply to this email directly, view it on GitHub <#1837 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALMIOZJW3S55J7BYN2N3KD3VHW6ZAVCNFSM6AAAAACHUZ3OQ6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTINJUG44TQNY> . You are receiving this because you were mentioned.Message ID: ***@***.*** .com>

[TomCools]

Hmm, the annotation @ShadowVariable has been in our code for many years. Surprised this only pops up now.
I would recommend following up on https://github.com/orgs/community/discussions/174585, which should give you more control on what kinds of notifications you'll get. We can't really enforce people to not accidently tag you I'm afraid.

[zzk0803]

I’m starting to realize that my problem might be far more difficult and complex than I originally thought — or perhaps it’s actually simple, and I just haven’t managed to break through that final layer of understanding.

Until now, I’ve been quietly proud of my little “clever hack”: using @PlanningVariable + a custom VariableListener + a TreeMap to make everything work. But this discussion has made me suddenly aware that this very “cleverness” might one day come back to haunt me like a time bomb.

I don’t deny the authority of @PlanningListVariable in modeling sequential chains — in fact, I really wish I could use it. It would save me lines of code and align with best practices. Unfortunately, reality has been disappointing: my try_xxx branches are now filled with commits trying every possible way to combine declarative shadow variables with the model I stubbornly insist on — and every single attempt ends with a commit message saying “not work”.

Score corruption!
Either in the shadow variables, or in the constraint streams.

Timefold is magic — but even magic has its fundamental laws.
To get it right, everything must be perfect:
entity design, data preparation, constraint stream logic, shadow variable suppliers, and solver configuration.
One tiny mistake anywhere, and the whole thing falls apart.

Thankfully, there’s still that “unconventional” approach — the hard-won little trick I finally figured out. It works. It runs. I understand it. It solves my problem. It doesn’t throw errors.

So, thank you — truly — to all the maintainers and experts on the front lines of development for taking the time to answer what might seem like a naive or trivial question. I’ve learned something valuable from this exchange.

No matter what, I’ll keep following this gem of an open-source project. I haven’t studied anything — not even Spring or Tomcat — with this level of persistence.

I think I’ll end the discussion here.
Thank you all, again.