Just wondering when a 4.x "Stable" version is targeted for 🙌 #233
Replies: 49 comments 42 replies
-
Thanks for the interest.
So where are things at?
There are a couple of major pieces still to be done.
I'm working on the process execution paths .run .start etc and hoping to
have these working next week.
I still need a solution for namedlocks. I'm thinking of using native
locking mechanisms e.g. lock from the posix package and something from the
win32 package.
If you want to have a crack at solving that?
I also need a windows implementation of the mailbox code.
Moderation guide would also be great.
the unit test need to be worked through to insure they are all functioning
but this probably needs to wait until I've completed work in process
execution.
A review of the API looking for inconsistencies and missing outdated API
documentation.
A review of what functions belong in each of the new sub packages.
General sanity checks.
Key changes
the breaking up into multiple packages with the CLI tooling now in dcli_sdk.
A number of functions have been renamed with a Async suffix.
...
I'm using the alpha in some of my code. The core hurdle is the process
execution and the reality that I might still tweak the APIs given the above
reviews.
…On Fri, 8 Mar 2024, 12:40 am Tsavo Knott, ***@***.***> wrote:
Would love to know how "stable" the core functionality of 4.x Alpha is and
if it is too early to adopt/what we can do to help contribute & or create a
migration guide @bsutton <https://github.com/bsutton>!
Absolutely huge fan of this entire project and I'd love to jump in where I
can. Primarily working on server-side AOT GRPC engines and using DCLI to
manage all types of things from automated CI/CD to DevEx @ pieces.app 🙌
—
Reply to this email directly, view it on GitHub
<#233>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG32OH2DVLHH4S5ERCB6PLYXBU4VAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZWGMZTMOBUG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
I don't have a specific spot for tracking progress. If it suites we could just post updates here, the other option is to create a discussion or I could start a web page. Thoughts? |
Beta Was this translation helpful? Give feedback.
-
Is the plan to move most functions to async functions or is there a workaround to continue with the sync API? I'm all for moving to async! The whole |
Beta Was this translation helpful? Give feedback.
-
@passsy so the async/sync debate is a really interesting question that has kept me up at nights. A core reason for going sync originally was that dart did a terrible job reporting missing awaits and in cli programming one missing await could cause serious data loss. So going sync side stepped this issue. Since then dart has gotten much better at reporting missing awaits (providing you use the correct lints) with a few exceptions. For much of the existing api we have been able to retain it's sync style. There were quite a few xxxxSync methods in the dart api that I wasn't taking advantage of. Here are the big issues that I'm working through and could do with some help on. Review of callbacks If the 'action/callback' argument to one of these makes an async call and you don't mark the callback as async, then you are in for a world of grief - for instance the withTempDir function will delete the temp dir before your async callback completes. At this point I've created alternate variants of these methods: 'withTempDir' and 'withTempDirAsync' and added a warning to the sync version. I'm still not convinced that leaving the sync version in the api is safe (this is also an issue with the 'ask' function validator callbacks). I have added warnings to the sync version and a 'see also' to the async version, but I'm not certain if this is enough. Thoughts? Sync vs Async processes. The stream method (which was experimental) is moving to async - because it never made sense for it to be sync. My feeling is that I should still support the sync versions of the start/run functions (and friends) and I believe that I have path forward on these methods (functions like .run are already working). Mailbox support on Windows. Support for the Mailbox class (and hence sync process) on Windows will take some more work but I think this is just find the right ffi calls for windows. Mailbox macos support I do think there is an argument for an async version of the process calls. I did look to integrating the cli_scripts package into dcli but felt that there was an impedance mismatch in the way the api is written. At this point I'm inclined to keep pushing on getting the sync version of the process calls working - because I can see the light at then end of the tunnel and then review the async question post 4.x. Review/cleanup of pubspec_manager Linter Review of package structure It would be useful if someone could review the set of sub-packages for the set of methods each exposes and whether each method is in the correct package. I think that is all of the big issues. |
Beta Was this translation helpful? Give feedback.
-
Hey all! @bsutton do you have an idea of where the mailboxes are currently being used? (I can also go dig but a direction pointer would be great) Here was some quick notes I got from ChatGPT to start evaluating the usage of mailboxes:
Happy to jump in here as I have had a lot of experience with Dart FFI but definitely wanna plan this out a bit to know exactly how it's intending to be used and how we might make a cross platform abstraction. Let me know thoughts & happy to chat live as well. |
Beta Was this translation helpful? Give feedback.
-
At this point that is the only place that I intend to use mailboxes. Good to hear its working on mac os. I do my primary testing on ubuntu.
Given that the mailbox code was contributed by @mraleph there might be an argument that we directly use the native_synchronization package and perhaps contribute some of my changes back to that repo. Only issues it that it is marked experimental so may go away at some point. Happy for you to make the call. |
Beta Was this translation helpful? Give feedback.
-
@bsutton the refactor & implementation is complete. I have tested it internally with our CI/CD. We are smooth sailing. This test passes with flying colors: .dcli/test/src/process/process/synchronous_test.dart Will make a PR shortly in a fresh fork from master 🤝 |
Beta Was this translation helpful? Give feedback.
-
Wow, that's fantastic.
Appreciate the work.
If you are looking for another challenge, NamedLock is the next issue. It
uses a TCP socket as a interprocess lock (in the hardlock function). We
need to change this to a native lock function on all platforms. The posix
package may have a lock method exposed ( I'm they maintainer of the posix
package so we can patch that if necessary).
@mralph appreciate you contributions. We wouldn't have got this far without
your work!
…On Thu, 21 Mar 2024, 6:39 am Tsavo Knott, ***@***.***> wrote:
@bsutton <https://github.com/bsutton> the refactor & implementation is
complete. I have tested it internally with our CI/CD. We are smooth sailing.
This test passes with flying colors:
.dcli/test/src/process/process/synchronous_test.dart
Both locally:
Screenshot.2024-03-20.at.3.38.07.PM.png (view on web)
<https://github.com/onepub-dev/dcli/assets/102485237/e1c903ec-5722-440e-a915-afe5f30b9068>
And on all platforms:
Screenshot.2024-03-20.at.3.34.40.PM.png (view on web)
<https://github.com/onepub-dev/dcli/assets/102485237/e5ca696f-3c04-4efb-ba86-5c4899e1b7c7>
Will make a PR shortly in a fresh fork from master 🤝
—
Reply to this email directly, view it on GitHub
<#233 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG32OC6VAPKS252F25FYFTYZHQV3AVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQNJXGQ3DO>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Yes, please!
…On Thu, 21 Mar 2024, 10:54 am Tsavo Knott, ***@***.***> wrote:
@bsutton <https://github.com/bsutton> regarding a solution for named
locks:
Take a look at this *relatively simple* rust package
<https://crates.io/crates/named-lock> repo here
<https://github.com/oblique/named-lock>(1.6M Downloads)
lib.rs NamedLock Implementation
<https://github.com/oblique/named-lock/blob/9ddc8536c1e6884a88a402012c84eb4b5f90ce89/src/lib.rs#L69>
windows.rs: RawNamedLock
<https://github.com/oblique/named-lock/blob/9ddc8536c1e6884a88a402012c84eb4b5f90ce89/src/windows.rs#L21>
unix.rs RawNamedLock
<https://github.com/oblique/named-lock/blob/9ddc8536c1e6884a88a402012c84eb4b5f90ce89/src/unix.rs#L15>
Note its description:
On UNIX this is implemented by using files and flock
<https://linux.die.net/man/2/flock>. The path of the created lock file
will be $TMPDIR/.lock, or /tmp/.lock if TMPDIR environment variable is not
set.
On Windows this is implemented by creating named mutex with CreateMutexW
<https://docs.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-createmutexw>
.
I can reimplement this in Dart in about a day if it looks like a good
solution. Lmk 🫡
-T
—
Reply to this email directly, view it on GitHub
<#233 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG32OFPQMAPZUC3IQABBHDYZIOTFAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQNJZGEYDQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Yes, that's it.
…On Thu, 21 Mar 2024, 10:58 am Tsavo Knott, ***@***.***> wrote:
Is that what we're looking for?
I can rip it 🤝
—
Reply to this email directly, view it on GitHub
<#233 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG32OEWMRKGBU4TVK6IU2LYZIPBTAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQNJZGEZTM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
There is certainly an issue with PowerShell. If you can take a look at it that would be appreciated.
You can do a local activation via:
This is a function of dart pub not a dcli function. On windows it creates a .bat file which then runs some sort of semi compiled object - sorry can't remember the right term.
What is the concern here? If we wanted to go down the compiled path on install we probably need to do something like what the dswitch package does (I'm the maintainer). It does a aot compile and then adds the executable to the path.
Greatly appreciate any help I can get. DCli is a big project and so far it has mostly been just me. |
Beta Was this translation helpful? Give feedback.
-
@tsavo-at-pieces |
Beta Was this translation helpful? Give feedback.
-
I've added a link to a wiki page which lists the outstanding issues. |
Beta Was this translation helpful? Give feedback.
-
I've just finished publishing dcli alpha.11 You can now successfully activate dcli under dart 3.x - still lots of issues to solve. dart pub global activate dcli 4.0.1-alpha.11 |
Beta Was this translation helpful? Give feedback.
-
The rust part makes me anxious as using a second language makes ongoing
support harder.
…On Tue, 26 Mar 2024, 2:45 am Tsavo Knott, ***@***.***> wrote:
Good morning @bsutton <https://github.com/bsutton>!
Couple of updates - Spent a good bit of time last Thursday and Friday
beginning the initial port of the Rust package above... after implementing
all of the windows functionality through FFI similar to the
native_synchronization <https://pub.dev/packages/native_synchronization>
package I quickly realized that the named-lock rust package
<https://crates.io/crates/named-lock> uses additional rust packages
internally ... specifically parking_lot
<https://crates.io/crates/parking_lot> and once_cell
<https://crates.io/crates/once_cell>... which would also need to be
ported 🫠
SOOO - game plan number two: Leverage Flutter Rust Bridge
<https://cjycode.com/flutter_rust_bridge> to bind to the rust named-lock
package APIs so that they can be synchronously called in Dart.
Fingers crossed this should be done by EOD and will move MUCH faster than
porting all the rust directly into Dart.
Will update progress here in a bit!
Cheers,
-T
—
Reply to this email directly, view it on GitHub
<#233 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG32OFBJUCXT37E2S6TGSLY2BBDDAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSMBUGQ3TO>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
One other small question - if you take a look at the GitHub actions on
Windows it looks like it is unable to find DCLI on the path after global
activation.
Specifically it looks like DCLI is getting installed at at a different
location than expected see more here:
https://github.com/open-runtime/dcli-4.x/actions/runs/8698076609/job/23854484016#step:15:1
I'll look into it more tomorrow but just thought I'd throw it out here for
any thoughts.
All other platforms are looking excellent:
https://github.com/open-runtime/dcli-4.x/actions/runs/8698076609
Cheers,
-T
…On Mon, Apr 15, 2024, 9:42 PM Tsavo Knott ***@***.***> wrote:
Okay fantastic - last question!
Within a given isolate does each function call need to be atomic or can a
series of function calls all be ran within a single lock?
I'll read into it but how exactly are you leveraging the counter to enable
a reentrant behavior (if you know off the top of your head)?
I have a wide open day tomorrow to get this refactored and it's going to
be great!
Appreciate any and all considerations/pointers/nuances as I get into it 🤝
Cheers,
-T
On Mon, Apr 15, 2024, 9:33 PM Brett Sutton ***@***.***>
wrote:
> We are locking across processes and isolates. Within an isolate we are
> reentrant.
>
> You understanding of warmup and locking is correct.
>
> To state the possibly obvious, each lock with a different 'name' is not
> affected by locks with a different name.
>
> On Tue, 16 Apr 2024, 10:56 am Tsavo Knott, ***@***.***> wrote:
>
> > Okay - just wanting to make sure I fully understand the use case
> scenario:
> >
> > It looks like we're using NamedLocks for things like
> >
> > _pubget, _pubupgrade, clean, and warmup within the DartProject class.
> >
> > I am assuming this is to ensure that we don't run anything concurrently
> > which could cause race conditions i.e. running a clean which happens to
> > overlap with a warmup thus causing reliability issues?
> >
> > I am also assuming that this is also to prevent issues where you may be
> > running multiple dcli compile -i commands in parallel that are trying
> to
> > run _pubget at the same time - also potentially causing issues although
> > parallel dart pub get(s) may not conflict
> > <https://melos.invertase.dev/configuration/overview#runpubgetoffline>
> if
> > handled in the same regard as with something like the Melos Monorepo
> > Manager Package <https://pub.dev/packages/melos>
> >
> > Another question I'd love your thoughts on is around concurrency
> > limitations? Should the locks be leveraged in a binary capacity i.e.
> only
> > one process/thread can hold the lock at a time or is there a world
> where
> > multiple processes/threads can access the resource concurrently (up to
> a
> > limit).
> >
> > Lastly, it looks like NamedLocks hold a count specifically so they can
> be
> > "Reentrant", is this so the current isolate can call .withLock multiple
> > times without getting interrupted by another process competing to lock
> the
> > resource and interrupting the execution of the current isolate?
> >
> > I guess main theme I'm looking to understand is are we leveraging
> > NamedLock primarily to prevent cross-process conflicts or inter-process
> > conflicts or both?
> >
> > Here is a super simple example of how my current native_semaphore
> > <https://pub.dev/packages/runtime_native_semaphores> can be leveraged
> as
> > a locking mechanism:
> >
> > import 'dart:isolate';import
> 'package:runtime_native_semaphores/runtime_native_semaphores.dart' show
> NativeSemaphore;
> > void main() {
> > // Create a unique identifier for the semaphore
> > // I's suggest using an safe integer identifier from
> > // [safe_int_id](https://pub.dev/packages/safe_int_id)
> > final String name = 'my-native-named-semaphore-identifier';
> >
> > spawnIsolate(name, 1);
> > spawnIsolate(name, 2);
> > // Add more isolates as needed
> > }
> > Future<void> spawnIsolate(String name, int isolate) async {
> > void isolateEntryPoint(SendPort sendPort) {
> > final sem = NativeSemaphore(identifier: name);
> >
> > if (!sem.lock()) {
> > throw Exception("Failed to lock semaphore in isolate $isolate");
> > }
> >
> > // Perform work here...i.e. a random blocking operation
> > sleep(Duration(milliseconds: Random().nextInt(500)));
> >
> > if (!sem.unlock()) {
> > throw Exception("Failed to unlock semaphore in isolate $isolate");
> > }
> >
> > sendPort.send(true);
> > }
> >
> > final receivePort = ReceivePort();
> > await Isolate.spawn(isolateEntryPoint, receivePort.sendPort);
> > await receivePort.first;
> > //...
> >
> > // Cleanup
> > sem.dispose();
> > receivePort.close();
> >
> > }
> >
> > I also saw that there is a background parameter on the warmup function.
> > i.e. warmup({bool background = false, bool upgrade = false}) and I'm
> > assuming this is one of the cases where we need to have interprocess
> locks
> > as to not call another conflicting command while that background
> process is
> > running.
> >
> > Here is a link to the GitHub Repository for the native_semaphores
> > <https://github.com/open-runtime/native_semaphores/tree/main/test>
> >
> > If we need to have counters we can absolutely do this cross process as
> > well.
> >
> > I'll keep studying and when I have a little more direction/constraint
> set
> > I'll be able to get it in very quickly (as in perhaps EOD tomorrow)!
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <
> #233 (reply in thread)>,
>
> > or unsubscribe
> > <
> https://github.com/notifications/unsubscribe-auth/AAG32OGKKDMYJ7MT4WSTPM3Y5RZNBAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRTHE4TG>
>
> > .
> > You are receiving this because you were mentioned.Message ID:
> > ***@***.***>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <#233 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AYN4Z5IW6SIZUS4OTCEJ6DLY5R5VDAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRUGE3TE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
Beta Was this translation helpful? Give feedback.
-
The aim is to allow nested locks within an isolate.
Given the Async model of dart, any Async call is non atomic by nature, do
we can't enforce the rule.
…On Tue, 16 Apr 2024, 11:43 am Tsavo Knott, ***@***.***> wrote:
Okay fantastic - last question!
Within a given isolate does each function call need to be atomic or can a
series of function calls all be ran within a single lock?
I'll read into it but how exactly are you leveraging the counter to enable
a reentrant behavior (if you know off the top of your head)?
I have a wide open day tomorrow to get this refactored and it's going to
be
great!
Appreciate any and all considerations/pointers/nuances as I get into it 🤝
Cheers,
-T
On Mon, Apr 15, 2024, 9:33 PM Brett Sutton ***@***.***> wrote:
> We are locking across processes and isolates. Within an isolate we are
> reentrant.
>
> You understanding of warmup and locking is correct.
>
> To state the possibly obvious, each lock with a different 'name' is not
> affected by locks with a different name.
>
> On Tue, 16 Apr 2024, 10:56 am Tsavo Knott, ***@***.***> wrote:
>
> > Okay - just wanting to make sure I fully understand the use case
> scenario:
> >
> > It looks like we're using NamedLocks for things like
> >
> > _pubget, _pubupgrade, clean, and warmup within the DartProject class.
> >
> > I am assuming this is to ensure that we don't run anything
concurrently
> > which could cause race conditions i.e. running a clean which happens
to
> > overlap with a warmup thus causing reliability issues?
> >
> > I am also assuming that this is also to prevent issues where you may
be
> > running multiple dcli compile -i commands in parallel that are trying
to
> > run _pubget at the same time - also potentially causing issues
although
> > parallel dart pub get(s) may not conflict
> > <https://melos.invertase.dev/configuration/overview#runpubgetoffline>
> if
> > handled in the same regard as with something like the Melos Monorepo
> > Manager Package <https://pub.dev/packages/melos>
> >
> > Another question I'd love your thoughts on is around concurrency
> > limitations? Should the locks be leveraged in a binary capacity i.e.
> only
> > one process/thread can hold the lock at a time or is there a world
where
> > multiple processes/threads can access the resource concurrently (up to
a
> > limit).
> >
> > Lastly, it looks like NamedLocks hold a count specifically so they can
> be
> > "Reentrant", is this so the current isolate can call .withLock
multiple
> > times without getting interrupted by another process competing to lock
> the
> > resource and interrupting the execution of the current isolate?
> >
> > I guess main theme I'm looking to understand is are we leveraging
> > NamedLock primarily to prevent cross-process conflicts or
inter-process
> > conflicts or both?
> >
> > Here is a super simple example of how my current native_semaphore
> > <https://pub.dev/packages/runtime_native_semaphores> can be leveraged
> as
> > a locking mechanism:
> >
> > import 'dart:isolate';import
> 'package:runtime_native_semaphores/runtime_native_semaphores.dart' show
> NativeSemaphore;
> > void main() {
> > // Create a unique identifier for the semaphore
> > // I's suggest using an safe integer identifier from
> > // [safe_int_id](https://pub.dev/packages/safe_int_id)
> > final String name = 'my-native-named-semaphore-identifier';
> >
> > spawnIsolate(name, 1);
> > spawnIsolate(name, 2);
> > // Add more isolates as needed
> > }
> > Future<void> spawnIsolate(String name, int isolate) async {
> > void isolateEntryPoint(SendPort sendPort) {
> > final sem = NativeSemaphore(identifier: name);
> >
> > if (!sem.lock()) {
> > throw Exception("Failed to lock semaphore in isolate $isolate");
> > }
> >
> > // Perform work here...i.e. a random blocking operation
> > sleep(Duration(milliseconds: Random().nextInt(500)));
> >
> > if (!sem.unlock()) {
> > throw Exception("Failed to unlock semaphore in isolate $isolate");
> > }
> >
> > sendPort.send(true);
> > }
> >
> > final receivePort = ReceivePort();
> > await Isolate.spawn(isolateEntryPoint, receivePort.sendPort);
> > await receivePort.first;
> > //...
> >
> > // Cleanup
> > sem.dispose();
> > receivePort.close();
> >
> > }
> >
> > I also saw that there is a background parameter on the warmup
function.
> > i.e. warmup({bool background = false, bool upgrade = false}) and I'm
> > assuming this is one of the cases where we need to have interprocess
> locks
> > as to not call another conflicting command while that background
process
> is
> > running.
> >
> > Here is a link to the GitHub Repository for the native_semaphores
> > <https://github.com/open-runtime/native_semaphores/tree/main/test>
> >
> > If we need to have counters we can absolutely do this cross process as
> > well.
> >
> > I'll keep studying and when I have a little more direction/constraint
> set
> > I'll be able to get it in very quickly (as in perhaps EOD tomorrow)!
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <
>
#233 (reply in thread)>,
>
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/AAG32OGKKDMYJ7MT4WSTPM3Y5RZNBAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRTHE4TG>
>
> > .
> > You are receiving this because you were mentioned.Message ID:
> > ***@***.***>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <
#233 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AYN4Z5IW6SIZUS4OTCEJ6DLY5R5VDAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRUGE3TE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
—
Reply to this email directly, view it on GitHub
<#233 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG32OASQ3TT5V266PG74V3Y5R637AVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRUGI2DG>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Okay solid - this helps.
So within the same isolate we need previous functions that are running as
locked to finish up before the next function attempts to run some code
leveraging the same lock.
We can do this 🤝
…On Mon, Apr 15, 2024, 9:58 PM Brett Sutton ***@***.***> wrote:
The aim is to allow nested locks within an isolate.
Given the Async model of dart, any Async call is non atomic by nature, do
we can't enforce the rule.
On Tue, 16 Apr 2024, 11:43 am Tsavo Knott, ***@***.***> wrote:
> Okay fantastic - last question!
>
> Within a given isolate does each function call need to be atomic or can
a
> series of function calls all be ran within a single lock?
>
> I'll read into it but how exactly are you leveraging the counter to
enable
> a reentrant behavior (if you know off the top of your head)?
>
> I have a wide open day tomorrow to get this refactored and it's going to
> be
> great!
>
> Appreciate any and all considerations/pointers/nuances as I get into it
🤝
>
> Cheers,
> -T
>
>
> On Mon, Apr 15, 2024, 9:33 PM Brett Sutton ***@***.***> wrote:
>
> > We are locking across processes and isolates. Within an isolate we are
> > reentrant.
> >
> > You understanding of warmup and locking is correct.
> >
> > To state the possibly obvious, each lock with a different 'name' is
not
> > affected by locks with a different name.
> >
> > On Tue, 16 Apr 2024, 10:56 am Tsavo Knott, ***@***.***> wrote:
> >
> > > Okay - just wanting to make sure I fully understand the use case
> > scenario:
> > >
> > > It looks like we're using NamedLocks for things like
> > >
> > > _pubget, _pubupgrade, clean, and warmup within the DartProject
class.
> > >
> > > I am assuming this is to ensure that we don't run anything
> concurrently
> > > which could cause race conditions i.e. running a clean which happens
> to
> > > overlap with a warmup thus causing reliability issues?
> > >
> > > I am also assuming that this is also to prevent issues where you may
> be
> > > running multiple dcli compile -i commands in parallel that are
trying
> to
> > > run _pubget at the same time - also potentially causing issues
> although
> > > parallel dart pub get(s) may not conflict
> > > <https://melos.invertase.dev/configuration/overview#runpubgetoffline>
> > if
> > > handled in the same regard as with something like the Melos Monorepo
> > > Manager Package <https://pub.dev/packages/melos>
> > >
> > > Another question I'd love your thoughts on is around concurrency
> > > limitations? Should the locks be leveraged in a binary capacity i.e.
> > only
> > > one process/thread can hold the lock at a time or is there a world
> where
> > > multiple processes/threads can access the resource concurrently (up
to
> a
> > > limit).
> > >
> > > Lastly, it looks like NamedLocks hold a count specifically so they
can
> > be
> > > "Reentrant", is this so the current isolate can call .withLock
> multiple
> > > times without getting interrupted by another process competing to
lock
> > the
> > > resource and interrupting the execution of the current isolate?
> > >
> > > I guess main theme I'm looking to understand is are we leveraging
> > > NamedLock primarily to prevent cross-process conflicts or
> inter-process
> > > conflicts or both?
> > >
> > > Here is a super simple example of how my current native_semaphore
> > > <https://pub.dev/packages/runtime_native_semaphores> can be
leveraged
> > as
> > > a locking mechanism:
> > >
> > > import 'dart:isolate';import
> > 'package:runtime_native_semaphores/runtime_native_semaphores.dart'
show
> > NativeSemaphore;
> > > void main() {
> > > // Create a unique identifier for the semaphore
> > > // I's suggest using an safe integer identifier from
> > > // [safe_int_id](https://pub.dev/packages/safe_int_id)
> > > final String name = 'my-native-named-semaphore-identifier';
> > >
> > > spawnIsolate(name, 1);
> > > spawnIsolate(name, 2);
> > > // Add more isolates as needed
> > > }
> > > Future<void> spawnIsolate(String name, int isolate) async {
> > > void isolateEntryPoint(SendPort sendPort) {
> > > final sem = NativeSemaphore(identifier: name);
> > >
> > > if (!sem.lock()) {
> > > throw Exception("Failed to lock semaphore in isolate $isolate");
> > > }
> > >
> > > // Perform work here...i.e. a random blocking operation
> > > sleep(Duration(milliseconds: Random().nextInt(500)));
> > >
> > > if (!sem.unlock()) {
> > > throw Exception("Failed to unlock semaphore in isolate $isolate");
> > > }
> > >
> > > sendPort.send(true);
> > > }
> > >
> > > final receivePort = ReceivePort();
> > > await Isolate.spawn(isolateEntryPoint, receivePort.sendPort);
> > > await receivePort.first;
> > > //...
> > >
> > > // Cleanup
> > > sem.dispose();
> > > receivePort.close();
> > >
> > > }
> > >
> > > I also saw that there is a background parameter on the warmup
> function.
> > > i.e. warmup({bool background = false, bool upgrade = false}) and I'm
> > > assuming this is one of the cases where we need to have interprocess
> > locks
> > > as to not call another conflicting command while that background
> process
> > is
> > > running.
> > >
> > > Here is a link to the GitHub Repository for the native_semaphores
> > > <https://github.com/open-runtime/native_semaphores/tree/main/test>
> > >
> > > If we need to have counters we can absolutely do this cross process
as
> > > well.
> > >
> > > I'll keep studying and when I have a little more
direction/constraint
> > set
> > > I'll be able to get it in very quickly (as in perhaps EOD tomorrow)!
> > >
> > > —
> > > Reply to this email directly, view it on GitHub
> > > <
> >
>
#233 (reply in thread)>,
>
> >
> > > or unsubscribe
> > > <
> >
>
https://github.com/notifications/unsubscribe-auth/AAG32OGKKDMYJ7MT4WSTPM3Y5RZNBAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRTHE4TG>
>
> >
> > > .
> > > You are receiving this because you were mentioned.Message ID:
> > > ***@***.***>
> > >
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <
>
#233 (comment)>,
>
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/AYN4Z5IW6SIZUS4OTCEJ6DLY5R5VDAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRUGE3TE>
>
> > .
> > You are receiving this because you were mentioned.Message ID:
> > ***@***.***>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <
#233 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AAG32OASQ3TT5V266PG74V3Y5R637AVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRUGI2DG>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
—
Reply to this email directly, view it on GitHub
<#233 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AYN4Z5OZLJID75KN6FO3TFDY5SAWDAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRUGMZTC>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
So within the same isolate we need previous functions that are running as
locked to finish up before the next function attempts to run some code
leveraging the same lock.
We can do this 🤝
Sorry some miss communication
Within an isolate we need to allow nested locks which means the outer
action will be running when the inner action runs.
So overlapping actions needs to be allowed, otherwise we end up with a dead
lock.
…On Tue, 16 Apr 2024, 12:12 pm Tsavo Knott, ***@***.***> wrote:
Okay solid - this helps.
So within the same isolate we need previous functions that are running as
locked to finish up before the next function attempts to run some code
leveraging the same lock.
We can do this 🤝
On Mon, Apr 15, 2024, 9:58 PM Brett Sutton ***@***.***> wrote:
> The aim is to allow nested locks within an isolate.
>
> Given the Async model of dart, any Async call is non atomic by nature,
do
> we can't enforce the rule.
>
>
> On Tue, 16 Apr 2024, 11:43 am Tsavo Knott, ***@***.***> wrote:
>
> > Okay fantastic - last question!
> >
> > Within a given isolate does each function call need to be atomic or
can
> a
> > series of function calls all be ran within a single lock?
> >
> > I'll read into it but how exactly are you leveraging the counter to
> enable
> > a reentrant behavior (if you know off the top of your head)?
> >
> > I have a wide open day tomorrow to get this refactored and it's going
to
> > be
> > great!
> >
> > Appreciate any and all considerations/pointers/nuances as I get into
it
> 🤝
> >
> > Cheers,
> > -T
> >
> >
> > On Mon, Apr 15, 2024, 9:33 PM Brett Sutton ***@***.***> wrote:
> >
> > > We are locking across processes and isolates. Within an isolate we
are
> > > reentrant.
> > >
> > > You understanding of warmup and locking is correct.
> > >
> > > To state the possibly obvious, each lock with a different 'name' is
> not
> > > affected by locks with a different name.
> > >
> > > On Tue, 16 Apr 2024, 10:56 am Tsavo Knott, ***@***.***> wrote:
> > >
> > > > Okay - just wanting to make sure I fully understand the use case
> > > scenario:
> > > >
> > > > It looks like we're using NamedLocks for things like
> > > >
> > > > _pubget, _pubupgrade, clean, and warmup within the DartProject
> class.
> > > >
> > > > I am assuming this is to ensure that we don't run anything
> > concurrently
> > > > which could cause race conditions i.e. running a clean which
happens
> > to
> > > > overlap with a warmup thus causing reliability issues?
> > > >
> > > > I am also assuming that this is also to prevent issues where you
may
> > be
> > > > running multiple dcli compile -i commands in parallel that are
> trying
> > to
> > > > run _pubget at the same time - also potentially causing issues
> > although
> > > > parallel dart pub get(s) may not conflict
> > > > <
https://melos.invertase.dev/configuration/overview#runpubgetoffline>
>
> > > if
> > > > handled in the same regard as with something like the Melos
Monorepo
> > > > Manager Package <https://pub.dev/packages/melos>
> > > >
> > > > Another question I'd love your thoughts on is around concurrency
> > > > limitations? Should the locks be leveraged in a binary capacity
i.e.
> > > only
> > > > one process/thread can hold the lock at a time or is there a world
> > where
> > > > multiple processes/threads can access the resource concurrently
(up
> to
> > a
> > > > limit).
> > > >
> > > > Lastly, it looks like NamedLocks hold a count specifically so they
> can
> > > be
> > > > "Reentrant", is this so the current isolate can call .withLock
> > multiple
> > > > times without getting interrupted by another process competing to
> lock
> > > the
> > > > resource and interrupting the execution of the current isolate?
> > > >
> > > > I guess main theme I'm looking to understand is are we leveraging
> > > > NamedLock primarily to prevent cross-process conflicts or
> > inter-process
> > > > conflicts or both?
> > > >
> > > > Here is a super simple example of how my current native_semaphore
> > > > <https://pub.dev/packages/runtime_native_semaphores> can be
> leveraged
> > > as
> > > > a locking mechanism:
> > > >
> > > > import 'dart:isolate';import
> > > 'package:runtime_native_semaphores/runtime_native_semaphores.dart'
> show
> > > NativeSemaphore;
> > > > void main() {
> > > > // Create a unique identifier for the semaphore
> > > > // I's suggest using an safe integer identifier from
> > > > // [safe_int_id](https://pub.dev/packages/safe_int_id)
> > > > final String name = 'my-native-named-semaphore-identifier';
> > > >
> > > > spawnIsolate(name, 1);
> > > > spawnIsolate(name, 2);
> > > > // Add more isolates as needed
> > > > }
> > > > Future<void> spawnIsolate(String name, int isolate) async {
> > > > void isolateEntryPoint(SendPort sendPort) {
> > > > final sem = NativeSemaphore(identifier: name);
> > > >
> > > > if (!sem.lock()) {
> > > > throw Exception("Failed to lock semaphore in isolate $isolate");
> > > > }
> > > >
> > > > // Perform work here...i.e. a random blocking operation
> > > > sleep(Duration(milliseconds: Random().nextInt(500)));
> > > >
> > > > if (!sem.unlock()) {
> > > > throw Exception("Failed to unlock semaphore in isolate $isolate");
> > > > }
> > > >
> > > > sendPort.send(true);
> > > > }
> > > >
> > > > final receivePort = ReceivePort();
> > > > await Isolate.spawn(isolateEntryPoint, receivePort.sendPort);
> > > > await receivePort.first;
> > > > //...
> > > >
> > > > // Cleanup
> > > > sem.dispose();
> > > > receivePort.close();
> > > >
> > > > }
> > > >
> > > > I also saw that there is a background parameter on the warmup
> > function.
> > > > i.e. warmup({bool background = false, bool upgrade = false}) and
I'm
> > > > assuming this is one of the cases where we need to have
interprocess
> > > locks
> > > > as to not call another conflicting command while that background
> > process
> > > is
> > > > running.
> > > >
> > > > Here is a link to the GitHub Repository for the native_semaphores
> > > > <https://github.com/open-runtime/native_semaphores/tree/main/test>
> > > >
> > > > If we need to have counters we can absolutely do this cross
process
> as
> > > > well.
> > > >
> > > > I'll keep studying and when I have a little more
> direction/constraint
> > > set
> > > > I'll be able to get it in very quickly (as in perhaps EOD
tomorrow)!
> > > >
> > > > —
> > > > Reply to this email directly, view it on GitHub
> > > > <
> > >
> >
>
#233 (reply in thread)>,
>
> >
> > >
> > > > or unsubscribe
> > > > <
> > >
> >
>
https://github.com/notifications/unsubscribe-auth/AAG32OGKKDMYJ7MT4WSTPM3Y5RZNBAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRTHE4TG>
>
> >
> > >
> > > > .
> > > > You are receiving this because you were mentioned.Message ID:
> > > > ***@***.***>
> > > >
> > >
> > > —
> > > Reply to this email directly, view it on GitHub
> > > <
> >
>
#233 (comment)>,
>
> >
> > > or unsubscribe
> > > <
> >
>
https://github.com/notifications/unsubscribe-auth/AYN4Z5IW6SIZUS4OTCEJ6DLY5R5VDAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRUGE3TE>
>
> >
> > > .
> > > You are receiving this because you were mentioned.Message ID:
> > > ***@***.***>
> > >
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <
>
#233 (comment)>,
>
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/AAG32OASQ3TT5V266PG74V3Y5R637AVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRUGI2DG>
>
> > .
> > You are receiving this because you were mentioned.Message ID:
> > ***@***.***>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <
#233 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AYN4Z5OZLJID75KN6FO3TFDY5SAWDAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRUGMZTC>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
—
Reply to this email directly, view it on GitHub
<#233 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG32OGY3NQU7E3A242QSY3Y5SCIZAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRUGQZDG>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Good evening from Melbourne Australia
…On Tue, 16 Apr 2024, 7:47 pm Tsavo Knott, ***@***.***> wrote:
Fantastic - good morning from Ohio!
Getting into it first thing this morning 🫡
—
Reply to this email directly, view it on GitHub
<#233 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG32OAHLRGGQYTMSZ65YTLY5TXTHAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRYGE3TS>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
@bsutton couple of quick things if you have a final moment this evening: Any chance you have specific unit tests that test to ensure things like No problem if not - I'd like to write some unit tests to simulate usage that would cause problems without a NamedLock implementation. I've seen the NamedLock unit test, but I'm looking for something (if existing) that tries to cause a race condition i.e. from the users perspective as they could run into in the wild. I guess we would have something like:
Other small things I'm wrapping my mind around: It looks to be the case that the only functionality that leverages named locks is Lastly, I just want to double check that NamedLock? __lock;
static const _lockName = 'script.lock';
NamedLock get _lock => __lock ??= NamedLock(suffix: _lockName, lockPath: pathToProjectRoot);
// ...
Future<void> warmup({bool background = false, bool upgrade = false}) async {
await _lock.withLock(
() async {
try {
if (background) {
//...
Future<void> clean() async {
await _lock.withLock(
() async {
// ...
But... it seems that unlike Future<void> _pubget() async {
await NamedLock(
suffix: _lockName,
lockPath: pathToProjectRoot,
)
//...
Future<void> _pubupgrade() async {
await NamedLock(
suffix: _lockName,
lockPath: pathToProjectRoot,
)
//...
Anything else that was notably important to continue to verify last time you were working on this would be legendary 🤝 I'll be getting after it today! |
Beta Was this translation helpful? Give feedback.
-
inline
On Tue, Apr 16, 2024 at 8:49 PM Tsavo Knott ***@***.***> wrote:
@bsutton <https://github.com/bsutton> couple of quick things if you have
a final moment this evening:
Any chance you have specific unit tests that test to ensure things like
warmup and clean are never conflicting cross process?
No, just the named lock tests themselves.
No problem if not - I'd like to write some unit tests to simulate usage
that would cause problems without a NamedLock implementation.
I've seen the NamedLock unit test, but I'm looking for something (if
existing) that tries to cause a race condition i.e. from the users
perspective as they could run into in the wild.
I think there is a thrash test in the NamedLock unit test which attempts to
do this.
I guess we would have something like:
1. Run Warmup, Clean in parallel and expect proper completion and
sequencing of both
2. Run Warmup, and Warmup with Upgrade at the same time on a given
script and expect proper completion and sequencing of both
3. Test the visa versa of the above as well
I'm not convinced we need to worry about warm up specifically, but rather
more generic NamedLock test.
Other small things I'm wrapping my mind around:
It looks to be the case that the only functionality that leverages named
locks is _pubget, _pubupgrade, clean, and warmup where warmup can start a
background subprocess of itself or from its' locked execution, calls
another locked sub execution _pubupgrade/_pubget.
The primary purpose for NamedLock is as part of our public api. I only use
it in warmup/clean as a convenience to make my coding easier.
Lastly, I just want to double check that clean and warmup run on the same
named _lock i.e.
NamedLock? __lock;
static const _lockName = 'script.lock';
NamedLock get _lock => __lock ??= NamedLock(suffix: _lockName, lockPath: pathToProjectRoot);
// ...
Future<void> warmup({bool background = false, bool upgrade = false}) async {
await _lock.withLock(
() async {
try {
if (background) {
//...
Future<void> clean() async {
await _lock.withLock(
() async {
// ...
But... it seems that unlike clean and warmup above the _pubget and
_pubupgrade methods (although not referencing the _lock instance
directly) appear to point to the same named lock as above based on suffix
and path. Was/is there a reason for this nuance that I should be aware of
about?
I don't remember, but it would probably make sense the lock out anything
that is updating the pubspec.lock file. There is a possibility that we are
duplicating code in the pub command is it likely puts its own lock on the
pubspec.lock file.
… Future<void> _pubget() async {
await NamedLock(
suffix: _lockName,
lockPath: pathToProjectRoot,
)
//...
Future<void> _pubupgrade() async {
await NamedLock(
suffix: _lockName,
lockPath: pathToProjectRoot,
)
//...
Anything else that was notably important to continue to verify last time
you were working on this would be legendary 🤝
I'll be getting after it today!
—
Reply to this email directly, view it on GitHub
<#233 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG32OE25UJOFF7PS63PQJ3Y5T6Z5AVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRYG4YTS>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Fantastic.
Are you able to fix this issue
#128
…On Wed, 17 Apr 2024, 12:16 pm Tsavo Knott, ***@***.***> wrote:
Good news! I have a good amount of unit tests that are passing for
NativeSemaphore.guard()
Here are some of the unit tests that ensure reentrant behavior and
cross-isolate coordination:
group('Testing Cross-Isolate Named Semaphore', () {
test('Several Isolates Accessing Same Named Semaphore, waiting random durations and then unlocking.', () async {
Future<bool> spawn_isolate(String name, int id) async {
// The entry point for the isolate
void isolate_entrypoint(SendPort sender) {
// Captures the call frame here, put right right inside the method entrypoint
CapturedCallFrame frame = CapturedCallFrame();
SemaphoreIdentity identity = SemaphoreIdentity(semaphore: name, frame: frame);
NativeSemaphore sem = NativeSemaphore(identity: identity);
// Lock
bool locked = sem.lock();
locked || (throw Exception("Lock in isolate $id should have succeeded"));
sleep(Duration(milliseconds: Random().nextInt(1000)));
// Unlock
bool unlocked = sem.unlock();
unlocked || (throw Exception("Unlock in isolate $id should have succeeded"));
// Dispose
bool disposed = sem.dispose();
disposed || (throw Exception("Dispose in isolate $id should have succeeded"));
sender.send(true);
}
// Create a receive port to get messages from the isolate
final ReceivePort receiver = ReceivePort();
// Spawn the isolate
await Isolate.spawn(isolate_entrypoint, receiver.sendPort);
// Wait for the isolate to send its message
return await receiver.first;
}
String name = '${safeIntId.getId()}_named_sem';
// Spawn the first helper isolate
final result_one = spawn_isolate(name, 1);
final result_two = spawn_isolate(name, 2);
final result_three = spawn_isolate(name, 3);
final result_four = spawn_isolate(name, 4);
// Wait for both isolates to complete their work
final outcomes = await Future.wait([result_one, result_two, result_three, result_four]);
expect(outcomes, everyElement(equals(true)));
});
});
group('Testing Reentrant Named Semaphore Behavior', () {
test('Several Isolates Accessing Same Named Semaphore, waiting random durations and then unlocking.', () async {
String name = '${safeIntId.getId()}_named_sem';
bool nested_lock() {
// Captures the call frame here, put right right inside the method entrypoint
CapturedCallFrame _frame = CapturedCallFrame();
SemaphoreIdentity _identity = SemaphoreIdentity(semaphore: name, frame: _frame);
NativeSemaphore _sem = NativeSemaphore(identity: _identity);
// Lock
bool locked = _sem.lock();
locked ||
(throw Exception(
"Nested Lock should have succeeded with true as it is the second lock and therefore already locked"));
sleep(Duration(milliseconds: Random().nextInt(1000)));
// Unlock
bool unlocked = _sem.unlock();
!unlocked ||
(throw Exception("Nested Unlock should have returned false as it will get unlocked on the upper level"));
// Dispose
bool disposed = _sem.dispose();
!disposed ||
(throw Exception("Nested Dispose should have returned false as it will get disposed on the upper level"));
return true;
}
// Captures the call frame here, put right right inside the method entrypoint
CapturedCallFrame frame = CapturedCallFrame();
SemaphoreIdentity identity = SemaphoreIdentity(semaphore: name, frame: frame);
NativeSemaphore sem = NativeSemaphore(identity: identity);
// Lock
bool locked = sem.lock();
locked || (throw Exception("Parent Lock should have succeeded"));
nested_lock();
// Unlock
bool unlocked = sem.unlock();
unlocked || (throw Exception("Parent Unlock should have succeeded"));
// Dispose
bool disposed = sem.dispose();
disposed || (throw Exception("Parent Dispose should have succeeded"));
});
test('Several Isolates Running Guard, waiting random durations and then unlocking.', () async {
String name = '${safeIntId.getId()}_named_sem';
bool nested_lock() {
// Captures the call frame here, put right right inside the method entrypoint
CapturedCallFrame _frame = CapturedCallFrame();
SemaphoreIdentity _identity = SemaphoreIdentity(semaphore: name, frame: _frame);
NativeSemaphore _sem = NativeSemaphore(identity: _identity);
// Lock
bool locked = _sem.lock();
locked ||
(throw Exception(
"Nested Lock should have succeeded with true as it is the second lock and therefore already locked"));
sleep(Duration(milliseconds: Random().nextInt(1000)));
// Unlock
bool unlocked = _sem.unlock();
!unlocked ||
(throw Exception("Nested Unlock should have returned false as it will get unlocked on the upper level"));
// Dispose
bool disposed = _sem.dispose();
!disposed ||
(throw Exception("Nested Dispose should have returned false as it will get disposed on the upper level"));
return true;
}
Future<bool> spawn_isolate(String name, int id) async {
// The entry point for the isolate
void isolate_entrypoint(SendPort sender) {
final NativeSemaphoreGuardedExecution<bool> returnable =
NativeSemaphore.guard(NativeSemaphoreGuardedExecution<bool>(
identifier: name,
callable: () {
sleep(Duration(milliseconds: Random().nextInt(100)));
nested_lock();
return true;
}));
sender.send(returnable.value);
}
// Create a receive port to get messages from the isolate
final ReceivePort receiver = ReceivePort();
// Spawn the isolate
await Isolate.spawn(isolate_entrypoint, receiver.sendPort);
// Wait for the isolate to send its message
return await receiver.first;
}
// Spawn the first helper isolate
final result_one = spawn_isolate(name, 1);
final result_two = spawn_isolate(name, 2);
final result_three = spawn_isolate(name, 3);
final result_four = spawn_isolate(name, 4);
// Wait for both isolates to complete their work
final outcomes = await Future.wait([result_one, result_two, result_three, result_four]);
expect(outcomes, everyElement(equals(true)));
});
});
On deck for tomorrow will be to simply update the internal contents of:
Future<void> withLock(
Future<void> Function() action, {
String? waiting,
})
and make sure some of the other properties are lined up and we will be
good to go!
I'll be sure to keep the NamedLock external properties and methods stable
as they have been and simply update the internals to leverage these
NativeNamed semaphores - I have updated the NativeNamed semaphores to
persist statuses i.e. lock counts etc at the file level and I just need to
do some small work around global cleanup i.e. on stray processes if in the
unlikely case that happens.
Looking forward to getting this in - day tomorrow is clear sailing so
should be finally G2G!
Feeling great about this ✅
—
Reply to this email directly, view it on GitHub
<#233 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG32ODKFP6LVCG72P63MA3Y5XLQNAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMZWGY2DM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Absolutely - That won't be an issues with the updates!
I'll try and get to some specific docker CI tests tomorrow but that should
no longer be a problem!
On Tue, Apr 16, 2024, 10:21 PM Brett Sutton ***@***.***>
wrote:
… Fantastic.
Are you able to fix this issue
#128
On Wed, 17 Apr 2024, 12:16 pm Tsavo Knott, ***@***.***> wrote:
> Good news! I have a good amount of unit tests that are passing for
> NativeSemaphore.guard()
>
> Here are some of the unit tests that ensure reentrant behavior and
> cross-isolate coordination:
>
> group('Testing Cross-Isolate Named Semaphore', () {
> test('Several Isolates Accessing Same Named Semaphore, waiting random
durations and then unlocking.', () async {
> Future<bool> spawn_isolate(String name, int id) async {
> // The entry point for the isolate
> void isolate_entrypoint(SendPort sender) {
> // Captures the call frame here, put right right inside the method
entrypoint
> CapturedCallFrame frame = CapturedCallFrame();
> SemaphoreIdentity identity = SemaphoreIdentity(semaphore: name, frame:
frame);
> NativeSemaphore sem = NativeSemaphore(identity: identity);
>
> // Lock
> bool locked = sem.lock();
> locked || (throw Exception("Lock in isolate $id should have
succeeded"));
>
> sleep(Duration(milliseconds: Random().nextInt(1000)));
>
> // Unlock
> bool unlocked = sem.unlock();
> unlocked || (throw Exception("Unlock in isolate $id should have
succeeded"));
>
> // Dispose
> bool disposed = sem.dispose();
> disposed || (throw Exception("Dispose in isolate $id should have
succeeded"));
>
> sender.send(true);
> }
>
> // Create a receive port to get messages from the isolate
> final ReceivePort receiver = ReceivePort();
>
> // Spawn the isolate
> await Isolate.spawn(isolate_entrypoint, receiver.sendPort);
>
> // Wait for the isolate to send its message
> return await receiver.first;
> }
>
> String name = '${safeIntId.getId()}_named_sem';
>
> // Spawn the first helper isolate
> final result_one = spawn_isolate(name, 1);
> final result_two = spawn_isolate(name, 2);
> final result_three = spawn_isolate(name, 3);
> final result_four = spawn_isolate(name, 4);
>
> // Wait for both isolates to complete their work
> final outcomes = await Future.wait([result_one, result_two,
result_three, result_four]);
>
> expect(outcomes, everyElement(equals(true)));
> });
> });
>
> group('Testing Reentrant Named Semaphore Behavior', () {
> test('Several Isolates Accessing Same Named Semaphore, waiting random
durations and then unlocking.', () async {
> String name = '${safeIntId.getId()}_named_sem';
>
> bool nested_lock() {
> // Captures the call frame here, put right right inside the method
entrypoint
> CapturedCallFrame _frame = CapturedCallFrame();
> SemaphoreIdentity _identity = SemaphoreIdentity(semaphore: name, frame:
_frame);
> NativeSemaphore _sem = NativeSemaphore(identity: _identity);
>
> // Lock
> bool locked = _sem.lock();
> locked ||
> (throw Exception(
> "Nested Lock should have succeeded with true as it is the second lock
and therefore already locked"));
>
> sleep(Duration(milliseconds: Random().nextInt(1000)));
>
> // Unlock
> bool unlocked = _sem.unlock();
> !unlocked ||
> (throw Exception("Nested Unlock should have returned false as it will
get unlocked on the upper level"));
>
> // Dispose
> bool disposed = _sem.dispose();
> !disposed ||
> (throw Exception("Nested Dispose should have returned false as it will
get disposed on the upper level"));
>
> return true;
> }
>
> // Captures the call frame here, put right right inside the method
entrypoint
> CapturedCallFrame frame = CapturedCallFrame();
> SemaphoreIdentity identity = SemaphoreIdentity(semaphore: name, frame:
frame);
> NativeSemaphore sem = NativeSemaphore(identity: identity);
>
> // Lock
> bool locked = sem.lock();
> locked || (throw Exception("Parent Lock should have succeeded"));
>
> nested_lock();
>
> // Unlock
> bool unlocked = sem.unlock();
> unlocked || (throw Exception("Parent Unlock should have succeeded"));
>
> // Dispose
> bool disposed = sem.dispose();
> disposed || (throw Exception("Parent Dispose should have succeeded"));
> });
>
> test('Several Isolates Running Guard, waiting random durations and then
unlocking.', () async {
> String name = '${safeIntId.getId()}_named_sem';
>
> bool nested_lock() {
> // Captures the call frame here, put right right inside the method
entrypoint
> CapturedCallFrame _frame = CapturedCallFrame();
> SemaphoreIdentity _identity = SemaphoreIdentity(semaphore: name, frame:
_frame);
> NativeSemaphore _sem = NativeSemaphore(identity: _identity);
>
> // Lock
> bool locked = _sem.lock();
> locked ||
> (throw Exception(
> "Nested Lock should have succeeded with true as it is the second lock
and therefore already locked"));
>
> sleep(Duration(milliseconds: Random().nextInt(1000)));
>
> // Unlock
> bool unlocked = _sem.unlock();
> !unlocked ||
> (throw Exception("Nested Unlock should have returned false as it will
get unlocked on the upper level"));
>
> // Dispose
> bool disposed = _sem.dispose();
> !disposed ||
> (throw Exception("Nested Dispose should have returned false as it will
get disposed on the upper level"));
>
> return true;
> }
>
> Future<bool> spawn_isolate(String name, int id) async {
> // The entry point for the isolate
> void isolate_entrypoint(SendPort sender) {
> final NativeSemaphoreGuardedExecution<bool> returnable =
> NativeSemaphore.guard(NativeSemaphoreGuardedExecution<bool>(
> identifier: name,
> callable: () {
> sleep(Duration(milliseconds: Random().nextInt(100)));
> nested_lock();
> return true;
> }));
>
> sender.send(returnable.value);
> }
>
> // Create a receive port to get messages from the isolate
> final ReceivePort receiver = ReceivePort();
>
> // Spawn the isolate
> await Isolate.spawn(isolate_entrypoint, receiver.sendPort);
>
> // Wait for the isolate to send its message
> return await receiver.first;
> }
>
> // Spawn the first helper isolate
> final result_one = spawn_isolate(name, 1);
> final result_two = spawn_isolate(name, 2);
> final result_three = spawn_isolate(name, 3);
> final result_four = spawn_isolate(name, 4);
>
> // Wait for both isolates to complete their work
> final outcomes = await Future.wait([result_one, result_two,
result_three, result_four]);
>
> expect(outcomes, everyElement(equals(true)));
> });
> });
>
>
> On deck for tomorrow will be to simply update the internal contents of:
>
> Future<void> withLock(
> Future<void> Function() action, {
> String? waiting,
> })
>
> and make sure some of the other properties are lined up and we will be
> good to go!
>
> I'll be sure to keep the NamedLock external properties and methods
stable
> as they have been and simply update the internals to leverage these
> NativeNamed semaphores - I have updated the NativeNamed semaphores to
> persist statuses i.e. lock counts etc at the file level and I just need
to
> do some small work around global cleanup i.e. on stray processes if in
the
> unlikely case that happens.
>
> Looking forward to getting this in - day tomorrow is clear sailing so
> should be finally G2G!
>
> Feeling great about this ✅
>
> —
> Reply to this email directly, view it on GitHub
> <
#233 (reply in thread)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AAG32ODKFP6LVCG72P63MA3Y5XLQNAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMZWGY2DM>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
—
Reply to this email directly, view it on GitHub
<#233 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AYN4Z5OGX5ZP5H3TOOROA43Y5XMB3AVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMZWGY3DQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Throwing this here (haven't looked into it as I'm wrapping up NamedLock refactor) but this is one of the last errors we're getting on windows: Let me know if anything comes to mind! Will hopefully take a look at this EOD! Looks like an issue when calling Leaving the GH Action Run |
Beta Was this translation helpful? Give feedback.
-
@tsavo-at-pieces I've just pushed a fixed for the windows registry issue. |
Beta Was this translation helpful? Give feedback.
-
Hey @bsutton -- apologies for the delay on all of this. I got really sick the past 2 weeks and then had a bunch of stuff going on across the rest of the team. Nevertheless - I am back on this and have reentrant named locks g2g for UNIX it seems - see example unit test here Working to layer this into my DCLI branch and test now! Should have a PR up soon. The API feel stable now - just have to do some adjustments for Windows and ensure its running properly but we are looking good. I'll pull your latest down and try to get this PR wrapped up ASAP. Good news is I have a very clear day tomorrow and Thursday to get this over the finish line. Fingers crossed & thank you again for the patience. Apologies I was down for the count brotha 😭 |
Beta Was this translation helpful? Give feedback.
-
No worries, thanks for the update!
I'm working on halfpipe to replace the pipe logic in dcli. Making good
progress.
…On Wed, 1 May 2024, 8:42 am Tsavo Knott, ***@***.***> wrote:
Hey @bsutton <https://github.com/bsutton> -- apologies for the delay on
all of this. I got really sick the past 2 weeks and then had a bunch of
stuff going on across the rest of the team.
Nevertheless - I am back on this and have reentrant named locks g2g for
UNIX it seems - see example unit test here
<https://github.com/open-runtime/named_locks/blob/8ce9d92724ffbd1814cef2215680b98756e961de/test/named_lock_test.dart>
Working to layer this into my DCLI branch and test now! Should have a PR
up soon. The API feel stable now - just have to do some adjustments for
Windows and ensure its running properly but we are looking good.
I'll pull your latest down and try to get this PR wrapped up ASAP. Good
news is I have a very clear day tomorrow and Thursday to get this over the
finish line. Fingers crossed & thank you again for the patience. Apologies
I was down for the count brotha 😭
—
Reply to this email directly, view it on GitHub
<#233 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAG32OFRQX4XOY6RMLKO4TTZAAM4FAVCNFSM6AAAAABELAW3CGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TEOBQGQYTM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
We are targeting Wednesday for a 4.0 release. Still a few hanging issues but nothing critical. |
Beta Was this translation helpful? Give feedback.
-
DCLI 4.0.0 has been released ! It's not perfect but it's good enough for a first release post waitFor. Thanks for everyone's input and in particular to @tsavo-at-pieces and @mraleph who made significant contributions to to this released. It wouldn't have happened without their help. As always if you find any problems please raise an new issue.. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Edit by bsutton the DCLI maintainer.
I've added a wiki page on outstanding issues. If you can pick one of these up let me know and I will mark it as in progress.
https://github.com/onepub-dev/dcli/wiki
Original question:
Would love to know how "stable" the core functionality of 4.x Alpha is and if it is too early to adopt/what we can do to help contribute & or create a migration guide @bsutton!
Absolutely huge fan of this entire project and I'd love to jump in where I can. Primarily working on server-side AOT GRPC engines and using DCLI to manage all types of things from automated CI/CD to DevEx @ pieces.app 🙌
Beta Was this translation helpful? Give feedback.
All reactions