dcli 2.0 ideas and problems.

[bsutton]

I'm currently working on dcli 2.0.

There are several objectives with 2.0 which will include breaking changes to the API which I'm hoping we can do without causing too much pain.

1. create Async version of the core libraries.
   dcli can currently only be used in console apps because of the user of waitFor. We have had a fair no of request to make key features of the API available to non console apps as well as simply the ability to make calls asynchronously.

With this in mind I'm creating a separate package called dcli_async. This package will contain most of the dcli code. The current dcli library will be gutted and make calls into the Async library. The current dcli API will remain intact with the exceptions noted below.

The intent is to start this process within the existing dcli 1.x project as much of this work can be done without breaking the current api.

2. improve the async nature of synchronous methods.
   So this sounds like oxymoron but let me explain.

One of the most used functions in DCli is the find function.

find('*', 'workingDirectory: '\', recursive: true ).forEach((line) => print);

Today in DCli 1.x the above command is going to crash your dart script.
Essentially the find command immediately starts scanning your disk looking for every single file.
Internally the find command adds each file to a stream.
The problem is that because find is a synchronous call it must complete a full scan of your disk before it returns and only then does it start calling forEach for each file it found.
The result is that the internal stream grows so large that you script runs out of memory and crashes.

The primary DCli package however must by synchronous so how do we fix this problem.

In 1.x I experimented with allowing the user to pass in a progress:

find('*', 'workingDirectory: '\', recursive: true, progress: (line) => print);

This sort of worked but made for inconsistencies as the find command must also return a progress. In the above example we return the same progress that the user passed in.
The issue with this is that the user may try to use the returned progress and that doesn't work as the progress is already 'consumed' when it returns.

So what is the solution.
The idea that I'm experimenting with is to change the find command (and all similar commands) to just 'prime the pump' in that it will create and return a new class FindProgress instance. The FindProgress class will contain all of the argument that you pass to the find command.
The find command won't actually start the disk scan.

Instead when you call one of the methods on the FindProgress the scan will start.

final progress = find('*', 'workingDirectory: '\', recursive: true );
progress.forEach((line) => print);

Of course this can be simplified to:

find('*', 'workingDirectory: '\', recursive: true ).forEach((line) => print);

Which is the original form of the command. So this changes is non-breaking and now the find command will actually complete without causing your script to run out of memory. (I have a working version of this and have scanned my entire PC for the first time).

This design pattern may however cause some issues with other apis and we need to consider how we go about changing these. There is also the cli_script issue to consider - more later

3. inclusion of cli_script
   [email protected] has written the rather excellent cli_script package and I've had discussion with Natalie about including cli_script into DCli. cli_scripts brings some really nice features such as piping between scripts and dart code blocks.
   To fully support the piping concept we should really look at changing the api to functions such as 'find', 'tail', 'head', 'cat' etc.
   There are some impedance mismatches between the two libraries that will need to be considered.

   • cli_scripts will output errors to the console that occur during piping.
     • this breaks the implied contract that dcli won't output to the console unless explicitly asked to. This also doesn't work for non-console apps which we are looking to support with the dcli_async package.
   • The argument passing method when calling external apps is different. We take a string or an array, where as cli_scripts allows both.

4. review and remove inconsistencies in the api.
   TODO: note what these are

5. review the usage of Progress
   This is probably one of the most used/useful parts of DCli but as per 2) it is broken and it also encourages users to attempt to use it consistently.
   It is also inconsistent with a clean integration with cli_script piping.

6. Review the collection of withXXX functions
   We have functions such as withTempFile and withTempDir.
   The cli_script project has withEnv.

We also have been experimenting with withOpenFile and withOpenLineFile.

I've built these as possible replacements to the 'string_as_process'.write, append, truncate methods.
Should we have both (the string over loads) and the 'withXXX' options?
The 'withXXX' options provide better performance (as we hold the file open).

I'm concerned the withOpenXXFile might be too complex. We shouldn't be trying to replace the existing dart file code. We should only provide additional functionality if it brings it in line with the 'sync by default' nature of dcli and if we greatly simplify the 90% of common use cases that we are targeting.

There are also questions about whether we need to consider the cli_script pipes.

6. TODO what other objectives do we have?

[bsutton]

Review all static methods to see if we should move them to members

e.g.
Ansi.strip -> Ansi().strip

The main driver is making unit testing easier.

[bsutton]

Should be able to use .parser to read contents of file. This should fit in with the concept of piping code and file io not just executable output.

[bsutton]

The cli_script package capture method outputs to the cli when an error occurs.
This breaks the dcli contract of not outputting to the console unless the user explicitly requests it.

[samuelstroschein]

create Async version of the core libraries.
dcli can currently only be used in console apps because of the user of waitFor. We have had a fairy no of request to make key lovers of the API available to non console apps as well as simply the ability to make calls asynchronously.

When I played around with DCLI some months ago, not being able to call/use async functions was a big turn-off. A thumbs up from me with a follow-up question: Why separate the packages though?

I maybe found the following explanation(s) in the documentation:

They aim of DCli is to create a Bash like simplicity cli apps. Futures are great and all but they do make the code more complex and harder to read.

Coming from Python, JavaScript, and Dart itself Futures/Promises/Async are the norm.

Futures also can make your scripts a little dangerous. If you copy a file and then want to append to the copied file, you had better be certain that the copy command has completed before you start the append.

Yes, everyone will most likely once in their programming career run into this problem. A lint rule activated in DCLI environments could completely partially solve this problem -> "You are not awaiting this Future. Are you sure?" with an potential ignore comment above the line,

DCli's global functions remove those complications.

Vice-versa for me (and likely others). It adds complexity because I had to look into the documentation for DCLI specific functions as standard Dart (async) function where not working.

---

Are the reasons above are the predominant reasons for splitting the packages?

[bsutton]

When I played around with DCLI some months ago, not being able to call/use async functions was a big turn-off. A thumbs up from me with a follow-up question: Why separate the packages though?

DCli doesn't stop you using async functions. You can use any dart code/packages that work in a cli app without limitations. What led you to the conclusion that you couldn't use async functions? I ask because it sounds like we have a problem with the documentation.

Yes, everyone will most likely once in their programming career run into this problem. A lint rule activated in DCLI environments

Currently there isn't a lint rule that covers warning a user if they call an async function from a sync method. I've raised this issue to have that problem addressed: https://github.com/dart-lang/linter/issues/2953 If you like the idea give it a thumbs up.

Vice-versa for me (and likely others). It adds complexity because I had to look into the documentation for DCLI specific functions as standard Dart (async) function where not working.

This is unlikely to be a DCli issue as dcli uses async functions all over the place (as does my own code which uses DCli) . If you have a reproducible problem please raise an issue so we can have a look at it. Of course it could be the 'fairies' I referred to causing the problem :) The reason to split the package is that dcli can only be used in console apps. There are many functions in dcli which are useful in other domains. By splitting out the libraries then desktop and flutter users can use functions like copy, move, find etc. S. Brett Sutton Noojee Contact Solutions 03 8320 8100

…

On Tue, 26 Oct 2021 at 09:28, Samuel Stroschein ***@***.***> wrote: *create Async version of the core libraries.* dcli can currently only be used in console apps because of the user of waitFor. We have had a fairy no of request to make key lovers of the API available to non console apps as well as simply the ability to make calls asynchronously. When I played around with DCLI some months ago, not being able to call/use async functions was a big turn-off. A thumbs up from me with a follow-up question: Why separate the packages though? I maybe found the following explanation(s) <https://dcli.noojee.dev/dart-basics/futures> in the documentation: They aim of DCli is to create a Bash like simplicity cli apps. Futures are great and all but they do make the code more complex and harder to read. Coming from Python, JavaScript, and Dart itself Futures/Promises/Async are the norm. Futures also can make your scripts a little dangerous. If you copy a file and then want to append to the copied file, you had better be certain that the copy command has completed before you start the append. Yes, everyone will most likely once in their programming career run into this problem. A lint rule activated in DCLI environments could completely partially solve this problem -> "You are not awaiting this Future. Are you sure?" with an potential ignore comment above the line, DCli's global functions remove those complications. Vice-versa for me (and likely others). It adds complexity because I had to look into the documentation for DCLI specific functions as standard Dart (async) function where not working. ------------------------------ Are the reasons above are the predominant reasons for splitting the packages? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#159 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG32OBRD3SH3DI33DD5LX3UIXK2VANCNFSM5EP32YHQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.