pubspec: Add bin_dependencies or global_dependencies

[DartBot]

<img src="https://avatars.githubusercontent.com/u/444270?v=3" align="left" width="96" height="96"hspace="10"> Issue by seaneagan
Originally opened as dart-lang/sdk#22054

---

Many packages expose command-line apps as well as dart APIs to that same functionality. Often the command-line apps require additional dependencies such as args, unscripted, ansicolor, prompt, etc. Similar to how dev_dependencies allow you to specify dependencies only needed by package developers, bin_dependencies could allow specifying dependencies only needed by folks running command-line apps from the package.

For pub global activate / pub global run it would be easy, because those use a self-contained dependency graph.

But for pub run, it would require separating the dependencies into two separate graphs, e.g. packages/pubspec.lock and bin_packages/pubspec_bin.lock.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/4865287?v=3" align="left" width="48" height="48"hspace="10"> Comment by lrhn

---

Added Area-Pub, Triaged labels.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

---

When we were first designing the package system, we intentionally decided to limit the number of different ways a dependency could be declared to encourage both conceptual clarity and straightforward implementation. I think that principle applies here: there isn't enough benefit from not having some dependencies downloaded under some circumstances to justify the complexity cost.

---

Added NotPlanned label.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/444270?v=3" align="left" width="48" height="48"hspace="10"> Comment by seaneagan

---

Example of why this would be useful:

https://github.com/seaneagan/which.dart/issues/3

[DartBot]

<img src="https://avatars.githubusercontent.com/u/444270?v=3" align="left" width="48" height="48"hspace="10"> Comment by seaneagan

---

Version lock is probably the bigger concern here, not downloading extra dependencies. This bit me again, causing me to have to put my executable and library in separate packages:

http://github.com/seaneagan/den
http://github.com/seaneagan/den_api

And it just came up again here:

https://chromiumcodereview.appspot.com/1022963002/

[DartBot]

<img src="https://avatars.githubusercontent.com/u/1081711?v=3" align="left" width="48" height="48"hspace="10"> Comment by jmesserly

---

cc @munificent.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/444270?v=3" align="left" width="48" height="48"hspace="10"> Comment by seaneagan

---

I think this would get much easier to implement once we have the package spec:

https://github.com/lrhn/dep-pkgspec

Then all the deps can still be downloaded into the same place, but pub run (when running a script from another package's 'bin') and pub global run just use a different package spec which includes bin_dependencies.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

---

I'm going to re-open this because I'm increasingly hearing this is a problem and I don't want people to solve it by splitting their packages into pure lib / command line halves. Totally agree that package-spec will help here.

---

Added Triaged label.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/67586?v=3" align="left" width="48" height="48"hspace="10"> Comment by pq

---

Yes please! This is something we're tangling significantly with in analyzer-land.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

---

A package's executables are part of its public API; it's valid for one package to depend on another because it wants to run that dependency's executable. If that executable's dependencies are distinct from its normal dependencies, that use-case breaks.

This is analogous to a package exposing two classes, one of which has a dependency that the other does not. If a downstream package only uses one class, sure, it would be nice for it not to use the other class's dependency, but that's not how our dependency system works, and for good reason: if every package has to spell out exactly which dependencies are used by which pieces of its API, and which pieces of the APIs of its dependencies it uses, the whole system becomes untenably complex.

And just like the multiple-classes case, if the dependencies or versioning story of two pieces of an API are different enough that they're causing problems, I don't think there's anything wrong with splitting them apart. We're planning on doing this with unittest, for example: there will be one "frontend" package for defining tests and a separate "backend" package for manipulating the infrastructure behind the definition, separated out so they can be versioned separately.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/444270?v=3" align="left" width="48" height="48"hspace="10"> Comment by seaneagan

---

A package's executables are part of its public API; it's valid for one package to depend on another because it wants to run that dependency's executable.

A package shouldn't depend on a package solely for its executable, that's what pub global activate is for.

If that executable's dependencies are distinct from its normal dependencies, that use-case breaks.

I think "breaks" is too strong. As mentioned in the original post, yes, it's harder for pub run, but you didn't address my proposal to build two separate package graphs, one which excludes bin_dependencies, and one which includes them.

This is analogous to a package exposing two classes, one of which has a dependency that the other does not. If a downstream package only uses one class, sure, it would be nice for it not to use the other class's dependency, but that's not how our dependency system works, and for good reason: if every package has to spell out exactly which dependencies are used by which pieces of its API, and which pieces of the APIs of its dependencies it uses, the whole system becomes untenably complex.

I don't think anyone has or would suggest such a complex partitioning of dependencies. A better analogy is with dev_dependencies. Those are useful because for example packages want to co-locate their tests, build scripts, etc. with their public API, using the standard pub package layout conventions (/lib, /test, /tool), but they don't want dependers to inherit e.g. unittest, grinder, etc. dependencies. Similarly, packages want to co-locate one cohesive public API including an executable and a dart API, using the standard pub package layout conventions (/lib, /bin) without having dependers inherit e.g. args, prompt, and ansicolor.

And just like the multiple-classes case, if the dependencies or versioning story of two pieces of an API are different enough that they're causing problems, I don't think there's anything wrong with splitting them apart. We're planning on doing this with unittest, for example: there will be one "frontend" package for defining tests and a separate "backend" package for manipulating the infrastructure behind the definition, separated out so they can be versioned separately.

That sounds like a good solution, but to a problem that's specific to the unittest package. Exposing executables however is a problem shared by a large percentage of packages. It's already partly solved by the /bin convention, pub run, etc. But partitioning bin dependencies is still a missing piece.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

---

A package's executables are part of its public API; it's valid for one package to depend on another because it wants to run that dependency's executable.

A package shouldn't depend on a package solely for its executable, that's what pub global activate is for.

Not true. It's helpful to depend on packages that you only use executables from because it ensures everyone hacking on your package can easily run the right version of those executables. You can just tell people:

1. Clone the package.
2. Run pub get.
3. Run pub run some_dep:some_exe

However, I'm not opposed to something like bin_dependencies, though it can be a tricky to implement correctly.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/188?v=3" align="left" width="48" height="48"hspace="10"> Comment by nex3

---

A package shouldn't depend on a package solely for its executable, that's what pub global activate is for.

This is incorrect. "pub global activate" is for end-users to install executables onto their systems for their direct use. If one package needs to use another's executable, it absolutely should depend on that package; making its users run "pub global activate" is an annoyingly manual installation step that interferes with the user's system state and doesn't respect version constraints.

I want to emphasize again that we consider a package's executables to be part of its API, and this model is not likely to change. Even if we do decide to add some notion of bin dependencies, it will be in a way that preserves works with this model, rather than against it.

you didn't address my proposal to build two separate package graphs, one which
excludes bin_dependencies, and one which includes them.

How would this help? The bin-inclusive package graph would be identical to package graphs today, and if it experiences version lock, installation would still fail.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/405837?v=3" align="left" width="48" height="48"hspace="10"> Comment by zoechi

---

I'm fighting a lot with package dependencies lately and every single dependency adds more headaches. Here its with examples https://bitbucket.org/andersmholmgren/shelf_auth/issue/6/please-make-it-compatible-with-shelf-060 (see end of the discussion)
I think the dependencies should be at least be split for use as dependency in another package and everything else (test, bin, example, tool)

[DartBot]

<img src="https://avatars.githubusercontent.com/u/46275?v=3" align="left" width="48" height="48"hspace="10"> Comment by munificent

---

I think the dependencies should be at least be split for use as dependency in another package and everything else (test, bin, example, tool)

This is exactly what dev_dependencies do, except that bin shouldn't be in that list. But for things in test, example, and tool, by all means do use dev_dependencies for those.

[DartBot]

<img src="https://avatars.githubusercontent.com/u/444270?v=3" align="left" width="48" height="48"hspace="10"> Comment by seaneagan

---

It's helpful to depend on packages that you only use executables from because it ensures everyone hacking on your package can easily run the right version of those executables.

I agree that that's a nice to have, but if folks are experiencing version lock they won't even get past the install, and thus won't be able to run the executables at all. This is not a problem with pub global activate, which leads me to my proposal!

Instead of trying to jam in the dependencies of all the binaries of all your transitive dependencies into a single dependency graph, similar to pub global activate, each of your (immediate) dependencies binaries deserve their own dependency graph. Consider that a large percentage of the time, a depender doesn't want or need to call the binaries of its dependencies, so it should only have to bear the cost in cases where it does. The cost I propose is that in the workflow you mentioned:

You can just tell people:

1. Clone the package.
2. Run pub get.
3. Run pub run some_dep:some_exe

We add a step between 2. and 3., which is to run a new command:

  pub activate some_dep

This is just like:

  pub global activate some_dep <some version or path or git repo>

except that you never need to specify <some version or path or git repo> since that is already done for you by the pubspec. It only uses the dependencies of some_dep, including its bin_dependencies, and thus some_dep is allowed to make different choices about the dependencies of its binaries than the local package that depends on it, and from binaries of any other dependencies. Also, bin_dependencies of any transitive dependencies never even come into play. This also increases the reliability and testability of the binaries, since the dependency graphs are the same regardless of whether they are activated globally or in the context of a local package.

If there is desire to easily activate all the binaries of one's immediate dependencies that could just be a pub activate command (without an argument), but I'm not sure if that's necessary.

The lock files of the pub activated binaries could be stored in a sub-directory of the .pub directory. There shouldn't be any need to check them in, since checking in the main pubspec.lock would already pin you to a specific version of the binary. Presumably the "package spec" files for each binary (once they are a thing) would co-located there.

Then when pub get or pub upgrade are later run again, there would be two options:

1. Mark the pub activated packages as dirty, so that later when they are again pub run, an error message tells the user that they need to re-pub activate the dependency.
2. Automatically re-pub activate any already activated packages for the user.

I personally like 2. a lot.

And pub activate can be added in a backwards compatible way. When pub run some_dep:some_exe is executed, it should check if some_dep has been pub_activated, and if not, it falls back to the existing behavior of using the local package's dependency graph. But this would be deprecated, and in dart 2.0 running pub activate before pub run will be required.

What do you all think?

[DartBot]

<img src="https://avatars.githubusercontent.com/u/1081711?v=3" align="left" width="48" height="48"hspace="10"> Comment by jmesserly

---

fyi, proposal doc: https://docs.google.com/document/d/1lvbzyivCejhTuHLG4WI3YVThbQqJhPUrwajwTC-sXWU/edit

looks great to me

[DartBot]

<img src="https://avatars.githubusercontent.com/u/444270?v=3" align="left" width="48" height="48"hspace="10"> Comment by seaneagan

---

Another example:

https://github.com/dart-lang/analyzer_cli

[DartBot]

<img src="https://avatars.githubusercontent.com/u/444270?v=3" align="left" width="48" height="48"hspace="10"> Comment by seaneagan

---

Another package:args conflict in issue dart-lang/sdk#23349.

[harryterkelsen]

This would be very useful for the dart2js_info package here: https://github.com/dart-lang/dart2js_info

The package basically consists of a library that reads/writes a dump-info file and a set of binaries that does useful analyses on the dump-info file. This package is bundled with the SDK since dart2js uses it, but dart2js only uses the 'library' aspect of it. I added a dependency in one of the binaries and now have to add that dependency to the SDK, even though the SDK will never use a binary in dart2js_info.

Since we have package specs now, isn't it not really true that a library's binaries are part of it's public API?

[nex3]

Since we have package specs now, isn't it not really true that a library's binaries are part of it's public API?

That doesn't really affect it one way or another.

[srawlins]

Oops! I just specced out another proposal at dart-lang/sdk#34837, which allows each executable to specify its own specs.

As for the idea that an entire package's contents are part of its public API, yeah that's tricky. Technically, today, a dependant could even reach out and load a test file, or read a file in tool/ or data/, since pub get downloads all files of a package that were published, not just lib/.

[jakemac53]

As for the idea that an entire package's contents are part of its public API, yeah that's tricky. Technically, today, a dependant could even reach out and load a test file, or read a file in tool/ or data/, since pub get downloads all files of a package that were published, not just lib/.

Note that you can't access anything outside of lib with a package: uri though (the ../ ends up replacing the package name and you end up with an absolute uri that doesn't work).

Sure you can put an absolute uri import to a non-lib dir in your pubcache but that does not need to continue to work imo.

[srawlins]

I'm curious about whether we can specify "public API" a little more tightly. Here's our current text on Public tools in package layout:

Dart scripts placed inside of the bin directory are public. Any package that depends on your package can run scripts from your package’s bin directory using pub run. Any package can run scripts from your package’s bin directory using pub global.

However, as @jakemac53 writes above, "you can't access anything outside of lib with a package: uri", so... what we're really saying here is that the developers of a downstream package, which depends on your package, can run scripts from your package's bin/ directory using pub run. So it seems to me you could define scripts in bin/ as being "public" for direct dependents, but "not public" in any other scenario, including for indirect dependents.

I think this means we could lay out an "executable dependencies" API (like mine 😄 dart-lang/sdk#34837), wherein pub should fetch all executable dependencies for each dev_dependency, so that a developer could run tools like grinder, test, etc.

One wrench in this plan would be packages like mockito, which both depends on test for its API (it imports package:test/test in lib/ code), and for its own tests, where I need to run pub run test. Hmm...

[greglittlefield-wf]

We sometimes hit version lock due to incompatible transitive dependencies, and a good portion of the time it's due to packages that declare executables.

Some packages like analyzer bump major versions frequently, and other packages like ansicolor and args bumping major versions to support Dart 2 have been problematic recently.

Here's another idea, similar to @seanegan's proposal above that would solve parts of my problem as well as parts of the problem discussed above. It also maintains that a package's executables are part of its API, and keeps them as dependencies.

• In pubspec.yaml, have a separate section (say, dev_executable_dependencies) to declare dependencies on packages whose executables (and not Dart APIs) you wish to use
  • Allow running these executables via pub run <package_name>, just as you can with existing dependencies
  • Don't allow consumption of those packages' Dart APIs
• Resolve each of those dependencies in its own isolated graph, similar to pub global activate, to prevent version conflicts between transitive dependencies
• Write these resolved dependencies to a separate executables-specific lockfile that can be checked in, to ensure contributors to a package are all using the same versions of executables and their dependencies.

Example:

# pubspec.yaml
dev_executable_dependencies:
  a: ^1.1.1
  b: ^2.2.2

# pubspec_executables.lock
executables:
  a:
    version: "1.0.0"
    packages:
      c:
        version: "1.0.0" # incompatible with package b's constraints, but that's
                         # okay since the dependency graphs are isolated
      d:
        version: "4.1.0"
  b:
    version: "1.0.0"
    packages:
      c:
        version: "2.0.0" # incompatible with package a's constraints, but that's
                         # okay since the dependency graphs are isolated
      d:
        version: "4.2.0"

There are some flaws with this approach, though.

This won't help for packages that expose both Dart APIs and executables. We would have to rely on those maintainers splitting out their APIs and executables into separate packages.

It might be difficult in practice to ensure compatibility between split up API/executable packages whose behavior is dependent on a transitive dependency. Say, for the example above, we want to ensure a and b resolve to the same version of d. We would likely have to come up with a way to constrain transitive dependencies to ensure compatibility. Declaring that constraint might get messy, especially if it falls on the consuming package to do so.

Also, having a separate lockfile wouldn't be ideal.

Anyways, I was hoping that sharing an idea for a different approach to this problem might spark some better ideas 😄.

[sigurdm]

@jonasfj has an alternative proposal called "private dependencies" that should cover this use case as well.

Could you open an issue with a rough description, and close this one in favour of the other?