Feature Request: remove need for multiple `spago.dhall` config files by defining output targets in `spago.dhall`

[JordanMartinez]

In an attempt to solve #416, I described in this comment the idea of "output targets" (since target already refers to the output file one produces when bundling an app).

Problem

If you want to add "developer dependencies," "test dependencies," or do a mono-repo, you have to define one spago.dhall config file for each output target. Hence, one may find a spago.dhall file, a test.dhall file, an examples.dhall file, and sometimes a benchmark.dhall file.

Rather than duplicating our work, this feature would define all of those targets within a single spago.dhall file.

Below is an example of what spago.dhall would look like after this change:

{-
Welcome to a Spago project!
You can edit this file as you like.
-}
let packages = ./packages.dhall
let main = { packages = packages
           , dependencies = [ "dep1" ,"dep2" ]
           , sources = [ "src/**/*.purs" ]
           }
let test = { packages = packages
           , dependencies = main.dependencies # [ "spec" ]
           , sources = main.sources # [ "test/**/*.purs" ]
           }
-- let otherTarget = --
in { name = "my-project"
   , outputs =
      { main = main
      , test = test
  --  , otherTarget = otherTarget
      }
   }

Unanswered Questions

1. When would we want to implement this change and how would we handle the transition process since it will be a big breaking change?
2. How should publishing change in light of this? Could we publish multiple projects from the same repo? (e.g. Halogen Hooks and it's corresponding testing library)
3. What would the command line argument look like?

For 3, we could "desugar" commands:

• spago build -> spago --output main build
• spago run -> spago -o main run
• spago test -> spago -o test run

I don't think this feature should remove the -x config.dhall part. Both features could exist at the same time.

[JordanMartinez]

I spent some time looking at this issue again.

I don't think a target should have a packages field. There's two reasons for this:

1. I can't think of a situation where one would want to use a different package set between two targets. Perhaps it might be done if one had an implementation for one backend and another for another backend, but I think that should stay outside the scope of this feature for the time being.
2. spago ls packages would need a target name to know which package set to grep (e.g. it "desugars" to spago ls packages --output targetName where targetName is likely main). I think a similar situation arises with verify and verify-set. If targets don't include a packages field, then it removes this issue, and I think we should keep this feature's scope smaller for the time being.

Specifying different spago.dhall File Formats

Single Target Format

Current spago.dhall content/format (i.e Single Target Format):

{-
Welcome to a Spago project!
You can edit this file as you like.
-}
let packages = ./packages.dhall
in { name = "my-project"
   , dependencies = [ "dep1" ,"dep2" ]
   , packages = packages
   , sources = [ "src/**/*.purs" ]
   }

Multi Target Format

Below is an example of what spago.dhall would look like after this change (i.e. Multi Target Format):

{-
Welcome to a Spago project!
You can edit this file as you like.
-}
let packages = ./packages.dhall
let main = { dependencies = [ "dep1" ,"dep2" ]
           , sources = [ "src/**/*.purs" ]
           }
let test = { dependencies = main.dependencies # [ "spec" ]
           , sources = main.sources # [ "test/**/*.purs" ]
           }
-- let otherTarget = --
in { name = "my-project"
   , packages = packages
   , outputs =
      { main = main
      , test = test
  --  , otherTarget = otherTarget
      }
   }

To reduce breakage, Spago could check to see whether the given spago.dhall file matches the Single Target Format and automatically migrate it to the Multi Target Format. If this fails, then Spago prints an error to the user and indicates it must be done manually. In other words, this spago.dhall file

in
   { name = "my-project"
   , dependencies = [ "dep1" ,"dep2" ]
   , packages = packages
   , sources = [ "src/**/*.purs", "test/**/*.purs" ]
   }

is migrated to this file:

let mainDeps = [ "dep1" ,"dep2" ]
in
   { name = "my-project"
   , outputs =
     { main = 
       { dependencies = mainDeps
       , sources = [ "src/**/*.purs" ]
       }
     , test =
       { dependencies= mainDeps
       , sources = [ "test/**/*.purs" ]
       }
     }
   , packages = packages
   }

The test dependencies will still need to be moved to the test target's dependency list, but all Spago commands should continue to work (see next section).

Target Naming Conventions and definitions

To help ease the breakage, all Spago commands that depend on the dependencies and sources fields in the Single Target Format should default to using a specific target name in the Multi Target Format. For example, if Spago is called with one of the below commands and an output is not specified, Spago should default to the following target names (where -o indicates the target name):

• spago install = spago install -o 'main'
• spago build = spago build -o 'main'
• spago docs = spago docs -o 'main'
• spago run = spago run -o 'main'
• spago bundle-module = spago bundle-module -o 'main'
• spago bundle-app = spago bundle-app -o 'main'
• spago search = spago search -o 'main'
• spago sources = spago sources -o 'main'
• spago ls deps = spago ls deps -o 'main'
• spago repl = spago repl -o 'main' (only if a spago.dhall file exists in the current working directory)
• spago bump-version = spago bump-version -o 'main'
• spago test = spago test -o 'test'

I believe I've covered every command here.

Due to the above "desugar to a default target name" requirement, when defining your spago.dhall file, I think Spago should make the following recommendations for target names and their definitions. However, to ensure we can support monorepos, I don't think the target names, main and test, should be required. If one is using a monorepo, they will likely have scripts that run spago build -o lib1-main and spago test -o lib1-test.

let packages = ./packages.dhall

{- `main` = what `spago.dhall` in the Single Target Format would refer to. -}
let main = { packages = packages
           , dependencies = [ "dep1" ,"dep2" ]
           , sources = [ "src/**/*.purs" ]
           }

{- `test` = what `test.dhall` in the Single Target Format typically refers to. -}
let test = { packages = packages
           , dependencies = main.dependencies # [ "spec" ]
           , sources = main.sources # [ "test/**/*.purs" ]
           }

{- `examples` = what `examples.dhall` in the Single Target Format typically refers to. -}
let examples = { packages = packages
               , dependencies = main.dependencies
               , sources = main.sources # [ "examples/**/*.purs" ]
               }

{- `benchmark` = what `benchmark.dhall` in the Single Target Format typically refers to. -}
let benchmark = { packages = packages
                , dependencies = main.dependencies # [ "minibench" ] {- benchotron is not in package set -}
                , sources = main.sources # [ "benchmark/**/*.purs" ]
                }
in
  { outputs =
    { main = main
    , test = test
    , examples = examples
    , benchmark = benchmark
    }
  }

The Publication Question

1. How should publishing change in light of this? Could we publish multiple projects from the same repo? (e.g. Halogen Hooks and it's corresponding testing library)

Since all tooling expects a library to published via a single repo, I don't think we should worry about this. As the Registry gets figured out more, we can reconsider this in the future. In other words, this feature does not mean we can publish multiple packages via a single repo. Rather, it enables production code to use a single spago.dhall file in a larger monorepo that isn't being published anywhere but simply used to build code.

[f-f]

Thanks for taking a look at this @JordanMartinez! I have a few thoughts on this (that I hope won't shuffle much the implementation work you already started) 😊

I originally envisioned this to be part of the config format redesign that will happen once the Registry ships (as I describe better in #766 (comment)).
This is because adding this will likely mean supporting yet another way that a config could be different, and might mean more trouble when we do the migration to the new config format.
I guess we could also ship targets now, but I might recruit you to help with the migration to the new config 😄

I can't think of a situation where one would want to use a different package set between two targets. Perhaps it might be done if one had an implementation for one backend and another for another backend

This is exactly the usecase that I had in mind with the packages key there. I postulate that today it is uncommon because

1. using alternate backends is not an exceptionally smooth experience
2. using an alternate backend while sharing code with the JS frontend is even harder

So the idea behind this was to make this process smoother.
I do agree though that implementing this is not strictly necessary at this stage.

spago ls packages --output targetName

I am not sure what's the rationale behind using the output flag here, but:

1. the compiler already has an output flag and it means a totally different thing
2. we want to support that same flag here in Spago (and it's low hanging fruit these days), see bundle-* commands don't support the purs --output option #216

I'd like to use -t and --target for the flag - if the -t shorthand is taken we can most likely make space for it (i.e. move the flag that uses it to only take the long version, freeing the shorthand)

Similarly, I think we should use targets in the config, rather than output.

this spago.dhall file ... is migrated to this file:

I wouldn't like us to migrate user files until we do the Final Migration Once The Registry Ships:tm:, as I feel that's going to add so much code in order to support multiple migrations, etc.
In the case of this feature it's entirely possible to just read the single-target format and convert it to the multi-target format (with a single target) in memory (maybe even warning the user about the existence of multitargets!)
We already do this kind of migration for old configs (so that users don't have to change their config files right away), see here:

spago/src/Spago/Config.hs

Lines 362 to 371 in 7728ce4

	isConfigV1, isConfigV2 :: Dhall.Map.Map Text v -> Bool
	isConfigV1 (Set.fromList . Dhall.Map.keys -> configKeySet) =
	let configV1Keys = ["name", "dependencies", "packages"]
	in configKeySet == configV1Keys
	isConfigV2 (Set.fromList . Dhall.Map.keys -> configKeySet) =
	let configV2Keys = ["name", "dependencies", "packages", "sources"]
	optionalKeys = ["backend", "license", "repository"]
	in Set.difference configKeySet optionalKeys == configV2Keys

In other words, this feature does not mean we can publish multiple packages via a single repo. Rather, it enables production code to use a single spago.dhall file in a larger monorepo that isn't being published anywhere but simply used to build code.

This is fair, but something that we should definitely avoid would be to implement this in a way that we'd have to break later, once we want to support publishing of multiple packages from a single config.
I think so far this is all fine, but we have to be careful about this aspect all the time, or we might have unpleasant amounts of breakage in the future, and I wouldn't like that.

---

Other than the above concerns, this looks good. Implementation-wise it might be troublesome to patch the part that manipulates the raw Dhall AST, that is called in spago install $package. Let me know if you'd need input for that (or anything else really)

[JordanMartinez]

I decided to try tackling this while on vacation when I had a few spare moments. I didn't have access to the Internet all the time, so it made doing some of this harder. So, that's why I didn't respond immediately to your comment above.

Was your above comment meant to be added to my PR or to this issue?

Also, should I wait until the Registry is out before continue work on this? I get the concern for migrating spago.dhall files from one format to another, but I also know that this is a very nice feature to have now, not later.

I am not sure what's the rationale behind using the output flag here, but:

Originally, I was going to use --target/-t but -t was already used in the bundle-* commands. My PR now uses --target, but not -t. Also, it seems like the -t arg is passed to purs' corresponding -o arg, so have 'outputs' already been supported and the linked issue's comments are old? Or am I just misunderstanding something in that comment thread?

In the case of this feature it's entirely possible to just read the single-target format and convert it to the multi-target format (with a single target) in memory (maybe even warning the user about the existence of multitargets!)
We already do this kind of migration for old configs (so that users don't have to change their config files right away), see here:

Yeah, I've implemented that in my PR. Both v1 and v2 config formats can be migrated to the multi-target one. I think I did this after seeing your comment above.

Implementation-wise it might be troublesome to patch the part that manipulates the raw Dhall AST, that is called in spago install $package

That's been done, too. I update the dependencies based using this resolution method:

1. lookup targets.<targetName>
2. if it's a literal record, then do the following:
   1. lookup the 'dependencies' field
   2. if it's a [] (i.e. ListLit), then just add the new dependency there
   3. if it's a list1 # list2 # ... [] (i.e. one or more ListAppends), then update the right-most ListLit to include that new package
3. if it's a binding (e.g. 'main' in the Dhall expression, let main = { ... } .... in { ..., targets = { main } }), then do the following:
   1. recurse "up" the dhall expression until finding the corresponding binding name
   2. do the same steps as the literal record

This resolution process covers all of these possibilities:

-- literal target with no dependencies on earlier targets
{ targets = 
  { main = 
    { dependencies = ["foo"]
    , sources = ["src/**/*.purs"] 
    } 
  } 
}

-- literal target with a dependency on earlier target
let dep = 
    { dependencies = ["foo"]
    , sources = ["src/**/*.purs"] 
    } 
in
{ targets = 
  { main = 
    { dependencies = dep.dependencies # ["foo"]
    , sources = dep.sources # ["src/**/*.purs"] 
    } 
  } 
}

-- bound target with literal target
let dep = 
    { dependencies = ["foo"]
    , sources = ["src/**/*.purs"] 
    } 
in
{ targets = 
  { main = { dep }
  } 
}

-- bound target with dependency on earlier target
let main = 
    { dependencies = ["foo"]
    , sources = ["src/**/*.purs"] 
    } 
let test = 
    { dependencies = main.dependencies # ["spec"]
    , sources = main.sources # ["testc/**/*.purs"] 
    } 
in
{ targets = 
  { main
  , test
  } 
}

[wclr]

Where those different "targets" are going to be build to (to what output directory) by the compiler?

[JordanMartinez]

For now, I've assumed that all targets' compiled output is stored in the same output directory. Since module names must be unique, I don't think that causes any problems.

[JordanMartinez]

Also, one issue I've come across while working on this. If we have multiple targets, how would that work with getting autocompletion, etc. via the IDE?

For example, let's say I have 4 targets. Since the IDE expects there to be one command that it runs to compile the code, I wouldn't want to run spago --target main build because that would exclude the other three. Autocomplete would work for src, but not for test, examples, or whatever other directories the targets use.

If one is using multiple targets, they likely have some sort of dependency chain between the targets, such that one target depends on all others and using that target would for spago --target targetName build would suffice. However, this would create a new problem. In Project A with two targets (i.e. main and test), the target name is "test". In Project B with three targets (i.e. main, test, and examples), the target name is "examples". Switching from one project to another would require one to change the target used in the command the IDE runs. I'll call this the "project-specific target name build command" problem.

Moreover, having a dependency chain between targets is not always guaranteed. In a monorepo setup, there might be multiple directories with each doing one thing separate from the others. In that case, one would need to combine all the targets into one by defining a fifth target. We could call this fifth target all:

let all =
  { dependencies = target1.dependencies # target2.dependencies # ... # target4.dependencies
  , sources = target1.sources # target2.sources # ... # target4.sources
  }
in
  { ...
  , targets = { ..., all }
  }

This all target solves the monorepo situation well. It's also the only workaround to the "project-specific target name build command" problem mentioned above.

However, I don't like this workaround. Every time I add a new target, I would need to update all to include the new target. Since all's definition is predictable, I instead propose the idea of a "virtual target". "Virtual targets" are

• targets that do not exist in spago.dhall but which Spago can create in memory based on what's inside of spago.dhall.
• targets into which the user cannot install additional dependencies

Using the above example, Spago could support a virtual target called "all" that combines the dependencies and sources of all other targets within the spago.dhall file. Then I could run spago --target all build, which would effectively do the same as what I wrote above for the all target. Since the all target would be a virtual target, I think it should be distinguished in some other way? Perhaps it could be a separate CLI arg entirely (e.g. spago --virtual-target all build)?

Another idea related to this one. I hate having to "install" psci-support to get REPL access. It spams a spago.dhall file in my opinion. With the idea of targets, it's now possible to have a main target that does not have REPL access because psci-support is not installed there and a main-repl target that adds REPL access because it adds psci-support. For example:

let main = { dependencies = [ "prelude" ], sources = [ "src/**/*.purs" ] }
let main-repl = main.with dependencies = main.dependencies # [ "psci-support" ]

However, now we have a new problem. If I want to get REPL access to my other targets, I need to add a *-repl target for each target. Why do I have to maintain these *-repl targets if all they are doing is adding a psci-support package to the dependencies field? I feel like this would be better implemented via a virtual target.

The difference between the all virtual target and this hypothetical repl virtual target is that the repl one builds off of an existing target. So, if I run spago --target foo repl, Spago should implicitly install the psci-support package and add that in memory to the target before running the REPL. Since this repl virtual target only works for the repl, there's no reason to run something like spago --virtual-target foo-repl build

[wclr]

I generally like the idea of multiple targets in a single spago config, as it can give an additional level of flexibility and avoid having separate configs if they are not really needed. At the same time, I think there is no need to go too far with this feature.

Trying to be too specific about what those targets are and how they should be used will inevitably lead to issues with different scenarios and accidental complications and workarounds (like "virtual-targets") as @JordanMartinez described in the previous post.

What I think how this feature should be generally designed:

• Multiple target should be optional, and default should "a single target" with common keys (src, dependencies, etc) in the root, as it is now.
• Targets just define additional install/build configurations that can be run independently.
• Targets configs should be explicit, though they probably should take the missing keys from the root config (which are present there).
• There should be optional key defaultTarget that would be used when spago run without specifying the target.
• I think there is a need to introduce output key that will define the output directory for purs compiler, and this key could be specified for a target.

As for the development/IDE process in a default assumed workflow the "default" target should be used, and the default target should really accumulate all the (needed) dependencies and sources from other targets, but it should be stated in config explicitly. For example, I do it now this way for my monorepo setup and have no problems with it except the one issue I created recently.

So I just have dhall config for each package with its deps (which will be published anyway in its own git repos to be consumed as the current packaging infrastructure assumes) and the root config that is used for development, which happens to include all the deps from packages and additional common deps for the project (e.g, deps for running tests):

let package1 = ./packages/package1/spago.dhall
let package2 = ./packages/package2/spago.dhall

in  { name = "monorepo"
    , dependencies =
          [ "spec"
          , "psci-support"
          ]
        # package1.dependencies
        # package2.dependencies
    , packages = ./packages.dhall
    , sources =
      [ "packages/**/src/**/*.purs"
      , "packages/**/test/**/*.purs"
      ]
    }

With targets support I could probably have it all in one config if it would be appropriate (which I'm not sure of yet) and that will also depend on how publishing is functioning.

[JordanMartinez]

Below is my understanding of your comment above. Correct me where my understanding is wrong:

• Targets configs should be explicit, though they probably should take the missing keys from the root config (which are present there).
• Targets just define additional install/build configurations that can be run independently.
• I think there is a need to introduce output key that will define the output directory for purs compiler, and this key could be specified for a target.

If I'm understanding you correctly, this sounds like what I will call a "target-then-root lookup resolution." If TargetA does not have a dependencies key, then got to--let's call it--the "root target" and see what it's dependencies key is. This would enable one to write targets like the following:

{ dependencies = [ "foo" ]
, sources = [ "src/**/*.purs" ] 
, output = "./output"
, targets = 
  { main = { }
  , test = { sources = [ "test/**/*.purs" ] }
  , foo = { output = "./barOutput", sources = [ "bar/**/*.purs" ] }
  }
}

main is defined as a target, but all it does is refer to the root target. test is a target, but all it does is add the test/**/*.purs source paths to the root target. foo is a target whose sources are src/**/*.purs and bar/**/*.purs; purs compiles its output to the barOutput folder rather than the output folder.

• Multiple target should be optional, and default should [be] "a single target" with common keys (src, dependencies, etc) in the root, as it is now.

Because the targets key is optional, the only target below is the "root target." This is quite similar to the current v2 config that Spago uses. The only addition is the output key.

{ dependencies = [ "foo" ]
, sources = [ "src/**/*.purs" ] 
, output = "./output"
, {- other fields -}
}

• There should be [an] optional key defaultTarget that would be used when spago run without specifying the target.

This is basically your solution to the "which target does Spago use when one isn't specified" problem.

As for the development/IDE process in a default assumed workflow the "default" target should be used, and the default target should really accumulate all the (needed) dependencies and sources from other targets, but it should be stated in config explicitly.

This is your solution to the "which target does an IDE use so that all desired code is compiled" problem.

---

Based on my understanding above, I'll now be responding to your comment in general, and then to specific points. If my understanding is wrong, then my arguments will be pointless.

I dislike most of this idea for one reason: it would make Spago do work that should be done via Dhall. I don't think Spago should need to do anything more than just lookup a key and use it.

For example, rather than having Spago figure out what some target's dependency field is via this "target-then-root lookup resolution," I can run normalize dhallExpr and then lookup the dependency value and get everything there. For normalize to work, the "root target" would need to be defined as a let binding, which is what the multi-target format basically does. Spago is only somewhat utilizing the features in Dhall. When the registry gets added, we will be able to use even more Dhall features. Going with your approach would likely stop that possibility from occurring. Rather, I imagine that it would make our usage of Dhall function more like a weird JSON rather than the programmable typed JSON that Dhall actually is.

I also dislike this idea because it's not clear what to do when you run spago install. Does Spago install the dependencies in the main target? Or in the root target? This is a problem completely avoided via "virtual targets."

The "target-then-root lookup resolution" also reminds me of sbt@2.x.x (Scala Build Tool), which I did not understand nor like in my brief venture with Scala.

• Multiple target should be optional, and default should [be] "a single target" with common keys (src, dependencies, etc) in the root, as it is now.

I disagree with this because I don't see why multiple targets should be optional.

First, a "single target" format is isomorphic to a "multi-target format" that has a single target. If we go with #811, the "single target" format below is the same as the "multi-target" format that follows.

{- single-target (current Spago format) -}
{ dependencies = [ "foo" ]
, sources = [ "src/**/*.purs" ] 
}

{- multi-targets with one literal target -}
{ targets = 
  { main = 
    { dependencies = [ "foo" ]
    , sources = [ "src/**/*.purs" ] 
    }
  } 
}

Second, if migrating is a concern, "single target" formats can be migrated to "multi-target" formats easily. In fact, what we have now (v2 config) was previously migrated from a different format (v1 config). The "multi-targets" format would simply be a v3 config.

Third, people using Spago almost always use spago init to set up a project. It's easier to delete stuff you don't need (e.g. the test dir and the test target) than to add this later. AFAICT, people are only using single targets right now because Spago doesn't support multi-target formats. That's why testing libraries are included in a project's dependencies even though they aren't needed.

There should be optional key defaultTarget that would be used when spago run without specifying the target.

I don't think your proposed solution solves the "which target does Spago use by default" problems completely. All spago commands (e.g. spago build, spago docs, etc.) use what would be the main target in the multi-target format, except for spago test. Only spago test uses what would be the test target.

So, in your proposed idea, what would happen if you run spago test?

• Would spago test always require a target as a CLI arg? If so, why have the command in the first place since it's basically spago run --target name?
• Would you drop spago test command completely? If so, why break users' expectations when a build tool doesn't have a test subcommand?
• Would spago test assume that test/**/*.purs sources exist as they do now and cause an error when they aren't found? If so, why provide the user a command that assumes a test/**/*.purs source path exists if that problem can be avoided completely via the multi-target format?

As for the development/IDE process in a default assumed workflow the "default" target should be used, and the default target should really accumulate all the (needed) dependencies and sources from other targets, but it should be stated in config explicitly.

This doesn't seem to solve the problem completely. Is there ever a time where you'd want the "default target" to refer to one target for spago build but a different target when running spago via the IDE? For example, if you have a src directory and a test directory, you likely want the IDE to build both. However, when you run spago build, you're not always wanting spago to build the test directory, especially if you have just done some major refactoring and know the tests will not compile.

I think there is a need to introduce [an] output key that will define the output directory for purs compiler, and this key could be specified for a target.

I think this is a valid point and the only part in your idea that I agree on 😄. I've assumed that all targets should share the same output directory, but there might be some targets that have the same module name because their code should never be run together. For example, let's say we have two targets, generateIcons and webapp. Both have a type class called Defaults for generating default values because their code is not intended to be used as a library elsewhere, so the type class is purely for convenience. generateIcons might be a script written in PureScript that generates some icons for the web app, and webapp could be the actual web application code. Outputting both of their outputs into the same folder may accidentally cause generateIcons' Defaults module to be packed up as the webapp's Defaults module. By specifying an output field, one can better support these monorepo / "one spago.dhall file used for multiple projects" setups.

[wclr]

First, I don't think that the points I laid out above go in contradiction with discussed proposal here. It is just my vision of the feature that can be wrong in some way and can be corrected, so I just wanted to stress some points that just should be considered more carefully.

If I'm understanding you correctly, this sounds like what I will call a "target-then-root lookup resolution." If TargetA does not have a dependencies key, then got to--let's call it--the "root target" and see what it's dependencies key is.

I've accented the word possibly in that point because it should be considered if it really the correct approach to extend target config with the values from the root, or to force the explicit configuration of the targets without involving any kind of inheritance.

I dislike most of this idea for one reason: it would make Spago do work that should be done via Dhall. I don't think Spago should need to do anything more than just lookup a key and use it.

I propose to come from the user perspective first of all, not from the dhall features standpoint. If it is reasonable from the user standpoint to inherit some values from the root then I believe it should be considered. Though I'm not sure that any values e.g, dependencies should be taken from the root (or maybe they should be even appended?), for example, the output property seems logical to inherit from for all the targets that do not specify it.

I also dislike this idea because it's not clear what to do when you run spago install. Does Spago install the dependencies in the main target? Or in the root target? This is a problem completely avoided via "virtual targets."

There are some questions that should be answered:

• Should multiple targets be required config? Then it is a breaking change.

• Should targets be optional, and then root properties could be interpreted as a build target (at least when there are not "targets" configured), this would allow keeping config backward compatible. If so what happens if targets are present, should root properties be in any case be considered as a target? Or in this case, only targets are considered (I would probably go this way).

• What should be the default target? Should be by default "main" target? Or maybe target named "default"? Should it be (optionally?) configurable via some key "defaultTarget/mainTarget". I actually think that word "default" in the context of build configs may be more appropriate than "main", besides you already have --main flag to point to the module to run.

So when those questions are resolved then it becomes clear what target runs "by default" (when no target is specified).

I didn't like this thing called virtual-target, not the idea itself. Maybe just to introduces a special arg --target-all that will combine all the targets (deps/sources) and perform the job, if it is really solving a common use case. I don't see a reason to call it specially "virtual target", only if you don't assume to have some other kinds of "virtual targets". And in this case, whether you call it virtual target or not it would make spago do some more sophisticated work than even just inheriting properties from the root to targets.

Also, there may be idea of combined targets specified in the config like list of other defined targets:

targets = 
  { test = { ... }
  , rest = { ... }
   ...
  , combined = [ "test", "rest" ]
  }

I just don't know if dhall supports this kind of config and if it is really appropriate and needed from a use case perspective.

This doesn't seem to solve the problem completely. Is there ever a time where you'd want the "default target" to refer to one target for spago build but a different target when running spago via the IDE?

As I stated it is assumed as the "default approach", if you need something else you can configure IDE tooling to run differently than the default build command.

For example, if you have a src directory and a test directory, you likely want the IDE to build both. However, when you run spago build, you're not always wanting spago to build the test directory, especially if you have just done some major refactoring and know the tests will not compile.

If my test won't compile I probably wouldn't want IDE to build them either because it would break build for everything else too as IDE is supposed to run just the same spago build command and handle its results. I just don't see the problem from IDE standpoint here, by default it would run "default" target, if you want you may reconfigure it to run something different. At the same time, it would be possible for the IDE tooling to configure them to run multiple commands for different targets (though need to explore and experiement if it really would be appropriate).

So, in your proposed idea, what would happen if you run spago test?

If this convention is considered to be useful, it should exist as probably a shortcut for spago run --target test, I don't see issues with this and how my points make any difference here. Though note that spago test also assumes the default main module name "Test.Main" to run, this also raises the question if the default main module for a run should be configured optionally in targets?

[f-f]

Thank you for the great discussion here!

I'll try to summarize here my wishes for how this should look like:

• I agree with Jordan's approach of using Dhall's features rather than having Spago do more work
• I'm onboard with the idea of a virtual target called all. I wouldn't like to have a separate flag for this (we'd use --target) and to prevent clashes we can just forbid users from using it as a target name
• a target should have an optional output key that maps to the output folder for that target. We'd default to output, and we'd pass that to the compiler (we currently don't do that at all, only the user can. We'd forbid that from the user and always pass it on the Spago side). However I think we should leave this out for now, and implement it in a later step.
• a target should have a required entrypoint key, that we'd use for spago run, spago test, spago bundle-app and so on
• I think we can just avoid having to install psci-support from the user side, and include it automatically when starting a REPL. This removes any interplay issue between the REPL and the targets. The code for the REPL is fairly self-contained, so we can just add psci-support to the dependencies before launching the REPL instead of erroring out as we do now

[wclr]

I agree with Jordan's approach of using Dhall's features rather than having Spago do more work

I still didn't get what nice dhall feature I was neglecting in my points. It sounds it is about some complex operations ("work") that could be avoided, but I just was proposing to inherit some values from the config root if it is appropriate and make sense (sematically) from the usage standpoint. If inheritance from the root is no good because it adds implicitity, it ok that everything should be explicitly encoded in config for all targets. Thought it may bring other problems, e.g, if user forgets to put some optional key in target config, say "output", it would be nice just have this property in the root and share its value between the targets, not to force all the targets to include that key.

I'm onboard with the idea of a virtual target called all. I wouldn't like to have a separate flag for this (we'd use --target) and to prevent clashes we can just forbid users from using it as a target name

I wouldn't like to oppose here much because it makes no big difference, but to me this seems to be a bit inconsistent. To me reserving a symbolic name that assumes a special kind of operation messes things up. A special flag reflects the special purpose and underlying operations. If there are supposed to exist other "virtual targets" (besides "all") a special flag like "--virtual-target" would be justified, at least it doesn't mess things up. And I think that from the user perspective a simple flag-like e.g., just --all for such a special case build would be more appreciated.

a target should have a required entrypoint key, that we'd use for spago run, spago test, spago bundle-app and so on

Not sure why should it be required if for example a target is not supposed to be run, for example, just to be built? I believe there are may exist targets that are not a subject for a run at all, or maybe the user won't be using spago run command (when other tools are used to work with the compiled code like bundlers, or some other runners). Also, the name entrypoint doesn't correspond to the currently used now flag called "main", would you like to change it too?