[RFC] Import query params

[padmaia]

Problem Statement

Currently in Parcel v2, dependencies are resolved to files and the resulting file paths are matched against the configured globs in the .parcelrc file to see which pipeline the file should be transformed by. This works for most cases, but sometimes you might want to force a dependency down a specific pipeline. For instance, let's say you are building documentation for a React component. By default, importing the React component will bundle the component as a JavaScript module, but what if you additionaly want the component's source as plain text? Maybe you also want to run some transformer over the component that extracts what prop types it expects and bundle that as well.

A very naive solution that would work for any and all tooling would be to just create files for the source text and prop types in a prebuild step and then you can import three individual files. The downside to this approach is that you would then have two build steps, which means you have two build commands that you have to string together and two separate watchers if you want to build in watch mode. Imagine in this scenario you updated MyComponent.js. Both the prebuild and the bundle watchers would pick up the change and start rebuilding. Then when the prebuild step finishes, the derived files are updated, which causes the bundler to rebuild yet again. It makes for a much better developer experience if this can all be handled in one process.

With webpack, this is usually solved by using inline loaders.

import MyComponent from './MyComponent';
import MyComponentSrc from '!!raw-loader!./MyComponent';
import MyComponentPropTypes from `!!extract-react-types-loader!./MyComponent`;

One downside to this approach is that the detail that you are using webpack leaks into your source code. Ideally this could be done in a more bundler agnostic way.

Proposed Solution: Import Query Params

The ECMAScript modules standard (ESM for short) is now supported across all modern browsers and will soon be supported by Node.js as well. One neat thing about ESM is that it essentially treats all module specifiers as URLs, which means adding query parameters is perfectly valid syntax and not something bespoke to a specific bundler. This seems like a great way to specify how you would like a dependency loaded:

import MyComponent from './MyComponent.js';
import MyComponentSrc from './MyComponent.js?as=raw-text';
import MyComponentPropTypes from './MyComponent.js?as=react-types';

So what would it take for Parcel to support this? The most straightforward approach requires no changes to Parcel's configuration so it's got that going for it. All you would have to do to force a file to be transformed by a different pipeline is add a query param to the module specifier and configure a pipeline in your .parcelrc with a glob that matches it.

{
  "transforms": {
    "**/*.js\?as=react-types": ["parcel-transformer-extract-react-types"]
  }
}

The only change we would need to make in Parcel would be to slice off the query params from the module specifiers to resolve the file and then tack them back on to resolve the transformation pipeline. Perhaps it would be nice to have a special glob matcher that makes it easier to write globs of this type. Query params could also be passed on to transformers as well.

import profilePic from './profile.png?width=300&height=300';

Drawbacks

• Since this solution is based on ESM it makes it pretty JavaScript centric. Some other languages and module formats support module specifiers as URLs, but what about the ones that don't? Parcel will either end up encouraging developers to write non standard code that only works because it's run with Parcel or this behavior may not end up being supported in all types of files.
• Text matching can get hairy. It seems possible that this could cause Parcel config files to balloon especially considering there could be multiple query parameters that might influence what pipeline a file should be transformed by.
• This is more of a drawback for the transformation configuration model in general, but one thing it doesn't accommodate is influencing what pipeline to use based on the file that defined the dependency. For instance you might want something different when importing as=url from a JavaScript file vs a CSS file. This could be solved by using a more specific parameter like as=js-url, but the problem still remains for imports without query parameters. If we decided to change the configuration to allow for this type of thing then it could potentially lead to a cleaner/smaller/simpler config.

Alternatives

One alternative would be to make a change to Parcel's configuration. I do like how simple Parcel's configuration is at the moment, but I could imagine this not working perfectly for all cases. I'm not aware of any of these cases at the moment, but that's what the RFC is for. If no one brings up a compelling case that proves that a change to configuration would either be necessary or would be a much cleaner solution, then this seems like a reasonable place to start.

Another alternative could be to use a special extension like extensions leading with a certain number of underscores.

import MyComponent from './MyComponent.js';
import MyComponentSrc from './MyComponent.js.__raw';
import MyComponentPropTypes from './MyComponent.js.__react-types';

Parcel could split the module specifier like it would have to with query parameters and use the parts before the special extension to resolve the file and tack the special extension back on to resolve the pipeline. You could also have a prebuild step that actually creates these files which means the imports would still work with other tooling. One potential issue is that the special extension might conflict with legitimate extensions.

Open Questions

• ESM treats module specifiers as URLs but CommonJS does not. Should Parcel only support this feature on module specifiers that are known to be URLs or should it be less strict?
• Should this be supported in node_modules or only source code?
• What happens when the configured transformer changes the type of the asset? Do we use the query parameters to match the next pipeline or do they only apply to the first pipeline? (Seems like you would probably have to use them to match if you wanted any control over whether to jump to a new pipeline or not)
• Any examples where this type of configuration doesn't cut it?

cc @devongovett @wbinnssmith @jamiebuilds

[mischnic]

Should this be supported in node_modules

I think we shouldn't encourage publishing packages with this (currently) non standard code, at least in the beginning.

[devongovett]

ESM treats module specifiers as URLs but CommonJS does not. Should Parcel only support this feature on module specifiers that are known to be URLs or should it be less strict?

I think we should only support query params for specifiers that are URLs, so for JS, only ESM. CSS and HTML are all URLs, but langs like SASS and LESS may not be. Some research required. CJS shouldn't be supported - ? is a valid character in a filename.

Should this be supported in node_modules

Agreed with @mischnic.

What happens when the configured transformer changes the type of the asset?

I think it should only apply to the first pipeline, and should be stripped off afterward.

[devongovett]

I think in order to make the config easiest to use, we shouldn't require escaping the query params. We can build a custom matcher, that strips off the query param first and matches the glob on the left side, and then checks the query params separately if it matches.

[devongovett]

It would also be pretty nice if config like this worked:

{
  "transforms": {
    "*.png": ["parcel-transformer-png"],
    "**?as=url": ["parcel-transformer-js-url"],
    "**?as=base64": ["parcel-transformer-js-base64"]
  }
}

Currently, that wouldn't work - for a PNG file, only the .png pipeline would run, and not the url or base64 pipeline. It could be made to work if you configured the order differently:

{
  "transforms": {
    "**?as=url": ["...", "parcel-transformer-js-url"],
    "**?as=base64": ["...", "parcel-transformer-js-base64"],
    "*.png": ["parcel-transformer-png"]
  }
}

The ... would cause the next matching pipeline (*.png) to run first, and then continue with url or base64. But that's kinda confusing vs the first.

Any ideas on how we could make this work?

[rtsao]

It makes for a much better developer experience if this can all be handled in one process.

One alternative is to expose a Parcel API for conveniently allowing transformations to write files to disk. This way it's still a single process with a single command and single file watcher, but there's zero "magic" in the JS module bundling phase as only JS files are importing other JS files (as opposed to arbitrary file types). This seems the most bundler agnostic solution, as any other bundler could pick up these plain JS files.

One huge advantage to on-disk code generation is generated code can be statically typed. Type checkers are not going to be able to understand query parameters. However, the transformer could output a file on disk that any type checker will be able to understand.

The main potential drawback I can see is passing configuration at the import site might not be possible (as the files would have to exist already) -- so the config would have to be centralized. This might not necessarily be bad, though. The other drawback is potentially extra work compiling files before they are used. However, this might be avoided using conventions around filenames where transformed files are only generated on-demand.

[jakearchibald]

My opinions here may not fit with what Parcel is trying to achieve, so feel free to ignore.

I don't think build systems should hide. If a file is to be transformed in an uncommon way, it should be obvious from the import that something special is happening.

import text from './whatever.js?as=text';

The above doesn't really look special. It's just a regular URL.

If the above is passed to a system that doesn't recognise the 'as' query parameter, it'll interpret the module as JavaScript, which feels like a bad fallback. If the transform doesn't have an 'acceptable' fallback (if a file fails to minify, the result still works), the import should fail as soon as possible.

Other ideas:

import text from 'as-text:./whatever.js';

The as-text scheme is invalid in module identifiers, so it's obvious something special is happening. Without the transform, this will fail at the parse stage.

import text from './whatever.js/text';

This treats whatever.js as a directory, when it isn't. Transforms would define what a file could 'contain'. Without the transform, this will fail at the fetch stage. That might be too late, and the special nature of the import may not be obvious enough.

• Should this be supported in node_modules or only source code?

I'm not sure if files within node_modules should support this syntax. However, I think files outside of node_modules should be able to use it on files within node_modules.

Eg:

import text from 'whatever-package?as=text';

[devongovett]

That's interesting feedback @jakearchibald! One reason we were thinking about query parameters was to avoid making as "special". It's just an option that is passed to transformers, like other query params (e.g. image width/height). It could be called type or banana for all Parcel core cares.

If we went with something more specific like a URL protocol, it would also affect resolution in addition to transformation and would need to become a bit more integral to Parcel core. But I can see your point about fallback in case a build system isn't configured to use as vs failing early. We can think about that further.

As for the specific suggestions, protocols seem like an interesting idea, but feel a bit like a "source" rather than a transform. e.g. normal protocols like http:, file:, and even std: modules are all sources - you cannot request the same file from different protocols generally. The directory idea is hard as well because it requires cooperation from the resolver plugin as well as the transformer.

[jamiebuilds]

There's a good point made that "falling back" to JavaScript when ?as=text is not supported, is actually worse than not resolving to anything. If you're going to fail, fail early.

A couple constraints:

• It needs to be specified as part of a "URL" (or rather a string-based module specifier), we can't rely on additional syntax being available in the language, because it very well might not exist.
• We need to stick as close as possible to what is considered "normal" in a language, even if that means not supporting every feature everywhere.
  • Ex: Don't try to make Rust support importing JavaScript
  • Ex: CSS has a way to refer to images with url(...) so push people that direction.
• We can't use protocols:// because we allow importing URLs in many forms
• I think there needs to be an abstraction between module specifiers and transformer code. i.e. !raw-loader! is a bad solution because it specifies the transformer directly.

I was in favor of query parameters before, but @jakearchibald made a good point that unknown query parameters rarely change how a route is resolved by a server.

Right now I'm leaning more towards the additional extension:

import "./file.js.txt"

This has the benefit of reusing existing configuration:

{
  "*.{txt,text}": ["@parcel/transformer-raw"] 
}

The problem comes in trying to resolve it. How do you decide ./file.js.txt should resolve to ./file.js?

The answer could be stat-ing basename(name) recursively until you resolve to something, but I haven't seen anything that behaves like that before.

[jakearchibald]

@devongovett

As for the specific suggestions, protocols seem like an interesting idea, but feel a bit like a "source" rather than a transform. e.g. normal protocols like http:, file:, and even std: modules are all sources - you cannot request the same file from different protocols generally.

The only exception I can think of is view-source:http://example.com/ which a few browsers do.

@jamiebuilds

Right now I'm leaning more towards the additional extension:

Yeah, that could work. Would it be chainable? Say I had two transforms:

• Optimise image data
• Data metadata for image (width, height, file size etc)

…would I be able to do:

import data from "./file.png.img-opt.img-data";

[jamiebuilds]

For that specific case I don’t think we’d encourage the use of optimizers via file extension vs Parcel’s config format:

{
  "transforms": {...},
  "optimizers": {
    "*.png": ["@parcel/optimizer-pngmin"]
  }
}

However for the generic case of chaining I’m a little bit torn.

There is some precedence for this in build systems as well, Ruby on Rails’ asset pipeline uses extensions to turn on different file transforms (although that’s at the file system level, not at import-time).

We’d have to consider how the Parcel pipeline interacts with this:

import "./img.png.img-data"

If the config were to look something like this:

{
  "transforms": {
    "*.png": ["@parcel/transform-png"],
    "*.img-data": ["@parcel/transform-img-data"],
  },
  "optimizers": {
    "*.png": ["@parcel/optimizer-pngmin"]
  }
}

There’s a lot of questions about which plugins run and when they run:

• Does the png optimizer run before you get the image data?
• Should the *.png match .png.img-data?
• Should png transforms/optimizers be run at all? Maybe it makes sense in this case but if you were importing .js.raw-source would you want Babel to run first? (Probably not)

I’m not sure we can come up with a pipeline algorithm that will work for all cases. In which case I’d prefer forcing users to do a little more work to be explicit than to have them be frustrated with a bad abstraction.

[jamiebuilds]

Vague late night thoughts:

• I think we need a new plugin type for "viewers" (or something like that).
• This new plugin type would apply only at the import level.
• You would only be able to turn on one viewer per import.
• The viewer plugin would have (declarative) control over the transforms/optimizers to be run (level of control needed should be explored)
• I've flipped on file extensions, I was pulling it from RoR before, but thinking more about the requirements, I don't think it works that well.
• I vaguely like view-source:http://... but I think it implies it's a standard of some sort
• I actually think that Webpack's loader!./file syntax would be nice here, but I explicitly want to use a different syntax than them to avoid feature confusion.

Some ideas on syntaxes other than loader! that don't conflict with the URL standard or what I've seen in any programming language import specifiers, and which will fail to load if you try loading them outside Parcel:

import "viewer~specifier"
import "viewer^specifier"
import "viewer|specifier"

Example

import "image-data~./file.png" // runs optimizer-png, but not transform-png
import "raw-source~http://example.com/script.js" // doesn't run any optimizers or transforms

{
  "transforms": {
    "*.js": ["@parcel/transform-babel"],
    "*.png": ["@parcel/transform-png"]
  },
  "optimizers": {
    "*.js": ["@parcel/transform-uglify"],
    "*.png": ["@parcel/optimizer-pngmin"]
  },
  "viewers": {
    "image-data": ["@parcel/viewer-image-data"],
    "raw-source": ["@parcel/viewer-raw-source"]
  }
}