TSD Future

[blakeembrey]

Hey everyone!

I'm going to be jumping into this project and seeing what we can do to improve the sharing of TypeScript definitions. Development isn't (quite) going to happen instantly, so I wanted to submit a short issue that describes some of my immediate goals and see if there are other ideas of what they want to see moving forward. Feel free to create new issues with feature requests and we can label them for TSD Future.

Goal

The goal of TSD future is to seamlessly support any TypeScript definition files, from any source. By default, you should still be able to search the definitely typed repository and install from there. However, more focus will be trained onto support independent definitions, specifically ones bundled with modules. This is a primary focus because as the ecosystem grows, a single repo is going to be a challenge to scale and I would like to give more power to module authors.

Currently TSD (tsd@next) supports a feature called tsd link. It's a great first step, but comes with many drawbacks - mostly because it's just a really simple path copy and paste. It needs a bit more to be useful. In the future, you'll be able to tsd install and it will look up your dependencies via package.json or bower.json and it will install definition files. However, it'll then recursively resolve dependencies from there. Any dependencies the module has likely have their own definition files that they're depending on.

This will make tsd.d.ts less editor friendly, but the goal is to avoid having to manually edit this file and check in dependencies any more. Just as you don't check in your node_modules or bower_components, you shouldn't have to check in your type dependencies. As we resolve dependencies down the tree, if we encounter tsd.json files they'll also factor into the recursive resolution. This is important since not every module can or will adapt over night. If there are conflicts between sub-dependency definitions, we will provide a dialog for selecting the version that should be used (resulting in that flat structure we wanted, a la bower).

Doing this, we'll pull closer to the seamless usage of definition files. Similarly, the commands will be pulled closer to npm and bower. All that will be required from now on is tsd install. To remove old dependencies, you can tsd prune. Finally, tsd uninstall will remove a dependency from tsd.json. The tsd.d.ts file will always be kept in sync with hard module dependencies and the tsd.json.

To support all this, I think some changes to tsd.json are required. I'm going to change this last (avoid it if I can), but I think it's too tied to the one source approach which doesn't scale. From now on, manual definition file installations tsd install <source> should create an entry like package.json with "name": "source", where source can be any file/url (tsd.json or d.ts file). For example, the source when installed from the definitely typed repo will the raw git files. This allows anyone to host their own definitions. tsd install <module> will still fall back to the definitely typed repo and tsd search <module> will still search the repo. It is possible for this to be automated as well.

Having the name -> source mapping allows us a little more flexibility. For example, we can now check in our custom definition files and have them resolve according to the recursive algorithm since they are defined. On top of that, we can create aliases (which are supported by build tools like browserify and webpack) from one module to another. Finally, we can eventually support the module output format by TypeScript and automatically compile that into the namespaced module. That would personally be very cool to do, since it's a pretty big pain point for third-party module developers.

Ok, I think I'm done rambling but there will definitely be more to come.

Tracking

• TSD to have command parity with NPM and Bower TSD to have parity with NPM and Bower commands #142 Alphabetize entries in installed field of tsd.json #119 Modify install command to perform 'reinstall' if invoked with no arguments #122 Create uninstall command #6 tsd install should install every definition listed in tsd.json #116
• Perform a complete, recursive defintion resolution Packages with a / in the name don't work. #134 TS6053 File not found when import tsd.d.ts from bower distributed package #105 Does tsd link not work? #130 Resolve dependencies for linked bundles #123 Package linker's folder scan uses hardcoded folders #109 tsd reinstall does not remove stale typings #151
• Make sure we're running on all node versions >= 0.8 and iojs Update to be compatible with Node.js 0.12 #144
• Fix general inconsistencies First argument ignored when using flags #143 tsd --save flag modifies permissions of parent directories unnecessarily #125 'undefined' being written to tsd.d.ts #120 Overlap path value install library --save each time in tsd.d.ts #112 Fix CLI query on Max etc #62 Report errors more gracefull #75 Simplify installed property in tsd.json #113 Generating tsd.json changes its parent directory's permission to 0744  #95
• Automatically install unknown dependency definitions tsd link --implicit proposal #115
• Install dependencies from any source query definitions from multiple repo #97 Support non-Github repos. #135 Allow install from seperate repo like bower #87
• Start work on a wrapper that will accept any definition file and declare as a module name

Testing

I will start work in the next few days on the future branch.

npm install -g DefinitelyTyped/tsd#future

Related

There's a few proposals in TypeScript right now to fix the definitions situation that will probably (hopefully) make most of this redundant (microsoft/TypeScript#2338, microsoft/TypeScript#2839).

[5angel]

Glad to know this project still has traction. So when do you plan deaing with #144?

[blakeembrey]

@5angel It definitely still has traction, but I'm not currently focused on previous versions of TSD for a few reasons (namely still not having publish access to NPM yet, since the original author is MIA). That issue doesn't mean it's not working with 0.12 or any other version, it's just the 0.5.x branch was pinned to certain node versions (see https://github.com/DefinitelyTyped/tsd/blob/master/package.json#L37). You can still install, it should only be a warning. If you install with tsd@next it won't even warn you since it specifies >= 0.10.

[5angel]

@blakeembrey phew, thank you. It was thinking about cloning the rep and making a pull request (:

[basarat]

namely still not having publish access to NPM yet,

@blakeembrey We (you and me) have that now. I'll change the tag on 0.6 to be latest (instead of next). I'll also remove beta from it. That will ensure that people can continue to use what we have right now while you work on vnext 👍 🌹

[blakeembrey]

@basarat Fantastic, thank you 😄 I saw it and I'm glad everything is alright.

[basarat]

Alright done ... phew. 0.6.0 is now latest so npm install tsd -g should just work.

[Bartvds]

Author here, sorry for going AWOL and dropping off the grid, stuff went different then expected 😞

I kinda dropped the ball here, sadly can't commit OS coding time anymore so @blakeembrey has my full blessing to go ahead on this and take ownership. Thanks @basarat for being admirably more dependable contributor and getting stuff rolling.

Today I got an email from Angular guys who needed a sub-dependency mistake fixed: minitable. So I pushed an update for it to npm earlier but for some unknown reason I pinned the exact dependency here in TSD. So I think we need an TSD v0.6.1 soonish.. a PR will turn up any time.

Anyway, I'll have a look around the tickets soonish and see what's up. Its been a while since I ran TypeScript and node development though (doing python now (:open_mouth:) And I'll put some text together about project status and stuff.

[blakeembrey]

@Bartvds Nice to see your ok! I was a little worried, even though I'm yet to meet you. Feel free to keep push any TSD versions and I'll make sure it stays maintained until the next version is ready 😄

[Bartvds]

Cool, thanks. I got distracted. Not sure if I get back in TypeScript yet but at least TSD could have second revival (2nd yes).

It was a bit of wayback machine but here is TSD 0.6 bootcamp to help you going: Since I don't know when I got focus again I just typed a lot now:

• 0.3 and 0.4 where from maintainer before me, also got limboed.
• 0.5 was my full rewrite to link directly to github (I had too much time back then). 0.5.7 has been used up till this week.
• 0.6 was major cleanup to imports style and more npm modules (not all, not enough). it got stuck in limbo for release for a long time.

The main issues TSD faces and tries to solve are about user interface and conventions of the DefinitelyType repos. There was (is?) not a 100% consensus about how package naming and versioning are handled.

TSD works using the filenames in first level folder as the module name (npm name recommended), it also considers some legacy nested folders. The main folders are usefull for convenient bundeling but over whole of DT there is not a universal semantic pattern of putting definitions together in folders. Then parses for semvers (filename is leading). The information in actual header of file is mostly cosmetic for listing/search.

Then there is the use of bundles/linked definitions; TSD link and dts-bundle are an attempt but appear flawed. There is a prototype in 0.6 to read information from package.json, an approach to scan for tsd.json could be cool. It gets a bit fuzzy what defs/types you actually need, you might not need all of a packages tsd.json but only what is visible on it's exported API.

Versioning is an issue. It is a mix of the path/name/semver and git's commit system. Keep in mind the semver in path is of the module it describes, not of the definition. There is no guarantee a next commit on same (semvered) filename doesn't break peoples builds, so for safety TSD pins on filename in a specific commit.

There were ideas of adding metadata to either the headers or a separate file in DT repos, and some other ideas in various related repos (here, DT itself, definition-header, definition-tester etc).

Since 0.6 was oldish even before it got lost I was trying a prototype of a cleaner version that uses streams all the way. It supported adapters, to read trees and blobs from file system, github and local git. Most of it worked but I sorta got stuck on streamy http cache and some stream plumbing problems. Could be solved though.

Anyway, some random stuff about code:

• it's a lot of own code, at least it's strict formatted and linted (and typed 😉) and I hope you like generic promises..
• /src code is basically a few main packages:
  • tsd has the type-definition specific stuff (and main app)
  • xm folder has (own) old utils. i tried to replace much of it with npm modules in 0.6 but only got so far. usefull but mixed quality. would be nice to get rid of.
  • expose is an experimental CLI command router and does fancy layout stuff, generate help, and outputcolor/plain/html etc. not essential but looks cool.
  • http is a http downloader with local file cache, it does e-tags and gzip and some http options (proxy etc). it prunes cache periodically. a small flaw is that it buffers downloads (works ok in our use-case).
  • git has wrappers to access blobs and trees from github over http, it uses the http cache.
• the typings folder is epic, i used to edit DT contributions as hobby.
• tests are a odd located: the actual cases are in src/spec, and src/test holds support code and custom assertions. some fixtures are located elsewhere.
• some tests are a little involved, but most TSD/CLI specific ones allow to add many cases by repeating the pattern: after verifying updates you can copy new test output to the fixtures easily.
• the gruntfile uses a custom grunt config wrapper/generator to create partial-tests. it's still just regular grunt running the tasks.
• most of actual logical interface is implemented in tsd/API.ts (should be exported and documented for re-use).
• API uses a structure from tsd/logic/Core.ts holding main services (index, http, git, downloaders etc).
• sub-systems usually know about the structure of Core and may reach into/through it to get their own work done (fun times but TypeScript compiler keeps it together), still try to glue in API.
• there is a settings/env/path thing in tsd/context/Context.ts. its not pretty but has the stuff. annoyingly it does IO and holds a copy of the tsd.json information in memory (race much).
• cli is setup from tsd/CLI.ts, it should use the API. API then glues together different services..
• .. so basically start there and follow the calls, but don't get stuck in custom utils.
• i felt there is a little too much code and too many dependencies and typed wrappers for what it does. although it is expandable in it's way; just slap on new services and add the glue.
• I'm Dutch and way back before 0.5 I just stopped writing games in Flash AS3..

.. anyway, most of it is kinda replacable except /src/tsd, and even that is basically just looping directory trees and parsing the paths for information. Then filter/compare and move blobs and paths around (don't underestimate amount of functionality needed though, it is a lot of things adding up, it's kinda like a mini VCS).

Cheers. 👍

[blakeembrey]

Thanks for the amazing introduction! 😄

[clavecoder]

VS15 and VSCode depend on TSD now, correct? It looks like you guys have a lot of rewrite to do. Can you get it done? Are you preparing for the io.js merge, for example? I'm feeling queasy about depending on tsd for my projects after all this discussion and the degree of rewrite in #170.

[blakeembrey]

VS15 and VSCode depend on TSD now, correct?

To be honest, I haven't heard or seen of this. Do you have a source?

Can you get it done?

Yes, absolutely. There's a lot of depth to the project, but it's not huge. I've just been incredibly busy since I started to rewrite and explore on the future branch.

Are you preparing for the io.js merge, for example?

That will definitely be prepared for, but it's unlikely we'll need to take any major action.

I'm feeling queasy about depending on tsd for my projects after all this discussion and the degree of rewrite in #170.

At the end of the day, it's completely up to you. TSD is currently stable on 0.6 and any future versions will have a completely seamless upgrade path. Even in #170, the first thing I implemented was a tsd init --upgrade command to seamlessly update to it - and that will always be the goal, making it as easy to use as possible.

In any case, the project does need refactoring and does need new ideas. Whether it's directly from #170 or we just pull some things from it, it's there to present some alternatives and directions. We need this because, as a new user even myself, it's very difficult to actually get started. In my first week using TypeScript I ran into issues getting definitions, then I ran into issues where they were out of date and had to hack in references to custom directories and copy and paste code, then I wanted to publish definitions of my own projects that depend on other definitions and that's not possible. Basically, I quickly ended up in a mess that tooling should have fixed. That fix is the goal of this.