Searchable REPL thoughts

[vedam]

hey all,
I've been thinking a bit about a browsable REPL-structure and I would like to share some thoughts for the upcoming v3 and guide-overhaul.
I discover myself stumpling through unrecognizable bookmark-folders with bad named bookmarks to keep some scripts/examples/best-practises/solutions in mind.
In the meanwhile there are so much, somethimes outdated, that I have to open several to find the right one. In the best case. Most of the time, I'm not able to find what I was looking for.

Wouldn't it make sense for the saved gists to add some search or sort criteria to get/find to the examples the user needs? Kind of what is done currently with the menu.

What could be a structure that makes sense, not distracting with too much overhead or additional administration-expense?

• adding a category (i.e. official, best-practise, guide, basics, animate)
• adding predefined tags (i.e. need-help, solution)
• adding a short description

Maybe it's just me using the REPL this way. Maybe it's too much.
But as I notice, also from the repeadetly asked questions over @discord ('how can I do this', 'xy is not working', 'why?', 'how?'), the REPL could, maybe should be one central piece to make svelte a really good dev-experience.

Just a thought.
I know v3 requires all resources.

[mindrones]

Filing here some of ideas that came out while chatting about the REPL.

Retrieving gists

Gists made with the Svelte REPL should contain a meta.json file containing {"svelte": true} so in theory it's possible to find them and tag them based on some criteria.
Unfortunately the GIST API does not provide a search endpoint.

We tried using @jvcalderon's gist-client but so far it threw.

const GistClient = require("gist-client");

const gistClient = new GistClient();
const GITHUB_TOKEN = "GITHUB_TOKEN";

gistClient
.setToken(GITHUB_TOKEN)
.getAll({
    rawContent: true,
    filterBy: [
        {public: true},
        {content: "svelte"},
        {since: "2018-11-01T00:00:01Z"}
    ]
})
.then(gistList => {
    console.log(JSON.stringify(gistList))
})
.catch(err => {
    console.log(err)
});

The Github token shouldn't be required for public Gists but so far it throws if we don't provide a token.
It is promising though as it would seem that it is fetching all (max 3000) Gists and then filter them based on our criteria: using rawContent: true in combination with {content: "svelte"} in the filters should get gists containing "svelte" by making specific calls to retrieve files content in the filtered gists.
If we were able to solve this, we could start analyzing/tagging the current gists (Svelte 2) and think about a new UI/UX for searching them.

EDIT: as of today a simple search (https://gist.github.com/search?q=svelte) shows that there are 232 gists containing the word "svelte".

[mindrones]

Detecting Svelte features in a gist

Another part of the conversation has focused on the idea of retrieving features from gists (#if, /else, class:, etc).

This would let us search by feature, similarly to what has been done in blockbuilder (here @enjalot outlined some very interesting insights about the effort behind that kind of website).

@pngwn has written this walker to show how we can retrieve features from compiled templates 😎 .

const svelte = require("svelte");
const walk = require("estree-walker").walk;

const compiled = svelte.compile(`
<p class:myClass="condition" on:click="noop()"></p>
{#if condition}
<div>boo</div>
{:else}
<p>Hi</p>
{/if}

<script>
export default {
    data() {
        return {
            condition: true
        }
    },
    methods: {
        noop() {}
    }
};
</script>
`);

function walky(ast) {
    let features = [];
    walk( ast, {
        leave: function ( node, parent ) {
  	         if (node.type) {
                features.push(node.type);
             }
             if (node.attributes) {
                node.attributes.forEach(v => features.push(v.type))
             }
        }
    });
    return features;
}

console.log(compiled);
console.log(walky(compiled.ast.html))

In this example:

• IfBlock -> #if:
• ElseBlock -> /else
• Class -> class:
  etc.
  If the provided string was a REPL example, we could tag it with these Svelte features and think about the appropriate UI to search by these features.

[mindrones]

Meta file format

Another part of the conversation was about the meta.json.

Currently as said that file contains just {"svelte": true}, but in the future it might contain:

• the Svelte version,
• eventually the repl version (to detect files expected to contain certaion fields)
• the list of features detected in the gist
• gist: ID, author, etc
• ?

{
  "svelte": true,
  "versions": {
    "svelte": "3.0.0-alpha1",
    "repl": "1.1.1",
  },
  "features": ["#if:", "/else", "class:", "on:click"],
  "gist": {
    "id": 1233453564657,
    "author": "pngwn",
    "title": "foo bar",
    "forkof": 6578567456345345,
    ...
  }
}

Again, having these fields would permit to think about an enhanced/unified search:

• by author
• by strings in title
• by features
• by Svelte version
• ...

[mindrones]

Version switch

There has been some discussion about the possibility to switch Svelte version while editing the REPL, especially useful while learning how to upgrade to a breaking version.

Should we do this, the meta file would have to change depending on context.

[mindrones]

Gists publishing and meta files storing

We realised that if the interface let us to publish (and unpublish) gists, when a gist goes "Public" we can save its ID and its meta file somewhere.

As of now we don't expect a huge amount of gists to be published so probably we can just update a single gist with the contents of all the meta files. This would permity us to easily search by fetching and searching in that central gist in the client memory without using a DB for now.

Also, since the Svelte 2 gists have a minimal meta file (not particularly useful for these purposes), we could make so that the REPL saves this additional information only for Svelte version 3 to clearly mark a line between v2 and v3.

Users would have to actively update their gists to Svelte 3 using the REPL interface to mark an old gist as searchable (setting it to version 3+ and publishing it).

EDIT: this would avoid us the need to search for gists in the future.

[pngwn]

I think we can definitely improve the REPL, especially having more examples and making discoverability easier.

Searching Gists
I'm not sure if this will work unless we default gists made via the REPL to public (which I don't really like). People should feel free to mess around on the REPL and save their gists without worrying that it will be made public. Currently, gists made via the Svelte REPL are secret and I think it should stay that way. We should have a make public or publish option that flags them as public and adds that REPL example to the 'search database' (however that would be implemented). We could then search anything within this collection. A curated selection of them could make up a showcase/ demo minisite/section.

Feature Tags
I think this is a good idea, its a little more complex than the example above but it should be pretty doable. I think the returned AST gives enough information to work out what features are being used. V3 probably makes this easier because we don't have the distinction between different function/ method types (there are less anyway). We'll obviously need to walk the JS AST as well, not just the HTML, but it should be possible to add these tags automatically at/after compilation. I don't actually know how the REPL works so I'll need to look into it before I can say anything for certain.

Versioning
I think this is really a matter of philosophy. I don't know if Rich wants a clean break from v2, in which case the v2 REPL should go along with the v2 Documentation and never the twain shall meet. On the other hand, we could make it so that the user can select a compiler version within the REPL itself. I don't know how difficult or desirable this would be. We should definitely store more information in the metadata.json, that will just make our lives easier in the future (even if we only support one compiler version at a time, knowing that certain gists are not compatible will be useful).

Version Switching
I'm not convinced by this. I think its a cool idea but the svelte-upgrade package has its limitations and it might be frustrating for users if they keep getting partially coverted components. Even if we spend time writing the best codemod we can come up with, there will probably be caveats and I honestly don't think it is a great use of time. After a transitional period there will be very little value in it for most people and given the significant differences between v2 and v3 it wouldn't be the simplest of tasks. We will need comparison/ upgrade examples as part of an upgrade guide but we would be better off writing or at least finishing them by hand to ensure the clearest, most idiomatic v3 code. v3 is hugely different and we will probably write our code slightly different due to the freedom the new design brings. This should be reflected in the examples we write. I think this is a better compromise. Additionally, if we choose not to have v2 and v3 REPLs in the same REPL a convert option makes little sense.

Other Thoughts
One thing I've been thinking about is having REPL examples embedded in the documentation as examples that a user can freely edit (and maybe fork and save). These kinds of experiences always seem the best to me, giving the user the ability to play with the feature being spoken about. I don't know how realistic this would be or if it is practical from a performance point of view (multiple REPLs on a page, etc., there are resolutions to this problem but it can get complex very quickly). The docs are written in markdown and I think everyone would prefer that to continue so I guess we would need a way of injecting REPLs into markdown files. The obvious solution to this is for the REPL to be a svelte component and our markdown to be MDX for svelte. How this would work, I have no idea, but I've been thinking about it a lot recently. This could also bring other benefits, we wouldn't be limited by what Markdown allowed. If we wanted to use animations or interactive examples in the docs, this would allow us to do so. This is out of the scope of this issue but I thought I would mention it.

I also think I prefer the term 'playground' to 'REPL'.

[mindrones]

Renaming the meta file to svelte.json

Current limitations of the Gist API (no search endpoint) force us to retrieve every single file in a gist to parse its content, which makes us quickly incur in API limits.

One solution to this would be to rename meta.json to svelte.json so that to detect Svelte examples we would have to search gists containing a svelte.json file and inspect its content to check if it's indeed a Svelte REPL meta file.

[mindrones]

I also think I prefer the term 'playground' to 'REPL'.

@pngwn me too

[mindrones]

Listing and searching own gists

Once logged in, it would be quite useful to list and/or search our own gists (by title, used Svelte features, etc) directly from the REPL.

Searching a user's gists shouldn't be a heavy operation and it would be a matter of filtering those with a working svelte.json.

This functionality would help to revisit old tests and eventually turn them into public examples or delete them.

Saving a thumbnail of the rendered area

Eventually, to help the search, we might think about capturing a thumbnail of the rendered area to be shown while the searching.