Merge pull request #10401 from romanowski/scala3doc/our-doc

Provide basic documentation for scala3doc

Author: abgruszecki

docs/docs/usage/dottydoc.md

@@ -1,6 +1,6 @@
 ---
 layout: doc-page
-title: Dottydoc
+title: Dottydoc [Legacy]
 ---
 
 Dottydoc is a tool to generate a combined documentation and API reference for

docs/docs/usage/scala3doc/blog.md

@@ -0,0 +1,9 @@
+---
+title: Built-in blog
+---
+
+# {{page.title}}
+
+Scala3doc allows you to include a simple blog in your documentation. For now, it
+provides only basic features. In the future, we plan to include more advanced
+features like tagging or author pages.

docs/docs/usage/scala3doc/docComments.md

@@ -0,0 +1,96 @@
+---
+title: API Documentation
+---
+
+# {{ page.title }}
+
+Scala3doc's main feature is creating API documentation from code comments.
+
+By default, the code comments are understood as Markdown, though we also support
+Scaladoc's old [Wiki syntax](https://docs.scala-lang.org/style/scaladoc.html).
+
+## Syntax
+
+### Definition links
+
+Our definition link syntax is quite close to Scaladoc's syntax, though we have made some
+quality-of-life improvements.
+
+#### Basic syntax
+
+A definition link looks as follows: `[[scala.collection.immutable.List]]`.
+
+Which is to say, a definition link is a sequence of identifiers separated by
+`.`. The identifiers can be separated with `#` as well for Scaladoc compatibility.
+
+By default, an identifier `id` references the first (in source order) entity
+named `id`. An identifier can end with `$`, which forces it to refer to a value
+(an object, a value, a given); an identifier can also end with `!`, which forces
+it to refer to a type (a class, a type alias, a type member).
+
+The links are resolved relative to the current location in source. That is, when
+documenting a class, the links are relative to the entity enclosing the class (a
+package, a class, an object); the same applies to documenting definitions.
+
+Special characters in links can be backslash-escaped, which makes them part of
+identifiers instead. For example, `` [[scala.collection.immutable\.List]] ``
+references the class named `` `immutable.List` `` in package `scala.collection`.
+
+#### New syntax
+
+We have extended Scaladoc definition links to make them a bit more pleasant to
+write and read in source. The aim was also to bring the link and Scala syntax
+closer together. The new features are:
+
+1. `package` can be used as a prefix to reference the enclosing package
+    Example:
+    ```
+    package utils
+    class C {
+      def foo = "foo".
+    }
+    /** See also [[package.C]]. */
+    class D {
+      def bar = "bar".
+    }
+    ```
+    The `package` keyword helps make links to the enclosing package shorter
+    and a bit more resistant to name refactorings.
+1. `this` can be used as a prefix to reference the enclosing classlike
+    Example:
+    ```
+    class C {
+      def foo = "foo"
+      /** This is not [[this.foo]], this is bar. */
+      def bar = "bar"
+    }
+    ```
+    Using a Scala keyword here helps make the links more familiar, as well as
+    helps the links survive class name changes.
+1. Backticks can be used to escape identifiers
+    Example:
+    ```
+    def `([.abusive.])` = ???
+    /** TODO: Figure out what [[`([.abusive.])`]] is. */
+    def foo = `([.abusive.])`
+    ```
+    Scaladoc required backslash-escaping to reference such identifiers. Instead,
+    Scala3doc allows using the familiar Scala backtick quotation.
+
+#### Why keep the Wiki syntax for links?
+
+There are a few reasons why we've kept the Wiki syntax for documentation links
+instead of reusing the Markdown syntax. Those are:
+
+1. Nameless links in Markdown are ugly: `[](definition)` vs `[[definition]]`
+    By far, most links in documentation are nameless. It should be obvious how to
+    write them.
+2. Local member lookup collides with URL fragments: `[](#field)` vs `[[#field]]`
+3. Overload resolution collides with MD syntax: `[](meth(Int))` vs `[[meth(Int)]]`
+4. Now that we have a parser for the link syntax, we can allow spaces inside (in
+    Scaladoc one needed to slash-escape those), but that doesn't get recognized
+    as a link in Markdown: `[](meth(Int, Float))` vs `[[meth(Int, Float)]]`
+
+None of these make it completely impossible to use the standard Markdown link
+syntax, but they make it much more awkward and ugly than it needs to be. On top
+of that, Markdown link syntax doesn't even save any characters.

docs/docs/usage/scala3doc/index.md

@@ -0,0 +1,26 @@
+---
+title: Scala3doc
+---
+
+![Scala3doc logo](/images/scala3doc-logo.png)
+
+Scala3doc is tool to generate documentation for your Scala 3 projects. It provies similar features to `javadoc` or `scaladoc` as well as `jekyll` or `docusaurus`.
+
+As you probably have guessed, this whole site was created using Scala3doc.
+
+
+{% for post in site.posts %}
+## [{{ post.title }}](/{{ post.url }})
+
+{{ post.excerpt }}
+
+[ (read more) ](/{{ post.url }})
+
+{% endfor %}
+
+## Other extensions
+
+We would love to have your feedback on what you think would be good in order to
+render the documentation you want! Perhaps you would like to render method
+definitions or members? Do you want to have runnable code snippets? Let us know
+by filing [issues](https://github.com/lampepfl/dotty/issues/new)!

docs/docs/usage/scala3doc/specificTags.md

@@ -0,0 +1,15 @@
+---
+title: Scala3doc-specific Tags and Features
+---
+
+# {{page.title}}
+
+Scala3doc extends Markdown with additional features, such as linking
+to API definitions. This can be used from within static documentation and blog
+posts to provide blend-in content.
+
+## Linking to API
+
+Scala3doc allows linking to API documentation with Wiki-style links. Linking to
+`scala.collection.immutable.List` is as simple as
+`[[scala.collection.immutable.List]]`. For more information on the exact syntax, see [doc comment documentation](./docComments.html#definition-links).

docs/docs/usage/scala3doc/staticSite.md

@@ -0,0 +1,125 @@
+---
+title: Static docucmentation
+---
+
+# {{ page.title}}
+
+Scala3doc is able to generate static sites, known from [Jekyll](http://jekyllrb.com/) or [Docusaurus](https://docusaurus.io/).
+Having a combined tool allows to provide interaction between static documentation and API, thus allowing the two to blend naturally.
+
+Creating a site is just as simple as in Jekyll. The site root contains the
+layout of the site and all files placed there will be either considered static,
+or processed for template expansion.
+
+The files that are considered for template expansion must end in `*.{html,md}`
+and will from here on be referred to as "template files" or "templates".
+
+A simple "hello world" site could look something like this:
+
+```
+├── docs
+│   └── getting-started.md
+└── index.html
+```
+
+This will give you a site with the following files in generated documentation:
+
+```
+index.html
+docs/getting-started.html
+```
+
+Scala3doc can transform both files and directories (to organize your documentation into tree-like structure). By default directories has title based on file name and has empty content. There is an option to include `index.html` or `index.md` (not both) to provide both content and properties like title (see [Properties](#properties)).
+
+## Properties
+
+Scala3doc uses the [Liquid](https://shopify.github.io/liquid/) templating engine
+and provides a number of custom filters and tags specific to Scala
+documentation.
+
+In Scala3doc, all templates can contain YAML front-matter. The front-matter
+is parsed and put into the `page` variable available in templates via Liquid.
+
+Scala3doc uses some predefined properties to controls some aspect of page.
+
+Predefined properties:
+
+ - **title** provide page title that will be used in navigation and html metadata.
+ - **extraCss** additional `.css` files that will be included in this page. Paths should be relative to documentation root. **This setting is not exported to template engine.**
+ - **extraJs** additional `.js` files that will be included in this page. Paths should be relative to documentation root. **This setting is not exported to template engine.**
+ - **hasFrame** when set to `false` page will not include default layout (navigation, breadcrumbs etc.) but only token html wrapper to provide metadata and resources (js and css files). **This setting is not exported to template engine.**
+- **layout** - predefined layout to use, see below. **This setting is not exported to template engine.**
+
+
+## Using existing Templates and Layouts
+
+To perform template expansion, Dottydoc looks at the `layout` field in the front-matter.
+Here's a simple example of the templating system in action, `index.html`:
+
+```html
+---
+layout: main
+---
+
+<h1>Hello world!</h1>
+```
+
+With a simple main template like this:
+
+{% raw %}
+```html
+<html>
+    <head>
+        <title>Hello, world!</title>
+    </head>
+    <body>
+        {{ content }}
+    </body>
+</html>
+```
+
+Would result in `{{ content }}` being replaced by `<h1>Hello world!</h1>` from
+the `index.html` file.
+{% endraw %}
+
+Layouts must be placed in a `_layouts` directory in the site root:
+
+```
+├── _layouts
+│   └── main.html
+├── docs
+│   └── getting-started.md
+└── index.html
+```
+
+Sidebar
+=======
+Scala3doc by default uses layout of files in `docs` directory to create table of content. There is also ability to override it by providing a `sidebar.yml` file in the site root:
+
+```yaml
+sidebar:
+    - title: Blog
+      url: blog/index.html
+    - title: Docs
+      url: docs/index.html
+    - title: Usage
+      subsection:
+        - title: Dottydoc
+          url: docs/usage/dottydoc.html
+        - title: sbt-projects
+          url: docs/usage/sbt-projects.html
+```
+
+The `sidebar` key is mandatory, as well as `title` for each element. The
+default table of contents allows you to have subsections - albeit the current
+depth limit is 2 however it accepts both files and directories and latter can be used to provide deeper structures.
+
+The items which have on the `subsection` level does not accepts `url`.
+
+```
+├── blog
+│   └── _posts
+│       └── 2016-12-05-implicit-function-types.md
+├── index.html
+└── sidebar.yml
+```

docs/images/scala3doc-logo.png

@@ -15,7 +15,9 @@ sidebar:
           url: docs/usage/language-versions.html
         - title: cbt-projects
           url: docs/usage/cbt-projects.html
-        - title: Dottydoc
+        - title: Scala3doc
+          url: docs/usage/scala3doc
+        - title: Dottydoc [Legacy]
           url: docs/usage/dottydoc.html
     - title: Reference
       subsection:

docs/sidebar.yml

@@ -20,7 +20,7 @@
                         </a>
                     </li>
                     <li class="nav-item">
-                        <a class="nav-link" href="docs/blog/index.html">
+                        <a class="nav-link" href="blog/index.html">
                             Blog
                         </a>
                     </li>

scala3doc/scala3-docs/index.html

@@ -168,9 +168,9 @@ class DottyDokkaPlugin extends DokkaJavaPlugin:
       .overrideExtension(dokkaBase.getLocationProvider)
   )
 
-  extension (ctx: DokkaContext):
-    def siteContext: Option[StaticSiteContext] = ctx.getConfiguration.asInstanceOf[DottyDokkaConfig].staticSiteContext
-    def args: Args = ctx.getConfiguration.asInstanceOf[DottyDokkaConfig].docConfiguration.args
+extension (ctx: DokkaContext):
+  def siteContext: Option[StaticSiteContext] = ctx.getConfiguration.asInstanceOf[DottyDokkaConfig].staticSiteContext
+  def args: Args = ctx.getConfiguration.asInstanceOf[DottyDokkaConfig].docConfiguration.args
 
 // TODO (https://github.com/lampepfl/scala3doc/issues/232): remove once problem is fixed in Dokka
 extension [T]  (builder: ExtensionBuilder[T]):

scala3doc/src/dotty/dokka/DottyDokkaPlugin.scala

@@ -20,11 +20,18 @@ case class LazyEntry(getKey: String, value: () => String) extends JMapEntry[Stri
 case class LoadedTemplate(templateFile: TemplateFile, children: List[LoadedTemplate], file: File):
 
   private def brief(ctx: StaticSiteContext): String =
-    val code = Jsoup.parse(resolveToHtml(ctx).code)
-    code.select("p").first().outerHtml()
+    try
+      val code = Jsoup.parse(resolveToHtml(ctx).code)
+      Option(code.select("p").first()).fold("...")(_.outerHtml())
+    catch
+      case e: Throwable =>
+        // TODO (https://github.com/lampepfl/scala3doc/issues/238): provide proper error handling
+        println(s"[ERROR] Unable to process brief for ${templateFile.file}")
+        e.printStackTrace()
+        "..."
 
   def lazyTemplateProperties(ctx: StaticSiteContext): JMap[String, Object] = new java.util.AbstractMap[String, Object]():
-    def entrySet(): JSet[JMapEntry[String, Object]] =
+    lazy val entrySet: JSet[JMapEntry[String, Object]] =
       val site = templateFile.settings.getOrElse("page", Map.empty).asInstanceOf[Map[String, Object]]
       site.asJava.entrySet() ++ JSet(
         LazyEntry("url", () => ctx.relativePath(LoadedTemplate.this).toString),

scala3doc/src/dotty/dokka/site/LoadedTemplate.scala

@@ -44,23 +44,28 @@ class StaticSiteContext(val root: File, sourceSets: Set[SourceSetWrapper], args:
 
   lazy val mainPages: Seq[StaticPageNode] = templates.map(templateToPage)
 
+  val docsPath = root.toPath.resolve("docs")
+
   lazy val allPages: Seq[StaticPageNode] = sideBarConfig.fold(mainPages){ sidebar =>
     def flattenPages(p: StaticPageNode): Set[Path] =
       Set(p.template.file.toPath) ++ p.getChildren.asScala.collect { case p: StaticPageNode => flattenPages(p) }.flatten
 
     val mainFiles = mainPages.toSet.flatMap(flattenPages)
-    val docsPath = root.toPath.resolve("docs")
+
     val allPaths =
       if !Files.exists(docsPath) then Nil
       else Files.walk(docsPath, FileVisitOption.FOLLOW_LINKS).iterator().asScala.toList
 
-    val orphanedFiles = allPaths.filterNot(mainFiles.contains).filter { p =>
+    val orphanedFiles = allPaths.filterNot { p =>
+       def name = p.getFileName.toString
+       def isMain = name == "index.html" || name == "index.md"
+       mainFiles.contains(p) || (isMain && mainFiles.contains(p.getParent))
+    }.filter { p =>
         val name = p.getFileName.toString
         name.endsWith(".md") || name.endsWith(".html")
     }
 
     val orphanedTemplates = orphanedFiles.flatMap(p => loadTemplate(p.toFile, isBlog = false))
-
     mainPages ++ orphanedTemplates.map(templateToPage)
   }
 
@@ -114,13 +119,17 @@ class StaticSiteContext(val root: File, sourceSets: Set[SourceSetWrapper], args:
   private def loadSidebarContent(entry: Sidebar): LoadedTemplate = entry match
     case Sidebar.Page(title, url) =>
       val isBlog = title == "Blog"
-      val path = if isBlog then "blog" else url.stripSuffix(".html") + ".md"
-      val file = root.toPath.resolve(path) // Add support for .html files!
-      val LoadedTemplate(template, children, tFile) = loadTemplate(file.toFile, isBlog).get // Add proper logging if file does not exisits
-      LoadedTemplate(template.copy(settings = template.settings + ("title" -> title)), children, tFile)
+      val path = if isBlog then "blog" else
+        if Files.exists(root.toPath.resolve(url)) then url
+        else url.stripSuffix(".html") + ".md"
+
+      val file = root.toPath.resolve(path).toFile
+      val LoadedTemplate(template, children, _) = loadTemplate(file, isBlog).get // Add proper logging if file does not exisits
+      LoadedTemplate(template.copy(settings = template.settings + ("title" -> title), file = file), children, file)
+
     case Sidebar.Category(title, nested) =>
       // Add support for index.html/index.md files!
-      val fakeFile = new File(root, title)
+      val fakeFile = new File(new File(root, "docs"), title)
       LoadedTemplate(emptyTemplate(fakeFile, title), nested.map(loadSidebarContent), fakeFile)
 
   private def loadAllFiles() =