Server-side syntax highlighting

[silverwind]

Syntax highlighing is currently performed client-side which has a few drawbacks:

• We need to ship a 700kB library to the client which hurts performance
• There is a flash of unhighlighted code on every page load
• Language detection of highlight.js does not work based on filenames but on the code itself, giving a lot of false highlights (we could probably fix by mapping more file extensions, thought).

It may be worth investigating integrating https://github.com/alecthomas/chroma as server-side highlight. It supports language detection based on filename out of the box.

[lafriks]

I would like to have this but it needs to be checked on how that performs, especially on large files and diffs.
For language detection src-d/enry could be used that I'm using also for language stats

[silverwind]

src-d/enry

Chroma has such a filename/extension detection built-in and I would prefer those to be bundled together because if we use another module, those will eventually get out of sync and we'd have to start remapping languagues a lot.

I don't think content-based detection is actually necessary, GitHub doesn't do it too.

[lafriks]

Github does that using linguist imho same that enry is based on

[silverwind]

Github does that using linguist imho

I don't think GitHub does any content-based detection. Check out this file, it is YAML but not highlighted.

how that performs

I tested with the sqlite amalgamation which takes around 20s to render in the Chroma playground. I'd say its somewhat comparable to our rendering which in total takes around the same time, while syntax highlighting alone is a bit faster (10s), but the page flashes empty content multiple times for some reason.

[lafriks]

Github does that using linguist imho

I don't think GitHub does any content-based detection. Check out this file, it is YAML but not highlighted.

I think it does: https://github.com/go-gitea/gitea/blob/801843b0115e29ba2304fa6a5bea1ae169a58e02/contrib/init/debian/gitea

Yaml is just hard to have content detection on as it does not have specific keywords to detect just structure

[silverwind]

Interesting. I think that is some rudimentry detection based on shebang for shell scripts. Certainly a good thing to try on extensionless files.

[silverwind]

I think generally we want to try detecting the language in this order:

1. Detect based on file name
2. Detect based on file extension
3. Detect based on content

Chroma looks to combine steps 1 and 2 into one function.

[lafriks]

src-d/enry does language detection based on:

var DefaultStrategies = []Strategy{
	GetLanguagesByModeline,
	GetLanguagesByFilename,
	GetLanguagesByShebang,
	GetLanguagesByExtension,
	GetLanguagesByContent,
	GetLanguagesByClassifier,
}

[ukos-git]

from various syntax highlighting language experiences (pygments,chroma,vim,linguist (old tmlanguage, rouge) I really would recommend taking a look at rouge which is used by gitlab and has a really good and extendable highlighting interface that also accepts rather complex statements.

[lafriks]

@ukos-git it's in Ruby so we can't really use it

[lunny]

@ukos-git that's a ruby library but Gitea backend is written by Golang

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.