JSON-LD Context processing in HTML Documents

[msporny]

From this issue in the Verifiable Claims Working Group with regard to the new "full Processor" conformance class: w3c/vc-data-model#585

@gkellogg wrote:

As this is something that may not be necessary in certain embedded environments, the notion of processor classes was introduced to allow a pure JSON Processor to conform without processing HTML. But, a full Processor is expected to do this.

@msporny wrote:

To be clear, I really dislike this feature of JSON-LD 1.1 because it raises the burden of Full JSON-LD 1.1 processors to contain an HTML processor (which is a massive requirement) on top of doing JSON-LD processing. I also think this is going to really damage the adoption of JSON-LD 1.1 and make it so much easier for people to argue against it... hell, even I would argue against "Full JSON-LD processors" (and plan to if this feature goes to REC).

@gkellogg wrote:

I appreciate your position, but JSON-LD in HTML is probably the biggest use case right now (although that will likely change with adoption of VC and WoT). JSON-LD in HTML is a reality that the spec needs to recognize and legitimize.

I agree that processing JSON-LD content in HTML is a primary use case and the WG should support it.

I disagree that people are publishing JSON-LD Contexts in HTML, that came out of nowhere. I can see what the WG is trying to do, but this issue is an example of my concern: w3c/vc-data-model#585

You have someone suggesting that we pull in a JSON-LD Context file via an HTML document without understanding the technical burden in doing so. They don't understand that publishing a JSON-LD Context as an HTML document will not require full processors.

I also note that expressing JSON-LD Contexts in HTML was not contemplated in any of the input documents to the JSON-LD WG and as such, the group is skirting very close to being in violation of their charter by adding this feature:

https://www.w3.org/2018/03/jsonld-wg-charter.html
https://github.com/json-ld/json-ld.org/wiki/Changes-in-Community-Group-Drafts-Targeted-for-1.1
https://json-ld.org/presentations/JSON-LD-Update-TPAC-2017/assets/player/KeynoteDHTMLPlayer.html

There are two major issues with this new set of features:

• Enabling JSON-LD Contexts in HTML documents will silently increase the burden of consuming JSON-LD by small form factor implementations (IoT, WoT, etc.). I haven't even considered the security implications here, but I can probably create something where a native JS processor uses a different JSON-LD Context than one that doesn't do DOM processing.
• There is an implied hierarchy of "good" and "not as good" in the new conformance classes. For example, it sounds like having a "full Processor" would be better than having a "pure JSON Processor".

Making the following changes to the specification would be an improvement:

• Rename "full Processor" to "HTML Processor".
• Remove the ability to use text/html files as JSON-LD Contexts as pure JSON Processors are not capable of processing them, which will lead to a variety of issues related to developer ergonomics.

[pchampin]

Rename "full Processor" to "HTML Processor".

I agree that the original naming may give the wrong impression (that other processors are somehow incomplete), and discourage some people from adopting JSON-LD. "HTML Processor" is a little misleading, but a better name could indeed be found.

Remove the ability to use text/html files as JSON-LD Contexts as pure JSON Processors are not capable of processing them, which will lead to a variety of issues related to developer ergonomics.

The argument was raised that JSON-LD Contexts are bona fide JSON-LD documents, and so it would be difficult to argue that a Full "Extended" processor could sometimes load JSON-LD from HTML, and sometimes not... I think this is a valid argument.

That being said, we could address your concern by replacing the Note, at the beginning of section 7, by a Warning, stating "not available in a Pure JSON-LD Processor" rather than "available in a Full Processor". And possibly hinting that content-negotiation is a more "portable" solution?...

[dlongley]

@pchampin,

And possibly hinting that content-negotiation is a more "portable" solution?...

I don't think we should merely "possibly hint" at this; my preference would be to make it a requirement that you MUST make your @context available as JSON. But, short of my own preferences, we should be very clear that you SHOULD do so and that if you don't, your @context won't work with every JSON-LD processor, only those that add the extra HTML feature set. I think we should be strongly encouraging JSON over HTML, but allow HTML for documentation purposes.

[msporny]

And possibly hinting that content-negotiation is a more "portable" solution?...

I feel stronger about this than @dlongley does... don't open up the Pandora's box of reading JSON-LD Context's from HTML. Remove the feature. The only argument that I can see for it is that it's a "neat feature" in the academic completeness sense... but JSON-LD was never meant to be an academically complete mechanism... it was supposed to help developers publish JSON-LD, but not become so complex that it blows your foot off when you try to use it. Having this feature means that developers will inevitably publish their JSON-LD Context as HTML only, which will cause a split in the ecosystem between "We expect you to publish via HTML" and "We expect you to publish no via HTML".

[msporny]

"HTML Processor" is a little misleading, but a better name could indeed be found.

Isn't the only feature that the "full" processor has over the JSON-only one the fact that it parses stuff from HTML?

[pchampin]

Isn't the only feature that the "full" processor has over the JSON-only one the fact that it parses stuff from HTML?

Yes, but "HTML Processor" makes it sound like it can only process HTML...

[gkellogg]

And possibly hinting that content-negotiation is a more "portable" solution?...

I feel stronger about this than @dlongley does... don't open up the Pandora's box of reading JSON-LD Context's from HTML. Remove the feature. The only argument that I can see for it is that it's a "neat feature" in the academic completeness sense... but JSON-LD was never meant to be an academically complete mechanism... it was supposed to help developers publish JSON-LD, but not become so complex that it blows your foot off when you try to use it. Having this feature means that developers will inevitably publish their JSON-LD Context as HTML only, which will cause a split in the ecosystem between "We expect you to publish via HTML" and "We expect you to publish no via HTML".

This was not added because it's a "neat feature", but as a response to concerns raised in #43. If JSON had a built-in commenting feature, it would be likely not necessary.

Because of this, and the need to normatively describe the in-the-wild JSON-LD in HTML scenarios provided a mechanism to do this. Once you describe JSON-LD in HTML, then allowing that for contexts and frames is a logical progression, particularly when the extraction is described in the document loader, which is the standard way to fetch all remote content.

The fact that it came up in w3c/vc-data-model#585 just goes to show a general need to be able to document contexts, and containing the context in the documenting HTML is likely a better way to keep them from diverging than using different resource formats.

I agree with @dlongley that we should better describe the potential for splitting the eco-system by recommending (SHOULD) that publishers provide an application/ld+json version via content-negotiation and not depend on a processor's conformance with HTML processing.

[azaroth42]

With chair hat on...

I also note that expressing JSON-LD Contexts in HTML was not contemplated in any of the input documents to the JSON-LD WG and as such, the group is skirting very close to being in violation of their charter

Could you point out where in the charter it says that we can only introduce features described in input documents to the WG? Because that would also preclude features like @protected, as far as I'm aware. I don't think that's, thus, relevant here unless you can find somewhere that says we're constrained in this way?

And with chair hat off ...

I agree with @gkellogg that if we say that a context is JSON-LD, and that JSON-LD can be expressed in a script element of an HTML page, then the implication is that a context can be expressed in a script element of an HTML page. If I recall correctly, @danbri has brought up his issue as a frustration of web developers.

The possible routes forward seem to be:

• Allow contexts in HTML with warnings and processor conformance statement [current]
• Keep contexts as JSON-LD, but introduce a rule that they cannot be in HTML [my understanding of Manu's proposal]
• Make contexts a separate, non-JSON-LD media type, with a rule they must be separate documents. [Alternative that I don't think anyone actually likes, and arguably backwards incompatible]
• Remain silent [I think it's too late for this - the issue has been raised, we have to address it]

I agree with @pchampin that "extended" is better than "full", along with a big warning about contexts in HTML being complicated in the spec.

[BigBlueHat]

JSON-LD in HTML exists and even informatively--when viewed from the HTML-perspective: https://html.spec.whatwg.org/#the-script-element:attr-script-type-4

In the current spec-space, it's already possible to extract JSON-LD from HTML and use it as JSON(-LD)--because that's how data blocks work with any embedded format (CSV, YAML, etc.).

We have gone beyond simply echoing that fact in the syntax document and instead baked additional processing steps into the API.

Shifting things into the documentLoader space does help from an architectural layering concern, but this "context in HTML" usage raises a whole host of architectural and community concerns. It effectively moves us from the current world of extracting-then-using the embedded JSON-LD into one where HTML becomes a valid representation of JSON-LD itself.

We need to work to re-narrow our focus at this stage, and go back to the "simplest thing that could possibly work."

[iherman]

This issue was discussed in a meeting.

• No actions or resolutions

View the transcript

Contexts in HTML
Ivan Herman: #172
Rob Sanderson: Summary: in the spec we say that (normatively) json-ld can be included in script el. There is now a requirement on <base>. It was noted that contexts are also jsonld. Hence, it is permissible to have contexts embedded in script tags inside html. This means that processors need to be able to process that.
… We all agree that this is an extension to normal proc mode. Either we need to say that contexts have a special role, contexts are not jsonld, or we need to accept that contexts can be embedded in html and processors should have to be able to say that they support processing them.
Manu Sporny: Some context wrt VC. Purely json-based processors find information using context. Someone said it would be nice to have human-readable context. Argument in favor of this feature.
… Person said, It would be nice to see jsonld in html. But I don’t want the burden of jsonld processor supporting html.
… We all agree that JSON-LD in HTML is a huge use case (e.g. schema.org)
… I think pulling in contexts from html is controversial
… 2 questions
… 1: does jsonld context in html greatly increase jsonld usage?
… I think the answer is no
… There are other ways to solve issues people would have to want html for contexts.
… 2: is this going to create interop issues?
… Is this going to cause ecosystem to change by other processors to start failing?
… I think this is going to create issues.
… Some people are going to start publishing contexts as html only.
… Even if we say you should not do this.
… The damage this feature could create is far greater than possible benefits.
… I have more reasons, but this is the biggest argument.
… We should wait until there is more demand for this feature. We could do it in the future if really needed.
Benjamin Young: “This means that processors need to be able to process that.”
Benjamin Young: +1 to everything Manu said.
… This is the part of what Rob said in start that jsonld in html normative somehow begets this notion that we have to …
… jsonld in html has always been normative thanks to the data block in script tag
… we just described it better
… comes from HTML5 spec.
… Using single URL to specify context and its documentation is interesting. (Conneg can be used)
… Overhead of making this possible is too big for processors.
… This is a nuclear weapon to kill a small bird.
… There are less damaging ways to solve the problems discussed.
Dave Longley: +1 to manu and bigbluehat
Manu Sporny: +1 to bigbluehat !
Benjamin Young: We need to be more careful than we have before, before introducing new things like these.
Rob Sanderson: ref - https://www.w3.org/TR/2018/WD-json-ld11-20181214/#embedding-json-ld-in-html-documents
Rob Sanderson: in 1.1, we made it our problem, so we have to solve it.
… I want to channel danbri. Search engines want to include info in their knowledge graph that they find on the web as jsonld.
… schema.org, or at least the engines, currently assume do not process contexts at all.
… If you have a template in your website, with multiple schema.org definitions, you could put into your CMS as a data script block to push this into every single page.
… search engines would be able to see these blocks
… by having google’s clusters waiting to process jsonld in page. Publishers would be required to not embed into page.
… why not have it as include contexts object?: when multiple people responsible for editing context. Also, if there are templare-driven CMSs being used, you want to stripe jsonld over different templates being used. This would cause data blocks being used multiple times.
Dave Longley: Many of these things can be solved by saying that serving should happen with application/ld+json
… I think there are many cases not being considered wrt complexity
… many use cases not covered on template-based html pages
… Such as dynamic pages when generated client-side with javascript
… We shouldn’t get into that space.
… We should say that context MUST be server with proper content type
Manu Sporny: I could not follow schema.org use case. Danbri should write this up. We should do a deep analysis on this use case, to see what could address his concern.
… There are a bunch of assumptions in that use case
… e.g. people could create their own non-schema.org contexts. This would add a huge amount of complexity.
… it would be good to have dan involved.
Rob Sanderson: +1 to dlongley and manu
Manu Sporny: Also, it feels like this is migrating away from BPs.
… We are learning a lot from security around publishing contexts.
… We had discussions on the type of attacks, if you could publish contexts as html.
… So there are security concerns around this feature
… Concern around complexity, interoperability, …
… A long list of reasons for saying that this is not spec-ready.
… So we have to get use-case right. And see if it can be solved with current feature-set. Only if needed, we should look further into this html issue.
Ivan Herman: Manu said many things what I wanted to say. We need danbri to raise his voice.
… We have to rely on documented requirements
Rob Sanderson: I agree
Benjamin Young: I think what you described, if it’s on danbri’s previous desire to have this in jsonld, then this is a request. Dan has expressed multiple times that search engines want to understand page contents. This is different to giving identifier that serves contexts in html.
… We are going to end up with RDF dataset that is compiled of multiple contexts.
Gregg Kellogg: no, doesn’t work that way
Benjamin Young: Generating a graph is not about coupling jsonld context identifier algorithm.
Gregg Kellogg: I don’t think it is practical for many CMSs to do content negotiation (like github pages)
… we have to re-characterize what jsonld in html is.
… I agree that these things start to increase complexity and raising barrier.
… We need to reconsider what processing jsonld in html means.
Manu Sporny: +1 to re-characterize how processors process JSON-LD in HTML.
Rob Sanderson: We are not going to solve this today.
… I will reach out to danbri to see if he wants to engage.
Gregg Kellogg: He may be at WebConf

[azaroth42]

From @danbri, posted with permission, after discussion with @gkellogg:

1. We are uncomfortable that our site (by virtue of our context url) has implicitly become a software component in a system where we don't even really know the other software components. I am considering turning off the context serving at weekend to encourage caching and more robust clients.

1b. Aside: it could be interesting to have a best practice note about how software components fetching contexts might identify themselves incl versions in http requests (user agent)

2. We are unhappy that the expectation of content negotiation on our home page blocks us from moving to 100% static-served site.

3. If we could have a small snippet of jsonld in our homepage, pointing off to a separate url with our giant big context file, that would be great

4. We are not interested in putting the whole context into our homepage; it is way too big. Similar issues may hold for Wikidata at some point.

5. We appreciate the reluctance to entangle the pure json nature of json-ld with html, but note that the success of json-ld was achieved in large part through just such an entangling

[rubensworks]

1b. Aside: it could be interesting to have a best practice note about how software components fetching contexts might identify themselves incl versions in http requests (user agent)

👍 Related to this, that best practise note should also talk about caching of contexts.

We are unhappy that the expectation of content negotiation on our home page blocks us from moving to 100% static-served site.

One possible solution for this would be to allow a link header to be added to HTML documents that points towards contexts.
(This may not solve all static site use cases though, as platforms like GitHub pages don't support custom link headers AFAIK)

[msporny]

We are uncomfortable that our site (by virtue of our context url) has implicitly become a software component in a system where we don't even really know the other software components. I am considering turning off the context serving at weekend to encourage caching and more robust clients.

I think that this would be a good thing to do. Provide guidance on aggressively caching the schema.org context (or packaging it with software implementations).

[msporny]

We are unhappy that the expectation of content negotiation on our home page blocks us from moving to 100% static-served site.

Then state that the new schema.org context will be served from: "https://schema.org/v1" -- make that the context, say that "https://schema.org/" is an alias for "https://schema.org/v1" and note that you will turn off content negotiation for "https://schema.org/" at the beginning of 2020.