RDF Deserialization and @index

[vcharpenay]

The use case I am going to present comes from the Web of Things W3C WG. Its main deliverable is the Thing Description model, for which we would like to define a JSON-LD serialization.

Currently, we have a pre-processing algorithm that adds @id, @type and @context keys to model elements. The model includes a number of indexed definitions, for which @index works just fine. However, it happens that @index keys are also used as identifiers for the definition they map to and we would like to be able to query them among a collection of TD documents stored in an RDF store. For instance, assuming the following TD document:

{
  "@context": "http://www.w3.org/ns/td",
  "@type": "Thing",
  "properties": {
    "temperature": { "type": "number" },
    "onOff": { "type": "boolean" }
  }
}

we'd like the following query to return something like "temperature" instead of a blank node:

select ?index where {
  ?thing a td:Thing ;
      td:properties ?index .
  ?index a jsonschema:NumberDataSchema .
}

In other words, we'd like to have a way to keep @index keys in the RDF graph after JSON-LD deserialization. As far as I know, it is currently not possible. We thought of two possible approaches: ID indexing or extended RDF deserialization.

ID indexing

Properties in the example above could be indexed by @id and not by @index. However, this solution comes with 2 issues: first, we would need to define a @base for these identifiers (ideally the root node's @id0 and, second, we have stumbled many times upon the problem that identifiers collide because the same index key is used several times in the same TD document. Our current solution is to assign a unique JSON pointer to all indexed definitions. Can this approach be standardized?

Extended RDF deserialization

Index values could be included in the RDF graph, using some RDF term defined under the jsonld namespace. Something like:

_:thing a td:Thing ;
    td:properties _:p1, _:p2 .
_:p1 a jsonschema:NumberDataSchema ;
    jsonld:indexKey "temperature"^^xsd:string .
_:p2 a jsonschema:BooleanDataSchema ;
    jsonld:indexKey "onOff"^^xsd:string .

Is that something that is within the scope of JSON-LD 1.1? It would make the full transformation to RDF fully bidirectional, as far as the TD model is concerned.

(See also the related issue w3c/wot-thing-description#444.)

[pchampin]

ID Indexing

How is it different from https://w3c.github.io/json-ld-syntax/#node-identifier-indexing ?

Extended RDF deserialization

This looks like a nasty can of worm. What if some bnodes miss the jsonld:indexKey? What if some have multiple values?

[vcharpenay]

Thanks for your comment!

How is it different from https://w3c.github.io/json-ld-syntax/#node-identifier-indexing ?

yes, that is essentially it. I raised that issue to introduce @container: @id some time ago but it turned out it did not solve completely our use case.

This looks like a nasty can of worm. What if some bnodes miss the jsonld:indexKey? What if some have multiple values?

I agree it looks a bit esoteric, but the same rules would apply as when compacting nodes without @index member : if the triple is not present, the node get indexed by @none and if there is more than one node under the same index key, an array is instantiated. By the way, I was glad to see that this corner case was already addressed in JSON-LD 1.1 for pure compaction.

[gkellogg]

I could see a toRdf option which would serialize @index as something like jsonld:index with a string value, which would require a reciprocal option to fromRdf to reverse it. As it is @index is the only syntactic element which is not described by the RDF model.

As it's simply an extra triple, we could conceivably do it without an option (or with an option defaulting to true), as generating more triples is typically not an issue for serializers.

[gkellogg]

if the triple is not present, the node get indexed by @none and if there is more than one node under the same index key, an array is instantiated.

Note that this won't be the case for expansion on serialization to RDF, as the @none key is used only in the index-map case to index nodes without an @index.

As an alternate, we could re-consider the notion of indexing on an arbitrary property. There were no good use cases, and it would add a fair bit of complexity, so it was not followed up on.

You might also consider @type indexing, which allows you to use the vocabulary space for values. It would require assigning an appropriate type to each entity, but you can use scoped contexts to help manage that. See Note Type Indexing.

[iherman]

As it's simply an extra triple, we could conceivably do it without an option (or with an option defaulting to true), as generating more triples is typically not an issue for serializers.

I'd prefer it this way. An extra flag may become a drag, easy to forget and leading to unexpected errors. An extra triple isn't a big deal.

The only caveat: as far as I know, jsonld:index would be the only RDF resource appearing in the generated graph in a jsonld namespace. That may open the floodgates for other terms...

[azaroth42]

I would prefer an option defaulting to True to reflect the index term into the graph. This way it can be turned off if needed, but to Ivan's point you don't need to remember to turn it on.

[pchampin]

@iherman wrote:

The only caveat: as far as I know, jsonld:index would be the only RDF resource appearing in the generated graph in a jsonld namespace. That may open the floodgates for other terms...

I'm not a big fan of this either. Ideally, I would much rather have "a notion of indexing on an arbitrary property" as suggested by @gkellogg. Although this looks like a can of worm we may not want to open...

[gkellogg]

I'd say keep it simple, and issue such an extra triple. There are plenty of precedents for specs using their own namespace for creating such types or properties; of course an option could default to the jsonld:index IRI, and allow null or some other IRI, but that would seem to be overly flexible.

[davidlehn]

Take into consideration the effect on canonicalization since triples are being added.

[iherman]

Take into consideration the effect on canonicalization since triples are being added.

which probably means that this should not be an optional feature.

[pchampin]

The more I think about it, the more I think that the "indexing by arbitrary property" approach is better.

It could work like this:

{
  "@context": {
    "ex": "http://ex.co/",
    "foo": {
      "@id": "ex:foo",
      "@container": "@index",
      "@index": "ex:myIndex"  // <--- this is the proposed new feature
    }
  },
  "foo": {
    "bar": {
      "@id": "#BAR"
    },
    "baz": {
      "@id": "#BAZ"
    }
  }
}

This would expand to

[
  {
    "http://ex.co/foo": [
      {
        "@id": "https://json-ld.org/playground/#BAR",
        "http://ex.co/myIndex": "bar"  // <--- instead of "@index": "bar"
      },
      {
        "@id": "https://json-ld.org/playground/#BAZ",
        "http://ex.co/foo": "baz"   // <--- instead of "@index": "baz"
      }
    ]
  }
]

and so additional triples would be materialized, holding the index values.

This proposal does not add any keyword (we just reuse @index in a different place). It extends the current spec in a rather natural way, in my opinion. What I might underestimate is the amount of complexity it adds to the standard algoithms in the API document...

[iherman]

I like this. What this means is that the author would have to make one step further indicating (a) whether he/she wants to have the index appear in the generated RDF and (b) using what name.

@vcharpenay would that be a viable option for you?

[vcharpenay]

Thanks for the discussion, I like the various suggestions without any particular preference. I must say, though, that @pchampin's proposal turns out to be very close to something I suggested some time ago: see json-ld/json-ld.org#430 (case 2).

However, @gkellogg already spent some time trying to implement it and it indeed comes with significant complexity, in particular w.r.t. compaction. See his comment.

[gkellogg]

I agree that it would be better to allow indexing on arbitrary properties. When looking at it, I thought that it had added a fair amount of complexity to the expansion/compaction algorithms, but it may end up being simpler than that. Ultimately, we'll need to decide if the added complexity is worth the value.

Looking at @pchampin's proposal, it looks to me that specifying a value for @index in the term definition would be interpreted to mean use the value of that property (if any) instead of the value of @index when expanding and compacting, which also leaves some consideration if there are multiple such values.