Get Unique Entity Key String

[shaunc869]

When I view an entity in the web based datastore viewer I see a big long guid-esq string that I can normally use in Python to walk back to the entity without knowing its type, for example:

ahNzfnByb21wdGFwcG9pbnRtZW50chULEghTY2hlZHVsZRiAgICD2e7xxxxx

get me back a specific entity and with Python I can get to this by saying:

my_entity = ndb.Key(urlsafe="ahNzfnByb21wdGFwcG9pbnRtZW50chULEghTY2hlZHVsZRiAgICD2xxxx")

Can this be done with the gcloud nodejs library and/or is this entirely a bad method for some reason to begin with? Thanks!

[jgeewax]

That's a good question. What you're looking at is really just a (mostly) base64 encoded version of the entity_pb.Reference protobuf. Since a key can be encoded and decoded anyway you want, and the path (ie, ['Person', 1, 'Playlist', 1234]) is unique to the entity, you might want to consider just serializing the path and using that as a unique identifier.

So for example:

var path = ['Person', 1, 'Playlist', 1234];

var encodedPath = btoa(JSON.stringify(path)); // "WyJQZXJzb24iLDEsIlBsYXlsaXN0IiwxMjM0XQ=="
var decodedPath = JSON.parse(atob(encodedPath)); // ["Person",1,"Playlist",1234]

@ryanseys @stephenplusplus : Do you think it's worthwhile to add this method to the Key class so we don't make people implement it themselves?

[jgeewax]

(For reference, here is the code that deals with the urlsafe stuff in NDB: https://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/key.py#803)

[ryanseys]

Hmm, trying to see the use case for this. I think we'd suggest that gcloud-node be used primarily driven by API calls, and I don't see that this information (base64'd reference protobuf stuff) is directly accessible through the API, rather you'd have to do extraneous work to put it in that format or manually extract it from the web view, so it's likely an edge case we won't directly support. @jgeewax does it sound like I am understanding you correctly given what you've said above?

[shaunc869]

I can understand if this is an edge case, but in general I have worked with the datastore in both Java and Python, but it looks like the return values in both of these languages when trying to convert them to json come out a lot different than gcloud is there a reason why in Java and Python the format is a bit flatter and the keys don't include paths (see to_dict() method for example in Python)? I was thinking this might be why the guid keys were being used to keep the json a bit flatter, maybe?

Also I tried running the code above on a guid key and it doesn't seem to come out right, is that more pseudo code or should it work on a full key guid string?

Thanks for all the help!

[jgeewax]

@ryanseys If we think of this use case as ... "Make it easy for people to pass around key=" and retrieve it on a request, I think it'd be useful to have a method that serializes a string in something that is URL safe... That is, in express...

var express = require('express');
var gcloud = require('gcloud')({ projectId: 'my-project' });

var app = express();
var dataset = gcloud.datastore.dataset();

// GET /?key=WyJQZXJzb24iLDEsIlBsYXlsaXN0IiwxMjM0XQ==
app.get('/', function(req, res){
  dataset.get(dataset.key({encodedValue: req.query.key}, function(err, entity) {
    res.redirect('/otherPath?key=' + entity.key.encodedValue());
  });
});

app.listen(3000);

@shaunc869 What problems are you seeing with that same code...? Can you be more specific? I'm just using Javascript to JSON stringify a list and base-64 encode it... nothing gcloud-specific here.

[shaunc869]

@jgeewax I agree, that helper method would be awesome. I am trying to copy and past the entity key guid string from the datastore entity editor on the developers console and then base64 decode it and I am not getting the full path for some reason.

[jgeewax]

@shaunc869 That code wasn't a way of decoding from the exact same format as the entity editor -- it was just an example demonstrating how we might go about this. You could certainly figure it out assuming it's using the same encoding style that NDB does (NDB's decoding logic is here: https://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/key.py#803).

[shaunc869]

Ah that makes sense, I will try my best with that, but I generally wait for much smarter developers like you to figure this stuff out. I will take a crack at the ndb stuff and see what I can come up. If this was to be implemented here would you use the same algorithm as ndb, I was assuming this was some kind of standard?

[jgeewax]

I believe (I could be wrong) but all that's happening is base64 encoding with a couple find/replace characters (-'s and /'s?) and then chopping off trailing ='s.

[shaunc869]

I copied the exact code from that ndb code and ran it against the app engine console string and it does not come as a readable path. What am I doing wrong?

[jgeewax]

To help you, I'd need to see the exact code you're talking about... @shaunc869

[shaunc869]

I grabbed the ndb code:

def _DecodeUrlSafe(urlsafe):
  """Decode a url-safe base64-encoded string.

  This returns the decoded string.
  """
  if not isinstance(urlsafe, basestring):
    raise TypeError('urlsafe must be a string; received %r' % urlsafe)
  if isinstance(urlsafe, unicode):
    urlsafe = urlsafe.encode('utf8')
  mod = len(urlsafe) % 4
  if mod:
    urlsafe += '=' * (4 - mod)
  # This is 3-4x faster than urlsafe_b64decode()
  return base64.b64decode(urlsafe.replace('-', '+').replace('_', '/'))

And if you try to run this you don't get a path like I would expect, I must be missing something?

[jgeewax]

It's not returning the path -- it's returning the binary of a Reference protobuf. What I wrote above:

What you're looking at is really just a (mostly) base64 encoded version of the entity_pb.Reference protobuf.

So you'd need to then create the protobuf object ... and then read the path property from that... :(

It doesn't look like it'd be fun code to write or use (and seems like it's totally overkill when a path suffices) so it's really really unlikely that gcloud-node would go that route when "coming up with a serializable version" of a key. (We'd likely go the route of just serializing the path, since that's unique...)

Is there some specific reason that you need to emulate the exact same encoding of the key from the Datastore UI?