I am attempting to apply @benlaurie's objecthash idea to protos:
https://github.com/benlaurie/objecthash
I would like to use Merkle-ish hash trees to compute unique digests for every proto in a nested object graph. This allows protos to be authenticated without canonicalization, while also allowing portions of them to be authenticated in Merkle inclusion proofs e.g. by choosing a child object in an object tree and providing the necessary digests of the other objects in the tree to get back to the root hash of the object graph. This is described as "redaction" in objecthash.
Unfortunately, there's a pretty big snag to actually pulling this off in practice: extension fields. Without knowledge of the schema, older clients can't compute the objecthash of fields they don't understand, since they need to be able to see into the nested structure of the protos to compute the ultimate hash. One option is for older clients to ask for a proto containing only the fields they understand, but that only works in scenarios where they're consuming protos from an interactive service, and not if they were pulling them from e.g. a stream processing system like Kafka.
The rather unfortunate thing is we could almost compute objecthash for a proto in absence of the schema, save for one small annoying thing: it's not possible to tell, in absence of the schema, whether a given field contains a nested proto, or whether it's any other type of length-delimited data. They all use wire type 2:
https://developers.google.com/protocol-buffers/docs/encoding#structure
If there were a different wire type for telling "nested proto" versus "any other type of length-delimited data" this would be possible.
Has this been considered before?
I am attempting to apply @benlaurie's objecthash idea to protos:
https://github.com/benlaurie/objecthash
I would like to use Merkle-ish hash trees to compute unique digests for every proto in a nested object graph. This allows protos to be authenticated without canonicalization, while also allowing portions of them to be authenticated in Merkle inclusion proofs e.g. by choosing a child object in an object tree and providing the necessary digests of the other objects in the tree to get back to the root hash of the object graph. This is described as "redaction" in objecthash.
Unfortunately, there's a pretty big snag to actually pulling this off in practice: extension fields. Without knowledge of the schema, older clients can't compute the objecthash of fields they don't understand, since they need to be able to see into the nested structure of the protos to compute the ultimate hash. One option is for older clients to ask for a proto containing only the fields they understand, but that only works in scenarios where they're consuming protos from an interactive service, and not if they were pulling them from e.g. a stream processing system like Kafka.
The rather unfortunate thing is we could almost compute objecthash for a proto in absence of the schema, save for one small annoying thing: it's not possible to tell, in absence of the schema, whether a given field contains a nested proto, or whether it's any other type of length-delimited data. They all use wire type 2:
https://developers.google.com/protocol-buffers/docs/encoding#structure
If there were a different wire type for telling "nested proto" versus "any other type of length-delimited data" this would be possible.
Has this been considered before?