Repository navigation
BlobReadChannel should fail if content comes from different generations. #359
Description
Activity
I think we should do 2 things (could be done independently).
-
Provide a way to specify generation in
Storage.readerand if provided stick to it (We should include generation as part of the BlobId. #363 should help with that). -
If a generation is not provided, fail if we detect that generation was changed between rpc reads
(this could be done by changingStorageRpc.readto return a tuple (StorageObject,byte[]).
-
- added this to the milestone
on Nov 12, 2015 Regarding 2) I could not find a way of getting both data and metadata with the same request. If this is not possible (please confirm) should we implement 2) by getting the metadata before getting blob's data? I was thinking of something like (only if generation is null):
- First request
- Get blob latest metadata and store generation
- Get blob data for generation
- Return a pair (generation, data)
- Other requests
- Get Blob latest metadata and check that generation == generation (if not throw)
- Get blob data for generation
- Return a pair (generation, data)
- First request
@Capstan I also could not find a way to get both metadata and content in one request (I mean besides using a batch which I think is not going to work due to race condition).
Also, I don't think you can do that with the XML API (or at least I was not able to).
However, with some code change, we can get the etag (in the XML api we can also get X-Goog-Hash).
Do you think using (and comparing) the etag would be a sufficient check instead of using the generation value?If so, @mziccard this code can return it:
Get req = storage.objects().get(from.getBucket(), from.getName()); req.setIfMetagenerationMatch(IF_METAGENERATION_MATCH.getLong(options)) .setIfMetagenerationNotMatch(IF_METAGENERATION_NOT_MATCH.getLong(options)) .setIfGenerationMatch(IF_GENERATION_MATCH.getLong(options)) .setIfGenerationNotMatch(IF_GENERATION_NOT_MATCH.getLong(options)); StringBuilder range = new StringBuilder(); range.append("bytes=").append(position).append("-").append(position + bytes - 1); req.getRequestHeaders().setRange(range.toString()); ByteArrayOutputStream output = new ByteArrayOutputStream(); req.executeMedia().download(output); String etag = req.getLastResponseHeaders().getETag(); byte[] bytes = output.toByteArray();
BTW, the above code also "fix" #290 by not using the
MediaHttpDownloader.I know the XML API yields the generation you happen to read in the x-goog-generation header on data read responses.
Sorry @Capstan I am not very familiar with the XML API. It seems to me that it does not support the "*generation-not-match" options in general and that Object Download does not support the x-goog-if-metageneration-match, can you confirm any of this?
If that's true switching to the XML api for range reads will force us to drop some of the options that we now allow users to set.
True, it does not support the *generation-not-match options. Object download should support x-goog-if-metageneration-match. If not, that's a bug (certainly a docs bug, possibly a service bug).
sigh Found the internal issue tracking it. :/
You can specify not via preconditions, but via the generation urlparam to specifically get that generation.
Oh, yes, I see now that x-goog-generation is returned together with the content for XML API.
@Capstan though not a big code change I would rather not use the XML API for read operations if that does not support all the options that the Json API supports.
Do you think that using the object's etag (which is available in the response of the Json API) for checking
that content has not changed between read calls is not sufficient or not going to work?That should be sufficient. We don't claim that the ETag won't change if the content stays the same, but the content changes, the ETag should change.
Oh, but we don't want to have false negative as that would trigger an undesired read failure.
How likely is that ("etag changes but content is the same") to happen and what can trigger it?
10 remaining items
- added a commit that references this issue
on Jul 8, 2022 - added a commit that references this issue
on Aug 9, 2022 - added 2 commits that reference this issue
on Oct 5, 2022 - added 2 commits that reference this issue
on Feb 1, 2023 - added a commit that references this issue
on Jan 6, 2026 - added a commit that references this issue
on Jan 22, 2026 - added a commit that references this issue
on Mar 23, 2026 - added a commit that references this issue
on Mar 30, 2026 - added a commit that references this issue
on Apr 1, 2026
Feedback from @Capstan:
If someone updates the object when the client is in the middle of a read and hadn’t specified a precondition, you will simply read from the new object. You need to record the generation upon the first read and then choose what the behavior is for future reads… if the generation still exists (e.g., it’s now a history object), do you still allow reads or do you balk (because the original request is talking about the “current” object which has changed)?