feat(storage): support bucket ip filter - #34612
shubhangi-google wants to merge 30 commits into
Conversation
|
Here is the summary of changes. You are about to add 6 region tags.
This comment is generated by snippet-bot.
|
Removed redundant explanation about uniform bucket-level access.
Removed unnecessary line break before accessing the IP filter.
There was a problem hiding this comment.
We cannot access the bucket after IP filter is enabled, right? So, we can still test enabling part.
There was a problem hiding this comment.
in tests we are testing for disabled only, in samples user can change it according to there requirement
added this particular test as it was requested
| # limitations under the License. | ||
|
|
||
| # [START storage_enable_bucket_ip_filter] | ||
| def enable_bucket_ip_filter bucket_name:, mode: "Enabled" |
There was a problem hiding this comment.
If we're enabling the ip_filter, we won't need to accept mode separately. Just like we have it for previous sample "disabling_ip_filter` .
There was a problem hiding this comment.
we included the mode parameter (which defaults to "Enabled") specifically for our acceptance tests. In the tests, we override this to "Disabled". If we enable the IP filter for real during the test run, it can lock the test runner out of the bucket, causing 403 Forbidden.
reference taken from PHP samples PR here: GoogleCloudPlatform/php-docs-samples#2221 (in enable_ip_filtering.php).
exact same pattern to get around the test lockout issue
There was a problem hiding this comment.
The CI test runner lockout concern makes sense, and parameterizing mode: "Enabled" with a default is a reasonable workaround.
To keep the snippet and test output consistent when mode: "Disabled" is passed, could we update the log message to dynamically reflect the mode (e.g. puts "Set IP filter mode to #{mode} for bucket #{bucket_name}." or puts "Configured IP filter for bucket #{bucket_name} (mode: #{mode}).") and adjust the test assertion accordingly?
| ip_filter = { | ||
| mode: "Disabled", | ||
| public_network_source: { | ||
| allowed_ip_cidr_ranges: [ |
There was a problem hiding this comment.
Shouldn't public_network_source be nil and vpc_network_sources be [] when disabling ip_filter?
There was a problem hiding this comment.
We are intentionally keeping mode: "Disabled" here while still providing the allowed_ip_cidr_ranges configuration. Just like with the enable sample, this is a workaround for our automated tests. We want to demonstrate to users how to attach an IP filter configuration during bucket creation, if we set the mode to "Enabled" in the snippet, the test runner immediately locks itself out of the new bucket and fails with a 403 Forbidden.
| mode: mode, | ||
| allow_all_service_agent_access: true, | ||
| public_network_source: { | ||
| allowed_ip_cidr_ranges: ["0.0.0.0/0", "::/0"] |
There was a problem hiding this comment.
Can you please also add a test to include multiple allowed_ip_cidr_ranges ?
|
QQ: Do we not use |
we can provide IP-filter configuration with mode Disabled |
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # [START storage_create_bucket_with_ip_filter] |
There was a problem hiding this comment.
Please update the sample region tags to match the standard Cloud Storage sample registry (storage.txtpb). Here and elsewhere in this PR.
| # limitations under the License. | ||
|
|
||
| # [START storage_delete_ip_filtering_rules] | ||
| def delete_bucket_ip_filter bucket_name: |
There was a problem hiding this comment.
Here, the sample should demonstrate removing specific CIDR/VPC rules from an existing configuration rather than setting mode: "Disabled".
| # [START storage_list_bucket_ip_filters] | ||
| def list_bucket_ip_filters | ||
| # The ID of your GCP project | ||
| # project_id = "your-project-id" |
There was a problem hiding this comment.
Please remove the unused # project_id = "your-project-id" comment to stay consistent with other list_buckets samples.
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| require_relative "../storage_helper" |
There was a problem hiding this comment.
Please change require_relative "../storage_helper" to require "storage_helper" in for consistency with all other acceptance tests in the repository.
| # limitations under the License. | ||
|
|
||
| # [START storage_enable_bucket_ip_filter] | ||
| def enable_bucket_ip_filter bucket_name:, mode: "Enabled" |
There was a problem hiding this comment.
The CI test runner lockout concern makes sense, and parameterizing mode: "Enabled" with a default is a reasonable workaround.
To keep the snippet and test output consistent when mode: "Disabled" is passed, could we update the log message to dynamically reflect the mode (e.g. puts "Set IP filter mode to #{mode} for bucket #{bucket_name}." or puts "Configured IP filter for bucket #{bucket_name} (mode: #{mode}).") and adjust the test assertion accordingly?
| mode: "Disabled", | ||
| public_network_source: { | ||
| allowed_ip_cidr_ranges: [ | ||
| "8.8.8.8/32" |
There was a problem hiding this comment.
This sample should focus on setting mode: "Disabled" without assigning arbitrary dummy CIDRs.
| if ip_filter | ||
| puts "Bucket #{bucket.name} has IP filter mode: #{ip_filter.mode}." | ||
| if ip_filter.public_network_source | ||
| puts "Allowed public network CIDR ranges: #{ip_filter.public_network_source.allowed_ip_cidr_ranges.join(', ')}." |
There was a problem hiding this comment.
allowed_ip_cidr_ranges may be nil when a network source is present without ranges, causing .join(', ') to raise a NoMethodError.
There was a problem hiding this comment.
handled the scenario
There was a problem hiding this comment.
Could we add a unit test for assigning bucket.ip_filter = hash directly outside of bucket.update? This will ensure test coverage for patch_gapi! :ip_filter when passing a Hash payload.
| require "google/cloud/storage" | ||
|
|
||
| storage = Google::Cloud::Storage.new | ||
| bucket = storage.bucket bucket_name |
There was a problem hiding this comment.
This does not pass projection: "full", which causes bucket.ip_filter to always return nil and silently skips the deletion block while still printing success. Additionally, we should only report success and trigger an update if the target CIDR was actually matched and removed from the configuration.
Could we fetch the bucket using projection: "full" and guard the update/log behind a check that confirms a rule was removed (similar to https://github.com/GoogleCloudPlatform/php-docs-samples/blob/main/storage/src/delete_ip_filtering_rules.php)?
|
|
||
| storage = Google::Cloud::Storage.new | ||
| ip_filter = { | ||
| mode: "Disabled", |
There was a problem hiding this comment.
mode: "Disabled" is hardcoded here, which means users copying the sample from documentation will create buckets with IP filtering disabled.
Could we use the same parameter pattern as enable_bucket_ip_filter (def create_bucket_with_ip_filter bucket_name:, mode: "Enabled") so the documentation defaults to "Enabled" while acceptance tests can pass "Disabled"?
| expected = "Deleted IP filter rule for bucket #{bucket_name}.\n" | ||
| retry_resource_exhaustion do | ||
| assert_output expected do | ||
| delete_bucket_ip_filter bucket_name: bucket_name |
There was a problem hiding this comment.
delete_bucket_ip_filter runs after disable_ip_filtering has already cleared the network sources, meaning no rule was present or deleted during the test.
Once delete_bucket_ip_filter is updated with projection: "full", could we reorder the steps or re-populate a rule before calling delete, and assert that the rule is absent from the bucket's refreshed metadata rather than asserting only on console output?
|
|
||
| # [START storage_list_buckets_ip_filtering] | ||
| def list_bucket_ip_filters | ||
| # The ID of your GCP project |
There was a problem hiding this comment.
This does not have an associated variable.
| # the bucket will include additional metadata, such as ACL policies and | ||
| # IP filter settings. | ||
| # | ||
| # @param [Hash] ip_filter The bucket's IP filter configuration. |
There was a problem hiding this comment.
ip_filter is typed as @param [Hash], but the parameter also accepts Google::Apis::StorageV1::Bucket::IpFilter. Could we update the type tag to @PARAM [Google::Apis::StorageV1::Bucket::IpFilter, Hash] ip_filter to match Bucket#ip_filter=?
kalragauri
left a comment
There was a problem hiding this comment.
The PR description still mentions "although some debugging code is still present, suggesting it is a work-in-progress" and omits the new projection: parameter added to Project#bucket, Project#buckets, Project#create_bucket, Bucket#reload!, and Bucket::List#next.
Please update the description to remove the WIP note and mention projection: support.
| ip_filter = bucket.ip_filter | ||
|
|
||
| if ip_filter | ||
| if ip_filter.public_network_source |
There was a problem hiding this comment.
Removing puts "Bucket #{bucket.name} has IP filter mode: #{ip_filter.mode}." means this sample no longer reports whether IP filtering is enabled or disabled, and produces no output at all when ip_filter is present without public_network_source (such as after disabling IP filtering or when only VPC sources are configured).
Could we restore printing ip_filter.mode under if ip_filter and update the expected output in samples/acceptance/buckets_test.rb to match both lines?
| end | ||
|
|
||
| describe "storage_bucket_ip_filter" do | ||
| let(:bucket_name) { random_bucket_name } |
There was a problem hiding this comment.
Because samples/acceptance/helper.rb uses minitest/hooks/default, after :all executes on a different instance than the it block where let(:bucket_name) is memoized. Calling bucket_name in after :all evaluates random_bucket_name a second time with a new random name, leaving the created test bucket undeleted.
Could we change after :all do to after do (matching storage_quickstart_test.rb) so teardown runs on the same test instance?
| bucket = storage_client.bucket bucket_name, projection: "full" | ||
| if bucket.ip_filter&.public_network_source&.allowed_ip_cidr_ranges | ||
| ranges = bucket.ip_filter.public_network_source.allowed_ip_cidr_ranges | ||
| refute_includes ranges, "0.0.0.0/0" |
There was a problem hiding this comment.
Wrapping refute_includes in if bucket.ip_filter&.public_network_source&.allowed_ip_cidr_ranges allows the test to pass without running the assertion if allowed_ip_cidr_ranges is nil, even though "::/0" should still be present.
Could we assert ranges = bucket.ip_filter.public_network_source.allowed_ip_cidr_ranges directly (e.g., refute_nil ranges and refute_includes ranges, "0.0.0.0/0")?
| assert_includes out, "Bucket Name: #{bucket_name}, IP Filtering Mode: Disabled" | ||
| end | ||
|
|
||
| # Deletes IP filter of an existing bucket (MOVED UP) |
There was a problem hiding this comment.
Pls remove (MOVED UP) from this comment.
There was a problem hiding this comment.
There are a few trailing whitespace and indentation issues in samples/storage_list_bucket_ip_filters.rb and samples/acceptance/buckets_test.rb. Please clean those up.
This pull request introduces the capability to configure IP filters for Google Cloud Storage buckets directly through the Ruby client library. It extends existing bucket management functionalities to allow specifying IP filter settings during bucket creation and provides methods to update these settings on existing buckets. The changes also include new sample code to demonstrate the usage of this feature.
Highlights
ip_filtergetter and setter methods to theGoogle::Cloud::Storage::Bucketclass, enabling programmatic access and modification of IP filter settings.create_bucketmethod inGoogle::Cloud::Storage::Projectto accept anip_filterparameter, allowing IP filter configuration during bucket creation.