Sitelet https://web.archive.org/web/20230106042727/https://github.com/sourcegraph/sourcegraph/pull/45311
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

all: entirely remove minio support #45311

Merged
merged 2 commits into from Dec 7, 2022
Merged

all: entirely remove minio support #45311

merged 2 commits into from Dec 7, 2022

Conversation

slimsag
Copy link
Member

@slimsag slimsag commented Dec 7, 2022 •

IMPORTANT: Please note this change is NOT active for customers until the CHANGELOG is updated and clearly states the removal of this for a specific version of Sourcegraph.

This change entirely removes minio support from our codebase, in favor of the new Apache-licensed sourcegraph/blobstore Docker image:

  • All deployment types already no longer use minio, including: * Single container sourcegraph/server * Docker Compose * Pure Docker * Kubernetes deployment types (helm, kustomize, vanilla, etc.) * AMI/image-based 'one click' deployments and script-install deployments
  • Removed the Docker image so we no longer distribute any minio code for new Sourcegraph versions in the future.
  • Removed the ability of Sourcegraph to even configure/use minio in any form.
  • Updated third-party license information to reflect the change.

Helps #44254

Signed-off-by: Stephen Gutekanst stephen@sourcegraph.com

Test plan

Existing tests / CI

IMPORTANT: Please note this change is NOT active for customers until the
CHANGELOG is updated and clearly states the removal of this for a specific
version of Sourcegraph.

This change entirely removes minio support from our codebase, in favor
of the new Apache-licensed `sourcegraph/blobstore` Docker image:

* All deployment types already no longer use minio, including:
    * Single container `sourcegraph/server`
    * Docker Compose
    * Pure Docker
    * Kubernetes deployment types (helm, kustomize, vanilla, etc.)
    * AMI/image-based 'one click' deployments and script-install deployments
* Removed the Docker image so we no longer _distribute_ any minio code for
  new Sourcegraph versions in the future.
* Removed the ability of Sourcegraph to even configure/use minio in any form.
* Updated third-party license information to reflect the change.

Helps #44254

Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
@slimsag slimsag requested review from efritz, eseliger and bobheadxi Dec 7, 2022
@cla-bot cla-bot bot added the cla-signed label Dec 7, 2022
@sourcegraph-bot
Copy link
Contributor

sourcegraph-bot commented Dec 7, 2022 •

Codenotify: Notifying subscribers in CODENOTIFY files for diff 710ae45...1ba9b22.

Notify File(s)
@Strum355 enterprise/internal/codeintel/shared/lsifuploadstore/config.go
enterprise/internal/codeintel/shared/lsifuploadstore/config_test.go
@bobheadxi enterprise/dev/ci/images/images.go
monitoring/definitions/containers.go
@efritz enterprise/internal/codeintel/shared/lsifuploadstore/config.go
enterprise/internal/codeintel/shared/lsifuploadstore/config_test.go
@sourcegraph/delivery doc/admin/observability/dashboards.md
monitoring/definitions/containers.go

@sourcegraph-bot
Copy link
Contributor

sourcegraph-bot commented Dec 7, 2022 •

Codenotify: Notifying subscribers in OWNERS files for diff 710ae45...1ba9b22.

Notify File(s)
@sourcegraph/dev-experience enterprise/dev/ci/images/images.go

Copy link
Member

@bobheadxi bobheadxi left a comment

:blobdance:

@@ -10,5 +10,5 @@ Zoekt,Apache License 2.0,https://github.com/sourcegraph/zoekt
Comby,Apache License 2.0,https://github.com/comby-tools/comby/
Alpine Linux,GNU General Public License,https://alpinelinux.org/
Tini,MIT License,https://github.com/krallin/tini
Minio,GNU Affero General Public License v3.0,https://min.io/
Copy link
Contributor

@efritz efritz Dec 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

efritz
efritz approved these changes Dec 7, 2022
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
@slimsag slimsag merged commit d859067 into main Dec 7, 2022
18 checks passed
@slimsag slimsag deleted the sg/no-minio branch Dec 7, 2022
slimsag added a commit that referenced this pull request Dec 7, 2022
* all: entirely remove minio support

IMPORTANT: Please note this change is NOT active for customers until the
CHANGELOG is updated and clearly states the removal of this for a specific
version of Sourcegraph.

This change entirely removes minio support from our codebase, in favor
of the new Apache-licensed `sourcegraph/blobstore` Docker image:

* All deployment types already no longer use minio, including:
    * Single container `sourcegraph/server`
    * Docker Compose
    * Pure Docker
    * Kubernetes deployment types (helm, kustomize, vanilla, etc.)
    * AMI/image-based 'one click' deployments and script-install deployments
* Removed the Docker image so we no longer _distribute_ any minio code for
  new Sourcegraph versions in the future.
* Removed the ability of Sourcegraph to even configure/use minio in any form.
* Updated third-party license information to reflect the change.

Helps #44254

Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
sashaostrikov pushed a commit that referenced this pull request Dec 7, 2022
* all: entirely remove minio support

IMPORTANT: Please note this change is NOT active for customers until the
CHANGELOG is updated and clearly states the removal of this for a specific
version of Sourcegraph.

This change entirely removes minio support from our codebase, in favor
of the new Apache-licensed `sourcegraph/blobstore` Docker image:

* All deployment types already no longer use minio, including:
    * Single container `sourcegraph/server`
    * Docker Compose
    * Pure Docker
    * Kubernetes deployment types (helm, kustomize, vanilla, etc.)
    * AMI/image-based 'one click' deployments and script-install deployments
* Removed the Docker image so we no longer _distribute_ any minio code for
  new Sourcegraph versions in the future.
* Removed the ability of Sourcegraph to even configure/use minio in any form.
* Updated third-party license information to reflect the change.

Helps #44254

Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
slimsag added a commit that referenced this pull request Dec 7, 2022
* all: introduce sourcegraph/blobstore; add option to cmd/server (#44255)
* all: add new sg blobstore backend to supersede minio
* cmd/server: optionally run sg blobstore backend over minio
* lsifuploadstore: add minio back to help text
* sg generate
* uploadstore: add DeleteObjects / NewListObjectsV2Paginator to s3API interface
* uploadstore: regenerate mocks
* uploadstore: add gcsAPI.Objects listing API
* uploadstore: regenerate mocks
* uploadstore: add Store.ExpireObjects stub implementation (thanks eric)
* uploadstore: implement ExpireObjects in GCS/S3/Lazy interfaces
* worker: integrate lsifuploadstore expirer
* uploadstore: update test API calls
* uploadstore: add TODO for myself
* uploadstore: batch S3 object deletions
* uploadstore: use operation logger instead of log15
* cmd/server: use sglog
* cmd/server: fix image build
* blobstore: build s3proxy from source ourselves; use our base alpine image (#44976)
* blobstore: build s3proxy from source ourselves; use our base alpine image
* sg.config: remove ineffective arguments
* cmd/server: remove minio support (default to blobstore) (#44979)
* cmd/server: remove minio support (default to blobstore)
* revert both blob store PRs (#44991)
* Revert "blobstore: build s3proxy from source ourselves; use our base alpine image (#44976)"

This reverts commit c5181d0.

* Revert "cmd/server: remove minio support (default to blobstore) (#44979)"

This reverts commit e49509f.

* server: use sg/log for minio
* blobstore: bring changes back (#45014)
* bring back blob store PRs (#44991)

This reverts commit 5c5ccff.

* blobstore: add missing env var
* Revert "blobstore: bring changes back (#45014)" (#45024)

This reverts commit e03ca0f.

* cmd/server: correct usage of blobstore backend (#45029)
* Revert "Revert "blobstore: bring changes back (#45014)" (#45024)"

This reverts commit afc1201.

* cmd/server: correct usage of blobstore backend
* cmd/server: correct s3proxy authorization method
* cmd/server: fix blobstore port

* cmd/server: correct blobstore env vars / fix CI (#45037)

cmd/server: correct blobstore env vars

* dev: cache blobstore image building in dev (#45065)

* {blobstore, uploadstore}: general cleanup (#45042)

* blobstore: default S3PROXY_AUTHORIZATION=none
* cmd/server: write blobstore env var overrides + default values more cleanly
* uploadstore: remove lifecycle configuration in favor of self-managing expiration

* blobstore: use port 9000 by default (#45105)

All deployments will have blobstore serve on port 9000, so default to
this in the Dockerfile so it doesn't need to be configured everywhere.

Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>

* make dev blobstore data/log dirs writable (#45124)

For me, the ~/.sourcegraph-dev/data/blobstore directory was chmod 775, which meant that the blobstore Docker container could not write to it. This caused errors:

- `failed to create bucket: operation error S3: CreateBucket, https response error StatusCode: 403` from the precise-code-intel-worker service
- `sendSimpleErrorResponse: 403 AccessDenied Forbidden` in the blobstore Docker container logs

Making these directories world-writable fixes the problem.

* fix s3proxy (blobstore) port in sg config (#45117)

The blobstore Docker image specifies a container listen port of 9000 (`S3PROXY_ENDPOINT="http://0.0.0.0:9000"`), but this `docker run --publish` flag in sg.config.yaml points to a container port of 80. This commit updates sg.config.yaml to use the correct port (9000).

* blobstore: fix /data file permissions (#45226)

As Quinn noted in an earlier PR, if `/data` is not writable by the container
then we encounter these errors:

- `failed to create bucket: operation error S3: CreateBucket, https response error StatusCode: 403` from the precise-code-intel-worker service
- `sendSimpleErrorResponse: 403 AccessDenied Forbidden` in the blobstore Docker container logs

Docker inherits the permissions of the folder if it is created in the Dockerfile
and I forgot to include the creation of the `/data` folder previously. Fixing
this means we don't e.g. have to manually correct permissions in Docker Compose
or pure-docker deployment types.

* all: entirely remove minio support (#45311)

* all: entirely remove minio support

IMPORTANT: Please note this change is NOT active for customers until the
CHANGELOG is updated and clearly states the removal of this for a specific
version of Sourcegraph.

This change entirely removes minio support from our codebase, in favor
of the new Apache-licensed `sourcegraph/blobstore` Docker image:

* All deployment types already no longer use minio, including:
    * Single container `sourcegraph/server`
    * Docker Compose
    * Pure Docker
    * Kubernetes deployment types (helm, kustomize, vanilla, etc.)
    * AMI/image-based 'one click' deployments and script-install deployments
* Removed the Docker image so we no longer _distribute_ any minio code for
  new Sourcegraph versions in the future.
* Removed the ability of Sourcegraph to even configure/use minio in any form.
* Updated third-party license information to reflect the change.

Helps #44254

Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>
Co-authored-by: Kalan <51868853+kalanchan@users.noreply.github.com>
Co-authored-by: Noah S-C <noah@sourcegraph.com>
Co-authored-by: Quinn Slack <quinn@slack.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants