fix: retry the Go module fetch in every Go sample's Dockerfile - #245
Open
slayerjain wants to merge 3 commits into
Open
slayerjain wants to merge 3 commits into
slayerjain wants to merge 3 commits into
Conversation
proxy.golang.org now and then resets an HTTP/2 stream partway through a download: go: golang.org/x/text@v0.31.0: read "https://proxy.golang.org/...": stream error: stream ID 187; INTERNAL_ERROR; received from peer The next attempt succeeds. But each Go sample's Dockerfile fetched its modules exactly once, with a bare `go mod download`, a bare `go mod tidy`, or a `go build` that fetches what it needs. So one reset failed the image build, and with it the CI lane of whichever repo was building the sample: the CI of keploy/keploy and of other keploy repositories clones this repo and runs `docker build` on these Dockerfiles. Each fetch now runs in a loop: 5 attempts, 5-20s apart, and then the build fails. A module fetch is idempotent and fails because of the network, so retrying it is safe. A missing module or a bad go.mod fails every attempt and still fails the build, about 50s later. - The 19 bare `go mod download`s and go-grpc's two `go mod tidy`s run in the loop. - echo-sql, gin-redis, mux-mysql, sse-svelte and users-profile fetched inside `go build`. They now run the looped `go mod download` first. - nethttp-mysql's `go mod download || true` ignored a failed fetch and left it to an un-retried `go mod tidy`. Both run in the loop now, so a fetch that fails five times fails the build. Built all 27 Dockerfiles: 25 build, and gin-pulsar and gin-redis fail for the same reason as before this change: their image's Go is older than their go.mod requires, which a separate commit fixes. With a GOPROXY that cuts the first download of one module partway, the original go-docker-timefreeze and go-grpc server builds fail ("unexpected EOF" on golang-jwt/jwt v5.3.0 and on grpc v1.69.2), and the looped ones recover on the second attempt. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Neither image built. gin-redis's go.mod says `go 1.23.0` and `toolchain go1.23.1`, which the Go in golang:1.20 cannot parse: go: errors parsing go.mod: /app/go.mod:3: invalid go version '1.23.0': must match format 1.23 /app/go.mod:5: unknown directive: toolchain gin-pulsar's go.mod says `go 1.25.0`, which golang:1.23 refuses: go: go.mod requires go >= 1.25.0 (running go 1.23.12; GOTOOLCHAIN=local) Each now builds on the Go minor release its go.mod requires, golang:1.23 and golang:1.25, and both images build. gin-pulsar's README asked for Go 1.23+ as well; it now asks for 1.25+. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
build.yml runs `go build ./...` in every top-level sample with a
go.mod, and that build fetched the sample's modules from
proxy.golang.org with no retry. One reset stream ("stream error: stream
ID N; INTERNAL_ERROR; received from peer") would fail the whole job.
Each sample now runs `go list -deps ./...` first, in a loop: 5
attempts, 5-20s apart, and then the job fails. `go list -deps` loads
the packages `go build` loads, so it fetches the same modules and no
more. `go mod download` would also fetch every module go.mod requires,
whether the build needs it or not.
Run as build.yml runs it, with Go 1.27.0 and an empty module cache, all
44 modules build, both with and without this change. With a GOPROXY
that cuts the first download of golang-jwt/jwt partway, the old step
fails on go-docker-timefreeze ("unexpected EOF"), and the new one
recovers on the second attempt.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
proxy.golang.org now and then resets an HTTP/2 stream partway through a module download:
The next attempt succeeds. But every Go sample's Dockerfile here fetched its modules exactly once, using a bare
go mod download, a barego mod tidy, or ago buildthat fetches what it needs. One reset therefore fails the image build. It also fails the CI lane of whichever keploy repo was runningdocker buildon the sample: keploy/keploy's workflows and other keploy CI clone this repo atmainand build these Dockerfiles. This repo's ownbuild.ymlhas the same exposure through its barego build ./....Changes
fix: retry the Go module fetch in every Go sample's Dockerfile(27 Dockerfiles). Each fetch now runs in a loop: 5 attempts, 5-20s apart, then the build fails.go mod downloads and go-grpc's twogo mod tidys run in the loop.go build. They now run the loopedgo mod downloadfirst.go mod download || trueignored a failed fetch and left it to an un-retriedgo mod tidy. Both are looped now.A module fetch is idempotent and fails only because of the network, so retrying it is safe. A missing module or a bad go.mod fails every attempt and still fails the build.
fix(gin-redis, gin-pulsar): build them on a Go their go.mod accepts. Neither image built before this PR. gin-redis's go.mod needs Go 1.23 but the image wasgolang:1.20; gin-pulsar's needs Go 1.25 but the image wasgolang:1.23.ci: retry each sample's module fetch before building it.build.ymlnow runsgo list -deps ./...in a loop, the same 5 attempts, before eachgo build ./....go list -depsloads the same packagesgo buildloads, so it fetches nothing more.Verification
docker build --checkreports no warnings on any of them.go-docker-timefreezeandgo-grpcserver Dockerfiles fail withunexpected EOF(on golang-jwt/jwt v5.3.0 and on grpc v1.69.2). The looped ones recover on the second attempt.build.yml's step: the old step fails on go-docker-timefreeze, and the new one recovers.build.yml's step was run as written, with Go 1.27.0 and an empty module cache: all 44 modules build, both with and without this change.sededits to these Dockerfiles. The only one rewrites go-docker-timefreeze'sRUN CGO_ENABLED=0 GOOS=linux go build -o /main .line, and this PR leaves that line unchanged.