Sitelet https://github.com/kubernetes/kops/pull/18658/files
Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions docs/examples/kops-tests-private-net-bastion-host.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ export KOPS_STATE_STORE=s3://my-kops-s3-bucket-for-cluster-state
Some things to note from here:

- "NAME" will be an environment variable that we'll use from now in order to refer to our cluster name. For this practical exercise, our cluster name is "privatekopscluster.k8s.local".
- Because we'll use `--dns=none` instead of a valid DNS domain on AWS ROUTE53 service, our cluster name need to include the string **".k8s.local"** at the end (this is covered on our AWS tutorials). You can see more about this on our [Getting Started Doc.](../getting_started/aws.md)
- Because no DNS zone is configured, the cluster defaults to None-DNS. The **".k8s.local"** suffix is only part of this example name and is not required. See the [Getting Started Guide](../getting_started/aws.md) for more information.


## KOPS PRIVATE CLUSTER CREATION:
Expand Down Expand Up @@ -79,7 +79,7 @@ A few things to note here:
- Because we are just doing a simple LAB, we are using "t3.micro" machines. Please DON'T USE t3.micro on real production systems. Start with "t3.medium" as a minimum realistic/workable machine type.
- And finally, the "--networking kopeio-vxlan" argument. With the private networking model, we need to tell kOps which networking subsystem to use. More information about kOps supported networking models can be obtained from the [KOPS Kubernetes Networking Documentation](../networking.md). For this exercise we'll use "kopeio-vxlan" (or "kopeio" for short).

**NOTE**: You can add the "--bastion" argument here if you are not using "gossip dns" and create the bastion from start, but if you are using "gossip-dns" this will make this cluster to fail (this is a bug we are correcting now). For the moment don't use "--bastion" when using gossip DNS. We'll show you how to get around this by first creating the private cluster, then creation the bastion instance group once the cluster is running.
**NOTE**: This guide adds the bastion instance group after the initial cluster creation to demonstrate that workflow. You can instead pass `--bastion` to `kops create cluster`.

With those points clarified, let's deploy our cluster:

Expand Down Expand Up @@ -127,7 +127,7 @@ But, all the cluster instances (masters and worker nodes) will have private IP's

## ADDING A BASTION HOST TO OUR CLUSTER.

We mentioned earlier that we can't add the "--bastion" argument to our "kops create cluster" command if we are using "gossip dns" (a fix it's on the way as we speaks). That forces us to add the bastion afterwards, once the cluster is up and running.
This example adds the bastion after the cluster is running. Passing `--bastion` to `kops create cluster` creates it with the cluster instead.

Let's add a bastion here by using the following command:

Expand Down Expand Up @@ -168,7 +168,6 @@ kops update cluster ${NAME} --yes
You will see an output like the following:

```bash
I0828 13:06:33.153920 16528 apply_cluster.go:420] Gossip DNS: skipping DNS validation
I0828 13:06:34.686722 16528 executor.go:91] Tasks: 0 done / 116 total; 40 can run
I0828 13:06:36.181677 16528 executor.go:91] Tasks: 40 done / 116 total; 26 can run
I0828 13:06:37.602302 16528 executor.go:91] Tasks: 66 done / 116 total; 34 can run
Expand Down
5 changes: 0 additions & 5 deletions hooks/nvidia-device-plugin/image/files/02-nvidia-docker.sh
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,6 @@ EOF
# Note that the nvidia-docker version must match the docker-ce version
# --force-confold prevents prompt for replacement of daemon.json
apt-get -y update
# Stop protokube to ensure not bring kubelet up again
systemctl stop protokube
# Stop kubelet to ensure not bring stopped containers up again and leak
# them as orphan containers
systemctl stop kubelet
Expand All @@ -85,7 +83,4 @@ apt-get install -y --allow-downgrades -o Dpkg::Options::="--force-confold" \
systemctl mask cloud-init.service
systemctl mask kops-configuration.service

# Restore protokube and protokube will bring up kubelet
systemctl start protokube
# Seems protokube won't bring up kubelet, so start kubelet separately
systemctl start kubelet
21 changes: 1 addition & 20 deletions pkg/apis/kops/cluster.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@ import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/intstr"
"k8s.io/kops/pkg/apis/kops/util"
"k8s.io/kops/pkg/dns"
"k8s.io/kops/upup/pkg/fi/utils"
)

Expand Down Expand Up @@ -945,35 +944,17 @@ func (c *Cluster) AzureNetworkSecurityGroupName() string {
}

func (c *Cluster) PublishesDNSRecords() bool {
if c.UsesNoneDNS() || dns.IsGossipClusterName(c.Name) {
return false
}
return true
}

func (c *Cluster) UsesLegacyGossip() bool {
if c.UsesNoneDNS() || !dns.IsGossipClusterName(c.Name) {
return false
}
return true
return !c.UsesNoneDNS()
}

func (c *Cluster) UsesPublicDNS() bool {
if c.UsesLegacyGossip() {
// Gossip clusters have public/private DNS topology set
return false
}
if c.Spec.Networking.Topology == nil || c.Spec.Networking.Topology.DNS == "" || c.Spec.Networking.Topology.DNS == DNSTypePublic {
return true
}
return false
}

func (c *Cluster) UsesPrivateDNS() bool {
if c.UsesLegacyGossip() {
// Gossip clusters have public/private DNS topology set
return false
}
if c.Spec.Networking.Topology != nil && c.Spec.Networking.Topology.DNS == DNSTypePrivate {
return true
}
Expand Down
6 changes: 5 additions & 1 deletion pkg/apis/kops/validation/validation.go
Original file line number Diff line number Diff line change
Expand Up @@ -566,6 +566,10 @@ func validateTopology(c *kops.Cluster, topology *kops.TopologySpec, fieldPath *f
return allErrs
}

func usesLegacyGossip(c *kops.Cluster) bool {
return !c.UsesNoneDNS() && strings.HasSuffix(strings.TrimSuffix(c.Name, "."), ".k8s.local")
}

func validateCloudDNSTopology(c *kops.Cluster, fieldPath *field.Path) field.ErrorList {
type dnsTopologies struct {
none bool // api server and kops-controller have a stable address
Expand Down Expand Up @@ -593,7 +597,7 @@ func validateCloudDNSTopology(c *kops.Cluster, fieldPath *field.Path) field.Erro
}

switch {
case c.UsesLegacyGossip():
case usesLegacyGossip(c):
return field.ErrorList{field.Forbidden(fieldPath,
"gossip DNS support was removed in kOps 1.37; migrate the cluster to dns=none or a hosted DNS zone using kOps 1.36 before upgrading (see https://kops.sigs.k8s.io/gossip/)")}
case c.UsesNoneDNS():
Expand Down
26 changes: 0 additions & 26 deletions pkg/dns/gossip.go

This file was deleted.

52 changes: 0 additions & 52 deletions pkg/dns/gossip_test.go

This file was deleted.

16 changes: 8 additions & 8 deletions pkg/kubeconfig/create_kubecfg_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -156,10 +156,10 @@ func TestBuildKubecfg(t *testing.T) {

publicCluster := buildMinimalCluster("testcluster", "testcluster.test.com", false, false)
emptyMasterPublicNameCluster := buildMinimalCluster("emptyMasterPublicNameCluster", "", false, false)
k8sLocalCluster := buildMinimalCluster("testgossipcluster.k8s.local", "", false, true)
k8sLocalCluster := buildMinimalCluster("testcluster.k8s.local", "", false, true)
certCluster := buildMinimalCluster("testcluster", "testcluster.test.com", true, false)
certNLBCluster := buildMinimalCluster("testcluster", "testcluster.test.com", true, true)
certK8sLocalNLBCluster := buildMinimalCluster("testgossipcluster.k8s.local", "", true, true)
certK8sLocalNLBCluster := buildMinimalCluster("testcluster.k8s.local", "", true, true)

fakeStatus := fakeStatusCloud{
GetApiIngressStatusFn: func(cluster *kops.Cluster) ([]fi.ApiIngressStatus, error) {
Expand Down Expand Up @@ -296,11 +296,11 @@ func TestBuildKubecfg(t *testing.T) {
status: fakeStatus,
},
want: &KubeconfigBuilder{
Context: "testgossipcluster.k8s.local",
Context: "testcluster.k8s.local",
Server: "https://elbHostName",
TLSServerName: "api.internal.testgossipcluster.k8s.local",
TLSServerName: "api.internal.testcluster.k8s.local",
CACerts: []byte(nextCertificate + certData),
User: "testgossipcluster.k8s.local",
User: "testcluster.k8s.local",
},
wantClientCert: false,
},
Expand Down Expand Up @@ -359,11 +359,11 @@ func TestBuildKubecfg(t *testing.T) {
},
},
want: &KubeconfigBuilder{
Context: "testgossipcluster.k8s.local",
Context: "testcluster.k8s.local",
Server: "https://elbHostName:8443",
TLSServerName: "api.internal.testgossipcluster.k8s.local",
TLSServerName: "api.internal.testcluster.k8s.local",
CACerts: []byte(nextCertificate + certData),
User: "testgossipcluster.k8s.local",
User: "testcluster.k8s.local",
},
wantClientCert: true,
},
Expand Down
4 changes: 1 addition & 3 deletions pkg/resources/aws/aws.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ import (
route53types "github.com/aws/aws-sdk-go-v2/service/route53/types"
"k8s.io/apimachinery/pkg/util/sets"
"k8s.io/klog/v2"
"k8s.io/kops/pkg/dns"
"k8s.io/kops/pkg/featureflag"
"k8s.io/kops/pkg/resources"
"k8s.io/kops/pkg/resources/spotinst"
Expand All @@ -59,7 +58,6 @@ type listFn func(fi.Cloud, string, string) ([]*resources.Resource, error)

func ListResourcesAWS(cloud awsup.AWSCloud, clusterInfo resources.ClusterInfo) (map[string]*resources.Resource, error) {
clusterName := clusterInfo.Name
clusterUsesNoneDNS := clusterInfo.UsesNoneDNS

resourceTrackers := make(map[string]*resources.Resource)

Expand Down Expand Up @@ -93,7 +91,7 @@ func ListResourcesAWS(cloud awsup.AWSCloud, clusterInfo resources.ClusterInfo) (
ListEventBridgeRules,
}

if !dns.IsGossipClusterName(clusterName) && !clusterUsesNoneDNS {
if clusterInfo.PublishesDNSRecords() {
// Route 53
listFunctions = append(listFunctions, ListRoute53Records)
}
Expand Down
5 changes: 5 additions & 0 deletions pkg/resources/clusterinfo.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,8 @@ type ClusterInfo struct {
AzureNetworkShared bool
AzureRouteTableShared bool
}

// PublishesDNSRecords reports whether resource discovery should query the cloud DNS provider.
func (c ClusterInfo) PublishesDNSRecords() bool {
return !c.UsesNoneDNS
}
56 changes: 56 additions & 0 deletions pkg/resources/clusterinfo_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
/*
Copyright 2026 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package resources

import "testing"

func TestClusterInfoPublishesDNSRecords(t *testing.T) {
tests := []struct {
name string
clusterInfo ClusterInfo
want bool
}{
{
name: "hosted DNS",
clusterInfo: ClusterInfo{Name: "cluster.example.com"},
want: true,
},
{
name: "hosted DNS with k8s.local name",
clusterInfo: ClusterInfo{Name: "cluster.k8s.local"},
want: true,
},
{
name: "None DNS with hosted-style name",
clusterInfo: ClusterInfo{Name: "cluster.example.com", UsesNoneDNS: true},
want: false,
},
{
name: "None DNS with k8s.local name",
clusterInfo: ClusterInfo{Name: "cluster.k8s.local", UsesNoneDNS: true},
want: false,
},
}

for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
if got := test.clusterInfo.PublishesDNSRecords(); got != test.want {
t.Errorf("PublishesDNSRecords() = %t, want %t", got, test.want)
}
})
}
}
13 changes: 6 additions & 7 deletions pkg/resources/digitalocean/resources.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,14 @@ func ListResources(cloud do.DOCloud, clusterInfo resources.ClusterInfo) (map[str
listFunctions := []listFn{
listVolumes,
listDroplets,
listDNS,
}
if clusterInfo.PublishesDNSRecords() {
listFunctions = append(listFunctions, listDNS)
}
listFunctions = append(listFunctions,
listLoadBalancers,
listVPCs,
}
)

for _, fn := range listFunctions {
rt, err := fn(cloud, clusterName)
Expand Down Expand Up @@ -147,11 +151,6 @@ func listDNS(cloud fi.Cloud, clusterName string) ([]*resources.Resource, error)
}

if domainName == "" {
if strings.HasSuffix(clusterName, ".k8s.local") {
klog.Info("Domain Name is empty. Ok to have an empty domain name since the cluster does not publish DNS records.")
return nil, nil
}

return nil, fmt.Errorf("failed to find domain for cluster: %s", clusterName)
}

Expand Down
4 changes: 1 addition & 3 deletions pkg/resources/gce/gce.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@ import (
clouddns "google.golang.org/api/dns/v1"
"google.golang.org/api/iam/v1"
"k8s.io/klog/v2"
"k8s.io/kops/pkg/dns"
"k8s.io/kops/pkg/resources"
"k8s.io/kops/pkg/truncate"
"k8s.io/kops/upup/pkg/fi"
Expand Down Expand Up @@ -63,7 +62,6 @@ const maxGCERouteNameLength = 63

func ListResourcesGCE(gceCloud gce.GCECloud, clusterInfo resources.ClusterInfo) (map[string]*resources.Resource, error) {
clusterName := clusterInfo.Name
clusterUsesNoneDNS := clusterInfo.UsesNoneDNS

ctx := context.TODO()
region := gceCloud.Region()
Expand Down Expand Up @@ -116,7 +114,7 @@ func ListResourcesGCE(gceCloud gce.GCECloud, clusterInfo resources.ClusterInfo)
return d.listRoutes(ctx, allResources)
},
}
if !dns.IsGossipClusterName(clusterName) && !clusterUsesNoneDNS {
if clusterInfo.PublishesDNSRecords() {
listFunctions = append(listFunctions, d.listGCEDNSZone)
}

Expand Down
Loading
Loading