Your business is successful and you need to go global. How do you scale your app across multiple regions? How do you handle deployments to multiple clusters? How do you provide great user experience with low latency and resilience while maintaining low cost? One way is to use Kubernetes Federations. You deploy multiple clusters, join them in a federation and sync the API resources. Federations, however, are still in beta version and are not recommended for production usage. If you go up another layer, you can manage the multi-cluster control plane with a service mesh like Istio.
Kubernetes Federation is an open source project that focuses on making it easy to manage multiple clusters. It does so by providing
2 major building blocks:
Sync resources across clusters: Federation provides the ability to keep resources in multiple clusters in sync. For example, you can ensure that the same app deployment exists in multiple clusters.
Cross-cluster discovery: Federation provides the ability to auto-configure DNS servers and load balancers with backends from all clusters.
Some other use cases that federation enables are:
Federation is not helpful unless you have multiple clusters. Some of the reasons why you might want multiple clusters are:
While there are a lot of attractive use cases for federation, there are also some caveats:
Federations of Kubernetes Clusters can include clusters running on different cloud providers (e.g. Google Cloud, AWS), and on-premises (e.g. on OpenStack). Kubefed is the recommended way to deploy federated clusters.
Thereafter, your API resources can span different clusters and cloud providers.
Kubernetes Federation is currently considered alpha for many of its features, and there is no clear path to evolve the API to GA. I would not recommend using Kubernetes Federation for your production systems. Ingresses typically don’t work even when you are using a simple federation of k8s cluster from one public provider. Managing more ingresses with Hybrid cloud could be an awful pain.
Federation uses Public DNS and IP adresses with external LoadBalancer for cross-cluster service discovery, which is usually a quite expensive option. I didn’t find out how to make it work on a private network as one cluster does not see the other cluster’s k8s services, but pods only. Moreover Kubernetes Federation's project development seems rather stale. 143 stars on Github? Seriously?
You want the same features as Kubernetes Federation with a more stable and mature solution? Check out Istio’s multi-cluster support.
Multi-cluster functions by enabling Kubernetes control planes running a remote configuration to connect to one Istio control plane. Once one or more remote Kubernetes clusters are connected to the Istio control plane, Envoy can then communicate with the single Istio control plane and form a mesh network across multiple Kubernetes clusters.
This guide describes how to install a multi-cluster Istio topology using the manifests and Helm charts provided within the Istio repository.
I made a github repo for an easy provisioning of the whole system on GCP based on the previously mentioned guide.
We will deploy the Bookinfo application to two GKE clusters. All the services will run in one cluster, only the Reviews-3 will run in the other. We leverage the GKE’s alias IPs feature, where pods in one cluster can communicate with pods in the other cluster, using just private IPs on a private network.
Product page requests will be load balanced across all the reviews’ versions, even though it runs on a different cluster, in a different zone, region, continent…
As mentioned above, you typically don’t want your services to communicate cross-cluster to different zones/regions as it usually causes higher latency and network bandwidth fees. A typical use case would be if you had a central cluster close to your HQ - say in Frankfurt - and you had customers not only in Europe but in Brazil as well. You could deploy a smaller cluster to Brazil for public-facing frontend APIs and some subset of services and the rest (like payment-gateway APIs, databases, …) will run in Frankfurt only to save some costs.
You can autoscale the services in each cluster independently, depending on a local cluster’s traffic needs - there is no need for overprovisioning.
One disadvantage of this setup is that the Istio’s ingress-gateway is deployed as a LoadBalancer only in the master cluster. That means all traffic is being proxied through the master cluster, and even if your client is in Brazil, the request he makes goes to Frankfurt and back to Brazil. You could possibly avoid this by deploying more Istio masters.
I will play with this a little bit more in the future. I’d like to use Google https LoadBalancer with Istio ingress-gateway and have all the frontends deployed to all clusters.
Even nowadays with all the clouds, k8s and service meshes, multiple clusters are still hard. But it's 2018 and we can do better! Leveraging the advantages of having multi-cluster setups can benefit our business greatly. Kubernetes Federations might not be the perfect way to setup such an ecosystem, so take a look at Istio and see for yourself. It is definitely worth trying!
https://istio.io/docs/setup/kubernetes/multicluster-install/