Agones – Advanced

Docs: Scheduling and Autoscaling

Thu, 03 Jan 2019 05:45:05 +0000

Cluster Autoscaler

Kubernetes has a cluster node autoscaler that works with a wide variety of cloud providers.

The default scheduling strategy (Packed) is designed to work with the Kubernetes autoscaler out of the box.

The autoscaler will automatically add Nodes to the cluster when GameServers don’t have room to be scheduled on the clusters, and then scale down when there are empty Nodes with no GameServers running on them.

This means that scaling Fleets up and down can be used to control the size of the cluster, as the cluster autoscaler will adjust the size of the cluster to match the resource needs of one or more Fleets running on it.

To enable and configure autoscaling on your cloud provider, check their connector implementation, or their cloud specific documentation.

Google Kubernetes Engine

Amazon Elastic Kubernetes Service

Cluster Autoscaler for EKS

Azure Kubernetes Service

Cluster Autoscaler on Azure Kubernetes Service (AKS) - Preview

Fleet Autoscaling

Fleet autoscaling is the only type of autoscaling that exists in Agones. It is currently available as a buffer autoscaling strategy or as a webhook driven strategy, such that you can provide your own autoscaling logic.

Have a look at the Create a Fleet Autoscaler quickstart, the Create a Webhook Fleet Autoscaler quickstart, and the Fleet Autoscaler Specification for details.

Autoscaling Concepts

To facilitate autoscaling, we need to combine several concepts and functionality, as described below.

Allocation Scheduling

Allocation scheduling refers to the order in which GameServers, and specifically their backing Pods are chosen from across the Kubernetes cluster within a given Fleet when allocation occurs.

Pod Scheduling

Each GameServer is backed by a Kubernetes Pod. Pod scheduling refers to the strategy that is in place that determines which node in the Kubernetes cluster the Pod is assigned to, when it is created.

Fleet Scale Down Strategy

Fleet Scale Down strategy refers to the order in which the GameServers that belong to a Fleet are deleted, when Fleets are shrunk in size.

Fleet Scheduling

There are two scheduling strategies for Fleets - each designed for different types of Kubernetes Environments.

Packed

apiVersion: "agones.dev/v1"
kind: Fleet
metadata:
  name: simple-game-server
spec:
  replicas: 100
  scheduling: Packed
  template:
    spec:
      ports:
      - containerPort: 7654
      template:
        spec:
          containers:
          - name: simple-game-server
            image: us-docker.pkg.dev/agones-images/examples/simple-game-server:0.15

This is the default Fleet scheduling strategy. It is designed for dynamic Kubernetes environments, wherein you wish to scale up and down as load increases or decreases, such as in a Cloud environment where you are paying for the infrastructure you use.

It attempts to pack as much as possible into the smallest set of nodes, to make scaling infrastructure down as easy as possible.

This affects the Cluster autoscaler, Allocation Scheduling, Pod Scheduling and Fleet Scale Down Scheduling.

Cluster Autoscaler

When using the “Packed” strategy, Agones will ensure that the Cluster Autoscaler doesn’t attempt to evict and move GameServer Pods onto new Nodes during gameplay.

Warning

The eviction API feature is currently Beta, and while it is enabled by default it may change in the future.

Use the Feature Gate SafeToEvict to disable this feature.

See the Feature Gate documentation for details on how to disable features.

If a gameserver can tolerate being evicted (generally in combination with setting an appropriate graceful termination period on the gameserver pod) and you want the Cluster Autoscaler to compact your cluster by evicting game servers when it would allow the Cluster Autoscaler to reduce the number of nodes in the cluster, Controlling Disruption describes how to choose the .eviction setting appropriate for your GameServer or Fleet.

Allocation Scheduling Strategy

Under the “Packed” strategy, allocation will prioritise allocating GameServers to nodes that are running on Nodes that already have allocated GameServers running on them.

Pod Scheduling Strategy

Under the “Packed” strategy, Pods will be scheduled using the PodAffinity with a preferredDuringSchedulingIgnoredDuringExecution affinity with hostname topology. This attempts to group together GameServer Pods within as few nodes in the cluster as it can.

Note

The default Kubernetes scheduler doesn’t do a perfect job of packing, but it’s a good enough job for what we need - at least at this stage.

Fleet Scale Down Strategy

With the “Packed” strategy, Fleets will remove Ready GameServers from Nodes with the least number of Ready and Allocated GameServers on them. Attempting to empty Nodes so that they can be safely removed.

Distributed

apiVersion: "agones.dev/v1"
kind: Fleet
metadata:
  name: simple-game-server
spec:
  replicas: 100
  scheduling: Distributed
  template:
    spec:
      ports:
      - containerPort: 7654
      template:
        spec:
          containers:
          - name: simple-game-server
            image: us-docker.pkg.dev/agones-images/examples/simple-game-server:0.15

This Fleet scheduling strategy is designed for static Kubernetes environments, such as when you are running Kubernetes on bare metal, and the cluster size rarely changes, if at all.

This attempts to distribute the load across the entire cluster as much as possible, to take advantage of the static size of the cluster.

This affects Allocation Scheduling, Pod Scheduling and Fleet Scale Down Scheduling.

Cluster Autoscaler

Since this strategy is not aimed at clusters that autoscale, this strategy does nothing for the cluster autoscaler.

Allocation Scheduling Strategy

Under the “Distributed” strategy, allocation will prioritise allocating GameServers to nodes that have the least number of allocated GameServers on them.

Pod Scheduling Strategy

Under the “Distributed” strategy, Pod scheduling is provided by the default Kubernetes scheduler, which will attempt to distribute the GameServer Pods across as many nodes as possible.

Fleet Scale Down Strategy

With the “Distributed” strategy, Fleets will remove Ready GameServers from Nodes with at random, to ensure a distributed load is maintained.

Docs: High Availability Agones

Fri, 10 Feb 2023 00:00:00 +0000

Warning

The split controller and extensions feature is currently Alpha, not enabled by default, and may change in the future.

Use the FeatureGate SplitControllerAndExtensions to enable and test this feature.

See the Feature Gate documentation for details on how to enable features.

High Availability for Agones Controller

When SplitControllerAndExtensions is enabled, the agones-controller responsibility is split up into agones-controller, which enacts the Agones control loop, and agones-extensions, which acts as a service endpoint for webhooks and the allocation extension API. Splitting these responsibilities allows the agones-extensions pod to be horizontally scaled, making the Agones control plane highly available and more resiliant to disruption.

GKE Autopilot Clusters

SplitControllerAndExtensions must be enabled for GKE Autopilot.

Extension Pod Configrations

The agones-extensions binary has a similar helm configuration to agones-controller, see here. If you previously overrode agones.controller.* settings, you may need to override the same agones.extensions.* setting.

To change controller.numWorkers to 200 from 100 values and through the use of helm --set, add the follow to the helm command:

Important: This will not have any effect on any extensions values!

 ...
 --set agones.controller.numWorkers=200
 ...

An important configuration to note is the PodDisruptionBudget fields, agones.extensions.pdb.minAvailable and agones.extensions.pdb.maxUnavailable. Currently, the agones.extensions.pdb.minAvailable field is set to 1.

Deployment Considerations

When SplitControllerAndExtensions is enabled, what was previously a single agones-controller pod is deployed as agones-controller and 2 agones-extensions pods. For example:

NAME                                 READY   STATUS    RESTARTS   AGE
agones-allocator-78c6b8c79-h9nqc     1/1     Running   0          23h
agones-allocator-78c6b8c79-l2bzp     1/1     Running   0          23h
agones-allocator-78c6b8c79-rw75j     1/1     Running   0          23h
agones-controller-fbf944f4-vs9xx     1/1     Running   0          23h
agones-extensions-5648fc7dcf-hm6lk   1/1     Running   0          23h
agones-extensions-5648fc7dcf-qbc6h   1/1     Running   0          23h
agones-ping-5b9647874-2rrl6          1/1     Running   0          27h
agones-ping-5b9647874-rksgg          1/1     Running   0          27h

The number of replicas for agones-extensions can be set using helm variable agones.extensions.replicas, but the default is 2.

We expect the aggregate memory consumption of the pods will be slightly higher than the previous singleton pod, but as the responsibilities are now split across the pods, the aggregate CPU consumption should also be similar.

Feature Design

SplitControllerAndExtensions represents phase 1 of HA Agones. The remaining phases are not yet implemented.

Docs: Controlling Disruption

Tue, 24 Jan 2023 20:15:26 +0000

Disruption in Kubernetes

A Pod in Kubernetes may be disrupted for involuntary reasons, e.g. hardware failure, or voluntary reasons, such as when nodes are drained for upgrades.

By default, Agones assumes your game server should never be disrupted voluntarily and configures the Pod appropriately - but this isn’t always the ideal setting. Here we discuss how Agones allows you to control the two most significant sources of voluntary Pod evictions, node upgrades and Cluster Autoscaler, using the eviction API on the GameServer object.

Warning

The eviction API feature is currently Beta, and while it is enabled by default it may change in the future.

Use the Feature Gate SafeToEvict to disable this feature.

See the Feature Gate documentation for details on how to disable features.

Benefits of Allowing Voluntary Disruption

It’s not always easy to write your game server in a way that allows for disruption, but it can have major benefits:

Compaction of your cluster using Cluster Autoscaler can lead to considerable cost savings for your infrastructure.
Allowing automated node upgrades can save you management toil, and lowers the time it takes to patch security vulnerabilites.

Considerations

When discussing game server pod disruption, it’s important to keep two factors in mind:

TERM signal: Is your game server tolerant of graceful termination? If you wish to support voluntary disruption, your game server must handle the TERM signal (even if it runs to completion after receiving TERM).
Termination Grace Period: After receiving TERM, how long does your game server need to run? If you run to completion after receiving TERM, this is equivalent to the session length - if not, you can think of this as the cleanup time. In general, we bucket the grace period into “less than 10 minutes”, “10 minutes to an hour”, and “greater than an hour”. (See below if you are curious about grace period considerations.)

`eviction` API

The eviction API is specified as part of the GameServerSpec, like:

apiVersion: "agones.dev/v1"
kind: GameServer
metadata:
  name: "simple-game-server"
spec:
  eviction:
    safe: Always
  template:
    [...]

You can set eviction.safe based on your game server’s tolerance for disruption and session length, based on the following diagram:

In words:

Does the game server support TERM and terminate within ten minutes?
- Yes to both: Set safe: Always, and set terminationGracePeriodSeconds to the session length or cleanup time.
- No to either: Does the game server support TERM and terminate within an hour?
  - Yes to both: Set safe: OnUpgrade, and configure terminationGracePeriodSeconds to the session length or cleanup time.
  - No to either: Set safe: Never. If your game server does not terminate within an hour, see below.

Note

To maintain backward compatibility with Agones prior to the introduction of the SafeToEvict feature gate, if your game server previously configured the cluster-autoscaler.kubernetes.io/safe-to-evict: true annotation, we assume eviction.safe: Always is intended.

Note

GKE Autopilot supports only Never and Always, not OnUpgrade.

What’s special about ten minutes and one hour?

Ten minutes: Cluster Autoscaler respects ten minutes of graceful termination on scale-down. On some cloud products, you can configure --max-graceful-termination-sec to change this, but it is not advised: Cluster Autoscaler is currently only capable of scaling down one node at a time, and larger graceful termination windows slow this down farther (see autoscaler#5079). If the ten minute limit does not apply to you, generally you should choose between safe: Always (for sessions less than an hour), or see below.
One hour: On many cloud products, PodDisruptionBudget can only block node upgrade evictions for a certain period of time - on GKE this is 1h. After that, the PDB is ignored, or the node upgrade fails with an error. Controlling Pod disruption for longer than one hour requires cluster configuration changes outside of Agones - see below.

Considerations for long sessions

Outside of Cluster Autoscaler, the main source of disruption for long sessions is node upgrade. On some cloud products, such as GKE Standard, node upgrades are entirely within your control. On others, such as GKE Autopilot, node upgrade is automatic. Typical node upgrades use an eviction based, rolling recreate strategy, and may not honor PodDisruptionBudget for longer than an hour. Here we document strategies you can use for your cloud product to support long sessions.

On GKE

On GKE, there are currently two possible approaches to manage disruption for session lengths longer than an hour:

(GKE Standard/Autopilot) Blue/green deployment at the cluster level: If you are using an automated deployment process, you can:
- create a new, green cluster within a release channel e.g. every week,
- use maintenance exclusions to prevent node upgrades for 30d, and
- scale the Fleet on the old, blue cluster down to 0, and
- use multi-cluster allocation on Agones, which will then direct new allocations to the new green cluster (since blue has 0 desired), then
- delete the old, blue cluster when the Fleet successfully scales down.
(GKE Standard only) Use node pool blue/green upgrades

Other cloud products

The blue/green cluster strategy described for GKE is likely applicable to your cloud product.

We welcome contributions to this section for other products!

Implementation / Under the hood

Each option uses a slightly different permutation of:

the safe-to-evict annotation to block Cluster Autoscaler based eviction, and
the agones.dev/safe-to-evict label selector to select the agones-gameserver-safe-to-evict-false PodDisruptionBudget. This blocks Cluster Autoscaler and (for a limited time) disruption from node upgrades.
- Note that PDBs do influence pod preemption as well, but it’s not guaranteed.

As a quick reference:

evictions.safe setting	`safe-to-evict` pod annotation	`agones.dev/safe-to-evict` label
`Never` (default)	`false`	`false` (matches PDB)
`OnUpdate`	`false`	`true` (does not match PDB)
`Always`	`true`	`true` (does not match PDB)

Docs: Limiting CPU & Memory

Thu, 03 Jan 2019 05:45:15 +0000

As a short description:

CPU Requests are limits that are applied when there is CPU congestion, and as such can burst above their set limits.
CPU Limits are hard limits on how much CPU time the particular container gets access to.

This is useful for game servers, not just as a mechanism to distribute compute resources evenly, but also as a way to advice the Kubernetes scheduler how many game server processes it is able to fit into a given node in the cluster.

It’s worth reading the Managing Compute Resources for Containers Kubernetes documentation for more details on “requests” and “limits” to both CPU and Memory, and how to configure them.

GameServers

Since the GameServer specification provides a full PodSpecTemplate, we can take advantage of both resource limits and requests in our GameServer configurations.

For example, to set a CPU limit on our GameServer configuration of 250m/0.25 of a CPU, we could do so as followed:

apiVersion: "agones.dev/v1"
kind: GameServer
metadata:
  name: "simple-game-server"
spec:
  ports:
  - name: default
    containerPort: 7654
  template:
    spec:
      containers:
      - name: simple-game-server
        image: us-docker.pkg.dev/agones-images/examples/simple-game-server:0.15
        resources:
          limits:
            cpu: "250m" #this is our limit here

If you do not set a limit or request, the default is set by Kubernetes at a 100m CPU request.

SDK GameServer sidecar

You may also want to tweak the CPU request or limits on the SDK GameServer sidecar process that spins up alongside each game server container.

You can do this through the Helm configuration when installing Agones.

By default, this is set to having a CPU request value of 30m, with no hard CPU limit. This ensures that the sidecar always has enough CPU to function, but it is configurable in case a lower, or higher value is required on your clusters, or if you desire hard limit.

Docs: Allocator Service

Tue, 19 May 2020 05:45:05 +0000

To allocate a game server, Agones provides a gRPC and REST service with mTLS authentication, called agones-allocator that can be used instead of GameServerAllocations .

Both gRPC and REST are accessible through a Kubernetes service that can be externalized using a load balancer. By default, gRPC and REST are served from the same port. However, either service can be disabled or the services can be served from separate ports using the helm configuration.

Warning

If gRPC and REST are served using the same port, then an http multi-plexer is used along with an experimental gRPC server which has noticeably worse performance than using the standard gRPC server.

If you require a fully compatible or feature compatible gRPC server implementation, you must separate the gRPC port from the REST port or disable the REST service.

For requests to either service to succeed, a client certificate must be provided that is in the authorization list of the allocator service. The remainder of this article describes how to manually make a successful allocation request using the API.

The guide assumes you have command line tools installed for jq, go and openssl.

`GameServerAllocation` vs Allocator Service

There are several reasons you may prefer to use the Allocator Service over the GameServerAllocation custom resource definition, depending on your architecture and requirements:

A requirement to do multi-cluster allocation.
Want to create Allocations from outside the Agones Kubernetes cluster.
Prefer SSL based authentication over Kubernetes RBAC.
Prefer a gRPC or REST based API over an integration with the Kubernetes API.

Find the external IP

The service is hosted under the same namespace as the Agones controller. To find the external IP of your allocator service, replace agones-system namespace with the namespace to which Agones is deployed and execute the following command:

kubectl get service agones-allocator -n agones-system

The output of the command should look like:

NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)            AGE
agones-allocator            LoadBalancer   10.55.251.73    34.82.195.204   443:30250/TCP      7d22h

Server TLS certificate

If the agones-allocator service is installed as a LoadBalancer using a reserved IP, a valid self-signed server TLS certificate is generated using the IP provided. Otherwise, the server TLS certificate should be replaced. If you installed Agones using helm, you can easily reconfigure the allocator service with a preset IP address by setting the agones.allocator.service.loadBalancerIP parameter to the address that was automatically assigned to the service and helm upgrade:

EXTERNAL_IP=$(kubectl get services agones-allocator -n agones-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
helm upgrade my-release agones/agones -n agones-system --wait \
   --set agones.allocator.service.loadBalancerIP=${EXTERNAL_IP} \
   ...

Warning

The parameter used to automatically replace the certifate changed in Agones 1.18.0. If you are using an older version of Agones you should pass the parameter agones.allocator.http.loadBalancerIP instead. If you need your script to work with both older and newer versions of Agones, you can pass both parameters as only one of them will effect the helm chart templates.

Another approach is to replace the default server TLS certificate with a certificate with CN and subjectAltName. There are multiple approaches to generate a certificate. Agones recommends using cert-manager.io solution for cluster level certificate management.

In order to use the cert-manager solution, first install cert-manager on the cluster. Then, configure an Issuer/ClusterIssuer resource and last configure a Certificate resource to manage allocator-tls Secret. Make sure to configure the Certificate based on your system’s requirements, including the validity duration.

Here is an example of using a self-signed ClusterIssuer for configuring allocator-tls Secret:

#!/bin/bash
# Create a self-signed ClusterIssuer
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}
EOF

EXTERNAL_IP=$(kubectl get services agones-allocator -n agones-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

# for EKS use hostname
# HOST_NAME=$(kubectl get services agones-allocator -n agones-system -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

# Create a Certificate with IP for the allocator-tls secret
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: allocator-tls
  namespace: agones-system
spec:
  commonName: ${EXTERNAL_IP}
  ipAddresses:
    - ${EXTERNAL_IP}
  secretName: allocator-tls
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer
EOF

# Wait for the allocator-tls Secret
sleep 1
TLS_CA_VALUE=$(kubectl get secret allocator-tls -n agones-system -ojsonpath='{.data.ca\.crt}')

# Add ca.crt to the allocator-tls-ca Secret
kubectl get secret allocator-tls-ca -o json -n agones-system | jq '.data["tls-ca.crt"]="'${TLS_CA_VALUE}'"' | kubectl apply -f -
echo $TLS_CA_VALUE | base64 -d > ca.crt
# In case of MacOS
# echo $TLS_CA_VALUE | base64 -D > ca.crt

Bring Your Own Certificates (advanced)

If you would like to completely manage the tls secrets outside of helm, you can create them in the namespace where agones is going to be installed, and then set the helm value agones.allocator.disableSecretCreation to true. This method will also work with the cert-manager method, as long as your certificate and secret are created ahead of time, and you populate the allocator-tls-ca and allocator-client-ca yourself.

Client Certificate

Because agones-allocator uses an mTLS authentication mechanism, a client must provide a certificate that is accepted by the server.

If Agones is installed using Helm, you can leverage a default client secret, allocator-client.default, created in the game server namespace and allowlisted in allocator-client-ca Kubernetes secret. You can extract and use that secret for client side authentication, by following the allocation example.

Otherwise, here is an example of generating a client certificate using openssl.

#!/bin/bash

EXTERNAL_IP=$(kubectl get services agones-allocator -n agones-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout client.key -out client.crt -addext 'subjectAltName=IP:'${EXTERNAL_IP}''

CERT_FILE_VALUE=$(cat client.crt | base64 -w 0)

# In case of MacOS
# CERT_FILE_VALUE=$(cat client.crt | base64)

# allowlist client certificate
kubectl get secret allocator-client-ca -o json -n agones-system | jq '.data["client_trial.crt"]="'${CERT_FILE_VALUE}'"' | kubectl apply -f -

The last command creates a new entry in the secret data map for allocator-client-ca for the client CA. This is for the agones-allocator service to accept the newly generated client certificate.

Send allocation request

After setting up agones-allocator with server certificate and allowlisting the client certificate, the service can be used to allocate game servers. Make sure you have a fleet with ready game servers in the game server namespace.

Set the environment variables and store the client secrets before allocating using gRPC or REST APIs:

NAMESPACE=default # replace with any namespace
EXTERNAL_IP=$(kubectl get services agones-allocator -n agones-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
KEY_FILE=client.key
CERT_FILE=client.crt
TLS_CA_FILE=ca.crt

# allocator-client.default secret is created only when using helm installation. Otherwise generate the client certificate and replace the following.
# In case of MacOS replace "base64 -d" with "base64 -D"
kubectl get secret allocator-client.default -n "${NAMESPACE}" -ojsonpath="{.data.tls\.crt}" | base64 -d > "${CERT_FILE}"
kubectl get secret allocator-client.default -n "${NAMESPACE}" -ojsonpath="{.data.tls\.key}" | base64 -d > "${KEY_FILE}"
kubectl get secret allocator-tls-ca -n agones-system -ojsonpath="{.data.tls-ca\.crt}" | base64 -d > "${TLS_CA_FILE}"

Using gRPC

To start, take a look at the allocation gRPC client examples in golang and C# languages. In the following, the golang gRPC client example is used to allocate a Game Server in the default namespace.

#!/bin/bash

go run examples/allocator-client/main.go --ip ${EXTERNAL_IP} \
    --port 443 \
    --namespace ${NAMESPACE} \
    --key ${KEY_FILE} \
    --cert ${CERT_FILE} \
    --cacert ${TLS_CA_FILE}

Using REST

#!/bin/bash

curl --key ${KEY_FILE} \
     --cert ${CERT_FILE} \
     --cacert ${TLS_CA_FILE} \
     -H "Content-Type: application/json" \
     --data '{"namespace":"'${NAMESPACE}'"}' \
     https://${EXTERNAL_IP}/gameserverallocation \
     -X POST

You should expect to see the following output:

{"gameServerName":"game-server-name","ports":[{"name":"default","port":7463}],"address":"1.2.3.4","nodeName":"node-name"}

Sending Data to the Game Server

The service accepts a metadata field, which can be used to apply labels and annotations to the allocated GameServer. The old metaPatch fields is now deprecated, but can still be used for compatibility. If both metadata and metaPatch fields are set, metaPatch is ignored.

Secrets Explained

agones-allocator has a dependency on three Kubernetes secrets:

allocator-tls - stores the server certificate.
allocator-client-ca - stores the allocation authorized client CA for mTLS to allowlist client certificates.
allocator-tls-ca (optional) - stores allocator-tls CA.

The separation of CA secret from the private secret is for the security reason to avoid reading the private secret, while retrieving the allocator CA that is used by the allocation client to validate the server. It is optional to set or maintain the allocator-tls-ca secret.

Troubleshooting

If you encounter problems, explore the following potential root causes:

Check server certificate - Using openssl you can get the certificate chain for the server.
```
EXTERNAL_IP=$(kubectl get services agones-allocator -n agones-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
openssl s_client -connect ${EXTERNAL_IP}:443
```
- Inspect the server certificate by storing the certificate returned, under Server certificate and validating using openssl x509 -in tls.crt -text -noout.
- Make sure the certificate is not expired and the Subject Alternative Name is set.
- If the issuer is CN = allocation-ca, the certificate is generated using Agones helm installation.
Check client certificate
- You may get an error such as rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection closed, make sure your client certificate is allowlisted by being added to allocator-client-ca.
```
kubectl get secret allocator-client-ca -o json -n agones-system
```
- If the server certificate is not accepted by the client, you may get an error such as rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority", depending on the client. In this case, verify that the TLS CA file matches the server certificate.
```
kubectl get secret allocator-tls -n agones-system -ojsonpath="{.data.tls\.crt}" | base64 -d > tls.crt
openssl verify -verbose -CAfile ca.crt tls.crt
tls.crt: OK
```

Make sure the service is up and running.

kubectl get pod -n agones-system | grep agones-allocator
agones-allocator-59b4f6b5c6-86j62      1/1     Running     0          6m36s
agones-allocator-59b4f6b5c6-kbqrq      1/1     Running     0          6m45s
agones-allocator-59b4f6b5c6-trbkl      1/1     Running     0          6m28s

kubectl get service agones-allocator -n agones-system
agones-allocator   LoadBalancer   10.55.248.14   34.82.195.204    443:32468/TCP   6d23h

API Reference

The AllocationService API is located as a gRPC service here . Additionally, the REST API is available as a Swagger API .

Docs: Multi-cluster Allocation

Fri, 25 Oct 2019 05:45:05 +0000

There may be different types of clusters, such as on-premise, and Google Kubernetes Engine (GKE), used by a game to help with the cost-saving and availability. For this purpose, Agones provides a mechanism to define priorities on the clusters. Priorities are defined on GameServerAllocationPolicy agones CRD. A matchmaker can enable the multi-cluster rules on a request and target agones-allocator endpoint in any of the clusters and get resources allocated on the cluster with the highest priority. If the cluster with the highest priority is overloaded, the allocation request is redirected to the cluster with the next highest priority.

The remainder of this article describes how to enable multi-cluster allocation.

Define Cluster Priority

GameServerAllocationPolicy is the CRD defined by Agones for setting multi-cluster allocation rules. In addition to cluster priority, it describes the connection information for the target cluster, including the game server namespace, agones-allocator endpoint and client K8s secrets name for redirecting the allocation request. Game servers will be allocated from clusters with the lowest priority number. If there are no available game servers available in clusters with the lowest priority number, they will be allocated from clusters with the next lowest priority number. For clusters with the same priority, the cluster is chosen with a probability relative to its weight.

Here is an example of setting the priority for a cluster and it’s connection rules. One such resource should be defined per cluster.

In the following example the policy is defined for cluster B in cluster A.

cat <<EOF | kubectl apply -f -
apiVersion: multicluster.agones.dev/v1
kind: GameServerAllocationPolicy
metadata:
  name: allocator-cluster-b
  namespace: cluster-a-ns
spec:
  connectionInfo:
    allocationEndpoints:
    - 34.82.195.204
    clusterName: "clusterB"
    namespace: cluster-b-ns
    secretName: allocator-client-to-cluster-b
    serverCa: c2VydmVyQ0E=
  priority: 1
  weight: 100
EOF

To define the local cluster priority a GameServerAllocationPolicy should be defined without an allocationEndpoints field. If the local cluster priority is not defined, the allocation from the local cluster happens only if allocation from other clusters with the existing allocation rules is unsuccessful.

Allocation requests with multi-cluster allocation enabled but with only the local cluster available (e.g. in development) must have a local cluster priority defined, or the request fails with the error “no multi-cluster allocation policy is specified”.

The namespace field in connectionInfo is the namespace that the game servers will be allocated in, and must be a namespace in the target cluster that has been previously defined as allowed to host game servers. The Namespace specified in the allocation request (below) is used to refer to the namespace that the GameServerAllocationPolicy itself is located in.

serverCa is the server TLS CA public certificate, set only if the remote server certificate is not signed by a public CA (e.g. self-signed). If this field is not specified, the certificate can also be specified in a field named ca.crt of the client secret (the secret referred to in the secretName field).

Establish trust

To accept allocation requests from other clusters, agones-allocator for cluster B should be configured to accept the client’s certificate from cluster A and the cluster A’s client should be configured to accept the server TLS certificate, if it is not signed by a public Certificate Authority (CA).

Follow the steps to configure the agones allocator gRPC service. The client certificate pair in the mentioned document is stored as a K8s secret. Here are the secrets to set:

1.Client certificate to talk to other clusters:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: allocator-client-to-cluster-b
  namespace: cluster-a-ns
type: Opaque
data:
  tls.crt: <REDACTED>
  tls.key: <REDACTED>
  ca.crt: <REDACTED>
EOF

The certificates are base 64 string of the certificate file e.g. cat ${CERT_FILE} | base64 -w 0

Agones recommends using cert-manager.io solution for generating client certificates.

2.Add client CA to the list of authorized client certificates by agones-allocator in the targeted cluster.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: allocator-client-ca
  namespace: agones-system
type: Opaque
data:
  client1.crt: <REDACTED>
  client2.crt: <REDACTED>
  …
  clientN.crt: <REDACTED>
EOF

Allocate multi-cluster

To enable multi-cluster allocation, set multiClusterSetting.enabled to true in allocation.proto and send allocation requests. For more information visit agones-allocator. In the following, using allocator-client sample , a multi-cluster allocation request is sent to the agones-allocator service. If the allocation succeeds, the AllocationResponse will contain a Source field which indicates the endpoint of the remote agones-allocator.

Set the environment variables and store the client secrets before allocating using gRPC or REST APIs

#!/bin/bash

NAMESPACE=default # replace with any namespace
EXTERNAL_IP=$(kubectl get services agones-allocator -n agones-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
KEY_FILE=client.key
CERT_FILE=client.crt
TLS_CA_FILE=ca.crt

# allocator-client.default secret is created only when using helm installation. Otherwise generate the client certificate and replace the following.
# In case of MacOS replace "base64 -d" with "base64 -D"
kubectl get secret allocator-client.default -n "${NAMESPACE}" -ojsonpath="{.data.tls\.crt}" | base64 -d > "${CERT_FILE}"
kubectl get secret allocator-client.default -n "${NAMESPACE}" -ojsonpath="{.data.tls\.key}" | base64 -d > "${KEY_FILE}"
kubectl get secret allocator-tls-ca -n agones-system -ojsonpath="{.data.tls-ca\.crt}" | base64 -d > "${TLS_CA_FILE}"

#!/bin/bash

go run examples/allocator-client/main.go --ip ${EXTERNAL_IP} \
    --namespace ${NAMESPACE} \
    --key ${KEY_FILE} \
    --cert ${CERT_FILE} \
    --cacert ${TLS_CA_FILE} \
    --multicluster true

If using REST use

#!/bin/bash

curl --key ${KEY_FILE} \
     --cert ${CERT_FILE} \
     --cacert ${TLS_CA_FILE} \
     -H "Content-Type: application/json" \
     --data '{"namespace":"'${NAMESPACE}'", "multiClusterSetting":{"enabled":true}}' \
     https://${EXTERNAL_IP}/gameserverallocation \
     -X POST

Troubleshooting

If you encounter problems, explore the following potential root causes:

Make sure single cluster allocation works for each cluster using this troubleshooting.
For each cluster, make sure there is a GameServerAllocationPolicy resource defined in the game server cluster.
Inspect the .spec.connectionInfo for GameServerAllocationPolicy for each cluster. Use the cluster connection information in that field to verify that single cluster allocation works. Use the information to verify the connection:

POLICY_NAME=<policy-name>
POLICY_NAMESPACE=<policy-namespace>

NAMESPACE=$(kubectl get gameserverallocationpolicy ${POLICY_NAME} -n ${POLICY_NAMESPACE} -ojsonpath={.spec.connectionInfo.namespace})
EXTERNAL_IP=$(kubectl get gameserverallocationpolicy ${POLICY_NAME} -n ${POLICY_NAMESPACE} -ojsonpath={.spec.connectionInfo.allocationEndpoints\[0\]})
CLIENT_SECRET_NAME=$(kubectl get gameserverallocationpolicy ${POLICY_NAME} -n ${POLICY_NAMESPACE} -ojsonpath={.spec.connectionInfo.secretName})

KEY_FILE=client.key
CERT_FILE=client.crt
TLS_CA_FILE=ca.crt

# In case of MacOS replace "base64 -d" with "base64 -D"
kubectl get secret "${CLIENT_SECRET_NAME}" -n "${POLICY_NAMESPACE}" -ojsonpath="{.data.tls\.crt}" | base64 -d > "${CERT_FILE}"
kubectl get secret "${CLIENT_SECRET_NAME}" -n "${POLICY_NAMESPACE}" -ojsonpath="{.data.tls\.key}" | base64 -d > "${KEY_FILE}"
kubectl get secret "${CLIENT_SECRET_NAME}" -n "${POLICY_NAMESPACE}" -ojsonpath="{.data.ca\.crt}" | base64 -d > "${TLS_CA_FILE}"

#!/bin/bash

go run examples/allocator-client/main.go --ip ${EXTERNAL_IP} \
    --port 443 \
    --namespace ${NAMESPACE} \
    --key ${KEY_FILE} \
    --cert ${CERT_FILE} \
    --cacert ${TLS_CA_FILE}

Docs: GameServer Pod Service Accounts

Thu, 14 Mar 2019 04:30:37 +0000

Default Settings

By default, Agones sets up service accounts and sets them appropriately for the Pods that are created for GameServers.

Since Agones provides GameServer Pods with a sidecar container that needs access to Agones Custom Resource Definitions, Pods are configured with a service account with extra RBAC permissions to ensure that it can read and modify the resources it needs.

Since service accounts apply to all containers in a Pod, Agones will automatically overwrite the mounted key for the service account in the container that is running the dedicated game server in the backing Pod. This is done since game server containers are exposed publicly, and generally don’t require the extra permissions to access aspects of the Kubernetes API.

Bringing your own Service Account

If needed, you can provide your own service account on the Pod specification in the GameServer configuration.

For example:

apiVersion: "agones.dev/v1"
kind: GameServer
metadata:
  generateName: "simple-game-server-"
spec:
  ports:
  - name: default
    containerPort: 7654
  template:
    spec:
      serviceAccountName: my-special-service-account # a custom service account
      containers:
      - name: simple-game-server
        image: us-docker.pkg.dev/agones-images/examples/simple-game-server:0.15

If a service account is configured, the mounted key is not overwritten, as it assumed that you want to have full control of the service account and underlying RBAC permissions.

Agones – Advanced

Docs: Scheduling and Autoscaling

Cluster Autoscaler

Google Kubernetes Engine

Amazon Elastic Kubernetes Service

Azure Kubernetes Service

Fleet Autoscaling

Autoscaling Concepts

Allocation Scheduling

Pod Scheduling

Fleet Scale Down Strategy

Fleet Scheduling

Packed

Cluster Autoscaler

Warning

Allocation Scheduling Strategy

Pod Scheduling Strategy

Note

Fleet Scale Down Strategy

Distributed

Cluster Autoscaler

Allocation Scheduling Strategy

Pod Scheduling Strategy

Fleet Scale Down Strategy

Docs: High Availability Agones

Warning

High Availability for Agones Controller

GKE Autopilot Clusters

Extension Pod Configrations

Deployment Considerations

Feature Design

Docs: Controlling Disruption

Disruption in Kubernetes

Warning

Benefits of Allowing Voluntary Disruption

Considerations

eviction API

Note

Note

What’s special about ten minutes and one hour?

Considerations for long sessions

On GKE

Other cloud products

Implementation / Under the hood

Further Reading

Docs: Limiting CPU & Memory

GameServers

SDK GameServer sidecar

Docs: Allocator Service

Warning

GameServerAllocation vs Allocator Service

Find the external IP

Server TLS certificate

Warning

Bring Your Own Certificates (advanced)

Client Certificate

Send allocation request

Using gRPC

Using REST

Sending Data to the Game Server

Secrets Explained

Troubleshooting

API Reference

Docs: Multi-cluster Allocation

Define Cluster Priority

Establish trust

Allocate multi-cluster

Troubleshooting

Docs: GameServer Pod Service Accounts

Default Settings

Bringing your own Service Account

`eviction` API

`GameServerAllocation` vs Allocator Service