Akka Clustering with Kubernetes
klusterd is an example Akka Cluster project that is packaged using Docker and deployed to Kubernetes.
Taking klusterd for a test drive
First, clone the project from GitHub.
git clone https://github.com/vyshane/klusterd.git
We’ll use sbt to package the application as a Docker image.
$ cd klusterd
$ sbt "docker:publishLocal"
This creates a local Docker image for us:
$ docker images | grep klusterd
vyshane/klusterd 1.0 43ee995e8adb 1 minute ago 690.7 MB
To deploy our klusterd image to our Kubernetes cluster:
$ cd ../deployment
$ ./up.sh
We can see that a klusterd pod comes up:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
klusterd-6srxl 1/1 Running 0 1m
Let’s tail its log:
$ kubectl logs -f klusterd-6srxl
We can see that klusterd is running at the IP address 172.17.0.3 and listening on port 2551.
INFO 14:58:06.022UTC akka.remote.Remoting - Starting remoting
INFO 14:58:06.235UTC akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://klusterd@172.17.0.3:2551]
Since we have only launched one klusterd node, it is its own cluster seed node.
INFO 14:58:06.446UTC akka.actor.ActorSystemImpl(klusterd) - Configured seed nodes: akka.tcp://klusterd@172.17.0.3:2551
We have a cluster of one.
INFO 14:58:06.454UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.3:2551] - Node [akka.tcp://klusterd@172.17.0.3:2551] is JOINING, roles []
INFO 14:58:06.482UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.3:2551] - Leader is moving node [akka.tcp://klusterd@172.17.0.3:2551] to [Up]
INFO 14:58:06.505UTC akka.tcp://klusterd@172.17.0.3:2551/user/cluster-monitor - Cluster member up: akka.tcp://klusterd@172.17.0.3:2551
Now, let’s scale this cluster up. Let’s ask Kubernetes for 2 more nodes:
$ kubectl scale --replicas=3 rc klusterd
We can see from klusterd-6srxl's logs that two more nodes join our cluster shortly after.
INFO 15:11:46.525UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.3:2551] - Node [akka.tcp://klusterd@172.17.0.4:2551] is JOINING, roles []
INFO 15:11:46.533UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.3:2551] - Node [akka.tcp://klusterd@172.17.0.5:2551] is JOINING, roles []
INFO 15:11:47.484UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.3:2551] - Leader is moving node [akka.tcp://klusterd@172.17.0.4:2551] to [Up]
INFO 15:11:47.505UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.3:2551] - Leader is moving node [akka.tcp://klusterd@172.17.0.5:2551] to [Up]
INFO 15:11:47.507UTC akka.tcp://klusterd@172.17.0.3:2551/user/cluster-monitor - Cluster member up: akka.tcp://klusterd@172.17.0.4:2551
INFO 15:11:47.507UTC akka.tcp://klusterd@172.17.0.3:2551/user/cluster-monitor - Cluster member up: akka.tcp://klusterd@172.17.0.5:2551
We can scale the cluster down:
$ kubectl scale --replicas=2 rc klusterd
And a node leaves the cluster. In our case, the original pod klusterd-6srxl was killed. Here are the logs from another node showing what happened:
WARN 15:19:31.855UTC akka.tcp://klusterd@172.17.0.4:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fklusterd%40172.17.0.3%3A2551-0 - Association with remote system [akka.tcp://klusterd@172.17.0.3:2551] has failed, address is now gated for [5000] ms. Reason: [Disassociated]
WARN 15:19:35.376UTC akka.tcp://klusterd@172.17.0.4:2551/user/cluster-monitor - Cluster member unreachable: akka.tcp://klusterd@172.17.0.3:2551
WARN 15:19:35.405UTC akka.tcp://klusterd@172.17.0.4:2551/system/cluster/core/daemon - Cluster Node [akka.tcp://klusterd@172.17.0.4:2551] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://klusterd@172.17.0.3:2551, status = Up)]
INFO 15:19:45.384UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.4:2551] - Leader is auto-downing unreachable node [akka.tcp://klusterd@172.17.0.3:2551]
INFO 15:19:45.387UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.4:2551] - Marking unreachable node [akka.tcp://klusterd@172.17.0.3:2551] as [Down]
INFO 15:19:46.412UTC akka.cluster.Cluster(akka://klusterd) - Cluster Node [akka.tcp://klusterd@172.17.0.4:2551] - Leader is removing unreachable node [akka.tcp://klusterd@172.17.0.3:2551]
INFO 15:19:46.414UTC akka.tcp://klusterd@172.17.0.4:2551/user/cluster-monitor - Cluster member removed: akka.tcp://klusterd@172.17.0.3:2551
WARN 15:19:52.756UTC akka.tcp://klusterd@172.17.0.4:2551/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fklusterd%40172.17.0.3%3A2551-3 - Association with remote system [akka.tcp://klusterd@172.17.0.3:2551] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://klusterd@172.17.0.3:2551]] Caused by: [No response from remote for outbound association. Associate timed out after [15000 ms].]
INFO 15:19:52.758UTC akka.tcp://klusterd@172.17.0.4:2551/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fklusterd%40172.17.0.3%3A2551-4 - No response from remote for outbound association. Associate timed out after [15000 ms].
To turn off everything:
./down.sh