Running akka cluster on docker swarm

Benefits of running an akka cluster are not so obvious while your app is stateless, you just run as many instances as you want and balance the traffic between them, but once state comes in - clustering can really help you solve various problems.

Expressjs hello world index page

You may have some shared state and - “you may need to run a single actor instance”, which I’ll use as demo case in this post.

The purpose of this note is to walk you through the process of setting up and run an akka cluster on a docker swarm, and rolling updates with (almost?) zero downtime. Why almost? - most of the time this downtime criteria depends on specifics of your app, ex: detecting a node is down - takes time, scheduling an actor on another node also takes some time, and if you have hard limits on that actor response latency - the app will be “down” for short interval, to fix it - you’ll need to go deeper on this topic, avoid single point of failures, configure timeouts and implement retries.

If you’re reding this - you’re probably already know what akka is, so no details are following.

Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications for Java and Scala

Application

The demo used in this setup - is a simple akka application running single TickActor which logs TIC or TAC text periodically. The actor is scheduled using ClusterSingletonManager

val tickActor: ActorRef = system.actorOf(
    ClusterSingletonManager.props(
      singletonProps = Props(classOf[TickActor], executionContext),
      terminationMessage = End,
      settings = ClusterSingletonManagerSettings(system).withRole("worker")
    ),
    name = "counter"
)

Application is assembled into a docker container using sbt-assembly and sbt-docker plugins, and published to docker hub: https://hub.docker.com/r/lostintime/hello-akka/.

Basically, to form an akka cluster - each application node need to know at least one other node, the list of seed nodes may be added to application.conf

akka {
  ...
  cluster {
    seed-nodes = [
        "akka.tcp://[email protected]:2552",
        "akka.tcp://[email protected]:2553"
    ]
  }
  ...
}

You need at least one seed node to start, but on production setups - is recomended to have 2 or more, to avoid split-brain states: if you will restart your single seed node - it will start a “new cluster” because it doesn’t know about the rest of the world. With 2 nodes, which knows each other - restart them sequentially so each will re-join existing cluster.

To define our application cluster - we will use a docker compose configuration file:

version: '3'

services:
  app-seed1:
    image: "lostintime/hello-akka:0.1.1"
    command: -Dakka.cluster.seed-nodes.0=akka.tcp://HelloAkka@app-seed1:2552
             -Dakka.cluster.seed-nodes.1=akka.tcp://HelloAkka@app-seed2:2552
             -jar /app/hello-akka.jar
    environment:
     - 'PUBLISH_HOST=app-seed1'
    stop_grace_period: 15s
    deploy:
      mode: replicated
      replicas: 1 # DO NOT SCALE THIS SERVICE!
      update_config:
        parallelism: 1
        delay: 15s
      restart_policy:
        condition: any
        delay: 20s
        max_attempts: 8
        window: 3m
  app-seed2:
    image: "lostintime/hello-akka:0.1.1"
    command: -Dakka.cluster.seed-nodes.0=akka.tcp://HelloAkka@app-seed1:2552
             -Dakka.cluster.seed-nodes.1=akka.tcp://HelloAkka@app-seed2:2552
             -jar /app/hello-akka.jar
    environment:
     - 'PUBLISH_HOST=app-seed2'
    stop_grace_period: 15s
    deploy:
      mode: replicated
      replicas: 1 # DO NOT SCALE THIS SERVICE!
      update_config:
        parallelism: 1
        delay: 15s
      restart_policy:
        condition: any
        delay: 20s
        max_attempts: 8
        window: 3m
  app:
    image: "lostintime/hello-akka:0.1.1"
    command: -Dakka.cluster.seed-nodes.0=akka.tcp://HelloAkka@app-seed1:2552
             -Dakka.cluster.seed-nodes.0=akka.tcp://HelloAkka@app-seed2:2552
             -Dakka.cluster.roles.0=worker
             -jar /app/hello-akka.jar
    stop_grace_period: 15s
    deploy:
      mode: replicated
      replicas: 1
      update_config:
        parallelism: 1
        delay: 15s
      restart_policy:
        condition: any
        delay: 20s
        max_attempts: 8
        window: 3m

So, in compose file we have 3 services:

app-seed1, app-seed2 - seed nodes services, you must run only 1 instance of each, otherwise you can end up in an inconsistent state, traffic will be split by docker between instances;
app service - our application which can we scaled as we need, it uses our seed services as seed nodes

Seed nodes configuration is passed using command line arguments:

    command: -Dakka.cluster.seed-nodes.0=akka.tcp://HelloAkka@app-seed1:2552
             -Dakka.cluster.seed-nodes.0=akka.tcp://HelloAkka@app-seed2:2552
             -Dakka.cluster.roles.0=worker
             -jar /app/hello-akka.jar"

app service have also an optional role set: -Dakka.cluster.roles.0=worker which is to limit nodes on which TickActor may run.

deploy: options are used only when you’re running a docker service on a swarm cluster with docker stack deploy command, a bit more details on those you can find in docker deployments with zero downtime post or compose file and docker service create documentation.

Pitfalls

Here are some issues you may have while trying to deploy this to production:

First Seed Node

As mentioned in docs - first seed node have a special role:

http://doc.akka.io/docs/akka/current/scala/cluster-usage.html

The seed nodes can be started in any order and it is not necessary to have all seed nodes running, but the node configured as the first element in the seed-nodes configuration list must be started when initially starting a cluster, otherwise the other seed-nodes will not become initialized and no other node can join the cluster. The reason for the special first seed node is to avoid forming separated islands when starting from an empty cluster.

First seed node will try by default to connect to other nodes, not to itself, only if it fails - then it will start a new cluster. This is what PUBLISH_HOST=app-seed1 environment variable is used for - so akka ca detect self as fist seed node.

Cure: configure one, and only one node having itself as first seed node, otherwise cluster may be split in few independent clusters.

Healthcheck

Docker HEALTHCHECK feature is another thing which may break cluster joining, ex: docker swarm will not configure dns for starting node until it’s healthy (which may take more time than starting akka system), but akka may already start cluster join routine and other nodes will not be able to reach it, so cluster join will fail.

Cure: set seed-node-timeout configuration option to an interval which exceeds node startup time to healthy state. Avoid using remote actors for defining node healthy state (kind of dead lock).

Downing

By default, if a node cannot reach other one - it will mark that node as unreachable and will wait indefinitely for it to re-join, but docker - may re-schedule an instance for a service with different IP address, so old node’s url will wait in unreachable list forever. There is auto-down-unreachable-after config option available, but it should not be used in production (same issue, split-brain state).

Cure: use custom downing which satisfy your needs :). You can find an interesting solution in this post: Building a docker container orchestrator with akka, the core idea is:

… we only mark down unreachable nodes if it detects more up nodes than unreachable nodes.

Source: https://github.com/roberveral/docker-akka/blob/master/src/main/scala/com/github/roberveral/dockerakka/cluster/CustomDowning.scala

Restart policy

In order to keep nodes alive - we usually configure service restart policy, on-falure may seem a logical choice, but! a node which was unreachable for a while (ex: network issues) then came back, but was already Downed by the cluster - will gracefully shut down (exit status 0), so docker will not try to restart it! cause it didn’t fail. At the end - all cluster nodes may gracefully go down.

Cure: use any as service restart condition. unless-stopped option is not available for services, but you also cannot stop services, only remove or scale, so - don’t need it.

Container Versioning

In manual seed nodes cluster setup mode - is important to restart services sequentially, so at least one seed node will be available, app-seed1 - will re-join cluster using app-seed2, then app-seed2 will join using app-seed1.

When you’re updating your services with docker stack deploy - be careful with changes done for every service in your compose or bundle file, change one by one!

Also - docker will first pull latest images from registry, and if image digest hash changed - it will start service update, so when you’re using latest tag for all services (seed1, seed2 and app) - all services will update same time.

Cure: use versioning, never update published containers, use new version tag so old containers never change.

Run

There are 2 options to run akka cluster on docker: using docker-compose or as a docker stack:

Run with docker-compose

$ docker-compose up

WARNING: Some services (app, app-seed1, app-seed2) use the 'deploy' key, which will be ignored. Compose does not support deploy configuration - use `docker stack deploy` to deploy to a swarm.
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating helloakka_app-seed1_1
Creating helloakka_app_1
Creating helloakka_app-seed2_1
Attaching to helloakka_app_1, helloakka_app-seed1_1, helloakka_app-seed2_1
app_1        | [INFO] [05/31/2017 06:26:26.409] [main] [akka.remote.Remoting] Starting remoting
app-seed1_1  | [INFO] [05/31/2017 06:26:26.440] [main] [akka.remote.Remoting] Starting remoting
app_1        | [INFO] [05/31/2017 06:26:26.631] [main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://[email protected]:2552]
app-seed1_1  | [INFO] [05/31/2017 06:26:26.704] [main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://HelloAkka@app-seed1:2552]
app_1        | [INFO] [05/31/2017 06:26:26.702] [main] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://[email protected]:2552] - Starting up...
app-seed1_1  | [INFO] [05/31/2017 06:26:26.722] [main] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed1:2552] - Starting up...
app-seed2_1  | [INFO] [05/31/2017 06:26:26.746] [main] [akka.remote.Remoting] Starting remoting
app_1        | [INFO] [05/31/2017 06:26:26.871] [main] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://[email protected]:2552] - Registered cluster JMX MBean [akka:type=Cluster]
app_1        | [INFO] [05/31/2017 06:26:26.871] [main] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://[email protected]:2552] - Started up successfully
app_1        | [INFO] [05/31/2017 06:26:26.919] [HelloAkka-akka.actor.default-dispatcher-4] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://[email protected]:2552] - Metrics collection has started successfully
app_1        | [INFO] [05/31/2017 06:26:26.935] [main] [Boot$(akka://HelloAkka)] server is online
app-seed1_1  | [INFO] [05/31/2017 06:26:26.946] [main] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed1:2552] - Registered cluster JMX MBean [akka:type=Cluster]
app-seed1_1  | [INFO] [05/31/2017 06:26:26.947] [main] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed1:2552] - Started up successfully
app-seed2_1  | [INFO] [05/31/2017 06:26:27.033] [main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://HelloAkka@app-seed2:2552]
app-seed1_1  | [INFO] [05/31/2017 06:26:27.089] [HelloAkka-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed1:2552] - Metrics collection has started successfully
app-seed1_1  | [INFO] [05/31/2017 06:26:27.098] [main] [Boot$(akka://HelloAkka)] server is online
app-seed2_1  | [INFO] [05/31/2017 06:26:27.125] [main] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed2:2552] - Starting up...
app-seed1_1  | [ERROR] [05/31/2017 06:26:27.115] [HelloAkka-akka.actor.default-dispatcher-18] [akka://HelloAkka/user/counter] requirement failed: This cluster member [akka.tcp://HelloAkka@app-seed1:2552] doesn't have the role [Some(worker)]
app-seed1_1  | akka.actor.ActorInitializationException: akka://HelloAkka/user/counter: exception during creation
app-seed1_1  |  at akka.actor.ActorInitializationException$.apply(Actor.scala:191)
app-seed1_1  |  at akka.actor.ActorCell.create(ActorCell.scala:608)
app-seed1_1  |  at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
app-seed1_1  |  at akka.actor.ActorCell.systemInvoke(ActorCell.scala:484)
app-seed1_1  |  at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:282)
app-seed1_1  |  at akka.dispatch.Mailbox.run(Mailbox.scala:223)
app-seed1_1  |  at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
app-seed1_1  |  at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
app-seed1_1  |  at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
app-seed1_1  |  at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
app-seed1_1  |  at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
app-seed1_1  | Caused by: java.lang.IllegalArgumentException: requirement failed: This cluster member [akka.tcp://HelloAkka@app-seed1:2552] doesn't have the role [Some(worker)]
app-seed1_1  |  at scala.Predef$.require(Predef.scala:277)
app-seed1_1  |  at akka.cluster.singleton.ClusterSingletonManager.<init>(ClusterSingletonManager.scala:427)
app-seed1_1  |  at akka.cluster.singleton.ClusterSingletonManager$.$anonfun$props$1(ClusterSingletonManager.scala:135)
app-seed1_1  |  at akka.actor.TypedCreatorFunctionConsumer.produce(IndirectActorProducer.scala:87)
app-seed1_1  |  at akka.actor.Props.newActor(Props.scala:213)
app-seed1_1  |  at akka.actor.ActorCell.newActor(ActorCell.scala:563)
app-seed1_1  |  at akka.actor.ActorCell.create(ActorCell.scala:589)
app-seed1_1  |  ... 9 more
app-seed1_1  | 
app-seed2_1  | [INFO] [05/31/2017 06:26:27.350] [main] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed2:2552] - Registered cluster JMX MBean [akka:type=Cluster]
app-seed2_1  | [INFO] [05/31/2017 06:26:27.351] [main] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed2:2552] - Started up successfully
app-seed2_1  | [INFO] [05/31/2017 06:26:27.374] [HelloAkka-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed2:2552] - Received InitJoin message from [Actor[akka.tcp://[email protected]:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#1923915470]], but this node is not initialized yet
app-seed2_1  | [INFO] [05/31/2017 06:26:27.384] [HelloAkka-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed2:2552] - Received InitJoin message from [Actor[akka.tcp://HelloAkka@app-seed1:2552/system/cluster/core/daemon/firstSeedNodeProcess-1#-1490393690]], but this node is not initialized yet
app-seed2_1  | [INFO] [05/31/2017 06:26:27.409] [main] [Boot$(akka://HelloAkka)] server is online
app-seed1_1  | [INFO] [05/31/2017 06:26:27.426] [HelloAkka-akka.actor.default-dispatcher-14] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed1:2552] - Received InitJoinNack message from [Actor[akka.tcp://HelloAkka@app-seed2:2552/system/cluster/core/daemon#1230459430]] to [akka.tcp://HelloAkka@app-seed1:2552]
app-seed2_1  | [ERROR] [05/31/2017 06:26:27.434] [HelloAkka-akka.actor.default-dispatcher-3] [akka://HelloAkka/user/counter] requirement failed: This cluster member [akka.tcp://HelloAkka@app-seed2:2552] doesn't have the role [Some(worker)]
app-seed2_1  | akka.actor.ActorInitializationException: akka://HelloAkka/user/counter: exception during creation
app-seed2_1  |  at akka.actor.ActorInitializationException$.apply(Actor.scala:191)
app-seed2_1  |  at akka.actor.ActorCell.create(ActorCell.scala:608)
app-seed2_1  |  at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
app-seed2_1  |  at akka.actor.ActorCell.systemInvoke(ActorCell.scala:484)
app-seed2_1  |  at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:282)
app-seed2_1  |  at akka.dispatch.Mailbox.run(Mailbox.scala:223)
app-seed2_1  |  at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
app-seed2_1  |  at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
app-seed2_1  |  at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
app-seed2_1  |  at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
app-seed2_1  |  at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
app-seed2_1  | Caused by: java.lang.IllegalArgumentException: requirement failed: This cluster member [akka.tcp://HelloAkka@app-seed2:2552] doesn't have the role [Some(worker)]
app-seed2_1  |  at scala.Predef$.require(Predef.scala:277)
app-seed2_1  |  at akka.cluster.singleton.ClusterSingletonManager.<init>(ClusterSingletonManager.scala:427)
app-seed2_1  |  at akka.cluster.singleton.ClusterSingletonManager$.$anonfun$props$1(ClusterSingletonManager.scala:135)
app-seed2_1  |  at akka.actor.TypedCreatorFunctionConsumer.produce(IndirectActorProducer.scala:87)
app-seed2_1  |  at akka.actor.Props.newActor(Props.scala:213)
app-seed2_1  |  at akka.actor.ActorCell.newActor(ActorCell.scala:563)
app-seed2_1  |  at akka.actor.ActorCell.create(ActorCell.scala:589)
app-seed2_1  |  ... 9 more
app-seed2_1  | 
app-seed1_1  | [INFO] [05/31/2017 06:26:27.450] [HelloAkka-akka.actor.default-dispatcher-14] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed1:2552] - Node [akka.tcp://HelloAkka@app-seed1:2552] is JOINING, roles []
app-seed2_1  | [INFO] [05/31/2017 06:26:27.453] [HelloAkka-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed2:2552] - Metrics collection has started successfully
app-seed1_1  | [INFO] [05/31/2017 06:26:27.465] [HelloAkka-akka.actor.default-dispatcher-14] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed1:2552] - Leader is moving node [akka.tcp://HelloAkka@app-seed1:2552] to [Up]
app-seed1_1  | [INFO] [05/31/2017 06:26:27.469] [HelloAkka-akka.actor.default-dispatcher-14] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed1:2552] - Received InitJoin message from [Actor[akka.tcp://HelloAkka@app-seed2:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#26925358]] to [akka.tcp://HelloAkka@app-seed1:2552]
app-seed1_1  | [INFO] [05/31/2017 06:26:27.469] [HelloAkka-akka.actor.default-dispatcher-14] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed1:2552] - Sending InitJoinAck message from node [akka.tcp://HelloAkka@app-seed1:2552] to [Actor[akka.tcp://HelloAkka@app-seed2:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#26925358]]
app-seed1_1  | [INFO] [05/31/2017 06:26:27.505] [HelloAkka-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed1:2552] - Node [akka.tcp://HelloAkka@app-seed2:2552] is JOINING, roles []
app-seed2_1  | [INFO] [05/31/2017 06:26:27.630] [HelloAkka-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed2:2552] - Welcome from [akka.tcp://HelloAkka@app-seed1:2552]
app-seed1_1  | [INFO] [05/31/2017 06:26:28.041] [HelloAkka-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed1:2552] - Leader is moving node [akka.tcp://HelloAkka@app-seed2:2552] to [Up]
app-seed2_1  | [INFO] [05/31/2017 06:26:32.432] [HelloAkka-akka.actor.default-dispatcher-18] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed2:2552] - Received InitJoin message from [Actor[akka.tcp://[email protected]:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#1923915470]] to [akka.tcp://HelloAkka@app-seed2:2552]
app-seed2_1  | [INFO] [05/31/2017 06:26:32.432] [HelloAkka-akka.actor.default-dispatcher-18] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed2:2552] - Sending InitJoinAck message from node [akka.tcp://HelloAkka@app-seed2:2552] to [Actor[akka.tcp://[email protected]:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#1923915470]]
app-seed2_1  | [INFO] [05/31/2017 06:26:32.456] [HelloAkka-akka.actor.default-dispatcher-4] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed2:2552] - Node [akka.tcp://[email protected]:2552] is JOINING, roles [worker]
app_1        | [INFO] [05/31/2017 06:26:32.479] [HelloAkka-akka.actor.default-dispatcher-19] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://[email protected]:2552] - Welcome from [akka.tcp://HelloAkka@app-seed2:2552]
app-seed1_1  | [INFO] [05/31/2017 06:26:34.027] [HelloAkka-akka.actor.default-dispatcher-13] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://HelloAkka@app-seed1:2552] - Leader is moving node [akka.tcp://[email protected]:2552] to [Up]
app_1        | [INFO] [05/31/2017 06:26:34.053] [HelloAkka-akka.actor.default-dispatcher-15] [akka.tcp://[email protected]:2552/user/counter] Singleton manager starting singleton actor [akka://HelloAkka/user/counter/singleton]
app_1        | [INFO] [05/31/2017 06:26:34.054] [HelloAkka-akka.actor.default-dispatcher-15] [akka.tcp://[email protected]:2552/user/counter] ClusterSingletonManager state change [Start -> Oldest]
app_1        | [INFO] [05/31/2017 06:26:39.071] [HelloAkka-akka.actor.default-dispatcher-16] [akka.tcp://[email protected]:2552/user/counter/singleton] TAC

app_1        | [INFO] [05/31/2017 06:26:49.068] [HelloAkka-akka.actor.default-dispatcher-3] [akka.tcp://[email protected]:2552/user/counter/singleton] TIC
app_1        | [INFO] [05/31/2017 06:26:59.068] [HelloAkka-akka.actor.default-dispatcher-15] [akka.tcp://[email protected]:2552/user/counter/singleton] TAC
app_1        | [INFO] [05/31/2017 06:27:09.068] [HelloAkka-akka.actor.default-dispatcher-2] [akka.tcp://[email protected]:2552/user/counter/singleton] TIC
app_1        | [INFO] [05/31/2017 06:27:19.068] [HelloAkka-akka.actor.default-dispatcher-15] [akka.tcp://[email protected]:2552/user/counter/singleton] TAC
app_1        | [INFO] [05/31/2017 06:27:29.068] [HelloAkka-akka.actor.default-dispatcher-16] [akka.tcp://[email protected]:2552/user/counter/singleton] TIC
app_1        | [INFO] [05/31/2017 06:27:39.068] [HelloAkka-akka.actor.default-dispatcher-18] [akka.tcp://[email protected]:2552/user/counter/singleton] TAC

You may see and ERROR there, akka.actor.ActorInitializationException - it happens because TickActor is allowe to run only on worker nodes, criteria which our seed services doesn’t match.

At the end of the log - you’ll see tic/tac messages, every 30 seconds.

Run with `docker stack`

$ docker stack deploy --compose-file docker-compose.yml hello

Creating network hello_default
Creating service hello_app
Creating service hello_app-seed1
Creating service hello_app-seed2

The output is shorter here, it doesn’t collect output from containers like docker-compose.

We can list our services:

$ docker service ls

ID            NAME             MODE        REPLICAS  IMAGE
mpyqzr25vkhs  hello_app-seed1  replicated  1/1       lostintime/hello-akka:0.1.0
pd29h05rzp7w  hello_app-seed2  replicated  1/1       lostintime/hello-akka:0.1.0
rc3k40c7sdxc  hello_app        replicated  1/1       lostintime/hello-akka:0.1.0

Some logs for app service (service are prefixed with stack name):

$ docker service logs hello_app

hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:16.600] [main] [akka.remote.Remoting] Starting remoting
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:16.780] [main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://[email protected]:2552]
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:16.829] [main] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://[email protected]:2552] - Starting up...
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:16.956] [main] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://[email protected]:2552] - Registered cluster JMX MBean [akka:type=Cluster]
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:16.956] [main] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://[email protected]:2552] - Started up successfully
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:17.000] [main] [Boot$(akka://HelloAkka)] server is online
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:17.006] [HelloAkka-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://[email protected]:2552] - Metrics collection has started successfully
hello_app.1.q6p1wootqh7k@moby    | [WARN] [05/31/2017 06:34:17.191] [HelloAkka-akka.remote.default-remote-dispatcher-6] [akka.tcp://[email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FHelloAkka%40app-seed2%3A2552-0] Association with remote system [akka.tcp://HelloAkka@app-seed2:2552] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://HelloAkka@app-seed2:2552]] Caused by: [app-seed2: Name or service not known]
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:17.204] [HelloAkka-akka.actor.default-dispatcher-15] [akka://HelloAkka/deadLetters] Message [akka.cluster.InternalClusterAction$InitJoin$] from Actor[akka://HelloAkka/system/cluster/core/daemon/joinSeedNodeProcess-1#1466612957] to Actor[akka://HelloAkka/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:22.061] [HelloAkka-akka.actor.default-dispatcher-17] [akka://HelloAkka/deadLetters] Message [akka.cluster.InternalClusterAction$InitJoin$] from Actor[akka://HelloAkka/system/cluster/core/daemon/joinSeedNodeProcess-1#1466612957] to Actor[akka://HelloAkka/deadLetters] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
hello_app.1.q6p1wootqh7k@moby    | [WARN] [05/31/2017 06:34:27.072] [HelloAkka-akka.actor.default-dispatcher-18] [akka.tcp://[email protected]:2552/system/cluster/core/daemon/joinSeedNodeProcess-1] Couldn't join seed nodes after [2] attempts, will try again. seed-nodes=[akka.tcp://HelloAkka@app-seed2:2552]
hello_app.1.q6p1wootqh7k@moby    | [WARN] [05/31/2017 06:34:27.085] [HelloAkka-akka.remote.default-remote-dispatcher-6] [akka.tcp://[email protected]:2552/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FHelloAkka%40app-seed2%3A2552-1] Association with remote system [akka.tcp://HelloAkka@app-seed2:2552] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://HelloAkka@app-seed2:2552]] Caused by: [app-seed2]
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:27.085] [HelloAkka-akka.actor.default-dispatcher-15] [akka://HelloAkka/deadLetters] Message [akka.cluster.InternalClusterAction$InitJoin$] from Actor[akka://HelloAkka/system/cluster/core/daemon/joinSeedNodeProcess-1#1466612957] to Actor[akka://HelloAkka/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
hello_app.1.q6p1wootqh7k@moby    | [WARN] [05/31/2017 06:34:32.091] [HelloAkka-akka.actor.default-dispatcher-14] [akka.tcp://[email protected]:2552/system/cluster/core/daemon/joinSeedNodeProcess-1] Couldn't join seed nodes after [3] attempts, will try again. seed-nodes=[akka.tcp://HelloAkka@app-seed2:2552]
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:32.243] [HelloAkka-akka.actor.default-dispatcher-14] [akka.cluster.Cluster(akka://HelloAkka)] Cluster Node [akka.tcp://[email protected]:2552] - Welcome from [akka.tcp://HelloAkka@app-seed2:2552]
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:34.044] [HelloAkka-akka.actor.default-dispatcher-17] [akka.tcp://[email protected]:2552/user/counter] Singleton manager starting singleton actor [akka://HelloAkka/user/counter/singleton]
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:34.047] [HelloAkka-akka.actor.default-dispatcher-17] [akka.tcp://[email protected]:2552/user/counter] ClusterSingletonManager state change [Start -> Oldest]
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:39.063] [HelloAkka-akka.actor.default-dispatcher-17] [akka.tcp://[email protected]:2552/user/counter/singleton] TAC
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:49.061] [HelloAkka-akka.actor.default-dispatcher-21] [akka.tcp://[email protected]:2552/user/counter/singleton] TIC
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:34:59.062] [HelloAkka-akka.actor.default-dispatcher-15] [akka.tcp://[email protected]:2552/user/counter/singleton] TAC
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:35:09.062] [HelloAkka-akka.actor.default-dispatcher-15] [akka.tcp://[email protected]:2552/user/counter/singleton] TIC
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:35:19.061] [HelloAkka-akka.actor.default-dispatcher-15] [akka.tcp://[email protected]:2552/user/counter/singleton] TAC
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:35:29.060] [HelloAkka-akka.actor.default-dispatcher-17] [akka.tcp://[email protected]:2552/user/counter/singleton] TIC
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:35:39.061] [HelloAkka-akka.actor.default-dispatcher-2] [akka.tcp://[email protected]:2552/user/counter/singleton] TAC
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:35:49.061] [HelloAkka-akka.actor.default-dispatcher-4] [akka.tcp://[email protected]:2552/user/counter/singleton] TIC
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:35:59.061] [HelloAkka-akka.actor.default-dispatcher-16] [akka.tcp://[email protected]:2552/user/counter/singleton] TAC
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:36:09.061] [HelloAkka-akka.actor.default-dispatcher-18] [akka.tcp://[email protected]:2552/user/counter/singleton] TIC
hello_app.1.q6p1wootqh7k@moby    | [INFO] [05/31/2017 06:36:19.061] [HelloAkka-akka.actor.default-dispatcher-18] [akka.tcp://[email protected]:2552/user/counter/singleton] TAC

As we started our services all same time - you can see that app service failed to join cluster 2 times, but then - succeed, that’s becuase app-seed1 and app-seed2 nodes was also starting same time.

And finally - scale it! (which you can originaly set with deploy.replicas option in compose-file.yml):

$ docker service scale hello_app=5

hello_app scaled to 5

After few moments - swarm will create 4 more app service instances:

$ docker service ls
ID            NAME             MODE        REPLICAS  IMAGE
mpyqzr25vkhs  hello_app-seed1  replicated  1/1       lostintime/hello-akka:0.1.0
pd29h05rzp7w  hello_app-seed2  replicated  1/1       lostintime/hello-akka:0.1.0
rc3k40c7sdxc  hello_app        replicated  5/5       lostintime/hello-akka:0.1.0

That’s actually it! Pretty easy, ha?!

Let’s thank to akka and docker teams and communities for that, You’re doing a great job guys!

See ya!.

Links:

Application source code: https://github.com/lostintime/hello-akka
Docker image: https://hub.docker.com/r/lostintime/hello-akka/
Deploy stack to swarm: https://docs.docker.com/engine/swarm/stack-deploy/
Docker compose file reference: https://docs.docker.com/compose/compose-file/
Akka networking: http://doc.akka.io/docs/akka/snapshot/scala/index-network.html

UPD: on docker version I’m using (17.03.1-ce) there are some connectivity issues: containers end up in a state when services host cannot be resolved for some containers, but it works on others, so cluster breaks :(. I still didn’t figure it out why this happens, so - TEST IT CAREFULLY before going to production!!! and please comment here if you found a solution).

This theoreticaly shouldn’t happen when using one of coordination solutions:

https://github.com/sclasen/akka-zk-cluster-seed
https://github.com/hseeberger/constructr

… it souldn’t use docker service resolving, I’m going to try this in one of my next posts.