[DCA] Orchestration

Nitrax · September 11, 2019, 2:34pm

Disclaimer: I’m not the author of the content below. I just aggregated public contents to create a knowledge base.

Hi fellas,

Here we go, the first article about the prior mentioned DCA series. Today, we will cover the points approached in the orchestration section of the exam.

Complete the setup of swarm mode cluster with managers and worker nodes

Create a Docker Swarm cluster

Make sure the Docker Engine daemon is started on the host machines.

Open a terminal and ssh into the machine where you want to run your manager node. This tutorial uses a machine named manager1. If you use Docker Machine, you can connect to it via SSH using the following command:

$ docker-machine ssh manager1

Run the following command to create a new swarm:

$ docker swarm init --advertise-addr <MANAGER-IP>

Note: If you are using Docker Desktop for Mac or Docker Desktop for Windows to test single-node swarm, simply run docker swarm init with no arguments. There is no need to specify –advertise-addr in this case. To learn more, see the topic on how to Use Docker Desktop or Mac or Docker Desktop for Windows with Swarm.

In the tutorial, the following command creates a swarm on the manager1 machine:

$ docker swarm init --advertise-addr 192.168.99.100
Swarm initialized: current node (dxn1zf6l61qsb1josjja83ngz) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \
    192.168.99.100:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

The –advertise-addr flag configures the manager node to publish its address as 192.168.99.100. The other nodes in the swarm must be able to access the manager at the IP address.

The output includes the commands to join new nodes to the swarm. Nodes will join as managers or workers depending on the value for the –token flag.

Run docker info to view the current state of the swarm:

$ docker info

Containers: 2
Running: 0
Paused: 0
Stopped: 2
  ...snip...
Swarm: active
  NodeID: dxn1zf6l61qsb1josjja83ngz
  Is Manager: true
  Managers: 1
  Nodes: 1
  ...snip...

Run the docker node ls command to view information about nodes:

$ docker node ls

ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
dxn1zf6l61qsb1josjja83ngz *  manager1  Ready   Active        Leader

The * next to the node ID indicates that you’re currently connected on this node.

Docker Engine swarm mode automatically names the node for the machine hostname. The tutorial covers other columns in later steps.

Add nodes to the swarm

Once you’ve created a swarm with a manager node, you’re ready to add worker nodes.

Open a terminal and ssh into the machine where you want to run a worker node. This tutorial uses the name worker1.
Run the command produced by the docker swarm init output from the Create a swarm tutorial step to create a worker node joined to the existing swarm:

$ docker swarm join \
  --token  SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \
  192.168.99.100:2377

This node joined a swarm as a worker.

If you don’t have the command available, you can run the following command on a manager node to retrieve the join command for a worker:

$ docker swarm join-token worker

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \
    192.168.99.100:2377

Open a terminal and ssh into the machine where you want to run a second worker node. This tutorial uses the name worker2.
Run the command produced by the docker swarm init output from the Create a swarm tutorial step to create a second worker node joined to the existing swarm:

$ docker swarm join \
  --token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \
  192.168.99.100:2377

This node joined a swarm as a worker.

Open a terminal and ssh into the machine where the manager node runs and run the docker node ls command to see the worker nodes:

ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
03g1y59jwfg7cf99w4lt0f662    worker2   Ready   Active
9j68exjopxe7wfl6yuxml7a7j    worker1   Ready   Active
dxn1zf6l61qsb1josjja83ngz *  manager1  Ready   Active        Leader

The MANAGER column identifies the manager nodes in the swarm. The empty status in this column for worker1 and worker2 identifies them as worker nodes.

Swarm management commands like docker node ls only work on manager nodes.

Demo

State the differences between running a container vs running a service

Docker run

The docker run command first creates a writeable container layer over the specified image, and then starts it using the specified command. That is, docker run is equivalent to the API /containers/create then /containers/(id)/start. A stopped container can be restarted with all its previous changes intact using docker start. See docker ps -a to view a list of all containers.

The docker run command can be used in combination with docker commit to change the command that a container runs. There is additional detailed information about docker run in the Docker run reference.

Docker services

When you deploy the service to the swarm, the swarm manager accepts your service definition as the desired state for the service. Then it schedules the service on nodes in the swarm as one or more replica tasks. The tasks run independently of each other on nodes in the swarm.

For example, imagine you want to load balance between three instances of an HTTP listener. The diagram below shows an HTTP listener service with three replicas. Each of the three instances of the listener is a task in the swarm.

A container is an isolated process. In the swarm mode model, each task invokes exactly one container. A task is analogous to a “slot” where the scheduler places a container. Once the container is live, the scheduler recognizes that the task is in a running state. If the container fails health checks or terminates, the task terminates.

Demo

Demonstrate steps to lock a swarm cluster

In Docker 1.13 and higher, the Raft logs used by swarm managers are encrypted on disk by default. This at-rest encryption protects your service’s configuration and data from attackers who gain access to the encrypted Raft logs. One of the reasons this feature was introduced was in support of the new Docker
secrets feature.

When Docker restarts, both the TLS key used to encrypt communication among swarm nodes, and the key used to encrypt and decrypt Raft logs on disk, are loaded into each manager node’s memory. Docker 1.13 introduces the ability to protect the mutual TLS encryption key and the key used to encrypt and decrypt Raft logs at rest, by allowing you to take ownership of these keys and to require manual unlocking of your managers. This feature is called autolock.

When Docker restarts, you must unlock the swarm first, using a key encryption key generated by Docker when the swarm was locked. You can rotate this key encryption key at any time.

Note: You don’t need to unlock the swarm when a new node joins the swarm,
because the key is propagated to it over mutual TLS.

Initialize a swarm with autolocking enabled

When you initialize a new swarm, you can use the –autolock flag to enable autolocking of swarm manager nodes when Docker restarts.

$ docker swarm init --autolock

Swarm initialized: current node (k1q27tfyx9rncpixhk69sa61v) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-0j52ln6hxjpxk2wgk917abcnxywj3xed0y8vi1e5m9t3uttrtu-7bnxvvlz2mrcpfonjuztmtts9 \
    172.31.46.109:2377

To add a manager to this swarm, run **docker swarm join-token manager** and follow the instructions.

To unlock a swarm manager after it restarts, run the **docker swarm unlock** command and provide the following key:

    SWMKEY-1-WuYH/IX284+lRcXuoVf38viIDK3HJEKY13MIHX+tTt8

Store the key in a safe place, such as in a password manager.

When Docker restarts, you need to unlock the swarm. A locked swarm causes an error like the following when you try to start or restart a service:

$ sudo service docker restart

$ docker service ls

Error response from daemon: Swarm is encrypted and needs to be unlocked before it can be used. Use "docker swarm unlock" to unlock it.

Enable or disable autolock on an existing swarm

To enable autolock on an existing swarm, set the autolock flag to true.

$ docker swarm update --autolock=true

Swarm updated.
To unlock a swarm manager after it restarts, run the ***docker swarm unlock*** command and provide the following key:

    SWMKEY-1-+MrE8NgAyKj5r3NcR4FiQMdgu+7W72urH0EZeSmP/0Y

Please remember to store this key in a password manager, since without it you will not be able to restart the manager.

To disable autolock, set –autolock to false. The mutual TLS key and theencryption key used to read and write Raft logs are stored unencrypted on disk. There is a trade-off between the risk of storing the encryption key unencrypted at rest and the convenience of restarting a swarm without needing to unlock each manager.

$ docker swarm update --autolock=false

Keep the unlock key around for a short time after disabling autolocking, in case a manager goes down while it is still configured to lock using the old key.

Unlock a swarm

To unlock a locked swarm, use docker swarm unlock.

$ docker swarm unlock

Please enter unlock key:

Enter the encryption key that was generated and shown in the command output when you locked the swarm or rotated the key, and the swarm unlocks.

View the current unlock key for a running swarm

Consider a situation where your swarm is running as expected, then a manager node becomes unavailable. You troubleshoot the problem and bring the physical node back online, but you need to unlock the manager by providing the unlock key to read the encrypted credentials and Raft logs.

If the key has not been rotated since the node left the swarm, and you have a quorum of functional manager nodes in the swarm, you can view the current unlock key using docker swarm unlock-key without any arguments.

$ docker swarm unlock-key

To unlock a swarm manager after it restarts, run the **docker swarm unlock** command and provide the following key:

    SWMKEY-1-8jDgbUNlJtUe5P/lcr9IXGVxqZpZUXPzd+qzcGp4ZYA

Please remember to store this key in a password manager, since without it you will not be able to restart the manager.

If the key was rotated after the swarm node became unavailable and you do not have a record of the previous key, you may need to force the manager to leave the swarm and join it back to the swarm as a new manager.

Rotate the unlock key

You should rotate the locked swarm’s unlock key on a regular schedule.

$ docker swarm unlock-key --rotate

Successfully rotated manager unlock key.

To unlock a swarm manager after it restarts, run the **docker swarm unlock** command and provide the following key:

    SWMKEY-1-8jDgbUNlJtUe5P/lcr9IXGVxqZpZUXPzd+qzcGp4ZYA

Please remember to store this key in a password manager, since without it you will not be able to restart the manager.

Warning:
When you rotate the unlock key, keep a record of the old key
around for a few minutes, so that if a manager goes down before it gets the new
key, it may still be unlocked with the old one.

Demo

Extend the instructions to run individual containers into running services under swarm

After you create a swarm, you can deploy a service to the swarm. For this tutorial, you also added worker nodes, but that is not a requirement to deploy a service.

Open a terminal and ssh into the machine where you run your manager node. For example, the tutorial uses a machine named manager1.
Run the following command:

$ docker service create --replicas 1 --name helloworld alpine ping docker.com

9uk4639qpg7npwf3fn2aasksr

The docker service create command creates the service.
The –name flag names the service helloworld.
The –replicas flag specifies the desired state of 1 running instance.
The arguments alpine ping docker.com define the service as an Alpine

Linux container that executes the command ping docker.com.

Run docker service ls to see the list of running services:

$ docker service ls

ID            NAME        SCALE  IMAGE   COMMAND
9uk4639qpg7n  helloworld  1/1    alpine  ping docker.com

Interpret the output of docker inspect commands

Required read: Docker inspect

Inspect a service on the swarm

When you have deployed a service to your swarm, you can use the Docker CLI to see details about the service running in the swarm.

If you haven’t already, open a terminal and ssh into the machine where you run your manager node. For example, the tutorial uses a machine named manager1.
Run docker service inspect --pretty to display the details about a service in an easily readable format.

To see the details on the helloworld service:

[manager1]$ docker service inspect --pretty helloworld

ID:		9uk4639qpg7npwf3fn2aasksr
Name:		helloworld
Service Mode:	REPLICATED
    Replicas:		1
Placement:
UpdateConfig:
    Parallelism:	1
ContainerSpec:
    Image:		alpine
    Args:	ping docker.com
Resources:
Endpoint Mode:  vip

Tip: To return the service details in json format, run the same command without the –pretty flag.

[manager1]$ docker service inspect helloworld
[
{
    "ID": "9uk4639qpg7npwf3fn2aasksr",
    "Version": {
        "Index": 418
    },
    "CreatedAt": "2016-06-16T21:57:11.622222327Z",
    "UpdatedAt": "2016-06-16T21:57:11.622222327Z",
    "Spec": {
        "Name": "helloworld",
        "TaskTemplate": {
            "ContainerSpec": {
                "Image": "alpine",
                "Args": [
                    "ping",
                    "docker.com"
                ]
            },
            "Resources": {
                "Limits": {},
                "Reservations": {}
            },
            "RestartPolicy": {
                "Condition": "any",
                "MaxAttempts": 0
            },
            "Placement": {}
        },
        "Mode": {
            "Replicated": {
                "Replicas": 1
            }
        },
        "UpdateConfig": {
            "Parallelism": 1
        },
        "EndpointSpec": {
            "Mode": "vip"
        }
    },
    "Endpoint": {
        "Spec": {}
    }
}
]

Run docker service ps to see which nodes are running the service:

[manager1]$ docker service ps helloworld

NAME                                    IMAGE   NODE     DESIRED STATE  CURRENT STATE           ERROR               PORTS
helloworld.1.8p1vev3fq5zm0mi8g0as41w35  alpine  worker2  Running        Running 3 minutes

In this case, the one instance of the helloworld service is running on the worker2 node. You may see the service running on your manager node. By default, manager nodes in a swarm can execute tasks just like worker nodes.

Swarm also shows you the DESIRED STATE and CURRENT STATE of the service task so you can see if tasks are running according to the service definition.

Run docker ps on the node where the task is running to see details about the container for the task.

Tip: If helloworld is running on a node other than your manager node, you must ssh to that node.

[worker2]$docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
e609dde94e47        alpine:latest       "ping docker.com"   3 minutes ago       Up 3 minutes                            helloworld.1.8p1vev3fq5zm0mi8g0as41w35

Convert an application deployment into a stack file using a YAML compose file with docker stack deploy

Required read Docker Stack Deploy
required read Service Configuration Reference

Demo

Manipulate a running stack of services

Required read Docker Stack Services

Demo

Increase number of replicas

The scale command enables you to scale one or more replicated services either up or down to the desired number of replicas. This command cannot be applied on services which are global mode. The command will return immediately, but the actual scaling of the service may take some time. To stop all replicas of a service while keeping the service active in the swarm you can set the scale to 0.

The following command scales the “frontend” service to 50 tasks.

$ docker service scale frontend=50

frontend scaled to 50

The following command tries to scale a global service to 10 tasks and returns an error.

$ docker service create --mode global --name backend backend:latest

b4g08uwuairexjub6ome6usqh

$ docker service scale backend=10

backend: scale can only be used with replicated mode

Directly afterwards, run docker service ls, to see the actual number of replicas.

$ docker service ls --filter name=frontend

ID            NAME      MODE        REPLICAS  IMAGE
3pr5mlvu3fh9  frontend  replicated  15/50     nginx:alpine

You can also scale a service using the docker service update command. The following commands are equivalent:

$ docker service scale frontend=50
$ docker service update --replicas=50 frontend

The docker service scale command allows you to set the desired number of tasks for multiple services at once. The following example scales both the backend and frontend services:

$ docker service scale backend=3 frontend=5

backend scaled to 3
frontend scaled to 5

$ docker service ls

ID            NAME      MODE        REPLICAS  IMAGE
3pr5mlvu3fh9  frontend  replicated  5/5       nginx:alpine
74nzcxxjv6fq  backend   replicated  3/3       redis:3.0.6

Demo

Add networks, publish ports

When you create a swarm service, you can publish that service’s ports to hosts outside the swarm in two ways:

You can rely on the routing mesh. When you publish a service port, the swarm makes the service accessible at the target port on every node, regardless of whether there is a task for the service running on that node or not. This is less complex and is the right choice for many types of services.
You can publish a service task’s port directly on the swarm node where that service is running. This feature is available in Docker 1.13 and higher. This bypasses the routing mesh and provides the maximum flexibility, including the ability for you to develop your own routing framework. However, you are responsible for keeping track of where each task is running and routing requests to the tasks, and load-balancing across the nodes.

Keep reading for more information and use cases for each of these methods.

Publish a service’s ports using the routing mesh

To publish a service’s ports externally to the swarm, use the –publish : flag. The swarm makes the service accessible at the published port on every swarm node. If an external host connects to that port on any swarm node, the routing mesh routes it to a task. The external host does not need to know the IP addresses or internally-used ports of the service tasks to interact with the service. When a user or process connects to a service, any worker node running a service task may respond. For more details about swarm service networking, see Manage swarm service networks.

Example: Run a three-task Nginx service on 10-node swarm

Imagine that you have a 10-node swarm, and you deploy an Nginx service running three tasks on a 10-node swarm:

$ docker service create --name my_web \
                        --replicas 3 \
                        --publish published=8080,target=80 \
                        nginx

Three tasks run on up to three nodes. You don’t need to know which nodes are running the tasks; connecting to port 8080 on any of the 10 nodes connects you to one of the three nginx tasks. You can test this using curl. The following example assumes that localhost is one of the swarm nodes. If this is not the case, or localhost does not resolve to an IP address on your host, substitute the host’s IP address or resolvable host name.

The HTML output is truncated:

$ curl localhost:8080

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...truncated...
</html>

Subsequent connections may be routed to the same swarm node or a different one.

Publish a service’s ports directly on the swarm node

Using the routing mesh may not be the right choice for your application if you need to make routing decisions based on application state or you need total control of the process for routing requests to your service’s tasks. To publish a service’s port directly on the node where it is running, use the mode=host option to the –publish flag.

Note: If you publish a service’s ports directly on the swarm node using mode=host and also set published= this creates an implicit limitation that you can only run one task for that service on a given swarm node. You can work around this by specifying published without a port definition, which causes Docker to assign a random port for each task.
In addition, if you use mode=host and you do not use the –mode=global flag on docker service create, it is difficult to know which nodes are running the service to route work to them.

Example: Run an `nginx` web server service on every swarm node

nginx is an open source reverse proxy, load balancer, HTTP cache, and a web server. If you run nginx as a service using the routing mesh,connecting to the nginx port on any swarm node shows you the web page for (effectively) a random swarm node running the service.

The following example runs nginx as a service on each node in your swarm and exposes nginx port locally on each swarm node.

$ docker service create \
  --mode global \
  --publish mode=host,target=80,published=8080 \
  --name=nginx \
  nginx:latest

You can reach the nginx server on port 8080 of every swarm node. If you add a node to the swarm, a nginx task is started on it. You cannot start another service or container on any swarm node which binds to port 8080.

Note: This is a naive example. Creating an application-layer routing framework for a multi-tiered service is complex and out of scope for this topic.

Connect the service to an overlay network

You can use overlay networks to connect one or more services within the swarm.

First, create overlay network on a manager node using the docker network create command with the –driver overlay flag.

$ docker network create --driver overlay my-network

After you create an overlay network in swarm mode, all manager nodes have access to the network.

You can create a new service and pass the –network flag to attach the service to the overlay network:

$ docker service create \
  --replicas 3 \
  --network my-network \
  --name my-web \
  nginx

The swarm extends my-network to each node running the service.

You can also connect an existing service to an overlay network using the –network-add flag.

$ docker service update --network-add my-network my-web

To disconnect a running service from a network, use the –network-rm flag.

$ docker service update --network-rm my-network my-web

For more information on overlay networking and service discovery, refer to Attach services to an overlay network and
Docker swarm mode overlay network security model.

Demo

Mount volumes

For best performance and portability, you should avoid writing important data directly into a container’s writable layer, instead using data volumes or bind mounts. This principle also applies to services.

You can create two types of mounts for services in a swarm, volume mounts or bind mounts. Regardless of which type of mount you use, configure it using the –mount flag when you create a service, or the –mount-add or –mount-rm flag when updating an existing service. The default is a data volume if you don’t specify a type.

Data volumes

Data volumes are storage that exist independently of a container. The lifecycle of data volumes under swarm services is similar to that under containers. Volumes outlive tasks and services, so their removal must be managed separately. Volumes can be created before deploying a service, or if they don’t exist on a particular host when a task is scheduled there, they are created automatically according to the volume specification on the service.

To use existing data volumes with a service use the –mount flag:

$ docker service create \
  --mount src=<VOLUME-NAME>,dst=<CONTAINER-PATH> \
  --name myservice \
  <IMAGE>

If a volume with the same does not exist when a task is scheduled to a particular host, then one is created. The default volume driver is local. To use a different volume driver with this create-on-demand pattern, specify the driver and its options with the –mount flag:

$ docker service create \
  --mount type=volume,src=<VOLUME-NAME>,dst=<CONTAINER-PATH>,volume-driver=<DRIVER>,volume-opt=<KEY0>=<VALUE0>,volume-opt=<KEY1>=<VALUE1>
  --name myservice \
  <IMAGE>

For more information on how to create data volumes and the use of volume drivers, see Use volumes.

Bind mounts

Bind mounts are file system paths from the host where the scheduler deploys the container for the task. Docker mounts the path into the container. The file system path must exist before the swarm initializes the container for the task.

The following examples show bind mount syntax:

To mount a read-write bind:

$ docker service create \
  --mount type=bind,src=<HOST-PATH>,dst=<CONTAINER-PATH> \
  --name myservice \
  <IMAGE>

To mount a read-only bind:

$ docker service create \
  --mount type=bind,src=<HOST-PATH>,dst=<CONTAINER-PATH>,readonly \
  --name myservice \
  <IMAGE>

Important: Bind mounts can be useful but they can also cause problems. In most cases, it is recommended that you architect your application such that mounting paths from the host is unnecessary. The main risks include the following:

If you bind mount a host path into your service’s containers, the path must exist on every swarm node. The Docker swarm mode scheduler can schedule containers on any machine that meets resource availability requirements and satisfies all constraints and placement preferences you specify.

The Docker swarm mode scheduler may reschedule your running service containers at any time if they become unhealthy or unreachable.

Host bind mounts are non-portable. When you use bind mounts, there is no guarantee that your application runs the same way in development as it does in production.

Demo

Illustrate running a replicated vs global service

There are two types of service deployments, replicated and global.

For a replicated service, you specify the number of identical tasks you want to run. For example, you decide to deploy an HTTP service with three replicas, each serving the same content.

A global service is a service that runs one task on every node. There is no pre-specified number of tasks. Each time you add a node to the swarm, the orchestrator creates a task and the scheduler assigns the task to the new node. Good candidates for global services are monitoring agents, an anti-virus scanners or other types of containers that you want to run on every node in the swarm.

The diagram below shows a three-service replica in yellow and a global service in gray.

Demo

Identify the steps needed to troubleshoot a service not deploying

A service may be configured in such a way that no node currently in the swarm can run its tasks. In this case, the service remains in state pending. Here are a few examples of when a service might remain in state pending.

Note: If your only intention is to prevent a service from being deployed, scale the service to 0 instead of trying to configure it in such a way that it remains in pending.

If all nodes are paused or drained, and you create a service, it is pending until a node becomes available. In reality, the first node to become available gets all of the tasks, so this is not a good thing to do in a production environment.
You can reserve a specific amount of memory for a service. If no node in the swarm has the required amount of memory, the service remains in a pending state until a node is available which can run its tasks. If you specify a very large value, such as 500 GB, the task stays pending forever, unless you really have a node which can satisfy it.
You can impose placement constraints on the service, and the constraints may not be able to be honored at a given time.

This behavior illustrates that the requirements and configuration of your tasks are not tightly tied to the current state of the swarm. As the administrator of a swarm, you declare the desired state of your swarm, and the manager works with the nodes in the swarm to create that state. You do not need to micro-manage the tasks on the swarm.

Demo

Apply node labels to demonstrate placement of tasks

Required read Docker node update

Placement constraints

Use placement constraints to control the nodes a service can be assigned to. In the following example, the service only runs on nodes with the label region set to east. If no appropriately-labelled nodes are available, tasks will wait in Pending until they become available. The –constraint flag uses an equality operator (== or !=). For replicated services, it is possible that all services run on the same node, or each node only runs one replica, or that some nodes don’t run any replicas. For global services, the service runs on every node that meets the placement constraint and any resource requirements.

$ docker service create \
  --name my-nginx \
  --replicas 5 \
  --constraint node.labels.region==east \
  nginx

You can also use the constraint service-level key in a docker-compose.yml file.

If you specify multiple placement constraints, the service only deploys onto nodes where they are all met. The following example limits the service to run on all nodes where region is set to east and type is not set to devel:

$ docker service create \
  --name my-nginx \
  --mode global \
  --constraint node.labels.region==east \
  --constraint node.labels.type!=devel \
  nginx

You can also use placement constraints in conjunction with placement preferences and CPU/memory constraints. Be careful not to use settings that are not possible to fulfill.

For more information on constraints, refer to the docker service create CLI reference.

Placement preferences

While placement constraints limit the nodes a service can run on, placement preferences try to place tasks on appropriate nodes in an algorithmic way (currently, only spread evenly). For instance, if you assign each node a rack label, you can set a placement preference to spread the service evenly across nodes with the rack label, by value. This way, if you lose a rack, the service is still running on nodes on other racks.

Placement preferences are not strictly enforced. If no node has the label you specify in your preference, the service is deployed as though the preference were not set.

Placement preferences are ignored for global services.

The following example sets a preference to spread the deployment across nodes based on the value of the datacenter label. If some nodes have datacenter=us-east and others have datacenter=us-west, the service is deployed as evenly as possible across the two sets of nodes.

$ docker service create \
  --replicas 9 \
  --name redis_2 \
  --placement-pref 'spread=node.labels.datacenter' \
  redis:3.0.6

Missing or null labels

Nodes which are missing the label used to spread still receive task assignments. As a group, these nodes receive tasks in equal proportion to any of the other groups identified by a specific label value. In a sense, a missing label is the same as having the label with a null value attached to it. If the service should only run on nodes with the label being used for the spread preference, the preference should be combined with a constraint.

You can specify multiple placement preferences, and they are processed in the order they are encountered. The following example sets up a service with multiple placement preferences. Tasks are spread first over the various datacenters, and then over racks (as indicated by the respective labels):

$ docker service create \
  --replicas 9 \
  --name redis_2 \
  --placement-pref 'spread=node.labels.datacenter' \
  --placement-pref 'spread=node.labels.rack' \
  redis:3.0.6

You can also use placement preferences in conjunction with placement constraints or CPU/memory constraints. Be careful not to use settings that are not possible to fulfill.

This diagram illustrates how placement preferences work:

When updating a service with docker service update, –placement-pref-add appends a new placement preference after all existing placement preferences. –placement-pref-rm removes an existing placement preference that matches the argument.

Demo

Sketch how a Dockerized application communicates with legacy systems

Required read Container networking

Paraphrase the importance of quorum in a swarm cluster

Maintain the quorum of managers

If the swarm loses the quorum of managers, the swarm cannot perform management tasks. If your swarm has multiple managers, always have more than two. To maintain quorum, a majority of managers must be available. An odd number of managers is recommended, because the next even number does not make the quorum easier to keep. For instance, whether you have 3 or 4 managers, you can still only lose 1 manager and maintain the quorum. If you have 5 or 6 managers, you can still only lose two.

Even if a swarm loses the quorum of managers, swarm tasks on existing worker nodes continue to run. However, swarm nodes cannot be added, updated, or removed, and new or existing tasks cannot be started, stopped, moved, or updated.

See Recovering from losing the quorum for troubleshooting steps if you do lose the quorum of managers.

Required read Add manager nodes for fault tolerance

Recover from losing the quorum

Swarm is resilient to failures and the swarm can recover from any number of temporary node failures (machine reboots or crash with restart) or other transient errors. However, a swarm cannot automatically recover if it loses a quorum. Tasks on existing worker nodes continue to run, but administrative tasks are not possible, including scaling or updating services and joining or removing nodes from the swarm. The best way to recover is to bring the missing manager nodes back online. If that is not possible, continue reading for some options for recovering your swarm.

In a swarm of N managers, a quorum (a majority) of manager nodes must always be available. For example, in a swarm with 5 managers, a minimum of 3 must be operational and in communication with each other. In other words, the swarm can tolerate up to (N-1)/2 permanent failures beyond which requests involving swarm management cannot be processed. These types of failures include data corruption or hardware failures.

If you lose the quorum of managers, you cannot administer the swarm. If you have lost the quorum and you attempt to perform any management operation on the swarm, an error occurs:

Error response from daemon: rpc error: code = 4 desc = context deadline exceeded

The best way to recover from losing the quorum is to bring the failed nodes back online. If you can’t do that, the only way to recover from this state is to use the –force-new-cluster action from a manager node. This removes all managers except the manager the command was run from. The quorum is achieved because there is now only one manager. Promote nodes to be managers until you have the desired number of managers.

# From the node to recover
docker swarm init --force-new-cluster --advertise-addr node01:2377

When you run the docker swarm init command with the –force-new-cluster flag, the Docker Engine where you run the command becomes the manager node of a single-node swarm which is capable of managing and running services. The manager has all the previous information about services and tasks, worker nodes are still part of the swarm, and services are still running. You need to add or re-add manager nodes to achieve your previous task distribution and ensure that you have enough managers to maintain high availability and prevent losing the quorum.

Demonstrate the usage of templates with docker service create

Required read Create services using templates

Demo

Sources

system · January 11, 2020, 6:34am

This topic was automatically closed after 121 days. New replies are no longer allowed.

[DCA] Orchestration

Complete the setup of swarm mode cluster with managers and worker nodes

Create a Docker Swarm cluster

Add nodes to the swarm

Demo

State the differences between running a container vs running a service

Docker run

Docker services

Demo

Demonstrate steps to lock a swarm cluster

Initialize a swarm with autolocking enabled

Enable or disable autolock on an existing swarm

Unlock a swarm

View the current unlock key for a running swarm

Rotate the unlock key

Demo

Extend the instructions to run individual containers into running services under swarm

Interpret the output of docker inspect commands

Inspect a service on the swarm

Convert an application deployment into a stack file using a YAML compose file with docker stack deploy

Demo

Manipulate a running stack of services

Demo

Increase number of replicas

Demo

Add networks, publish ports

Publish a service’s ports using the routing mesh

Example: Run a three-task Nginx service on 10-node swarm

Publish a service’s ports directly on the swarm node

Example: Run an nginx web server service on every swarm node

Connect the service to an overlay network

Demo

Mount volumes

Data volumes

Bind mounts

Demo

Illustrate running a replicated vs global service

Demo

Identify the steps needed to troubleshoot a service not deploying

Demo

Apply node labels to demonstrate placement of tasks

Placement constraints

Placement preferences

Demo

Sketch how a Dockerized application communicates with legacy systems

Paraphrase the importance of quorum in a swarm cluster

Maintain the quorum of managers

Recover from losing the quorum

Demonstrate the usage of templates with docker service create

Demo

Sources

Example: Run an `nginx` web server service on every swarm node