CoreOS cluster, etcd, a base for Kubernetes

Abstract

In this post, we will install the underlying requirements for a container cluster.

You can use several popular OSes, but if you want to save time, you will want to choose a lightweight OS, like DCOS, Atomic Host, or CoreOS.

In this example, we will choose CoreOS, a well known OS used for this purpose. A very simple os to install, embedding all the technologies for running a container cluster (etcd, ..). Well designed for openstack, it uses some mechanisms from it (user_data description files for lcoud-init..)

Planning your installation

You will need to design your network infrastructure before running the following steps.

For my needs, I will partition a /16 class A ipv4 network:

Network subnet and “range” Used for ..
10.0.0.0/16 Terminals network (laptops, …)
10.0.1.0/16 Servers and network appliances/routers
10.2.0.0/16 Container PODs network
10.3.0.0/24 Service IP range

Requirements (for this setup)

For this setup only because there are lots of choices that can be made, depending of your needs.

My needs : I’m at home, I want to understand, learn, and nevertheless use it my “home production” environnement to host applications (ECM, Mail, …), data, and secure them.

My choices :

  • 3 KVM Hosts, with Centos 7.3 64bit, see my datacenter in two boxes
    3 for computing nodes
    2 for storage nodes

    • First host : 16GB RAM Intel NUC, 120 GB SSD Storage
    • Second host : 64 GB RAM supermicro X10SDV 8 cores ITX, .. 25 TB storage (54 + 51)
    • Third host : 32 GB RAM supermicro X10SDV 4 cores ITX, .. 13 TB storage (25 + 31)
  • Shared storage

    • easy with Ceph. Following this link.

All the software will run in virtual machines on top of that bare metal hosts

IMPORTANT NOTES :

  • I have no special need at home about strong security, so i will install thi cluster without SSL/TLS mechanisms (as far as i can). Note that it’s strongly recommended to use encryption mechanisms in a real production environment, the links provided in this document can lead you to this kind of setup
  • For this example, I do not need two separate networks (public/private). Again, it’s not required at home, but in production environments, it’s strongly recommended.

Install the OSes

We will install one CoreOS VM node on each KVM host at first

The name of my hosts are hyp01,hyp02,hyp03.

The CoreOS part is inspired from another blog

CoreOS deployment

Create the environment

First, create a subdirectory in the directory used to store your VMs.

In my hypervisors, this directory is /mnt/images/libvirt/

Create it under the base directory

mkdir /mnt/images/libvirt/coreos

Download the current production image of CoreOS and name its image for kvm/qemu :

wget https://stable.release.core-os.net/amd64-usr/current/coreos_production_qemu_image.img.bz2 -O - | bzcat > coreos.img

CoreOS use some openstack mechanisms for its deployment process. Download the configuration template

wget https://gist.githubusercontent.com/dutchiechris/e5689a526d4651f6dec4880230fe3c89/raw/4576b956ab00430e644917f17080f4ccfd408615/user_data

Edit this file to follow your needs. Here are mine :

#cloud-config

ssh_authorized_keys:
 - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjgUAV27yJT6YRxzDfGiDB4fwZwmx7EWzcZU3LXRWjaaSgvlizQtHwG8OCJYQN0aG29CTQNgJs+EY40/VQyeidVOdVmaClzmVSMruB68msEuvMrz5DA/v1FXVrYJCAyy+3l719DI9eA++nYyDo//LEj5cf7/4Xcs+12o4ADCJzYMNXazQ8f/1d3EPqJhfcuL+spehCYzDGCYyDDGeIsUaZYnWdOY4z4+2wWtd++9WBfCrVE2g8I3k0+U0iVEM1tZXfrvIzj+fw17zs/FNuzAunQRAIagqUcIswc9aPJbN1H9W31N6gX1X5IV/SvqPl39QXV/wgXtc9M+0oEKJQOvOj core"

hostname: "@@VMNAME@@"

users:
  - name: "user"
    passwd: "$6$D6SWuIMCinNlXP4b$QXnT3RVvPwHcR6ElHP1hWRBxq03gA/TbM1iMKRz3NukT7AYGGf3uSGC9WIkBI1s0GlQ9a1wWUvil.e/OBu6ax/"
    groups:
      - "sudo"

write_files:
  - path: /etc/systemd/resolved.conf
    permissions: 0644
    owner: root
    content: |
      #  This file is part of systemd.
      #
      #  systemd is free software; you can redistribute it and/or modify it
      #  under the terms of the GNU Lesser General Public License as published by
      #  the Free Software Foundation; either version 2.1 of the License, or
      #  (at your option) any later version.
      #
      # Entries in this file show the compile time defaults.
      # You can change settings by editing this file.
      # Defaults can be restored by simply deleting this file.
      #
      # See resolved.conf(5) for details

      [Resolve]
      #DNS=
      #FallbackDNS=
      Domains=int.intra
      #LLMNR=yes
      #DNSSEC=allow-downgrade
      #Cache=yes
      #DNSStubListener=udp

coreos:
  update:
    reboot-strategy: off
  units:
    - name: systemd-networkd.service
      command: stop
    - name: static.network
      runtime: true
      content: |
        [Match]
        Name=eth0

        [Network]
        Address=@@IP@@/16
        Gateway=10.0.0.1
        DNS=10.0.0.1
    - name: down-interfaces.service
      command: start
      content: |
        [Service]
        Type=oneshot
        ExecStart=/usr/bin/ip link set eth0 down
        ExecStart=/usr/bin/ip addr flush dev eth0
    - name: systemd-networkd.service
      command: restart
  • The first line is REQUIRED for CoreOS to recognize a configuration file
  • The ssh_authorized_keys is the pub key of the user you would like to use when connecting to your nodes. I’ve defined this user (“core”) in an admin node, ran the command ssh-keygen and got this key in .ssh/id_rsa.pub
  • the hash of the password is generated with the help of a little python command
    python -c ‘import crypt,getpass; print(crypt.crypt(getpass.getpass(), crypt.mksalt(crypt.METHOD_SHA512)))’

Finally, download this script, thanks to the blog author, that will automate some things (virt-install..)

wget https://gist.githubusercontent.com/dutchiechris/98ba718278cca3c1b7ecbc546aad22f0/raw/a9b3707fd1fc6054b0e23ae308b473c0bec08fd1/manage.py chmod 755 manage.py

Edit this script to suit your needs (my coreos nodes will have 2 CPU and 2 4096MB of ram, you have to specify it in the script)

DNS records

Make the life more easy : declare the 3 nodes in your DNS, so that the script manage.py will be entirely automated.
It will search the DNS for the IPs to grant to the CoreOS hosts.

CoreOS deployment

For my example, I will run the following command on each of my kvm hosts.
Example follows for the 2nd one : the prefix of the host is “c” for CoreOS or Container, I start at 2 for 1 host, the name will be c2, and the script automatically gets the IP from the DNS according to the previous subchapter.

[root@hyp02 coreos]# ./manage.py --action create --start 2 --count 1 --prefix c --ip dns
INFO: c2: Creating...
INFO: c2: Using IP 192.168.10.192
INFO: c2: Creating cloud-config
INFO: c2: Creating ISO image of cloud-config
INFO: c2: Copying master boot image for VM

Starting install...
Creating domain...                                                                                                                                                                                                |    0 B  00:00:00     
Domain creation completed.
INFO: c2: VM created and booting...

Several seconds (not minutes..) later, you will be able to connect to your nodes with the user you declared before, with SSH, without any password thanks to your SSH public key

[core@admin .ssh]$ ssh core@c2
The authenticity of host 'c2 (192.168.10.192)' can't be established.
ECDSA key fingerprint is b2:18:42:8c:94:3f:c2:4d:29:08:e6:7e:44:9b:7f:1e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'c2,192.168.10.192' (ECDSA) to the list of known hosts.
Container Linux by CoreOS stable (1353.8.0)
Update Strategy: No Reboots
core@c2 ~ $ sudo sh
sh-4.3#

If you want to debug the cloud-init process, you can use this command :
journalctl –identifier=coreos-cloudinit

Done ! And done quickly …

Add ETCD to your config.

Once you have 3 running CoreOS nodes, you will have to deploy ETCD.

It’s strongly recommended to automate the installation. -> We will use the cloud config file (user_data) we have just created to deploy our CoreOS nodes.

… Yes ! you have to destroy the three nodes we’ve just created, but remember : it’s done and done quickly :

Destroy your VMS

On all nodes : (using my second KVM host for the example)

[root@hyp02 coreos]# cd /mnt/images/libvirt/coreos
[root@hyp02 coreos]# virsh destroy c2
[root@hyp02 coreos]# virsh undefine c2
[root@hyp02 coreos]# rm -Rf ../c2

It was really quickly done uh ?

configure ETCD discovery

I will use DNS for ETCD discovery (service that populates the nodes) instead of a web service. Look here

I will define this three hosts :

(I use bind on openwrt)

10.0/16 rev zone :
100.1.0.10.in-addr.arpa. IN PTR c0.int.intra.
101.1.0.10.in-addr.arpa. IN PTR c1.int.intra.
102.1.0.10.in-addr.arpa. IN PTR c2.int.intra.

main zone :
c0 A 10.0.1.100
c1 A 10.0.1.101
c2 A 10.0.1.102

Add SRV records :

`_etcd-server._tcp.int.intra. 300 IN  SRV  0 0 2380 c0.int.intra.
_etcd-server._tcp.int.intra. 300 IN  SRV  0 0 2380 c1.int.intra.
_etcd-server._tcp.int.intra. 300 IN  SRV  0 0 2380 c2.int.intra.
_etcd-client._tcp.int.intra. 300 IN SRV 0 0 2379 c0.int.intra.
_etcd-client._tcp.int.intra. 300 IN SRV 0 0 2379 c1.int.intra.
_etcd-client._tcp.int.intra. 300 IN SRV 0 0 2379 c2.int.intra.

And test :

[core@admin ~]$ dig +noall +answer SRV _etcd-client._tcp.int.intra
_etcd-client._tcp.int.intra. 300 IN    SRV    0 0 2379 c1.int.intra.
_etcd-client._tcp.int.intra. 300 IN    SRV    0 0 2379 c0.int.intra.
_etcd-client._tcp.int.intra. 300 IN    SRV    0 0 2379 c2.int.intra.
[core@admin ~]$ dig +noall +answer SRV _etcd-server._tcp.int.intra
_etcd-server._tcp.int.intra. 300 IN    SRV    0 0 2380 c0.int.intra.
_etcd-server._tcp.int.intra. 300 IN    SRV    0 0 2380 c1.int.intra.
_etcd-server._tcp.int.intra. 300 IN    SRV    0 0 2380 c2.int.intra.
[core@admin ~]$``

Bootstrap your coreos/etcdv3 cluster

Hmm, I’m writing this part of the job 2 days after beginning working on that. The documentation on coreos.com is not very “steb by step” like, and this post will be useful for you if you want to go fast and not loose your time.

This is the steps I followed. You can go directly to the last item (or etcd v3 in RKT container engine?) but knowing about the other options can be useful to U.

Another explanation : i’ve not sufficiently read and understood ( ;) ) all the documentation to get the steps required to get a cluster quickly
But I’m not alone.. : https://github.com/coreos/etcd/issues/7436

Use puppet ?

Declare an etcd class in your puppetmaster configuration

You already know how to do that

include etcd

class { 'etcd':
  ensure                     => 'latest',
  listen_client_urls          => 'http://0.0.0.0:2379',
  advertise_client_urls       => "http://${::fqdn}:2379,http://127.0.0.1:2379",
  listen_peer_urls            => 'http://0.0.0.0:2380',
  initial_advertise_peer_urls => "http://${::fqdn}:2380,http://127.0.0.1:2379",
  initial_cluster             => [
    "${::hostname}=http://${::fqdn}:2380",
    'c0=http://c0.int.intra:2380',
    'c1=http://c1.int.intra:2380',
    'c2=http://c2.int.intra:2380'
  ],
}

…but unfortunately, the existing documentation is not as complete as you might want.. i’ve spent 3 hours debugging the puppet docker agent, and finally found he advertised as OS “Linux” and not Redhat or Debian. the class etcd expect Redhat or Debian, on my puppetmaster (existing puppet master). Perhaps the agent must be used with the docker-based puppetmaster, i will test it later.

For the moment, I choosed a simplest way … see next subchapter..

or etcd v3 in RKT container engine…

ETCD v3 is the last version of this distributed registry, it’s supported by Kubernetes 1.6 and later.

Although CoreOS is distributed with etc and etc2, they might disappear at the beginning of 2018.

CoreOS choosed to embed ETCD v3 in RKT containers (not docker by default). It’s easy, but the step by step procedure is too much strong to find. The current documentation do not lead you to the result as quick as it should.

Please follow these steps to get a fully functionnal 3 nodes coreos/etcd cluster :

Enabling RKT to use DNS..

I’ve had to follow a lot of web pages to get it work :

3 hours later, one had found a cloud config file that does the trick :-) :

#cloud-config

ssh_authorized_keys:
 - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjgUAV27yJT6YRxzDfGiDB4fwZwmx7EWzcZU3LXRWjaaSgvlizQtHwG8OCJ    YQN0aG29CTQNgJs.....................v1FXVrYJCAyy+3l719DI9eA++nYyDo//LEj5cf7/4Xcs+12o4ADCJzYMNXazQ8f/1d3EPqJhfcuL+spehCYzDGCYyDDGeIsUaZYnWdOY4z4+2wWtd++9WBfCrVE2g8I3k0+U0iVEM1tZXfrvIzj+fw17zs/FNuzAunQRAIagqUcIswc9aPJbN1H9W31N6gX1X5IV/SvqPl39QXV/wgXtc9M+0oEKJQOvOj core"

hostname: "@@VMNAME@@"

users:
  - name: "user"
    passwd: "$6$D6SWuIMCinNlXP4b$QXnT3RVvPwHcR6ElHP1hWRBxq03gA/TbM1iMKRz3NukT7...a1wWUvil.e/OBu6ax/"
groups:
  - "sudo"

write_files:
  - path: /etc/systemd/resolved.conf
    permissions: 0644
    owner: root
    content: |
      #  This file is part of systemd.
      #
      #  systemd is free software; you can redistribute it and/or modify it
      #  under the terms of the GNU Lesser General Public License as published by
      #  the Free Software Foundation; either version 2.1 of the License, or
      #  (at your option) any later version.
      #
      # Entries in this file show the compile time defaults.
      # You can change settings by editing this file.
      # Defaults can be restored by simply deleting this file.
      #
      # See resolved.conf(5) for details

      [Resolve]
      #DNS=
      #FallbackDNS=
      Domains=int.intra
      #LLMNR=yes
      #DNSSEC=allow-downgrade
      #Cache=yes
      #DNSStubListener=udp

write_files:
  - path: /etc/systemd/system/etcd-member.service.d/override.conf
    permissions: 0644
    owner: root
    content: |
      [Service]
      Environment="ETCD_IMAGE_TAG=v3.2.0"
      Environment="ETCD_DATA_DIR=/var/lib/etcd"
      Environment="ETCD_SSL_DIR=/etc/ssl/certs"
      Environment="ETCD_OPTS=--name=@@VMNAME@@ --listen-client-urls=http://@@IP@@:2379    --advertise-client-urls=http://@@IP@@:2379 --listen-peer-urls=http://@@IP@@:2380     --initial-advertise-peer-urls=http://@@IP@@:2380 --initial-cluster-token=my-etcd-token --auto-compaction-retention=1 --discovery-srv=int.intra"

coreos:
  update:
    reboot-strategy: off
  units:
    - name: systemd-networkd.service
      command: stop
    - name: static.network
      runtime: true
  content: |
    [Match]
    Name=eth0

        [Network]
        Address=@@IP@@/16
        Gateway=10.0.0.1
        DNS=10.0.0.1
    - name: down-interfaces.service
      command: start
      content: |
        [Service]
        Type=oneshot
        ExecStart=/usr/bin/ip link set eth0 down
        ExecStart=/usr/bin/ip addr flush dev eth0
    - name: systemd-networkd.service
      command: restart
    - name: systemd-resolved.service
      command: restart
    - name: etcd-member.service
      command: start

Please take a look at all the config options of my user_data cloud-config file.
Replace any of my option with yours (int.intra is y internal DNS name, the IP are mine… )

Now just start your three nodes and hop.

Check your new ETCD v3 cluster :

You have to het access to a etcdctl binary. you can build it and run outside the cluster, or use the running containers.

For the example, i will run the embedded etcdctl of my brand new containers :

/ # c0 ~ # rkt list
UUID        APP    IMAGE NAME            STATE    CREATED        STARTED        NETWORKS
21c1654b    etcd    quay.io/coreos/etcd:v3.2.0    running    23 minutes ago    23 minutes ago
c0 ~ # rkt enter 21c1654b /bin/sh
/ #

And issue these commands :

c0 ~ # rkt enter 21c1654b /bin/sh
/ # export ETCDCTL_API=3 ; export ETCDCTL_ENDPOINTS=http://c0:2379;     /usr/local/bin/etcdctl member list
65ac5f05513b1262, started, c1, http://c1.int.intra:2380, http://10.0.1.101:2379
6c2733a4aad35dac, started, c0, http://c0.int.intra:2380, http://10.0.1.100:2379
84d5ec72bbd714d2, started, c2, http://c2.int.intra:2380, http://10.0.1.102:2379
/ # export ETCDCTL_API=3 ; export ETCDCTL_ENDPOINTS=http://c0:2379; /usr/local/bin/etcdctl put foo bar
OK
/ # export ETCDCTL_API=3 ; export ETCDCTL_ENDPOINTS=http://c0:2379; /usr/local/bin/etcdctl get foo
foo
bar
/ #

Verifying on another node…

/ #  export ETCDCTL_API=3 ; export ETCDCTL_ENDPOINTS=http://c2:2379; /usr/local/bin/etcdctl get foo
foo
bar

Good..

/ # export ETCDCTL_API=3 ; export ETCDCTL_ENDPOINTS=http://c0:2379; /usr/local/bin/etcdctl check perf
 60 / 60 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00%1m0s
PASS: Throughput is 150 writes/s
PASS: Slowest request took 0.264878s
PASS: Stddev is 0.033646s
PASS

Ready for Kubernetes !