firewalld custom rules for OpenShift Container Platform

If you end with below while trying to restart iptables, then firewalld is the service you will be looking at

# systemctl restart iptables 
Failed to restart iptables.service: Unit is masked.

Firewalld service has set of commands, but most notable one is firewall-cmd and if run in help mode, it will present iteself in whole messy glory … try and run!

# firewall-cmd -h

will give you all necessary to proceed to play with firewalld rules.

Useful ones are

# systemctl status firewalld
# firewall-cmd --get-zones
# firewall-cmd --list-all-zones
# firewall-cmd --get-default-zone
# firewall-cmd --get-active-zones
# firewall-cmd --info-zone=public

and hundredths of others, man firewal-cmd is man page to read.

If for some reason we have to change firewalld rules then that could be different experience than most linux users are get used.

In recent OpenShift installation you will notice many firewalld rules created by Openshift installation. An example of input chain is

Chain IN_public_allow (1 references)
  pkts bytes target     prot opt in     out     source               destination         

  598 31928 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22 ctstate NEW
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:2379 ctstate NEW
   24  1052 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 ctstate NEW
   34  1556 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 ctstate NEW
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8053 ctstate NEW
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:10255 ctstate NEW
 2669  160K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8443 ctstate NEW
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:4789 ctstate NEW
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:10250 ctstate NEW
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:10255 ctstate NEW
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8444 ctstate NEW
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:2380 ctstate NEW
13862 1488K ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:8053 ctstate NEW

Trying to add additional rule in IN_public_allow with classical iptables will not work. Firwealld has different approach.

ie. to add CNS ( Container Native Storage ) ports ( which are by default not open and that will be like that as long as CNS is not part of default OpenShift ansible installer ) then we need to run

# firewall-cmd --direct --add-rule ipv4 filter IN_public_allow 1 -m tcp -p tcp -m conntrack --ctstate NEW --dport 24007 -j ACCEPT
# firewall-cmd --direct --add-rule ipv4 filter IN_public_allow 1 -m tcp -p tcp -m conntrack --ctstate NEW --dport 24008 -j ACCEPT
# firewall-cmd --direct --add-rule ipv4 filter IN_public_allow 1 -m tcp -p tcp -m conntrack --ctstate NEW --dport 2222 -j ACCEPT
# firewall-cmd --direct --add-rule ipv4 filter IN_public_allow 1 -m tcp -p tcp -m conntrack --ctstate NEW -m multiport --dports 49152:49664 -j ACCEPT

keyword is --direct as it name says, it will interact with firewalld rules direct-ly. More about this here and here

After adding rules, if not saved with

# firewall-cmd --runtime-to--permanent

next restart of firewalld.service will clean ip them, so necessary to save rules. These rules will be written in /etc/firewalld/direct.xml

#fedora-2, #firewalld, #iptables, #openshift, #redhat

OpenShift : Error from server: User “$user” cannot list all nodes/pods in the cluster

The below OpenShift error messages can be quite annoying, and they appear if current login is not for system:admin
Example error messages

# oc get pods 
No resources found.
Error from server: User "system:anonymous" cannot list pods in project "default"
root@dhcp8-176: ~ # oc get nodes 
No resources found.
Error from server: User "system:anonymous" cannot list all nodes in the cluster

Trying to login with user admin will not help

# oc login -u admin 
Authentication required for https://dhcp8-144.example.net:8443 (openshift)
Username: admin
Password: 
Login successful.
You don't have any projects. You can try to create a new project, by running
    oc new-project 

root@dhcp8-176: ~ # oc get pods
No resources found.
Error from server: User "admin" cannot list pods in project "default"
root@dhcp8-176: ~ # oc get nodes 
No resources found.
Error from server: User "admin" cannot list all nodes in the cluster

To get rid of it, login as system:admin

# oc login -u system:admin

what it does and what certificates reads in order to succeed is possible to see if last command run with --loglevel=10

# oc login -u system:admin --logleve=10

#kubernetes, #openshift

etcd error message “etcd failed to send out hearbeat on time”

… etcd distributed key value store that provides a reliable way to store data across a cluster of machines per 1 and 2. ETCD is very sensitive on delays in networks, and not only in networks but all kind of overlay sluggishness of etcd cluster nodes can lead to complete kubernets cluster functionality problems.

At time when OpenShift/Kubernetes cluster starts reporting error messages as showed below, cluster will already behave inappropriate and pods scheduling / deleting will not work as expected and problems will be more than visible

Sep 27 00:04:01 dhcp7-237 etcd: failed to send out heartbeat on time (deadline exceeded for 1.766957688s)
Sep 27 00:04:01 dhcp7-237 etcd: server is likely overloaded
Sep 27 00:04:01 dhcp7-237 etcd: failed to send out heartbeat on time (deadline exceeded for 1.766976918s)
Sep 27 00:04:01 dhcp7-237 etcd: server is likely overloaded

systemctl status etcd output

 systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2016-10-01 09:18:37 EDT; 5h 20min ago
 Main PID: 11970 (etcd)
   Memory: 1.0G
   CGroup: /system.slice/etcd.service
           └─11970 /usr/bin/etcd --name=dhcp6-138.example.net --data-dir=/var/lib/etcd/ --listen-client-urls=https://172.16.6.138:2379

Oct 01 14:38:55 dhcp6-138.example.net etcd[11970]: server is likely overloaded
Oct 01 14:38:56 dhcp6-138.example.net etcd[11970]: failed to send out heartbeat on time (deadline exceeded for 377.70994ms)
Oct 01 14:38:56 dhcp6-138.example.net etcd[11970]: server is likely overloaded
Oct 01 14:38:56 dhcp6-138.example.net etcd[11970]: failed to send out heartbeat on time (deadline exceeded for 377.933298ms)
Oct 01 14:38:56 dhcp6-138.example.net etcd[11970]: server is likely overloaded
Oct 01 14:38:58 dhcp6-138.example.net etcd[11970]: failed to send out heartbeat on time (deadline exceeded for 1.226630142s)
Oct 01 14:38:58 dhcp6-138.example.net etcd[11970]: server is likely overloaded
Oct 01 14:38:58 dhcp6-138.example.net etcd[11970]: failed to send out heartbeat on time (deadline exceeded for 1.226803192s)
Oct 01 14:38:58 dhcp6-138.example.net etcd[11970]: server is likely overloaded
Oct 01 14:39:07 dhcp6-138.example.net etcd[11970]: the clock difference against peer f801f8148b694198 is too high [1.078081179s > 1s]

# systemctl status etcd -l will also have similar messages,and check these too

ETCD configuration file is located in /etc/etcd/etcd.conf and has similar content as below, this one is from RHEL, other OSes can have it a bit changed

ETCD_NAME=dhcp7-237.example.net
ETCD_LISTEN_PEER_URLS=https://172.16.7.237:2380
ETCD_DATA_DIR=/var/lib/etcd/
ETCD_HEARTBEAT_INTERVAL=6000
ETCD_ELECTION_TIMEOUT=30000
ETCD_LISTEN_CLIENT_URLS=https://172.16.7.237:2379

ETCD_INITIAL_ADVERTISE_PEER_URLS=https://172.16.7.237:2380
ETCD_INITIAL_CLUSTER=dhcp7-241.example.net=https://172.16.7.241:2380,dhcp7-237.example.net=https://172.16.7.237:2380,dhcp7-239.example.net=https://172.16.7.239:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1
ETCD_ADVERTISE_CLIENT_URLS=https://172.16.7.237:2379


ETCD_CA_FILE=/etc/etcd/ca.crt
ETCD_CERT_FILE=/etc/etcd/server.crt
ETCD_KEY_FILE=/etc/etcd/server.key
ETCD_PEER_CA_FILE=/etc/etcd/ca.crt
ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt
ETCD_PEER_KEY_FILE=/etc/etcd/peer.key

bold parameters in above configuration files are ones we want to change ETCD_HEARTBEAT_INTERVAL and ETCD_ELECTION_TIMEOUT and there is not unified value for all, it is necessary to play with different values and find out what is best. For most cases default (500/2500) will be fine.

After changing /etc/etcd/etc.conf do not forget to restart etcd service

# systemctl restart etcd

Below issue affecting ETCD nodes can lead to problem described in this post

  • network latency
  • storage latency
  • combination of network latency and storage latency

if network latency is low, then check storage which is used by Kubernets/OpenShift ETCD servers. This is workaround for case when root cause is discovered and changes as stated in this post are performed in order to mitigate issue when no other option is possible. First and better solution would be to solve issue at its roots by fixing problematic subsystem(s).

In my particular case storage subsystem was slow and not possible to change that without bunch of $$$

References : etcd documentation

#etcd, #k8s, #kubernetes, #linux, #openshift, #redhat, #storage

Openshift v3 pod bulk creation with Amazon Elastic Block Store (EBS) – Block Storage as persistent storage

If you ever want to create many Openshift pods with one EBS volume attached to each pod then create_ebs_pod.py can help you with that. Help is showed below.
There are few steps which has to be taken in advance before running script

  • It is necessary to configure openshift nodes / masters to be able to talk with Amazon EC2. This blog post can give you some ideas what to do to achieve this aws-ebs-as-persistent-storage-for-openshift
  • boto3 has to be installed on machine where create_ebs_pod.py is supposed to run pip install boto3 will get it
  • Ensure to run
    # pip install awscli 
    # aws configure 
    
  • and follow steps to configure Amazon cli client. aws configure is going to ask questions regarding aws credentials which needs to be provided

    After all above is satisfied, running below

     ./create-ebs-class.py --volumesize=1 --image=fedora  --tagprefix=fedora_test --minpod=1 --maxpod=11 --pvfile=pv.json --pvcfile=pvc.json --podfile=pod.json 

    will create 10 pods based on image fedora, with each pod having mounted one EBS volume to /mnt/persistentvolume. Each EBS volume will have tag value of fedora_test. The EBS tag is not used for anything except to easier delete EBS volumes later on, for example using script from delete-ebs-volumes-based-on-ebs-volume-tag

    pv.json, pvc.json, pod.json are generic files which examples can be found here

    #python create_ebs_pod.py -h
    usage: create-ebs-class.py [-h] [--volumesize VOLUMESIZE] [--vtype VTYPE]
                               [--region REGION] [--image IMAGE]
                               [--tagprefix TAGPREFIX] [--mountpoint MOUNTPOINT]
                               [--minpod MINPOD] [--action ACTION] --maxpod MAXPOD
                               --pvfile PVFILE --pvcfile PVCFILE --podfile PODFILE
    
    Script to create OSE pods and attach one EBS volume per pod as persistent
    storage
    
    optional arguments:
      -h, --help            show this help message and exit
      --volumesize VOLUMESIZE
                            size of EBS voluems - in GB
      --vtype VTYPE         EBS volume type, default is gp2
      --region REGION       Amazon region where to connect
      --image IMAGE         docker image to use
      --tagprefix TAGPREFIX
                            tag prefix for EBS volumes, default tag is openshift-
                            testing-EBS_volume_id
      --mountpoint MOUNTPOINT
                            mount point inside pod where EBS volume will be
                            mounted, default is /mnt/persistentvolume
      --minpod MINPOD       minimum number of pods to create - default is 1 - so
                            minimum one pod will be created
      --action ACTION       what to do - either to create pods or delete pods
      --maxpod MAXPOD       maximum number of pods to create - required parameter
      --pvfile PVFILE       persistent volume definition json file - required
                            parameter
      --pvcfile PVCFILE     persistent volume claim definition json file -
                            required parameter
      --podfile PODFILE     pod definition json file - required parameter
    

    #amazon, #aws, #ebs, #linux, #openshift, #python

    AWS EBS as persistent storage for Openshift

    Openshift supports different persistent storage backends. Ceph, gluster, Amazon EBS, just to name some of them. In using ceph as persistent storage for openshift pods blog post I described how to hook openshift installation to use CEPH cluster as persistent storage for openshift pods. Here will be described process how configure Openshift to use EBS AWS based storage as persistent volumes for openshift pods

    It is assumed here that as starting point is working Openshift environment. In order to use Amazon EBS storage with openshift, it is necessary to do small changes in openshift configuration

  • Openshift master and all Openshift nodes need to be configured to support AWS EBS. Openshift documentation gives nice overview of necessary changes
  • After configuring openshift master/nodes to be able to reach / allocate EBS, next steps are straightforward.
    Examples of PersistentVolume , PersistentVolumeClaim, and Pod json file are showed below

  • PersistenVolume
  • PersistentVolumeClaim
  • pod file
  • In above .json files is necessary to adapt them with proper pv name, size, pvc name, size, pod name and etc to correspond environment where used. After adapting them, create PersistentVolume and PersistentVolumeClaim

    # oc create -f pv.json 
    # oc create -f pvc.json 
    
     # oc get pv
    NAME       LABELS    CAPACITY     ACCESSMODES     STATUS    CLAIM               REASON      AGE
    pvolume1               1          RWO             Bound     default/pvclaim1                3h
    # oc get pvc 
    NAME       LABELS     STATUS    VOLUME      CAPACITY      ACCESSMODES     AGE
    pvclaim1              Bound     pvolume1    1             RWO             3h
    

    Now we can create pod which will use above created PersistentVolume and PersistentVolumeClaim

    # oc create -f pod.json
    # oc get pods | grep pod1
    pod1                      1/1       Running   0          30s
    

    Now, on node where pod runs, we will see

    # fdisk -l | grep xvd
    WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.
    Disk /dev/xvda: 10.7 GB, 10737418240 bytes, 20971520 sectors
    Disk /dev/xvdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
    /dev/xvdb1            2048    20971519    10484736   8e  Linux LVM
    Disk /dev/xvdf: 1073 MB, 1073741824 bytes, 2097152 sectors ---- this is EBS device used inside pod 
    

    /dev/xvdf is device mapped to node, I know this as it is only 1GB device on my system. Further, device is visible in mount output

    # mount | grep xvdf
    /dev/xvdf on /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/vol-3f7210c9 type ext4 (rw,relatime,seclabel,data=ordered)
    /dev/xvdf on /var/lib/origin/openshift.local.volumes/pods/3a467bfd-eeb7-11e5-9e13-02982d3b6367/volumes/kubernetes.io~aws-ebs/pvolume1 type ext4 (rw,relatime,seclabel,data=ordered)
    

    it is mounted at mount point under /var/lib/origin/openshift.local.volumes …

    If we take closer look, we will see there, EBS volume id ( vol-3f7210c ), then plugin used ( aws-ebs ) – this can be useful if it is necessary to do sorting / searching devices allocated on same Openshift node and commit from two different storage backennds ( eg, EBS and CEPH )
    In pod we see

     # oc exec  pod1 -- mount  | grep xvdf
    /dev/xvdf on /mnt/persistentvolume type ext4 (rw,relatime,seclabel,data=ordered)
    

    For every pod using EBS as persistant storage pair of lines similar to ones above will be visible on Openshift node

    #amazon-ebs, #linux, #openshift, #persistantvolume, #persistantvolumeclaim, #pod, #storage

    ceph rbd block device as persistent storage for openshift

    After Fedora 23 – ceph installation and Fedora 23 – Openshift installation setup it is now time to hook openshift environment to use CEPH storage backend

    Openshift pods will be using CEPH rados block devices as persistent storage and to achieve this one option is to follow below steps

  • create ceph pool and desired number of images on top of it, this can be done manually or using ceph-pool-setup.sh script. If ceph-pool-setup.sh is used,read README before running it.
  • create ceph-secret file. As an example is possible to use ceph-secret
  • define persistent volume and persistent volume claim. Yaml example files ceph-pv and ceph-pv-claim
  • create pod file adapting it to use ceph pv and ceph pv claims pod-pv-pvc-ceph
  • In above examples is necessary to change variables to suit different environments ( ceph pool name, ceph monitor(s) ip addresses … )

    Once all is in place, then running at below on Ceph cluster and after Openshift master will create pod which will in return start using rbd as persistent

    Ceph side

    # ./ceph-pool-setup -a c -p mypool -i 1 -r 3 -s 1
    

    This will create three way replicated ceph poll with name mypool, with one image on top of it with size of 1 GB

    Openshift side

    # oc create -f ceph-secrets.yaml
    # oc create -f ceph-pv.yaml 
    # oc create -f ceph-pv-claim.yaml
    # oc create -f pod-pv-pvc-ceph.json
    

    If all is fine pod should start and mount rbd inside pod with ext4 file system preformanted

    # oc rsh pod 
    # mount | grep rbd 
    /dev/rbd0 on /mnt/ceph type ext4 (rw,relatime,seclabel,stripe=1024,data=ordered)
    

    This setup will enable openshift pods to use ceph rbd device as persistent storage, and in case pod is removed, and started at some other openshift node it will get same data in case it has access to rbd device which was used before pod was deleted. As name said, this is persistent volume and it should persist across pod re-creation.

    #ceph, #ceph-rbd, #ceph-storage, #kubenetes, #openshift

    install openshift origin / OS Fedora 23

    Installing Openshift origin on Fedora 23 is showed below,overall not an difficult task to get test environment for openshift origin which then can be used for testing
    ( really only for testing – as this is going to be master / node on one machine and in kvm guest )

    Following below steps will lead to test openshift origin environment.

    Openshift origin publish bits are github openshift releases bellow is what I did to get it working under 10 mins.

    # dnf install -y docker; systemctl enable docker; systemctl start docker 
    # mkdir /root/openshift
    # cd /root/openshift
    # wget https://github.com/openshift/origin/releases/download/v1.1.1/openshift-origin-server-v1.1.1-e1d9873-linux-64bit.tar.gz
    # tar -xaf openshift-origin-server-v1.1.1-e1d9873-linux-64bit.tar.gz
    # cd openshift-origin-server-v1.1.1-e1d9873-linux-64bit
    # ./openshift start &
    

    After this, beside files delivered after unpacking source archive, there will be created in openshift directory openshift configuration files

    # ls -l 
    drwxr-xr-x. 4 root root        46 Jan 19 18:44 openshift.local.config
    drwx------. 3 root root        20 Jan 19 20:03 openshift.local.etcd
    drwxr-x---. 4 root root        33 Jan 19 18:44 openshift.local.volumes
    

    From here, it is necessary to export paths to keys and certificated

    # export KUBECONFIG="$(pwd)"/openshift.local.config/master/admin.kubeconfig
    $ export CURL_CA_BUNDLE="$(pwd)"/openshift.local.config/master/ca.crt
    $ sudo chmod +r "$(pwd)"/openshift.local.config/master/admin.kubeconfig
    

    That is! Follow rc-local Fedora 23 to make it to start on boot, or write systemd files using as starting points openshift-master-service and openshift-node-service – what should work with small tweaks

    #openshift, #openshift-origin