본문 바로가기
CLOUD/Openshift

OCP 4.10.23 설치(openshift) - 3. HA proxy, image mirror(cfssl)

1000maru 2022. 8. 25. 14:01
반응형

HAproxy

•LB 용도로서 HAproxy 사용
•Master와 bootstrap은 6443, 22623 port 등록
•Worker와 infra는 80, 443 port 등록

 

[bastion]

 

HAproxy 설치

yum -y install haproxy

 

config 

vi /etc/haproxy/haproxy.cfg

#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   https://www.haproxy.org/download/1.8/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

    # utilize system-wide crypto-policies
#    ssl-default-bind-ciphers PROFILE=SYSTEM
#    ssl-default-server-ciphers PROFILE=SYSTEM

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 4000


#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend static
    balance     roundrobin
    server      static 127.0.0.1:4331 check

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
frontend openshift-api-server
        bind *:6443
        default_backend openshift-api-server
        mode tcp
        option tcplog

backend openshift-api-server
        balance source
        mode tcp
        server bootstrap 172.16.2.190:6443 check
        server master01 172.16.2.193:6443 check
        server master02 172.16.2.194:6443 check
        server master03 172.16.2.195:6443 check

frontend machine-config-server
        bind *:22623
        default_backend machine-config-server
        mode tcp
        option tcplog

backend machine-config-server
        balance source
        mode tcp
        server bootstrap 172.16.2.190:22623 check
        server master01 172.16.2.193:22623 check
        server master02 172.16.2.194:22623 check
        server master03 172.16.2.195:22623 check

frontend ingress-http
        bind *:80
        default_backend ingress-http
        mode tcp
        option tcplog

backend ingress-http
        balance source
        mode tcp
        server infra01 172.16.2.198:80 check
        server infra02 172.16.2.199:80 check
        server worker01 172.16.2.196:80 check
        server worker02 172.16.2.197:80 check

frontend ingress-https
        bind *:443
        default_backend ingress-https
        mode tcp
        option tcplog

backend ingress-https
        balance source
        mode tcp
        server infra01 172.16.2.198:443 check
        server infra02 172.16.2.199:443 check
        server worker01 172.16.2.196:443 check
        server worker02 172.16.2.197:443 check

서비스 시작

systemctl enable haproxy.service --now
systemctl status haproxy.service

 

도구 Download

[registry]

 

[bastion]

 

tool install

tar -xvf openshift-client-linux-4.10.23.tar.gz -C /usr/local/bin/
tar -xvf openshift-install-linux-4.10.23.tar.gz -C /usr/local/bin/

 

Image mirror 구성

•Disconnected 환경이라 registry에서 mirror 구성 후 bastion에 load 합니다.
•Login 정보는 htpasswd 사용합니다.
•인증서는 cfssl로 생성합니다.
•Pull-secre에 bastio과 registry 인증 정보를 추가합니다.

 

[registry, bastion]

 

폴더 생성 및 login 정보 생성

mkdir -p /opt/registry/{auth,data,certs}
htpasswd -bBc /opt/registry/auth/htpasswd admin new1234!
echo -n 'admin:new1234!' | base64 -w0

 

[registry]

 

bastion이 disconnect 상태이기 때문에 registry 서버에서 넘겨준다.

podman pull docker.io/library/registry:2
podman save -o registry.tar docker.io/library/registry:2
scp registry.tar bastion:/root/

 

[bastion]

 

image load

podman load -i registry.tar
podman images

 

[registry]

 

인증서 생성

wget --quiet https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64 -O /usr/local/bin/cfssljson
wget --quiet https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64 -O /usr/local/bin/cfssl
chmod 755 /usr/local/bin/cfssl /usr/local/bin/cfssljson
cfssl version ; cfssljson --version

cd /opt/registry/certs/
cat << EOF > ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "server": {
        "expiry": "87600h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth"
        ]
      },
      "client": {
        "expiry": "87600h",
        "usages": [
          "signing",
          "key encipherment",
          "client auth"
        ]
      }
    }
  }
}
EOF

cat << EOF > ca-csr.json
{
  "CN": "Red Hat GPS",
  "hosts": [
    "registry.maru.ocp4.com"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "KR",
      "ST": "Seoul",
      "L": "Gangnam",
      "OU": "GPS"
    }
  ]
}
EOF

cat << EOF > server.json
{
  "CN": "Red Hat GPS",
  "hosts": [
    "registry.maru.ocp4.com"
  ],
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
  "names": [
    {
      "C": "KR",
      "ST": "Seoul",
      "L": "Gangnam",
      "OU": "GPS"
    }
  ]
}
EOF


cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server.json | cfssljson -bare server

cp /opt/registry/certs/ca.pem /etc/pki/ca-trust/source/anchors/registry.pem
update-ca-trust extract

 

[bastion]

 

인증서 생성

wget http://registry.maru.ocp4.com:58080/repo/cfssl_1.6.1_linux_amd64 -O /usr/local/bin/cfssl
wget http://registry.maru.ocp4.com:58080/repo/cfssljson_1.6.1_linux_amd64 -O /usr/local/bin/cfssljson
chmod 755 /usr/local/bin/cfssl /usr/local/bin/cfssljson
cfssl version ; cfssljson --version


cd /opt/registry/certs/
cat << EOF > ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "server": {
        "expiry": "87600h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth"
        ]
      },
      "client": {
        "expiry": "87600h",
        "usages": [
          "signing",
          "key encipherment",
          "client auth"
        ]
      }
    }
  }
}
EOF

cat << EOF > ca-csr.json
{
  "CN": "Red Hat GPS",
  "hosts": [
    "bastion.maru.ocp4.com"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "KR",
      "ST": "Seoul",
      "L": "Gangnam",
      "OU": "GPS"
    }
  ]
}
EOF

cat << EOF > server.json
{
  "CN": "Red Hat GPS",
  "hosts": [
    "bastion.maru.ocp4.com"
  ],
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
  "names": [
    {
      "C": "KR",
      "ST": "Seoul",
      "L": "Gangnam",
      "OU": "GPS"
    }
  ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server.json | cfssljson -bare server

cp /opt/registry/certs/ca.pem /etc/pki/ca-trust/source/anchors/bastion.pem
update-ca-trust extract

 

 

각 서버에서 서로의 인증서 복사

 

[registry]

 

scp /etc/pki/ca-trust/source/anchors/registry.pem bastion:/etc/pki/ca-trust/source/anchors/

 

[bastion]

 

scp /etc/pki/ca-trust/source/anchors/bastion.pem registry:/etc/pki/ca-trust/source/anchors/

 

[registry, bastion]

 

registry pod 올리기 및 체크

update-ca-trust extract

cd /opt/registry/

cat << EOF > registry.sh
podman run --name mirror-registry -p 5000:5000 \
-v /opt/registry/data:/var/lib/registry:z \
-v /opt/registry/auth:/auth:z \
-e "REGISTRY_AUTH=htpasswd" \
-e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
-e "REGISTRY_HTTP_SECRET=ALongRandomSecretForRegistry" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
-v /opt/registry/certs:/certs:z \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/server.pem \
-e REGISTRY_HTTP_TLS_KEY=/certs/server-key.pem \
-d docker.io/library/registry:2
EOF

sh registry.sh 
podman ps
curl -u admin:new1234! -k https://registry.maru.ocp4.com:5000/v2/_catalog
curl -u admin:new1234! -k https://bastion.maru.ocp4.com:5000/v2/_catalog

 

[registry]

확인

 

[bastion]

확인

 

 

 

[registry]

 

pull secret 생성

 

Redhat console에서 받은 pull-secret에 인증정보 추가

Redhat consol에서 받은 pull-secret에 인증정보 추가

cat << EOF > pull-secret
{"auths":{"cloud.openshift.com":{"auth":"..."email":"maru@mail.co.kr"},"bastion.maru.ocp4.com:5000":{"auth":"YWRtaW46bmV3MTIzNCE="},"registry.maru.ocp4.com:5000":{"auth":"YWRtaW46bmV3MTIzNCE="}}}
EOF

 

 

mirror 구성 스크립트

vi mirror.sh
OCP_RELEASE="4.4.5-x86_64"
LOCAL_REGISTRY='registry.maru.ocp4.com:5000'
LOCAL_REPOSITORY='ocp4/openshift4'
PRODUCT_REPO='openshift-release-dev'
RELEASE_NAME="ocp-release"
LOCAL_SECRET_JSON=/opt/registry/pull-secret
oc adm -a ${LOCAL_SECRET_JSON} release mirror \
--from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE} \
--to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
--to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}

 

mirror 구성 

sh -x /opt/registry/mirror.sh

 

확인

curl -u admin:new1234! -k https://bastion.maru.ocp4.com:5000/v2/_catalog
curl -u admin:new1234! -k https://registry.maru.ocp4.com:5000/v2/_catalog

ocp4/openshift4 보이면 성공

 

login 확인

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

반응형

'CLOUD > Openshift' 카테고리의 다른 글

OCP 4.10.23 설치(openshift) - 6. pv  (0) 2022.08.28
OCP 4.10.23 설치(openshift) - 5. infra node  (0) 2022.08.27
OCP 4.10.23 설치(openshift) - 4. Install-config, bootstrap, master, worker  (0) 2022.08.26
OCP 4.10.23 설치(openshift) - 2. DNS  (2) 2022.08.25
OCP 4.10.23 설치(openshift) - 1. private registry, timezone, chrony  (0) 2022.08.24

'CLOUD/Openshift' Related Articles

티스토리툴바