k8s实战案例之部署Nginx+Tomcat+NFS实现动静分离
2023-06-05 06:27:59
来源：博客园
1、基于镜像分层构建及自定义镜像运行Nginx及Java服务并基于NFS实现动静分离1.1、业务镜像设计规划
1.2、Nginx+Tomcat+NFS实现动静分离架构图根据业务的不同，我们可以导入官方基础镜像，在官方基础镜像的基础上自定义需要用的工具和环境，然后构建成自定义出自定义基础镜像，后续再基于自定义基础镜像，来构建不同服务的基础镜像，最后基于服务的自定义基础镜像构建出对应业务镜像；最后将这些镜像上传至本地harbor仓库，然后通过k8s配置清单，将对应业务运行至k8s集群之上；
2、自定义centos基础镜像构建客户端通过负载均衡器的反向代理来访问k8s上的服务， nginx pod和tomcat pod 由k8s svc 资源进行关联；所有数据（静态资源和动态资源）通过存储挂载至对应pod中；nginx作为服务入口，它负责接收客户端的请求，同时响应静态资源（到存储上读取，比如js文件，css文件，图片等）；后端动态资源，由nginx将请求转发至后端tomcat server 完成（tomcat负责数据写入，比如用户的上传的图片等等）；
root@k8s-master01:~/k8s-data/dockerfile/system/centos# lsCentOS7-aliyun-Base.repo CentOS7-aliyun-epel.repo Dockerfile build-command.sh filebeat-7.12.1-x86_64.rpmroot@k8s-master01:~/k8s-data/dockerfile/system/centos# cat Dockerfile #自定义Centos 基础镜像FROM centos:7.9.2009 ADD filebeat-7.12.1-x86_64.rpm /tmp# 添加阿里源ADD CentOS7-aliyun-Base.repo CentOS7-aliyun-epel.repo /etc/yum.repos.d/ # 自定义安装工具和环境RUN yum makecache &&yum install -y /tmp/filebeat-7.12.1-x86_64.rpm vim wget tree lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop && rm -rf /etc/localtime /tmp/filebeat-7.12.1-x86_64.rpm && ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && useradd nginx -u 2088root@k8s-master01:~/k8s-data/dockerfile/system/centos# cat build-command.sh #!/bin/bash#docker build -t harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009 .#docker push harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009/usr/local/bin/nerdctl build -t harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009 ./usr/local/bin/nerdctl push harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009root@k8s-master01:~/k8s-data/dockerfile/system/centos# 在harbor上验证镜像是否正常上传？运行镜像为容器，验证对应镜像是否有我们添加的工具和环境？
(相关资料图)
root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/nginx-base# lsDockerfile build-command.sh nginx-1.22.0.tar.gzroot@k8s-master01:~/k8s-data/dockerfile/web/pub-images/nginx-base# cat Dockerfile #Nginx Base Image# 导入自定义centos基础镜像FROM harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009 # 安装编译环境RUN yum install -y vim wget tree lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop# 添加nginx源码至/usr/local/src/ADD nginx-1.22.0.tar.gz /usr/local/src/# 编译nginxRUN cd /usr/local/src/nginx-1.22.0 && ./configure && make && make install && ln -sv /usr/local/nginx/sbin/nginx /usr/sbin/nginx &&rm -rf /usr/local/src/nginx-1.22.0.tar.gz root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/nginx-base# cat build-command.sh #!/bin/bash#docker build -t harbor.ik8s.cc/pub-images/nginx-base:v1.18.0 .#docker push harbor.ik8s.cc/pub-images/nginx-base:v1.18.0nerdctl build -t harbor.ik8s.cc/pub-images/nginx-base:v1.22.0 .nerdctl push harbor.ik8s.cc/pub-images/nginx-base:v1.22.0root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/nginx-base# 验证nginx基础镜像是否上传至harbor？把nginx基础镜像运行为容器，看看nginx是否正常安装？
3.3、构建自定义nginx业务镜像能够将nginx基础镜像运行为容器，并在容器内部启动nginx，表示nginx基础镜像就构建好了；
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/nginx# lsDockerfile app1.tar.gz build-command.sh index.html nginx.conf webapproot@k8s-master01:~/k8s-data/dockerfile/web/magedu/nginx# cat Dockerfile #Nginx 1.22.0# 导入nginx基础镜像FROM harbor.ik8s.cc/pub-images/nginx-base:v1.22.0 # 添加nginx配置文件ADD nginx.conf /usr/local/nginx/conf/nginx.conf# 添加业务代码ADD app1.tar.gz /usr/local/nginx/html/webapp/ADD index.html /usr/local/nginx/html/index.html# 创建静态资源挂载路径RUN mkdir -p /usr/local/nginx/html/webapp/static /usr/local/nginx/html/webapp/images # 暴露端口EXPOSE 80 443# 运行nginxCMD ["nginx"] root@k8s-master01:~/k8s-data/dockerfile/web/magedu/nginx# cat nginx.conf user nginx nginx;worker_processes auto;daemon off;events { worker_connections 1024;}http { include mime.types; default_type application/octet-stream; #log_format main "$remote_addr - $remote_user [$time_local] "$request" " # "$status $body_bytes_sent "$http_referer" " # ""$http_user_agent" "$http_x_forwarded_for""; #access_log logs/access.log main; sendfile on; #tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 65; upstream tomcat_webserver { server magedu-tomcat-app1-service.magedu:80; } server { listen 80; server_name localhost; location / { root html; index index.html index.htm; } location /webapp { root html; index index.html index.htm; } location /app1 { proxy_pass http://tomcat_webserver; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; } error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } }}root@k8s-master01:~/k8s-data/dockerfile/web/magedu/nginx# cat build-command.sh #!/bin/bashTAG=$1#docker build -t harbor.ik8s.cc/magedu/nginx-web1:${TAG} .#echo "镜像构建完成，即将上传到harbor"#sleep 1#docker push harbor.ik8s.cc/magedu/nginx-web1:${TAG}#echo "镜像上传到harbor完成"nerdctl build -t harbor.ik8s.cc/magedu/nginx-web1:${TAG} .nerdctl push harbor.ik8s.cc/magedu/nginx-web1:${TAG}root@k8s-master01:~/k8s-data/dockerfile/web/magedu/nginx# 3.4、验证自定义nginx业务镜像上述Dockerfile中主要基于nginx基础镜像添加业务代码，添加配置，以及定义运行nginx和暴露服务端口；
验证nginx业务镜像是否上传至harbor？运行nginx业务镜像为容器，看看对应业务是否能够正常访问？
4、基于自定义centos基础镜像构建tomcat镜像4.1、基于自定义centos基础镜像构建jdk基础镜像这里提示找不到magedu-tomcat-app1-service.magedu:80这个upstream ，这是因为我们在配置文件中写死了nginx调用后端tomcat在k8s中 svc的地址；所以该镜像只能运行在k8s环境中，并且在运行该镜像前对应k8s环境中，tomcat的svc必须存在；
root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/jdk-1.8.212# lltotal 190464drwxr-xr-x 2 root root 4096 Jun 4 04:04 ./drwxr-xr-x 6 root root 4096 Aug 9 2022 ../-rw-r--r-- 1 root root 389 Jun 4 04:04 Dockerfile-rw-r--r-- 1 root root 259 Jun 4 04:02 build-command.sh-rw-r--r-- 1 root root 195013152 Jun 22 2021 jdk-8u212-linux-x64.tar.gz-rw-r--r-- 1 root root 2105 Jun 22 2021 profileroot@k8s-master01:~/k8s-data/dockerfile/web/pub-images/jdk-1.8.212# cat Dockerfile #JDK Base Image# 导入自定义centos基础镜像FROM harbor.ik8s.cc/baseimages/magedu-centos-base:7.9.2009 # 安装jdk环境ADD jdk-8u212-linux-x64.tar.gz /usr/local/src/RUN ln -sv /usr/local/src/jdk1.8.0_212 /usr/local/jdk ADD profile /etc/profileENV JAVA_HOME /usr/local/jdkENV JRE_HOME $JAVA_HOME/jreENV CLASSPATH $JAVA_HOME/lib/:$JRE_HOME/lib/ENV PATH $PATH:$JAVA_HOME/binroot@k8s-master01:~/k8s-data/dockerfile/web/pub-images/jdk-1.8.212# cat build-command.sh #!/bin/bash#docker build -t harbor.ik8s.cc/pub-images/jdk-base:v8.212 .#sleep 1#docker push harbor.ik8s.cc/pub-images/jdk-base:v8.212nerdctl build -t harbor.ik8s.cc/pub-images/jdk-base:v8.212 .nerdctl push harbor.ik8s.cc/pub-images/jdk-base:v8.212root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/jdk-1.8.212# 验证jdk基础镜像是否上传至harbor?运行jdk基础镜像为容器，看看jdk环境是否安装？
4.2、基于自定义jdk镜像构建tomcat基础镜像能够正常在容器内部执行java 命令表示jdk镜像构建没有问题；
root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43# lltotal 9508drwxr-xr-x 2 root root 4096 Jun 4 04:23 ./drwxr-xr-x 6 root root 4096 Aug 9 2022 ../-rw-r--r-- 1 root root 390 Jun 4 04:22 Dockerfile-rw-r--r-- 1 root root 9717059 Jun 22 2021 apache-tomcat-8.5.43.tar.gz-rw-r--r-- 1 root root 275 Jun 4 04:23 build-command.shroot@k8s-master01:~/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43# cat Dockerfile #Tomcat 8.5.43基础镜像# 导入自定义jdk镜像FROM harbor.ik8s.cc/pub-images/jdk-base:v8.212# 创建tomcat安装目录、数据目录和日志目录RUN mkdir /apps /data/tomcat/webapps /data/tomcat/logs -pv # 安装tomcatADD apache-tomcat-8.5.43.tar.gz /appsRUN useradd tomcat -u 2050 && ln -sv /apps/apache-tomcat-8.5.43 /apps/tomcat && chown -R tomcat.tomcat /apps /data -Rroot@k8s-master01:~/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43# cat build-command.sh #!/bin/bash#docker build -t harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43 .#sleep 3#docker push harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43nerdctl build -t harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43 .nerdctl push harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43root@k8s-master01:~/k8s-data/dockerfile/web/pub-images/tomcat-base-8.5.43# 验证自定义tomcat基础镜像是否上传至harbor？将自定义tomcat镜像运行为容器，看看tomcat是否可正常访问呢？
root@k8s-node01:~# nerdctl run -it --rm -p 8080:8080 harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43 /bin/bashWARN[0000] skipping verifying HTTPS certs for "harbor.ik8s.cc" harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43: resolved |++++++++++++++++++++++++++++++++++++++| manifest-sha256:ba362a6c099d965ffae43aae04e3ecb15e86e39d9d88259f4be72378b75bd1e5: done |++++++++++++++++++++++++++++++++++++++| config-sha256:1d5e5f5e1dcb8ce1c7f7d6ce9fcc967275309c14baaaddbf3aa86de26054d1d5: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:fa5fdb4dc02a5e79b212be196324b9936efbc850390f86283498a0e01b344ec3: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:00bab63d153828acf58242f1781bd40769cc8b69659f37a2a49918ff3bfca68c: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:6ddb864b9c4e53f3079ec4839ea3bace75b0d5d3daeae0ae910c171646bc4f96: done |++++++++++++++++++++++++++++++++++++++| elapsed: 2.4 s total: 18.6 M (7.7 MiB/s) [root@f5752bba588f /]# ll /apps/total 4drwxr-xr-x 1 tomcat tomcat 4096 Jun 4 12:25 apache-tomcat-8.5.43lrwxrwxrwx 1 tomcat tomcat 26 Jun 4 12:25 tomcat -> /apps/apache-tomcat-8.5.43[root@f5752bba588f /]# cd /apps/tomcat/bin/[root@f5752bba588f bin]# lsbootstrap.jar catalina.sh commons-daemon-native.tar.gz configtest.sh digest.sh shutdown.bat startup.sh tool-wrapper.bat version.shcatalina-tasks.xml ciphers.bat commons-daemon.jar daemon.sh setclasspath.bat shutdown.sh tomcat-juli.jar tool-wrapper.shcatalina.bat ciphers.sh configtest.bat digest.bat setclasspath.sh startup.bat tomcat-native.tar.gz version.bat[root@f5752bba588f bin]# ./catalina.sh runUsing CATALINA_BASE: /apps/tomcatUsing CATALINA_HOME: /apps/tomcatUsing CATALINA_TMPDIR: /apps/tomcat/tempUsing JRE_HOME: /usr/local/jdk/jreUsing CLASSPATH: /apps/tomcat/bin/bootstrap.jar:/apps/tomcat/bin/tomcat-juli.jar04-Jun-2023 12:40:29.845 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.5.4304-Jun-2023 12:40:29.851 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Jul 4 2019 20:53:15 UTC04-Jun-2023 12:40:29.851 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number: 8.5.43.004-Jun-2023 12:40:29.851 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux04-Jun-2023 12:40:29.852 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 5.15.0-72-generic04-Jun-2023 12:40:29.852 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd6404-Jun-2023 12:40:29.852 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/local/src/jdk1.8.0_212/jre04-Jun-2023 12:40:29.852 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_212-b1004-Jun-2023 12:40:29.853 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation04-Jun-2023 12:40:29.853 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /apps/apache-tomcat-8.5.4304-Jun-2023 12:40:29.853 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /apps/apache-tomcat-8.5.4304-Jun-2023 12:40:29.854 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/apps/tomcat/conf/logging.properties04-Jun-2023 12:40:29.854 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager04-Jun-2023 12:40:29.855 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=204804-Jun-2023 12:40:29.855 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources04-Jun-2023 12:40:29.855 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=002704-Jun-2023 12:40:29.856 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=04-Jun-2023 12:40:29.856 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/apps/tomcat04-Jun-2023 12:40:29.856 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/apps/tomcat04-Jun-2023 12:40:29.857 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/apps/tomcat/temp04-Jun-2023 12:40:29.857 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]04-Jun-2023 12:40:30.166 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]04-Jun-2023 12:40:30.193 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read04-Jun-2023 12:40:30.232 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-8009"]04-Jun-2023 12:40:30.236 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read04-Jun-2023 12:40:30.237 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 1262 ms04-Jun-2023 12:40:30.303 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]04-Jun-2023 12:40:30.303 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.4304-Jun-2023 12:40:30.324 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/apps/apache-tomcat-8.5.43/webapps/docs]04-Jun-2023 12:40:30.918 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/apps/apache-tomcat-8.5.43/webapps/docs] has finished in [593] ms04-Jun-2023 12:40:30.919 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/apps/apache-tomcat-8.5.43/webapps/examples]04-Jun-2023 12:40:31.524 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/apps/apache-tomcat-8.5.43/webapps/examples] has finished in [605] ms04-Jun-2023 12:40:31.524 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/apps/apache-tomcat-8.5.43/webapps/ROOT]04-Jun-2023 12:40:31.540 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/apps/apache-tomcat-8.5.43/webapps/ROOT] has finished in [16] ms04-Jun-2023 12:40:31.540 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/apps/apache-tomcat-8.5.43/webapps/manager]04-Jun-2023 12:40:31.565 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/apps/apache-tomcat-8.5.43/webapps/manager] has finished in [25] ms04-Jun-2023 12:40:31.565 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/apps/apache-tomcat-8.5.43/webapps/host-manager]04-Jun-2023 12:40:31.597 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/apps/apache-tomcat-8.5.43/webapps/host-manager] has finished in [32] ms04-Jun-2023 12:40:31.600 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]04-Jun-2023 12:40:31.608 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]04-Jun-2023 12:40:31.611 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 1373 ms访问tomcat
4.3、基于自定义tomcat镜像构建tomcat业务镜像能够正常访问tomcat,说明tomcat基础镜像就构建的没有问题；
root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1# lltotal 23588drwxr-xr-x 3 root root 4096 Jun 4 04:58 ./drwxr-xr-x 11 root root 4096 Aug 9 2022 ../-rw-r--r-- 1 root root 603 Jun 4 04:57 Dockerfile-rw-r--r-- 1 root root 144 May 11 06:26 app1.tar.gz-rwxr-xr-x 1 root root 261 Jun 4 04:58 build-command.sh*-rwxr-xr-x 1 root root 23611 Jun 22 2021 catalina.sh*-rw-r--r-- 1 root root 24086235 Jun 22 2021 filebeat-7.5.1-x86_64.rpm-rw-r--r-- 1 root root 667 Oct 24 2021 filebeat.ymldrwxr-xr-x 2 root root 4096 May 11 06:26 myapp/-rwxr-xr-x 1 root root 372 Jan 22 2022 run_tomcat.sh*-rw-r--r-- 1 root root 6462 Oct 10 2021 server.xmlroot@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1# cat Dockerfile #tomcat web1# 导入自定义tomcat镜像FROM harbor.ik8s.cc/pub-images/tomcat-base:v8.5.43 # 添加启动脚本和配置文件ADD catalina.sh /apps/tomcat/bin/catalina.shADD server.xml /apps/tomcat/conf/server.xmlADD run_tomcat.sh /apps/tomcat/bin/run_tomcat.sh# 添加业务代码ADD app1.tar.gz /data/tomcat/webapps/app1/#ADD filebeat.yml /etc/filebeat/filebeat.yml RUN chown -R nginx.nginx /data/ /apps/#ADD filebeat-7.5.1-x86_64.rpm /tmp/#RUN cd /tmp && yum localinstall -y filebeat-7.5.1-amd64.deb# 暴露端口EXPOSE 8080 8443# 启动tomcatCMD ["/apps/tomcat/bin/run_tomcat.sh"]root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1# cat build-command.sh #!/bin/bashTAG=$1#docker build -t harbor.ik8s.cc/magedu/tomcat-app1:${TAG} .#sleep 3#docker push harbor.ik8s.cc/magedu/tomcat-app1:${TAG}nerdctl build -t harbor.ik8s.cc/magedu/tomcat-app1:${TAG} .nerdctl push harbor.ik8s.cc/magedu/tomcat-app1:${TAG}root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1# root@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1# cat run_tomcat.sh #!/bin/bash#echo "nameserver 223.6.6.6" > /etc/resolv.conf#echo "192.168.7.248 k8s-vip.example.com" >> /etc/hosts#/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat &su - nginx -c "/apps/tomcat/bin/catalina.sh start"tail -f /etc/hostsroot@k8s-master01:~/k8s-data/dockerfile/web/magedu/tomcat-app1#验证自定义tomcat业务镜像是否上传至harbor？
将自定义tomcat业务镜像运行为容器，看看对应业务是否正常访问？访问业务
5、在k8s环境中运行tomcat能够正常访问app1说明业务容器的镜像构建没有问题；
root@k8s-master01:~/k8s-data/yaml/namespaces# lsmagedu-ns.yamlroot@k8s-master01:~/k8s-data/yaml/namespaces# cat magedu-ns.yaml apiVersion: v1kind: Namespacemetadata: name: mageduroot@k8s-master01:~/k8s-data/yaml/namespaces# cd ../magedu/tomcat-app1/root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# lltotal 16drwxr-xr-x 2 root root 4096 Jun 4 06:16 ./drwxr-xr-x 12 root root 4096 Aug 9 2022 ../-rw-r--r-- 1 root root 596 Jun 22 2021 hpa.yaml-rw-r--r-- 1 root root 1849 Jun 4 05:18 tomcat-app1.yamlroot@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# cat hpa.yaml #apiVersion: autoscaling/v2beta1apiVersion: autoscaling/v1 kind: HorizontalPodAutoscalermetadata: namespace: magedu name: magedu-tomcat-app1-podautoscaler labels: app: magedu-tomcat-app1 version: v2beta1spec: scaleTargetRef: apiVersion: apps/v1 #apiVersion: extensions/v1beta1 kind: Deployment name: magedu-tomcat-app1-deployment minReplicas: 2 maxReplicas: 20 targetCPUUtilizationPercentage: 60 #metrics: #- type: Resource # resource: # name: cpu # targetAverageUtilization: 60 #- type: Resource # resource: # name: memoryroot@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# cat tomcat-app1.yaml kind: Deployment#apiVersion: extensions/v1beta1apiVersion: apps/v1metadata: labels: app: magedu-tomcat-app1-deployment-label name: magedu-tomcat-app1-deployment namespace: mageduspec: replicas: 1 selector: matchLabels: app: magedu-tomcat-app1-selector template: metadata: labels: app: magedu-tomcat-app1-selector spec: containers: - name: magedu-tomcat-app1-container image: harbor.ik8s.cc/magedu/tomcat-app1:v1 #command: ["/apps/tomcat/bin/run_tomcat.sh"] imagePullPolicy: IfNotPresent #imagePullPolicy: Always ports: - containerPort: 8080 protocol: TCP name: http env: - name: "password" value: "123456" - name: "age" value: "18" #resources: # limits: # cpu: 1 # memory: "512Mi" # requests: # cpu: 500m # memory: "512Mi" volumeMounts: - name: magedu-images mountPath: /usr/local/nginx/html/webapp/images readOnly: false - name: magedu-static mountPath: /usr/local/nginx/html/webapp/static readOnly: false volumes: - name: magedu-images nfs: server: 192.168.0.42 path: /data/k8sdata/magedu/images - name: magedu-static nfs: server: 192.168.0.42 path: /data/k8sdata/magedu/static# nodeSelector:# project: magedu# app: tomcat---kind: ServiceapiVersion: v1metadata: labels: app: magedu-tomcat-app1-service-label name: magedu-tomcat-app1-service namespace: mageduspec: type: NodePort ports: - name: http port: 80 protocol: TCP targetPort: 8080 nodePort: 30092 selector: app: magedu-tomcat-app1-selectorroot@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# root@harbor:~# mkdir -pv /data/k8sdata/magedu/imagesmkdir: created directory "/data/k8sdata/magedu"mkdir: created directory "/data/k8sdata/magedu/images"root@harbor:~# mkdir -pv /data/k8sdata/magedu/staticmkdir: created directory "/data/k8sdata/magedu/static"root@harbor:~# cat /etc/exports # /etc/exports: the access control list for filesystems which may be exported# to NFS clients. See exports(5).## Example for NFSv2 and NFSv3:# /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)## Example for NFSv4:# /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)# /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)#/data/k8sdata/kuboard *(rw,no_root_squash)/data/volumes *(rw,no_root_squash)/pod-vol *(rw,no_root_squash)/data/k8sdata/myserver *(rw,no_root_squash)/data/k8sdata/mysite *(rw,no_root_squash)/data/k8sdata/magedu/images *(rw,no_root_squash)/data/k8sdata/magedu/static *(rw,no_root_squash)root@harbor:~# exportfs -avexportfs: /etc/exports [1]: Neither "subtree_check" or "no_subtree_check" specified for export "*:/data/k8sdata/kuboard". Assuming default behaviour ("no_subtree_check"). NOTE: this default has changed since nfs-utils version 1.0.xexportfs: /etc/exports [2]: Neither "subtree_check" or "no_subtree_check" specified for export "*:/data/volumes". Assuming default behaviour ("no_subtree_check"). NOTE: this default has changed since nfs-utils version 1.0.xexportfs: /etc/exports [3]: Neither "subtree_check" or "no_subtree_check" specified for export "*:/pod-vol". Assuming default behaviour ("no_subtree_check"). NOTE: this default has changed since nfs-utils version 1.0.xexportfs: /etc/exports [4]: Neither "subtree_check" or "no_subtree_check" specified for export "*:/data/k8sdata/myserver". Assuming default behaviour ("no_subtree_check"). NOTE: this default has changed since nfs-utils version 1.0.xexportfs: /etc/exports [5]: Neither "subtree_check" or "no_subtree_check" specified for export "*:/data/k8sdata/mysite". Assuming default behaviour ("no_subtree_check"). NOTE: this default has changed since nfs-utils version 1.0.xexportfs: /etc/exports [7]: Neither "subtree_check" or "no_subtree_check" specified for export "*:/data/k8sdata/magedu/images". Assuming default behaviour ("no_subtree_check"). NOTE: this default has changed since nfs-utils version 1.0.xexportfs: /etc/exports [8]: Neither "subtree_check" or "no_subtree_check" specified for export "*:/data/k8sdata/magedu/static". Assuming default behaviour ("no_subtree_check"). NOTE: this default has changed since nfs-utils version 1.0.xexporting *:/data/k8sdata/magedu/staticexporting *:/data/k8sdata/magedu/imagesexporting *:/data/k8sdata/mysiteexporting *:/data/k8sdata/myserverexporting *:/pod-volexporting *:/data/volumesexporting *:/data/k8sdata/kuboardroot@harbor:~# root@k8s-master01:~/k8s-data/yaml# cd namespaces/root@k8s-master01:~/k8s-data/yaml/namespaces# lsmagedu-ns.yamlroot@k8s-master01:~/k8s-data/yaml/namespaces# kubectl apply -f magedu-ns.yaml namespace/magedu createdroot@k8s-master01:~/k8s-data/yaml/namespaces# lsmagedu-ns.yamlroot@k8s-master01:~/k8s-data/yaml/namespaces# cd ../magedu/tomcat-app1/root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# lshpa.yaml tomcat-app1.yamlroot@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# kubectl apply -f .horizontalpodautoscaler.autoscaling/magedu-tomcat-app1-podautoscaler createddeployment.apps/magedu-tomcat-app1-deployment createdservice/magedu-tomcat-app1-service createdroot@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1#root@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# kubectl get pods -n mageduNAME READY STATUS RESTARTS AGEmagedu-tomcat-app1-deployment-7754c8549c-prglk 1/1 Running 0 2m2smagedu-tomcat-app1-deployment-7754c8549c-xmg9l 1/1 Running 0 2m17sroot@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# kubectl get svc -n magedu NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEmagedu-tomcat-app1-service NodePort 10.100.129.23 80:30092/TCP 2m23sroot@k8s-master01:~/k8s-data/yaml/magedu/tomcat-app1# 访问k8s集群节点的30092端口，看看对应tomcat服务是否能够正常访问？
5.4、更改tomcat service 类型为无头服务，重新apply配置清单，让后端tomcat服务只能在k8s内部环境访问6、在k8s环境中运行nginx，实现nginx+tomcat动静分离能够通过k8s集群节点访问tomcat服务，说明tomcat服务已经正常部署至k8s环境；
root@k8s-master01:~/k8s-data/yaml/magedu/nginx# cat nginx.yamlkind: DeploymentapiVersion: apps/v1metadata: labels: app: magedu-nginx-deployment-label name: magedu-nginx-deployment namespace: mageduspec: replicas: 1 selector: matchLabels: app: magedu-nginx-selector template: metadata: labels: app: magedu-nginx-selector spec: containers: - name: magedu-nginx-container image: harbor.ik8s.cc/magedu/nginx-web1:v1 #command: ["/apps/tomcat/bin/run_tomcat.sh"] #imagePullPolicy: IfNotPresent imagePullPolicy: Always ports: - containerPort: 80 protocol: TCP name: http - containerPort: 443 protocol: TCP name: https env: - name: "password" value: "123456" - name: "age" value: "20" resources: limits: cpu: 500m memory: 512Mi requests: cpu: 500m memory: 256Mi volumeMounts: - name: magedu-images mountPath: /usr/local/nginx/html/webapp/images readOnly: false - name: magedu-static mountPath: /usr/local/nginx/html/webapp/static readOnly: false volumes: - name: magedu-images nfs: server: 192.168.0.42 path: /data/k8sdata/magedu/images - name: magedu-static nfs: server: 192.168.0.42 path: /data/k8sdata/magedu/static #nodeSelector: # group: magedu---kind: ServiceapiVersion: v1metadata: labels: app: magedu-nginx-service-label name: magedu-nginx-service namespace: mageduspec: type: NodePort ports: - name: http port: 80 protocol: TCP targetPort: 80 nodePort: 30090 - name: https port: 443 protocol: TCP targetPort: 443 nodePort: 30091 selector: app: magedu-nginx-selectorroot@k8s-master01:~/k8s-data/yaml/magedu/nginx# root@k8s-master01:~/k8s-data/yaml/magedu/nginx# kubectl apply -f .deployment.apps/magedu-nginx-deployment createdservice/magedu-nginx-service createdroot@k8s-master01:~/k8s-data/yaml/magedu/nginx# kubectl get pod -n mageduNAME READY STATUS RESTARTS AGEmagedu-nginx-deployment-5589bbf4bc-6gd2w 1/1 Running 0 14smagedu-tomcat-app1-deployment-7754c8549c-c7rtb 1/1 Running 0 8m7smagedu-tomcat-app1-deployment-7754c8549c-prglk 1/1 Running 0 19mroot@k8s-master01:~/k8s-data/yaml/magedu/nginx# kubectl get svc -n mageduNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEmagedu-nginx-service NodePort 10.100.94.118 80:30090/TCP,443:30091/TCP 24smagedu-tomcat-app1-service ClusterIP 10.100.129.23 80/TCP 19mroot@k8s-master01:~/k8s-data/yaml/magedu/nginx# 6.3、通过nginx 访问tomcat服务，看看tomcat是否能够被nginx代理？能够通过访问k8s集群节点的30090正常访问到nginx，说明nginx服务已经正常部署至k8s环境；
7、在负载均衡器上代理nginx服务能够通过nginx访问后端tomcat服务，说明nginx能够正常代理tomcat服务；
root@k8s-ha01:~# cat /etc/haproxy/haproxy.cfgglobal log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners stats timeout 30s user haproxy group haproxy daemon # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-ticketsdefaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.httplisten k8s_apiserver_6443bind 192.168.0.111:6443mode tcp#balance leastconnserver k8s-master01 192.168.0.31:6443 check inter 2000 fall 3 rise 5server k8s-master02 192.168.0.32:6443 check inter 2000 fall 3 rise 5server k8s-master03 192.168.0.33:6443 check inter 2000 fall 3 rise 5listen nginx-svc-80bind 192.168.0.111:80mode tcp server k8s-node01 192.168.0.34:30090 check inter 2000 fall 3 rise 5server k8s-node02 192.168.0.35:30090 check inter 2000 fall 3 rise 5server k8s-node03 192.168.0.36:30090 check inter 2000 fall 3 rise 5root@k8s-ha01:~# systemctl restart haproxyroot@k8s-ha01:~# root@k8s-master01:~/ubuntu/html# scp -rp * 192.168.0.42:/data/k8sdata/magedu/images/root@192.168.0.42"s password: 1.jpg 100% 40KB 8.6MB/s 00:00 index.html 100% 277 282.5KB/s 00:00 root@k8s-master01:~/ubuntu/html# 验证资源是否正常上传至存储对应目录？
root@harbor:~# ll /data/k8sdata/magedu/images/total 16drwxr-xr-x 3 root root 4096 Jun 4 07:56 ./drwxr-xr-x 4 root root 4096 Jun 4 06:32 ../drwxr-xr-x 2 root root 4096 May 31 18:37 images/-rw-r--r-- 1 root root 277 Aug 5 2022 index.htmlroot@harbor:~# 访问nginx服务，看看对应资源是否能够被读取？