Kubernetes使用

使用前提

已部署Kubernetes集群或minikube

具备访问Kubernetes集群的kubectl

采集Kubernetes容器日志

1. 创建部署iLogtail的命名空间

   将下面内容保存为ilogtail-ns.yaml

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ilogtail

   您也可以直接从下面的地址下载示例配置。

   wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-ns.yaml

   应用上述配置

   kubectl apply -f ilogtail-ns.yaml

2. 创建配置iLogtail的ConfigMap和Secret

   将下面内容保存为ilogtail-user-configmap.yaml。该ConfigMap后续将作为配置目录挂载到iLogtail容器中，因此可包含多个采集配置。

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: ilogtail-user-cm
     namespace: ilogtail
   data:
     nginx_stdout.yaml: |
       enable: true
       inputs:
         - Type: service_docker_stdout
           Stderr: false
           Stdout: true                # only collect stdout
           IncludeK8sLabel:
             app: nginx                # choose containers with this label
       processors:
         - Type: processor_regex       # structure log
           SourceKey: content
           Regex: '([\d\.:]+) - (\S+) \[(\S+) \S+\] \"(\S+) (\S+) ([^\\"]+)\" (\d+) (\d+) \"([^\\"]*)\" \"([^\\"]*)\" \"([^\\"]*)\"'
           Keys:
             - remote_addr
             - remote_user
             - time_local
             - method
             - url
             - protocol
             - status
             - body_bytes_sent
             - http_referer
             - http_user_agent
             - http_x_forwarded_for
       flushers:
         - Type: flusher_stdout
           OnlyStdout: true

   将下面内容保存为ilogtail-secret.yaml。该Secret为可选，当需要将日志写入SLS时会用到。

   apiVersion: v1
   kind: Secret
   metadata:
     name: ilogtail-secret
     namespace: ilogtail
   type: Opaque
   data:
     access_key_id:  # base64 accesskey id if you want to flush to SLS
     access_key:     # base64 accesskey secret if you want to flush to SLS

   您也可以直接从下面的地址下载示例配置。

   wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-user-configmap.yaml
   wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-secret.yaml

   应用上述配置

   kubectl apply -f ilogtail-user-configmap.yaml
   kubectl apply -f ilogtail-secret.yaml

3. 创建iLogtail DaemonSet

   将下面内容保存为ilogtail-daemonset.yaml。

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: ilogtail-ds
     namespace: ilogtail
     labels:
       k8s-app: logtail-ds
   spec:
     selector:
       matchLabels:
         k8s-app: logtail-ds
     template:
       metadata:
         labels:
           k8s-app: logtail-ds
       spec:
         tolerations:
           - operator: Exists                    # deploy on all nodes
         containers:
           - name: logtail
             env:
               - name: ALIYUN_LOG_ENV_TAGS       # add log tags from env
                 value: _node_name_|_node_ip_
               - name: _node_name_
                 valueFrom:
                   fieldRef:
                     apiVersion: v1
                     fieldPath: spec.nodeName
               - name: _node_ip_
                 valueFrom:
                   fieldRef:
                     apiVersion: v1
                     fieldPath: status.hostIP
               - name: cpu_usage_limit           # iLogtail's self monitor cpu limit
                 value: "1"
               - name: mem_usage_limit           # iLogtail's self monitor mem limit
                 value: "512"
               - name: default_access_key_id     # accesskey id if you want to flush to SLS
                 valueFrom:
                   secretKeyRef:
                     name: ilogtail-secret
                     key: access_key_id
                     optional: true
               - name: default_access_key        # accesskey secret if you want to flush to SLS
                 valueFrom:
                   secretKeyRef:
                     name: ilogtail-secret
                     key: access_key
                     optional: true
             image: >-
               sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:latest
             imagePullPolicy: IfNotPresent
             resources:
               limits:
                 cpu: 1000m
                 memory: 1Gi
               requests:
                 cpu: 400m
                 memory: 384Mi
             volumeMounts:
               - mountPath: /var/run                       # for container runtime socket
                 name: run
               - mountPath: /logtail_host                  # for log access on the node
                 mountPropagation: HostToContainer
                 name: root
                 readOnly: true
               - mountPath: /usr/local/ilogtail/checkpoint # for checkpoint between container restart
                 name: checkpoint
               - mountPath: /usr/local/ilogtail/config/local # mount config dir
                 name: user-config
                 readOnly: true
         dnsPolicy: ClusterFirstWithHostNet
         hostNetwork: true
         volumes:
           - hostPath:
               path: /var/run
               type: Directory
             name: run
           - hostPath:
               path: /
               type: Directory
             name: root
           - hostPath:
               path: /etc/ilogtail-ilogtail-ds/checkpoint
               type: DirectoryOrCreate
             name: checkpoint
           - configMap:
               defaultMode: 420
               name: ilogtail-user-cm
             name: user-config

   您也可以直接从下面的地址下载示例配置。

   wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/ilogtail-daemonset.yaml

   添加参数示例

           - name: logtail
             command:
             - /usr/local/ilogtail/ilogtail_control.sh
             args:
             - "start_and_block"
             - "-enable_containerd_upper_dir_detect=true"
             - "-dirfile_check_interval_ms=5000"
             - "-logtail_checkpoint_check_gc_interval_sec=120"

   应用上述配置

   kubectl apply -f ilogtail-daemonset.yaml

4. 部署用来测试的nginx

   将下面内容保存为nginx-deployment.yaml。

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nginx
     namespace: default
     labels:
       app: nginx
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
           - image: 'nginx:latest'
             name: nginx
             ports:
               - containerPort: 80
                 name: http
                 protocol: TCP
             resources:
               requests:
                 cpu: 100m
                 memory: 100Mi

   您也可以直接从下面的地址下载示例配置。

   wget https://raw.githubusercontent.com/alibaba/ilogtail/main/example_config/start_with_k8s/nginx-deployment.yaml

   应用上述配置

   kubectl apply -f nginx-deployment.yaml

5. 发送请求构造示例日志

   kubectl exec nginx-<pod-id> -- curl localhost/hello/ilogtail

6. 查看采集到的测试容器标准输出日志

   kubectl logs ilogtail-ds-<pod-id> -n ilogtail

   结果为

   2022-07-14 16:36:50 {"_time_":"2022-07-15T00:36:48.489153485+08:00","_source_":"stdout","_image_name_":"docker.io/library/nginx:latest","_container_name_":"nginx","_pod_name_":"nginx-76d49876c7-r892w","_namespace_":"default","_pod_uid_":"07f75a79-da69-40ac-ae2b-77a632929cc6","_container_ip_":"10.223.0.154","remote_addr":"::1","remote_user":"-","time_local":"14/Jul/2022:16:36:48","method":"GET","url":"/hello/ilogtail","protocol":"HTTP/1.1","status":"404","body_bytes_sent":"153","http_referer":"-","http_user_agent":"curl/7.74.0","http_x_forwarded_for":"-","__time__":"1657816609"}

采集模版

了解采集配置结构：采集配置

查看更多K8s采集日志模版（从容器中采集文件日等）：https://github.com/alibaba/ilogtail/blob/main/k8s_templates

参考更多可组合的日志采集和处理配置样例：https://github.com/alibaba/ilogtail/blob/main/example_config