【云原生 | Kubernetes 系列】—K8S部署logstach,实现es数据迁移

🍇 logstach简介

  Logstash是具有实时流水线能力的开源的数据收集引擎。Logstash可以动态统一不同来源的数据，
并将数据标准化到您选择的目标输出。它提供了大量插件，可帮助我们解析，丰富，转换和缓冲任何
类型的数据。
  管道（Logstash Pipeline）是Logstash中独立的运行单元，每个管道都包含两个必须的元素输
入（input）和输出（output），和一个可选的元素过滤器（filter），事件处理管道负责协调它
们的执行。 输入和输出支持编解码器，使您可以在数据进入或退出管道时对其进行编码或解码，而不
必使用单独的过滤器。如：json、multiline等

🍋 k8s部署logstach,实现es集群的数据同步

部署

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-configmap
  namespace: es7-cluster
data:
  logstash.yml: |-
    http.host: "0.0.0.0"
    path.config: /usr/share/logstash/pipeline
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch.username: username
    xpack.monitoring.elasticsearch.password: password
    xpack.monitoring.elasticsearch.hosts: [ "http://127.0.0.1:9200" ]
  logstash.conf: |-
    input {
      elasticsearch {
        hosts => ["127.0.0.1:9200"]
        user => "user"
        password => "password"
        index => "索引名称"
        size => 1000
        scroll => "1m"
      }
    }
    filter {
      mutate {
          remove_field => ["@timestamp", "@version"] #去除多余字段
      }
      fingerprint {
          source => ["id"]　　#根据哪些字段去重，此处根据id和时间字段
          target => "[@metadata][fingerprint]"
          method => "MURMUR3"　　#设置唯一id的方法
          concatenate_sources => true　　#将多个字段拼接为唯一id
      }
    }
    output {
       elasticsearch {
          hosts => ["127.0.0.1:9200"]
          user => "user"
          password => "password"
          index => "索引名称"
          document_id => "%{[@metadata][fingerprint]}"
       }
    }

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash-deployment
  namespace: es7-cluster
spec:
  selector:
    matchLabels:
      app: logstash
  replicas: 1
  template:
    metadata:
      labels:
        app: logstash
    spec:
      containers:
        - name: logstash
          image: logstash:7.17.4
          ports:
            - containerPort: 5044
          volumeMounts:
            - name: config-volume
              mountPath: /usr/share/logstash/config
            - name: logstash-pipeline-volume
              mountPath: /usr/share/logstash/pipeline
            - mountPath: /etc/localtime
              name: localtime
      volumes:
        - name: config-volume
          configMap:
            name: logstash-configmap
            items:
              - key: logstash.yml
                path: logstash.yml
        - name: logstash-pipeline-volume
          configMap:
            name: logstash-configmap
            items:
              - key: logstash.conf
                path: logstash.conf
        - name: localtime
          hostPath:
            path: /etc/localtime


---
kind: Service
apiVersion: v1
metadata:
  name: logstash-service
  namespace: es7-cluster
spec:
  selector:
    app: logstash
  ports:
    - protocol: TCP
      port: 5044
      targetPort: 5044

部署并查看

[root esxpack]# kubectl  apply   -f  logstach.yaml 
configmap/logstash-configmap created
deployment.apps/logstash-deployment created
service/logstash-service created
[root esxpack]# kubectl   get  po  -n  es7-cluster
NAME                                   READY   STATUS    RESTARTS   AGE
elasticsearch-master-0                 1/1     Running   0          63d
elasticsearch-master-1                 1/1     Running   0          63d
elasticsearch-master-2                 1/1     Running   0          63d
kibana-6975f8bc8-bk4kh                 1/1     Running   0          63d
logstash-deployment-7778479476-g2dc9   1/1     Running   0          45m

🥭 注意事项

• es和logstach版本要对应 点击查看
• es若开启了xpack认证,需要配置账号密码
• 注意去除重复数据,和多余的字段