ElasticSearch :: Ay Docs

ES [Local Disk]

Mon, 07 Oct 2024 19:58:45 +0800

Preliminary ElasticSearch has installed, if not check link The elasticsearch.yml has configed path.repo, which should be set the same value of settings.location (this will be handled by helm chart, don’t worry) ES argocd-app yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: elastic-search spec: syncPolicy: syncOptions: - CreateNamespace=true project: default source: repoURL: https://charts.bitnami.com/bitnami chart: elasticsearch targetRevision: 19.11.3 helm: releaseName: elastic-search values: | global: kibanaEnabled: true clusterName: elastic image: registry: m.zjvis.net/docker.io pullPolicy: IfNotPresent security: enabled: false service: type: ClusterIP extraConfig: path: repo: /tmp ingress: enabled: true annotations: cert-manager.io/cluster-issuer: self-signed-ca-issuer nginx.ingress.kubernetes.io/rewrite-target: /$1 hostname: elastic-search.dev.72602.online ingressClassName: nginx path: /?(.*) tls: true master: masterOnly: false replicaCount: 1 persistence: enabled: false resources: requests: cpu: 2 memory: 1024Mi limits: cpu: 4 memory: 4096Mi heapSize: 2g data: replicaCount: 0 persistence: enabled: false coordinating: replicaCount: 0 ingest: enabled: true replicaCount: 0 service: enabled: false type: ClusterIP ingress: enabled: false metrics: enabled: false image: registry: m.zjvis.net/docker.io pullPolicy: IfNotPresent volumePermissions: enabled: false image: registry: m.zjvis.net/docker.io pullPolicy: IfNotPresent sysctlImage: enabled: true registry: m.zjvis.net/docker.io pullPolicy: IfNotPresent kibana: elasticsearch: hosts: - '{{ include "elasticsearch.service.name" . }}' port: '{{ include "elasticsearch.service.ports.restAPI" . }}' esJavaOpts: "-Xmx2g -Xms2g" destination: server: https://kubernetes.default.svc namespace: application diff from oirginal file : extraConfig: path: repo: /tmp

ES [S3 Compatible]

Mon, 07 Oct 2024 19:58:45 +0800

Preliminary ElasticSearch has installed, if not check link ES argocd-app yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: elastic-search spec: syncPolicy: syncOptions: - CreateNamespace=true project: default source: repoURL: https://charts.bitnami.com/bitnami chart: elasticsearch targetRevision: 19.11.3 helm: releaseName: elastic-search values: | global: kibanaEnabled: true clusterName: elastic image: registry: m.zjvis.net/docker.io pullPolicy: IfNotPresent security: enabled: true service: type: ClusterIP extraEnvVars: - name: S3_ACCESSKEY value: admin - name: S3_SECRETKEY value: ZrwpsezF1Lt85dxl extraConfig: s3: client: default: protocol: http endpoint: "http://:9090" path_style_access: true initScripts: configure-s3-client.sh: | elasticsearch_set_key_value "s3.client.default.access_key" "${S3_ACCESSKEY}" elasticsearch_set_key_value "s3.client.default.secret_key" "${S3_SECRETKEY}" hostAliases: - ip: hostnames: - minio-api.dev.72602.online ingress: enabled: true annotations: cert-manager.io/cluster-issuer: self-signed-ca-issuer nginx.ingress.kubernetes.io/rewrite-target: /$1 hostname: elastic-search.dev.72602.online ingressClassName: nginx path: /?(.*) tls: true master: masterOnly: false replicaCount: 1 persistence: enabled: false resources: requests: cpu: 2 memory: 1024Mi limits: cpu: 4 memory: 4096Mi heapSize: 2g data: replicaCount: 0 persistence: enabled: false coordinating: replicaCount: 0 ingest: enabled: true replicaCount: 0 service: enabled: false type: ClusterIP ingress: enabled: false metrics: enabled: false image: registry: m.zjvis.net/docker.io pullPolicy: IfNotPresent volumePermissions: enabled: false image: registry: m.zjvis.net/docker.io pullPolicy: IfNotPresent sysctlImage: enabled: true registry: m.zjvis.net/docker.io pullPolicy: IfNotPresent kibana: elasticsearch: hosts: - '{{ include "elasticsearch.service.name" . }}' port: '{{ include "elasticsearch.service.ports.restAPI" . }}' esJavaOpts: "-Xmx2g -Xms2g" destination: server: https://kubernetes.default.svc namespace: application diff from oirginal file : extraEnvVars: - name: S3_ACCESSKEY value: admin - name: S3_SECRETKEY value: ZrwpsezF1Lt85dxl extraConfig: s3: client: default: protocol: http endpoint: "http://:9090" path_style_access: true initScripts: configure-s3-client.sh: | elasticsearch_set_key_value "s3.client.default.access_key" "${S3_ACCESSKEY}" elasticsearch_set_key_value "s3.client.default.secret_key" "${S3_SECRETKEY}" hostAliases: - ip: hostnames: - minio-api.dev.72602.online

ES Auto BackUp

Mon, 07 Oct 2024 19:58:45 +0800

Preliminary ElasticSearch has installed, if not check link We use local disk to save the snapshots, more deatils check link And the security is enabled. ES argocd-app yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: elastic-search spec: syncPolicy: syncOptions: - CreateNamespace=true project: default source: repoURL: https://charts.bitnami.com/bitnami chart: elasticsearch targetRevision: 19.11.3 helm: releaseName: elastic-search values: | global: kibanaEnabled: true clusterName: elastic image: registry: m.zjvis.net/docker.io pullPolicy: IfNotPresent security: enabled: true tls: autoGenerated: true service: type: ClusterIP extraConfig: path: repo: /tmp ingress: enabled: true annotations: cert-manager.io/cluster-issuer: self-signed-ca-issuer nginx.ingress.kubernetes.io/rewrite-target: /$1 hostname: elastic-search.dev.72602.online ingressClassName: nginx path: /?(.*) tls: true master: masterOnly: false replicaCount: 1 persistence: enabled: false resources: requests: cpu: 2 memory: 1024Mi limits: cpu: 4 memory: 4096Mi heapSize: 2g data: replicaCount: 0 persistence: enabled: false coordinating: replicaCount: 0 ingest: enabled: true replicaCount: 0 service: enabled: false type: ClusterIP ingress: enabled: false metrics: enabled: false image: registry: m.zjvis.net/docker.io pullPolicy: IfNotPresent volumePermissions: enabled: false image: registry: m.zjvis.net/docker.io pullPolicy: IfNotPresent sysctlImage: enabled: true registry: m.zjvis.net/docker.io pullPolicy: IfNotPresent kibana: elasticsearch: hosts: - '{{ include "elasticsearch.service.name" . }}' port: '{{ include "elasticsearch.service.ports.restAPI" . }}' esJavaOpts: "-Xmx2g -Xms2g" destination: server: https://kubernetes.default.svc namespace: application diff from oirginal file : security: enabled: true extraConfig: path: repo: /tmp

Example Shell Script

Thu, 14 Mar 2024 15:00:59 +0800

Init ES Backup Setting create an ES backup setting in s3, and make an snapshot after creation #!/bin/bash ES_HOST="http://192.168.58.2:30910" ES_BACKUP_REPO_NAME="s3_fs_repository" S3_CLIENT="default" ES_BACKUP_BUCKET_IN_S3="es-snapshot" ES_SNAPSHOT_TAG="auto" CHECK_RESPONSE=$(curl -s -k -X POST "$ES_HOST/_snapshot/$ES_BACKUP_REPO_NAME/_verify?pretty" ) CHECKED_NODES=$(echo "$CHECK_RESPONSE" | jq -r '.nodes') if [ "$CHECKED_NODES" == null ]; then echo "Doesn't exist an ES backup setting..." echo "A default backup setting will be generated. (using '$S3_CLIENT' s3 client and all backup files will be saved in a bucket : '$ES_BACKUP_BUCKET_IN_S3'" CREATE_RESPONSE=$(curl -s -k -X PUT "$ES_HOST/_snapshot/$ES_BACKUP_REPO_NAME?pretty" -H 'Content-Type: application/json' -d "{\"type\":\"s3\",\"settings\":{\"bucket\":\"$ES_BACKUP_BUCKET_IN_S3\",\"client\":\"$S3_CLIENT\"}}") CREATE_ACKNOWLEDGED_FLAG=$(echo "$CREATE_RESPONSE" | jq -r '.acknowledged') if [ "$CREATE_ACKNOWLEDGED_FLAG" == true ]; then echo "Buckup setting '$ES_BACKUP_REPO_NAME' has been created successfully!" else echo "Failed to create backup setting '$ES_BACKUP_REPO_NAME', since $$CREATE_RESPONSE" fi else echo "Already exist an ES backup setting '$ES_BACKUP_REPO_NAME'" fi CHECK_RESPONSE=$(curl -s -k -X POST "$ES_HOST/_snapshot/$ES_BACKUP_REPO_NAME/_verify?pretty" ) CHECKED_NODES=$(echo "$CHECK_RESPONSE" | jq -r '.nodes') if [ "$CHECKED_NODES" != null ]; then SNAPSHOT_NAME="meta-data-$ES_SNAPSHOT_TAG-snapshot-$(date +%s)" SNAPSHOT_CREATION=$(curl -s -k -X PUT "$ES_HOST/_snapshot/$ES_BACKUP_REPO_NAME/$SNAPSHOT_NAME") echo "Snapshot $SNAPSHOT_NAME has been created." else echo "Failed to create snapshot $SNAPSHOT_NAME ." fi