Skip to content

elasticsearch cloud deploy for 8.x#123

Open
shaigbdb wants to merge 13 commits intomasterfrom
feat-8x
Open

elasticsearch cloud deploy for 8.x#123
shaigbdb wants to merge 13 commits intomasterfrom
feat-8x

Conversation

@shaigbdb
Copy link
Copy Markdown
Contributor

I've checked out the current master branch to elasticsearch-7.x .

In this PR:

  • 8.x revision for GCP, AWS
  • yml definitions, ES api calls conform to 8.x
  • data nodes are now hot-content (h+c) rather than data (d). This helps add tiering to the cluster.
  • heap dumps are disabled - those usually just exploded disks out of nowhere.
  • conformed to new GCP API filter syntax for unattached volume check
  • improved packer gcp image naming
  • GCP nodes now on ubuntu 20.04
  • grafana installation removed. I've left in optional elasticsearch monitoring in case someone still wants it, but didn't check whether it works
  • added encryption keys for security, saved objects, reports
  • moved es password in Kibana to keystore
  • AWS deployment conforms to new terraform
  • data-voters only in GCP for now
  • Azure untouched and probably not working

known issues:

  • 1-master clusters don't get bootstrapped well (but 1-master + 2 data voters do, and 3-master clusters work well). I suspect that was an existing issue.

FYI @LiorF-BDBQ @synhershko

@shaigbdb shaigbdb requested a review from LiorF-BDBQ February 22, 2024 16:49
Comment thread assets/scripts/client.sh Outdated
Comment on lines +56 to +58
cat <<'EOF' >>/etc/elasticsearch/elasticsearch.yml
xpack.security.http.ssl.enabled: false
EOF
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this makes sense, I think we should configure it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll open a separate issue for this, but I understand this is not the focus of the current task - @synhershko
Either way in 8.x the flag must be either false or true.

Comment thread assets/scripts/common/config-es.sh Outdated
Comment thread assets/scripts/data.sh Outdated
Comment thread assets/scripts/gcp/config-es-discovery.sh
Comment thread assets/scripts/singlenode.sh Outdated
Comment thread packer/README.md Outdated
Comment thread packer/elasticsearch8-node.packer.json Outdated
Comment thread terraform-aws/vpc.tf
Comment thread terraform-gcp/main.tf Outdated
shaigbdb added 7 commits March 6, 2024 10:09
# Conflicts:
#	terraform-gcp/main.tf
…dpoint ports, adding packer manifest, adding debug bootstrap mode, fixing redundant provider, adding certs renewal
# Conflicts:
#	.gitignore
#	assets/scripts/bootstrap.sh
#	assets/scripts/common/config-clients.sh
#	assets/scripts/common/config-cluster.sh
#	assets/scripts/common/config-es.sh
#	assets/scripts/gcp/autoattach-disk.sh
#	packer/README.md
#	packer/elasticsearch8-node.packer.json
#	packer/kibana8-node.packer.json
#	templates/aws_user_data.sh
#	templates/gcp_user_data.sh
#	terraform-aws/main.tf
#	terraform-aws/variables.tf
#	terraform-aws/vpc.tf
#	terraform-gcp/main.tf
#	terraform-gcp/variables.tf
# Conflicts:
#	templates/gcp_user_data.sh
Comment thread terraform-aws/lock.old Outdated
Comment on lines +1 to +6
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/aws" {
version = "5.37.0"
hashes = [
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to keep this file? Lets add a required_providers block with all providers we use here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants