Just making some notes for the upgrade steps for Zookeeper as a memo.

Prerequisites

• Terraform work complete to create Zookeeper Auto Scaling Group (ASG)

Autoscaling

Full autoscaling Zookeeper (actually adding new nodes to the Zookeeper ensemble automatically) is a bit tricky currently as a full rolling restart of the Zookeeper cluster would have to be orchestrated without ever losing quorum. Zookeeper 3.5 introduced dynamic reconfiguration which would most likely make this significantly easier. There are some other options for autoscaling Zookeeper 3.4 but we’d need to weigh up whether that additional complexity is really worth it.

Puppet

Using the puppet module: deric/puppet-zookeeper

Background

Let’s say there’s a legacy platform called XXX, and it still poses an operational cost for SRE to manage the monitoring / alerting / upgrade etc.

At present, XXX’s ElasticSearch architecture consists of 3 hosts (m4.xlarge) acting as master, client and data nodes, for XXX’s searching functionality.

All of them are using Ubuntu Trusty (14.04) image, which is no longer supported and need to be migrated to Xenial (16.04). Its usage seems pretty low and cost wise so we would like to migrate the XXX ES to AWS managed ElasticSearch Service to reduce operational overhead.

This doc will cover the project plan / checklist for migration.

Solution overview

Elasticsearch (ES) indexes can be migrated with following steps:

• Create baseline indexes

  • Create a snapshot repository and associate it to an AWS S3 Bucket.
  • Create the first snapshot of the indexes to be migrated, which is a full snapshot on EC2 - The snapshot will be automatically stored in the AWS S3 bucket created in the first step.
  • Restore this full snapshot to the AWS ES.

• Periodic incremental snapshots

  • Repeat serval incremental snapshot and restore.

• Final snapshot and service switchover

  • Stop services which can modify index data.
  • Create a final incremental snapshot of the EC2.
  • Perform service switchover to the AWS ES.

Background

When using make plan command, if there’re too many resources need to be applied, together with other unrelated resources that we don’t want to touch, then we need to use Terraform resource targeting:

1

PLAN_OPTIONS="-target="A" -target="B" ... -target="N"" make plan

But Terraform doesn’t support –exclude feature for the target at the moment (and we don’t want to copy & paste over 50 times for the targets), that’s why need to find a way to implement exclude / include features within Makefile.

This document outlines the steps to move existing terraform resources to different remote states.

Reference: https://medium.com/@lynnlin827/moving-terraform-resources-states-from-one-remote-state-to-another-c76f8b76a996

Steps

1. Move Terraform files to the new folder

2. Run make plan to get a list of resources that would be deleted

sample to generate desired commands used for next step

1

terraform plan | grep '\s-\s' | sed -e 's/\s-\s//g' | awk '{print "terraform state mv -state-out=monitoring/.terraform/terraform.tfstate "$1,$1}'

Glue - Official FAQ

The official doc for troubleshooting could be found here

Troubleshooting - Notes & Tips

Notes and tips for Glue when implementing the ETL process:

Naming rules / conventions for AWS services

• S3 bucket name can either NOT uppercase nor NOT contain “_”
• Dynamo DB table / Name can NOT contain “-“
• ES / Name: can NOT contain “-“

ACM SSL Cert

Using us-east-1 region for AWS CloudFront (certificate)

This document outlines the steps to clean or regenerate puppet agent certificates in a traditional master/client setup.

First thing is to ssh into the agent

Then, delete all *.pem files in /var/lib/puppet/ssl associated to your instance.
e.g they should be in the form of .pem

1
2
3
4
5
6
7
8
9
10
11

root@gateway:/var/lib/puppet/ssl# hostname -f
gateway.ap-southeast-2.aws.ci.xxxxxx.com

root@gateway:/var/lib/puppet/ssl# find . -type f | grep "gateway.ap-southeast-2.aws.ci.xxxxxx.com"
./public_keys/gateway.ap-southeast-2.aws.ci.xxxxxx.com.pem
./private_keys/gateway.ap-southeast-2.aws.ci.xxxxxx.com.pem
./certificate_requests/gateway.ap-southeast-2.aws.ci.xxxxxx.com.pem
./certs/gateway.ap-southeast-2.aws.ci.xxxxxx.com.pem

# to delete them
find . -type f | grep "gateway.ap-southeast-2.aws.ci.xxxxxx.com" | xargs rm -rf

Next step is to ssh into the puppet master and do the same

Errors

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

root@elastic-data:/mnt/elasticsearch/nodes/0/indices# ls -rthl
...
drwxr-xr-x   4 elasticsearch elasticsearch 4.0K Apr 15 14:55 _nGsAa--TU-Yt8zAsPWzBw
drwxr-xr-x   5 elasticsearch elasticsearch 4.0K Apr 15 14:55 -hqCV4oLRG-ttGc8A93BVw
drwxr-xr-x   5 elasticsearch elasticsearch 4.0K Apr 15 14:55 y4ORbH-YRL65u5JPRWyM3A
...

root@elastic-data:/mnt/elasticsearch/nodes/0/indices# du -sh *
du: invalid option -- '6'
du: invalid option -- 'w'
du: invalid option -- 'U'
du: invalid option -- 'g'
du: invalid option -- 'E'
du: invalid option -- 'Q'
du: invalid option -- 'q'
du: invalid option -- 'I'
du: invalid option -- 'R'
du: invalid option -- 'f'
du: eQWyw: No such file or directory
du: invalid option -- 'F'
du: xFiduSIGlLbpGulXyqA: No such file or directory
du: invalid option -- 'q'
du: invalid option -- 'C'
du: invalid option -- 'V'
du: invalid option -- '4'
du: invalid option -- 'o'
du: invalid option -- 'R'
du: invalid option -- 'G'
du: invalid option -- '-'
du: invalid -t argument 'tGc8A93BVw'