Installing and configuring Elasticsearch for WhatsUp Gold

That was fun.

I wanted to learn how difficult it was to install Elasticsearch on Linux as compared to Windows. In the end, we have an installation script that does exactly what we need to do to install and configure an Elasticsearch backend as required by WhatsUp Gold Log Management. There are even more secure ways of configuring Elasticsearch. The method used within the example script provided uses no passwords on the certificates, which are only required to enable encrypted communications.

Automating Elasticsearch Installation with SSL & Passwordless Authentication on RHEL

When it comes to deploying Elasticsearch securely, automating the process ensures consistency, speed, and reduced chances for human error. In this blog post, we’ll walk through the steps we took to automate the installation and configuration of Elasticsearch with SSL using a self-signed certificate, along with passwordless authentication on a RHEL-based system (Rocky Linux in our case). This process ensures that Elasticsearch is set up securely for production use.

Background

Our goal was to automate the entire installation of Elasticsearch version 8.15.2, configure SSL/TLS encryption for HTTP and transport communications using a self-signed certificate, and ensure passwordless authentication where appropriate. The automation also handled necessary firewall adjustments and reset the elastic user password, verifying that everything worked as expected.

We hit several hurdles along the way—particularly with SSL certificate generation and password management—but in the end, we were able to develop a robust bash script that met our goals.

Step-by-Step Breakdown

Step 1: Setting Up Elasticsearch

The journey started with automating the installation of Elasticsearch itself. We created a bash script that checks if Elasticsearch is installed on the machine, and if not, it installs the latest version (in our case, version 8.15.2). This step ensures that even in future deployments, the script will handle the installation cleanly.

bash
# Check if Elasticsearch is already installed
if rpm -q elasticsearch &>/dev/null; then
    echo "Elasticsearch is already installed. Skipping installation."
else
    echo "Installing Elasticsearch $ELASTIC_VERSION..."
    sudo dnf install -y https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-$ELASTIC_VERSION-x86_64.rpm
fi

Step 2: Generating the SSL Certificates

One of the critical parts of this automation is setting up SSL encryption using a self-signed certificate. We initially tried using openssl directly, but ultimately, using elasticsearch-certutil made things easier.

Our script ensures the CA and certificates are regenerated every time, guaranteeing that you always get fresh, valid certificates.

bash
# Always generate the CA
echo "Generating self-signed CA using elasticsearch-certutil..."
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca --out "$CA_FILE" --pass ""

# Always generate the certificate for the node with SAN support
echo "Generating the certificate for $NODE_IP_HOSTNAME with no password using the CA..."
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca "$CA_FILE" --out "$CERT_FILE" --pass "" --ip "$NODE_IP" --dns "$NODE_IP_HOSTNAME"

We encountered an issue where previously generated certificates were reused due to conditional logic that skipped regenerating them. This was fixed by ensuring the certificates and CA were always re-generated.

Step 3: Configuring Elasticsearch

To enable SSL, we need to modify the elasticsearch.yml file with appropriate configurations. The script automates this by writing the necessary configuration directly into the file, ensuring that both HTTP and transport layers are encrypted.

bash
# Step 4: Configure Elasticsearch (always rewrite config)
cat <<EOF | sudo tee /etc/elasticsearch/elasticsearch.yml > /dev/null
# ---------------------------------- Network -----------------------------------
http.host: 0.0.0.0

# ---------------------------------- Cluster -----------------------------------
cluster.initial_master_nodes: ["$NODE_IP_HOSTNAME"]

# ---------------------------------- Security -----------------------------------
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "$CERT_FILE"
xpack.security.http.ssl.truststore.path: "$CERT_FILE"
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: "$CERT_FILE"
xpack.security.transport.ssl.truststore.path: "$CERT_FILE"
EOF

Step 4: Resetting the Keystore Passwords

A recurring issue we faced was Elasticsearch failing to start due to keystore password issues. To resolve this, we ensured that any existing keystore passwords were removed so we could run the system passwordless for SSL. This helped fix recurring errors where the system could not decrypt or validate certificates.

bash

# Step 5: Remove existing keystore passwords
sudo /usr/share/elasticsearch/bin/elasticsearch-keystore remove xpack.security.http.ssl.keystore.secure_password || echo "No password to remove for HTTP keystore."
sudo /usr/share/elasticsearch/bin/elasticsearch-keystore remove xpack.security.transport.ssl.keystore.secure_password || echo "No password to remove for transport keystore."
sudo /usr/share/elasticsearch/bin/elasticsearch-keystore remove xpack.security.transport.ssl.truststore.secure_password || echo "No password to remove for transport truststore."

Step 5: Resetting the elastic User Password

Finally, we automated the resetting of the elastic user password. This was critical to ensure secure access to Elasticsearch, and our script captures the newly generated password to allow seamless access later.

bash
# Step 9: Reset 'elastic' user password
ELASTIC_PASSWORD=$(sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -b -s)
if [[ $? -eq 0 ]]; then
    echo "Elastic user password: $ELASTIC_PASSWORD"
else
    echo "Failed to reset elastic user password."
    exit 1
fi

Step 6: Testing Connectivity

The last part of the script verifies that Elasticsearch is running and that we can successfully connect using the elastic user. It uses the newly generated password in a curl request to verify that the service is up and running securely.

bash
# Step 10: Verify access using curl with elastic user
curl -k -u elastic:"$ELASTIC_PASSWORD" https://$NODE_IP:9200

Lessons Learned

1. Regenerating Certificates: It’s important to always regenerate the CA and certificates to avoid SSL errors, especially when rerunning scripts on the same node.
2. Passwordless Keystore: Removing keystore passwords is critical for setting up passwordless SSL, and this fixed several issues we encountered.
3. Automating User Management: Automating the reset of the elastic user password streamlined the process, ensuring a fresh and secure environment after each run.

Conclusion

By the end of this process, we had a fully automated Elasticsearch setup, complete with SSL encryption, passwordless authentication for certificates, and automated password management for the elastic user. This approach is ideal for deploying Elasticsearch on RHEL or Rocky Linux systems, ensuring a secure and scalable installation.

Instructions

1. Login as a user capable of running ‘as root’
2. Create the install script by running: vi elasticsearch-install.sh
3. Copy the code below, hit the i key, and then right-click to paste
4. Type :w to write the file
5. Type :q to quit vi
6. Run the command: chmod +x elasticsearch-install.sh
7. Run the script: ./elasticsearch-install.sh
8. Hit Enter when prompted for a password (if you enter one, it’s not going to work!)
9. Wait for the script to complete and give you the elastic password
10. Drop the username, password, and IP address into WhatsUp Gold

#!/bin/bash

# Variables
CERT_DIR="/etc/elasticsearch/certs"
CERT_FILE="${CERT_DIR}/elastic-certificates.p12"
CA_FILE="${CERT_DIR}/elastic-stack-ca.p12"
NODE_IP_HOSTNAME="$(hostname -f)"  # Automatically gets the hostname
NODE_IP="$(hostname -I | awk '{print $1}')"  # Automatically gets the first IP address
ELASTIC_VERSION="8.15.2"
FIREWALL_ENABLED=true

# Stop Elasticsearch (if running)
echo "Stopping Elasticsearch if running..."
systemctl stop elasticsearch || echo "Elasticsearch service not running, continuing..."

# Check if Elasticsearch is already installed
if rpm -q elasticsearch &>/dev/null; then
    echo "Elasticsearch is already installed. Skipping installation."
else
    echo "Installing Elasticsearch $ELASTIC_VERSION..."
    sudo dnf install -y https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-$ELASTIC_VERSION-x86_64.rpm
fi

# Ensure certificate directory exists and is fresh
echo "Cleaning up and preparing certificate directory..."
sudo rm -rf "$CERT_DIR"  # Always regenerate everything
sudo mkdir -p "$CERT_DIR"

# Step 1: Generate the CA (ALWAYS regenerate)
echo "Generating self-signed CA using elasticsearch-certutil..."
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca --out "$CA_FILE" --pass ""

# Step 2: Generate certificate for the node with SAN (hostname and IP) - always regenerate
echo "Generating the certificate for $NODE_IP_HOSTNAME with no password using the CA..."
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca "$CA_FILE" --out "$CERT_FILE" --pass "" --ip "$NODE_IP" --dns "$NODE_IP_HOSTNAME"

# Step 3: Fix permissions on certificates
echo "Setting permissions on the certificate file..."
sudo chown elasticsearch:elasticsearch "$CERT_FILE"
sudo chmod 600 "$CERT_FILE"

# Step 4: Configure Elasticsearch (always rewrite config)
echo "Configuring Elasticsearch..."
cat <<EOF | sudo tee /etc/elasticsearch/elasticsearch.yml > /dev/null
# ---------------------------------- Network -----------------------------------
http.host: 0.0.0.0

# ---------------------------------- Cluster -----------------------------------
cluster.initial_master_nodes: ["$NODE_IP_HOSTNAME"]

# ---------------------------------- Security -----------------------------------
xpack.security.enabled: true

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "$CERT_FILE"
xpack.security.http.ssl.truststore.path: "$CERT_FILE"

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: "$CERT_FILE"
xpack.security.transport.ssl.truststore.path: "$CERT_FILE"

# ---------------------------------- Authentication (Optional for Kibana) -----------------------------------
xpack.security.authc.token.enabled: true

path.logs: /var/log/elasticsearch
path.data: /var/lib/elasticsearch
EOF

# Step 5: Remove existing keystore passwords (for passwordless SSL)
echo "Removing keystore passwords for SSL..."
sudo /usr/share/elasticsearch/bin/elasticsearch-keystore remove xpack.security.http.ssl.keystore.secure_password || echo "No password to remove for HTTP keystore."
sudo /usr/share/elasticsearch/bin/elasticsearch-keystore remove xpack.security.transport.ssl.keystore.secure_password || echo "No password to remove for transport keystore."
sudo /usr/share/elasticsearch/bin/elasticsearch-keystore remove xpack.security.transport.ssl.truststore.secure_password || echo "No password to remove for transport truststore."

# Step 6: Open necessary firewall ports (Always open)
if [ "$FIREWALL_ENABLED" = true ]; then
    echo "Opening firewall ports..."
    sudo firewall-cmd --add-port=9200/tcp --permanent
    sudo firewall-cmd --add-port=9300/tcp --permanent
    sudo firewall-cmd --reload
fi

# Step 7: Start Elasticsearch service
echo "Starting Elasticsearch service..."
sudo systemctl start elasticsearch

# Step 8: Check if Elasticsearch service is active before proceeding
echo "Checking if Elasticsearch started successfully..."
if ! systemctl is-active --quiet elasticsearch; then
    echo "Elasticsearch failed to start. Exiting."
    exit 1
fi

# Step 9: Reset 'elastic' user password
echo "Resetting password for the 'elastic' user..."
ELASTIC_PASSWORD=$(sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -b -s)
if [[ $? -eq 0 ]]; then
    echo "Elastic user password: $ELASTIC_PASSWORD"
else
    echo "Failed to reset elastic user password."
    exit 1
fi

# Step 10: Verify access using curl with elastic user
echo "Verifying access to Elasticsearch with 'elastic' user credentials..."
curl -k -u elastic:"$ELASTIC_PASSWORD" https://$NODE_IP:9200

echo "Elasticsearch setup complete. You can check the status using 'systemctl status elasticsearch' or view logs in '/var/log/elasticsearch/'."