Salt on Graviton

2023-01-27 6 min read 3sky

Welcome

It will be nice to solve another real-world problem, isn’t it? For example machines in private subnets, behind the NAT, or on rack shelves in an office. All of them need to be managed - at least from time to time.

Ansible is great, I appreciated the creators of this tool. It helps me a lot! In different cases, environment. I remember when I was a bit younger, a very popular interview question was - what is the difference between Ansible and Chef/Puppet ?. If we skip the part, that Chef/Puppet isn’t very popular in Poland, the main and always correct answer was that Ansible is agentless.

Then I met a guy. Do you know what he said? Agentless? Phi, what about all these community roles and packages, python packages, etc? Your workstation is an agent. That for an unknown reason was a game-changer for me. From that point in time I desired an agent.

It solves the ingress issue, but the need for an experiment was stronger!

Issue

Let’s talk about the ingress issue for a moment. According to security best practices we need to take care of a lot of incoming traffic. CIDR ranges, NATs, VPN, private subnets, bastion hosts, or System Manager which is awesome (however good luck with running Ansible through it!). Security is great, unfortunately, it makes our life only harder. That’s why I as a smart guy decided to use agent-based SaltStack. I started with one machine behind the NAT, now I have over 100 agents around the world. In the cloud, as well as in data centers.

Tools used in this episode

git
saltstack
terrafrom

Scenario

Let’s assume that we’re running 5 machines in AWS. To make it fancier, let’s use Gravitons2 - AWS Arm64-based instances. Also, two of our machines will be placed Data Center, ah and one small server in the office. Do you need a diagram for it? Probably, not. However diagrams are awesome, so here we go:


Mondodraw based picture

First problem

In the beginning went I started playing with Salt, there were no pre-built agents. Seems that today it’s the same. What should be done then? Compile from the source - nothing special in the ARM ecosystem. How to do it? It’s very simple:

echo ${host} > /etc/salt/minion_id
curl -fsSL https://bootstrap.saltproject.io -o install_salt.sh
sh install_salt.sh -x python3 -i ${host} git

That’s it! It is that simple. In full picture it looks like that:

Spin new EC2 instance with pre-baked AMI
Provide user-data script
accept agent on salt-master side
Run all needed formulas

And the code:

# hansolo.tf
[...]
user_data = templatefile("./init.sh.tpl", {
    host       = "hansolo"
    saltmaster = "204.0.1.12"
  })
[...]

#!/bin/bash
# install satlstack minion
sudo su
apt-get update
apt-get install git curl
hostname ${host}
echo ${host} > /etc/salt/minion_id
curl -fsSL https://bootstrap.saltproject.io -o install_salt.sh
sh install_salt.sh -x python3 -i ${host}  git
sed -i "/#master: salt/c\master: ${saltmaster}" /etc/salt/minion
# temporary fix on templates rendering
sed -i -e '101,107d' /usr/local/lib/python3.8/dist-packages/salt/utils/templates.py
systemctl restart salt-minion

Tips

As we’re running an agent-based solution, we need to take care of versions of our components. In general, we have two options here:

Update master often
Point install_salt.sh to dedicated tag

In general, I prefer the first option, however, it’s worth remembering.

Basic Salt syntax

In the begining directory structure, in my case looks like that:

root@saltmaster:/srv/salt# tree -L 1
.
├── configs # keeps configs
├── setups # formulas
├── top.sls # top file
└── users # user

Then we can take a look at the basic setup, for example, Jenkins control plane.

{% set plugin_manager_version = '2.12.3' %}

Add Jenkins Repo:
  pkgrepo.managed:
    - humanname : Jenkins LTS
    - name: deb [arch=all] https://pkg.jenkins.io/debian-stable binary/
    - file: /etc/apt/sources.list.d/jenkins.list
    - gpgcheck: 1
    - key_url: https://pkg.jenkins.io/debian-stable/jenkins.io.key

Update Repos:
  pkg.uptodate:
    - refresh : True

Install Jenkins Packages:
  pkg.installed:
    - pkgs:
      - openjdk-11-jdk
      - jenkins
      - nginx
      - certbot
      - python3-certbot-nginx

Set JAVA_HOME:
  file.append:
    - require:
      - pkg: Install Jenkins Packages
    - name: /var/lib/jenkins/.bash_profile
    - text: export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

Get Plugin Manager:
  file.managed:
    - name: /var/lib/jenkins/jenkins-plugin-manager.jar
    - source: https://github.com/jenkinsci/plugin-installation-manager-tool/releases/download/{{ plugin_manager_version }}/jenkins-plugin-manager-{{ plugin_manager_version }}.jar
    - skip_verify: true
    - user: jenkins
    - group: jenkins

Copy plugin file:
  file.managed:
    - require:
      - file: /var/lib/jenkins/jenkins-plugin-manager.jar
    - source:
      - salt://configs/plugins.txt
    - name: /var/lib/jenkins/plugins.txt
    - user: jenkins
    - group: jenkins
    - mode: 644

Generate latest plugin:
  cmd.run:
    - name: java -jar /var/lib/jenkins/jenkins-plugin-manager.jar -f /var/lib/jenkins/plugins.txt --war /usr/share/java/jenkins.war --available-updates --output txt > /var/lib/jenkins/plugins-new.txt
    - runas: jenkins

Install plugin:
  cmd.run:
    - name: java -jar /var/lib/jenkins/jenkins-plugin-manager.jar -f /var/lib/jenkins/plugins-new.txt --war /usr/share/java/jenkins.war --plugin-download-directory /var/lib/jenkins/plugins/
    - runas: jenkins

Copy JaaC:
  file.managed:
    - source:
      - salt://configs/jenkins.yaml
    - name: /var/lib/jenkins/jenkins.yaml
    - user: jenkins
    - group: jenkins
    - mode: 644
    - template: jinja
    - defaults:
        akv_secret: {{ pillar['akv_secret'] }}

Copy Jenkins server settings:
  file.managed:
    - source:
      - salt://configs/jenkinsai.master.config
    - name: /etc/default/jenkins

Restart Jenkins:
  service.running:
    - name: jenkins
    - enable: True
    - watch:
      - file: /var/lib/jenkins/jenkins.yaml

Create cert dir:
  file.directory:
    - name: /etc/nginx/certs
    - user: root

Copy fullchain:
  file.managed:
    - source:
      - salt://configs/wildcard.3sky.fullchain.pem
    - name: /etc/nginx/certs/fullchain.pem

Copy privkey:
  file.managed:
    - source:
      - salt://configs/wildcard.3sky.privkey.pem
    - name: /etc/nginx/certs/privkey.pem

Copy SSL config:
  file.managed:
    - source:
      - salt://configs/options-ssl-nginx.conf
    - name: /etc/nginx/options-ssl-nginx.conf

Add dhparam:
  cmd.run:
    - name: openssl dhparam -out /etc/nginx/ssl-dhparams.pem 2048
    - creates: /etc/nginx/ssl_dhparam.pem

/etc/nginx/sites-available/jenkinsai.3sky.com:
  file.managed:
    - require:
      - pkg: Install Jenkins Packages
    - source:
      - salt://configs/jenkinsai.3sky.com
    - user: root
    - group: root
    - mode: 644


/etc/nginx/sites-enabled/jenkinsai.3sky.com:
  file.symlink:
    - require:
      - file: /etc/nginx/sites-available/jenkinsai.3sky.com
    - name: /etc/nginx/sites-enabled/jenkinsai.3sky.com
    - target: /etc/nginx/sites-available/jenkinsai.3sky.com

/etc/nginx/sites-enabled/default:
  file.absent:
    - name: /etc/nginx/sites-enabled/default

reload_jenkins:
  service.running:
    - name: jenkins
    - enable: True
    - reload: True

reload_nginx:
  service.running:
    - name: nginx
    - enable: True
    - reload: True
    - watch:
      - file: /etc/nginx/sites-enabled/jenkinsai.3sky.com

Yp, that’s true - the whole standalone Jenkins setup in one file. Awesome right?
Then I need to configure my top.sls.

base:
  'hansolo':
    - setups.upgrade
    - setups.init_setup
    - setups.jenkins_main

And execute one simple command:

salt -L 'hansolo' state.apply

That will apply all formulas on my brand-new VM. Cool!

Summary

What I can say about this article? I hope it will be useful if some would like to play a bit with ARM-based Salt agents. It takes me some time to figure out, how all this needs to be set up in a clear and easy-to-use way.

I’m also aware, that I can run it over dedicated AMI. The thing is that my fleet is rather stable, I treat them as pets with a name, etc. That’s related to the specific workflow of one of my clients. We like stable long-living and very stable machines, that’s why we have bare metal servers as well.

That is the first article this year, so Happy New Year!. I’d like to bring my blog to life again and write one post per month. Where will go? We will see.