Exploring ELK (Elastic) Stack for hack-a-thon

At my current gig, our group finally got to do hack-a-thon week and I joined a team project that tied together a few of the technologies I've been testing and sporadically using the last five to ten years of my career in automation and quality assurance testing.

Ultimately, the purpose of this hack-a-thon project I joined was to explore the collection, pushing, indexing, and analysis/charting from logs that we collect from our own Docker swarm containers and storage nodes.

It's funny that I haven't used a stack like this until now - having heard about it plenty in the past - but sometimes things come together nicely where all the technologies do mesh well; especially within the orchestration framework of Kubernetes and containerization support of Docker.

Like any stack exploration, there were some parts of ELK I hadn't really used before: Logstash and Kibana. But to have experience in other parts of the stack made the process go a little easier to get a bit more done for week of work with a POC.

 Technologies used:

1. Elasticsearch (v. 7.9.2)
   
2. Logstash (v. 7.9.2)
   
3. Kibana (v. 7.9.2)
   
4. Java (openjdk version "1.8.0_265")
   
5. Nginx (nginx version: nginx/1.14.0 (Ubuntu) )
   
6. Docker (Docker version 19.03.12, build 48a66213fe)
   
7. Ubuntu 18.04 (gotten from docker hub)
   
8. Kubernetes (10.4.1)
   
9. Artifactory (v 6.8.7)
   

I had the thought before starting with ELK that Grafana with something like Prometheus would also serve this purpose well, but ultimately my mind was changed and I was really psyched to have revisited and tied all these ideas together for something that could be matured for better log analysis and alerting.

The most rewarding parts of this project was seeing how well stuff could work in one container (not having to assign one process to an individual container - for the time being)  and letting the deployed container run in a Kubernetes namespace with plenty of RAM and volume space.

The biggest gotchas were with Kubernetes and making sure that I had the proper kubeconfig file, namespace created and a proper YAML file for creating a service and deployment to a node within my namespace.

 

Using Ansible with vCenter and SSH calls

Can't believe it took me so long to explore Ansible for configuration management and automation.

I heard about it back in 2014 and tested against it since we used it to install our micro services framework. 

I used a good intro video tutorial to try making a vCenter connection and listing hosts.

Problems encountered were the YAML syntax, using local versus remote connections, SSH with sudo, small syntax errors and using the ansible.cfg and hosts file under /etc/ansible, which I created manually

Initial vCenter Tests:

Here's some initial code for connecting to vCenter:

cat auth_vcenter.yml
---
  - hosts: localhost
    vars:
      user_readme: 'Welcome to vCenter tests'
    tasks:
    - debug:
        msg: "Starting test aginst vcenter"
    - name: Including Secret Environment Items
      include_vars:
        file: secret67.yml
        name: secret
    - name: vcenter login
      uri:
        url: "https://{{secret.vcenter}}/rest/com/vmware/cis/session"
        force_basic_auth: yes
        method: POST
        user: "{{secret.username}}"
        password: "{{secret.password}}"
        status_code: 200
        validate_certs: no
      register: login
    - name: Get hosts from vCenter
      uri:
        url: "https://{{secret.vcenter}}/rest/vcenter/host"
        force_basic_auth: yes
        validate_certs: no
        headers:
          Cookie: "{{login.set_cookie}}"
      register: vchosts
    - debug: var=login
    - debug: var=vchosts

This works with a YAML secrets file which for initial testing is fine but passwords should be encrypted so it's not a longterm solution:

cat secret67.yml
---
username: administrator@vsphere.local
password:
vcenter: 10.117.180.10

Initial SSH Tests:

Here's some other code for trying out an automated SSH connection with some initial commands

One that that got me was authentication errors so to make the login process work I had to run:

ssh-copy-id -i ~/.ssh/id_rsa.pub username@10.117.155.123

cat testone.yml
---
  - hosts: 10.117.155.123
    vars:
      user_readme: 'Welcome to this machine!!!! Hey good!'
    tasks:
    - debug:
        msg: "Starting test against server"
    - name: List contents
      command: ls -lta
      register: out1
    - name: Touch file
      file:
        path: $HOME/test_server.txt
        state: touch
      register: out2
    - name: Create target directory
      file: path=~/testit state=directory mode=0755
      register: out3
    - name: simple file try
      copy:
        dest: ~/testit/README.txt
        content:  " {{ user_readme }} "
      register: out4
    - debug: var=out1
    - debug: var=out2
    - debug: var=out3
    - debug: var=out4

Hosts and config Files:

For the SSH testing these initial files seems to work for me. Very brief and to the point.

$ cat ansible.cfg
[defaults]
remote_user=

$ cat hosts
all:
  vars:
    ansible_ssh_user=
    ansible_ssh_pass=
  hosts:
    10.117.180.10

Summary:

To watch the video, setup Ansible on Mac, and get these tests working should take a day or two, max, in my opinion.

Next steps are to see about using Ansible for further automation testing.