Exploring ELK (Elastic) Stack for hack-a-thon

At my current gig, our group finally got to do hack-a-thon week and I joined a team project that tied together a few of the technologies I've been testing and sporadically using the last five to ten years of my career in automation and quality assurance testing.

Ultimately, the purpose of this hack-a-thon project I joined was to explore the collection, pushing, indexing, and analysis/charting from logs that we collect from our own Docker swarm containers and storage nodes.

It's funny that I haven't used a stack like this until now - having heard about it plenty in the past - but sometimes things come together nicely where all the technologies do mesh well; especially within the orchestration framework of Kubernetes and containerization support of Docker.

Like any stack exploration, there were some parts of ELK I hadn't really used before: Logstash and Kibana. But to have experience in other parts of the stack made the process go a little easier to get a bit more done for week of work with a POC.

 Technologies used:

1. Elasticsearch (v. 7.9.2)
   
2. Logstash (v. 7.9.2)
   
3. Kibana (v. 7.9.2)
   
4. Java (openjdk version "1.8.0_265")
   
5. Nginx (nginx version: nginx/1.14.0 (Ubuntu) )
   
6. Docker (Docker version 19.03.12, build 48a66213fe)
   
7. Ubuntu 18.04 (gotten from docker hub)
   
8. Kubernetes (10.4.1)
   
9. Artifactory (v 6.8.7)
   

I had the thought before starting with ELK that Grafana with something like Prometheus would also serve this purpose well, but ultimately my mind was changed and I was really psyched to have revisited and tied all these ideas together for something that could be matured for better log analysis and alerting.

The most rewarding parts of this project was seeing how well stuff could work in one container (not having to assign one process to an individual container - for the time being)  and letting the deployed container run in a Kubernetes namespace with plenty of RAM and volume space.

The biggest gotchas were with Kubernetes and making sure that I had the proper kubeconfig file, namespace created and a proper YAML file for creating a service and deployment to a node within my namespace.

 

Using Ansible with vCenter and SSH calls

Can't believe it took me so long to explore Ansible for configuration management and automation.

I heard about it back in 2014 and tested against it since we used it to install our micro services framework. 

I used a good intro video tutorial to try making a vCenter connection and listing hosts.

Problems encountered were the YAML syntax, using local versus remote connections, SSH with sudo, small syntax errors and using the ansible.cfg and hosts file under /etc/ansible, which I created manually

Initial vCenter Tests:

Here's some initial code for connecting to vCenter:

cat auth_vcenter.yml
---
  - hosts: localhost
    vars:
      user_readme: 'Welcome to vCenter tests'
    tasks:
    - debug:
        msg: "Starting test aginst vcenter"
    - name: Including Secret Environment Items
      include_vars:
        file: secret67.yml
        name: secret
    - name: vcenter login
      uri:
        url: "https://{{secret.vcenter}}/rest/com/vmware/cis/session"
        force_basic_auth: yes
        method: POST
        user: "{{secret.username}}"
        password: "{{secret.password}}"
        status_code: 200
        validate_certs: no
      register: login
    - name: Get hosts from vCenter
      uri:
        url: "https://{{secret.vcenter}}/rest/vcenter/host"
        force_basic_auth: yes
        validate_certs: no
        headers:
          Cookie: "{{login.set_cookie}}"
      register: vchosts
    - debug: var=login
    - debug: var=vchosts

This works with a YAML secrets file which for initial testing is fine but passwords should be encrypted so it's not a longterm solution:

cat secret67.yml
---
username: administrator@vsphere.local
password:
vcenter: 10.117.180.10

Initial SSH Tests:

Here's some other code for trying out an automated SSH connection with some initial commands

One that that got me was authentication errors so to make the login process work I had to run:

ssh-copy-id -i ~/.ssh/id_rsa.pub username@10.117.155.123

cat testone.yml
---
  - hosts: 10.117.155.123
    vars:
      user_readme: 'Welcome to this machine!!!! Hey good!'
    tasks:
    - debug:
        msg: "Starting test against server"
    - name: List contents
      command: ls -lta
      register: out1
    - name: Touch file
      file:
        path: $HOME/test_server.txt
        state: touch
      register: out2
    - name: Create target directory
      file: path=~/testit state=directory mode=0755
      register: out3
    - name: simple file try
      copy:
        dest: ~/testit/README.txt
        content:  " {{ user_readme }} "
      register: out4
    - debug: var=out1
    - debug: var=out2
    - debug: var=out3
    - debug: var=out4

Hosts and config Files:

For the SSH testing these initial files seems to work for me. Very brief and to the point.

$ cat ansible.cfg
[defaults]
remote_user=

$ cat hosts
all:
  vars:
    ansible_ssh_user=
    ansible_ssh_pass=
  hosts:
    10.117.180.10

Summary:

To watch the video, setup Ansible on Mac, and get these tests working should take a day or two, max, in my opinion.

Next steps are to see about using Ansible for further automation testing.

Pycharm IDE. How to fix missing source directories in Project view. Doesn't show file or folder structure.

Possibly due to a git merge between two of my computers on the same GIT project I lost the project structure. <

I was able to research this and re-add the content root under Preferences > Project > Project Structure. I wasn't as intuitive as I initially thought but here is the dialogue where I add it.

All fixed!

Being more pythonic with the shortcut conditional : A if C else B

I had these two blocks of "if else" code that I found too crude.

test_case_data = None
test_case_data2 = None

if test_already_exists(tests_object_list, test1_name) is not None:
  print(">>>> Test 1 already exists. skipping the add")
else:
  print(">>>> Test 1 doesn't exist. adding")
  test_case_data = add_test_case(api_conn, int(project_id), test1_name)

if test_already_exists(tests_object_list, test2_name) is not None:
  print(">>>> Test 2 already exists. skipping the add")
else:
  print(">>>> Test 2 doesn't exist. adding")
  test_case_data2 = add_test_case(api_conn, int(project_id), test2_name) 

 

I took the code and applied Guido's "ternary solution," Link here.

 
All of the above code became three-four lines of code and I was able to apply a one-line Pythonic truthy and falsy evaluation on the return object. Very lambda-ish. 

[Correction: What I had earlier with data = func_call if True else other_func_call, was incorrect. Corrected below. So it's really A if A (when A is not None) else B (which in this case creates a new test case object.]

for current_test in tests_lists:
  test_exists_data = test_already_exists() # Note returns None or or existing test data
  test_case_data = test_exists_data if test_exists_data else add_test_case()

  ...further code that doesn't contribute to A if C else B 

 

As a side note: This same elegance in code I also found with the use of the "with," statement using the Context Manager. But for this, a lot more compact, readable and Pythonic.

When the default pip install won't work. Using pip directly with a github repo

I work at a company where we cannot publish some (or all) Python modules to pyPI.

95% of the time before last week I would use:

pip/pip3 install [module name]

to install modules that I needed.

If that doesn't work and you have access to the git repo, another command that might be successful is installing directly from the repo. Here's an example:

pip3 install git+https://github.com/[username]/repo

And if you've downloaded the repo using git clone and there's a setup.py, which should be standard for a properly maintained module - you can run:

pip3 install -e .

Remember, sometimes you need to work around less-standard, less user-friendly modules so thankfully there are other pip options that can hopefully serve your needs.

Trying out my Python 3 skills on HackerRank

I've been using HackerRank to get my Python coding chops up a little better then where they've been before.

Tried a medium challenge on HackerRank called Piling Up

My solution isn't as elegant as others but it works and constantly reduces the size of the cube size list that need to be rearranged from horizontal to vertical where the block size needs to be equal or smaller than the block below it.

for tests in range(int(input())):

    cube_length = int(input())

    list_of_cubes = list(map(int, input().split()))

    last_value_placed_on_stack = 0

    all_items_on_placed_on_stack = False

    cannot_stack_anymore = False

    a = 0

    z = int(len(list_of_cubes)) - 1

    while a < int(len(list_of_cubes)):

        if cannot_stack_anymore:

            break

        if int(len(list_of_cubes)) == 1:

            if int(list_of_cubes[a]) <= last_value_placed_on_stack:

                all_items_on_placed_on_stack = True

                break

            else:

                break

        while z > 0:

            if list_of_cubes[a] >= int(list_of_cubes[z]):

                if list_of_cubes[a] <= last_value_placed_on_stack or last_value_placed_on_stack == 0:

                    last_value_placed_on_stack = list_of_cubes[a]

                    list_of_cubes.pop(a)

                    z = z - 1

                    break

                else:

                    cannot_stack_anymore = True

                    break

            if list_of_cubes[z] >= int(list_of_cubes[a]):

                if list_of_cubes[z] <= last_value_placed_on_stack or last_value_placed_on_stack == 0:

                    last_value_placed_on_stack = list_of_cubes[z]

                    list_of_cubes.pop(z)

                    z = z - 1   

                    break

                else:

                    cannot_stack_anymore = True

                    break

    if all_items_on_placed_on_stack:

        print("Yes")

    else:

        print("No")

Computing Fibonacci in Go

Had a coding whiteboard one of my first interviews back in early April.

It's nice to use Go Playground to prove it out and validate it after you get home.

For the whiteboard I had to do it iteratively. I added the recursive func after the fact

Here's the Go Playground link: https://play.golang.org/p/KDQaQlq8ONl

package main

import (
"fmt"
)

func main() {
fmt.Println("Hello, playground. Test for computing Fibonacci numbers")

computeFibonacci(10)
computeFibonacciRecursively(10, 0, 1)
}

func computeFibonacci(numberOfNumbers int) {

penultimate := 0
ultimate := 1

fmt.Printf("\nNext number (iterative): %d", penultimate)
fmt.Printf("\nNext number (iterative): %d", ultimate)

for i := 2; i < numberOfNumbers; i++ {

currentSum := penultimate + ultimate
fmt.Printf("\nNext number (iterative): %d", currentSum)
penultimate = ultimate
ultimate = currentSum

}

}

func computeFibonacciRecursively(numberOfNumbers int, penultimate int, ultimate int) {

if numberOfNumbers == 0 {
return
} else {
fmt.Printf("\nNext number (recurse method): %d", penultimate)
numberOfNumbers--
computeFibonacciRecursively(numberOfNumbers, ultimate, penultimate+ultimate)

}

}

Powershell problem : With remote PSSession, Running EXE in path with spaces and getting back response using Write-Output

I think some of the most trouble I've recently had in coding Powershell has been trying to run an executable on a remote Hyper-V VM that was located under Program Files and then actually getting the response back properly so I could iterate though the results.

The first trickiness was with running the command properly when there were spaces in the path.

In double quotes you have to use the ampersand (&) and then write the path to the EXE in single quotes and then at the end of the string before the closing quotes character, append an argument.

For example :

$commandLocation = "& 'C:\Program Files\....\application_name.exe' argument"

This is the proper syntax that you can then pass to an Invoke-Expression that's inside a ScriptBlock called by Invoke-Command. Invoke-Expression is a very helpful commandlet to run EXEs, when, for example, running Powershell on a remote machine.

The second issue was understanding how I could use Write-Output to pipe the remote call output back a Powershell variable within the ScriptBlock that I could then use in the calling function.

I read that you could use 4>&1 at the end of the ScriptBlock but never found it useful in my case.

In my case, I just had a ScriptBlock assigned to a variable and then ran the ScriptBlock using an Invoke-Command on a separate line.

There was some funkiness with WriteHost appending to the invoke-command so you have to disable that and then return the pure unadulterated Invoke-Command Result that I assigned to a variable.

Then I could iterate over the lines in the method that called the method below.

Here's the example code :

Function RunCommandOnRemoteMachine([string] $blahParameter, [string] $remoteMachineAddress)
{

    $defaultBlahCommandLocation = "& 'C:\Program Files\...\blah.exe' "

    WriteLogAndConsole "Default blah directory : $defaultBlahCommandLocation"
 
    $fullCommand = $defaultBlahCommandLocation +  $blahParameter

    WriteLogAndConsole "Full command with arguments being run : $fullCommand"

    $scriptToExecute = { param($passedCommand)
        begin {
            Write-Host "Start of command:"
        }
        process {
            $consoleOutput = Invoke-Expression -Verbose -Command $passedCommand
        }
        end {
            Write-Host "Output from remote command:"
            Write-Host $consoleOutput
            Write-Host "************"
            Write-Output $consoleOutput
            }
     }

    $blahCommandResult = Invoke-Command -Session $global:session -Verbose -ScriptBlock $scriptToExecute -ArgumentList $fullCommand

    #Write-Host "Result of blah argument command: " $blahCommandResult

    return $blahCommandResult

}

Powershell New-PS Drive using Administrator credentials. User name or password is incorrect

Sometimes you forget the obvious when your doing scripting.

Although not recommended, I was using powershell to create a new psdrive session using the Administrator.

Problem is that with just Administrator and the password, powershell was complaining about a bad username and password combination.

Since the VM was connected to a development domain I wasn't qualifying the Administrator name therefore confusing powershell.

I add the computername\Administrator and tried again. Much better success

Elasticsearch analogies

I work a lot with Elasticsearch.

At my job we have migrated from using SQL Server to Elasticsearch for log storage.

Sometimes, when you're learning new grammar for a new technology you're working on it's nice to have metaphors or similes to use when learning the component parts.

One of the best similies I've heard of for what Elasticsearch represents is that it's like an index at the end of a book. Additionally, Elasticsearch by many, even those who work at Elastic, consider it not a DB.

If you start using Elasticsearch a lot more you'll begin to work with types, templates and mappings.

A type could be analogous to a regular SQL table.

Mappings could be similar to a a table column that stores a field a certain way that could be something like a string, int, varchar.

How mappings and settings are created and loaded has changed from Elasticsearch 1.x to 2.x+. Now that we use Elasticsearch 2.x we have to load template files that are used when new indices are created. They combine settings and the mappings for the fields that get parsed out and placed into an index.

Thanks to this website for giving me those Aha! analogies.

http://swops.com/blog/elasticsearch-mappings-and-templates

Rebuild project of 18 year old Mansfield Toilet

I just completed a toilet rebuild last weekend. The toilet began leaking a year or two ago; especially during warm weather.

Although I wasn't working the whole time the total time from start to finish was a half day. I did initial work and investigations starting Friday night, I let the tank dry out overnight, and I finished about 1 PM the following day having started around 10 AM with the installation of the new parts on the tank.

Actual time working was probably around 4 hours.

Work entailed replacing the fill valve, flush valve, new bolts attaching the tank to the bowl, new shutoff valve and supply line too.

New shutoff valve

New Supply Line to Tank

All parts replaced in dry tank

Tank full with water

All parts were ordered from Mansfield and the cost was just under 50 bucks. I went to Home Depot for the new shutoff valve and supply line.

Total cost was about 60 bucks.

Issues encountered after the replacement was slight leaking from one of the bolts, which I had to tighten down more to prevent further leakage.

As mentioned on my Twitter post, I'd say this home project was a 2-3 out of 10.

Perl : subroutine redefined, prototype mismatch

I was refactoring my code and placing some common methods into a base module in my Perl lib directory.

Unfortunately some of my new method names had name collisions with possible subroutines in other packages.

I was able to rename the methods that I was exporting for use in my PL files and the errors went away.

Python Indentation Error

I've only been using Python for a year.

Before that, my focuses were Java and Perl.

An interesting error I came across, since Python has stricter criteria for formatting was: "IndentationError: expected an indented block"

The code in question:

The error might be considered a red herring since the problem with the else: statement was that there was no real code to run after the if statement.

Once I added a logging statement:

The error went away.

Setting up a new hyper-v Windows 2012 server to be pingable and allow RDP connections

At my current company we are needing to test deployments not only to Linux but Windows too.

To quickly have preconfigured systems come up we use virtualization (Hyper-V) along with other technologies like Vagrant and Ansible with other configuration and testing scripts like Python and/or Powershell.

I just setup a new Windows 2012 R2 Machine in Hyper-V using a ISO image.

By default, some features aren't enabled to make my testing easier. For example, to be able to ping the machine or RDP into it for GUI related tasks.

To enable Windows to respond to a ping you have to go into Windows Firewall with Advanced Security > Inbound Rules and enable File and Printer Sharing (Echo Request - ICMPv4-In) by right clicking on the rule name.

For remote desktop connections I had to do two things:

1) Type SystemPropertiesRemote and select "Allow remote connections to this computer."  I also deselected the "Allow connections only..." beneath it

2) I ran this command: netsh advfirewall firewall set rule group="Remote Desktop" new enable=yes

If for some reason I disabled Allow remote connections again I would have to do the two steps again to get RDP to allow my remote client to connect.

Caveats/setup notes for running pytest with Jenkins on Windows 2012 R2 server

I've been writing test code in Python for the last few weeks.  Although it's been great to have Perl and Java as my predominant testing languages, it is nice to try a new scripting language on for size.

I first used Python on Linux and this last week - due to testing requirements - I needed to use it on Windows.

Python Installation :

Installing Python is pretty straightforward. Go to python.org. and download it. Installation is very easy.

After the install, I needed to add the Python to the System Path.

Pip and Pytest Installation :

Pip seemed a little different to install -vs- Linux. With Linux I could install with yum and call pip directly. To use Pip on Windows I had to call it through Python first

Installing Pytest then required me calling Python to call pip and then install PyTest. See below:

python -m pip install pytest

With Pytest installed I needed to incorporate it into my Python test class:

1) Import PyTest in the Python script using : import pytest.
2) Use an assert statement for test results.
3) Place the test statements in a test method (def testSomething(): )  For the tests to work correctly with JUnit output I needed to put the asserts in a test method.
4) Create JUnit output for Jenkins. When I did run my basic test class without outputting to an XML file it seemed to work just fine, but if I wanted to get results to Jenkins I needed to get an XML file for Jenkins to read.
5) Once I solved the above issues I needed to make sure a JUnit XML file was created. This is how I called it below:

python -m pytest scripts\CheckMSIBuildStatus.py --junitxml=results.xml

Running a java test class in a maven project on the command line

Sometimes I need to write a quick one-off Java test class against a developer's maven project or class within a maven project.

Easy to do locally on my Mac but my Mac is not a production environment.

In this case this necessitates that I run this test class on a remote machine (UNIX or Windows) where I'd rather not setup a full development environment and most likely have command line access.

Given a minimal production environment the requirements are : I can compile the developer's class, supporting classes and libraries, my test class, and then test the resulting service; in this case a small micro service the developer created.

Pre-requisites, in my case, needed on the remote machine are :

1) Java

2) GIT

3) Maven

4) If needed, the service deployed and running under systemd. 

Once I have the first three components installed, I clone the GIT repository and then try to build the maven project, usually with a mvn clean install -Dmaven.test.skip=true.

If I have any dependencies to this project, I try to use a library manager referenced in a settings.xml file under .m2 (like Artifactory) to make the project compile.

If this succeeds, I try to compile my stand-alone test class. Since I'd rather not run the developer's unit integration tests I run this command :

mvn install -DskipTests

This allows my class to compile. If this works I then I run this command :

mvn exec:java -Dexec.mainClass="com.[path to my test class].[my test class name]" -Dexec.classpathScope=test -e

Running this command allowed me to launch the Java driver program that then - in this case - created a new instance of the developer's monitor class that would run in it's own Thread through Runnable, which I could subsequently test.

GIT : getting the lastest changes from your upstream master when you have no changes in your repo

I clone a lot of our micro service repositories locally into my own branch, but then really don't add to them. A few months pass and then I need to re-sync.

GIT seems to make it a little harder to do it compared, to say, Perforce, another source control program. At a high level you have to get the latest changes from your upstream master, that you forked from earlier, fetch and then merge.

I've detailed the more individual steps below, and what I encountered. To start, you may not even have your upstream master defined!

Here are the high level command line steps I took on my Mac Book Pro.

git fetch upstream

   I got > fatal: 'upstream' does not appear to be a git repository. So here upstream is not defined.

git remote -v

   With this command I noted that no upstream was listed. Guess I have to create it then.

git remote add upstream http://github[remainder of URI path]/[repository name].git

   This command allowed me to define my upstream 

git remote -v

   Verification step that upstream remote was now defined

git fetch upstream

   This got me the latest changes, but stored it locally in a repo called upstream/master.

git log

   This verified that the latest changes - in my local master repo - were still from a few months ago so a couple more steps needed.

git checkout master

   Although not needed in my case, just a verification I was on the right branch. This command will move you to the master branch in case you might be on another branch.

git merge upstream/master

   This command finally synced my local unchanged branch to the latest changes in the HEAD, aka upstream.

git log

   With git log I saw the latest changes were from today.


Hope this helps.

Pad Kee Mao Gai

I created a separate cooking blog...

The blog writeup will move permanently here >

http://andresfernandezcooking.blogspot.com/


*************



I've been cooking Thai cuisine the last 2 years and have learned quite a few dishes.

I've done Lad Na, Pad Thai, Masaman, Pad Prik Khing, and other recipes, but wanted to try something new with Thai basil.

I order a lot from importfood.com, and wanted to use their recipe on the website but did a little more searching to see what the recipe would look like and see the possible variations.

I tried this recipe here : Alosha's Kitchen and with a couple of small modifications, I thought it was a very good recipe.

To keep the heat down to medium, I used only three Thai chilies along with 12 ounces of wide noodles and a pound of chicken. I did use a whole small red onion and a red and yellow pepper; so twice the amount of onion.

If you're using 12 ounces of wide noodles I would recommend a 1/4 cup of fish sauce, slightly a little less of the golden mountain and thick soy sauce and directly mix it with the chicken before putting it in the pan. I also mix my palm sugar with water for easy use and store it in the fridge so I can just pour it on the dish as I'm creating it.

Here's a photo of the finished dish :

Comparing UNIX networking commands : netstat, lsof and ss

I've realized recently there's a lot more availability in UNIX commands to look at your network (open ports, etc...).

Some sample commands that do the same thing, which I'll investigate later :

netstat -a -p  | grep 13100

ss -all | grep 13100

lsof -n -i4TCP | grep 13100

These three commands do a lot of similar things, albeit with some minor tweaks; and then pipe it to grep to look at a specific port.

Currently, my favorite for information and formatting is lsof.

I'll write more later...

The simplest resolution or explanation is usually the most common solution

In engineering, we have to diagnose and fix issues all the time.

In my case I've found that the problems I encounter when I find a build broken, a class that won't compile, or a problematic configuration that won't make a application run, is most likely due to a slight misconfiguration, misspelling, or not reading the directions thoroughly.

In other words, the easiest solution is usually the most common fix to a problem you've encountered.

Maybe this is why we have Occam's Razor as a principle?