At my current gig, our group finally got to do hack-a-thon week and I joined a team project that tied together a few of the technologies I've been testing and sporadically using the last five to ten years of my career in automation and quality assurance testing.
Ultimately, the purpose of this hack-a-thon project I joined was to explore the collection, pushing, indexing, and analysis/charting from logs that we collect from our own Docker swarm containers and storage nodes.
It's funny that I haven't used a stack like this until now - having heard about it plenty in the past - but sometimes things come together nicely where all the technologies do mesh well; especially within the orchestration framework of Kubernetes and containerization support of Docker.
Like any stack exploration, there were some parts of ELK I hadn't really used before: Logstash and Kibana. But to have experience in other parts of the stack made the process go a little easier to get a bit more done for week of work with a POC.
Technologies used:
- Elasticsearch (v. 7.9.2)
- Logstash (v. 7.9.2)
- Kibana (v. 7.9.2)
- Java (openjdk version "1.8.0_265")
- Nginx (nginx version: nginx/1.14.0 (Ubuntu) )
- Docker (Docker version 19.03.12, build 48a66213fe)
- Ubuntu 18.04 (gotten from docker hub)
- Kubernetes (10.4.1)
- Artifactory (v 6.8.7)
I had the thought before starting with ELK that Grafana with something like Prometheus would also serve this purpose well, but ultimately my mind was changed and I was really psyched to have revisited and tied all these ideas together for something that could be matured for better log analysis and alerting.
The most rewarding parts of this project was seeing how well stuff could work in one container (not having to assign one process to an individual container - for the time being) and letting the deployed container run in a Kubernetes namespace with plenty of RAM and volume space.
The biggest gotchas were with Kubernetes and making sure that I had the proper kubeconfig file, namespace created and a proper YAML file for creating a service and deployment to a node within my namespace.
