Have no doubt, OpenShift is a great platform for running microservices architectured applications! However, when it comes to Application Performance Management (APM) some may think there’s a lack because no components are coming off-the-shelf in the platform. Indeed this makes sense cause APM is a tricky problem and may be addressed in many mays depending on application technology – and after all microservices architecture is also about picking the right techno for the right problem! 😉 However, this article introduces a quite non-intrusive solution for Spring Boot (and more generally Servlet based) microservices by using Hawkular APM.
Hawkular APM ?
OpenShift provides a built-in monitoring tool called Hawkular. That tool is in charge of collecting metrics from Docker containers through Kubernetes interface and storing, aggregating, visualizing them. The metrics collected are CPU, Memory, Disk and Network usage. Hawkular offers a “black-box” view of container performances but does not deal with application metrics like service performance or distribution of response time through application layers. For that specific cases, Hawkular community is working on another module called Hawkular APM that provides insight into the way an application executes across multiple (micro) services in a distributed (e.g. cloud) environment. Hawkular APM supports agent-based instrumention for Java or API based metrics collection for other languages. It also supports distributed tracing frameworks like ZipKin or OpenTracing. It is not provided as a built-in module in OpenShift but it’s really easy to setup on OpenShift and having Java application being monitored without touching the source code.
OpenShift part setup
Assuming you already have an OpenShift cluster running (through a complete installation, a local
oc cluster up, the Red Hat Container Development Kit or something like Minishift), the first step is to install the Hawkular APM server that will be responsible of storing, aggregating and visualizing the application metrics.
The following commands can be issued in any OpenShift project/namespace of your choice.
[root@rhel72-openshift Development]# oc create -f https://raw.githubusercontent.com/jboss-dockerfiles/hawkular-apm/master/openshift-templates/hawkular-apm-server-deployment.yml secret "hawkular-apm-admin-account" created route "hawkular-apm" created deploymentconfig "hawkular-apm" created deploymentconfig "hawkular-apm-es" created serviceaccount "hawkular-apm" created serviceaccount "hawkular-apm-es" created service "hawkular-apm" created service "hawkular-apm-es" created [root@rhel72-openshift Development]#
It’s just as easy as this! OpenShift template for Hawkular APM creates deployments for server and Elasticsearch backend, services and route for accessing server. Just use the created route within a browser and access the UI after having authenticated with default admin/password created. Here is the result on below screenshot:
Container part setup
The second part of the process is now to prepare a base Docker file for OpenShift that will be used for containerizing our Spring Boot applications from sources. In OpenShift, this process of directly using sources and handling the Dockerization is called Source-to-image. For our Spring Boot / APM purpose, we will start from a base image handling basic Java Dockerization.
The goal of customization is just to retrieve Hawkular APM Java agent and to make it available as a lib within the base image that will later contain our Spring Boot fat jar application. A little trick here is to temporary switch to
root user to chmod correctly the Jar file. Otherwise, JVM that will execute with other user will not be able to read Java agent Jar during the boot. Here’s the DockerFile:
FROM docker.io/fabric8/s2i-java:1.3.6 ENV APM_VERSION=0.13.0.Final ENV APM_AGENT=/libs/hawkular-apm-agent.jar ADD https://repository.jboss.org/nexus/service/local/artifact/maven/redirect?r=releases&g=org.hawkular.apm&a=hawkular-apm-agent&v=$APM_VERSION&e=jar $APM_AGENT # Temporary switch to root USER root RUN chmod 444 /libs/hawkular-apm-agent.jar # S2I requires a numeric, non-0 UID. This is the UID for the jboss user in the base image USER 185
Now that you’ve got this file, you can produce your own base Docker image and store it into a Docker registry for a later use from OpenShift. You can use for example the Docker Hub registry.
$> docker build -t lbroudoux/s2i-java-apm . $>... $> docker push lbroudoux/s2i-java-apm $>...
Finally, in order to use this newly built image into OpenShift and to use it conveniently, we may want to create some ImageStream and Template. We’re are not going in details on how to create those files. The one I have used for testing and demonstrating are just available in my GitHub sample repository in the
Just use these 2 files to declare resources into your OpenShift installation that way:
[root@rhel72-openshift Development]# oc create -f https://raw.githubusercontent.com/lbroudoux/spring-boot-hello/master/openshift/spring-boot-apm-is.json -n openshift imagestream "spring-boot-apm" created [root@rhel72-openshift Development]# oc create -f https://raw.githubusercontent.com/lbroudoux/spring-boot-hello/master/openshift/spring-boot-apm-template.json -n openshift template "spring-boot-apm" created
Bringing it all together!
It’s now time to create a Spring Boot application to check everything is ok. For that, go to the project/namespace of your choice and use the
spring-boot-apm template that is now available. The default of this template uses the sample Spring Boot microservices of my GitHub repository: https://github.com/lbroudoux/spring-boot-hello. You have nothing to change to have something running. After having pressed the “Create” button, build starts (checking out sources, invoking maven build and so on…) and deployment should end-up in a few minutes.
At that time, application is up and you should have a microservice running at
http://spring-boot-app-project.domain.com/api/hello?name=World. Check your exact route configuration through OpenShift console. However if you check Hawkular APM console back again, you’ll did not notice anything new… That’s because application performance management has not been turned on, it is disabled by default. In order to turn it on, go to your application deployment configuration and manage to add the environment variables like below:
In order to get APM up, you have to add these 4 variables:
HAWKULAR_APM_URI: the HTTP URI for accessing your hawkular server (make sur to not use the HTTPS URL, I did not manage to got it working for now),
HAWKULAR_APM_USERNAME: the username for connecting APM server, use default admin user,
HAWKULAR_APM_PASSWORD: the password for connecting APM server, use default admin password,
JAVA_OPTIONS: the JVM options for enabling Java agent, use
-javaagent:/libs/hawkular-apm-agent.jar=boot:/libs/hawkular-apm-agent.jarthat refers to the Hawkular agent previously added to image libs.
Once you’ve hit the “Save” button, deployment is updated, a new container is deployed and the Spring Boot application is now instrumented by Hawkular Java agent. Make few calls to the previous “Hello World” microservice and go check the results in Hawkular APM…. tadam !