Build, test and deploy Apama projects using the Apama Builder Image

Introduction

There’s a new Docker image in the Docker Store for Software AG’s 10.3 release called Apama Builder. It differs from the Apama Correlator image in that it can be used to build Apama projects into new images via multi-stage builds. In this blog we will explore how leveraging the Apama Builder along with other common tools, we can easily create an Apama application in a container that is self-testing, self-deploying, and subject to continuous integration.

The Apama Builder image contains a suite of tools that enable the creation of images derived from the Apama Correlator image containing tested and deployed Apama applications. It provides the usual tools such as engine_deployengine_inject, the correlator and PySys used during the normal workflow. Additionally, it contains tools such as Ant, Java and Python and can easily be extended to contain any elements required by your build by installing them as part of the Docker build process.

To help explain the process and to demonstrate how Continuous Integration (CI) can be implemented using Travis, this article will refer to a project in GitHub that can be found here:

https://github.com/CaribouJohn/ApamaBuilderExample

This uses a simple Java plug-in which needs to be built from Java sources, combined with an Apama application that uses the plug-in and a PySys test that will verify the application. The key to all of this is the authoring of a Dockerfile for your project which uses the Apama Builder image to produce the desired image.

Creating the Dockerfile

For your own project you will need to author a Dockerfile to produce the desired application image.

The above example uses substitution variables to define the image names used to generate the final image. It defaults the version to 10.3 and the images to the Docker Store versions for convenience. The various steps required to build, test and deploy the application need to be inserted into the Dockerfile.

Alternatively if you use Software AG Designer (available on Windows), there’s a new context menu in 10.3: Apama > Add Docker Support. This will add a Dockerfile to the top level of your project. For the majority of EPL-only Apama projects, the default Dockerfile this creates is all you need. This will correctly sequence injections and event files at startup, and includes all your configuration, with no changes required. The Designer Dockerfile will be similar in structure to the example above, and will default the images and version.

Building the image

Once you have added a Dockerfile, you can use the Docker client to build an image from the project. On Windows, you can use the Docker client for Windows and either a local Linux Docker installation or a remote Docker server. It can also be done from Linux as part of a CI pipeline.

By default, this will use the public Docker Store images for Apama:

However, you can change this by using the build_arg parameter

The advantages of this process are that steps such as the build, the tests and deployment will all cause the process to fail if they fail. This can be used to ensure that the final image is fully working before it is generated. Additionally, this method of multi-stage building is efficient because it will discard the intermediate layers which are not included in the final image.

Apama project

Regardless of Windows or Linux operating system, and whether or not Designer is used to create the project, you can automate the apama-builder image in a GitHub project repository and integrate it with TravisCI using the following example as a guide:  https://github.com/CaribouJohn/ApamaBuilderExample

There is no need to install Software AG Apama locally on your machine, and you do not need a Java JDK (or JRE) installation locally either, as these are only needed inside the container where the Apama correlator will run the sample, and the Docker image already includes them.

Follow the instructions in the README.md in the GitHub repository, which cover all the steps required to build, deploy and run the sample in the container. They are more or less repeated here for convenience:

1) From the command line, navigate to a chosen working directory and git clone (or svn checkout) the URL which can be copied from the “Clone or Download” control on the GitHub URL above.

Or:

They both have a similar effect. The svn checkout command creates an ApamaBuilderExample.git folder under which is a trunk folder, while git clone creates an ApamaBuilderExample folder which contains the same files as the Subversion trunk folder. Navigate into that, and build (and test) the image with:

2) Run the image, picking a number for a port, in this case 40000, for the container which you can then access from another command prompt, and another, 15903, for the correlator inside it:

When the container runs, the complex.yaml file in the project causes the correlator to be initialized with the complex_plugin.jar built and copied into the APAMA_WORK directory, and the copied ComplexPlugin.mon (see the Dockerfile).

3) In another prompt, use

to confirm the presence of the running container, and discover its name and container identifier. Doing

will show the same output as on the screen in the first prompt, and is comparable to ComplexPluginSample.txt reference. The single PySys test is pretty rudimentary and only checks that after the injection, there are no ERROR or FATAL loggings. It doesn’t compare the output from the plug-in with the text file reference, for example.

4) Finally, stop the container cleanly with

Linking to Travis for continuous integration.

Linking the project to Travis ( https://docs.travis-ci.com/user/tutorial/ ) means that whenever changes are made to the project in GitHub, the whole build, test and deploy process is triggered automatically. Failure at any stage will cause the ci process to fail which can then trigger actions to identify what commit caused the failure. The process of adding Travis is as easy as registering with the site, linking your Github repository and adding a configuration file to the project at the top level.

The example above triggers Travis to use docker to build the project using the supplied script and adds a specific command to do a docker login so that the images can be pulled for the build. The variables named in the before_install line are defined on the Travis website and are not shown in logs or output from the build. See https://docs.travis-ci.com/user/docker/ for more details.

If you want a quick start for your own project you can fork the example repository into your own GitHub account. This appears on the surface to take Travis configuration with it, but it requires further setup. Create a Travis account synchronized with your GitHub account, and activate Travis for the forked project.

Note that the grey/green link to Travis from the GitHub project page is linked to the original project’s Travis setup, and is not instantly useful in the forked repository. Edit the URL behind it (accessing Readme.txt via the GitHub UI and the edit button may be the most convenient way) to be the URL for your Travis CI build for the project, and commit. This edit is likely to be to the two instances of username and project name in the line at the top of the file, which starting with the encoding “[![Build Status]”.

Go the the project in Travis, and under “More Options” select Settings and add environment variables called DOCKER_PASSWORD and DOCKER_USERNAME (see .travis.yml) and assign values to them (with “Display value in build log” turned off). The Docker username and password just need to be for any Docker account (create one if necessary) as this account will be sure to have access to the (public) Software AG Apama 10.3 Docker images, including apama-builder.

Clicking the “Restart build” button for the project in Travis will now build and test in the copied repository, and the build status link should now be for your project (if you edited it correctly) and be grey/green build/passing. Follow the link to the Travis log to confirm things (URL, date/time when the build and test ran, etc). Another way to trigger a Travis build is to commit a change to one or more files in the repository.

Working on the copy in the local file system, edit the EPL and Java plug-in code, add further modules, event files, extra plug-ins, configuration etc. as desired. Make sure the project basically does what you want when it runs. Also edit and add to the PySys tests, and debug those until they all run too. Submit the changes as an update to the new repository on GitHub, and see if the subsequent Travis build succeeds and passes.

Debug as required, commit more changes, add more features, and let the spec.creep and code bloat begin!

Additional tips

Consult Travis, Docker, Apama and Git documentation on other forums for full guidance.

Useful Commands

To check which version of Java the container is running, run

docker ps

as per instructions above to confirm that the container is running, and note its randomly assigned name, under the NAMES column, and do

For 10.3, it will show you OpenJDK version “1.8.0_181”.

You can also do

to reveal the full version number of Apama which is installed in the container.

If you do happen to have a local Software AG installation, and source apama_env (in the Apama/bin directory), then while the container is running (and outputting status lines) you can use a separate connection to

to inject the EPL again and see the same output again. Use

to view all the output. To inject different EPL files to the same correlator in the container, though, even ones existing in the local Apama/samples/correlator_plugin/java folder, you would first need to include them in the image build.

Finally, instead of

or

doing

will stop the correlator within the container, and since the correlator is running as PID 1 in the container, the container will also stop.