# Solo and System Groovy for Jenkins source: https://developer.mastercard.com/blog/solo-and-groovy-jenkins/index.md Authors: Kenny Hung. Real Time Payments Dev-ops Instant payment systems are a part of the critical infrastructure of a country: they offer a mechanism for banks within a country to transfer money to each other immediately. Typically, these systems are used to enable payments to be made within a few seconds 24 hours a day, seven days a week, 365 days a year. The Real Time Payments service offered by Mastercard is an example of such a system. The Clearing House alone has made over 40 million transactions totalling more than 20 billion USD just in the third quarter of 2022. The Real Time Payments service is used by central banks, including The Clearing House in the US, the P27 system used by the Nordic region, and the Cámera de Compensación Electrónica in Perú. Our team's purpose is to provide tooling support for our large team of developers during the development process. We have internally deployed instances of Github for managing our code, Jira for managing our teams, and Confluence for managing our documentation, as well as systems and automated processes to allow our engineers to build, run, test and deploy their code. The development of such a large, complex system has necessitated a strong team to support our developers in deploying our code to allow testing of new features rigorously before we release them to production systems. This has meant that our team have developed a comprehensive suite of tools accessed by our colleagues, a large, international team of developers, testing and deploying our code thousands of times every week. Synopsis Having decided to use Groovy to do most of the work in our Jenkins instances, we needed a facility to trigger Groovy code on Jenkins Agents from the Controller that allows us to pass data in both directions, and to take advantage of third-party libraries available to the JVM. This allows the following: Keep Groovy code on the Jenkins Controller to a minimum in terms of computational power required and complexity. Develop Groovy code to be run on the Jenkins Agent with a process similar to any other software development project. Allow data to be passed from the Jenkins Agent to the Controller (so that code running on the Agent can get context about the pipeline that it's running in), and back again (so that the Controller can present the results of our code to the end user). Use third-party libraries and SDK available to interface with other systems, like AWS, or databases. What is Jenkins Jenkins is an open source automation server. It is used extensively in the industry, and it is what we use on the Real Time Payments project at Mastercard to expose an interface to our developers to deploy their code into working test environments in AWS. We also use it to run automated tests against code when our developers push and merge their code: this enables us to protect against regressions in behaviour and unintentional bugs in an automated manner, and nudge the quality of our code forever upwards. In addition, we use Jenkins to automate the cleaning up of test environments, creation of new repositories, and general reporting and housekeeping of our test environments estate. More generally, anything we find that we or our engineers are doing regularly, we try and automate using Jenkins. What is Groovy? Groovy is a JVM-based language that has been in use since around 2007, but first implemented in 2003. From a bird's eye view, Groovy was designed originally to overcome some of the perceived shortcomings of Java: in particular, the Grails web framework took as inspiration the Ruby on Rails framework that was gaining traction at the time. It's fairly clear that a lot of the innovations in Groovy are inspired by their necessity in implementing a Rails-like framework, but using Groovy as a base rather than Ruby. In particular, Groovy has implemented dynamic duck typing (like in Ruby) and domain specific language (DSL) features. These DSL features are what makes it viable for using in Jenkinsfiles. In a nutshell, Groovy is primarily an Object Oriented (OOP) language, is dynamically-typed and has a large collection of functional features which were not available in Java at the time (including anonymous functions). Why we chose to use Groovy Groovy is also the first-class citizen used by Jenkins: its Jenkinsfiles are implemented by a DSL. For this reason, we chose to write our entire Jenkins pipelines functionality in Groovy, slowly migrating our existing (mostly bash) code base towards the more OOP style of our shared library. Although other languages are equally (or more) capable compared to Groovy (Python is an obvious alternative), the fact that Jenkins uses Groovy throughout makes development of our pipelines friction-free: we use the same language all the way through from gathering information from the Jenkins UI to talking to AWS using the SDK provided by Amazon. At Mastercard, we are constantly looking at new technologies to nudge ourselves towards a better work environment. For instance, Dagger is something we're keeping an eye on. How it's designed to work Jenkins Pipeline overview On the Real Time Payments project at Mastercard, we use Jenkins extensively to provide a framework for our development team to spin up test environments for their code. With a fairly complicated system architecture, and with multiple end customers with differing requirements, the system that we use has evolved into quite a complex beast, and requires us to run Groovy both on the Jenkins Controller and Agents. Typically, the code running on the Controller is to instruct the Agent to run a command. We use declarative pipelines to define our Jenkins jobs (and manage with version control), and we have a sophisticated shared library running on the Controller that allows us to modularize and unit test our code, thus making our pipelines more resilient and less prone to breakages. System Groovy on the Controller, Solo Groovy on the Agent System Groovy Groovy, as run by the Jenkins Agent (formally known as the Slave), is significantly different to Groovy as run by the Jenkins Controller (formally known as the Master). The Controller runs its Groovy code (from the Jenkinsfile to our shared library) through an interpreter called groovy-cps. The shortcomings that this entails is generally the price that has to be paid for the resilience that CPS in Jenkins gives us so that we can recover from a catastrophic failure in the Jenkins Controller. CPS as a concept is a style that is borrowed from Functional languages. In Jenkins, a CPS translator is used to parse Groovy code so that Jenkins can save the state of the execution after every step of the code, which allows Jenkins to resume jobs exactly (down to the atomic Groovy instruction) where it was interrupted in case of unexpected failure (e.g. power loss, system crash). This brings with it quite a few restrictions. In particular, it is difficult to bring in third-party libraries into our Groovy scripts on the Controller, and some features of the Groovy language are disabled (either by design, or minor bugs). Aside from the lack of third-party libraries, most differences fall into the category of CPS method mismatch. Moreover, the version of Groovy that the Jenkins Controller runs is quite old (2.4.21 at the time of writing). Solo Groovy Jenkins recommends keeping the use of System Groovy to a minimum, and that anything complex or computationally heavy should be performed on the Jenkins Agent. To this end, we have developed a framework to trigger Groovy code on the Jenkins Agent from the Controller, and we call this Groovy 'Solo Groovy' (in that it runs on its own, without CPS, or requiring pre-compiling). Groovy (more specifically, the DSL from the Jenkinsfile runner) on the Controller supplies us with withGroovy and sh methods which provides access to the groovy executable, and allows us to run a shell command respectively. With that, we can run a standalone script on the Agent just by invoking the following from the Jenkinsfile. withGroovy { sh 'groovy NameOfScript.groovy' } Unlike Java, Groovy has three ways of deciding whether a file is executable as a standalone script. A single class with a static main method class Script { static void main(String\[\] args) { println 'Hello' } } Implementing the Runnable interface class Script implements Runnable { void run() { println 'Hello' } } Imperative style println 'Hello' (There are some other ways as well, but these were the three that we considered). External libraries A dependency problem A problem that we quickly came up against when running Groovy scripts on Agents was that using third-party libraries was a little difficult. The standard way of doing this was to decorate our scripts with @Grab annotations, using Groovy Grapes. However, given the complexity of what we were trying to do, these annotations quickly became unmanagable: We were having to update version numbers across multiple scripts, for instance. Having these dependencies stated in every script meant that there was a lot of boilerplate code. Configuring our code repositories was quite difficult. Having scripts that inherited code from other scripts was a difficult thing. (This made applying OOP principles to our code rather difficult.) Unit testing our code was difficult, and probably requires a POM file anyway. IDEs really don't deal with dependencies with Grapes very well. Much more preferable would be a more standard Enterprise project file layout, much akin to that used by any standard Spring Boot project. Our scripts then looked much like this. class ExampleScript { private final Logger logger private final S3Client s3Client ExampleScript(Logger logger, S3Client s3Client) { this.logger = logger this.s3Client = s3Client } void run() { logger.info('Doing stuff') s3Client.doStuff() logger.success('Stuff now done') } static void main(String\[\] args) { Logger logger = new Logger() S3Client s3Client = new S3Client() ExampleScript instance = new ExampleScript(logger, s3Client) instance.run() } } Having the code structured as it is above allows us to invoke it directly (because of the main method). Having the code in a class instance, and injecting its dependencies via the constructor allows us to mock the code's dependencies, and implement an effective set of unit tests. The problem here, though, is that we cannot just plainly run the script: its dependencies (namely the Logger and the S3Client classes) are not known by the Groovy runtime if we were to invoke the script directly. Our solutions We opted to try and implement a Maven-based solution. POM-files are used extensively on our development projects: some of our projects have been around since before Gradle was widely used, and there are no plans to migrate to it: the advantages of using Gradle over Maven are still being discussed, and certainly Maven serves our needs well. We tentatively place a pom.xml file in the root of our shared library repository, shove all our dependencies into it, and find a way to invoke our Groovy script with those dependencies loaded ready to instantiate at runtime. Fortunately, we found a very neat solution that worked on our development machines. We tentatively put this into a Jenkinsfile for us to watch the execution of our Groovy code with grape-less dependencies. String repositoryRoot = '/path/to/repository' String executeGroovyScript = \[ 'MVN_OUTPUT="$(mvn dependency:build-classpath -Dscope=runtime -q -Dmdep.includeScope=runtime -Dmdep.outputFile=/dev/stdout)"', "CLASSPATH=\\"$repositoryRoot/src:$repositoryRoot/solosrc/main/groovy:\\$MVN_CLASSPATH\\"", 'export CLASSPATH' "groovy \\"$repositoryRoot/solosrc/main/groovy/path/to/script.groovy\\"" \].join(System.lineSeparator()) ''' pipeline { agent any stages { stage('Test') { steps { ws(repositoryRoot) { withGroovy { withMaven( mavenSettingsConfig: 'config-id', maven: '3.6.3' ) { sh executeGroovyScript } } } } } } post { cleanup { cleanWs() } } } Our eyes weep when we see it crash and burn. ++ mvn dependency:build-classpath -Dscope=runtime -q -Dmdep.includeScope=runtime -Dmdep.outputFile=/dev/stdout Picked up JAVA_TOOL_OPTIONS: -Dsome.random.stuff + MVN_OUTPUT='----- withMaven Wrapper script ----- Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) Maven home: /jenkins-agent/tools/hudson.tasks.Maven_MavenInstallation/3.6.3 Java version: 1.8.0_311, vendor: Oracle Corporation, runtime: /jenkins-agent/java/jdk1.8.0_311/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-1160.53.1.el7.x86_64", arch: "amd64", family: "unix" /home/jenkins-agent/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:\*\*\*/lots/of/other/dependencies\*\*\*' + CLASSPATH=/src:/solosrc/main/groovy:, + export CLASSPATH + groovy /solosrc/main/groovy/path/to/script.groovy It took quite a few times to spot this little gem: but mvn, as provided on Jenkins Agents, is actually provided by a wrapper script that logs out useful information about the environment under which it is running. Usually, this is pretty useful in the logs, but in our case (where we simply want the classpath), it made our exported CLASSPATH invalid. A little bit of digging pulls up this issue. Further searching unearths similar issues elsewhere, with seemingly no way to turn this extra output off, and no plans to allow this in the near future. Fortunately, it's is easy enough to fix (or, rather, hack): we simply loop through all lines in the MVN_OUTPUT and discard everything except the final line. String executeGroovyScript = \[ 'MVN_OUTPUT="$(mvn dependency:build-classpath -Dscope=runtime -q -Dmdep.includeScope=runtime -Dmdep.outputFile=/dev/stdout)"', ''' for MVN_CLASSPATH in $MVN_OUTPUT; do # Ignore all lines except the last : done ''', "CLASSPATH=\\"$repositoryRoot/src:$repositoryRoot/solosrc/main/groovy:\\$MVN_CLASSPATH\\"", 'export CLASSPATH' "groovy \\"$repositoryRoot/solosrc/main/groovy/path/to/script.groovy\\"" \].join(System.lineSeparator()) The bash for loop here ignores all of the output of the mvn command except the last one, which is the one that has all the classpath goodness for appending to our CLASSPATH environment variable. System \& Solo arguments This solution for calling solo scripts worked for quite a long time. However, two minor annoyances quickly start to get larger as the number of solo scripts grows. The main method is a piece of boilerplate code that we would like to not have to have. We would like to pass data to and from our solo scripts back to the shared library that we have running on the Jenkins Controller. The data we pass to and from our solo scripts fall into two categories. Configuration from the Jenkins Controller e.g. AWS profile and region to use. These need to passed to all (or a large number of) scripts. Script specific data e.g. the name of the test environment that we are changing. We settle on the following solution: Solo Groovy Invocation Serialize context and script data (i.e. 3.1 and 3.2 above) into JSON files with UUID filenames. Scripts need not implement any interface, or have a main method: instead, they should contain a single run method that accepts a single argument, and has a serializable return type. Write a script that does the following. Accepts the fully-qualified class name containing the code to be executed, along with the filenames of the context and script JSON files, and the expected output filename. Use reflection to find out what data the run method Deserialize the context to a context class instance, and the data to the expected data class instance. Instantiate the class, injecting any dependencies into the constructor. Invoke the run method with the data instance. Serialize the return value to the output file. Our implementation of the following encapsulates the following. A Context class. Its definition is present in both the shared library on the Controller and the solo scripts on the Agent. This was using a singleton pattern until we recently implemented a more sophisticated Inversion of Control container. This class is Mastercard-specific, so I haven't included the code here. The ExecuteScript class. A method that runs on the Controller side. The ExecuteScript class is below: package com.vocalink.solo import com.fasterxml.jackson.databind.ObjectMapper import groovy.json.JsonOutput import java.lang.reflect.Method class ExecuteScript { private final static String IMPLEMENTATION_USAGE = ''' When implementing SoloScript, you must have a single run method taking a single request type and a non-void response type ''' private final static ObjectMapper MAPPER = new ObjectMapper() private final Class clazz private final Context context private final File requestFile private final File responseFile private final soloScriptInstance ExecuteScript( Class clazz, Context context, File requestFile, File responseFile ) { this.clazz = clazz this.context = context this.requestFile = requestFile this.responseFile = responseFile this.soloScriptInstance = clazz.getConstructor().newInstance() } void run() { // Find the run method on the target class Collection runMethods = clazz .methods .findAll { it.name == 'run' } .findAll { it.parameterCount == 1 } .findAll { it.returnType != void } if (runMethods.size() != 1) { throw new UsageException(IMPLEMENTATION_USAGE) } def runMethod = runMethods.first() // Read the request parameter from the supplied file def request = runMethod.parameters.first().type.with { MAPPER.readValue(requestFile.text, it) } // Invoke the run method with the request parameter, and write the result to the output file def response = runMethod.invoke(soloScriptInstance, request) responseFile.write(JsonOutput.toJson(response)) } static void main(String... args) { // Collect the arguments provided from the Controller if (args.length != 5) { throw new UsageException(''' Expected exactly four parameters passed through to this script. They should be: \* fully qualified class name of the script to be run \* path to the JSON file containing the Context object \* path to the JSON file containing the request object \* path that the response object will be serialized to ''') } String className = args\[ 0 \] String contextFile = args\[ 1 \] String requestFile = args\[ 2 \] String responseFile = args\[ 3 \] Context.singleton = deserialize(contextFile, Context) def executor = new ExecuteScript( getClazz(className), context, new File(requestFile), new File(responseFile) ) executor.run() } private static Class getClazz(className) { try { this.classLoader.loadClass(className) } catch (ClassNotFoundException ignored) { throw new UsageException("Could not find class '$className': does it exist, and is it fully-qualified?") } } private static T deserialize(String filePath, Class clazz) { new File(filePath).text.with { MAPPER.readValue(it, clazz) } } } The method to invoke an Agent-side groovy script from the Controller is included below. In our current solution, this method is defined in a class that encapsulates the tasks we ask the Jenkins Agent to do. We have also abstracted away the inner workings of the shell invocation, but that's probably a story for another day. The workflowScript object that is passed in is the object that supplies all the Jenkinsfile methods (and is the value of this inside a Jenkinsfile). def TResponse executeGroovyScript( Object workflowScript, String script, Object request ) { String repositoryRoot = "root/of/repository" String contextFile = "$repositoryRoot/${UUID.randomUUID()}_context.json" String requestFile = "$repositoryRoot/${UUID.randomUUID()}_request.json" String responseFile = "$repositoryRoot/${UUID.randomUUID()}_response.json" String contextJson = JsonOutput.toJson(context) String requestJson = JsonOutput.toJson(request) writeFile("$repositoryRoot/$requestFile", requestJson) writeFile("$repositoryRoot/$contextFile", contextJson) String executeScriptCommand = ''' MVN_OUTPUT="$(mvn dependency:build-classpath -Dscope=runtime -q -Dmdep.includeScope=runtime -Dmdep.outputFile=/dev/stdout)" for MVN_CLASSPATH in $MVN_OUTPUT; do # Ignore all lines except the last : done ''' + """ CLASSPATH="$repositoryRoot/src:$repositoryRoot/solosrc/main/groovy:\\$MVN_CLASSPATH" export CLASSPATH groovy "$repositoryRoot/solosrc/main/groovy/com/vocalink/solo/ExecuteScript.groovy" "$script" "$contextFile" "$requestFile" "$responseFile" """ String output = shell() .withMaven() .withGroovy() .currentDirectory(repositoryRoot) .execute(executeScriptCommand) workflowScript.ws("$repositoryRoot") { workflowScript.withGroovy { workflowScript.withMaven( mavenSettingsConfig: 'config-id', maven: '3.6.3' ) { workflowScript.sh executeGroovyScript String responsePayload = workflowScript.readFile("$repositoryRoot/$responseFile") new JsonSlurperClassic().parseText(responsePayload) as TResponse } } } } The sequence diagram below illustrates the flow of events between solo groovy and system groovy: Execute script overview Summary Our usage of Groovy in Jenkins is advanced, certainly the most advanced that I've ever seen in my career. It's necessarily complex because of the number of clients we serve, the complexity of the product that we build, the sheer number of developers whose work we support, the number of clients we have and the number of tasks we have automated (we have over 200 different individual pipeline definitions, plus an automated build/test/CI pipeline on all of our repositories). Jenkins is a superb tool, but at the level of complexity that we have to operate at, we have needed to apply some Enterprise development architecture to our instance of it. Groovy is a great language that allows us to do that, and Jenkins' use of Groovy at its heart is what makes us able to have such a sophisticated system at the heart of what we do.