Hedge – Building a new cloud deployment tool with service abstraction

At the start of my Siili career as an apprentice, our group was introduced to a mysterious tool: Hedge. Our apprentice project was to build an internal tool using Hedge as a cloud platform abstraction library and a deployment tool. After our apprentice program had come to an end, I was offered an opportunity to continue the development of Hedge in an internal project. Because we had already used Hedge I had some experience as a framework user. The task I was offered was to implement basic Amazon Web Services(AWS) support into Hedge.

What is this Hedge you are talking about?

Hedge offers automated serverless function code generated from common ring compatible handler methods, libraries for various levels of abstractions, and a set of common commands to build handlers, create artifacts, and to deploy created artifacts. Hedge is an open source software and is available in our github.

“Hedge is a platform agnostic ClojureScript framework for deploying ring compatible handlers to various environments with focus on serverless deployments.”

Hedge developer

Back to my journey with Hedge development

The scope of platform agnostic in Hedge was uncertain for me when I started the development. First, I implemented a simple feature parity for AWS code creation and deployment with Serverless Framework. Later the definition of the platform agnostic, roadmap, and context of Hedge became more clear.

When I started working with Hedge I had little Clojure experience and almost no hands-on experience with AWS. For AWS deployment I had checked how other tools do deployment and had chats with my peers. These information sources gave good advice on how to handle deployment.

Why Clojure(Script)? ClojureScript in backend?!

Clojure(Script) is popular and it is in high demand in Finland and at Siili Solutions. Using ClojureScript gives access to ClojureScript and JavaScript libraries. The functional paradigm has steep learning curve, but after learning the basics it is easy to realize that immutable data fits well with serverless handlers. Multithread-safety adds more security if one of the supported clouds is re-using processes with multiple threads.

One of the Clojure build tools, Boot, is also extensively used with Hedge. Boot tasks are great for creating and chaining commands. It is fast and easy to develop a set of tasks which for example build input files, create all artifacts, and finally deploy artifacts to cloud. Then those tasks can be combined into one large task which does all with one command or uses small tasks to store artifacts into disk and later deploys artifacts to cloud.

Why platform agnostic framework?

Following code snippets reveals the first problems with current serverless platforms:

AWS function handler example

'use strict';
module.exports.hello =
(event, context, callback) =>; {
  console.log(’log msg!')
  const response = {
    statusCode: 200,
    body: JSON.stringify({
      message: 'Go Serverless!',
      input: event,
    }),
   };
 callback(null, response);
};

Azure function handler example

'use strict';
module.exports.hello =
(context, req) => {
  context.log(’log msg!');
  context.res = {
    // status: 200, /* Defaults to 200 */
    body: {
      message: 'Go Serverless!',
      input: event,
    }
  };
  context.done();
};

Above code snippets are simple Hello World serverless function handlers but the problem is already visible.

  • Handler function signatures(line 3)
  • Logging(line 4)
  • Output(line 5) handling(line 12)
  • Exit condition signaling(line 12)

all are different.

The main goal of Hedge is platform agnosticism: once the code and configuration has been written for one cloud platform, it is re-deployable to another cloud provider by changing deployment command and a few configuration directives. This improves code re-usability and limits the risk of being locked with specific cloud provider.

To make the implementation easier, Hedge supports only a small common subset of provided cloud features. Limiting number of supported features might be a risk and might lower acceptance rate amongst developers. It is unknown if developers will use a framework which allows using only a small fraction of cloud features. The small number of features definitely narrows down the creativity of developers and might also limit usage of Hedge for some projects.

Our implementation

Hedge adds abstraction layers for function handler code, infrastructure and deployment. Following code snippets are examples of abstraction layers of current development version.

Unified Hedge handler example

(defn hello
[req]
  (info "log msg!")
  {:status 200
   :body "Go Clojure!"})

Above example shows how Hedge unifies function handler features:

  • Handler function signature(lines 1 & 2)
  • Logging (line 3)
  • Payload creation and emit(lines 4 & 5)
  • Function exit condition signalling(line 5)
  • Persistent storage and queues(Still in WIP list)

Infrastructure abstaction

{:api {
       "hello-json" {:handler handler.core/hello-json}
       "calc" {:handler handler.core/calc}
       "fail-hard" {:handler handler.core/fail-hard}}
 :timer {"timer" {:handler handler.core/hello
                  :cron "*/15 * * * *"}}}

Infrastructure abstraction is done as EDN configuration file. Hedge will create platform specific templates from EDN.

Deployment abstraction


$ boot help
...
 aws/deploy                  ** Build and deploy function app(s) **
 aws/deploy-from-directory   ** Deploy files from directory. **
 aws/deploy-to-directory     ** removed for readability **

 azure/deploy                  ** Build and deploy function app(s) **
 azure/deploy-from-directory   ** Deploy files from directory **
 azure/deploy-to-directory     ** removed for readability **
...
$ boot aws/deploy -n my-stack
Deploying to AWS
Stack is ready!
API endpoint base URL :
https://iftc4hog23.execute-api.eu-west-3.amazonaws.com/Prod

cloud-abstractionAbove snippet is example how to run Hedge. Help command lists all available command and simple aws/deploy or azure/deploy commands can be used to deploy project to selected cloud.

Deployment is abstracted with boot tasks. Identical set of command can be used for build, artifact creation and deployment.

Hedge HTTP function handlers resembles Ring handers which is well-known abstraction for HTTP in Clojure. Hedge reads user-supplied configuarion files and serververless function hander then creates cloud-specific wrapper code between a user supplied handler function and cloud’s native handler entry points. The wrapped code handles differences between cloud providers and in the future, there will be libraries for the rest of code abstraction.

Hedge is currently under heavy development and some of the listed features are still on roadmap.

Lessons learned

During this project I had to learn how create Cloudformation templates and stacks from template. Cloudformation templates for lambda functions with API Gateway endpoints first seemed overwhelming, but AWS SAM simplified template creation. AWS SAM is still a young technology and I did not find any big projects using it.

I had to find one important piece of information from StackOverflow. Cloudformations templates using SAM must be deployed using change sets. Luckily it was documented there, since otherwise I probably would have spent a lot of time debugging a feature which was not supposed to work.

As the documentation of SAM is still vague and mostly in GitHub only, I personally can recommend using SAM for serverless Cloudformation templates.

See also:
“Modern Application Development: Should you skip microservices and go directly to serverless?”

A Guidance Framework for Architecting Portable Cloud and Multicloud Applications

Multi-cloud, what are the options? – Low level abstraction libraries

Pros and Cons of a Multi Cloud approach

AWS Helsinki Meetup January 2018 slides

(<3 siili_ clojure) – hope you do too!

My take on ‘Why Clojure?’

Both homoiconic nature and functional side make clojure very distinctive language. They are both inherent in making clojure what it is and i would not take one before the other.

As we have blog posts covering the reasons for FP here i’m going to take FP as a paradigm more or less granted in this post.

Why do i see clojure as a great language in the context of functional programming?

Learning and unlearning

First of all it’s hard to learn a programming paradigm in a language that does not embrace it. A wrong tool drives you to make wrong choices. It leaves you without actually learning the paradigm. It is by far the best thing to do to learn a paradigm using tools meant for it. Eric Normand wrote a good blog post about this problem.

To understand the benefits of FP, you have to use languages meant for it. Languages like Haskell (and by extension Elm and Purescript), Clojure, F#, Scala, Other lisps, Erlang, or OCaml.

I take it that you’re not interested in why not Haskell family, F# or OCaml, but i’ll anyhow go through that part. Most of these languages have very limited use in actual business right now. Currently Haskell and OCaml don’t have any place in programming for business. Which is a pity and no fault of the languages . Elm is currently used in front end and as such is still lacking in backend. Purescript seems to be the most versatile and available of these. It works well both in front and on node.js making it a viable language used in professional fullstack dev.

Scala then is a very solid language running on an widely accepted platform: the JVM. It is a very good all around language, there is no doubt about that. It also has a very good yet complex type system. System that can guide a good FP programmer to discover things about their domain. There are yet few things making it lacking in comparison to Clojure.

  1. Main problem in learning is unlearning. The familiar look of Scala drives people to not learn it. It’s easy to hack something with it but actually using Java with different syntax. Learning first Clojure leads to faster understanding of Scala itself. Adding to this rudimentaries of Haskell helps even more. Learn You a Haskell for Great Good! is a good resource for this.
  2. The unique way of thinking about problems and the language that describes them. This way of thinking is very valid and a good tool to learn for any programmer. Biggest reason for this is the homoiconic nature of the language. The great ability it brings in metaprogramming.
  3. Scala is currently only viable for backend use. Unlike scala Clojure has many supported platforms. It supports JVM, JS including node and browser, and CLR. All which acceptable platforms for business use. This makes it possible to share the same code in front end and backend. With it you can make a consistent whole. Creating front end components that are in sync with their microservice counterparts. In scala this is not possible as Scala.js is lacking in actual production grade stability.

So yes scala is an interesting language but not the best to learn FP. Especially so for Java developers.

Motivation as a driver for effectiveness

It’s actually quite hard to have valid measurements for coder efficiency. Yet one thing we do know. Study after study has found that developer motivation is elemental to productivity.

With unenthustiastic devs its best to inspire them. One of the main benefits comes from the fact that clojure is easy to use and makes coding actually fun again. This is a quote you do hear quite a lot from clojure devs.

As this is about inspiring people, you should go with what is good and nice and feels good and brings joy to you. As then you’ll be more inspiring.

Final words

It also goes down to the provable fact that Imperative Paradigm is more complex than simpler FP. Inside FP clojure is a very very very easy way to learn it. Much easier one to reap the rewards than most others.

If you have a bunch of coders struggling with imperative stuff. And you want to get the benefits of FP as fast as possible. Clojure is the fastest one to learn.