Don’t Fear the Agile

When truly Agile, we developers are involved less in projects we don’t believe in. We can influence on what the outcome will be. But it demands courage and will to live in a constant change.

Billiards_balls

Agile and billiards – or pool – has things in common. At the beginning of a game the balls are in neat, predefined order. During the game however their position on the table changes in unpredictable ways, ways that nobody could see. The best action is chosen according to the new situation. This is the idea behind Agile too. Details are decided as the development progresses instead of heavy planning at the beginning.

Back in the days project managers tried to plan software projects from start to finish with all the details. This worked in those rare occasions when own people made no mistakes and other players acted completely according to ‘the script’. One can count these with his or her middle fingers.

Planning a software project beforehand with all the details is practically impossible. Especially if you want to create an outcome, that fills users needs and produces significant value to the customer who pays the bill. Something unforeseen always comes up during the project. Something that needs responding. In agile these situations are solved by changing the game plan.

Agile Gives You Better Solutions

During my 15 year career as software craftsman I’ve been involved in hundreds of bigger and smaller development actions, projects and ventures. I have also seen that agile methodologies scares some people. They feel the development, though falsely, a lot safer, if all things can be planned precisely.

DQmQezCasSoZs5f1i9Lms9EKLy2jfnNNfrLsRutH1y7uLpV

The Agile way is a huge opportunity for us developers. Yes, it demands skill and will to regenerate constantly. But with it, we can create solutions that exceeds the expectations.

I was working in a software project, where we knowingly skipped planning of an uncertain part which were to be implemented only after few months. During these months the customer scoped that part completely out of the product. Being agile and co-operating with the customer we avoided redundant design and work. These decisions also prevented creating unnecessary complex system. We saved customers money and created a better outcome.

Sometimes also totally new requirements emerge. Depending on their size, they might even paralyse the traditional software development process. At least they will delay it significantly.

Agile Necessitates Continues Learning

Agile has enabled me to try out new tools and techniques and learn new stuff constantly. Sometimes experiments have not produced the expected outcome, but they have never been useless. So called failures also provides lots of information for the next steps to take. As in billiards, learning how the spins work in general and in different situations, I can rectify errors in previous shots in the upcoming ones.

td-deep-learning

Continuous learning and improving is part of developers DNA. Without running experiments and learning, you can only achieve a good solution to a single problem. And that is already in the past.

Don’t go to conference!

The title is from a pitching session in Continuous Integration and Testing -conference (CITCON18) by Open Information Foundation. The announcer of the pitch wanted to create discussion about benefits of traditional conferences versus Open Space events. CITCON itself represents an Open Space format event, which is an awesome way to share ideas and networking for IT professionals.

CITCON gathered in the spring of Vienna, Austria, this time with around 60 people, of whom six were Hedgehogs from our Software Automation Tribe. Open Space events are most effective with 50-100 people, since the sessions are organized on the spot in an agile manner. The participants are mostly software and testing specialists, but the topics contain more human sciences than you’d find in traditional IT conferences.

The event starts with a quick introduction of participants. Followed by inventing short pitch talks about the topic for upcoming discussion sessions. You are allowed to present one pitch at a time, but can you join the line of presenters again after you are done with your first pitch. People are also encouraged to ask clarifying questions if the idea is hard to grasp. Post-It notes with the topic candidate are added on a whiteboard after a pitch is made. Each session is meant to last one hour.

IMG_20180420_194213
Tommi Oinonen pitching his session topic

When the line of presenters had finished, the amount of Post-It notes had grown up to around 50. Next, everyone could vote on the topics their are interested in with a pencil, but only one vote at a time. There were five time slots and five rooms booked for the sessions next day. Hence, there were 25 sessions in total. When all the voters had visited the whiteboard, the most popular topics began to move to a timetable chart. This happened also in a self-organizing way, meaning that everybody could move the notes and the most voted ones found their time slots. Similar topics could end up bundled into the same session. The final timetable might not have been formulated earlier than just before the sessions the next day.

The time table is then used by people to plan their attendance to different sessions. However, the most important rule with Open Space sessions is the law of two feet: If the current session isn’t interesting, leave it.

IMG-20180421-WA0000
Agile timetable

The sessions were held in classrooms, where chairs were organized in a circular form in order to maintain best possible connection between participants. The classroom also contained a white board for demonstrating purposes. The topics varied from more technical software problem solving in IT to sea shanties (!?). The most popular topics were about communication skills in work life. These sessions were facilitated by professionals, who also solve those issues in their everyday work. The session I liked the most was one where communication related challenges were asked from the participants and then solved via a role playing style dialog.

As an example of a more technical session topic, Tommi Oinonen from Siili wanted to obtain insights about his master’s thesis pertaining to metrics of software test automation and version control. Tommi truly achieved some good philosophical conversation and opinions to bring home from CITCON.

If I had to list some negatives about this Open Space event, I felt that people were even too eager to create session topics. Some general level subjects without deeper focus on some strictly defined problem ended up leading conversations into an academic monologue.

Open Space events encourage people to create sessions of their own kind where everyone can have interesting discussions. Compared especially to lecture-liked conferences, interactions between even shyer engineers increases. As an organizer encapsulated in his opening speech: If you did not get what you were looking for here, blame the organizers, that means yourself!

Pekka Rantala

A Thought on Writing Tests That Suck Less

 

I visited the ClojureD conference in Berlin on 24.2.2018 during some cold winter days. I went with my colleagues to gather some insights and listen to interesting talks on different subjects around Clojure. As a result I decided to challenge myself (and tests I write), inspired by Torsten Mangners good presentation titled:
Writing tests that suck less – “What” vs “How”.
Disclaimer: This post takes no credit whatsoever on the subject or claims to be a best way of doing things. This text relies on Torsten Mangners presentation but there is also some personal interpretation.

The Gauntlet

As Torsten reasoned that writing pure unit tests in Clojure (or any other language) is usually a small task. You invoke your unit with inputs and assert the results. Tests ideally document our software (as an added side-effect of verification).
Let’s see a simple, self-documenting unit tests.
;; the implementation (the unit, the HOW)
(defn add-numbers
  [numbers]
  (apply + numbers))

;; the test that should describe to a (technical) reader WHAT we are testing
(deftest add-two-numbers-together
  (testing "If two numbers can be added together"
    (is (= 3 (add-numbers '(1 2))))))
So it’s pretty straightforward, we don’t need to know the HOW or any other implementation details and because we are testing a pure unit with no dependencies we don’t need to prepare anything obfuscating and arcane.
Writing tests should not be an effort for a developer – but it starts to feel so when we run out of pure functions and go over to integration testing. It shouldn’t be too hard to write the actual tests – we just end up with a bunch of namespace_test.clj(s) files that can end up being many times longer than the actual code including a lot of HOW:s. Arcane and magical ceremonies, where we roll out mocks, stub and whatnot – finally ending in one or a few strange assertions.
If you jump into a project and find out that the documentation basically is the source code, you would be glad to find out that the tests would clearly tell us WHAT they test and prove. We don’t want to investigate and reverse-engineer the HOW part. We could just read what the software internals in it’s current state is documented and supposed to do.
At some point in time, as a software developer, you would need to know the HOW but that’s more of the part of writing the implementation (and the tests).

Throwing The Gauntlet

Thus I challenge myself to separate the bloat of HOW out my tests that repeatedly ended up looking like this:
(deftest handler-post-to-user-success
  (testing "calling handler posting message to user"
    (async done
      ; channel open
      (.mockOnce xhr-mocklet "POST" #".*"
        (mocked-response (clj->js {:ok true}) 200 "application/json"))
      ; post
      (.mockOnce xhr-mocklet "POST" #".*"
        (mocked-response (clj->js {:ok true}) 200 "application/json"))
      (go
        (let [response (<! (api/handler 
          {:body {"data" {"message" "testmessage"
                          "user-id" "user-id"}}}))]
          (is (utils.http/request-success? response)))           
          (is (= "post message to channel success" 
                 (-> response :body :message))))
    (done)))))

This test mocks two successful HTTP calls, invokes the handler (entry point of the serverless function) and performs some assertions.

For those who are curious about what’s going on in here – this is a test for a serverless REST API handler in a Slack Bot backend. It’s written in ClojureScript and runs on Javascript function runtime on Azure.  See Siili Solutions / Hedge on GitHub – a serverless framework to deploy ClojureScript functions on Azure and AWS.

This test checks how our serverless API responds to clients if it succeeds when contacting the Slack API.

So is this test complex? Maybe not too complex. But when you test more complicated logic it probably will be. Also there’s lot of potential arcane ceremony that doesn’t add to test readability like async done, (done), go, <! that is purely about the HOW in my tests. And those things just repeat over and over, multiple times per test namespace.

The Dust Settles

So this was my quick try to find a general, more WHAT for my tests.

(deftest handler-post-to-user-success
  (testing-handler "calling handler, posting message to user succeeds"
    :incoming-request {:body {"data" {"message" "testmessage"
                                      "user-id" "user-id"}}}
    :ext-http-calls-return #(do (channel-open-success)
                                (channel-post-success))
    :assert #(do (is (utils.http/request-success? %))
                 (is (= "post message to channel success"
                        (-> % :body :message))))))

Maybe it wasn’t state of the art of perfect, but I hope you see the intent. So the summary of the WHAT in my case:

  • We are testing the handler (the serverless functions entry point)
  • The input values are presented in :incoming-request
  • :ext-http-calls-return describes what is going on in the “external” dependencies during this run (mocks are instantiated)
  • :assert contains the assertions that should be done for this test

And therefore:

  • Less noise
  • Less boilerplate
  • More documentation

So I re-wrapped testing to do the things that are repeated over and over in all the tests:

(defn testing-handler [message & {:keys [ext-http-call-returns
                                         incoming-request
                                         assert]}]
  (testing message
    (async done
      ;; install mocks
      (ext-http-call-returns)
      (go
        ;; invoke the handler
        (let [response (<! (api/handler incoming-request))]
          ;; assert
          (assert response)
    (done))))))

And my mocks are just a different combination of successes and failures, so they were just wrapped into their own functions.

Conclusion

You could still reason that this could even be more simplified. Macros could help me to get rid of even more noise. But in the end of the day,  Torstens presentation at ClojureD  made me more motivated about trying to write tests that suck less. Or die trying.

ClojureD conference in Berlin

In the end of February five Siilis took off from Helsinki to visit Berlin. ClojureD conference to be exact. Two tracks, plenty of talks and focus in Clojure. This is an overview from the talks that we visited and because Joy Clark had awesome notes on talks too I asked permission to share her notes here as well. Big thumbs up for the visual content!

Teaching Clojure

For me the first talk of the day was about teaching. We have ran our junior program with Clojure for two years as we speak so new views and how others find teaching in this domain was interesting. Mike Sperber shared his point on systematic thinking and how it affects the way we build software.

teaching_clj

Recap in drawing from Joy Clark

Maria: a beginner-friendly coding environment for Clojure

Next talk was about Maria by Dave Liepmann. A simple really beginner friendly place to learn fundamentals in Clojure. In his talk there was quite a few demos on how fast one can get on with coding.

Fast results are important when you start something new and Maria offers fast response to your coding and shows results immediately. Of course when you start doing bigger and more advanced stuff your pace of getting things done and ready will be different.

mariacloud.jpg

Recap in drawing from Joy Clark

 Writing test that suck less

Last talk before lunch was about testing by Torsten Mangner. This talk dived a little deeper to the fundamentals of testing. Talk began wit a confession and gave us the baseline.

“Writing unit tests in Clojure is easy, since testing pure functions is trivial. But the more high-level our tests become, the more they have to deal with the state and side-effects of your application. It becomes harder and harder to write proper tests. And worst of all: they are harder and harder to understand and to maintain, greatly diminishing the value of those tests as a documentation of your software.”

confession.jpg

A Dynamic, Statically Typed Contradiction

After lunch we had the pleasure of diving into mathematics. As Andrew Mcveigh talked about lambda calculus and its application in a “Hindley-Milner“ based type system and how it can be mapped onto Clojure. This solution can also be used to “type-chec” a subset of Clojure code.

Unfortunately my attention was to fully hang on with the maths and our ways with Joy had departed so, no good pictures from this talk. Sorry!

Onyx, virtual machines, jokes and afterparty

Next up for me was Vijay talking about Onyx and its uses for data crunching. An overall look for the uses. Masterless systems, stream and batch processing, flow control and principles of Clojure.

After coffee we got the pleasure of seeing a walkthrough how to implement Clojure on a new virtual machine. Heavy stuff and a great idea.

Last official talk before the lightning talks (I had to skip those because I was hungry and needed a Döner 😉 ) was called defjoke. Live coding a fully compliant spec for humour.

All talks were taped and are available soon in http://clojured.de/ check it out. Other great photos and one-liners can be found in https://twitter.com/clojuredconf

Thanks from Siili to great organisers and speakers. Hoping to see you soon in other adventures!

 

-e

 

Serverlessconf

I remember last year binge-watching the Serverlessconf recordings online thinking “If there is one conference I want to go this is the one!”. The talks were so utterly good that I felt bad for not having witnessed it live. I was looking for the coming Serverlessconfs but they seemed to be coming nowhere near me. Then suddenly they announced the event in Paris and I went crazy. Let´s do this!

1

On day #1 there were multiple workshops on various technology stacks. AWS, Microsoft Azure, Google Cloud, OpenFaaS and IBM Cloud Functions.

I chose a workshop that was organized by A Cloud Guru where we built a Youtube like video service website with AWS API Gateway, Lambda, Elastic transcoder, Google Firebase and Auth0.

2

The present and future of Serverless observability | Yan Cui

The first presentation of the day was about the present and future of Serverless observability. Yan Cui was pleasant to listen as he had a very fluent presentation style and insightful observations. He also pointed out an interesting whitepaper Serverless Application Lens for the AWS Well-architected framework.

The document covers common serverless applications scenarios. I encourage you to read it to understand AWS best practices and strategies to use when designing architectures for serverless applications.

The next presentation was given by Subbu Allamaraju from Expedia Inc. Expedia is American travel company that owns some of the most popular travel sites like Hotels.com and Trivago. He approached the concept from an interesting perspective highlighting some points on benefits of the serverless paradigm and in contrast common counter arguments like today’s limitations and constraints, personal perspectives of cloud, portability, lock-in, cost etc. He highlighted how past habits, culture and legacy can really slow down adoption.

The presentation provided ideas on how to tackle these blockers on the journey towards a platform of the future, where functions and events fit, and how to prepare ones colleagues for that future.

3

Serverless tools with Google Cloud and Firebase | Bret McGowen & Monika Nawrot

Bret McGowen & Monika Nawrot showed demos around the Google Cloud offering. Monika explained how Google Serverless offering has grown over the years and had a Google Cloud Functions demo with thumbnail creation function, storage & logging. She highlighted also that there is a possibility to use a Function emulator for debugging functions.

Bret McGowen vent quickly through some of the best Backend as a Service (BaaS) offerings out there and then focused on the Google Firebase. He explained that the Realtime database was the first Firebase feature but there are many more built-in features like Cloud storage, Functions, Hosting, etc.

He also made a bold statement which people seemed to find quite amusing. “More serverless than serverless!” =) . It sounded like he was selling artificial sweetener by claiming “Sweeter than sugar” and I found it triggering my bullshit detector. Luckily the demos totally convinced everybody as he showed the Firebase realtime update capability. Apparently Firebase is also handling situations when a client connection drops. The data syncs instantly when a client gets back online. All the Firebase features can be implemented directly in the client-side code using an SDK. The sugar on the top was BigQuery that could query strings on pretty much the entire universe in just mere seconds.

4

https://firebase.google.com/

Visualizing Serverless Architectures: What does a healthy serverless app look like? | Kassandra Perch

Kassandra Perch talked about visualizing Serverless architectures and what does a healthy serverless app look like. Her demo included inspecting for example error rates and counting cold starts. She showed how to include IopipeLib, trace and profiler in a lambda function & how to use those. Interestingly the IOPipe wraps the whole lambda handler. The recommended tool IOPipe has a slogan “See inside you Lambda functions” You can find it here: https://www.iopipe.com/

IBM Cloud Functions & Apache OpenWhisk: Overview and customer scenarios | Frédéric Lavigne

Frédéric Lavigne introduced IBM´s Serverless offering and clarified the difference between IBM Cloud Functions & Apache OpenWhisk. OpenWhisk is open source and used in the IBM managed service. The IBM Cloud team contributes to the open source project. Runtimes include: Node 6, Node 8, PHP 7.1, Python 3 and Swift. Additionally, OpenWhisk can run any custom code put in a Docker container.

In the demo he defined a smiley face as SVG inside the Function code. This was enabled by the RAW HTTP handling setting. Unfortunately, there was some demo effect with API Gateway as he tried to show how to use Web actions and API Gateway. IBM Cloud Functions have some built-in event sources such as Cloudant, Message Hub and Github. Currently they are developing a composer for cloud function flow control.

The presentation included some interesting customer cases such as the DOV-E Hypersonic-based Mobile Device Communication. They had Coca Cola TV advertise trigger a hypersonic message on the viewers phone. Viewer could not hear the communication, but they received a message on the phone asking whether they would like to drink some Coca Cola right there and then.

Event specifications, state of the serverless landscape, and other news from the CNCF Serverless Working Group | Daniel Krook

Daniel Krook had a very informative talk about Event specifications, state of the serverless landscape, and other news from the CNCF Serverless Working Group. CNCF stands for Cloud Native Computing Foundation which drives the adoption of the new Serverless computing paradigm.

They have multiple projects under their wings such as Kubernetes, Prometheus, Fluentd to name a few. However, currently there are no serverless projects. They consist of four active CNCF Working groups (Continuous Integration, Networking, Storage, Serverless) and have now a whitepaper available for review. It is hosted in Github for collaboration purposes and they are accepting pull requests to keep up with the fast phase of development in Serverless. Please find it here: http://www.github.com/cncf/wg-serverless

Accelerating DevOps with Serverless | Michael H. Oshita

Michael H. Oshita talked about Accelerating DevOps with Serverless. His excellent demo showcased how to provide automated branch testing infrastructure for Devs without burdening Ops. Technically the setup uses Github hooks that trigger Circle CI, provisions using Terraform templates and Lambda functions are orchestrated with Step Functions. The infrastructure is created on pull request and destroyed when branch is merged and deleted. See the ECS Deity -project here: http://www.github.com/ijin/ecs-deity

End-To-End Serverless | Randall Hunt

The most entertaining talk of the day was a live coding session with technical evangelist Randal Hunt. He´s opening line was surprisingly in French and the goofball show made everyone crack up. He also asked audience for suggestions on what to build there on the spot in front of the audience. Not something most presenters would want to try I imagine. The randomness didn´t seem to bother viewers and he did manage to finish the live demo just in time which made the organizers happy. The energetic personality and laid back attitude with relevant content was a great combination and I hope to see more of this in the future. If there is one recording you should watch, it would be this one in my opinion.

5

With Great Scalability Comes Great Responsibility | Dana Engebretson

Dana Engebretson had a pleasant talk telling an interesting story about her journey to Serverless architecture for analytical purposes. Humorous croissant baking tips and to-the-point analogies made it inspiring and entertaining.

6

Overall the talks were versatile and excellent. There were also some less technical talks but nothing really outside the Serverless focus which was good as it didn´t seem like the organizers had struggled finding relevant and good presentations for the event. I recommend getting involved in your local cloud computing communities. Serverless development is super-efficient and enjoyable so if you haven´t tried it I suggest you do so without further due.


The author Jouni Leino has 15 years of experience in IT. He lives in Stuttgart Germany working as a Field Application Engineer in the Automotive industry. As Serverless Competence Lead he´s mission is to guide customers and colleagues towards a more cost effective and productive future.
Follow in Twitter
LinkedIn 

Snips and Snails, and Puppy Dog Tails, is that what Agile Leadership is made of?

Let’s face it, we are all doomed. But it’ll take some time before this madness ends. And during that time the world is becoming increasingly complex. The speed of change is faster than ever and more and more is demanded from the leaders. Managing actions and expecting results won’t get you there. Leaders who affect on experiences and beliefs are creating culture that welcomes change. It is quite easy to change those in present and in the future. Have an open mind, search for positive exceptions and find positive explanations. You will create more positive culture for learning and creativity. And by using techniques from NLP it is possible to even change the meanings of one’s experiences and beliefs in the past.
And this is why Agile Leadership is needed. These are some of the topics we went through on our two and a half day course of Certified Agile Leadership by Agile42.

Cynefin – Embrace the Suck

complexity-cynefin.jpgI’d like to return on one word from the past chapter: complex. It is the most interesting domains of the Cynefin Framework. The field of unknown unknowns is where one can

47e6eb3a1b82db01831fa600c120635f

deduce cause and effect only in retrospect. There are no right answers. Thus we as leaders need to create safe to fail experiments. We cannot solve complex problems with best practices. Software development is very much in the complex domain. Agile offers many tools for solving problems in that domain: reflection and improving, collaboration and continuous delivery. If you’re good, you will push many problems to complicated or even obvious domain for example by utilizing automation. Welcome the complex and do some magic!

Giving Your Team the AMP up

team-motivation.jpg
Motivating your team is sometimes difficult. Relying on extrinsic motivators such as money or using the good-old carrot and stick approach does not work when your tasks call for even rudimentary cognitive skill. And even when they do work – the mechanical tasks – you are using a shrinking pie. Offering teams the AMP – Autonomy, Mastery, Purpose – utilizes intrinsic motivators. And they do work! And yet again Agile has tricks for this one too. It introduces pull and self-organization for autonomy. It offers collaboration, feedback, and trial & error for mastery. And it gives you holistic responsibility for purpose. And there are many factors in this growing pie of motivators. Listen to your team and you will learn what will make them tick!

All Hands on DECK!

self-organization.jpg
Self-Organization is extremely important. The more responsibility the team has the more flexible, robust and innovative the team will be. As a leader one should make sure that the teams has a right direction and suitable environment. Removing unnecessary constraints is equally vital. The team will increase it’s own knowledge base as it goes on. And this will speed up the development. Aiming to Self-Designing Teams should be a standard in any agile organization. This means for example that team itself designs who belongs to the team and who does not. Of course those teams also manage their progress and process. And execute their tasks.

When Your Heart and Brain Are Happy Together

emotional-intelligence.jpg
To be a successful leader in an agile context one needs to have emotional intelligence. It falls into self- and social oriented parts. It also splits into how aware one is of oneself and how one handles things. So, emotional intelligence consists of self and social awareness, self management and social skills. Does things like self-confidence, empathy, self-control, initiative, visionary and good communication skills sound like the qualities of a great leader? Another wonderful thing is that one can develop each of these attributes.

Agile Meets Antifragile

Let’s step onwards from the team level to consider the whole organization. We should think more on what happens to the organization when risks realize or some crisis emerge. A fragile organization worries nothing and acts in a naive way. In an emergency it breaks, even in a catastrophic way. A robust organization can resist known stressors, and it should be a minimal level for any company. In a contrast, a resilient system reforms itself during disturbances. Finally it retains the same identity than before. Antifragile systems are the sovereign champions in the organizational map. They can increase their capabilities and grow stronger in response to stress. They are the truest of agile and learning organizations!

antifragility.jpg
To achieve this, there are some things one could experiment. Engage people more. Be empirical and apply local validation. Introduce teams to iterative & incremental development. Have some defined metrics. Create emergent standardization when you can. And this is basically what Agile Leadership is made of.

But Wait! There’s More!

Ok, say that you agree on all the above. But to shine and thrive as an agile leader you need good knowledge and understanding of agile values and principles. You need to deal less with things that are going wrong and help things go right more. Systems thinking will be useful. You should:
“Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.”
You should know at least three different leadership styles and be able to choose the right one for each occasion. Practice Benjamin Franklin’s Thirteen Virtues. Follow simple advice of Michelle Eileen McNamara “It’s chaos. Be kind.”

 

Hedge – Building a new cloud deployment tool with service abstraction

At the start of my Siili career as an apprentice, our group was introduced to a mysterious tool: Hedge. Our apprentice project was to build an internal tool using Hedge as a cloud platform abstraction library and a deployment tool. After our apprentice program had come to an end, I was offered an opportunity to continue the development of Hedge in an internal project. Because we had already used Hedge I had some experience as a framework user. The task I was offered was to implement basic Amazon Web Services(AWS) support into Hedge.

What is this Hedge you are talking about?

Hedge offers automated serverless function code generated from common ring compatible handler methods, libraries for various levels of abstractions, and a set of common commands to build handlers, create artifacts, and to deploy created artifacts. Hedge is an open source software and is available in our github.

“Hedge is a platform agnostic ClojureScript framework for deploying ring compatible handlers to various environments with focus on serverless deployments.”

Hedge developer

Back to my journey with Hedge development

The scope of platform agnostic in Hedge was uncertain for me when I started the development. First, I implemented a simple feature parity for AWS code creation and deployment with Serverless Framework. Later the definition of the platform agnostic, roadmap, and context of Hedge became more clear.

When I started working with Hedge I had little Clojure experience and almost no hands-on experience with AWS. For AWS deployment I had checked how other tools do deployment and had chats with my peers. These information sources gave good advice on how to handle deployment.

Why Clojure(Script)? ClojureScript in backend?!

Clojure(Script) is popular and it is in high demand in Finland and at Siili Solutions. Using ClojureScript gives access to ClojureScript and JavaScript libraries. The functional paradigm has steep learning curve, but after learning the basics it is easy to realize that immutable data fits well with serverless handlers. Multithread-safety adds more security if one of the supported clouds is re-using processes with multiple threads.

One of the Clojure build tools, Boot, is also extensively used with Hedge. Boot tasks are great for creating and chaining commands. It is fast and easy to develop a set of tasks which for example build input files, create all artifacts, and finally deploy artifacts to cloud. Then those tasks can be combined into one large task which does all with one command or uses small tasks to store artifacts into disk and later deploys artifacts to cloud.

Why platform agnostic framework?

Following code snippets reveals the first problems with current serverless platforms:

AWS function handler example

'use strict';
module.exports.hello =
(event, context, callback) =>; {
  console.log(’log msg!')
  const response = {
    statusCode: 200,
    body: JSON.stringify({
      message: 'Go Serverless!',
      input: event,
    }),
   };
 callback(null, response);
};

Azure function handler example

'use strict';
module.exports.hello =
(context, req) => {
  context.log(’log msg!');
  context.res = {
    // status: 200, /* Defaults to 200 */
    body: {
      message: 'Go Serverless!',
      input: event,
    }
  };
  context.done();
};

Above code snippets are simple Hello World serverless function handlers but the problem is already visible.

  • Handler function signatures(line 3)
  • Logging(line 4)
  • Output(line 5) handling(line 12)
  • Exit condition signaling(line 12)

all are different.

The main goal of Hedge is platform agnosticism: once the code and configuration has been written for one cloud platform, it is re-deployable to another cloud provider by changing deployment command and a few configuration directives. This improves code re-usability and limits the risk of being locked with specific cloud provider.

To make the implementation easier, Hedge supports only a small common subset of provided cloud features. Limiting number of supported features might be a risk and might lower acceptance rate amongst developers. It is unknown if developers will use a framework which allows using only a small fraction of cloud features. The small number of features definitely narrows down the creativity of developers and might also limit usage of Hedge for some projects.

Our implementation

Hedge adds abstraction layers for function handler code, infrastructure and deployment. Following code snippets are examples of abstraction layers of current development version.

Unified Hedge handler example

(defn hello
[req]
  (info "log msg!")
  {:status 200
   :body "Go Clojure!"})

Above example shows how Hedge unifies function handler features:

  • Handler function signature(lines 1 & 2)
  • Logging (line 3)
  • Payload creation and emit(lines 4 & 5)
  • Function exit condition signalling(line 5)
  • Persistent storage and queues(Still in WIP list)

Infrastructure abstaction

{:api {
       "hello-json" {:handler handler.core/hello-json}
       "calc" {:handler handler.core/calc}
       "fail-hard" {:handler handler.core/fail-hard}}
 :timer {"timer" {:handler handler.core/hello
                  :cron "*/15 * * * *"}}}

Infrastructure abstraction is done as EDN configuration file. Hedge will create platform specific templates from EDN.

Deployment abstraction


$ boot help
...
 aws/deploy                  ** Build and deploy function app(s) **
 aws/deploy-from-directory   ** Deploy files from directory. **
 aws/deploy-to-directory     ** removed for readability **

 azure/deploy                  ** Build and deploy function app(s) **
 azure/deploy-from-directory   ** Deploy files from directory **
 azure/deploy-to-directory     ** removed for readability **
...
$ boot aws/deploy -n my-stack
Deploying to AWS
Stack is ready!
API endpoint base URL :
https://iftc4hog23.execute-api.eu-west-3.amazonaws.com/Prod

cloud-abstractionAbove snippet is example how to run Hedge. Help command lists all available command and simple aws/deploy or azure/deploy commands can be used to deploy project to selected cloud.

Deployment is abstracted with boot tasks. Identical set of command can be used for build, artifact creation and deployment.

Hedge HTTP function handlers resembles Ring handers which is well-known abstraction for HTTP in Clojure. Hedge reads user-supplied configuarion files and serververless function hander then creates cloud-specific wrapper code between a user supplied handler function and cloud’s native handler entry points. The wrapped code handles differences between cloud providers and in the future, there will be libraries for the rest of code abstraction.

Hedge is currently under heavy development and some of the listed features are still on roadmap.

Lessons learned

During this project I had to learn how create Cloudformation templates and stacks from template. Cloudformation templates for lambda functions with API Gateway endpoints first seemed overwhelming, but AWS SAM simplified template creation. AWS SAM is still a young technology and I did not find any big projects using it.

I had to find one important piece of information from StackOverflow. Cloudformations templates using SAM must be deployed using change sets. Luckily it was documented there, since otherwise I probably would have spent a lot of time debugging a feature which was not supposed to work.

As the documentation of SAM is still vague and mostly in GitHub only, I personally can recommend using SAM for serverless Cloudformation templates.

See also:
“Modern Application Development: Should you skip microservices and go directly to serverless?”

A Guidance Framework for Architecting Portable Cloud and Multicloud Applications

Multi-cloud, what are the options? – Low level abstraction libraries

Pros and Cons of a Multi Cloud approach

AWS Helsinki Meetup January 2018 slides

(<3 siili_ clojure) – hope you do too!