Cigars and Serverless IoT

Say what? Cigars and Serverless? What on earth could those two have in common? Maybe not much but bare with me and I’ll let you know.

The background

Some time ago I happened to run into a new Finnish open-source sensor beacon platform. I really wanted to give it a try. What could I do with it? Enter cigars! I came up with a requirement and created a user story that I wanted to implement: “As a cigar owner I want to be able to monitor the temperature and humidity of my humidor regardless of my location so that I know when to add purified water into the humidifier“. My current solution required me to open the humidor and check the meter inside it.

Screen Shot 2017-10-17 at 10.04.16

The brainstorm

Two important non-functional requirements were wireless communication from the humidor so I wouldn’t have to do any physical modifications to the humidor and a long battery life so I wouldn’t have to replace it too often. Both were met by the chosen sensor.

So what else would I need to make it happen? A place where I could process and save the sensor data and visualize it for devices regardless of their (or my) location. Enter cloud! I also had a couple of Raspberry Pi boards lying around so when the sensors arrived I was good to go.

AWS has an IoT service that can listen to messages from things (as in Internet of Things) you have registered to it. You can then do whatever you wish with those messages: save them into DynamoDB, process them with Lambda, forward them to Kinesis etc. Just what I needed. Enter serverless! Have to say I was very exited. I had never done any IoT stuff before so this was going to be a learning experience for me as well.

The solution

First thing I wanted to do was to read the sensor from the RasPi. A little bit of web surfing revealed a small but enthusiastic community around the sensor and I found a python script doing exactly what I wanted. The communication technology would be BLE.

Next thing was to connect the RasPi to AWS IoT service. That was also a no-brainer thanks to AWS documentation. AWS creates certificates and keys for the thing to be authenticated with and an endpoint for the messages. AWS processes the incoming messages with Rules. A rule defines a query that parses the incoming message and action(s) to be performed. The thing publishes it’s messages into a named MQTT topic via the given endpoint and the rule is a subscriber to the same topic.

I chose to save the data into DynamoDB by my IoT rule and implement a serveless website using S3 and Lambda. S3 is an object storage that is perfect for hosting static html files and Lambda is a compute service to run code without having to worry about any infrastructure. My Lambda function fetches the data from DynamoDB table and is called by ajax from html through API gateway.

Finally I wanted the lightest and the simpliest javascript graph library to visualize the sensor data from my humidor. See the screenshot  above. A bit boring graph I know. But luckily it is not an EKG!

At the time of writing this the data is flowing once every 15 minutes from the sensor into DynamoDB and it is read from there by Lambda whenever the html page is loaded. Maybe I’ll implement some alarms next?

Screen Shot 2017-10-17 at 8.27.36

What did I learn?

First of all I learned once again that serverless services are extremely fast and easy to implement for example for prototyping. They enable individuals and businesses to do things that have been impossible or at least expensive in the past. They also make it easy to explore new ways of doing thing and doing business. My rough cost estimation for this solution is 1-2€ per month after the AWS free tier has been eaten. So it is fast, easy and cheap as well.

Secondly I learned a lot about how DynamoDB works. There was quite a few tricks on the way. For example is allows you to set a TTL attribute for a field containing epoch seconds as a string but it won’t do anything.

Resources:
AWS Services: IoT, Lambda, S3, DynamoDB
Sensor: Ruuvitag
Hardware: Raspberry Pi
Source and more details: Github

Serverless 101 and Siili CraftCon

The second official Siili CraftCon was held before summer holidays 2017. It is an internal craftmanship conference for all craftmen and -women in Siili. This time it was half days and three tracks worth of pure skillz with topics such as “how to be a tech lead“, “data driven design“, “RPA” and more.

I had the pleasure of speaking about serverless architecture to the whole crowd as a closing presentation. Since I am a keen agile/lean fan I am also totally in love in serverless architecture and everything it has to offer in terms of reacting to changes and feedback and the ease of implementing new features and trying out new things.

Serverless means that you only need to think of you business logic. Everything else is taken care of by your chosen vendor. All big name vendors have their own serverless platform and services. In this context I am talking about PaaS (Platform as a Service) and FaaS (Function as a Service) side of serverless.

Some may include SaaS and (m)BaaS solutions into serverless context. SaaS stands for Software as a Service and like the name implies it includes software you configure for your needs. Some examples are Google Apps, Dropbox and Slack. (m)BaaS is (mobile) Backend as a Service and it provides some backend services, such as authentication, mainly for mobile applications.

FaaS is a subset of PaaS and means you write your function in you chosen language (or in a language that is supported by your chosen vendor) and deploy it. You also have to configure how the function is called. It can listen to events or can be triggered by an http request via an API gateway among other ways. Your vendor takes care of scaling it to your needs and you pay only for execution time. Wikipedia explains FaaS as “A category of cloud computing services that provides a platform allowing customers to develop, run, and manage application functionalities without the complexity of building and maintaining the infrastructure“.

PaaS includes lot more than just FaaS. Widely available PaaS services include messaging, databases, big data, analytics, file storage etc. They all are services you launch and configure. You can insert your FaaS function in a PaaS workflow and use all other available services with it. Again your vendor takes care of your infrastructure needs like scaling and backups and you pay for what you use. Wikipedia explains PaaS as “A category of cloud computing services that provides a platform allowing customers to develop, run, and manage applications without the complexity of building and maintaining the infrastructure“. See how it differes from FaaS?

In short, serverless means you are responsible only for your code and your data.

In Siili we have a lot of internal serverless development also on top of all the fancy stuff we create for customer. We also have cloud sandboxes freely available for all Siilis for learning and trying out serverless stuff.

Evolution of CI/CD pipeline

Back in the days there weren’t any tools for continuous integration or delivery (like Jenkins or Ansible etc). There was only ad-hoc build, manual testing and deployment done by developers from their own computers – and that was suprisingly ok (or they actually didn’t know the power of CI/CD tools yet). Actually it’s fine even today if we are looking CI/CD pipieline from a Lean perspective: to automate only the necessary parts of the value stream to get things done, BUT remembering to measure and improve the processes if needed (by creating a feedback loop and continuously measuring the process in retrospectives).

Sometimes it would be better to make your own tools instead of forcing to use ready made ones. I remember the times before Ansible when I was scripting a bit vague deployment pipeline with Powershell to deploy some PHP and Java artefacts to Windows environment alongside of .NET stuff packed into Nuget packages over TeamCity and Octopus Deploy. I know that was pretty questionable solution and I’m sure I’ll not do that again, but in the moment it felt a good enough (compromise) and I learned pretty much what it is to build things pretty much from the scratch. I found that a complex CI/CD processes could be an indicator of massive technical debt or cultural or process problems – but anyway, it was working and what was the most important thing. Customer got the solution and our team delivered a working software to production many times in a day.

Nowadays CI and CD pipelines can be presented as a code and stored to the VCS instead of managing configurations from the tool’s own GUI. Presenting configuration as a code is good because e.g. Ansible/Docker/Jenkins configurations are as valuable as the code of the main business logic and it’s fair to manage them over the same processes (for example pull requests, show-and-tell sessions , testing etc.). That approach turns the focus on the actual value and leads the change closer to DevOps culture.

Another good thing of modern CI is easier parallel processing set up. Today it’s possible to bring a flexible amount of Jenkins slaves in Docker containers if needed, or execute unit, integration and system tests in parallel by various tools like Gradle, Selenium Grid2 or Robot with pabot (for parallel end-to-end testing). Of course the cloud based CI services like Travis or CircleCI are pretty handy tools when project is suitable for using them.

But still the basic idea for me is to see the CI/CD pipeline as a part of the value stream. It’s all about choosing the right tools for the right case, or crafting the own tools if there’s no suitable one available.

More about configuration as a code: