Several months ago, we faced one interesting problem while we were working on a project using distributed approach. In our registered user scenario, we needed to register a user in several different systems.
We started building Lambda function for each system where a user needs to be registered. For example, we had one Lambda function for registration in Active directory, one for registration in system for Products and one for registration in system for Promotions.
In order to return immediate response to the user that is in process of registration and to do registration in asynchronous manner, we had one Lambda function that creates messages and pushes them to SQS. Then, SQS and SNS mechanism woke up Lambda functions for separate systems. So far – so good.
But, after everything was finished, we got a new requirement – notify the user when the registration is successfully finished. Interesting, isn’t it? – Keeping track of all Lambda functions that are working asynchronously. Ok, we can save the state in a database, introduce a custom solution that will handle different scenarios… everything is really time consuming and will require changes to our already existing Lambda functions.
AWS has the answer – Step functions. This AWS service allowed us to introduce another Lambda function for sending e-mail and also a rollback scenario for the entire registration flow. Let’s dig into the details of how Step functions helped us solve the problem.
Understand all parts of your business flow
In order to create well functioning step functions, you need to understand every part of your workflow and how they communicate to each other. From there, you can create separate Lambda functions with well defined input and output parameters.
In our case, we will have the following Lambda functions:
[table id=3 /]
When a user provides data for registration, in parallel we make calls to Active directory, Products and Promotions systems. When the operation in each system is done, we need to send a confirmation e-mail to the user. In case of an error – we need to rollback changes done in each system.
Create a state machine
Based on the flow, define your state machine using the Amazon States Language (ASL), and review the visual representation of your workflow. In our case it looks like this:
Follow the simple wizard that will guide you through the creation of a state machine. It will create the state machine and IAM role for allowing state machine to invoke your Lambda functions.
Lambda functions and a complete flow is defined using the following code snippet:
{
"Comment": "Parallel Example.",
"StartAt": "RegisterUser",
"States": {
"RegisterUser": {
"Type": "Parallel",
"Next": "SendEmailNotification",
"Catch": [{
"ErrorEquals": ["States.ALL"],
"Next": "RollbackRegistration"
}],
"Branches": [
{
"StartAt": "RegisterUserInAD",
"States": {
"RegisterUserInAD": {
"Type": "Task",
"Retry" : [
{
"ErrorEquals": [ "States.ALL" ],
"IntervalSeconds": 10,
"MaxAttempts": 2,
"BackoffRate": 1.5
}
],
"Resource":"arn:aws:lambda:us-east-1:<accountNumber>:function:register-user-in-AD",
"End": true
}
}
},
{
"StartAt": "RegisterUserInProducts",
"States": {
"RegisterUserInProducts": {
"Type": "Task",
"Retry" : [
{
"ErrorEquals": [ "States.ALL" ],
"IntervalSeconds": 10,
"MaxAttempts": 2,
"BackoffRate": 1.5
}
],
"Resource":"arn:aws:lambda:us-east-1:<accountNumber>:function:register-user-in-products",
"End": true
}
}
},
{
"StartAt": "RegisterUserInPromotions",
"States": {
"RegisterUserInPromotions": {
"Type": "Task",
"Retry" : [
{
"ErrorEquals": [ "States.ALL" ],
"IntervalSeconds": 10,
"MaxAttempts": 2,
"BackoffRate": 1.5
}
],
"Resource":"arn:aws:lambda:us-east-1:<accountNumber>:function:register-user-in-promotions",
"End": true
}
}
}
]
},
"SendEmailNotification": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:<accountNumber>:function:send-email-notification",
"End": true,
"Catch": [{
"ErrorEquals": ["States.ALL"],
"Next": "RollbackRegistration"
}]
},
"RollbackRegistration": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:<accountNumber>:function:rollback-register-flow",
"End": true
}
}
}What this code means
The code is written using Amazon States Language (ASL). Amazon States Language is a JSON-based, structured language used to define your state machine, a collection of states, that can do work (Task states), determine which states to transition to next (Choice states), stop an execution with an error (Fail states), and so on. In our case, we wanted to have parallel processing on three Lambda functions, so we used state of type parallel:
“States“: {
“RegisterUser”: {
“Type”: “Parallel“,
…..
Under branches we defined three parallel tasks. After the parallel state is finished, we want to send a notification. That is defined with “Next”: “SendEmailNotification“. In case of an error, we want to call rollback Lambda function:
"Catch": [{
"ErrorEquals": ["States.ALL"],
"Next": "RollbackRegistration"
}],
In each parallel task, we have the retry mechanism:
“Retry”:[
{
“ErrorEquals”: [“States.ALL”]
“IntervalSeconds”:10,
“MaxAttempts”:2,
“BackoffRate”:1.5
}
]Defined with 2 attempts in interval of 10 seconds in case of every error that may arise. ‘BackoffRate’ is multiplier by which the retry interval increases during each attempt. With the retry mechanism we try 2 times before we raise an error. More info about ASL can be found on the links at the end of the blog.
Execute and see the results
Now let’s try and see what step functions can do. Navigate to definitions of your step function:
and Start Execution by defining name of execution and input parameters. In our case input JSON is:
{
"firstname": "John",
"lastname": "Johonovan",
"username": "jjohn",
"email": "jjohn@test.com",
"phone":"123456"
}And here is the execution path:
If you select each state you can see details about input and output values:
State machine is taking care of exchanging input and output parameters between states. All you need to do is in your Lambda function to define expected input parameter and defined output as valid JSON. Example:
exports.handler = async (event) => {
…….
console.log('Event in register-user-in-AD is:', event);
….
if(event.error) {
throw new Error(event.error);
}
const response = {
status: true,
email: event.email
};
return response;
};Execution event history tab gives great info for I/O from each execution step.
Conclusion
Step Functions give out-of-the-box solution for state machine and save vast amount of time for setting your own one. Step functions allow you to focus on the development of business logic inside lambda functions and move the logic bricks in right flow. Everything else is done by the Step Functions itself.
More info on:
https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html https://docs.aws.amazon.com/step-functions/latest/dg/concepts-amazon-states-language.html
