Welcome to Cloud-Native Development with AWS Cloud9 & AWS CodeStar ⚡🎉

I have been experimenting with AWS Cloud9 since Werner Vogels announced it a few weeks ago at AWS re:Invent 2017 (keynote video here).

This article is the paraphrased version of my talk AWS Cloud9 & CodeStar for Serverless Apps, given at the AWS User Group in Milan on Jan 10th.

I am going to skip the “Serverless Disclaimer” section of my deck.If you are not familiar with Serverless, please have a look here, or here, or here, or here, or here, or here, or here.

If you are familiar with Serverless and you don’t like it, you may still enjoy this article and the benefits of Cloud9 and CodeStar. Just make sure you mentally replace “FaaS” with “Container”, and “SAM” with “CloudFormation” :)

What is AWS Cloud9?

AWS Cloud 9 is a “cloud IDE” for writing, running, and debugging code.

I’d start saying that most IDEs are fantastic tools to boost your productivity and the quality of your code if you can use them and invested a few months/years in learning them properly.

That being said, some IDEs offer more advanced and useful features than others (please don’t take it personally, unless you code in Word or Notepad).

AWS acquired Cloud9 in July 2016, which has now been rebranded as AWS Cloud9. Even though it looked brilliant on Werner’s browser, my first impression during the keynote was something along the lines of “Why should I pay to write code?”, immediately followed by “Does that mean I cannot code when I’m offline?”. Like most software engineers, I’ve used many IDEs in the last ten years, for free, and I’m used to coding a lot while I’m traveling.

Apparently, I was not alone, and many developers asked the same questions during my presentation. So let me briefly recap my arguments.

Regarding the cost, I believe it’s pretty much negligible for most organizations that already use AWS heavily (less than $2 per month for a t2.micro environment, 8 hours a day, 20 days a month). And without considering AWS Free Tier and automatic cost-saving settings (hibernation after 30min).

The “no coding offline” drawback is much harder to defend, but let me try.

Unless you are going for a hardcore debugging session over a well-established project, can you really code for more than 30min when you are offline? Can you git clone or git pull anything useful? Can you npm install or pip install the modules you need? Can you mock or ignore all the third-party services and APIs your application consumes? Can you avoid googling around your favorite framework’s documentation?

Sure, you could prepare for a 12h flight and download/install everything you need in advance. But how often does that happen? Simply put, I’ve seen the best developers and engineers give up and take a break when the network goes down.

On the other hand, AWS Cloud9 offers you a better alternative for when your machine gives up :) I could throw my dev machine out of the window right now, switch to my colleague’s notebook, login into my AWS Account and keep working on the very same Cloud9 session (which is saved and stored server-side). That means you could as well use a much cheaper machine such as a Chromebook or a tablet (or a phone?). Well, you could be 100% operational using the random machine of an internet cafe, or your grandmother’s computer :)

Of course, there are always exceptions, and I’ll make sure I’m ready to use my local IDE when Cloud9 is not an option. In the meantime, I hope AWS will work on some sort of client-side offline support (and maybe turn Cloud9 into a progressive web app?).

… and why does it matter for developers?

I think AWS Cloud9 solves a bunch of problems for the many organizations currently trying to set up elaborate stacks of tools on every developer’s machine, especially if the team is heterogeneous and/or distributed.

Let’s recap some of its features:

• It’s a full-fledged IDE (based on the Ace open-source editor)

• It comes with an integrated web terminal.It is a real ssh session directly your browser, without the need of managing and storing ssh keys or IAM credentials on your local machine.

• This web terminal can run on a new EC2 instance managed by AWS Cloud9 (EC2 env.), or you can bring your own instance (SSH env.)
• EC2 environments offer quite a handy cost-saving functionality, which means you can optionally configure them to hibernate after 30 minutes of inactivity (or more)

• EC2 environments are based on this Amazon Machine Image, which includes at least 90% of the dev tooling you need.For example, the AWS CLI, sam-local, git, gcc, c++, Docker, node.js, npm, nvm, CoffeeScript, Python, virtualenv, pip, pylint, boto3, PHP, MySQL, Apache, Ruby, Rails, Go, Java, etc.In case anything is missing, you can always install it :)

• AWS Cloud9 comes with the live debugging capabilities you’d expect from a modern IDE (currently only for Node.js, though)

• It enables collaborative coding and debugging sessions.You can share a Cloud9 environment with other IAM users and invite them with read-only or read-write permissions.

• It comes with built-in support for AWS Lambda.This greatly simplifies the process of creating new Lambda Functions, updating and testing their code locally, deploying new versions, etc.

• Plus, AWS SAM is part of the team too.AWS SAM — or Serverless Application Model — is an open specification that allows you to define serverless applications and related resources with a simplified CloudFormation syntax. Cloud9 natively integrates some of the functionalities offered by SAM Local, an open-source CLI tool written in Go by AWS to simplify local development and testing of Serverless applications. For example, you can invoke Functions locally and emulate API Gateway endpoints too.

I took the following screenshot during a live-debugging session of a very simple Lambda Function (note: I also spent 3 minutes customising theme & layout, according to my taste and needs).

AWS Cloud9 in action during a live-debugging session with Node.js

AWS Cloud9 Limitations and my personal “wishes”

I do have a few wishes for AWS Cloud9, and I’ve shared a few of them on Twitter (tweets below).

Let me discuss a few of them:

• The “numeric limitations” are pretty reasonable, in my opinion. You can create up to 20 environments per user (max 10 open concurrently), 100 per account, and you can invite up to 8 members in each environment.These are reasonable numbers since you can’t share environment across AWS accounts (yet?)

• Live debugging is terrific, but only available for Node.js. I’m looking forward to built-in support for Python (and all the others, of course).
• The same holds for some runtime customizations currently not supported. For example, I couldn’t find a way to change the default linting and code completion behavior for Python (only pylint arguments are supported). Without custom linting, Python3 developers cannot benefit from Python’s type hints capabilities (mypy is required).

• As discussed above, there is no support for “offline” development.Working offline may become a critical need in some scenarios/teams, although you can always download the whole environment to your local machine with one click (File > Download Project) and keep working locally.

• I think Cloud9 could be much better integrated with the AWS Console, especially with services such as Lambda and API Gateway. For example, there is no easy way to jump to the Lambda Console of a given Function (or API Gateway).
• The built-in Lambda integration is excellent, but it’s still not as productive as the native Lambda Console. For example, you can test a Function locally, but you can’t quickly pick the test event from a list of templates. For now, you can workaround this by running sam local generate-event in the terminal.

What is AWS CodeStar?

AWS CodeStar (aka Code-*) is a sort of “catch-all service” for the ever-expanding suite of tools for developers.It is a free service that lets you manage and link together services such as CodeCommit, CodeBuild, CodePipeline, CodeDeploy, Lambda, EC2, Elastic Beanstalk, CloudFormation, Cloud9, etc.

One of my 2018 new year resolutions is to use more memes in my decks (until someone decides to stop me, for some reason), so here’s how I presented some of the pain points that CodeStar can solve.

Too many projects tend to postpone CI/CD until it’s way too late and their productivity level is too low, just because it sounds hard. But it doesn’t have to be that hard, right?

Nobody likes building and maintaining real-time dashboards. But they are vital from day one, for a manager to assess the status of a project, as well as for a developer to monitor how the system is behaving.

This is what most developers want to do all day. Just write some code and push to master. How can you provide such a simple and frictionless experience without impacting quality and ownership?

Issue tracking can be a very frustrating experience, especially if not well integrated with source control, access control, team management, monitoring, etc.

Data-driven parenthesis: I can statistically confirm that the JIRA meme generated 42% more laughs than all others combined.

… and why does it matter for organizations?

CodeStar may not be the best fit for every project/organization, especially the most experienced and advanced ones, but it definitely provides some very good defaults to get started with. It’s worth noting that CodeStar is 100% free, and you only pay for the resources it will spin up on your behalf.

Let’s recap its features:

• CodeStar offers the concept of “project templates”.Each template represents a complete stack and includes a sample app, with a given backend, programming language, and framework.

• It supports three compute layers: EC2, Elastic Beanstalk, and Lambda.
• It supports six programming languages: C#, Java, Node.js, Python, PHP, and Ruby (plus plain HTML apps).

• It supports plenty of frameworks: Express, Spring, Django, Flask, ASP.NET Core, Laravel, etc.Note: AWS Lambda projects only support Express (Node.js) and Spring (Java), plus a few sample projects for Alexa Skills.

• Depending on the project template of your choice, CodeStar will spin up a CI/CD pipeline by linking together CodePipeline, CodeBuild, and CodeDeploy.
• Every CodeStar project starts with source control (git), either on AWS CodeCommit or GitHub. CodeStar will create the git repository for you (via OAuth in case of GitHub) and take care of triggers/hooks for CI/CD.
• You can optionally configure your own coding tools to work with CodeStar. Currently, only Cloud9, Eclipse, and VSCode are natively supported (plus the regular AWS CLI). You’ll read more on the Cloud9 integration later in this article.
• Even though many projects start as a one-person effort over the weekend, good projects tend to evolve and onboard more people quickly. CodeStar allows you to invite IAM users to your project with one of these roles: Owner (“God Mode”), Contributor (everything but team management), or Viewer (read-only dashboard access). More technical info here.
• As mentioned above, issue tracking is probably the most frustrating part of many projects, as it brings context switching, visibility, and misunderstanding issues into the equation. CodeStar allows you to integrate JIRA or GitHub Issues into your project dashboard, which might help in reducing the context switch and centralizing all the information.

• CodeStar provides a customisable app dashboard (screenshot below), which includes a project wiki section, a CloudWatch Metrics section, your project’s git history, API endpoints, open issues, Cloud9 environments, etc.You can drag-and-drop sections around as you wish and use this dashboard to check the overall status of your project quickly and effectively.

A typical CI/CD pipeline managed by CodeStar (AWS Lambda project)

A brand-new CodeStar dashboard (AWS Lambda project)

CodeStar’s team management and user roles

AWS CodeStar Limitations and a few “gotchas”

CodeStar can look like magic if you’ve never played with CodePipeline and CodeBuild, but unfortunately it’s not perfect yet. I’ve shared a few “wishes” on Twitter too (tweets below), and here’s a quick recap of what I’ve found.

• You can create up to 333 projects per account (it looks like the number came out of some kind of thoughtful calculation, right?), but you can only have 10 projects per user and 100 users per project.As for Cloud9, these seem like reasonable numbers, especially since CodeStar does not support federated users or temporary access credentials, which means you’ll need to create a lot of users if a few developers start collaborating across AWS Accounts.

• Although the default user roles seem to cover most use cases, you are stuck with the owner/contributor/viewer permissions as you can’t create custom roles. For example, you may want to have a “project manager” role with view-only and team-management permissions.
• Remember that CodeStar permissions are not related to Cloud9 permissions. If you want other owners or contributors to join your Cloud9 environment, you’ll have to invite them explicitly. Which makes sense, but I believe it could be improved/automated.
• The most critical limitation I’ve found is that there is no way to customise project templates. As I’ve mentioned already, CodeStar provides pretty good defaults for a lot of things, but you’ll probably need to change a few details here and there. For example, you may want to add a testing step to CodePipeline, add custom permissions to the default IAM role, edit the default build file, change the default API Gateway stage, etc. And once you’ve done that, there is no way to save your edits into a new custom project template so that your team will start from there. Which means you’ll have to apply your customizations to each new project. And you only have two options: 1) applying changes manually, or 2) automatically applying Change Sets to your CloudFormation Stacks (please note that every project will create one or more CloudFormation Stack and that the original templates are not versioned anywhere, so good luck with that!). Plus, see the next bullet point :)

• Some of the CodeStar functionalities are based on mysterious and undocumented CloudFormation magic such as theAWS::CodeStar::SyncResources resource and the AWS::CodeStar transform. They both sound pretty powerful, but there is no easy way to know what they are used for (or how we could use them).My current understanding is that AWS::CodeStar::SyncResources will wait for all the other resources to be deployed (i.e. DependsOn) and then make sure everything is okay (e.g. IAM permissions, project id, etc.). AWS::CodeStar seems to simply to inject SyncResources into the processed template so that we don’t have to.

What about AWS CodeStar + AWS Cloud9?

Cloud9 and CodeStar are pretty cool services on their own, and I was excited to see how they’ve been integrated. Or, better, how Cloud9 has been integrated into CodeStar.

You can associate AWS Cloud9 Environments to your CodeStar project natively. In case multiple developers are working on the same project, you can create and assign a Cloud9 Environment to each developer (eventually, they’ll collaborate and invite each other, if needed).

Once you open Cloud9, you’ll find your IAM credentials integrated with git (which does require some work) and your CodeCommit repository already cloned for you.

Unfortunately, this magic doesn’t happen if you choose GitHub (for now?).

As a couple of friends and colleagues pointed out, it’s not such a critical or technical complex integration, in the sense that you could have taken care of it yourself (as you’d do on your local machine). But I think it’s a great way to streamline the development experience and reduce the margin for error, especially when you work on multiple projects and multiple accounts.

For example, most developers make great use of AWS profiles when working on their local machine, and some of them also manage to remember which profile can do what, in which account, etc. With CodeStar+Cloud9 you won’t care anymore about profiles or local credentials since every Cloud9 environment is bound to a specific project and account. Also, since CI/CD is enabled by default, most of the time you will just write code, test with sam-local and git push 💛

Of course, you may also have a generic Cloud9 Environment (i.e. not related to a specific project) and use it with multiple profiles to manage unique resources or prototype new stuff.

Let me open a parenthesis: AWS SAM

I decided to conclude my presentation with a brief parenthesis about AWS SAM, which got a few mentions and therefore deserves some context.

** Serverless alert **

Meet SAM!

SAM stands for Serverless Application Model, and it’s an open specification whose goal is to offer a standard way to define serverless applications.

Technically speaking, it’s a CloudFormation Transform named AWS::Serverless that will convert special Serverless resources such as AWS::Serverless::Function into standard CloudFormation syntax.

You can think of Transforms as a way to augment the expressiveness of CloudFormation templates so that you can define complex resources and their relationships in a much more concise way.

If you are familiar with other tools such as the Serverless Framework, you’ll notice that the syntax is quite similar (there is even a plugin to convert your templates to SAM).

In fact, you can deploy SAM templates with AWS SAM Local, a CLI tool for local development written in Go and officially released by AWS.

You can use AWS SAM Local to test your Lambda Functions locally and emulate API Gateway endpoints too. The CLI tool is available by default on every Cloud9 EC2 Environment, and the UI already supports some of its functionalities.

A few SAM examples

A simple serverless Function (additional properties are available for IAM policies, VCP config, DLQ, tracing, etc.)

The same Function defined above, plus CodeDeploy Traffic Shifting (10% every 10min)

The same Function defined above, plus an API Gateway endpoint (nothing else needs to be defined)

A simplified DynamoDB definition (only primary key and throughput)

An API Gateway defined in swagger format (sam-local will package it to S3 before deploying!)

My personal “wishes” for AWS SAM

I have only one wish for AWS SAM: I would love to see more transparency and documentation related to the AWS::Serverless Transform.

Apr 2018 Update: AWS SAM is now open-source on GitHub!

And since I like dreaming, why not allowing custom CloudFormation Transforms too? I am almost ready to bet they are implemented with some kind of Lambda hook, and I can’t even start to imagine how many great things the community might be able to develop and share that way.

I hope you learned something new about AWS Cloud9 and CodeStar (please don’t confuse them and create weird hybrids such as “CloudStar” as I did a few times). I would recommend building a simple prototype or a sample project on CodeStar asap. You can get started here!

If you got this far, you probably enjoyed the article or feel like sharing your thoughts. Either way, don’t forget to recommend & share, and please do not hesitate to give feedback & share your ideas =)