Home » ci-cd
Tag:

ci-cd

GeneralTutorials

5 Best DevSecOps Tools in 2023

by corey.walker
written by corey.walker 9 minutes read

In today’s world, cybersecurity is constantly growing in importance for individuals as well as businesses. It is also constantly evolving and changing with different tactics and methods of exploitation by cybercriminals. However, it’s almost impossible to make sure every possible avenue of attack is fortified against exploitation. The bigger a software application becomes or the more apps & services are in use for a software development process, the bigger the opportunity for cyber criminals to find a weakness. That’s why automating as much cybersecurity into your software development process is ever-growing in importance. While automating a lot of the software development lifecycle is known as DevOps, including security-related functionality into that workflow is what’s known as DevSecOps.

DevSecOps is a great way to get a leg up on cybersecurity issues and catch them before they are available to be exploited by cybercriminals. Utilizing services and tools specific to cybersecurity within your software development lifecycle is not very difficult and can save a lot of manual effort and possible headaches from security incidents later on down the line. It just takes some upfront time to set up and configured correctly, and then just tweak things as you discover areas that need more or less stringent security based on your situation.

In this post, we will go over 5 of the best DevSecOps tools available to integrate into your software development lifecycle right now to greatly enhance your security posture for your organization without a ton of effort. They are in no particular order.

5 Best DevSecOps Tools in 2023

  1. OWASP ZAP
  2. SonarQube/SonarCloud
  3. Snyk/GitHub Dependabot
  4. Infrastructure as Code
  5. Monitoring

OWASP ZAP

OWASP is the Open Worldwide Application Security Project. It was founded on Dec 1, 2001, as a way to help individuals and businesses better secure themselves online.

OWASP annually comes out with a list of the most exploited security flaws in web applications across the entire internet. This is known as the OWASP top 10 and is used by the majority of cybersecurity personnel as a basis for what to focus on the most at any particular time. Cybersecurity professionals focus on many more areas than just these 10, but this list helps them determine what is being the most exploited security flaws at any given time.

OWASP created a piece of software called OWASP ZAP (Zed Attack Proxy) that can automatically crawl and scan a website and try multiple types of attacks against the site and report any findings it discovered may be an issue. It will try many of the same tactics used by cybercriminals when they try to attack websites. A lot of these automated tasks won’t help with more advanced attacks or attacks stemming from social engineering attacks, but they will help with more technical-related attacks.

When running this software against a website, it is possible that it can bring down your website or corrupt your data, so running it against your production website is ill-advised. How it should be used is deploying the website to a “staging” environment that is set up exactly like your production website. Make sure to NOT have this staging environment connected to production data sources but have testing data sources instead. Then run OWASP ZAP against the staging website and see what results come from the scan. If everything looks good, you can deploy changes to the production environment. If any findings are discovered by the scanner, review them and make any necessary changes, redeploy to the staging environment, scan once more, and then deploy to production once all issues are resolved.

This software is free to download and use and is open source and is a great addition to any website-related software projects.

SonarQube/SonarCloud

Whereas OWASP ZAP scans your website once it has been deployed (known as dynamic code scanning), SonarQube/SonarCloud is a product/service that will scan the source code itself before it is deployed and alert on any possible security issues related to the source code. This is known as static code scanning. It looks for things that can be exploited. Things such as not sanitizing user input which could cause things like XSS, CSRF, SQL injection, etc, or things like bad regular expressions (regex) that could cause things to consume way too many resources and cause a Denial of Service (DOS) attack, buffer overflows, etc.

This is a great addition to have in your DevOps pipeline since source code bugs have notoriously been a great source of exploited security issues. Put this hook in your SCM repository so any findings that are of a certain severity or higher will reject the change from being merged into the branch that gets pushed to production. You are guaranteed run see numerous false positives that you will have to mark as such and then run the scan again to be able to merge the code changes. But once you mark specific areas of code as “good” within SonarQube/SonarCloud, it will not flag them in subsequent scans.

This software is not free but SonarCloud can be as little as €10/month. SonarQube is software that you can license and run on your own hardware, whereas SonarCloud is Software as a Service (SaaS).

Snyk/GitHub Dependabot

Today software projects almost always have software package dependencies that are utilized to reduce the amount of code that has to be written for specific functionality. If someone else has already written something that does exactly what you need, and it doesn’t cost anything, why should someone spend the time and effort to write code that does the exact same thing? This is where things called package managers such as npm, yarn, pip, Apache Maven, etc come into play. They allow you to easily install packages that another application can easily utilize and save developers a lot of time and effort.

However, many times, these dependencies have security flaws in them that can also cause the application using them to have security issues. This is where Snyk and GitHub Dependabot come to the rescue. These services will scan all the packages that have been included in the host application and alert when there are updates available and when there are known security issues with the dependent software packages. This allows for the host application maintainers to either change the package being utilized or update the package to remediate the vulnerability. Without these automated scans and notifications, developers would have to constantly manually check all the dependent packages to see if there were any known vulnerabilities in any of them. When there are numerous projects being managed by a team and each project has numerous dependencies, that would be very inefficient and a waste of people hours.

These services have a limited free tier to use but offer paid plans for more features and more scans/repositories that can be scanned. So small teams with only a few repositories should be able to utilize the free plan(s), whereas larger organizations with more repositories or need additional scans will need to pay for the services.

Infrastructure as Code

A great way to make sure you have a consistent environment where applications are deployed is by utilizing what’s known as Infrastructure as Code. This is where all infrastructure is defined within files and stored within an SCM repository instead of being manually provisioned each time a change is needed or new infrastructure is needed. The files have everything needed declared within the files themselves and can be used multiple times to set up multiple instances the exact same way every time.

This makes it much easier to make sure everything is configured in a secure manner and replicated across every instance. It also allows for much quicker deployment of new resources when needed as each piece is not having to be done manually. Having infrastructure defined in source code files also allows for the history of changes to be tracked so that if anything negative happens after a change, it’s much easier to determine what may have caused the issue. It also gives the ability to quickly revert changes and redeploy using the previous version of the source files to a known working configuration.

There are multiple providers for Infrastructure as Code such as AWS CloudFormation, RedHat Ansible, HashiCorp Terraform, Puppet, Chef, and others. It is advised to research each to determine what is best for any given situation since each has pros and cons. Some of these also are not completely free while others are. There are also some that are specific to a particular cloud provider while others are provider agnostic.

In-Depth Monitoring

Security concerns do not stop after it’s deployed to production. Monitoring running applications is a must in order to see many issues that may not have been discovered before deploying to production.

If monitors discover a lot of outgoing traffic from your network from a particular set of IPs that have not been seen often before, it could be an indication that threat actors may be exporting a lot of private data and should be looked into immediately. Monitors may also be able to catch bugs in software that weren’t discovered in testing but are being seen by end users. This could be because initial testing didn’t think of the ways end users may either be trying to abuse the application or because how end users are just doing things not initially thought of by the development team. These monitors could show where crashes or issues are occurring that may pose a threat to security if cyber criminals were to discover and try to exploit them. This gives the chance to possibly correct these issues before any security incidents occur if the correct monitors and alerting are in place.

There are many platforms that can be utilized for monitoring and alerting. Some examples are New Relic, Datadog, AWS CloudWatch, Sentry, Dynatrace, and others. Again, these providers each have pros and cons related to pricing, offering, ad vendor lock-in. So research the options to see what may possibly be best for a given situation.

Conclusion

There are many aspects to helping to prevent cybersecurity incidents in any organization. DevSecOps is not the complete solution just as no single aspect can be. Cybersecurity has so many different aspects and avenues of attack that it takes everyone in an organization to always be thinking about it and trying to help prevent any attacks. That’s where internal company training is always needed.

However, automating as much security as possible is always a great investment to make in any organization. Having things set up to help protect you from specific types of attacks and not having to think about those particular pieces of the puzzle will greatly enhance the security posture of any organization and any software that the organization creates and customers utilize.

These may be the top 5 most important DevSecOps tools you can easily incorporate, but that doesn’t mean they should be the only ones. So now go off and secure those software development workflows!

0 comments
0 FacebookTwitterLinkedinTumblrRedditWhatsappTelegramCopy LinkPocketEmail
Getting Started with GitHub Actions
GeneralTutorials

Improve Your DevOps Workflow With GitHub Actions

by corey.walker
written by corey.walker 8 minutes read

GitHub Actions has quickly become one of the most popular Continuous Integration (CI) and Continuous Delivery (CD) service platforms. It provides many great features for automating your software development workflow as well as great integrations with GitHub itself. GitHub has long been the largest SCM provider available for many reasons. They have many great features that make the software development process easier, and more robust, tons of great integrations with other tools and services, as well as having a very good uptime record. Now, GitHub has also added the ability to control your CI/CD processes directly within GitHub itself without having to rely on another service or constantly swap screens between the multiple services. In this post, we will go over how to improve your DevOps workflow with GitHub Actions!

Current State of CI/CD Platforms

CI/CD services rely on adding configuration files to your project repository. These files are config files that will tell the CI/CD service exactly what you want it to do and when to do those actions. Actions can be configured to trigger upon every commit that is pushed to the repo, only when merge requests are merged into a certain branch, as well as many other options. The reason we put these config files in the repo itself is for a few reasons. 

  • You can keep track of all changes to the configuration files. So in the case any errors or issues arise after making changes, you can revert to the previous configuration easily
  • You can easily copy a configuration file to another project repository and make small changes if the project is similar enough to the original project. This cuts down on having to manually go through a UI form and set the same values over and over which is time-consuming and error-prone

Now, let’s see how we can improve our DevOps workflow with GitHub Actions!

GitHub Actions

GitHub Actions is no different in that it requires a specific directory structure to be in place for the different parts GitHub Actions provides. The different sections of GitHub Actions are as follows:

  • Workflows are under .github/workflows/
  • Actions are under .github/actions/

Workflows are the total combined steps you wish to perform upon a certain action being taken on the repo (commit push, pull request merged, etc). Actions are more specific steps for a particular piece of the workflow that is complex enough and used in multiple places so that it is better to be separated into their own action that can be referenced in multiple workflow files instead of repeating the same steps in multiple places. This cuts down on the problem where when something changes, you may miss updating certain places which can cause issues in any subsequent builds. 

Language Options

GitHub Actions supports a multitude of language options for builds. The list of all languages supported by GitHub Actions can be found here which includes almost all (if not all) of the most popular languages being used today.

Setting Up GitHub Actions

We will start off with the customary “Hello World!” example. First, create a directory in the root of your GitHub repo called .github/workflows. Inside that directory, create a file named example.yml. Inside that file, add the following:

name: example
# Run this workflow every time a new commit pushed to your repository 

on: push

jobs:   
  # Set the job key. The key is displayed as the job name   
  # when a job name is not provided   
  test:     
    # Name the Job     
    name: Run Example     
    # Set the type of machine to run on
    runs-on: ubuntu-latest
    
    steps:       
      # Checks out a copy of your repository on the ubuntu-latest machine       
      - name: Checkout code         
        uses: actions/checkout@v2
      - name: Say 'Hello World'
        run: 'echo "Hello World!"'

Let’s step through this line by line:

  • Line 1: This is just the name of the workflow. You can put anything you want here. Just make it something that makes sense for what this workflow does
  • Line 4: on: push tells GitHub Actions that this workflow is to be run anytime a commit is pushed to any branch in this repository. You can customize this to set it to only run on for certain branches etc. See additional options here.
  • Line 6: jobs are groupings of steps that are performed in sequence. Jobs run in parallel by default, but can be configured to be run sequentially
  • Lines 8 and 10: Set the name and what OS to run the steps on for the test job. 
  • Line 12: steps is the grouping of individual actions that are run on the specified runner. You can use shell commands, external scripts, or predefined/shard GitHub Actions for steps
  • Lines 14-17: Lines 14-15 are specifying a GitHub Action to use to check out the code for the repo onto the system running our workflow. We provide a name for it so it shows us as an understandable step in our workflow in the GitHub Actions UI. When specifying a GitHub Action, we use the term uses: instead of run: which is used for shell commands as seen in lines 16-17

Testing Out GitHub Actions

Now, add these files to a new commit and push them to your repo on GitHub. You may not see a run of these the first time you push since the configuration wasn’t present when you pushed this commit. So try adding a simple change to a file such as a README.md and push that commit to your repo. 

Now go to your repo in GitHub and click on the Actions tab which should be beside the Pull Requests tab. Click on the name of the workflow you provided in your configuration file (you can also just leave it on All workflows for now if you wish).  You should see something similar to the below screen.

GitHub Actions Workflow Screenshot
Click on the top workflow run entry in your list. You should now see a screen similar to this.

GitHub Actions job tasks screenshot
GitHub Actions job tasks screenshot

Now click on Run Example in the grey area to open up the actual steps performed. You should now see a screen similar to this

If you click on the Say ‘Hello World’ entry, you should see an output of Hello World to the console. Congrats! You have now successfully configured your repository to use GitHub Actions!


You can check out this example here.

Improving What We Already Have

Let’s improve this a little bit. Let’s say you want to add the “Hello World!” step to multiple workflows. Awesome, we can just copy that step name and command over to all other workflow files, right? Well, we could… But what if we instead wanted each workflow to say “Good [Morning|Afternoon|Evening] World!”? We would have to go and change every workflow file that we had copied that step over to. 

A better way would be to create a custom GitHub Action under .github/actions/ and then reference that action in all of our workflow files. That way, when we needed to update what we output in each workflow, we just update it in a single place and all workflow files are automatically updated! That’s much better don’t you think?!


Ok. So let’s create a new file .github/actions/greeting/action.yml. In that file, add the following: 

name: "Greeting" 
description: "Outputs a greeting in workflow output" 
outputs:   
  greeting: # id of output   
    description: "Final Greeting" 
runs:   
  using: 'node12'   
  main: 'greeting.js'

Now, create a new file named greeting.js in the same directory as the action.yml file above. Add the following to the file: 

const core = require('@actions/core'); 
const github = require('@actions/github');
try {   
  let timeOfDay = null;   
  const date = new Date();   
  if (date.getHours() <= 11) {     
    timeOfDay = 'Morning';   
  } else if (date.getHours() > 11 && date.getHours() <= 17) {     
    timeOfDay = 'Afternoon';   
  } else {     
    timeOfDay = 'Evening';   
  }   
  console.log(`Good ${timeOfDay} World!`);   
  core.setOutput("greeting", `Good ${timeOfDay} World!`); 
} catch (error) {   
  core.setFailed(error.message); 
}


Update the workflow file under .github/workflows/ and make it match the following: 

name: example

# Run this workflow every time a new commit pushed to your repository 
on: push

jobs:   
  # Set the job key. The key is displayed as the job name   
  # when a job name is not provided   
  test:     
    # Name the Job     
    name: Run Example     
    # Set the type of machine to run on     
    runs-on: ubuntu-latest
    
    steps:     
      # Checks out a copy of your repository on the ubuntu-latest machine     
      - name: Checkout code       
        uses: actions/checkout@v2     
      - name: Install dependencies for custom actions       
        run: npm i @actions/core @actions/github     
      - name: Display Greeting       
        uses: ./.github/actions/greeting


Now when your workflow runs, it will print a greeting based on what time of day it is! You can find this example configuration here.

Conclusion

That’s it! You have now created a customized GitHub Action that you used in a custom GitHub Action Workflow! You are now ready to improve your DevOps workflow with GitHub Actions! For more information on GitHub Actions and all the other options available as well as some security tips for Actions, check out GitHub Actions docs.

0 comments
0 FacebookTwitterLinkedinTumblrRedditWhatsappTelegramCopy LinkPocketEmail
How To Become a DevOps Engineer
Tutorials

How To Become A DevOps Engineer – 3 Focus Areas

by corey.walker
written by corey.walker 7 minutes read

DevOps is quickly becoming one of the most sought-after specialties in the software development world. DevOps professionals help organizations automate many processes they currently do manually or semi-manually. This saves companies a lot of money in man-hours as well as makes their processes less prone to human error. DevOps engineers also are responsible for setting up the monitoring of resources and services that the company relies on. Companies desperately need professionals to set up, configure, and maintain all alerting mechanisms so recovery is as quick as possible. Every minute these services are out costs the company money, thus they want to recover as quickly as possible. This is one of the primary focuses of DevOps engineers. 


In this post, I will go over 5 main areas of focus that DevOps engineers must be fluent in and know very well. Enough of the introduction, I bet you want to see what areas to focus your attention on huh?? Let’s jump in!

Check out our related YouTube video below:

Monitoring

Monitoring is one of the biggest and most important parts of maintaining software services. Computers are very predictable and repeatable compared to humans, but they do have errors that crop up for varying reasons. You can set up services to be robust enough to recover from some of the most common errors. However, there will ALWAYS be errors that come up that you don’t anticipate. Monitoring allows immediate detection when things are not functioning properly and immediately notify the correct team to fix the issue.


There are many popular monitoring platforms available to choose from. Some of the most popular are:

  • New Relic
  • DataDog
  • Sentry
  • Firebase
  • AWS CloudWatch

There are many ways to monitor different types of services. There isn’t a one-way fits-all to cover everything. You need to dig in and know what your service(s) are used for and the best way to monitor them. However, there are different avenues of monitoring that when combined offer the best coverage for your services. 


Internal Monitoring

Monitoring your services from within your network allows you to quickly notice any errors that may crop up before end-users see them. This can allow you to react quickly if caught soon enough and remediated quickly. However, monitoring from within the same stack/network has major downsides. If your network goes down or somehow loses network connectivity, your internal monitors won’t be able to detect that kind of issue. So let’s look at how we can get a more comprehensive monitoring solution in place. 


Internal monitors normally involve more details that alert for issues that don’t affect the whole service but can affect some users. The internal monitors include much more error details in the alerts as well as more frequent tests than external monitors.


External Monitoring

Internal monitoring is great and a needed aspect to monitor your resources and services. External monitors that are hosted by a third-party service or in a separate geographical region and/or separate network will need to be configured. Thus, if the whole network goes down, your external monitors can detect this and alert when internal monitors are unable to function properly. 


External monitors are normally not as intensive and frequent as internal monitors but instead focus more on making sure the overall service is functional and has a good user experience (speed, no consistent errors with good requests, etc). These are normally not run as frequently as internal monitors but are a lifesaver when internal monitors cannot detect issues.


Learning how to figure out what to monitor and how to set up these monitors and when to alert is part art/part science.. Experience with setting up monitors and alerts, seeing other errors that weren’t monitored, and then adding monitors for those errors, etc is a nonstop loop of learning and tweaking to get it working just the way you want. 

Continuous Integration and Continuous Delivery (CI/CD)

One of THE most important things a DevOps engineer should know well is Continuous Integration and Continuous Delivery (CI/CD). Git source code management allows for multiple people to work on different features or bugs in a project at the same time. Then once changes are ready for production, they can “merge” their code into the production branch. This is different compared to some source code management platforms that “lock” files when someone checks out a file. No one else can check out or edit the file until the original editor checks the file back in. This can slow down development if multiple people are trying to update the file at the same time. 
Some of the most popular CI/CD providers as of this writing are:

You can check out some of my posts comparing the above providers here.


Continuous Integration

Continuous Integration allows you to make smaller frequent updates to your codebase instead of many changes all at once. This is especially critical when working with Git as your source code repository tool of choice. Small frequent changes allow for much fewer conflicts when merging as well as easier conflict resolution when conflicts do occur. Unit tests and sometimes acceptance tests are automatically triggered to run when changes are pushed to a repository. This helps to ensure any new changes don’t break existing business logic. If changes do break tests, the changes are small enough that narrowing down the cause of failure is much easier.

Most software projects historically followed what is known as the waterfall method. Of course, this process RARELY ever worked efficiently. Developers would go back and forth multiple times to fix all errors and then try to combine their changes again. This normally took quite a long time and added a lot of time to the software development lifecycle. 


DevOps is focused more on a continuous flow instead of teams working in silos for extended periods of time. Continuous integration is part of that philosophy. This allows for teams to test out very small changes quickly and deploy those changes rather than fewer large changes. 


Continuous Delivery

You want to enable automatic deployments to your different environments once you have automated testing of changes in place. This allows you to manually test your changes in your development and staging environments before merging them into your production git branch in your repository. If you have adequate monitoring of your development and staging environments, these tests should let you know if your changes broke anything. This lets you verify everything works as expected before pushing it to production.

Automated Testing

You need to be sure to have very thorough automated tests in place when using CI/CD. That way, you can be very confident that your changes did not break any functionality when pushing new changes. Without thorough automated tests, you cannot fully experience the depth of what DevOps can provide over conventional software development lifecycles. 


Unit tests need to cover a very high percentage of your code base. You cannot be confident using only automated test results unless your tests cover a majority of lines in your code. You can utilize different code coverage frameworks and file output types when testing code coverage. There are many providers that allow you to upload those files to keep a history of code coverage changes throughout the history of your project. Some include Coveralls, Code Climate, Codecov, and more. 


With a very good suite of automated tests and high test coverage percentage, you should be comfortable utilizing continuous deployment to production without manual reviews. This happens a lot with many newer technology companies where they deploy to production numerous times a day. This allows for deployments that can possibly cause issues in production. However, when the development lifecycle is agile and quick from start to finish with small changes, a bug deployed to production can be quickly remediated and pushed to production as well.

0 comments
0 FacebookTwitterLinkedinTumblrRedditWhatsappTelegramCopy LinkPocketEmail
GitHub Actions vs Travis CI logos image
GeneralTechnology Comparisons

Which is Better? GitHub Actions vs Travis CI

by corey.walker
written by corey.walker 7 minutes read

In my previous post, I discussed the differences between GitHub Actions vs CircleCI. In earlier posts, I have also compared Travis CI vs CircleCI, Travis CI vs Jenkins, and CircleCI vs Jenkins. This post includes what is arguably the most popular newcomer to the group of CI/CD service providers, GitHub Actions. Do you know which is better for your needs? Let’s compare the two platforms and lay out the differences between each, and provide a list of what I believe are the pros and cons of each. Let’s get started!

GitHub Actions

This is an awesome new player in the CI/CD pipeline space provided by GitHub. They have been around for a long time and are also the most popular SCM provider in the world. GitHub offers great features and a great user experience for their SCM offerings. 

GitHub Actions recently joined the CI/CD pipeline space back in November 2019. It was an instant hit among many companies and people that were already using the platform. This allowed for easier integration with code already hosted on GitHub, as well as the option to reduce the number of services you need to subscribe to or purchase to have a complete CI/CD workflow. This is obviously a great feature. It also allows for much easier viewing of the pipeline status for each build since you do not have to leave GitHub’s website to see any logs and/or errors; the tab is right there as part of your repository’s UI. 

Screenshot of GitHub Actions tab in GitHub web UI

Features

You can utilize containers or VMs in your builds and also run on either GitHub.com or on-prem.

Use your own VMs, in the cloud or on-prem, with self-hosted runners.

There is support for the most popular programming languages today. These include Node.js, Python, Java, Ruby, PHP, Go, Rust, .NET, and more.

Just like Travis CI, GitHub Actions is also available to build on Linux, macOS, and Windows.

One great feature is the catalog of community-provided Actions you can add to your workflow without re-inventing the wheel. Don’t waste your precious time doing something that someone else has already done! Wondering if someone has already created the action you need in your workflow? Check out the catalog of available Actions here.

Build Matrix is another great available feature. This allows you to run multiple builds or tests simultaneously using different language versions, platform versions, etc. This can save quite a bit of time and also allow you to see if any certain language versions or platforms break with your new changes. For more information on creating a build matrix and what it can help with, check out this article.

Configuration

The configuration of GitHub Actions is very similar to that of CircleCI. You create a .github/workflows directory and create YAML files inside that directory. You can create different YAML files for each job or add them all to a single file if you wish. For a full list of available references that you can add to your config files, check out the reference guide. For a simple example of how to set up a configuration file, check out our example here.

Pricing

The pricing structure is based on paying for the amount of time your CI/CD pipelines are running per billing period. You get ~2K action minutes per month for free. Whereas Travis CI offers unlimited builds for their payment plans, they do not, however, offer a free tier except for open source projects (i.e. public repositories).

Travis CI

Travis CI is the oldest and most mature cloud-hosted CI/CD provider. It was founded in 2011 and is based in Berlin, Germany. Travis CI is among the most popular CI/CD platforms for open source projects. It has support for a variety of languages with multiple pricing plans available to suit most peoples’ needs. Travis CI is currently only available for GitHub and BitBucket.  

Configuration

Travis CI can be configured as easily as just creating a file in the root of your SCM repo called .travis.yml. You can find how to configure this file and all the available options here. This allows for the ability to copy the configuration file to multiple repositories and just edit the relevant details specific to that repo and everything will work. 


Some CI/CD tools require setting up the process manually to start with, and each time you create another project, you have to go through those same steps to set it up each time. Some steps could be overlooked or missed from time to time, which is not ideal. This can cause some of your automated processes to not work correctly, and thus possibly cause issues down the road if it goes unnoticed.

You can find steps to set up Travis CI with GitHub here. Steps to set up Travis CI with BitBucket are found here.  

Pricing


Travis CI is always free for open-source projects. For projects with private repositories, there is a free tier that offers 10K credits that once used up, you are unable to run any more builds for private repositories. This 10K credit allotment will allow for an estimated 1000 minutes of build time.

For unlimited builds for private repos that run on Windows, Linux, or FreeBSD, their pricing starts at $69/month for 1 concurrent build. For each paid plan, macOS builds now require an extra add-on of renewable credits. This means you no longer have unlimited macOS builds included in Travis CI’s paid plans. However, if you do not have any builds requiring macOS builds, or you have a limited amount of macOS builds, this may not be a huge change in pricing as $15 will get you @5K credits which will allow for roughly 500 macOS build minutes. See all details for pricing plans here.

Supported Languages

Travis CI supports a wide variety of programming languages. Currently, they support 32 different languages that are listed here and here. You can find docs for how to submit support for a new language here.  

Comparision

GitHub Actions Pros

  • One interface for both your source code repositories and your CI/CD pipelines
  • Only 1 bill to pay that includes both your SCM plan and your CI/CD service (if you pay for either now that is)
  • Build Matrix
  • Catalog of available Actions you can utilize without reinventing the wheel
  • Includes macOS builds as part of the free tier (macOS builds require 10 credits per build minute however compared to 1 credit per minute for Linux)
  • Free tier credits are renewed/reset each month 

GitHub Actions Cons

  • Newer platform. Thus not as quite many features as Travis CI as of yet (although they sure have had a strong start)
  • Only available for project repositories hosted on GitHub or GitHub Enterprise

Travis CI Pros

  • Build Matrix
  • Has a free tier (although not as many free build minutes). Private repo credits do not renew. However open-source projects are always free
  • More widely used and thus more community-based support
  • Can be hooked up to more SCMs than GitHub or GitHub Enterprise

Travis CI Cons

  • Free tier credits do not renew every month
  • Not as many available plugins to help ease the configuration of actions others have already completed
  • Need multiple subscriptions for both SCM and CI/CD services
  • Switching back and forth between CircleCI and SCM to see build logs and errors

Conclusion

The best platform for you all depends on your specific needs. If you host your code somewhere other than GitHub.com or GitHub Enterprise, you will be better off using Travis CI. However, if you host your code on GitHub.com or GitHub Enterprise, the choice can be a little more difficult. 

For most people, if they have a small number of projects whose CI/CD config files are relatively simple, I would suggest giving GitHub Actions a try. Don’t disconnect or delete your current pipelines until you try GA out for a project or two, however. Just add the config files in for GA and see how you like the flow and feel of it. Once you verify that GA has everything you need to do your builds properly and successfully, and they have been running for at least a week or two successfully, you can delete your other config files and cancel your previous CI/CD service.

If you are interested in any other CI/CD platforms and are curious about the differences in all of them, go check out my posts comparing GitHub Actions vs CircleCI, Travis CI vs CircleCI, Travis CI vs Jenkins, and CircleCI vs Jenkins.

Please let me know which platform you think is better or that you prefer using and why in the comments!

0 comments
0 FacebookTwitterLinkedinTumblrRedditWhatsappTelegramCopy LinkPocketEmail
GitHub Actions vs CircleCI logos image
GeneralTechnology Comparisons

Which is Better? GitHub Actions vs Circle CI

by corey.walker
written by corey.walker 9 minutes read

In earlier posts, I have compared Travis CI vs Circle CI, Travis CI vs Jenkins, and Circle CI vs Jenkins. In this article, I’ll be including what is arguably the most popular newcomer to the group of CI/CD service providers. Do you know which is better for your needs?

One of the newest platforms to offer the CI/CD pipeline space is GitHub Actions. Another option that has been around for a few years now and is a great candidate for any CI/CD pipelines is Circle CI. Circle CI is one of the most popular cloud-based CI/CD pipeline providers available today. In this post, I will compare the two platforms and lay out the differences between each, and provide a list of what I believe are the pros and cons of each. Let’s get started!

Check out our related YouTube video:

YouTube Thumbnail – GitHub Actions vs Circle CI

GitHub Actions

This is an awesome new player in the CI/CD pipeline space provided by GitHub. They have been around for a long time and are also the most popular SCM provider in the world. GitHub offers great features and a great user experience for their SCM offerings. 

GitHub Actions recently joined in the CI/CD pipeline space back in November 2019. It was an instant hit among many companies and people that already used the platform. This allowed for easier integration with code already hosted on GitHub, as well as the option to reduce the number of services you need to subscribe to or purchase to have a complete CI/CD workflow. This is obviously a great feature regarding GitHub Actions. It also allows for much easier viewing of the pipeline status for each build since you do not have to leave GitHub’s website to see any logs and/or errors; the tab is right there as part of your repository’s UI. 

Screenshot of GitHub Actions tab in GitHub web UI

Features

You can utilize containers or VMs in your builds and you can also use GitHub Actions to run on GitHub or on-prem.

Use your own VMs, in the cloud or on-prem, with self-hosted runners.

GitHub Actions also has support for the most popular programming languages today. These include Node.js, Python, Java, Ruby, PHP, Go, Rust, .NET, and more. Just like Circle CI, GitHub Actions is also available to build on Linux, macOS, and Windows. However, Circle CI does have a couple of extra options for building images/OS compared to GitHub Actions. 

Another great feature of GitHub Actions is the catalog of community-provided Actions you can add to your workflow without re-inventing the wheel. Don’t waste your precious time doing something that someone else has already done! Wondering if someone has already created the action you need in your workflow? Check out the catalog of available Actions here.

The build matrix is another great feature that GitHub Actions has but Circle CI lacks. This allows you to run multiple builds or tests simultaneously using different language versions, platform versions, etc. This can save quite a bit of time and also allow you to see if any certain language versions or platforms break with your new changes. For more information on creating a build matrix and what it can help with, check out this article.

Configuration

The configuration of GitHub Actions is very similar to that of Circle CI. You create a .github/workflows directory and create YAML files inside that directory. You can create different YAML files for each job or add them all to a single file if you wish. For a full list of available references, you can add to your GitHub Actions config files, check out the reference guide. For a simple example of how to set up a GitHub Actions configuration file, check out our example repo here.

Pricing

The pricing structure is similar to that of Circle CI. You pay for the amount of time your CI/CD pipelines are running per billing period. They also have a free tier that can be used indefinitely without any cost as long as you stay below the threshold for the free tier. GitHub Actions gives ~2K action minutes per month for free. 

Circle CI

Circle CI is one of the most popular cloud-hosted CI/CD providers. It was founded in 2011 and is based in San Francisco, CA. Circle CI is among the most popular CI/CD platforms for open source projects. It has support for a variety of languages with multiple pricing plans available to suit most peoples’ needs. Circle CI currently, as of this writing, is only available for repositories hosted on GitHub and BitBucket.

Configuration

You can configure Circle CI easily by just creating a file in the root of your SCM repo called .circleci/config.yml. You can find details on how to configure this file and all the available options here. This allows for the ability to copy the configuration file to multiple repositories and just edit the relevant details specific to that repo and everything will work. S

Some CI/CD tools require setting up the process manually to start with, and each time you create another project, you have to go through those same steps to set it up each time. Some steps could be overlooked or missed from time to time, which is not ideal. This can cause some of your automated processes to not work correctly, and thus possibly cause issues down the road if it goes unnoticed. You can find steps to set up Circle CI with GitHub here. You can find steps to set up Circle CI with BitBucket here.

Pricing

Circle CI is always free for open-source projects. For projects with private repositories, Circle CI has a free plan that has limited “credits” per week (as of this writing, it is 2500/week). For the default medium size Docker build container, that would give you about 250 minutes a week in build time since the medium docker image build container uses 10 credits per minute. You can use this free plan forever if your projects are not constantly running builds. If your projects only need a few builds a week, this is a great option! 

Below is a screenshot listing the majority of build container options available as of this writing and the number of credits per minute each one uses. One advantage Circle CI has over GitHub Actions is the GPU is they had GPU enhanced build option types. If your builds require the heavy lifting of GPU builds, these are a great option to have. 

Chart of credits used for different machine types
Chart of credits used for different machine types
Chart of credits used for different machine types
Chart of credits used for different machine types – Part 2

Beyond the free starter plan, Circle CI pricing is based on what they call “credits”. They do not have a paid plan that offers unlimited builds. This is something to keep in mind if you have many projects that are running builds on a consistent basis as this could spike the costs for your team if you do not estimate correctly. The paid plan starts at $30/month. You can find all pricing details for Circle CI here.

Supported Languages

Circle CI supports many of today’s most popular programming languages which can be found here.

Pros and Cons

GitHub Actions Pros

  • One interface for both your source code repositories and your CI/CD pipelines
  • Only 1 bill to pay that includes both your SCM plan and your CI/CD service (if you pay for either now that is)
  • Option to create a Build Matrix
  • You get around 2x the amount of build time per month on the free tier compared to Circle CI (2500 for GA and 1000 for Circle CI based on default configurations)
  • Catalog of available Actions you can utilize without reinventing the wheel

GitHub Actions Cons

  • Newer platform. Thus not as quite many features as Circle CI as of yet (although they sure have had a strong start)
  • Only available for project repositories hosted on GitHub or GitHub Enterprise
  • Does not have GPU-enhanced build options

Circle CI Pros

  • They have GPU-enhanced build options for projects needing the extra boot from GPU builds
  • Has a free tier (although not as many free build minutes as GitHub Actions)
  • More widely used than GitHub Actions (as of this writing) and thus more community-based support
  • Can be hooked up to more SCMs than just GitHub or GitHub Enterprise

Circle CI Cons

  • Not as many minutes on the free tier as GitHub Actions
  • Not as many available plugins to help ease the configuration of actions others have already completed
  • Need multiple subscriptions for both SCM and CI/CD services
  • Switching back and forth between Circle CI and SCM to see build logs and errors

Conclusion

The best platform for you all depends on your specific needs. If you host your code somewhere other than GitHub.com or GitHub Enterprise, you will be better off using Circle CI. However, if you host your code on GitHub.com or GitHub Enterprise, the choice can be a little more difficult. 

For most people, if they have a small number of projects whose CI/CD config files are relatively simple, I would suggest moving over to try GitHub Actions. Don’t disconnect or delete your current pipelines until you try GA out for a project or two, however. Just add the config files in for GA and see how you like the flow and feel of it. Once you verify that GA has everything you need to do your builds properly and successfully, and they have been running for at least a week or two successfully, you can delete your other config files and cancel your previous CI/CD service.

If you have complex projects or projects that need the hefty boost of GPU builds, I would suggest trying out or sticking with Circle CI. They have a few more options to help with more complex build scenarios since they are the mature CI/CD service provider. 

Go check out my posts comparing Travis CI vs Circle CI, Travis CI vs Jenkins, and Circle CI vs Jenkins to learn about other CI/CD platforms and the differences between them.

Please let me know which platform you think is better or that you prefer using and why in the comments!

0 comments
0 FacebookTwitterLinkedinTumblrRedditWhatsappTelegramCopy LinkPocketEmail
CircleCI Getting Started Image
Tutorials

How To Use Circle CI – Getting Started With CI/CD

by corey.walker
written by corey.walker 6 minutes read

Are you looking for a better way to manage your software development workflow? Tired of doing so many things manually or want to make sure bugs don’t get in the final product? That’s where DevOps comes into play! In this post, we will learn how to use Circle CI to improve our software development workflow!

Check out our related YouTube video below:

Creating an Account

The first thing we need to do is create an account. Head over to https://circleci.com/signup/ and create an account by using your GitHub or BitBucket account credentials. You will need to allow Circle CI to access some data in your SCM provider account of choice. This will allow Circle CI to be able to install webhooks and access data like your repository names, etc. This allows for automating processes such as build triggers when code is pushed to the repository.

Linking Your Repositories to Circle CI

You will need to link your repositories to Circle CI so builds can be triggered automatically when triggers you defined are triggered. Head over to the Circle CI Projects Dashboard and choose the user or organization you want to link to Circle CI. This will redirect you to a list of repositories that Circle CI is able to access under that account. For each repository, you will see a button/link that says “Set Up Project” as seen in the image below. Click on that button to start setting up your project

screenshot of CircleCI dashboard project list

This will bring up a webpage that has a code editor and some menu options. You may or may not need to change these based on the specific project you chose to set up. Make sure you have chosen the proper language in the upper left above the code editor. This will sometimes change the config shown in the code editor quite a bit, so make sure you have that correctly selected before moving forward. See Below

Dashboard of how to use Circle CI screenshot

Wrapping Up Initial Configuration

If the correct language has been selected and the selected config is showing in the editor, you can add the current config to your project to see the pipeline run. This default config will not do much except try to run unit tests. The default config will fail when it is run if you have a non-default way to run tests. You can at least see your pipeline triggered after a commit is pushed to make sure it’s hooked up correctly.

Setting Up A Basic Pipeline Config

Now let’s set up a basic pipeline config that we can actually have a pass when our Circle CI pipeline is triggered. The below example will print out a few stages in the pipeline and the last stage should be titled Print ‘Hello World’ in which it prints the words “Hello World”.

version: 2.1
orbs:
  node: circleci/[email protected]:
  build:
    docker:
      - image: circleci/node:12.20.1
    working_directory: /tmp/repo
    steps:
      - checkout
      - restore_cache:
          keys:
            - v1-dependencies-{{ checksum "package.json" }}
            - v1-dependencies-
      - run:
          name: "Install NPM Packages"
          command: "npm install"
      - run:
          name: "Print 'Hello World'"
          command: "echo 'Hello World'"
      - save_cache:
          paths:
            - node_modules
          key: v1-dependencies-{{ checksum "package.json" }}

There are many more advanced things you can add to your Circle CI pipeline. This is just the most simple example. In the next section, we will add a bit of customization to our config.

Customizing the Config File

We are going to customize our pipeline to sync all files in our project to an AWS S3 Bucket upon each commit and push. In order to do that, we will need to use a second Circle CI orb: aws-cli. To do this, add the following line to your config file:

version: 2.1
orbs:
  node: circleci/[email protected]
  aws-cli: circleci/[email protected] <<<<------ ADD THIS LINE

For the following steps to work, you will need a set of AWS IAM credentials you can use to upload files to S3. If you do not any AWS credentials, do that now by following the steps here.

Setting Up The Context

Next, we need to add what is called a context to our account. This is a set of credentials you can reference so you don’t hardcode them into your config file; which is ALWAYS a bad idea. You never want to check in files to your SCM tool with secret/sensitive values in them. So let’s add that context real quick.

Go to Circle CI Dashboard and then click Organization Settings in the left menu. Once you are in the Organization Settings menu, make sure that Contexts is selected on the left-hand side. Here is where we will create the context that holds our AWS credentials. In the main Contexts window, click the Create Contextbutton. Now give your context a name that is easy to remember. Now click on the newly created context. This will bring you to a screen that has 2 sections: Security and Environment Variables. 

Setting Up The AWS Environment Variables

In the Environment Variables section, click the Add Environment Variable button. We will add 2 environment variables to our context. First, add the variable named AWS_ACCESS_KEY_ID and then give it the value AWS_ACCESS_KEY_ID from your AWS IAM credential pair. Next, add a variable named AWS_SECRET_ACCESS_KEY and provide the corresponding value from your AWS IAM credential pair. Lastly, create an environment variable called AWS_DEFAULT_REGION and give it the value for one of the AWS regions you want to use. All of the AWS region names can be found here. Awesome! You’re done setting up your context!

Next, add a step in the build job to sync all of our files to an AWS S3 bucket. Add the following to your config file:

version: 2.1
orbs:
  node: circleci/[email protected]
  aws-cli: circleci/[email protected]
jobs:
  build:
    executor: aws-cli/default
    docker:
      - image: circleci/node:12.20.1
    working_directory: /tmp/repo
    steps:
      - checkout
      - aws-cli/setup
      - restore_cache:
          keys:
            - v1-dependencies-{{ checksum "package.json" }}
            - v1-dependencies-
      - run:
          name: "Install NPM Packages"
          command: "npm install"
      - run:
          name: "Print 'Hello World'"
          command: "echo 'Hello World'"
      - run:
          name: "Sync Files to AWS S3"
          command: aws s3 sync . s3://[bucket name goes here] --exclude "*node_modules/*"
      - save_cache:
          paths:
            - node_modules
          key: v1-dependencies-{{ checksum "package.json" }}

Now, the only thing left to do is to add a reference to the context so that the pipeline can access the AWS credentials it needs. Let’s do that now. Add the following to your config file:

version: 2.1
orbs:
  node: circleci/[email protected]
  aws-cli: circleci/[email protected]
jobs:
  build:
    docker:
      - image: circleci/node:12.20.1
    working_directory: /tmp/repo
    steps:
      - checkout
      - restore_cache:
          keys:
            - v1-dependencies-{{ checksum "package.json" }}
            - v1-dependencies-
      - run:
          name: "Install NPM Packages"
          command: "npm install"
      - run:
          name: "Print 'Hello World'"
          command: "echo 'Hello World'"
      - run:
          name: "Sync Files to AWS S3"
          command: aws s3 sync . s3://[bucket name goes here] --exclude "*node_modules/*"- save_cache:
          paths:
            - node_modules
          key: v1-dependencies-{{ checksum "package.json" }}
workflows:
  version: 2
  custom-workflow:
    jobs:
      - build:
          context: [your context name here]

Conclusion

That’s it! Your Circle CI pipeline should now auto-sync files to an AWS S3 bucket on each commit and push them to the remote repository! Now that you know how to use Circle CI, go improve your DevOps workflow!

0 comments
0 FacebookTwitterLinkedinTumblrRedditWhatsappTelegramCopy LinkPocketEmail
CircleCI Jenkins CI Logos Image
GeneralTechnology Comparisons

Which Is Better? Circle CI vs Jenkins

by corey.walker
written by corey.walker 5 minutes read

Software development and automation of the process have advanced a great deal in recent years. There are now many different platforms you can utilize to automate your software development workflow. Utilizing one of these CI (Continuous Integration) and CD (Continuous Delivery) tools is a best practice for the software development lifecycle. It helps automate things when new code is added to an SCM platform such as GitHub or BitBucket.

Two of the most popular CI/CI platforms today are Circle CI and Jenkins. In this post, I will compare the two platforms and lay out the differences between each, and provide a list of what I believe are the pros and cons of each. Let’s get started!

Check out the related YouTube video:

YouTube video for Circle CI vs Jenkins

Circle CI

Circle CI is one of the most popular cloud-hosted CI/CD providers. It was founded in 2011 and is based in San Francisco, CA. Circle CI is among the most popular CI/CD platforms for open source projects. It has support for a variety of languages with multiple pricing plans available to suit most peoples’ needs. Circle CI is currently only available for GitHub and BitBucket.

Configuration

Circle CI can be configured just by creating a file in the root of your SCM repo called .circleci/config.yml. Details can be found here on how to configure this file and all the available options. This allows for the ability to copy the configuration file to multiple repositories and just edit the relevant details specific to that repo and everything will work.

Some CI/CD tools require setting up the process manually to start with. Then each time you create another project, you have to go through those same steps to set it up each time. That is not really ideal as some steps could be overlooked or missed between projects. This can cause some of your automated processes to not work correctly, and thus possibly cause issues down the road if they go unnoticed.

You can find steps to set up Circle CI with GitHub here. You can find steps to set up Circle CI with BitBucket here.

Pricing

Circle CI is always free for open-source projects. For projects with private repositories, Circle CI has a free plan that has limited “credits” per week. You can use this free plan forever if your projects are not constantly running builds. If your projects only need a few builds a week, this is a great option!

Beyond the free starter plan, Circle CI pricing is based on what they call “credits”. They do not have a paid plan that offers unlimited builds. This is something to keep in mind if you have many projects that are running builds on a consistent basis as this could spike the costs for your team if you do not estimate correctly. The paid plan starts at $30/month. You can find all pricing details for Circle CI here.

Supported Languages

You can find a list of programming languages Circle CI supports here.

Jenkins

Jenkins is one of the oldest CI/CD platforms available which started back in 2005. It has been installed millions of times. This makes it one of the most, if not THE most, popular CI/CD platforms in use today. Jenkins has over 1500 plugins to choose from to customize your CI/CD needs. It is open-source and thus does not cost anything for the software itself. The only cost involved is the infrastructure you run it on.

Whereas Circle CI has both cloud-based and on-premise enterprise solutions, Jenkins is only available to be run on-premise. They do not have a Jenkins-hosted cloud-based CI/CD service that you can subscribe to. There are some third-party cloud-based offerings such as Google Cloud’s Jenkins Cloud offering and Cloudbees Jenkins Platform.

Jenkins allows simultaneous builds to be run concurrently by default if you have the infrastructure to support it. Numerous large companies use Jenkins for their CI/CD needs.

Although Jenkins does not have a native cloud-based offering directly, there are a few hosting providers that offer such an offering. One of the major draws of Jenkins, however, is the customizability using the multitude of plugins available.

Pros and Cons

Circle CI Pros

These are the benefits of Circle CI over Jenkins:

  • Cloud-based offering
  • No need to pay for, configure or maintain your own infrastructure
  • Large user base and well-vetted
  • Forever free starter plan for projects with private repositories

Circle CI Cons

  • Not as many plugins available to simplify the customization of CI/CD flow
  • Only available for repositories stored in GitHub or BitBucket
  • Not unlimited builds. Must keep purchasing credits which are used at varying rates based on OS and build image size

Jenkins Pros

  • Open-source software, and thus well vetted
  • Over 1,500 plugins to help simplify and customize CI/CD flow
  • Completely free software, only need to pay for the infrastructure you want to allocate
  • More widely used than Circle CI
  • More community-based support
  • More SCM integrations than just GitHub and BitBucket

Jenkins Cons

  • No cloud-based subscription offering directly from the vendor themselves. Requires third-party platforms hosting Jenkins in the cloud
  • Have to pay for, configure, and maintain your own infrastructure to run Jenkins
  • Bit of a learning curve to know which plugins to use for different scenarios

Conclusion

The best platform for you all depends on your specific needs. If you do not have a large monthly budget or extra man-hours to spend on infrastructure issues, then a cloud-based provider such as Circle CI will be a better choice. Just note that Circle CI only integrates with GitHub and Bitbucket as I write this (June 2020).

You would probably be better off using Jenkins for now if using SCM other than GitHub or BitBucket. Try one of the third-party cloud-based Jenkins offerings referenced earlier if on a budget.

You can also check out my posts comparing Travis CI vs Jenkins and Travis CI vs Circle CI.

Please let me know which platform you think is better or that you prefer using and why in the comments!

0 comments
0 FacebookTwitterLinkedinTumblrRedditWhatsappTelegramCopy LinkPocketEmail
TravisCI vs Jenkins Logos Image
GeneralTechnology Comparisons

Which Is Better? Travis CI vs Jenkins

by corey.walker
written by corey.walker 5 minutes read

Software development and automation of the process have advanced a great deal in recent years. There are now many different platforms you can utilize to automate your software development workflow. Utilizing one of these CI (Continuous Integration) and CD (Continuous Delivery) tools is a best practice for the software development lifecycle. It helps automate things when new changes are made. Do you know which is better for your needs? Do you know the differences between Travis CI vs Jenkins?

In this post, I will compare the two platforms and lay out the differences between each, and provide a list of what I believe are the pros and cons of each. Let’s get started!

Check out our related YouTube video:

YouTube video for Travis CI vs Jenkins

Travis CI

Founded in 2011, Travis CI is the oldest and most mature cloud-hosted CI/CD provider. They are based in Berlin, Germany. Travis CI is among the most popular CI/CD platforms for open source projects. It has support for a variety of languages with multiple pricing plans available to suit most peoples’ needs. Travis CI is currently only available for GitHub and BitBucket.

Configuration

Travis CI can be configured by just creating a file in the root of your SCM repo called .travis.yml. You can find details on how to configure this file and all the available options here. This allows for the ability to copy the configuration file to multiple repositories and just edit the relevant details specific to that repo and everything will work.

Some CI/CD tools require setting up the process manually to start with, and each time you create another project, you have to go through those same steps to set it up each time. That is not really ideal as some steps could be overlooked or missed from time to time. This can cause some of your automated processes to not work correctly, and thus possibly cause issues down the road if it goes unnoticed.

You can find steps to set up Travis CI with GitHub here. Steps to set up Travis CI with BitBucket are found here.

Pricing

Travis CI is always free for open-source projects. For projects with private repositories, your first 100 builds are free in order to allow you to try it out even for private repositories. For unlimited builds for projects with private repos, there are a variety of plans for developers and teams of all sizes starting at $65/month as of this writing (June 2020). You can find the full list of plan options for private repositories here.

Supported Languages

Travis CI supports a wide variety of programming languages. They support 32 different languages that are listed here and here. You can find the docs for how to submit support for a new language here.

Jenkins

Jenkins first started back in 2005 which makes it one of the oldest CI/CD platforms available. It has been installed millions of times. This makes it one of the most, if not THE most, popular CI/CD platforms in use today. It has over 1500 plugins to choose from to customize your CI/CD needs. Jenkins is open-source and does not cost anything for the software itself. The cost of infrastructure needed to run it is the only cost.

Jenkins is only available on-premise except through third-party providers. They do not have a Jenkins-hosted cloud-based CI/CD service that you can subscribe to. There are some offerings such as Google Cloud’s Jenkins Cloud offering and Cloudbees Jenkins Platform that allow you to use a cloud-based Jenkins solution, however.

Jenkins allows for simultaneous builds running concurrently by default if you have the infrastructure to support it. Numerous large companies use Jenkins for their CI/CD needs.

Although Jenkins does not have a cloud-based offering directly from Jenkins, there are a few hosting providers that offer such an offering. However, one of the major draws of Jenkins for users is the customizability of using the multitude of plugins available.

Pros and Cons

Travis CI Pros

The benefits of Travis CI over Jenkins are the following:

  • Cloud-based offering
  • No need to pay for, configure or maintain your own infrastructure
  • Build Matrices

Travis CI Cons

  • In order to get multiple concurrent builds, the monthly price can get expensive quickly
  • No completely free account for projects with private repositories
  • Not as many plugins available that can simplify the customization of CI/CD flow compared to Jenkins
  • Only available for repositories stored in GitHub or BitBucket

Jenkins Pros

  • Open-source software, well-vetted
  • Over 1,500 plugins to help simplify and customize CI/CD flow
  • Completely free software, only need to pay for the infrastructure you want to allocate
  • More widely used than Travis CI and thus more community-based support
  • More SCMs than just GitHub and BitBucket

Jenkins Cons

  • No cloud-based subscription offering directly from vendors themselves. Requires third-party platforms hosting Jenkins in the cloud
  • Have to pay for, configure, and maintain your own infrastructure to run Jenkins
  • With the multitude of plugins available, it can be a bit of a learning curve when learning which ones are best for your particular situation

Conclusion

The best platform for you all depends on your specific needs. If you do not have a large monthly budget or extra man-hours to spend on infrastructure issues, then a cloud-based provider such as Travis CI will be a better choice. Just note that Travis CI only integrates with GitHub and Bitbucket as I write this (June 2020).

If you are building projects that contain repositories stored somewhere other than GitHub or BitBucket, you would probably be better off using Jenkins for now until Travis CI expands its SCM integrations. If you are on a budget, try one of the third-party cloud-based Jenkins offerings I referenced earlier in this story.

You can also check out my post comparing Travis CI vs CircleCI and my comparison between CircleCI vs Jenkins.

Please let me know which platform you think is better or that you prefer using and why in the comments!

0 comments
0 FacebookTwitterLinkedinTumblrRedditWhatsappTelegramCopy LinkPocketEmail
TravisCI vs CircleCI Logos Image
GeneralTechnology Comparisons

Which Is Better? Travis CI vs Circle CI

by corey.walker
written by corey.walker 6 minutes read

Software development and automation of the process have advanced a great deal in recent years. There are now many different platforms you can utilize to automate your software development workflow. Utilizing one of these CI (Continuous Integration) and CD (Continuous Delivery) tools is a best practice for the software development lifecycle. It helps to automate things when new changes are added. Do you know which is better for your needs? Do you know the differences between Travis CI vs Circle CI?

In this post, I will compare the two platforms and lay out the differences between each, and provide a list of what I believe are the pros and cons of each. Let’s get started!

Check out our related YouTube video:

YouTube video for Travis CI vs Circle CI

Travis CI

Founded in 2011, Travis CI is the oldest and most mature cloud-hosted CI/CD provider. They are based in Berlin, Germany. They are among the most popular CI/CD platforms for open source projects. It has support for a variety of languages with multiple pricing plans available to suit most peoples’ needs. It is currently only available for GitHub and BitBucket.

Configuration

Configuration is as easy as just creating a file in the root of your SCM repo called .travis.yml. You can find details on how to configure this file and all the available options here. This allows for the ability to copy the configuration file to multiple repositories and just edit the relevant details specific to that repo and everything will work.

Some CI/CD tools require setting up the process manually to start with, and each time you create another project, you have to go through those same steps to set it up each time. That is not really ideal as some steps could be overlooked or missed from time to time. This can cause some of your automated processes to not work correctly, and thus possibly cause issues down the road if it goes unnoticed.

You can find steps to set up Travis CI with GitHub here and steps to set up with BitBucket here.

Pricing

Always free for open-source projects. For projects with private repositories, your first 100 builds are free in order to allow you to try it out even for private repositories. For unlimited builds for projects with private repos, there are a variety of plans for developers and teams of all sizes starting at $65/month as of this writing (June 2020). You can find the full list of plan options for private repositories here.

Supported Languages

They support a wide variety of programming languages. They support 32 different languages that can be found here and here. You can find the docs on how to submit support for a new language here.

They does not charge extra for access to macOS build environments. Circle CI does not charge “extra” per se, but requires more “credits” per minute than Linux build environments and thus can eat through your credits much faster than expected.

Circle CI

Also started in 2011, Circle CI has been around as long as Travis CI. They do not have quite as large of a customer base as Travis CI but are an equally powerful CI/CD platform that is based in San Francisco, CA. It also has support for a variety of languages with multiple pricing plans available to suit most peoples’ needs just as Travis CI does. Circle CI also is currently only available for GitHub and BitBucket.

Configuration

Configuration for Circle CI is just as easy by creating a directory in the root of your SCM repo called .circleci. Inside this directory, you create a file config.yml. You can find details on how to configure this file and all the available options here. This allows for the ability to copy the configuration file to multiple repositories and just edit the relevant details specific to that repo and everything will work just like Travis CI.

You can find steps to set up Circle CI with GitHub here. Steps to set up Circle CI with BitBucket are found here.

Pricing

Circle CI is always free for open-source projects. For projects with private repositories, you have multiple plans available with a Free plan available with limited credits each week. So if you have a project that doesn’t require a lot of changes and thus not many builds each week, this could be a permanent free solution.

Unlike Travis CI where you buy different plans which are all unlimited builds and build minutes but only differ by the number of concurrent builds, Circle CI plans are based on “credits”. This can allow you to auto-scale to the capacity of concurrent builds as needed by using more credits for each additional concurrent build. Pricing starts at $30/month which includes 3 users and 25,000 credits with each additional 25,000 credits costing $15 and an additional $15 per additional user.

Circle Ci charges more credits per minute for macOS builds than for the Linux Docker builds. The differences between normal Linux VM and macOS are minimal.

Supported Languages

Circle CI also supports a wide variety of programming languages. You can find information on Circle CI’s supported languages here.

Pros and Cons

Travis CI Pros

  • Unlimited builds for each paid plan
  • More languages supported by default
  • Build Matrices

Travis CI Cons

  • Higher starting price point for paid plans
  • No completely free account for projects with private repos
  • No concurrent builds until you pay more per month for higher-priced plans

Circle CI Pros

  • Completely free account for projects with private repos with limited build minutes each week
  • Cheaper starting price point for paid plans
  • Can automatically have multiple concurrent builds at the lowest price point

Circle CI Cons

  • Paid plans are based on “credits”. Credits are spent for each minute of build time
  • Credits are used at different rates depending on OS and VM size. Thus if you are building on Windows or macOS, you spend more credits per minute than Docker or Linux VM builds.

Conclusion

The best platform for you all depends on your specific needs. If you are only building projects that only have open source repositories, then either platform will be free. Thus, it’s all based on personal preference.

If you are building projects that contain private repositories and are heavily utilizing Windows or macOS VMs, it would most likely be cheaper, in the long run, to go with Travis CI. Circle CI will be cheaper if you have projects that are not updated very frequently and can be built on Linux, or if you can mostly utilize Circle Ci’s Docker Linux builds.

The other major thing to consider is the language being used for your projects. If one platform offers it and the other platform doesn’t, then the platform that does offer it would probably suit your needs better.

You can also check out my comparison between Travis CI and Jenkins or Circle CI and Jenkins.

Please let me know which platform you think is better and why in the comments!

0 comments
0 FacebookTwitterLinkedinTumblrRedditWhatsappTelegramCopy LinkPocketEmail
AWS Cloudwatch and Route53 logos image
Tutorials

How To Setup Multi-Region DNS Fail-over In AWS

by corey.walker
written by corey.walker 7 minutes read

Do you have an API hosted on AWS API Gateway? Do you wonder how you can deploy that API into multiple regions around the globe and have automatic fail-over protection in case one region goes offline so that your API keeps on working without any intervention? Well, you came to the right blog post. In this post, I will show you how to set up and configure your AWS Route53 DNS entries, health checks, and AWS API Gateway APIs in order to have that automatic multi-region DNS fail-over protection you’re looking for.

Initial Multi-Region DNS Fail-over Configuration

The first thing we need to do is have a domain setup in AWS Route 53. Once you have a domain, we also need to set up a hosted zone in Route 53 for that domain. Inside this hosted zone is where we will add the DNS records we need as part of our automatic DNS fail-over configuration. Once you have a domain name and hosted zone setup, let’s get going on configuring automatic DNS fail-over!

First things first, we need to have the desired API deployed to AWS API Gateway in each of the AWS regions we want in our setup. If you do not have that yet, you can follow this guide as an example of how to create an API using HTTP front-end with AWS Lambda as the back-end for processing the requests. You will need an endpoint on your API that will respond to GET requests and be able to return a status code of 2XX in order for our health checks to know that our API is healthy in each region.

Creating An SSL/TLS Certificate

Once you have that, you will need to create an SSL certificate on AWS Certificate Manager in each of the same regions to use with your domain which we will use in the next step. You will need to verify the entries you add to your SSL certificate. If you are using Route 53 as we have discussed in this article, you can expand each of the domain name listings and add it to Route 53. We then just need to wait on the verification to be complete before moving on to the next step.

Now that you have your API deployed to AWS API Gateway and we have an SSL certificate for our domain, we need to set up a custom domain name in AWS API Gateway in each of our AWS regions. Go to the AWS API Gateway console and click Create Custom Domain Name. It will open up a form to create a new custom domain which you should fill out as follows:

  • For Domain Name, fill in your desired domain for this API. (e.g. examplecorp.com)
  • Choose HTTP Protocol
  • Security Policy: TLS 1.2 (1.2 is the latest and most secure as of this writing. If there is a newer version as you’re reading this, choose it instead)
  • Endpoint Configuration: Regional (this selection is important!)
  • Choose the SSL certificate you created in the previous steps, then click Save

Cool. Now we have everything configured in API Gateway. Let’s move on to Route 53 to finish up our automatic multi-region DNS fail-over!

Configuring Health Checks

We now need to set up the health checks that allow us to automatically determine if a certain region is unavailable. We will use this to route our traffic accordingly. Visit the AWS Route 53 Console then click on Health Checks in the left-hand menu and then click Create health check.

Under the first section under Configure health check, give this health check a name that would specify the region, service, and environment this is being used for so we can keep things organized and easy to know what health check is monitoring what. A good name may be something like Production — US-East-1 — API for example. For What to monitor, select Endpoint.

Monitor Endpoint Configuration

Now, in the Monitor an endpoint section, use the following configuration:

  • Specify endpoint by: Domain Name
  • Protocol: HTTPS (this is required to be HTTPS because we use API Gateway API stage generated URLs which are HTTPS)
  • Domain Name: This will be the Invoke URL generated for the API stage in API Gateway such as dev, prod, etc after deploying an API. Template URL for where you can find it is https://console.aws.amazon.com/apigateway/home?region=us-east-1#/apis/{{API_ID}}/stages/dev and an example URL is dk0bnnfwn2.execute-api.us-east-1.amazonaws.com
  • Leave Port as default since it is determined by the protocol you selected
  • For Path, enter the stage name for the URL you copied above (e.g. dev, production, etc) followed by the endpoint path you want the health check to monitor. (The one I mentioned earlier that needed to have a GET method and return a 2XX status code). For example, a valid path could be dev/health

You can expand the Advanced Configuration section and change some of the options if you wish. One I would possibly really consider modifying is the list of Health Checker Regions. You have to choose at least 3 regions, choose 3 for development APIs, and additional for my production APIs.

CloudWatch Alarms

Click Next and on the next screen, you can set up CloudWatch alarms when the health check fails for the number of consecutive times specified in the Advanced Configuration section on the first page. Continue on and create the health check.

Repeat the above steps for each of the regions you wish to deploy your API, remembering to grab the correct Invoke URL for the API in each region. Now you should have all of your health checks in place. Let’s move on to the last piece of the puzzle, the DNS entries in your hosted zone for your domain.

Creating Route53 Record Sets

Let’s get the last piece configured now. Head over to the AWS Route 53 Console once again, and click on Hosted Zones in the left menu, then click on the hosted zone for your domain. Click on Create Record Set and enter the following details:

  • Name: desired domain/subdomain mapping you want for this API (e.g. api.examplecorp.com).
  • Record type: CNAME
  • Alias: No
  • TTL: 300 (5m)
  • Value: This is the value of Target Domain Name generated when you created the custom domain in API Gateway. Copy that value and paste it here
  • Routing Policy: Latency
  • Select the region that matches the custom domain Target Domain Nameyou just pasted, and then give this policy a name that distinguishes it from the other regions and environments you may use.
  • Associate With Health Check: Yes then select the health check that corresponds with the same region and environment that matches the other values for this CNAME record
  • Click Create
  • Follow the same steps for all other regions you deployed your API to

Conclusion

That’s it! Now whenever one of your regions is unavailable for whatever reason, it will take just a few minutes (<5 minutes) for your other regions to pick up the load. The response times may be a little slower than the desired region since these are latency-based routing rules. However, your application should keep on working after the health checks fail the specified number of times and fail-over occurs! Congratulations! You have now set up automatic multi-region DNS fail-over using AWS Route53 and API Gateway!

0 comments
0 FacebookTwitterLinkedinTumblrRedditWhatsappTelegramCopy LinkPocketEmail
Serverless Framework Tutorials Image
Tutorials

How To Deploy Multiple Environments Using Serverless

by corey.walker
written by corey.walker 4 minutes read

Do you currently use a serverless configuration for any applications or interested in migrating a current application to use serverless? Have you wondered how you can easily configure your CI/CD pipeline to easily deploy to multiple environments such as development, staging, and all production environments? With the Serverless Framework you can do just that. Continue reading to find out how.

Intro To Serverless Framework

Currently, I use the Serverless Framework to deploy to multiple environments and regions on the AWS cloud. I deploy to our development, staging, and production environments. Our development and staging environments are deployed in 2 different regions. Our production environment is deployed to 4 regions. So I have a serverless.yml file with many properties found in the docs for serverless.yml found here.

Configuration

  • All properties that remain the same for all environments and/or regions go into the serverless.yml base file
  • For all properties that are different between each environment such as development, staging, and production, I have another config file with the following naming convention under my config/ directory (the same naming I have is not required, but just what i use):
  • config/config.dev.yml, config/config.stg.yml, config/config.prod.yml, etc

Following this guide, I add all the needed custom properties to those environment-specific config files, and then in the main serverless.yml file, I create the same property but then reference the property from the environment/stage specified on the serverless deploy -s {stage}CLI command. I use a combination of both the stage CLI parameter and setting an ENV variable in a bash script before calling serverless deploy. Then I can also use ${env:ENV} to reference that value inside serverless.yml .

Environment Specific Configurations

Here’s an example of an environment-specific YAML configuration file I have for environment-specific variables:

stageSuffix: -${self:provider.stackSuffix, 'dev'}
lambda:
  versioning: false
prune:
  automatic: ${self:lambda.versioning}
  number: 5
api:
  domain: api.dev.${self:custom.domain}
storage:
  bucket: dev.${self:provider.region}.${self:custom.domain}
slackChannel: 'dev-slack-channel'

And inside serverless.yml, I reference those values by doing the following:

environment:
  ENV: ${env:ENV, 'dev'}
  STAGE: ${opt:stage, 'dev'}
  REGION: ${opt:region, 'us-east-1'}
  SLACK_CHANNEL: ${file(./config/config.${self:provider.stage}.yml):slackChannel}

Also, using a lot of shell environment-based variables allows you to make your serverless config even more decoupled away from specific projects/microservices since you then can just create a .env file specific to each project, create a bash script to run your serverless deploycommands, and before running serverless deploy, run source .env and many values can be populated easily by just supplying your .env file

Lastly, here’s an example of the bash script we use to run our serverless deploy commands that will supply many of our values in serverless.ymlwith values from a .env file instead of manually finding and replacing all the values between all our different microservices:

set -evif [[ -e "./secret.env" ]]; then
    source "./secret.env"
fi
if [[ -e "./.env" ]]; then
    source "./.env"
fiBRANCH="${CIRCLE_BRANCH:-$BRANCH}"if [[ $# -gt 0 ]]; then
    REGION="$1"
fiif [[ -n "$REGION" ]]; then
    REGION="$REGION"
else
    REGION="us-east-1"
fiif [[ "$BRANCH" == "development" ]] || [[ "$BRANCH" =~ ^issue-* ]]; then
    ENV="dev"
elif [[ "$BRANCH" == "staging" ]]; then
    ENV="stg"
elif [[ "$BRANCH" == "master" ]]; then
    ENV="prod"
fiif [[ "$BRANCH" =~ ^issue-* ]]; then
    export STAGE="feature"
    export STACK_SUFFIX="$BRANCH"
    export API_BASE_PATH_SUFFIX="-$BRANCH"
    "$(dirname "$0")"/create_domain.sh -s "$STAGE" -r "$REGION"
    "$NODE_MODULES_CLI_DIR"/serverless deploy -v -s "$STAGE" -r "$REGION"
else
    export STACK_SUFFIX="$ENV"
    "$(dirname "$0")"/create_domain.sh -s "$ENV" -r "$REGION"
    "$NODE_MODULES_CLI_DIR"/serverless deploy -v -s "$ENV" -r "$REGION"
fi

and the .env file we use to supply the variable values:

export BRANCH="$(git branch | grep \* | cut -d ' ' -f2)"
export ENV="dev"
export STACK_PREFIX="{STACK_PREFIX_HERE}"
export STACK_SUFFIX="$ENV"
export APP="{APP_ID_HERE}"
export APP_NAME="{APP_NAME_HERE}"
export COMPONENT="{COMPONENT_NAME_HERE}"
export REGION="us-east-1"
export DYNAMODB_ENDPOINT="dynamodb.$REGION.amazonaws.com"
export GITHUB_USERNAME="{GITHUB_USERNAME_HERE}"
export DB_TABLE_NAME="$COMPONENT"
export DOMAIN="{DOMAIN_NAME_HERE}"
export EMAIL_SUPPORT="support@${DOMAIN}"
export NODE_PRESERVE_SYMLINKS=1
export TEST_DEPLOYMENT_SUFFIX="-test"
export SLACK_CHANNEL="slack-channel-${ENV}"
export API_VERSION="v1"
export API_BASE_PATH=""
export API_BASE_PATH_PREFIX=""
export API_BASE_PATH_SUFFIX=""
export NODE_MODULES_CLI_DIR="./node_modules/.bin"

Conclusion

Let me know what you think and if you also see any improvement I myself could make to this process!

0 comments
0 FacebookTwitterLinkedinTumblrRedditWhatsappTelegramCopy LinkPocketEmail