forked from sebivenlo/esd-2024-esd-2024-esd-template
-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add comparison between GitHub actions, Jenkins and CircleCI
- Loading branch information
Showing
2 changed files
with
255 additions
and
125 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,255 @@ | ||
| # GitHub Actions vs Jenkins Comparison vs CircleCI | ||
|
|
||
| This document describes a comparison between [GitHub Actions](https://github.com/features/actions) and [Jenkins](https://www.jenkins.io/). | ||
|
|
||
| ## Similarities | ||
|
|
||
| - Build, test and deploy code changes, which in turn enables faster development cycles [^1], [^17] | ||
| - Allow developers to automate the process of integrating code changes[^17] from multiple team members [^1] | ||
| - Can deploy code to different environments such as: dev, staging (stage) and production (prod) [^1] | ||
| - Maintain code quality [^1], by running tests* and linters**. | ||
| - Keeping teams in sync [^1], by pushing internal libraries to a central server as soon they are finished | ||
|
|
||
| \* _such as [JUnit](https://junit.org/junit5/) in Java, [https://kotest.io/](https://kotest.io/) for Kotlin or [Rust Tests] https://doc.rust-lang.org/book/ch11-01-writing-tests.html_ | ||
| \** _such as SonarLint in Java, [Detekt](https://github.com/detekt/detekt) in Kotlin or [Clippy](https://github.com/rust-lang/rust-clippy) in Rust_ | ||
|
|
||
| ## Differences | ||
|
|
||
| ### Criteria | ||
|
|
||
| For the comparison I choose following criteria: | ||
|
|
||
| #### Setup | ||
|
|
||
| The required effort or knowledge is an important criteria in an organisational context. | ||
| The longer you have to work on something, the more you have to pay your employees. | ||
| The more they need to know, the higher your employees need to be educated, and in turn their salary is higher. | ||
| If setup is long and complex you need more effort and knowledge than with a simple setup or no setup. | ||
| A complex setup usually comes with the benefit that the program is more powerful. | ||
| If this is desired or not, depends on the context. | ||
|
|
||
| #### Price | ||
|
|
||
| Price is important factories for business, so it is important to keep it low. | ||
| Price here refers to the software itself and not additional costs for hosting. | ||
| This could be included, but an extensive price comparisons is out of scope, because there a lot of different | ||
| cloud providers. Price can vary a lot therefore. | ||
| A price range should be stated here. | ||
|
|
||
| #### Vendor Lock in | ||
|
|
||
| Vendor lock in means that you are forced to use the software by the same vendor, instead of different ones. | ||
| This is usually not desired, because it limits the flexibility of switching to other products, if your requirements changes. | ||
|
|
||
| #### Host Type | ||
|
|
||
| Host type refers to how the CI/CD is hosted. Self-hosted, cloud or both. | ||
| Some companies have very strict security guidelines or company secrets. | ||
| A cloud-hosted CI/CD could expose a risk for some companies | ||
|
|
||
| #### Scaling | ||
|
|
||
| Scaling is important, because team size and company size can vary. | ||
| At a certain point, you need to scale your CI/CD properly. | ||
| A CI/CD cluster will differ significantly between a student group and large automobile manufacturer. | ||
| It should be stated how easy it is to scale and also mention price, if possible | ||
|
|
||
| #### Extensibility | ||
|
|
||
| Extensibility refers to the possibility to customize CI/CD. | ||
| Some projects may require a certain software or in-house solutions. | ||
| In that case it is important to what extent CI/CD can be customized. | ||
|
|
||
| #### Execution Model | ||
|
|
||
| Execution Model explains the architecture and way of working. | ||
| This is important, because different projects may require different architecture. | ||
| An event-based architecture my draw too much performance when there are a lot of events. | ||
|
|
||
| #### Maintenance | ||
|
|
||
| Maintenance is important because CI/CD is setup once, and then maintained. | ||
| Maintenance means here how easy it is to debug or how much effort is required. | ||
|
|
||
| #### Ease of use | ||
|
|
||
| Ease of use means how easy it is to learn a CI/CD platform. | ||
| For small projects or prototypes a fast result may be important. | ||
| Usually, a steep learning curve implies more configuration. | ||
| So you have a trade-off between complexity and Ease of use | ||
|
|
||
| #### Integration with other platforms | ||
|
|
||
| Integration with other platforms is important, because different projects have different requirements. | ||
| This requirement could be a deal-breaker, if you already have a certain provider or supplier, which you cannot change. | ||
|
|
||
| #### Testing on different machines/environments | ||
| If you are building a cross-platform program, you need to verify it works on different machines. | ||
| If you are using Rust, you also need to compile per architecture and operating system [^3] | ||
| Even when you are using Java, you need to verify your program on different machines, because issues my occur, like | ||
| line ending mismatch. | ||
|
|
||
| ### Comparison | ||
|
|
||
| | | GitHub | Jenkins | CircleCI | | ||
| |--------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------| - | | ||
| | Setup | Installation only required for self-hosted runners[^10], otherwise none | Installation required [^6] | No installation, just requires linking your GitHub repository [^25] | ||
| | Price | Free for standard GitHub-hosted runners in public repositories, and for self-hosted runners [^4] <br> For private repositories, GitHub Free offers 500MB storage and 2000 minutes per month [^4] | Free, but requires self hosting | Free for up to 6.000 build minutes and 30x concurrent jobs [^26] | ||
| | Vendor Lock in | Practically yes\*, you need to use GitHub | No, you can install it to different vendors [^6] | No, offers connection to different VCS providers [^27] | ||
| | Scaling | Can autoscale self-hosted runners based on webhooks[^21]. Large runners[^24]. | Jenkins can create multiple controllers[^22] and agents[^23] | Scale plan available to grow with the needs of the team [^28] | ||
| | Extensibility | Huge community support and GitHub marketplace[^1], [^15] | Through the Plugin index [^14] | Use CircleCI orbs, which are sharable packages of configuration [^29] | ||
| | Maintenance | Detailed logs and visual representations of workflows [^1], [^9] | Yes, through BlueOcean [^11] or plugins like "Pipeline: Stage View"[^12] | Offers a UI for Insights on how your build is performing [^25], [^30] | ||
| | Ease of use | GitHub provides templates [^13] and will automatically suggest these based on your repository code, no installion requierd | Easy to install [^16] | Easy to use, no installion required [^25]. | ||
| | Integration with other platforms | Seamless integration wih GitHub | Compatible with popular DevOps tools like Maven, Gradle, Git, Docker, and Kubernetes.[^16], [^17] | Can integrate with different VCS providers [^27]. Runs on different operating systems [^32]. | ||
| | Testing on different machines/environments | GitHub matrix strategies [^2] | Distributed builds** [^1], [^17] | Uses matrix to run the same test with different parameters [^33], [^34]. | ||
|
|
||
| \* _Technically, nobody is stopping you from cloning a non-GitHub repository in a self-Hosted runner action_ | ||
| \** _Although [^1] and [^17] state that distributed builds can do that, the Jenkins Documentation for Distributed builds is outdated [^18], but the updated one is still under development [^19]_ | ||
|
|
||
|
|
||
| #### Setup | ||
| For setup, GitHub Actions requires no installation [^10]. | ||
| You can just create a workflow and upload it to your repository [^20]. | ||
|
|
||
| On the other hand, Jenkins requires an installation first [^6]. | ||
| This could be difficult and my introduce more sources of error, because you have to install it on some servers and take care | ||
| of possible dependencies. | ||
|
|
||
| CircleCI is similar easy to setup as GitHub actions, because it does not require an installation. | ||
| You just need to link it with your GitHub repository [^25], or any other supported VCS [^27]. | ||
|
|
||
| #### Price | ||
| Looking at the price alone, Jenkins and GitHub is free. | ||
| However, because Jenkins requires an installation, and therefore a server when deploying something online, Jenkins is not really free. | ||
| Because you always need to pay a server host when making your pipeline accessible from the internet. | ||
|
|
||
| GitHub is completely free for public repositories. | ||
| GitHub also offers self-hosted runners, but these require a server, like Jenkins also does. | ||
| For private repositories, GitHub offers free minutes, depending on your plan in GitHub. | ||
| The table belows shows the prices. | ||
|
|
||
| | Plan | Storage | Minutes (per month) | | ||
| |-------------------------------|---------|---------------------| | ||
| | GitHub Free | 500 MB | 2.000 | | ||
| | GitHub Pro | 1 GB | 3.000 | | ||
| | GitHub Free for organizations | 500 MB | 2,000 | | ||
| | GitHub Team | 2 GB | 3,000 | | ||
| | GitHub Enterprise Cloud | 50 GB | 50,000 | | ||
|
|
||
| You can see, that the different between the free and paid tier is not that large. | ||
| The jump in 'Enterprise Cloud', however, is large. | ||
|
|
||
| It seems to offer more free minutes, but these depend on what resource class you use [^26]. | ||
| It even states that their Insights tool is not accurate about credit usage [^30]. | ||
| CircleCI start with a free plan for up to 5 active user and up to 6000 build minutes. | ||
| The performance plan costs 15 dollars per month and include 30.000 credits [^26]. | ||
| However, it is not stated how many build minutes that are. | ||
| The Scale plan starts at 2000 dollars per month and has great customizations and even has GPUs available [^26]. | ||
|
|
||
| #### Vendor Lock in | ||
|
|
||
| If you are using GitHub for version control, it is really convenient to use GitHub actions, too. | ||
| Theoretically, you could pull projects from other sources (at the very least on a self-hosted runner) and | ||
| However, triggering a workflow might be more difficult then. | ||
| But generally speaking, you need to use GitHub if you want to use GitHub actions. | ||
|
|
||
| Jenkins does not vendor lock you in. | ||
| You can use any version control and can run on different operating systems [^6]. | ||
|
|
||
| CircleCI does not force you to use specific VCS provider [^27]. | ||
|
|
||
| #### Scaling | ||
|
|
||
| GitHub offers to auto-scale self-hosted runners[^21]. | ||
| This way, you can scale up your workflows if needed. | ||
| However, this now needs your own infrastructure. | ||
|
|
||
| Jenkins provides slightly better support by enabling you to have multiple controllers[^22] and agents[^23], | ||
| to enable proper secure scaling. | ||
|
|
||
| CircleCI gives the opportunity for a Scale Plan [^28]. | ||
| It also offers custom enterprise plans [^26]. | ||
|
|
||
| #### Extensibility | ||
|
|
||
| GitHub, Jenkins and CircleCI offers som kind of plugin registry. [^1], [^15], [^14], [^29]. | ||
| However, GitHub also has a lot of templates projects, too. | ||
| All together, all solutions provide large extendability. | ||
|
|
||
| #### Maintenance | ||
|
|
||
| In maintenance, Jenkins has the best insights when installing plugins [^12]. | ||
| GitHub and CircleCI just have basic insights, if their builds failed or how they are performing [^1], [^9], [^25], [30] | ||
|
|
||
| #### Ease of use | ||
|
|
||
| Ease of use is always hard to estimate, because it depends on the person. | ||
| Every person has different level of skill and knowledge. | ||
| But generally speaking, GitHub actions is easier to use, because it does not require an installation. | ||
| This is one step less. | ||
| Defining the ease of use for CircleCI is hard to estimate, because there are not many sources, and testing | ||
| that thoroughly is not scope of this document. | ||
|
|
||
| #### Integration with other platforms | ||
|
|
||
| For integration with other platforms, Jenkins and CircleCI score the highest. | ||
| Which makes senses, because they strive to be independent of VCS or specific vendors. | ||
| GitHub on the hand, has different goals. | ||
| Then want to offer a great experience with their website. | ||
|
|
||
|
|
||
| #### Testing on different machines/environments | ||
|
|
||
| GitHub actions, Jenkins and CircleCI support some sort of matrix job execution, which run the same jobs with different parameters. | ||
| All of them support different operating systems, too. CircleCI also explicit states on their pricing site, that they offer different architectures, too (ARM and x86) [^26]. | ||
|
|
||
| ## Advice based on the comparison | ||
|
|
||
| GitHub actions is suitable for small hobby projects, even if they are private, because it is free [^4] and easy to use [^13]. | ||
| Usually, small hobby projects use GitHub already, so it is really easy to integrate it with GitHub actions. | ||
|
|
||
| Small open-source teams may also use GitHub actions, when they are already on GitHub. | ||
| Because they are open-source, their GitHub repository is public, so GitHub actions is free to use [^4]. | ||
|
|
||
| Larger project, either closed or open-source, probably want to use Jenkins. | ||
| Jenkins provides extensive functionality for scaling [^22], [^23] while not forcing you to use GitHub. | ||
| Open-source project sometimes like to host everything by themselves. | ||
| So they could use GitLab and Jenkins, or GitLab and GitLab runners. | ||
|
|
||
| Larger project can use CircleCI, if they don't want to maintain their own CI/CD infrastructure, as CircleCI | ||
| offers to host all runners in the cloud, while still providing enough resources and customer support [^26]. | ||
|
|
||
| [^1]: https://testomat.io/blog/jenkins-vs-github-actions-which-cicd-tool-is-right/ | ||
| [^2]: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/running-variations-of-jobs-in-a-workflow#example-using-a-single-dimension-matrix | ||
| [^3]: https://forge.rust-lang.org/infra/other-installation-methods.html | ||
| [^4]: https://docs.github.com/en/billing/managing-billing-for-your-products/managing-billing-for-github-actions/about-billing-for-github-actions#about-billing-for-github-actions | ||
| [^5]: https://docs.github.com/en/actions/about-github-actions/understanding-github-actions#runners | ||
| [^6]: https://www.jenkins.io/doc/book/installing/ | ||
| [^7]: https://github.com/jenkinsci/jenkins | ||
| [^8]: https://github.com/jenkinsci/jenkins/blob/master/LICENSE.txt | ||
| [^9]: https://docs.github.com/en/actions/writing-workflows/quickstart#viewing-your-workflow-results | ||
| [^10]: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners | ||
| [^11]: https://www.jenkins.io/doc/book/blueocean/#blue-ocean-overview | ||
| [^12]: https://plugins.jenkins.io/pipeline-stage-view/ | ||
| [^13]: https://github.com/actions/starter-workflows | ||
| [^14]: https://plugins.jenkins.io/ | ||
| [^15]: https://github.com/marketplace | ||
| [^16]: https://www.accelq.com/blog/what-is-jenkins-in-devops/ | ||
| [^17]: https://dev.to/mcieciora/why-jenkins-is-still-mvp-among-cicd-tools-2npn | ||
| [^18]: https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds | ||
| [^19]: https://www.jenkins.io/doc/developer/distributed-builds/ | ||
| [^20]: https://docs.github.com/en/actions/writing-workflows/quickstart | ||
| [^21]: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners | ||
| [^22]: https://www.jenkins.io/doc/book/scaling/architecting-for-scale/#distributed-builds-architecture | ||
| [^23]: https://www.jenkins.io/doc/book/scaling/architecting-for-scale/#agent-communications | ||
| [^24]: https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners/about-larger-runners | ||
| [^25]: https://coderonfleek.medium.com/circleci-vs-github-actions-a-software-engineers-perspective-14567e539b9c | ||
| [^26]: https://circleci.com/pricing/ | ||
| [^27]: https://circleci.com/docs/version-control-system-integration-overview/ | ||
| [^28]: https://circleci.com/docs/plan-scale/ | ||
| [^29]: https://circleci.com/developer/orbs | ||
| [^30]: https://circleci.com/docs/insights/#overview | ||
| [^31]: https://medium.com/@afimaamedufie/circling-around-circleci-8a1a15d751ff | ||
| [^32]: https://circleci.com/pricing/price-list/ | ||
| [^33]: https://circleci.com/docs/using-matrix-jobs/ | ||
| [^34]: https://circleci.com/docs/configuration-reference/#matrix |
Oops, something went wrong.