-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
0 parents
commit cb59add
Showing
20 changed files
with
549 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
name: Continuous Integration | ||
on: push | ||
|
||
jobs: | ||
syntax-checks: | ||
runs-on: ubuntu-latest | ||
steps: | ||
- uses: actions/checkout@v4 | ||
|
||
- name: Code formatting using Ruff | ||
uses: chartboost/ruff-action@v1 # https://github.com/chartboost/ruff-action | ||
with: | ||
args: format --check | ||
|
||
- name: Code linting using Ruff | ||
uses: chartboost/ruff-action@v1 | ||
|
||
unit-testing: | ||
strategy: | ||
matrix: | ||
python-version: ["3.9", "3.10", "3.11"] | ||
runs-on: ubuntu-latest | ||
container: | ||
image: python:${{ matrix.python-version }} | ||
steps: | ||
- uses: actions/checkout@v4 | ||
- name: Install Python dependencies | ||
run: pip install .[dev,test] | ||
- name: Run tests using Pytest | ||
run: python -m pytest tests/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,102 @@ | ||
# Inspired by https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#checking-out-the-project-and-building-distributions | ||
|
||
name: Publish package to TestPyPI | ||
on: | ||
push: | ||
tags: | ||
- '*' | ||
|
||
jobs: | ||
build: | ||
name: 📦 Build package | ||
runs-on: ubuntu-latest | ||
steps: | ||
- uses: actions/checkout@v4 | ||
|
||
- name: Set up Python | ||
uses: actions/setup-python@v4 | ||
with: | ||
python-version: "3.x" | ||
|
||
- name: Install pypa/build | ||
run: >- | ||
python3 -m | ||
pip install | ||
build | ||
--user | ||
- name: Build a binary wheel and a source tarball | ||
run: python3 -m build | ||
|
||
- name: Store the distribution packages | ||
uses: actions/upload-artifact@v3 | ||
with: | ||
name: python-package-distributions | ||
path: dist/ | ||
|
||
publish-to-testpypi: | ||
name: Publish Python 🐍 distribution 📦 to TestPyPI | ||
needs: | ||
- build | ||
runs-on: ubuntu-latest | ||
|
||
environment: | ||
name: testpypi | ||
url: https://test.pypi.org/p/democorp_airflow | ||
|
||
permissions: | ||
id-token: write # IMPORTANT: mandatory for trusted publishing | ||
|
||
steps: | ||
- name: Download all the dists | ||
uses: actions/download-artifact@v3 | ||
with: | ||
name: python-package-distributions | ||
path: dist/ | ||
- name: Publish distribution 📦 to TestPyPI | ||
uses: pypa/gh-action-pypi-publish@release/v1 | ||
with: | ||
repository-url: https://test.pypi.org/legacy/ | ||
|
||
github-release: | ||
name: >- | ||
Sign the Python 🐍 distribution 📦 with Sigstore | ||
and upload them to GitHub Release | ||
needs: | ||
- publish-to-testpypi | ||
runs-on: ubuntu-latest | ||
|
||
permissions: | ||
contents: write # IMPORTANT: mandatory for making GitHub Releases | ||
id-token: write # IMPORTANT: mandatory for sigstore | ||
|
||
steps: | ||
- name: Download all the dists | ||
uses: actions/download-artifact@v3 | ||
with: | ||
name: python-package-distributions | ||
path: dist/ | ||
- name: Sign the dists with Sigstore | ||
uses: sigstore/[email protected] | ||
with: | ||
inputs: >- | ||
./dist/*.tar.gz | ||
./dist/*.whl | ||
- name: Create GitHub Release | ||
env: | ||
GITHUB_TOKEN: ${{ github.token }} | ||
run: >- | ||
gh release create | ||
'${{ github.ref_name }}' | ||
--repo '${{ github.repository }}' | ||
--notes "" | ||
- name: Upload artifact signatures to GitHub Release | ||
env: | ||
GITHUB_TOKEN: ${{ github.token }} | ||
# Upload to GitHub Release using the `gh` CLI. | ||
# `dist/` contains the built packages, and the | ||
# sigstore-produced signatures and certificates. | ||
run: >- | ||
gh release upload | ||
'${{ github.ref_name }}' dist/** | ||
--repo '${{ github.repository }}' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
*.egg-info/ | ||
*.pyc | ||
.DS_Store | ||
.idea/ | ||
.python-version | ||
__pycache__/ | ||
dist/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,143 @@ | ||
# Custom package demo | ||
|
||
``` | ||
THIS REPOSITORY SERVES AS A DEMO FOR HOW TO STRUCTURE A CUSTOM PACKAGE. | ||
FEEL FREE TO COPY AND ADJUST TO YOUR OWN NEEDS. | ||
``` | ||
|
||
This Python library hosts custom Airflow Operators, Sensors, Notifiers, etc. that are common across **[insert company name]** data teams. This guide will walk you through how this library works, how to use it in your project, making changes, running tests, and contributing your changes. | ||
|
||
## Table of Contents | ||
|
||
- [Installing the library](#installing-the-library) | ||
- [Using the library](#using-the-library) | ||
- [Versioning](#versioning) | ||
- [Continuous Integration/Continuous Deployment (CI/CD)](#continuous-integrationcontinuous-deployment-cicd) | ||
- [Contributing](#contributing) | ||
* [Getting started](#getting-started) | ||
* [Making changes](#making-changes) | ||
* [Testing](#testing) | ||
* [Pull requests](#pull-requests) | ||
* [Update your project dependency](#update-your-project-dependency) | ||
- [Library structure](#library-structure) | ||
- [How do I ...?](#how-do-i-) | ||
|
||
<sub>Table of contents generated using https://derlin.github.io/bitdowntoc.</sub> | ||
|
||
## Installing the library | ||
|
||
Since this repository is built only for example purposes, we publish a Python package only to [TestPyPI](https://test.pypi.org/project/democorp-airflow). Therefore, you'll need to add TestPyPI as an (extra) index URL. You can install the package using pip: | ||
|
||
```bash | ||
pip install -i https://test.pypi.org/simple/ democorp-airflow | ||
``` | ||
|
||
Or define the package in a requirements.txt file and install using `pip install -r requirements.txt`: | ||
|
||
``` | ||
--extra-index-url https://test.pypi.org/simple | ||
democorp_airflow | ||
``` | ||
|
||
## Using the library | ||
|
||
Import example: | ||
```python | ||
from democorp_airflow.operators.example import ExampleOperator | ||
``` | ||
|
||
## Versioning | ||
|
||
We use `setuptools-scm` to automatically manage versioning based on git tags. When you create a new tag, the library version is updated accordingly, and a release for the given tag is made. See the files in `.github/workflows` for inspiration. | ||
|
||
See [1](https://github.com/pypa/setuptools_scm/), [2](https://www.moritzkoerber.com/posts/versioning-with-setuptools_scm/) for more details. | ||
|
||
## Continuous Integration/Continuous Deployment (CI/CD) | ||
|
||
Our CI/CD pipeline is managed using GitHub Actions: | ||
|
||
- On every commit, syntax checks and unit tests are run to ensure code quality. See `.github/workflows/ci.yaml`. | ||
- When a new tag is pushed, the library is built, published to TestPyPI, and a GitHub release is created. See `.github/workflows/publish.yaml`. | ||
|
||
## Contributing | ||
|
||
We welcome contributions from every team to improve this library! Here's how you can get started: | ||
|
||
### Getting started | ||
|
||
1. Clone the repository to your local machine. | ||
1. Create a new branch for your changes: `git checkout -b my-new-feature`. | ||
|
||
### Making changes | ||
|
||
1. Make your desired changes to the library, following the structure and coding guidelines. | ||
1. Write unit tests for your changes in the `tests/` directory. | ||
|
||
### Testing | ||
|
||
Before submitting your changes, ensure that all tests pass: | ||
|
||
```bash | ||
pytest tests/ | ||
``` | ||
|
||
### Pull requests | ||
|
||
1. Commit your changes and push them to your branch. | ||
1. Create a pull request from your branch to the `main` branch. | ||
1. The CI/CD pipeline will automatically run tests on your pull request. | ||
1. Once the tests pass and your code is reviewed, your contribution will be merged into the main repository. | ||
|
||
### Update your project dependency | ||
|
||
Once a new version of the `custom-package-demo` library is released, you need to update your Airflow project to use it. | ||
|
||
1. Update the version constraint in your `requirements.txt` file. | ||
1. After updating the package, it's crucial to test your project to ensure that the new version works as expected and doesn't introduce any compatibility issues or bugs. Run `astro dev start` to test changes locally. | ||
1. Commit your changes to the requirements.txt file to Git so that other developers can easily reproduce your project environment. | ||
|
||
## Library structure | ||
|
||
The library is organized as follows: | ||
|
||
``` | ||
custom_package_demo/ | ||
├── hooks/ | ||
│ ├── __init__.py | ||
│ └── ... | ||
├── notifiers/ | ||
│ ├── __init__.py | ||
│ └── ... | ||
├── operators/ | ||
│ ├── __init__.py | ||
│ └── ... | ||
├── sensors/ | ||
│ ├── __init__.py | ||
│ └── ... | ||
├── tests/ | ||
│ ├── hooks/ | ||
│ │ ├── __init__.py | ||
│ │ └── ... | ||
│ ├── notifiers/ | ||
│ │ ├── __init__.py | ||
│ │ └── ... | ||
│ ├── operators/ | ||
│ │ ├── __init__.py | ||
│ │ └── ... | ||
│ ├── sensors/ | ||
│ │ ├── __init__.py | ||
│ │ └── ... | ||
├── README.md | ||
├── pyproject.toml | ||
└── setup.py | ||
``` | ||
|
||
- `tests/` directory contains unit tests for each component | ||
- `pyproject.toml` is used for package configuration and versioning | ||
- `setup.py` is used for backwards compatibility of newer `pyproject.toml` syntax | ||
|
||
Happy coding! | ||
|
||
## How do I ...? | ||
|
||
Explain usage of custom components here... |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
from importlib.metadata import PackageNotFoundError, version | ||
|
||
try: | ||
__version__ = version("afaas-custom-package") | ||
except PackageNotFoundError: | ||
__version__ = "unknown version" |
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
from airflow.hooks.base import BaseHook | ||
from airflow.models import Connection | ||
|
||
|
||
class ExampleHook(BaseHook): | ||
"""...""" | ||
|
||
def __init__(self, conn_id: str) -> None: | ||
""" | ||
Example Airflow hook. Implement your own business logic here. | ||
:param conn_id: Airflow connection id | ||
""" | ||
super().__init__() | ||
self._conn_id = conn_id | ||
|
||
self._conn: Connection | None = None | ||
|
||
@property | ||
def conn(self) -> Connection: | ||
""" | ||
Cache connection to avoid re-fetching multiple times. | ||
:return: Airflow connection object | ||
""" | ||
if self._conn is None: | ||
self._conn = self.get_connection(conn_id=self._conn_id) | ||
return self._conn | ||
|
||
def test_connection(self) -> tuple[bool, str]: | ||
""" | ||
Verify a connection. | ||
:return: | ||
""" | ||
try: | ||
# ... Implement your business logic to validate a connection here ... | ||
_ = self.conn | ||
return True, "Connection successful." | ||
except Exception as e: | ||
self.log.warning("Failed connection verification.") | ||
return False, str(e) |
Empty file.
Empty file.
Oops, something went wrong.