Skip to content

Commit

Permalink
First commit
Browse files Browse the repository at this point in the history
  • Loading branch information
BasPH committed Dec 11, 2023
0 parents commit cb59add
Show file tree
Hide file tree
Showing 20 changed files with 549 additions and 0 deletions.
30 changes: 30 additions & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: Continuous Integration
on: push

jobs:
syntax-checks:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Code formatting using Ruff
uses: chartboost/ruff-action@v1 # https://github.com/chartboost/ruff-action
with:
args: format --check

- name: Code linting using Ruff
uses: chartboost/ruff-action@v1

unit-testing:
strategy:
matrix:
python-version: ["3.9", "3.10", "3.11"]
runs-on: ubuntu-latest
container:
image: python:${{ matrix.python-version }}
steps:
- uses: actions/checkout@v4
- name: Install Python dependencies
run: pip install .[dev,test]
- name: Run tests using Pytest
run: python -m pytest tests/
102 changes: 102 additions & 0 deletions .github/workflows/publish.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# Inspired by https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#checking-out-the-project-and-building-distributions

name: Publish package to TestPyPI
on:
push:
tags:
- '*'

jobs:
build:
name: 📦 Build package
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.x"

- name: Install pypa/build
run: >-
python3 -m
pip install
build
--user
- name: Build a binary wheel and a source tarball
run: python3 -m build

- name: Store the distribution packages
uses: actions/upload-artifact@v3
with:
name: python-package-distributions
path: dist/

publish-to-testpypi:
name: Publish Python 🐍 distribution 📦 to TestPyPI
needs:
- build
runs-on: ubuntu-latest

environment:
name: testpypi
url: https://test.pypi.org/p/democorp_airflow

permissions:
id-token: write # IMPORTANT: mandatory for trusted publishing

steps:
- name: Download all the dists
uses: actions/download-artifact@v3
with:
name: python-package-distributions
path: dist/
- name: Publish distribution 📦 to TestPyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
repository-url: https://test.pypi.org/legacy/

github-release:
name: >-
Sign the Python 🐍 distribution 📦 with Sigstore
and upload them to GitHub Release
needs:
- publish-to-testpypi
runs-on: ubuntu-latest

permissions:
contents: write # IMPORTANT: mandatory for making GitHub Releases
id-token: write # IMPORTANT: mandatory for sigstore

steps:
- name: Download all the dists
uses: actions/download-artifact@v3
with:
name: python-package-distributions
path: dist/
- name: Sign the dists with Sigstore
uses: sigstore/[email protected]
with:
inputs: >-
./dist/*.tar.gz
./dist/*.whl
- name: Create GitHub Release
env:
GITHUB_TOKEN: ${{ github.token }}
run: >-
gh release create
'${{ github.ref_name }}'
--repo '${{ github.repository }}'
--notes ""
- name: Upload artifact signatures to GitHub Release
env:
GITHUB_TOKEN: ${{ github.token }}
# Upload to GitHub Release using the `gh` CLI.
# `dist/` contains the built packages, and the
# sigstore-produced signatures and certificates.
run: >-
gh release upload
'${{ github.ref_name }}' dist/**
--repo '${{ github.repository }}'
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
*.egg-info/
*.pyc
.DS_Store
.idea/
.python-version
__pycache__/
dist/
143 changes: 143 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
# Custom package demo

```
THIS REPOSITORY SERVES AS A DEMO FOR HOW TO STRUCTURE A CUSTOM PACKAGE.
FEEL FREE TO COPY AND ADJUST TO YOUR OWN NEEDS.
```

This Python library hosts custom Airflow Operators, Sensors, Notifiers, etc. that are common across **[insert company name]** data teams. This guide will walk you through how this library works, how to use it in your project, making changes, running tests, and contributing your changes.

## Table of Contents

- [Installing the library](#installing-the-library)
- [Using the library](#using-the-library)
- [Versioning](#versioning)
- [Continuous Integration/Continuous Deployment (CI/CD)](#continuous-integrationcontinuous-deployment-cicd)
- [Contributing](#contributing)
* [Getting started](#getting-started)
* [Making changes](#making-changes)
* [Testing](#testing)
* [Pull requests](#pull-requests)
* [Update your project dependency](#update-your-project-dependency)
- [Library structure](#library-structure)
- [How do I ...?](#how-do-i-)

<sub>Table of contents generated using https://derlin.github.io/bitdowntoc.</sub>

## Installing the library

Since this repository is built only for example purposes, we publish a Python package only to [TestPyPI](https://test.pypi.org/project/democorp-airflow). Therefore, you'll need to add TestPyPI as an (extra) index URL. You can install the package using pip:

```bash
pip install -i https://test.pypi.org/simple/ democorp-airflow
```

Or define the package in a requirements.txt file and install using `pip install -r requirements.txt`:

```
--extra-index-url https://test.pypi.org/simple
democorp_airflow
```

## Using the library

Import example:
```python
from democorp_airflow.operators.example import ExampleOperator
```

## Versioning

We use `setuptools-scm` to automatically manage versioning based on git tags. When you create a new tag, the library version is updated accordingly, and a release for the given tag is made. See the files in `.github/workflows` for inspiration.

See [1](https://github.com/pypa/setuptools_scm/), [2](https://www.moritzkoerber.com/posts/versioning-with-setuptools_scm/) for more details.

## Continuous Integration/Continuous Deployment (CI/CD)

Our CI/CD pipeline is managed using GitHub Actions:

- On every commit, syntax checks and unit tests are run to ensure code quality. See `.github/workflows/ci.yaml`.
- When a new tag is pushed, the library is built, published to TestPyPI, and a GitHub release is created. See `.github/workflows/publish.yaml`.

## Contributing

We welcome contributions from every team to improve this library! Here's how you can get started:

### Getting started

1. Clone the repository to your local machine.
1. Create a new branch for your changes: `git checkout -b my-new-feature`.

### Making changes

1. Make your desired changes to the library, following the structure and coding guidelines.
1. Write unit tests for your changes in the `tests/` directory.

### Testing

Before submitting your changes, ensure that all tests pass:

```bash
pytest tests/
```

### Pull requests

1. Commit your changes and push them to your branch.
1. Create a pull request from your branch to the `main` branch.
1. The CI/CD pipeline will automatically run tests on your pull request.
1. Once the tests pass and your code is reviewed, your contribution will be merged into the main repository.

### Update your project dependency

Once a new version of the `custom-package-demo` library is released, you need to update your Airflow project to use it.

1. Update the version constraint in your `requirements.txt` file.
1. After updating the package, it's crucial to test your project to ensure that the new version works as expected and doesn't introduce any compatibility issues or bugs. Run `astro dev start` to test changes locally.
1. Commit your changes to the requirements.txt file to Git so that other developers can easily reproduce your project environment.

## Library structure

The library is organized as follows:

```
custom_package_demo/
├── hooks/
│ ├── __init__.py
│ └── ...
├── notifiers/
│ ├── __init__.py
│ └── ...
├── operators/
│ ├── __init__.py
│ └── ...
├── sensors/
│ ├── __init__.py
│ └── ...
├── tests/
│ ├── hooks/
│ │ ├── __init__.py
│ │ └── ...
│ ├── notifiers/
│ │ ├── __init__.py
│ │ └── ...
│ ├── operators/
│ │ ├── __init__.py
│ │ └── ...
│ ├── sensors/
│ │ ├── __init__.py
│ │ └── ...
├── README.md
├── pyproject.toml
└── setup.py
```

- `tests/` directory contains unit tests for each component
- `pyproject.toml` is used for package configuration and versioning
- `setup.py` is used for backwards compatibility of newer `pyproject.toml` syntax

Happy coding!

## How do I ...?

Explain usage of custom components here...
6 changes: 6 additions & 0 deletions democorp_airflow/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
from importlib.metadata import PackageNotFoundError, version

try:
__version__ = version("afaas-custom-package")
except PackageNotFoundError:
__version__ = "unknown version"
Empty file.
42 changes: 42 additions & 0 deletions democorp_airflow/hooks/example.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
from airflow.hooks.base import BaseHook
from airflow.models import Connection


class ExampleHook(BaseHook):
"""..."""

def __init__(self, conn_id: str) -> None:
"""
Example Airflow hook. Implement your own business logic here.
:param conn_id: Airflow connection id
"""
super().__init__()
self._conn_id = conn_id

self._conn: Connection | None = None

@property
def conn(self) -> Connection:
"""
Cache connection to avoid re-fetching multiple times.
:return: Airflow connection object
"""
if self._conn is None:
self._conn = self.get_connection(conn_id=self._conn_id)
return self._conn

def test_connection(self) -> tuple[bool, str]:
"""
Verify a connection.
:return:
"""
try:
# ... Implement your business logic to validate a connection here ...
_ = self.conn
return True, "Connection successful."
except Exception as e:
self.log.warning("Failed connection verification.")
return False, str(e)
Empty file.
Empty file.
Loading

0 comments on commit cb59add

Please sign in to comment.