Enhancement/improved dependency management

[proudfeet]

Overview

This PR closes MindGPT-75 but is conditional upon PR MindGPT-73.

Updates pre-commit dependency versions and introduces improved dependency management between poetry and pre-commit.

Details

This PR aims to:

1. Improve the dependency management between poetry and pre-commit
2. Update the dev dependencies and reconcile any resulting linting changes required
3. Document the process for adding a new dev dependency

The main addition is the use of sync_with_poetry. This is a small meta-hook that is used to read the poetry.lock file and update the .pre-commit-config.yaml file to match.

Updates to any dependency specifications are inline with the poetry dependency spec documenation

What is included

• Adds the sync_with_poetry hook to pre-commit
• Added black and ruff to explicit dev dependencies
• Updated some existing dependency version specifications that were blocking updates ¹
• Updated the dev dependency versions in poetry.lock and .pre-commit-config.yaml ²
• Minor refactoring changes due to up-versioning of black and ruff
• Extension of the DEVELOPMENT.md file to document the dependency management process

What is not included

• Full update of the package dependencies (using poetry update)
• Automation of dependency updates through CI

Footnotes

1. Several versions were specified as "^0.X.Y", when the leading value is 0 this locks the package so it cannot exceed 0.X.Y.
   For example, "^0.14.2" would allow 0.14.2.1 -> 0.14.2.2 but not 0.14.2.1 -> 0.15.0, which I do not believe is the expected outcome here. Therefore I have truncated up one level to allow version increases. See here for more details. ↩

2. pre-commit-hooks (not pre-commit) has actually moved to v4, however, it is locked to a max of v3.* in pyproject.toml. I have left this restriction as the v3->v4 change introduces breaking changes so requires further investigation. Though I think it should be fine as the breaking change is the removal of flake8 which you do not use. ↩