Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add import_site_packages option #4

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 81 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ section:
| --- | ------- | ----------- |
| `generate_grpc` | `true` | Whether to generate gRPC output files. |
| `generate_pyi` | `true` | Whether to generate .pyi output files. Note that these are not generated for the gRPC output. You may want to use mypy-protobuf instead. |
| `proto_paths` | `["."]` or `["src"]` | An array of paths to search for `.proto` files. Also passed as `--proto_path` arguments to `protoc`. |
| `import_site_packages` | `false` | Adds your Python `site-packages` directory to `--proto_path`, so you can [`import` `.proto` files from installed Python packages](#include-proto-files-from-site-packages). This *does not* add individual `.proto` files in `site-packages` as arguments to `protoc`. |
| `proto_paths` | `["."]` or `["src"]` | An array of paths to search for `.proto` files. Also passed as `--proto_path` arguments to `protoc`. This does not follow symlinks. |
| `output_path` | `"."` or `"src"` | The default output directory. This can be overridden on a per-generator basis for custom generators. |

Hatch-protobuf will guess whether to use "src" as the default input/output directory in
Expand Down Expand Up @@ -73,3 +74,82 @@ outputs = ["{proto_path}/{proto_name}_pb2.pyi"]
name = "mypy_grpc"
outputs = ["{proto_path}/{proto_name}_pb2_grpc.pyi"]
```

### Import `.proto` files from `site-packages`

Setting `include_site_packages = true` causes the plugin to add your current
`site-packages` directory as a `--proto_path` when running `protoc`, *without*
trying to build a `_pb2.py` for *every* `.proto` file.

This allows your package to consume Protocol Buffer definitions from published
Python packages which include both `.proto` and `_pb2.py` files.

As an example, consider a gRPC service definition which includes a Protocol
Buffer message definiton from
[Google API common protos](https://github.com/googleapis/googleapis/tree/master/google/api):

```proto
// proto/example/v1/example.proto
syntax = "proto3";

package example.v1;

import "google/api/annotations.proto";

// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello(HelloRequest) returns (HelloReply) {
option (google.api.http) = {
get: "/say"
};
}
}

// The request message containing the user's name.
message HelloRequest {
string name = 1;
}

// The response message containing the greetings
message HelloReply {
string message = 1;
}
```

In this case, the
[`googleapis-common-protos` package](https://pypi.org/project/googleapis-common-protos/)
contains `annotations.proto` and a pre-built version of `annotations_pb2.py`,
which is normally installed in a directory tree layout which can be used by
*both* Python and `protoc`:

```
site-packages/google/api/
├── annotations_pb2.py
├── annotations.proto
├── auth_pb2.py
├── auth.proto
...
```

Setting `include_site_packages = true` makes your generated code contain imports
that reference the already-built bindings, and not rebuild them:

```python
# example/v1/example_pb2.py
from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
```

If you had added the directory containing `google/api/annotations.proto` to this
plugin's `proto_paths` option, this would cause it to re-build Python files for
*all* `.proto` files in that directory, not just the things used for your
package, and stomp all over your `site-packages` directory.

> [!NOTE]
> You can only `import` Protocol Buffer definitions in `.proto` files from
> *non-editable* dependencies (ie: from ordinary packages published on PyPI or
> private registries).
>
> *Editable* dependencies (eg: installed with `pip install -e` or using
> [`uv` workspaces](https://docs.astral.sh/uv/concepts/projects/dependencies/#editable-dependencies))
> use a different directory layout which can't be imported from by `protoc`.
4 changes: 4 additions & 0 deletions src/hatch_protobuf/plugin.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from dataclasses import dataclass
from functools import cached_property
from pathlib import Path
from sysconfig import get_path
from typing import Any, Dict, List

from hatchling.builders.hooks.plugin.interface import BuildHookInterface
Expand Down Expand Up @@ -47,6 +48,9 @@ def initialize(self, version: str, build_data: Dict[str, Any]) -> None:
for path in self._proto_paths:
args.append("--proto_path")
args.append(path)
if self.config.get("import_site_packages", False):
args.append("--proto_path")
args.append(get_path("purelib"))
for generator in self._generators:
args.append(f"--{generator.name}_out={generator.output_path}")

Expand Down