GraphQL: Support for filtering on custom fields

[emil-nasso]

NetBox version

v3.0.3

Feature type

New functionality

Proposed functionality

There doesn't seem to be a way to filter based on custom fields in the GraphQL-api.

The documentation mentions "The GraphQL API employs the same filtering logic as the UI and REST API" so one approach would be to just follow the same pattern where cf_foo_bar etc are exposed as arguments for the list query of an entity where foo_bar is a custom field.

(Reference to discussion about this: #7569 )

Use case

We have the use for this when listing tenants via the graphql API. We put the customers name in the name field but we have a custom field where we have an internal identifier that is used as a reference when linking the data in netbox up with other systems though integrations.

Database changes

No response

External dependencies

No response

[candlerb]

Another way might be via a graphql query on custom_fields. I note that you can ask for custom_fields in a response:

# curl -sS -X GET -H "Content-Type: application/json" -H "Accept: application/json" http://localhost:8001/graphql/ \
  --data '{"query": "query {device_list(id:\"5\") {id custom_fields}}"}' | python3 -m json.tool
{
    "data": {
        "device_list": [
            {
                "id": "5",
                "custom_fields": {
                    "snmp_module": [
                        "mikrotik_secret"
                    ]
                }
            }
        ]
    }
}

(There's also custom_field_data which returns a string holding JSON). But as far as I can tell, you can't currently filter on custom_fields in a query:

# curl -sS -X GET -H "Content-Type: application/json" -H "Accept: application/json" http://localhost:8001/graphql/ \
  --data '{"query": "query {device_list(custom_fields:{snmp_module:\"mikrotik_secret\"}) {id custom_fields}}"}' | python3 -m json.tool
{
    "errors": [
        {
            "message": "Unknown argument \"custom_fields\" on field \"device_list\" of type \"Query\".",
            "locations": [
                {
                    "line": 1,
                    "column": 20
                }
            ]
        }
    ]
}

It would of course be good if such a query could be pushed down to a SQL condition in postgres, rather than making Netbox retrieve all objects from the database and discard the ones which don't match.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our contributing guide.

[jeremystretch]

Blocked by #9856

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Do not attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our contributing guide.

[arthanson]

Blocked on #11949

[arthanson]

Adding blocked until #9856

[jeremypng]

I've got this ready to submit a PR. If someone (@jeremystretch or @arthanson) can assign it to me, I'll submit it. I ended up making a few more changes to get this working smoothly.

• The GraphiQL interface works fine with the custom schema entries now.
• A restart of netbox is required for the Strawberry schema to recognize the new field. I have not found a way to modify the GraphQL schema at runtime.
• Moved away from the deprecated filtering in Strawberry. This allows using the newer filter types and I expanded the map_strawberry_type function in filter_mixins.py.
• Created a resolver function in netbox/graphql/resolvers.py to handle custom field filtering on list queries.
• Modified each folder's schema.py list entries to use the new list resolver.
• BaseFilterMixin in the netbox/graphql/schema.py no longer does any filtering. This is handled through the newer Strawberry to Django coupling since deprecated filters are disabled. It is still in place to apply the strawberry.input decorator to all of filters derived from this class.
• Since BaseFilterMixin doesn't have a "filter_by_filterset" method anymore, we don't need the "should_create_function" logic in the autotype_decorator or map_strawberry_type functions.
• Autotype_decorator creates filter.annotations for model fields, declared filters, and then instantiates a filterset instance to get the custom field filters.
• The filterset instance requires database access and the migrations to be complete. Because the decorator loads at module import time, this requires a try/except pattern here to avoid crashing on startup if the migrations are missing or the database is not present (like in a testing only scenario). I tried delaying this until a later point, but the schema for graphql cannot be modified once it is loaded and it loads when the urls.py gets initialized, which is before the database is fully initialized. This is only a problem on first time startup and testing scenarios, but the try/except takes care of it. If there is a better way to solve this, I'm happy to make an adjustment.
• The custom field annotations are augmented with a netbox_field_map on the filter class. This is used by the resolver to transform the GraphQL filter name (ie: cf_cust_id) into the Django field name (ie: custom_field_data__cust_id).
• If a query contains a custom field, it is processed by the resolver function before being handed to Strawberry for normal processing. The resolver unsets the custom filter field entries because the normal Strawberry processing cannot map the custom fields to the custom_field_data field on the model. Permissions are enforced in the resolver, and the query syntax processing still happens using the Strawberry code. I'm just using that generated query and applying it against a queryset.filter() at that point.
• At this point all of the tests are passing. It could use some more testing, but it looks like the functionality I was looking for is there without breaking anything else.

Here is the link to my code at this point.

[jeremypng]

I have the custom field filtering itself working pretty well. However, the rest of the graphql implementation is in such bad shape, I'm going a different route. Introspecting the django fitlers and models to try to infer a graphql schema at import time is very complex and requires gutting Strawberry of most of its power.

I've cycled through most of the options outlined by @arthanson in Issue #17688 and come to some conclusions that there is a better way to solve this. I'll start a discussion around this.

[jeremystretch]

@jeremypng thanks for starting this discussion. I haven't had any time to dedicate to the theme of GraphQL recently and @arthanson is currently loaned out for another initiative, but I hope we can prioritize this work soon as it's clearly a pain point for NetBox users.

[jeremypng]

This is fixed in my branch here:
https://github.com/jeremypng/netbox/tree/refs/heads/graphql-filter-redesign

New query syntax for custom fields:

query GetTenants {
  tenant_list (filters: {custom_field_data: {path: "cust_id", lookup: {string_lookup: {exact:"YKY01"}}}}) {
    id
    name
    custom_field_data
  }
}

results:

{
  "data": {
    "tenant_list": [
      {
        "id": "9",
        "name": "Nakatomi Corportation",
        "custom_field_data": {
          "cust_id": "YKY01"
        }
      }
    ]
  }
}

If you'll assign this to me, I'll tag this issue in the PR.

[jeremystretch]

@jeremypng I've tagged this for v4.3 and assigned it to you. Could you open a PR from your branch into feature please?