Implement type = hybrid-bonded for metal_device_network_type

[displague]

Depends on equinixmetal-archive/packngo#231

With the upcoming https://feedback.equinixmetal.com/networking-features/p/mixed-mode-layer-23-for-hybrid-cloud feature, we'll need device support for network-type = "hybrid-bonded".

This is exciting because it means that in Terraform, we should not require the use of the metal_device_network_type resource to toggle resource modes for the most common uses.

One possible complication is that devices with metal_device_network_type resources defined as layer3 could be automatically transitioned to hybrid-bonded if a vlan is attached. Terraform would want to restore the layer3 value, which would entail detaching all existing vlans first.

Another complication is that a configuration like the following would succeed, but would have plan jitter because the network_type would be hybrid-bonded on subsequent reads.

resource "metal_device_network_type" "test" {
  device_id = metal_device.test.id
  type      = "layer3"
}

resource "metal_port_vlan_attachment" "test" {
  device_id = metal_device_network_type.test.id
  port_name = "eth1"
  vlan_vnid = metal_vlan.test.vxlan
}

Users would want to set type = "hybrid-bonded" in this case.

Notes

• When a server is provisioned, the initial state for the first bond is L3 Bonded. Other bond ports (if present) are initialized in L2 Bonded mode. (The only exception are x1.small.x86 / t1.small.x86 servers, as they only support Hybrid unbonded mode)
• Ports can not transition between some modes: ("any" means bonded and unbonded, list shows from -> to)
  • L3 -> L2 unbonded (also called "individual")
  • L2 unbonded -> L3
  • L2 (any) -> Hybrid (any)
  • Hybrid (any) -> L2 (any)
  • Hybrid (any) -> Hybrid (alternate)
• L3 is always bonded

Affected Resource(s)

Please list the resources as a list, for example:

• metal_device_network_type

[displague]

@t0mk and I have been going through some alternatives and teasing out requirements and features. The latest bit of thinking on this topic is expressed on the dependent packngo PR: equinixmetal-archive/packngo#239 (comment)

We’ve juggled the way TF users interact with these port/bond configurations a few times and are about to introduce another.

I have strong opinions on how Terraform should express this, and that is to say, Terraform hides nothing and simplifies nothing. If the API offers options X,Y,Z, Terraform offers options X,Y,Z. This is Hashicorp’s best-practice for implementing providers. No magic.

What we have now is a Terraform provider that disguises X,Y,Z port configurations as A,B,C network types, emulating the Equinix Metal portal UI.

The provider does not offer bond or port specific configuration and hides this behind a user-configurable “device network type”. Faced with the recent changes, the Terraform provider must move this configuration down from “device” to “port pair”, at least.

Arguably, the best configuration that we can offer is the truest to the API — “port” configuration, agnostic of what kind of port (bond or eth) we are dealing with. The TF user can effectively represent port state that maps to all of the port API calls, rather than hiding them behind a fake “network type”.

---

Copied from equinixmetal-archive/packngo#239 (comment)

resource "metal_port" "server-bond0" {
  port_id = metal_device.foo.network_ports.bond0.id
  bonded = true
  layer2 = false // this is default and represents either layer2 or layer3, leads to /port/id/convert-layer-X call on diff
  // network_type = "hybrid-bonded" // computed, read-only
  virtual_networks = [metal_vlan.foo.id]
  native_virtual_network = metal_vlan.native.id
  ip_address {
    type = "private_ipv4"
    cidr = 30
    // reservation_ids
    // cidr_notation from metal_ip_attachment
  }
}

resource "metal_port" "server-bond1" {
  port_id = metal_device.foo.network_ports.bond1.id
  bonded = true
  layer2 = true // representing the desired /port/id/convert-layer-X state
  // network_type = "layer2-bonded" // computed, read-only
  virtual_networks = [metal_vlan.foo.id]
}

[displague]

metal_device.foo.network_ports.bond0.id

This suggests a new computed network_ports field from metal_device. This is not strictly necessary today since the following can be used:

port_id = [for x in metal_device.foo.ports: x.id if x.name == "bond0"][0].id

[displague]

Reflecting on the first line of this resource:

resource "metal_port" "server-bond0" {
  port_id = metal_device.foo.network_ports.bond0.id

Ports can not be created or destroyed by users, they are made available through association with hardware_reservations or devices.

This makes me wonder if either:

• the resource should be called metal_port_settings because the name implies the dependent relationship
  • or should the resource be a network_ports property of the Device
• should the id be computed by lookup with the device_id or hardware_reservation_id and port_name fields required?
  • or should port_id be required and duplicated as id.

In either case, what does it mean to unset this port from the device, or destroy a port resource? Should Terraform revert the port to layer3/bonded/managed and configure other settings depending on if the port is eth0, eth1, bond0, etc? Or should unset/destroy mean that Terraform will make no further changes to the resource. I think the latter. If you don't define the settings you want, you get the settings you were given. Likewise, if you never defined port settings at all, you would have the same experience.

[displague]

One of the benefits of the independent resources (like the attachment and network_type resources) is how easily they can be used to trigger provisioners.

If all ports, bonding, and IP address assignments are made as updates to the device resource, null_resource provisioners would need to use triggers awaiting the specific port and state the provisioner is predicated on. This could lead to more accurate port-based provisioning patterns.

[displague]

If we receive a 403 during a port transition, the port resource (or device resource) would also produce a 403 - if the port is no longer accessible, it is indicative of the device no longer being accessible (moved to another user).

We will get this holistic delete detection behavior if port changes come through a port or device resource.

[displague]

If we move to metal_port, we can deprecate metal_device_network_type and replace it with a module that implements the resource through device (or port) data sources and metal_port resources.

[displague]

By moving to metal_port, we could also deprecate metal_port_vlan_attachment, again offering a module alternative.

[displague]

Support for metal_device_network_type type = "hybrid-bonded" was released in v1.1.0

Support for metal_port was added in v3.2.0

Usage is described in the following guide:
https://registry.terraform.io/providers/equinix/equinix/latest/docs/guides/network_types