Skip to content

Commit db9b419

Browse files
joeweschjoewesch
authored
Merge pull request #728 from nautobot/u/joewesch-inline-m2m-fields
Adding inline M2M fields
2 parents 3b9712c + fcfcb02 commit db9b419

52 files changed

Lines changed: 4230 additions & 658 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎changes/+m2m.added‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
Added the ability to assign secrets to a secrets group when using the `secrets_group` module.
2+
Added the ability to assign static group associations when using the `dynamic_group` module.
3+
Added the ability to assign prefixes to a cloud network when using the `cloud_network` module.
4+
Added the ability to assign cloud networks to a cloud service when using the `cloud_service` module.
5+
Added the ability to assign vrfs to a device, virtual machine, or virtual device context when using the respective module.
6+
Added the ability to assign clusters to a device when using the `device` module.
7+
Added the ability to assign ip addresses to a device interface or vm interface when using the respective module.
8+
Added the ability to assign prefixes or vlans to a location when using the `location` module.
9+
Added the ability to assign provider networks to a provider when using the `provider` module.
10+
Added the ability to manage custom field choices to a custom field when using the `custom_field` module.
11+
Added the ability to manage metadata choices to a metadata type when using the `metadata_type` module.

‎changes/+module.deprecated‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
Deprecated the `secrets_groups_association` module. Use the `secrets` option of the `secrets_group` module instead.
2+
Deprecated the `static_group_association` module. Use the `static_group_associations` option of the `dynamic_group` module instead.
3+
Deprecated the `cloud_network_prefix_assignment` module. Use the `prefixes` option of the `cloud_network` module instead.
4+
Deprecated the `cloud_service_network_assignment` module. Use the `cloud_networks` option of the `cloud_service` module instead.
5+
Deprecated the `vrf_device_assignment` module. Use the `vrfs` option of the `device`, `virtual_machine`, or `virtual_device_context` module instead.
6+
Deprecated the `device_cluster_assignment` module. Use the `clusters` option of the `device` module instead.
7+
Deprecated the `ip_address_to_interface` module. Use the `ip_addresses` option of the `device_interface` or `vm_interface` module instead.
8+
Deprecated the `prefix_location` module. Use the `prefixes` option of the `location` module instead.
9+
Deprecated the `vlan_location` module. Use the `vlans` option of the `location` module instead.
10+
Deprecated the `custom_field_choice` module. Use the `custom_field_choices` option of the `custom_field` module instead.
11+
Deprecated the `metadata_choice` module. Use the `metadata_choices` option of the `metadata_type` module instead.
12+
Deprecated the `provider_network` module. Use the `provider_networks` option of the `provider` module instead.

‎docs/getting_started/contributing/modules/architecture.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -291,3 +291,49 @@ The next few lines manipulate the data and prepare it for sending to Nautobot.
291291
- Converts any fields that are namespaced to prevent conflicts when searching for them (e.g. device_role, ipam_role, rack_group, etc.)
292292

293293
If all those pass, it sets the manipulated data to `self.data` that is used in the module util apps.
294+
295+
## Inline M2M Fields
296+
297+
+++ 6.2.0
298+
299+
Some parent modules support inline many-to-many (M2M) association fields. These are declared in the `M2M_FIELDS` global dict in `utils.py`, which maps each endpoint to its supported M2M fields and their association API endpoints:
300+
301+
```python
302+
M2M_FIELDS = {
303+
"devices": {
304+
"vrfs": "vrf_device_assignments",
305+
"clusters": "device_cluster_assignments",
306+
},
307+
"locations": {
308+
"prefixes": "prefix_location_assignments",
309+
"vlans": "vlan_location_assignments",
310+
},
311+
# ... etc.
312+
}
313+
```
314+
315+
### How M2M Fields Are Processed
316+
317+
During `__init__`, M2M fields are handled separately from regular fields to prevent collisions with `_find_ids`:
318+
319+
1. **Extraction**: M2M field data is popped from the data dict *before* `_find_ids` runs on the parent data. This prevents M2M field names (like `prefixes`) from being mistakenly resolved as direct FK relationships.
320+
2. **Child resolution**: Each M2M object's child key (e.g., `vrf`, `prefix`, `ip_address`) is individually run through `_find_ids` to resolve names to UUIDs. This supports both simple strings (`vrf: "My VRF"`) and dicts for disambiguation (`ip_address: {address: "10.0.0.1/24", namespace: "Global"}`).
321+
3. **Payload stripping**: M2M field names are added to `remove_keys` so they are stripped from the REST API payload sent for the parent object.
322+
323+
After the parent object is created or updated, `_ensure_object_exists` calls `_process_m2m_fields` which iterates through each M2M field and:
324+
325+
- Fetches current associations from the API
326+
- Compares desired vs current using normalized comparison keys
327+
- Applies the requested state (`merge`, `replace`, or `delete`) via bulk create/delete operations
328+
329+
### Adding M2M Support to a New Module
330+
331+
If a new association endpoint is added to Nautobot and you want to support inline management:
332+
333+
1. Add the mapping to `M2M_FIELDS` in `utils.py`
334+
2. Add the M2M argument to the parent module's `argument_spec` following the standard structure (state/objects/child_key suboptions)
335+
3. Add matching `DOCUMENTATION` with suboptions
336+
4. Ensure the child key is in `CONVERT_TO_ID` and has appropriate `QUERY_TYPES`, `ENDPOINT_NAME_MAPPING`, and `ALLOWED_QUERY_PARAMS` entries
337+
5. Add integration tests covering merge, replace, delete, and idempotency
338+
339+
No changes to the module's `main()` function or constructor are needed -- the framework auto-detects M2M fields from `M2M_FIELDS` based on `self.endpoint`.

‎docs/getting_started/how-to-use/inline_m2m.md‎

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
# Inline Many-to-Many Associations
2+
3+
+++ 6.2.0
4+
5+
Many Nautobot models have many-to-many (M2M) relationships with other models. For example, a device can be associated with multiple VRFs, a location can have multiple prefixes, and a secrets group can contain multiple secrets.
6+
7+
Previously, managing these associations required using separate standalone modules (e.g., `vrf_device_assignment`, `prefix_location`, `secrets_groups_association`). Now you can manage them inline on the parent module using the M2M field options.
8+
9+
## Supported Parent Modules and Fields
10+
11+
| Parent Module | M2M Field | Child Object Key | Description |
12+
|---|---|---|---|
13+
| `device` | `vrfs` | `vrf` | VRF assignments |
14+
| `device` | `clusters` | `cluster` | Cluster assignments |
15+
| `virtual_machine` | `vrfs` | `vrf` | VRF assignments |
16+
| `virtual_device_context` | `vrfs` | `vrf` | VRF assignments |
17+
| `device_interface` | `ip_addresses` | `ip_address` | IP address to interface |
18+
| `vm_interface` | `ip_addresses` | `ip_address` | IP address to VM interface |
19+
| `location` | `prefixes` | `prefix` | Prefix to location |
20+
| `location` | `vlans` | `vlan` | VLAN to location |
21+
| `cloud_network` | `prefixes` | `prefix` | Cloud network prefix assignments |
22+
| `cloud_service` | `cloud_networks` | `cloud_network` | Cloud service network assignments |
23+
| `secrets_group` | `secrets` | `secret` | Secrets group associations |
24+
| `dynamic_group` | `static_group_associations` | `associated_object_type` / `associated_object_id` | Static group memberships |
25+
| `custom_field` | `custom_field_choices` | `value` | Custom field choices |
26+
| `metadata_type` | `metadata_choices` | `value` | Metadata type choices |
27+
| `provider` | `provider_networks` | `name` | Provider networks |
28+
29+
## M2M Field Structure
30+
31+
All M2M fields follow the same structure:
32+
33+
```yaml
34+
<m2m_field>:
35+
state: merge # merge (default), replace, or delete
36+
objects:
37+
- <child_key>: "value"
38+
- <child_key>: "another value"
39+
```
40+
41+
### States
42+
43+
- **merge** (default): Adds the specified associations without removing existing ones. Safe for incremental changes.
44+
- **replace**: Enforces exactly the listed associations. Any existing associations not in the list are removed.
45+
- **delete**: Removes only the specified associations. Other existing associations are left intact.
46+
47+
## Basic Examples
48+
49+
### Adding VRFs to a Device
50+
51+
```yaml
52+
- name: Create a device with VRF associations
53+
networktocode.nautobot.device:
54+
url: "{{ nautobot_url }}"
55+
token: "{{ nautobot_token }}"
56+
name: "my-router"
57+
device_type: "Cisco CSR1000v"
58+
role: "Router"
59+
location: "Main Site"
60+
status: "Active"
61+
vrfs:
62+
objects:
63+
- vrf: "Management VRF"
64+
- vrf: "Production VRF"
65+
state: present
66+
```
67+
68+
Since no `state` is specified on the `vrfs` field, it defaults to `merge` -- the VRFs are added without affecting any other existing VRF associations.
69+
70+
### Adding IP Addresses to an Interface
71+
72+
The child key accepts either a simple string or a dictionary for more specific lookups:
73+
74+
```yaml
75+
# Simple string -- looks up the IP address by address
76+
- name: Associate IP addresses with an interface
77+
networktocode.nautobot.device_interface:
78+
url: "{{ nautobot_url }}"
79+
token: "{{ nautobot_token }}"
80+
device: "my-router"
81+
name: "GigabitEthernet0/0"
82+
ip_addresses:
83+
objects:
84+
- ip_address: "10.0.0.1/24"
85+
state: present
86+
87+
# Dictionary -- useful when disambiguation is needed (e.g., multiple namespaces)
88+
- name: Associate IP address with namespace specified
89+
networktocode.nautobot.device_interface:
90+
url: "{{ nautobot_url }}"
91+
token: "{{ nautobot_token }}"
92+
device: "my-router"
93+
name: "GigabitEthernet0/0"
94+
ip_addresses:
95+
objects:
96+
- ip_address:
97+
address: "10.0.0.1/24"
98+
namespace: "Production"
99+
state: present
100+
```
101+
102+
### Adding Secrets to a Secrets Group
103+
104+
Some M2M associations have extra fields beyond just the child identifier. For secrets group associations, `access_type` and `secret_type` are part of the association itself:
105+
106+
```yaml
107+
- name: Create secrets group with secret associations
108+
networktocode.nautobot.secrets_group:
109+
url: "{{ nautobot_url }}"
110+
token: "{{ nautobot_token }}"
111+
name: "Device Credentials"
112+
secrets:
113+
objects:
114+
- secret: "admin-username"
115+
access_type: "SSH"
116+
secret_type: "username"
117+
- secret: "admin-password"
118+
access_type: "SSH"
119+
secret_type: "password"
120+
state: present
121+
```
122+
123+
## Managing Association State
124+
125+
### Merge (Default)
126+
127+
Merge adds new associations without removing existing ones. Running the same task twice is idempotent.
128+
129+
```yaml
130+
# First run: adds VRF-A
131+
- name: Add first VRF
132+
networktocode.nautobot.device:
133+
url: "{{ nautobot_url }}"
134+
token: "{{ nautobot_token }}"
135+
name: "my-router"
136+
vrfs:
137+
objects:
138+
- vrf: "VRF-A"
139+
state: present
140+
141+
# Second run: adds VRF-B, VRF-A is untouched
142+
- name: Add second VRF
143+
networktocode.nautobot.device:
144+
url: "{{ nautobot_url }}"
145+
token: "{{ nautobot_token }}"
146+
name: "my-router"
147+
vrfs:
148+
objects:
149+
- vrf: "VRF-B"
150+
state: present
151+
# Result: device has both VRF-A and VRF-B
152+
```
153+
154+
### Replace
155+
156+
Replace enforces exactly the listed set of associations. Any existing associations not in the list are removed.
157+
158+
```yaml
159+
# Device currently has VRF-A and VRF-B
160+
- name: Replace all VRFs with only VRF-C
161+
networktocode.nautobot.device:
162+
url: "{{ nautobot_url }}"
163+
token: "{{ nautobot_token }}"
164+
name: "my-router"
165+
vrfs:
166+
state: replace
167+
objects:
168+
- vrf: "VRF-C"
169+
state: present
170+
# Result: device has only VRF-C (VRF-A and VRF-B removed)
171+
```
172+
173+
### Delete
174+
175+
Delete removes only the specified associations. Other associations are left intact.
176+
177+
```yaml
178+
# Device currently has VRF-A and VRF-B
179+
- name: Remove only VRF-B
180+
networktocode.nautobot.device:
181+
url: "{{ nautobot_url }}"
182+
token: "{{ nautobot_token }}"
183+
name: "my-router"
184+
vrfs:
185+
state: delete
186+
objects:
187+
- vrf: "VRF-B"
188+
state: present
189+
# Result: device has only VRF-A
190+
```
191+
192+
## Diff Output
193+
194+
When M2M fields change, the diff output includes the before and after state as sorted lists of child object UUIDs:
195+
196+
```json
197+
{
198+
"diff": {
199+
"before": {
200+
"vrfs": ["<uuid-of-vrf-a>"]
201+
},
202+
"after": {
203+
"vrfs": ["<uuid-of-vrf-a>", "<uuid-of-vrf-b>"]
204+
}
205+
}
206+
}
207+
```

‎meta/runtime.yml‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,77 @@ plugin_routing:
66
deprecation:
77
removal_version: "7.0.0"
88
warning_text: "Use networktocode.nautobot.graphql_info or networktocode.nautobot.graphql_facts instead."
9+
secrets_groups_association:
10+
deprecation:
11+
removal_version: "7.0.0"
12+
warning_text: >-
13+
Use the networktocode.nautobot.secrets_group module with the
14+
'secrets' option instead.
15+
static_group_association:
16+
deprecation:
17+
removal_version: "7.0.0"
18+
warning_text: >-
19+
Use the networktocode.nautobot.dynamic_group module with the
20+
'static_group_associations' option instead.
21+
cloud_network_prefix_assignment:
22+
deprecation:
23+
removal_version: "7.0.0"
24+
warning_text: >-
25+
Use the networktocode.nautobot.cloud_network module with the
26+
'prefixes' option instead.
27+
cloud_service_network_assignment:
28+
deprecation:
29+
removal_version: "7.0.0"
30+
warning_text: >-
31+
Use the networktocode.nautobot.cloud_service module with the
32+
'cloud_networks' option instead.
33+
vrf_device_assignment:
34+
deprecation:
35+
removal_version: "7.0.0"
36+
warning_text: >-
37+
Use the networktocode.nautobot.device, networktocode.nautobot.virtual_machine,
38+
or networktocode.nautobot.virtual_device_context module with the
39+
'vrfs' option instead.
40+
device_cluster_assignment:
41+
deprecation:
42+
removal_version: "7.0.0"
43+
warning_text: >-
44+
Use the networktocode.nautobot.device module with the
45+
'clusters' option instead.
46+
ip_address_to_interface:
47+
deprecation:
48+
removal_version: "7.0.0"
49+
warning_text: >-
50+
Use the networktocode.nautobot.device_interface or
51+
networktocode.nautobot.vm_interface module with the
52+
'ip_addresses' option instead.
53+
prefix_location:
54+
deprecation:
55+
removal_version: "7.0.0"
56+
warning_text: >-
57+
Use the networktocode.nautobot.location module with the
58+
'prefixes' option instead.
59+
vlan_location:
60+
deprecation:
61+
removal_version: "7.0.0"
62+
warning_text: >-
63+
Use the networktocode.nautobot.location module with the
64+
'vlans' option instead.
65+
custom_field_choice:
66+
deprecation:
67+
removal_version: "7.0.0"
68+
warning_text: >-
69+
Use the networktocode.nautobot.custom_field module with the
70+
'custom_field_choices' option instead.
71+
metadata_choice:
72+
deprecation:
73+
removal_version: "7.0.0"
74+
warning_text: >-
75+
Use the networktocode.nautobot.metadata_type module with the
76+
'metadata_choices' option instead.
77+
provider_network:
78+
deprecation:
79+
removal_version: "7.0.0"
80+
warning_text: >-
81+
Use the networktocode.nautobot.provider module with the
82+
'provider_networks' option instead.

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,7 @@ nav:
107107
- How to Use:
108108
- Modules: "getting_started/how-to-use/modules.md"
109109
- Inventory: "getting_started/how-to-use/inventory.md"
110+
- Inline M2M Associations: "getting_started/how-to-use/inline_m2m.md"
110111
- Advanced Usage - Modules: "getting_started/how-to-use/advanced.md"
111112
- EDA Event Source Plugin: "getting_started/how-to-use/eda.md"
112113
- Migrating from query_graphql: "getting_started/how-to-use/graphql_migration.md"

0 commit comments

Comments
 (0)
⚡