Skip to content

Commit e72c4d4

Browse files
committed
try to fix 223
1 parent a55a46d commit e72c4d4

12 files changed

Lines changed: 559 additions & 66 deletions

File tree

README.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -238,14 +238,20 @@ No modules.
238238
| [azurerm_log_analytics_solution.main](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/log_analytics_solution) | resource |
239239
| [azurerm_log_analytics_workspace.main](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/log_analytics_workspace) | resource |
240240
| [azurerm_role_assignment.acr](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
241+
| [azurerm_role_assignment.application_gateway_contributor](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
242+
| [azurerm_role_assignment.application_gateway_resource_group_reader](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
243+
| [azurerm_role_assignment.application_gateway_vnet_network_contributor](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
241244
| [azurerm_role_assignment.network_contributor](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
242245
| [azurerm_role_assignment.network_contributor_on_subnet](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
243246
| [null_resource.kubernetes_version_keeper](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource |
244247
| [null_resource.pool_name_keeper](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource |
245248
| [tls_private_key.ssh](https://registry.terraform.io/providers/hashicorp/tls/latest/docs/resources/private_key) | resource |
249+
| [azurerm_client_config.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/client_config) | data source |
246250
| [azurerm_log_analytics_workspace.main](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/log_analytics_workspace) | data source |
251+
| [azurerm_resource_group.ingress_appgw](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/resource_group) | data source |
247252
| [azurerm_resource_group.main](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/resource_group) | data source |
248253
| [azurerm_user_assigned_identity.cluster_identity](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/user_assigned_identity) | data source |
254+
| [azurerm_virtual_network.application_gateway_vnet](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/virtual_network) | data source |
249255

250256
## Inputs
251257

@@ -271,6 +277,7 @@ No modules.
271277
| <a name="input_agents_type"></a> [agents\_type](#input\_agents\_type) | (Optional) The type of Node Pool which should be created. Possible values are AvailabilitySet and VirtualMachineScaleSets. Defaults to VirtualMachineScaleSets. | `string` | `"VirtualMachineScaleSets"` | no |
272278
| <a name="input_api_server_authorized_ip_ranges"></a> [api\_server\_authorized\_ip\_ranges](#input\_api\_server\_authorized\_ip\_ranges) | (Optional) The IP ranges to allow for incoming traffic to the server nodes. | `set(string)` | `null` | no |
273279
| <a name="input_api_server_subnet_id"></a> [api\_server\_subnet\_id](#input\_api\_server\_subnet\_id) | (Optional) The ID of the Subnet where the API server endpoint is delegated to. | `string` | `null` | no |
280+
| <a name="input_application_gateway_for_ingress"></a> [application\_gateway\_for\_ingress](#input\_application\_gateway\_for\_ingress) | * `id` - (Required) The ID of the Application Gateway that be used as cluster ingress.<br>* `subnet_id` - (Optional) The ID of the Subnet which the Application Gateway is connected to. Must be set when `create_role_assignments` is `true`.<br>* `create_role_assignments` - (Optional) Whether to create the corresponding role assignments or not. Defaults to `true`. | <pre>object({<br> id = string<br> subnet_id = optional(string)<br> create_role_assignments = optional(bool, true)<br> })</pre> | `null` | no |
274281
| <a name="input_attached_acr_id_map"></a> [attached\_acr\_id\_map](#input\_attached\_acr\_id\_map) | Azure Container Registry ids that need an authentication mechanism with Azure Kubernetes Service (AKS). Map key must be static string as acr's name, the value is acr's resource id. Changing this forces some new resources to be created. | `map(string)` | `{}` | no |
275282
| <a name="input_auto_scaler_profile_balance_similar_node_groups"></a> [auto\_scaler\_profile\_balance\_similar\_node\_groups](#input\_auto\_scaler\_profile\_balance\_similar\_node\_groups) | Detect similar node groups and balance the number of nodes between them. Defaults to `false`. | `bool` | `false` | no |
276283
| <a name="input_auto_scaler_profile_empty_bulk_delete_max"></a> [auto\_scaler\_profile\_empty\_bulk\_delete\_max](#input\_auto\_scaler\_profile\_empty\_bulk\_delete\_max) | Maximum number of empty nodes that can be deleted at the same time. Defaults to `10`. | `number` | `10` | no |
@@ -310,7 +317,6 @@ No modules.
310317
| <a name="input_image_cleaner_enabled"></a> [image\_cleaner\_enabled](#input\_image\_cleaner\_enabled) | (Optional) Specifies whether Image Cleaner is enabled. | `bool` | `false` | no |
311318
| <a name="input_image_cleaner_interval_hours"></a> [image\_cleaner\_interval\_hours](#input\_image\_cleaner\_interval\_hours) | (Optional) Specifies the interval in hours when images should be cleaned up. Defaults to `48`. | `number` | `48` | no |
312319
| <a name="input_ingress_application_gateway_enabled"></a> [ingress\_application\_gateway\_enabled](#input\_ingress\_application\_gateway\_enabled) | Whether to deploy the Application Gateway ingress controller to this Kubernetes Cluster? | `bool` | `false` | no |
313-
| <a name="input_ingress_application_gateway_id"></a> [ingress\_application\_gateway\_id](#input\_ingress\_application\_gateway\_id) | The ID of the Application Gateway to integrate with the ingress controller of this Kubernetes Cluster. | `string` | `null` | no |
314320
| <a name="input_ingress_application_gateway_name"></a> [ingress\_application\_gateway\_name](#input\_ingress\_application\_gateway\_name) | The name of the Application Gateway to be used or created in the Nodepool Resource Group, which in turn will be integrated with the ingress controller of this Kubernetes Cluster. | `string` | `null` | no |
315321
| <a name="input_ingress_application_gateway_subnet_cidr"></a> [ingress\_application\_gateway\_subnet\_cidr](#input\_ingress\_application\_gateway\_subnet\_cidr) | The subnet CIDR to be used to create an Application Gateway, which in turn will be integrated with the ingress controller of this Kubernetes Cluster. | `string` | `null` | no |
316322
| <a name="input_ingress_application_gateway_subnet_id"></a> [ingress\_application\_gateway\_subnet\_id](#input\_ingress\_application\_gateway\_subnet\_id) | The ID of the subnet on which to create an Application Gateway, which in turn will be integrated with the ingress controller of this Kubernetes Cluster. | `string` | `null` | no |
Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
resource "kubernetes_namespace_v1" "example" {
2+
metadata {
3+
name = "example"
4+
}
5+
depends_on = [module.aks]
6+
}
7+
8+
resource "kubernetes_pod" "aspnet_app" {
9+
#checkov:skip=CKV_K8S_8:We don't need readiness probe for this simple example.
10+
#checkov:skip=CKV_K8S_9:We don't need readiness probe for this simple example.
11+
#checkov:skip=CKV_K8S_22:readOnlyRootFilesystem would block our pod from working
12+
#checkov:skip=CKV_K8S_28:capabilities would block our pod from working
13+
metadata {
14+
name = "aspnetapp"
15+
namespace = kubernetes_namespace_v1.example.metadata[0].name
16+
labels = {
17+
app = "aspnetapp"
18+
}
19+
}
20+
spec {
21+
container {
22+
name = "aspnetapp-image"
23+
image = "mcr.microsoft.com/dotnet/samples@sha256:7070894cc10d2b1e68e72057cca22040c5984cfae2ec3e079e34cf0a4da7fcea"
24+
image_pull_policy = "Always"
25+
security_context {}
26+
port {
27+
container_port = 80
28+
protocol = "TCP"
29+
}
30+
resources {
31+
requests = {
32+
cpu = "250m"
33+
memory = "256Mi"
34+
}
35+
limits = {
36+
cpu = "250m"
37+
memory = "256Mi"
38+
}
39+
}
40+
}
41+
}
42+
}
43+
44+
resource "kubernetes_service" "svc" {
45+
metadata {
46+
name = "aspnetapp"
47+
namespace = kubernetes_namespace_v1.example.metadata[0].name
48+
}
49+
spec {
50+
selector = {
51+
app = "aspnetapp"
52+
}
53+
port {
54+
port = 80
55+
target_port = 80
56+
protocol = "TCP"
57+
}
58+
}
59+
}
60+
61+
resource "kubernetes_ingress_v1" "ing" {
62+
metadata {
63+
name = "aspnetapp"
64+
namespace = kubernetes_namespace_v1.example.metadata[0].name
65+
annotations = {
66+
"kubernetes.io/ingress.class" : "azure/application-gateway"
67+
}
68+
}
69+
spec {
70+
rule {
71+
http {
72+
path {
73+
path = "/"
74+
backend {
75+
service {
76+
name = "aspnetapp"
77+
port {
78+
number = 80
79+
}
80+
}
81+
}
82+
path_type = "Exact"
83+
}
84+
}
85+
}
86+
}
87+
depends_on = [
88+
module.aks,
89+
]
90+
}
91+
92+
resource "time_sleep" "wait_for_ingress" {
93+
create_duration = var.use_existing_application_gateway ? "1m" : "15m"
94+
95+
depends_on = [kubernetes_ingress_v1.ing]
96+
}
97+
98+
data "kubernetes_ingress_v1" "ing" {
99+
metadata {
100+
name = "aspnetapp"
101+
namespace = kubernetes_namespace_v1.example.metadata[0].name
102+
}
103+
104+
depends_on = [time_sleep.wait_for_ingress]
105+
}
Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
resource "random_id" "prefix" {
2+
byte_length = 8
3+
}
4+
5+
resource "random_id" "name" {
6+
byte_length = 8
7+
}
8+
9+
resource "azurerm_resource_group" "main" {
10+
count = var.create_resource_group ? 1 : 0
11+
12+
location = var.location
13+
name = coalesce(var.resource_group_name, "${random_id.prefix.hex}-rg")
14+
}
15+
16+
locals {
17+
resource_group = {
18+
name = var.create_resource_group ? azurerm_resource_group.main[0].name : var.resource_group_name
19+
location = var.location
20+
}
21+
}
22+
23+
resource "azurerm_virtual_network" "test" {
24+
count = var.use_existing_application_gateway ? 1 : 0
25+
address_space = ["10.52.0.0/16"]
26+
location = local.resource_group.location
27+
name = "${random_id.prefix.hex}-vn"
28+
resource_group_name = local.resource_group.name
29+
}
30+
31+
resource "azurerm_subnet" "test" {
32+
count = var.use_existing_application_gateway ? 1 : 0
33+
address_prefixes = ["10.52.0.0/24"]
34+
name = "${random_id.prefix.hex}-sn"
35+
resource_group_name = local.resource_group.name
36+
virtual_network_name = azurerm_virtual_network.test[0].name
37+
}
38+
39+
resource "azurerm_subnet" "appgw" {
40+
count = var.use_existing_application_gateway ? 1 : 0
41+
42+
address_prefixes = ["10.52.1.0/24"]
43+
name = "${random_id.prefix.hex}-gw"
44+
resource_group_name = local.resource_group.name
45+
virtual_network_name = azurerm_virtual_network.test[0].name
46+
}
47+
48+
# Locals block for hardcoded names
49+
locals {
50+
backend_address_pool_name = try("${azurerm_virtual_network.test[0].name}-beap", "")
51+
frontend_ip_configuration_name = try("${azurerm_virtual_network.test[0].name}-feip", "")
52+
frontend_port_name = try("${azurerm_virtual_network.test[0].name}-feport", "")
53+
http_setting_name = try("${azurerm_virtual_network.test[0].name}-be-htst", "")
54+
listener_name = try("${azurerm_virtual_network.test[0].name}-httplstn", "")
55+
request_routing_rule_name = try("${azurerm_virtual_network.test[0].name}-rqrt", "")
56+
}
57+
58+
resource "azurerm_public_ip" "pip" {
59+
count = var.use_existing_application_gateway ? 1 : 0
60+
61+
allocation_method = "Static"
62+
location = local.resource_group.location
63+
name = "appgw-pip"
64+
resource_group_name = local.resource_group.name
65+
sku = "Standard"
66+
}
67+
68+
resource "azurerm_application_gateway" "appgw" {
69+
count = var.use_existing_application_gateway ? 1 : 0
70+
71+
location = local.resource_group.location
72+
#checkov:skip=CKV_AZURE_120:We don't need the WAF for this simple example
73+
name = "ingress"
74+
resource_group_name = local.resource_group.name
75+
76+
backend_address_pool {
77+
name = local.backend_address_pool_name
78+
}
79+
backend_http_settings {
80+
cookie_based_affinity = "Disabled"
81+
name = local.http_setting_name
82+
port = 80
83+
protocol = "Http"
84+
request_timeout = 1
85+
}
86+
frontend_ip_configuration {
87+
name = local.frontend_ip_configuration_name
88+
public_ip_address_id = azurerm_public_ip.pip[0].id
89+
}
90+
frontend_port {
91+
name = local.frontend_port_name
92+
port = 80
93+
}
94+
gateway_ip_configuration {
95+
name = "appGatewayIpConfig"
96+
subnet_id = azurerm_subnet.appgw[0].id
97+
}
98+
http_listener {
99+
frontend_ip_configuration_name = local.frontend_ip_configuration_name
100+
frontend_port_name = local.frontend_port_name
101+
name = local.listener_name
102+
protocol = "Http"
103+
}
104+
request_routing_rule {
105+
http_listener_name = local.listener_name
106+
name = local.request_routing_rule_name
107+
rule_type = "Basic"
108+
backend_address_pool_name = local.backend_address_pool_name
109+
backend_http_settings_name = local.http_setting_name
110+
priority = 1
111+
}
112+
sku {
113+
name = "Standard_v2"
114+
tier = "Standard_v2"
115+
capacity = 1
116+
}
117+
118+
lifecycle {
119+
ignore_changes = [
120+
tags,
121+
backend_address_pool,
122+
backend_http_settings,
123+
http_listener,
124+
probe,
125+
request_routing_rule,
126+
url_path_map,
127+
]
128+
}
129+
}
130+
131+
module "aks" {
132+
#checkov:skip=CKV_AZURE_141:We enable admin account here so we can provision K8s resources directly in this simple example
133+
source = "../.."
134+
135+
prefix = random_id.name.hex
136+
resource_group_name = local.resource_group.name
137+
kubernetes_version = "1.26" # don't specify the patch version!
138+
automatic_channel_upgrade = "patch"
139+
agents_availability_zones = ["1", "2"]
140+
agents_count = null
141+
agents_max_count = 2
142+
agents_max_pods = 100
143+
agents_min_count = 1
144+
agents_pool_name = "testnodepool"
145+
agents_pool_linux_os_configs = [
146+
{
147+
transparent_huge_page_enabled = "always"
148+
sysctl_configs = [
149+
{
150+
fs_aio_max_nr = 65536
151+
fs_file_max = 100000
152+
fs_inotify_max_user_watches = 1000000
153+
}
154+
]
155+
}
156+
]
157+
agents_type = "VirtualMachineScaleSets"
158+
azure_policy_enabled = true
159+
enable_auto_scaling = true
160+
enable_host_encryption = true
161+
http_application_routing_enabled = true
162+
application_gateway_for_ingress = {
163+
new_gw = var.use_existing_application_gateway ? null : {
164+
name = "ingress"
165+
subnet_cidr = "10.225.0.0/16"
166+
}
167+
existing_gw = var.use_existing_application_gateway ? {
168+
id = azurerm_application_gateway.appgw[0].id
169+
subnet_id = azurerm_subnet.appgw[0].id
170+
} : null
171+
}
172+
local_account_disabled = false
173+
log_analytics_workspace_enabled = false
174+
net_profile_dns_service_ip = "10.0.0.10"
175+
net_profile_service_cidr = "10.0.0.0/16"
176+
network_plugin = "azure"
177+
network_policy = "azure"
178+
os_disk_size_gb = 60
179+
private_cluster_enabled = false
180+
public_network_access_enabled = true
181+
rbac_aad = true
182+
rbac_aad_managed = true
183+
role_based_access_control_enabled = true
184+
sku_tier = "Standard"
185+
vnet_subnet_id = var.use_existing_application_gateway ? azurerm_subnet.test[0].id : null
186+
depends_on = [
187+
azurerm_subnet.test,
188+
]
189+
}
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
output "ingress_endpoint" {
2+
depends_on = [time_sleep.wait_for_ingress]
3+
value = "http://${data.kubernetes_ingress_v1.ing.status[0].load_balancer[0].ingress[0].ip}"
4+
}
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
terraform {
2+
required_version = ">=1.3"
3+
required_providers {
4+
azurerm = {
5+
source = "hashicorp/azurerm"
6+
version = ">= 3.51, < 4.0"
7+
}
8+
kubernetes = {
9+
source = "hashicorp/kubernetes"
10+
version = "2.22.0"
11+
}
12+
random = {
13+
source = "hashicorp/random"
14+
version = "3.3.2"
15+
}
16+
time = {
17+
source = "hashicorp/time"
18+
version = "0.9.1"
19+
}
20+
}
21+
}
22+
23+
provider "azurerm" {
24+
features {
25+
resource_group {
26+
prevent_deletion_if_contains_resources = false
27+
}
28+
}
29+
}
30+
31+
provider "kubernetes" {
32+
host = module.aks.admin_host
33+
client_certificate = base64decode(module.aks.admin_client_certificate)
34+
client_key = base64decode(module.aks.admin_client_key)
35+
cluster_ca_certificate = base64decode(module.aks.admin_cluster_ca_certificate)
36+
}
37+
38+
provider "random" {}
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
variable "create_resource_group" {
2+
type = bool
3+
default = true
4+
nullable = false
5+
}
6+
7+
variable "location" {
8+
default = "eastus"
9+
}
10+
11+
variable "resource_group_name" {
12+
type = string
13+
default = null
14+
}
15+
16+
variable "use_existing_application_gateway" {
17+
type = bool
18+
default = false
19+
}

0 commit comments

Comments
 (0)