feat: add kubernetes discovery module by zhixiongdu027 · Pull Request #4880 · apache/apisix

zhixiongdu027 · 2021-08-23T14:51:08Z

Signed-off-by: adugeek root@libssl.com

What this PR does / why we need it:

Pre-submission checklist:

Did you explain what problem does this PR solve? Or what new features have been added?
Have you added corresponding test cases?
Have you modified the corresponding document?
Is this PR backward compatible? If it is not backward compatible, please discuss on the mailing list first

#4388

todo:

add docs [done]
support namespace-selector [done]
support label-selector [done]

Signed-off-by: adugeek <root@libssl.com>

apisix/discovery/k8s.lua

ci/install-ext-services-via-docker.sh

tao12345666333 · 2021-08-24T03:15:42Z

Thanks for your contribution, I will review it today

Signed-off-by: adugeek <root@libssl.com>

t/APISIX.pm

Signed-off-by: adugeek <root@libssl.com>

tzssangglass · 2021-09-07T08:52:21Z

If you want to use environment variables in code, you can use: os.getenv()

If you want to use environment variables in test cases, you can

use --- main_config directive , example:

apisix/t/plugin/ext-plugin/sanity.t

Lines 391 to 392 in d3e104e

--- main_config

env MY_ENV_VAR=foo;
or define the environment variables in APISIX.pm just as you did

apisix/discovery/k8s.lua

Signed-off-by: adugeek <root@libssl.com>

zhixiongdu027

ci file is updated for mount kubernetes token file into centos container

apisix/discovery/k8s.lua

tao12345666333 · 2021-09-19T15:39:08Z

ci/install-ext-services-via-docker.sh

+curl -Lo ./jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64
+chmod +x ./jq
+
+until [[ $(curl 127.0.0.1:6445/api/v1/pods?fieldSelector=status.phase%21%3DRunning |./jq .items) == "[]" ]]; do


you can just using kubectl wait --for=condition=Ready pods --all -A instead of this.

In my test
When kubectl wait --for=condition=Ready pods --all -A is finished,
Still have pod Pending or ContainerCreating
This will cause the test to fail

@tao12345666333

maybe we can use like this:

echo "wait k8s start..." sleep 10 until [[ $(./kubectl get pods -A --field-selector 'status.phase!=Running' 2>&1) =~ "No resources found" ]]; do echo 'still wait k8s start...' sleep 1 done

ci/install-ext-services-via-docker.sh

Signed-off-by: adugeek <root@libssl.com>

membphis · 2022-02-09T04:26:59Z

apisix/discovery/kubernetes/init.lua

+    -- TODO: maybe we can read dict name from discovery config
+    endpoint_dict = ngx.shared.discovery
+    if not endpoint_dict then
+        error("failed to get ngx.shared.dict discovery")


failed to get Nginx shared dict: discovery, please check your APISIX version

zhixiongdu027 · 2022-02-14T01:10:41Z

@membphis @spacewander @tao12345666333 @crazyMonkey1995
Hi, I think the current code is more in line with the apisix style.
Please help start ci and reviews.
Many THK.

tao12345666333 · 2022-02-14T02:15:23Z

ok, thanks for your contribution.

tangzhenhuang · 2022-02-14T06:54:04Z

apisix/discovery/kubernetes/informer_factory.lua

+
+    informer.continue = data.metadata.continue
+    if informer.continue and informer.continue ~= "" then
+        list(httpc, informer)


Seems like the apiserver(the second parameter of the function) is missing here?

Sorry, It'a mistake

zhixiongdu027 · 2022-02-14T10:05:37Z

@crazyMonkey1995
HI, tks review verymuch .and mistake had fixed.
Please help review again.

tzssangglass · 2022-02-16T02:27:59Z

cc @tao12345666333 @membphis @spacewander @tokers @crazyMonkey1995

Looks like this PR is complete?

tangzhenhuang · 2022-02-16T02:36:47Z

cc @tao12345666333 @membphis @spacewander @tokers @crazyMonkey1995

Looks like this PR is complete?

I think it's ok.

tao12345666333 · 2022-02-16T02:38:02Z

It's on my list. I will review it ASAP

membphis

LGTM

tao12345666333

LGTM

Wouldn't it be better if namespaces were not directly hardcoded in the configuration file, but filtered by labels?

apisix/discovery/kubernetes/informer_factory.lua

docs/zh/latest/discovery/kubernetes.md

tao12345666333 · 2022-02-18T10:09:20Z

docs/zh/latest/discovery/kubernetes.md

+      schema: https #default https
+
+      # kubernetes apiserver host, options [ ipv4 | ipv6 | domain | env variable]
+      host: 10.0.8.95 #default ${KUBERNETES_SERVICE_HOST}


I suggest writing kubernetes.default directly here, what do you think?

@zhixiongdu027 what do you think?

We may have two operating environments:

Inside the Pod:
Both the apiserver host and apiserver port values are stored via environment variables.
Therefore, our configuration items must support the reference environment variable value, and the user needs to recognize the referenced environment variable value at a glance.
${} is a good form

Outside the Pod:
The apiserver host and apiserver port values are manually input by the user. We need a specific format to distinguish whether the value entered by the user is a domain name or an environment variable,
${} is also a good form

If we have next config:

kubernetes: service: host: kubernets.default

How can we tell if "kubernets.default" is a constant or a domain name?

In the cluster we can get the address of the kubernetes service through KUBERNETES_SERVICE_HOST, but once we really need to verify the SSL certificate, most of the time we do not issue a certificate for the IP address. Domain names are far more generic than IP addresses.

most of the time we do not issue a certificate for the IP address. Domain names are far more generic than IP addresses.

Usually this is the case.

But in the clusters I have seen, the san list of the certificates used by the apiserver all contain${KUBERNETES_SERVICE_HOST} and Host_IP,

so even if "ssl_verify=true "
I think there is no problem to use ${KUBERNETES_SERVICE_HOST} as the server address within the Pod.

maybe you know some exception?

@tao12345666333

We can keep the status for now.

zhixiongdu027 · 2022-02-19T03:19:43Z

@membphis @tao12345666333 @crazyMonkey1995 @tokers
Hi ,doc had update.
Please help start ci and review again .
Many TKS.

tao12345666333

We can keep this implementation for now, let's move on.

jianhaiqing · 2022-02-23T02:25:52Z

it's a big move. We keep eyes on this feature.

* upstream: (52 commits) feat: add kubernetes discovery module (apache#4880) docs: fix For L7 proxy -> For L4 proxy (apache#6423) fix(deps): upgrade jsonschema to 0.9.8 (apache#6407) docs: translate Chinese to English in en clickhouse-logger (apache#6416) docs: add zh proxy-control.md&modify other doc error (apache#6346) docs: update public API relative usage (apache#6318) docs(cn): remove datadog from sidebar & fix doc lint conf (apache#6411) fix(request-validation): should not limit the urlencoded post args number (apache#6396) docs: fix configuration file typo (apache#6395) docs(extern-plugin): the implementation of runner (apache#6336) docs: polishing skywalking-logger plugin's docs (apache#6377) doc: adjust the directory structure of observability's documents (apache#6391) change(admin): empty nodes should be encoded as array (apache#6384) fix: should not limit the header number (apache#6379) ci: remove unnecessary tmate action (apache#6367) fix(opentelemetry): batch_span_processor export zero length spans (apache#6349) feat(graphql): support http get and post json request (apache#6343) feat: support for configuring the number of etcd health check retries (apache#6322) feat(wasm): support getting request body (apache#6325) fix(hmac-auth): hmac-auth plugin sort array param (apache#6314) ...

feat: add k8s discovery module

aaa2642

Signed-off-by: adugeek <root@libssl.com>

spacewander requested review from tao12345666333 and tokers August 24, 2021 01:43

tokers reviewed Aug 24, 2021

View reviewed changes

apisix/discovery/k8s.lua Outdated Show resolved Hide resolved

apisix/discovery/k8s.lua Outdated Show resolved Hide resolved

apisix/discovery/k8s.lua Outdated Show resolved Hide resolved

zhixiongdu027 requested a review from tokers August 24, 2021 02:02

zhixiongdu027 commented Aug 24, 2021

View reviewed changes

ci/install-ext-services-via-docker.sh Outdated Show resolved Hide resolved

tao12345666333 self-assigned this Aug 24, 2021

zhixiogndu added 2 commits September 5, 2021 20:09

feat: add k8s discovery module

5a351d1

Signed-off-by: adugeek <root@libssl.com>

feat: add k8s discovery module

77dd742

Signed-off-by: adugeek <root@libssl.com>

starsz reviewed Sep 6, 2021

View reviewed changes

t/APISIX.pm Outdated Show resolved Hide resolved

feat: add k8s discovery module

45e1bcd

Signed-off-by: adugeek <root@libssl.com>

zhixiongdu027 requested a review from starsz September 6, 2021 05:57

zhixiogndu added 2 commits September 6, 2021 23:46

feat: add k8s discovery module

8042718

Signed-off-by: adugeek <root@libssl.com>

feat: add k8s discovery module

2694c7f

Signed-off-by: adugeek <root@libssl.com>

tzssangglass reviewed Sep 7, 2021

View reviewed changes

apisix/discovery/k8s.lua Outdated Show resolved Hide resolved

zhixiogndu added 4 commits September 8, 2021 08:32

feat: add k8s discovery module

046b65c

Signed-off-by: adugeek <root@libssl.com>

feat: add k8s discovery module

99aef06

Signed-off-by: adugeek <root@libssl.com>

feat: add k8s discovery module

276705a

Signed-off-by: adugeek <root@libssl.com>

feat: add k8s discovery module

9cbe29a

Signed-off-by: adugeek <root@libssl.com>

zhixiongdu027 mentioned this pull request Sep 19, 2021

request help: Set environment variable failed #5102

Closed

adugeek added 3 commits September 19, 2021 15:08

Merge branch 'apache:master' into master

3f5dbfd

feat: add k8s discovery module

a1d9b8e

Signed-off-by: adugeek <root@libssl.com>

feat: add k8s discovery module

35ad639

Signed-off-by: adugeek <root@libssl.com>

zhixiongdu027 commented Sep 19, 2021

View reviewed changes

tao12345666333 reviewed Sep 19, 2021

View reviewed changes

feat: add k8s discovery module

f665734

Signed-off-by: adugeek <root@libssl.com>

zhixiongdu027 requested review from tao12345666333 and tzssangglass September 20, 2021 13:05

zhixiongdu027 requested a review from membphis February 9, 2022 03:25

membphis reviewed Feb 9, 2022

View reviewed changes

membphis previously approved these changes Feb 9, 2022

View reviewed changes

feat: add kubernetes discovery module

02ef870

spacewander dismissed membphis’s stale review via 02ef870 February 11, 2022 03:41

feat: add kubernetes discovery module

fa017d2

zhixiongdu027 requested a review from membphis February 14, 2022 01:20

tangzhenhuang reviewed Feb 14, 2022

View reviewed changes

zhixiongdu added 3 commits February 14, 2022 17:52

feat: add kubernetes discovery module

57d7f91

feat: add kubernetes discovery module

5e85c1d

feat: add kubernetes discovery module

b2940ab

tzssangglass previously approved these changes Feb 16, 2022

View reviewed changes

membphis previously approved these changes Feb 16, 2022

View reviewed changes

tao12345666333 reviewed Feb 18, 2022

View reviewed changes

docs: improve kubernetes.md

2276693

zhixiongdu027 dismissed stale reviews from membphis and tzssangglass via 2276693 February 19, 2022 03:17

membphis approved these changes Feb 20, 2022

View reviewed changes

zhixiongdu027 requested a review from tao12345666333 February 21, 2022 01:12

tao12345666333 approved these changes Feb 23, 2022

View reviewed changes

spacewander merged commit f7b50f2 into apache:master Feb 23, 2022

Conversation

zhixiongdu027 commented Aug 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Pre-submission checklist:

todo:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tao12345666333 commented Aug 24, 2021

Uh oh!

Uh oh!

tzssangglass commented Sep 7, 2021

Uh oh!

Uh oh!

zhixiongdu027 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhixiongdu027 Sep 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhixiongdu027 Sep 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhixiongdu027 commented Feb 14, 2022

Uh oh!

tao12345666333 commented Feb 14, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhixiongdu027 commented Feb 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tzssangglass commented Feb 16, 2022

Uh oh!

tangzhenhuang commented Feb 16, 2022

Uh oh!

tao12345666333 commented Feb 16, 2022

Uh oh!

membphis left a comment

Choose a reason for hiding this comment

Uh oh!

tao12345666333 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhixiongdu027 Feb 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhixiongdu027 commented Feb 19, 2022

Uh oh!

tao12345666333 left a comment

Choose a reason for hiding this comment

Uh oh!

zhixiongdu027 commented Aug 23, 2021 •

edited

Loading

zhixiongdu027 Sep 20, 2021 •

edited

Loading

zhixiongdu027 Sep 20, 2021 •

edited

Loading

zhixiongdu027 commented Feb 14, 2022 •

edited

Loading

zhixiongdu027 Feb 19, 2022 •

edited

Loading