[Question] Stale neighbor/ARP cache with wrong MAC for reused pod IP causing intermittent connection timeouts

[slice-mohijeet]

Summary

We are investigating an intermittent pod-to-pod connectivity issue on EKS
with Amazon VPC CNI. After significant debugging, we have narrowed the
symptom to a stale neighbor (ARP) cache entry on the source node holding
a wrong MAC address for a destination pod IP. We believe this wrong MAC
may belong to a previously deleted node's ENI, and the entry remains in
REACHABLE state indefinitely due to Linux NUD behavior.

We are opening this as a question to understand:

1. Whether VPC CNI is expected to handle this via gratuitous ARP (GARP)
2. Whether this is a known gap or an already-tracked issue
3. What the recommended mitigation is

---

Environment

• Platform: AWS EKS
• Node OS: Bottlerocket
• CNI: Amazon VPC CNI (aws-node)
• Istio: Used as ingress gateway only (no sidecars, no ztunnel on
  affected path)
• Node type: Standard EC2 instances with multiple ENIs (VPC CNI
  secondary IP mode)

---

Observed Symptom

Intermittent connection timeouts from a specific source node/pod to a
specific destination pod IP. The same destination pod IP works fine from
other source nodes. Deleting the neighbor cache entry on the source node
with ip neigh del <dst-ip> dev eth0 fixes the issue immediately, and
after fresh ARP resolution a different (new) MAC is learned for the same IP.

Initially appeared as Istio ingress 503 UF / connection_timeout, but
direct curl from source node to destination pod IP:port reproduces the
failure, ruling out Istio configuration as the root cause.

---

Debugging Steps Performed

1. Direct pod IP reachability confirmed as the failure point

Direct curl from source node to destination pod IP:port timed out,
bypassing any VirtualService or Istio routing logic. Same curl from a
healthy source node to the same destination pod IP worked immediately.

2. Failure is source-node-specific and destination-IP-specific

• Source node A → destination pod IP X: fails
• Source node B → destination pod IP X: works
• Source node A → replacement pod IP Y (same destination node): works

This ruled out the destination application, destination node health, and
generic cluster-wide issues.

3. tcpdump on source node showed only SYN retransmits

10.131.60.193 -> 10.131.55.225:8080 [S]
10.131.60.193 -> 10.131.55.225:8080 [S] (retransmit)
10.131.60.193 -> 10.131.55.225:8080 [S] (retransmit)
...

No SYN-ACK was ever seen. The TCP handshake never completed.

4. conntrack showed SYN_SENT [UNREPLIED]

src=10.131.60.193 dst=10.131.55.225 dport=8080 SYN_SENT [UNREPLIED]

SYN left the source but no reply was tracked by conntrack.

5. Simultaneous behavior: existing TCP flow works, new SYN fails

In one capture, a pre-existing established TCP connection between the
same source and destination IPs was still exchanging data while new SYN
packets to the same destination IP kept retransmitting and never
completed. This suggested the issue affects new connection establishment
and path resolution, not total connectivity.

6. ip neigh showed wrong MAC as REACHABLE

$ ip -s neigh show 10.131.49.193
10.131.49.193 dev eth0 lladdr 0a:47:d5:85:ae:29 ref 1 used ... REACHABLE

After ip neigh del 10.131.49.193 dev eth0 and arping:

$ arping -c 5 -I eth0 10.131.49.193
Unicast reply from 10.131.49.193 [0a:04:bb:8d:7b:61]

A completely different MAC was resolved, and connectivity was immediately
restored. This pattern repeated across multiple destination pod IPs and
multiple incidents.

7. Asymmetric reachability observed

• Source node → destination pod IP: fails
• Destination pod → source node: works

This strongly suggests the issue is on the outbound path from the source
node, not a bidirectional break.

8. conntrack table exhaustion ruled out

nf_conntrack_count = 1343
nf_conntrack_max = 131072

9. iptables ruled out

Compared iptables rules on faulty vs healthy source nodes. No DROP/REJECT
rule specific to affected pod IPs or ports found.

---

Current Hypothesis (Not Confirmed)

We suspect the following sequence may be occurring:

1. A destination pod IP (secondary ENI IP) was previously assigned to an
   ENI on a now-deleted node. The source node learned the correct MAC for
   that IP at the time.

2. The node was deleted. The same pod IP was subsequently reassigned —
   either to a new pod on the same destination node via a different ENI,
   or to a pod on a different node entirely.

3. VPC CNI (or AWS fabric) updates L3 routing for the IP correctly, but
   the source node's neighbor cache is not invalidated. The old MAC —
   now belonging to a deleted node's ENI — remains cached.

4. A separate TCP flow exists where the destination pod has a connection
   to the source node (for an unrelated purpose). The destination pod
   sends packets to the source node via correct VPC L3 routing. The source
   node receives these packets.

5. Linux NUD treats these inbound packets as upper-layer confirmation
   (neigh_confirm()) for the destination IP's neighbor entry, resetting
   the REACHABLE timer — even though the outbound path uses a dead MAC.

6. The neighbor entry never goes STALE → never goes to PROBE → dead MAC
   is never re-ARPed → stale entry persists indefinitely.

7. New TCP connections from source to destination use the dead MAC, are
   silently dropped by the AWS fabric (no ICMP, no RST), and the SYN
   retransmits forever.

The core question is: In VPC CNI's IP reuse lifecycle (pod deletion →
IP released → IP reassigned to new pod/node), is a gratuitous ARP (GARP)
expected to be sent by VPC CNI or the AWS fabric to invalidate stale
neighbor entries on remote nodes?

---

Questions

1. Does VPC CNI send a gratuitous ARP when a secondary ENI IP is
   assigned to a new pod? If yes, is there a scenario where this GARP
   could be missed or not reach a particular source node?

2. Is net.ipv4.conf.eth0.arp_notify=1 expected to be set on EKS nodes
   with VPC CNI? This sysctl causes the kernel to send a GARP when an
   IP address is added to an interface, which would naturally invalidate
   stale entries on peer nodes.

3. Is there a known race condition or gap between IP reuse in VPC CNI
   and neighbor cache invalidation on remote nodes, particularly when the
   reused IP belongs to a completely different node than before?

4. Is the Linux NUD upper-layer confirmation behavior (where inbound
   TCP packets from a destination keep its neighbor entry REACHABLE on the
   source even when the outbound MAC is wrong) a known interaction with
   VPC CNI's IP reuse model? Is this tracked anywhere?

5. Are there recommended sysctl settings (base_reachable_time_ms,
   gc_stale_time, arp_notify) for EKS/VPC CNI nodes to reduce the
   window in which stale neighbor entries can cause this class of failure?

---

Workaround

Manually running the following on the source node restores connectivity
immediately:

ip neigh del <destination-pod-ip> dev eth0

After deletion, fresh ARP resolution retrieves the correct MAC and new
TCP connections succeed.

---

References / Related

• Linux NUD state machine and upper-layer confirmation:
  neigh_confirm() in net/core/neighbour.c
• net.ipv4.conf.all.arp_notify kernel documentation
• Other CNIs (Antrea, Calico, Flannel) explicitly send GARP on pod
  startup to handle this class of stale ARP problem
• Amazon VPC CNI repo: https://github.com/aws/amazon-vpc-cni-k8s

---

Additional Context

Happy to provide additional node diagnostics, kernel versions, VPC CNI
version, or packet captures if helpful for investigation.

[mefju]

@slice-mohijeet What version of VPC CNI are you using?

[slice-mohijeet]

we are using
Version : v1.20.1-eksbuild.1

Also i want to add few other findings

Observation

We observed a specific behavior in our EKS cluster that we want to
understand better before assuming it is a bug.

• EKS, Bottlerocket nodes
• Amazon VPC CNI, prefix delegation (/28)
• Istio ingress gateway with NodePort service
• NLB with source IP preservation enabled

---

Setup

Source node:      ip-10-131-xx-xxx  
Destination pod:  10.131.xx.xxx  (Java application)
NLB:              10.131.xx.xxx  (source IP preservation enabled)
NodePort:         80:3xxxx/TCP   on source node

The destination pod's Java application makes outbound calls to the NLB.
NLB forwards this to the NodePort on the source node with source IP
preserved as the pod's IP.

---

What We Observed

Neighbor cache had wrong MAC as REACHABLE

On source node:

$ ip -s neigh show <DST-POD-IP>
<DST-POD-IP> dev eth0 lladdr 0a:**:**:**:**:**
used 1926/1/1923 probes 4 REACHABLE

Key observations:

• confirmed = 1 second ago — continuously refreshed every few seconds
• probes 4 — kernel sent 4 unicast ARP probes, received no reply
  to any of them
• Entry still REACHABLE — probe cycle never completed to FAILED

Correct MAC confirmed via arping returned a completely different MAC
than what was cached in the neighbor table.

---

tcpdump confirmed two simultaneous behaviors

While wrong MAC was cached, tcpdump on source node showed:

Flow 1 — NLB-forwarded connection on NodePort — working:

<DST-POD-IP>:xxxxx → <SRC-NODE-IP>:3xxxx  [P.]  data
<SRC-NODE-IP>:3xxxx → <DST-POD-IP>:xxxxx  [.]   ACK

Fully bidirectional, continuously flowing for 1+ hours.

Flow 2 — New direct connection to pod port 8080 — failing:

<SRC-NODE-IP>:xxxxx → <DST-POD-IP>:8080  [S]
<SRC-NODE-IP>:xxxxx → <DST-POD-IP>:8080  [S]  retransmit
<SRC-NODE-IP>:xxxxx → <DST-POD-IP>:8080  [S]  retransmit
<SRC-NODE-IP>:xxxxx → <DST-POD-IP>:8080  [S]  retransmit

Zero SYN-ACKs. Both flows happening simultaneously to same
destination IP.

After arping — direct connection worked immediately:

$ curl -v <DST-POD-IP>:8080
* Established connection to <DST-POD-IP> port 8080
< HTTP/1.1 404 Not Found

Pod was healthy throughout. Problem was purely the wrong MAC in
neighbor cache.

---

tcpdump with -e flag showed three distinct MACs

Inbound  (<DST-POD-IP> → <SRC-NODE-IP>:NodePort):
  0a:**:**:**:**:** → 0a:**:**:**:**:**
  (unknown MAC → source node MAC)

Outbound (<SRC-NODE-IP>:NodePort → <DST-POD-IP>):
  0a:**:**:**:**:** → 0a:**:**:**:**:**
  (source node MAC → correct pod ENI MAC learned after arping)

Three distinct MACs observed:

MAC-A:  cached in neighbor table for DST-POD-IP  → wrong
MAC-B:  arriving on inbound packets from DST-POD-IP 
        → does NOT exist as any ENI in our AWS account
          (verified via aws ec2 describe-network-interfaces)
MAC-C:  correct pod ENI MAC returned by arping
        → direct connection worked after arping updated cache

MAC-A and MAC-B and MAC-C are all three different from each other.
MAC-B does not exist in AWS ENI inventory at all.

---

Identified the connection via pod namespace

We entered the pod network namespace on the destination node:

ESTAB  <DST-POD-IP>:52xxx  <NLB-IP>:443
users:(("java",pid=xxxxx,fd=xxx))

Source port exactly matched what we saw in tcpdump.
The Java app was making outbound HTTPS calls to the NLB,
which were being forwarded to the NodePort on the source node
with source IP preserved as the pod IP.

---

Neighbor cache GC thresholds

gc_thresh1 = 8096
gc_thresh2 = 12288
gc_thresh3 = 16384

Current neighbor table size: 1102 entries
→ GC never runs (1102 << 8096)

State breakdown:
  REACHABLE:   84
  STALE:     1009
  FAILED:       1
  DELAY:        8

---

Our Question

When inbound packets arrive at the source node with:

• Source IP = pod IP (preserved by NLB)
• Source MAC = MAC-B (not any known ENI in our account)

The Linux kernel calls neigh_confirm() for the pod IP, which resets
the REACHABLE timer on whatever MAC is currently cached — without
verifying that the outbound path using that MAC actually works.

The inbound and outbound paths are completely different:

Inbound:   pod → NLB → VPC fabric → source node
           works regardless of what MAC source node has cached

Outbound:  source node → uses cached MAC → direct to pod
           broken if cached MAC is wrong

The kernel cannot distinguish between these two paths. Both look
identical — same source IP. Both call neigh_confirm().

Specific question:

Is this a known interaction between NLB source IP preservation and
Linux NUD (neigh_confirm()) in VPC CNI's networking model?

When NLB preserves the source IP of a backend pod and forwards
traffic to a NodePort on another node, inbound packets from that
pod IP continuously call neigh_confirm() on the receiving node —
keeping whatever MAC is cached for that pod IP as REACHABLE
indefinitely, even when that MAC is wrong and direct pod
connectivity from that node is completely broken.

We observed this live:

• probes 4 — all unanswered — yet entry stayed REACHABLE
• Each probe cycle aborted by neigh_confirm() from NLB-forwarded
  inbound traffic before it could complete to FAILED
• Wrong MAC locked as REACHABLE for 1+ hours
• Only fixed by manual ip neigh del

Is this expected? Is there a recommended configuration to prevent
neigh_confirm() from being fed by NLB-forwarded traffic that goes
through a different L2 path than direct pod-to-pod connectivity?

---

[viveksb007]

@slice-mohijeet where is following config coming from?

gc_thresh1 = 8096

we have this config as 0 in bottlerocket AMIs (https://github.com/bottlerocket-os/bottlerocket-core-kit/blob/dfe9e5eef4cb3c6a24c1ec9740a02197226590e1/packages/release/release-sysctl.conf#L9)

Is there any override of these settings while launching nodes in your cluster?

[slice-mohijeet]

Yes, that net.ipv4.neigh.default.gc_thresh1 is explicitly overridden to 8096 by our configuration which we have made 0, while deep diving OS behaviour i suspect that eks supports multiple OS with VPC cni as ENI option and in all of those OS i think value is 128 as default not 0.

[viveksb007]

This setting is not override in EKS-AMI (https://github.com/awslabs/amazon-eks-ami) but comes from base AL2023 AMI.

If you boot-up vanilla AL2023 and do the following

sysctl -a | grep -E gc_thresh

net.ipv4.neigh.default.gc_thresh1 = 0

gc_thresh1 as 0 can be verified. Upstream linux has set that value as 128 but sysctl-defaults-1.0-3.amzn2023.noarch sets it as 0.

cat 00-defaults.conf
# Maximize console logging level for kernel printk messages
kernel.printk = 8 4 1 7

# Wait 5 seconds and then reboot
kernel.panic = 5

# Allow neighbor cache entries to expire even when the cache is not full
net.ipv4.neigh.default.gc_thresh1 = 0
net.ipv6.neigh.default.gc_thresh1 = 0

# Avoid neighbor table contention in large subnets
net.ipv4.neigh.default.gc_thresh2 = 15360
net.ipv6.neigh.default.gc_thresh2 = 15360
net.ipv4.neigh.default.gc_thresh3 = 16384
net.ipv6.neigh.default.gc_thresh3 = 16384

# Increasing to account for skb structure growth since the 3.4.x kernel series
net.ipv4.tcp_wmem = 4096 20480 4194304

# Set default TTL to 127.
net.ipv4.ip_default_ttl = 127

# Disable unprivileged access to bpf
kernel.unprivileged_bpf_disabled = 1

[slice-mohijeet]

Does it mean net.ipv4.neigh.default.gc_thresh1 = 0 comes as default with all OS supported by EKS and do not require any additonal changes ? in default parameters

[viveksb007]

No it doesn't mean all OS have that. We checked the Ubuntu EKS AMI, it has 128. Only Amazon AMIs have it as 0.

For this issue, ARP GC isn't cleaning the STALE entries because of overriden configs. From above output, table has 1009 STALE entries but they aren't getting cleaned as its less than 8096.