DubboMetadataServiceProxy.removeProxy Lead to cpu 100%

[wywanchongshan]

Environment

The framework I user is spring cloud alibaba

       <dependency>
                <groupId>com.alibaba.cloud</groupId>
                <artifactId>spring-cloud-alibaba-dependencies</artifactId>
                <version>2.2.2.RELEASE</version>
            </dependency>

The dubbo version information used internally is

      <dependency>
            <groupId>com.alibaba.cloud</groupId>
            <artifactId>spring-cloud-starter-dubbo</artifactId>
        </dependency>

The service registry uses nacos

       <dependency>
            <groupId>com.alibaba.cloud</groupId>
            <artifactId>spring-cloud-starter-alibaba-nacos-discovery</artifactId>
        </dependency>

• Dubbo version: 2.7.8
• Operating System version: Linux version 4.19.91-21.al7.x86_64 (mockbuild@koji-aliyun011016174077.na61) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)) 不同的group应该是不同的注册中心 #1 SMP Wed Sep 2 19:47:49 CST 2020
• Java version: 1.8.0_202

Steps to reproduce this issue

I have two projects, A and B, and A depends on project B. When I started Project A, there was no problem.

Dubbo related configuration

dubbo:
  scan:
    base-packages: xxx.xxx.dubbo.service
  protocol:
    port: -1
    name: dubbo
  provider:
    loadbalance: gray
  consumer:
    filter: dubboContextConsumerFilter
    check: false
  cloud:
    subscribed-services: B

But when I start project B, the nacos listener of project A will perform service subscription discovery processing when it is running. At this time, it will cause cpu 100% .
Note: No problems in development and test environment

top -H -p 4279

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4763 prod 20 0 3908772 509644 16708 R 99.9 6.5 25:18.14 java
4279 prod 20 0 3908772 509644 16708 S 0.0 6.5 0:00.00 java
4281 prod 20 0 3908772 509644 16708 S 0.0 6.5 0:09.95 java
4282 prod 20 0 3908772 509644 16708 S 0.0 6.5 0:00.21 java

jstack 4279 >> 1.txt

Stack information of the thread in question

"com.alibaba.nacos.naming.client.listener" #459 daemon prio=5 os_prio=0 tid=0x00007f236d0d7000 nid=0x129b runnable [0x00007f23557fc000]
   java.lang.Thread.State: RUNNABLE
	at java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1172)
	at java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
	at com.alibaba.cloud.dubbo.service.DubboMetadataServiceProxy.removeProxy(DubboMetadataServiceProxy.java:76)
	at com.alibaba.cloud.dubbo.metadata.repository.DubboServiceMetadataRepository.removeMetadataAndInitializedService(DubboServiceMetadataRepository.java:285)
	- locked <0x00000000c1100838> (a java.lang.Object)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry.notifyAllSubscribedURLs(DubboCloudRegistry.java:370)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry.subscribeDubboMetadataServiceURLs(DubboCloudRegistry.java:470)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry.subscribeDubboMetadataServiceURLs(DubboCloudRegistry.java:436)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry.doSubscribe(DubboCloudRegistry.java:174)
	at org.apache.dubbo.registry.support.FailbackRegistry.subscribe(FailbackRegistry.java:333)
	at org.apache.dubbo.registry.ListenerRegistryWrapper.subscribe(ListenerRegistryWrapper.java:105)
	at org.apache.dubbo.registry.integration.RegistryDirectory.subscribe(RegistryDirectory.java:185)
	at org.apache.dubbo.registry.integration.RegistryProtocol.doRefer(RegistryProtocol.java:469)
	at org.apache.dubbo.registry.integration.RegistryProtocol.refer(RegistryProtocol.java:454)
	at org.apache.dubbo.rpc.protocol.ProtocolListenerWrapper.refer(ProtocolListenerWrapper.java:72)
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper.refer(ProtocolFilterWrapper.java:161)
	at org.apache.dubbo.qos.protocol.QosProtocolWrapper.refer(QosProtocolWrapper.java:73)
	at org.apache.dubbo.rpc.Protocol$Adaptive.refer(Protocol$Adaptive.java)
	at org.apache.dubbo.config.ReferenceConfig.createProxy(ReferenceConfig.java:372)
	at org.apache.dubbo.config.ReferenceConfig.init(ReferenceConfig.java:305)
	- locked <0x00000000fef65188> (a org.apache.dubbo.config.spring.ReferenceBean)
	at org.apache.dubbo.config.ReferenceConfig.get(ReferenceConfig.java:205)
	- locked <0x00000000fef65188> (a org.apache.dubbo.config.spring.ReferenceBean)
	at com.alibaba.cloud.dubbo.service.DubboGenericServiceFactory.create(DubboGenericServiceFactory.java:81)
	at com.alibaba.cloud.dubbo.service.DubboMetadataServiceInvocationHandler.<init>(DubboMetadataServiceInvocationHandler.java:40)
	at com.alibaba.cloud.dubbo.service.DubboMetadataServiceProxy.createProxy(DubboMetadataServiceProxy.java:192)
	at com.alibaba.cloud.dubbo.service.DubboMetadataServiceProxy.lambda$createProxyIfAbsent$0(DubboMetadataServiceProxy.java:141)
	at com.alibaba.cloud.dubbo.service.DubboMetadataServiceProxy$$Lambda$954/913139131.apply(Unknown Source)
	at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
	- locked <0x00000000fef66d20> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
	at com.alibaba.cloud.dubbo.service.DubboMetadataServiceProxy.createProxyIfAbsent(DubboMetadataServiceProxy.java:140)
	at com.alibaba.cloud.dubbo.service.DubboMetadataServiceProxy.createProxyIfAbsent(DubboMetadataServiceProxy.java:130)
	at com.alibaba.cloud.dubbo.service.DubboMetadataServiceProxy.getProxy(DubboMetadataServiceProxy.java:101)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry.getProxy(DubboCloudRegistry.java:340)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry.getTemplateExportedURLs(DubboCloudRegistry.java:319)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry.lambda$cloneExportedURLs$6(DubboCloudRegistry.java:288)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry$$Lambda$952/1017604146.accept(Unknown Source)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry.cloneExportedURLs(DubboCloudRegistry.java:284)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry.getExportedURLs(DubboCloudRegistry.java:265)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry.subscribeURLs(DubboCloudRegistry.java:240)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry.subscribeURLs(DubboCloudRegistry.java:227)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry.lambda$subscribeURLs$2(DubboCloudRegistry.java:206)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry$$Lambda$676/451312813.accept(Unknown Source)
	at java.lang.Iterable.forEach(Iterable.java:75)
	at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry.subscribeURLs(DubboCloudRegistry.java:204)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry.lambda$subscribeURLs$0(DubboCloudRegistry.java:194)
	at com.alibaba.cloud.dubbo.registry.DubboCloudRegistry$$Lambda$679/1178290888.onApplicationEvent(Unknown Source)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:404)
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:361)
	at com.alibaba.cloud.dubbo.autoconfigure.DubboServiceDiscoveryAutoConfiguration.dispatchServiceInstancesChangedEvent(DubboServiceDiscoveryAutoConfiguration.java:175)
	at com.alibaba.cloud.dubbo.autoconfigure.DubboServiceDiscoveryAutoConfiguration.access$200(DubboServiceDiscoveryAutoConfiguration.java:108)
	at com.alibaba.cloud.dubbo.autoconfigure.DubboServiceDiscoveryAutoConfiguration$NacosConfiguration.lambda$subscribeEventListener$1(DubboServiceDiscoveryAutoConfiguration.java:557)
	at com.alibaba.cloud.dubbo.autoconfigure.DubboServiceDiscoveryAutoConfiguration$NacosConfiguration$$Lambda$866/1442236552.onEvent(Unknown Source)
	at com.alibaba.nacos.client.naming.core.EventDispatcher$Notifier.run(EventDispatcher.java:177)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[wywanchongshan]

The complete stack information is
all_stack.txt

[wywanchongshan]

My development machine also reproduced

[wywanchongshan]

I know the cause of this bug;

We know the call situation through the top thread stack

"com.alibaba.nacos.naming.client.listener" #459 daemon prio=5 os_prio=0 tid=0x00007f236d0d7000 nid=0x129b runnable [0x00007f23557fc000]
java.lang.Thread.State: RUNNABLE

When executing this code

at com.alibaba.cloud.dubbo.service.DubboMetadataServiceProxy.createProxyIfAbsent(DubboMetadataServiceProxy.java:140)

jdk 8 ConcurrentHashMap Will initialize invoke initTable(); This will cause an null key to be placed inside and map an null finger
And when the computeIfAbsent method of ConcurrentHashMap is executed, it will be called DubboMetadataServiceProxy class 141 line code; And this line of code will eventually execute

at java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1172)
	at java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
	at com.alibaba.cloud.dubbo.service.DubboMetadataServiceProxy.removeProxy(DubboMetadataServiceProxy.java:76)

That’s why data like hashMap happened when I was debugging
#7403 (comment)

[AlbumenJ]

it seems as if is the issue of spring cloud alibaba?

[wywanchongshan]

it seems as if is the issue of spring cloud alibaba?

Then how do I transfer this question to the spring cloud alibaba project?

[CrazyHZM]

I think the transfer has already been done, so close it first.