Add dependencies check script by htynkn · Pull Request #3941 · apache/dubbo

htynkn · 2019-04-27T13:43:46Z

What is the purpose of the change

Run script to get dependencies license list

For example:
licenseCheck.sh - will check whole project
licenseCheck.sh module-name - will only check specific module

Result will save in license-list.txt file

Brief changelog

Add script
Add license plugin config
Add missing file

Verifying this change

test script

Follow this checklist to help us incorporate your contribution quickly and easily:

Make sure there is a GITHUB_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
Run mvn clean install -DskipTests=false & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
If this contribution is large, please follow the Software Donation Guide.

…ependencies-check

htynkn · 2019-04-27T13:44:11Z

Close #3840
Close #3790

htynkn · 2019-04-27T13:45:00Z

Some example for how license-list.txt file looks like

==============================================
./dubbo-cluster/target/generated-sources/license/THIRD-PARTY.txt

Lists of 41 third-party dependencies.
     (Apache License, Version 2.0) Code Generation Library (cglib:cglib-nodep:2.2 - http://cglib.sourceforge.net/)
     (Apache License, Version 2.0) fastjson (com.alibaba:fastjson:1.2.46 - https://github.com/alibaba/fastjson)
     (Apache License, Version 2.0) Hessian Lite(Alibaba embed version) (com.alibaba:hessian-lite:3.2.5 - https://github.com/dubbo/hessian-lite)
     (Apache License, Version 2.0) java-util (com.cedarsoftware:java-util:1.9.0 - https://github.com/jdereg/java-util)
     (Apache License, Version 2.0) json-io (com.cedarsoftware:json-io:2.5.1 - https://github.com/jdereg/json-io)
     (3-Clause BSD License) Kryo (com.esotericsoftware:kryo:4.0.1 - https://github.com/EsotericSoftware/kryo/kryo)
     (The 3-Clause BSD License) MinLog (com.esotericsoftware:minlog:1.3.0 - https://github.com/EsotericSoftware/minlog)
     (The 3-Clause BSD License) ReflectASM (com.esotericsoftware:reflectasm:1.11.3 - https://github.com/EsotericSoftware/reflectasm)
     (Apache License, Version 2.0) Jackson-core (com.fasterxml.jackson.core:jackson-core:2.8.6 - https://github.com/FasterXML/jackson-core)
     (Apache License, Version 2.0) Guava: Google Core Libraries for Java (com.google.guava:guava:20.0 - https://github.com/google/guava/guava)
     (Apache License, Version 2.0) Apache Commons Logging (commons-logging:commons-logging:1.2 - http://commons.apache.org/proper/commons-logging/)
     (Apache License, Version 2.0) kryo serializers (de.javakaffee:kryo-serializers:0.42 - https://github.com/magro/kryo-serializers)
     (Apache License, Version 2.0) fst (de.ruedigermoeller:fst:2.48-jdk-6 - http://ruedigermoeller.github.io/fast-serialization/)
     (The 3-Clause BSD License) JLine (jline:jline:0.9.94 - http://jline.sourceforge.net)
     (Apache License, Version 2.0) Apache Log4j (log4j:log4j:1.2.16 - http://logging.apache.org/log4j/1.2/)
     (Apache License, Version 2.0) Byte Buddy (without dependencies) (net.bytebuddy:byte-buddy:1.9.3 - http://bytebuddy.net/byte-buddy)
     (Apache License, Version 2.0) Byte Buddy Java agent (net.bytebuddy:byte-buddy-agent:1.9.3 - http://bytebuddy.net/byte-buddy-agent)
     (Apache License, Version 2.0) Curator Client (org.apache.curator:curator-client:4.0.1 - http://curator.apache.org/curator-client)
     (Apache License, Version 2.0) Curator Framework (org.apache.curator:curator-framework:4.0.1 - http://curator.apache.org/curator-framework)
     (Apache License, Version 2.0) dubbo-common (org.apache.dubbo:dubbo-common:2.7.2-SNAPSHOT - https://github.com/apache/incubator-dubbo/dubbo-common)
     (Apache License, Version 2.0) dubbo-configcenter-api (org.apache.dubbo:dubbo-configcenter-api:2.7.2-SNAPSHOT - https://github.com/apache/incubator-dubbo/dubbo-configcenter/dubbo-configcenter-api)
     (Apache License, Version 2.0) dubbo-remoting-api (org.apache.dubbo:dubbo-remoting-api:2.7.2-SNAPSHOT - https://github.com/apache/incubator-dubbo/dubbo-remoting/dubbo-remoting-api)
     (Apache License, Version 2.0) dubbo-rpc-api (org.apache.dubbo:dubbo-rpc-api:2.7.2-SNAPSHOT - https://github.com/apache/incubator-dubbo/dubbo-rpc/dubbo-rpc-api)
     (Apache License, Version 2.0) dubbo-serialization-api (org.apache.dubbo:dubbo-serialization-api:2.7.2-SNAPSHOT - https://github.com/apache/incubator-dubbo/dubbo-serialization/dubbo-serialization-api)
     (Apache License, Version 2.0) Apache Yetus - Audience Annotations (org.apache.yetus:audience-annotations:0.5.0 - https://yetus.apache.org/audience-annotations)
     (Apache License, Version 2.0) zookeeper (org.apache.zookeeper:zookeeper:3.4.13 - no url defined)
     (Apache License, Version 2.0) org.apiguardian:apiguardian-api (org.apiguardian:apiguardian-api:1.0.0 - https://github.com/apiguardian-team/apiguardian)
     (The 3-Clause BSD License) Hamcrest All (org.hamcrest:hamcrest-all:1.3 - https://github.com/hamcrest/JavaHamcrest/hamcrest-all)
     (Apache License, Version 2.0) (LGPL 2.1) (MPL 1.1) Javassist (org.javassist:javassist:3.20.0-GA - http://www.javassist.org/)
     (Eclipse Public License v2.0) JUnit Jupiter API (org.junit.jupiter:junit-jupiter-api:5.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Jupiter Engine (org.junit.jupiter:junit-jupiter-engine:5.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Jupiter Params (org.junit.jupiter:junit-jupiter-params:5.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Platform Commons (org.junit.platform:junit-platform-commons:1.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Platform Engine API (org.junit.platform:junit-platform-engine:1.4.0 - https://junit.org/junit5/)
     (The MIT License) mockito-core (org.mockito:mockito-core:2.23.4 - https://github.com/mockito/mockito)
     (Apache License, Version 2.0) Objenesis (org.objenesis:objenesis:2.6 - http://objenesis.org)
     (Apache License, Version 2.0) org.opentest4j:opentest4j (org.opentest4j:opentest4j:1.1.1 - https://github.com/ota4j-team/opentest4j)
     (The 3-Clause BSD License) ASM Core (org.ow2.asm:asm:5.0.4 - http://asm.objectweb.org/asm/)
     (The MIT License) SLF4J API Module (org.slf4j:slf4j-api:1.7.25 - http://www.slf4j.org)
     (The MIT License) SLF4J LOG4J-12 Binding (org.slf4j:slf4j-log4j12:1.7.25 - http://www.slf4j.org)
     (Apache License, Version 2.0) SnakeYAML (org.yaml:snakeyaml:1.20 - http://www.snakeyaml.org)
==============================================
./dubbo-container/dubbo-container-spring/target/generated-sources/license/THIRD-PARTY.txt

Lists of 34 third-party dependencies.
     (Apache License, Version 2.0) Code Generation Library (cglib:cglib-nodep:2.2 - http://cglib.sourceforge.net/)
     (Apache License, Version 2.0) fastjson (com.alibaba:fastjson:1.2.46 - https://github.com/alibaba/fastjson)
     (Apache License, Version 2.0) Hessian Lite(Alibaba embed version) (com.alibaba:hessian-lite:3.2.5 - https://github.com/dubbo/hessian-lite)
     (Apache License, Version 2.0) java-util (com.cedarsoftware:java-util:1.9.0 - https://github.com/jdereg/java-util)
     (Apache License, Version 2.0) json-io (com.cedarsoftware:json-io:2.5.1 - https://github.com/jdereg/json-io)
     (3-Clause BSD License) Kryo (com.esotericsoftware:kryo:4.0.1 - https://github.com/EsotericSoftware/kryo/kryo)
     (The 3-Clause BSD License) MinLog (com.esotericsoftware:minlog:1.3.0 - https://github.com/EsotericSoftware/minlog)
     (The 3-Clause BSD License) ReflectASM (com.esotericsoftware:reflectasm:1.11.3 - https://github.com/EsotericSoftware/reflectasm)
     (Apache License, Version 2.0) Jackson-core (com.fasterxml.jackson.core:jackson-core:2.8.6 - https://github.com/FasterXML/jackson-core)
     (Apache License, Version 2.0) Apache Commons Logging (commons-logging:commons-logging:1.2 - http://commons.apache.org/proper/commons-logging/)
     (Apache License, Version 2.0) kryo serializers (de.javakaffee:kryo-serializers:0.42 - https://github.com/magro/kryo-serializers)
     (Apache License, Version 2.0) fst (de.ruedigermoeller:fst:2.48-jdk-6 - http://ruedigermoeller.github.io/fast-serialization/)
     (Apache License, Version 2.0) Apache Log4j (log4j:log4j:1.2.16 - http://logging.apache.org/log4j/1.2/)
     (Apache License, Version 2.0) Byte Buddy (without dependencies) (net.bytebuddy:byte-buddy:1.9.3 - http://bytebuddy.net/byte-buddy)
     (Apache License, Version 2.0) Byte Buddy Java agent (net.bytebuddy:byte-buddy-agent:1.9.3 - http://bytebuddy.net/byte-buddy-agent)
     (Apache License, Version 2.0) dubbo-common (org.apache.dubbo:dubbo-common:2.7.2-SNAPSHOT - https://github.com/apache/incubator-dubbo/dubbo-common)
     (Apache License, Version 2.0) dubbo-container-api (org.apache.dubbo:dubbo-container-api:2.7.2-SNAPSHOT - https://github.com/apache/incubator-dubbo/dubbo-container/dubbo-container-api)
     (Apache License, Version 2.0) org.apiguardian:apiguardian-api (org.apiguardian:apiguardian-api:1.0.0 - https://github.com/apiguardian-team/apiguardian)
     (The 3-Clause BSD License) Hamcrest All (org.hamcrest:hamcrest-all:1.3 - https://github.com/hamcrest/JavaHamcrest/hamcrest-all)
     (Apache License, Version 2.0) (LGPL 2.1) (MPL 1.1) Javassist (org.javassist:javassist:3.20.0-GA - http://www.javassist.org/)
     (Eclipse Public License v2.0) JUnit Jupiter API (org.junit.jupiter:junit-jupiter-api:5.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Jupiter Engine (org.junit.jupiter:junit-jupiter-engine:5.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Jupiter Params (org.junit.jupiter:junit-jupiter-params:5.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Platform Commons (org.junit.platform:junit-platform-commons:1.4.0 - https://junit.org/junit5/)
     (Eclipse Public License v2.0) JUnit Platform Engine API (org.junit.platform:junit-platform-engine:1.4.0 - https://junit.org/junit5/)
     (The MIT License) mockito-core (org.mockito:mockito-core:2.23.4 - https://github.com/mockito/mockito)

codecov-io · 2019-04-30T07:53:13Z

Codecov Report

Merging #3941 into master will decrease coverage by 0.07%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #3941      +/-   ##
============================================
- Coverage     63.94%   63.87%   -0.08%     
  Complexity       98       98              
============================================
  Files           715      715              
  Lines         31493    31493              
  Branches       5074     5074              
============================================
- Hits          20139    20115      -24     
- Misses         9052     9074      +22     
- Partials       2302     2304       +2

Impacted Files	Coverage Δ	Complexity Δ
.../apache/dubbo/qos/protocol/QosProtocolWrapper.java	`64.1% <0%> (-17.95%)`	`0% <0%> (ø)`
...ache/dubbo/remoting/p2p/support/AbstractGroup.java	`45.45% <0%> (-11.37%)`	`0% <0%> (ø)`
.../apache/dubbo/remoting/transport/AbstractPeer.java	`63.04% <0%> (-8.7%)`	`0% <0%> (ø)`
...ng/exchange/support/header/HeartbeatTimerTask.java	`73.68% <0%> (-5.27%)`	`0% <0%> (ø)`
...pache/dubbo/registry/support/AbstractRegistry.java	`78.16% <0%> (-4.22%)`	`0% <0%> (ø)`
.../exchange/support/header/HeaderExchangeServer.java	`66.98% <0%> (-1.89%)`	`0% <0%> (ø)`
...dubbo/rpc/protocol/dubbo/CallbackServiceCodec.java	`80.88% <0%> (+0.73%)`	`0% <0%> (ø)`	⬇️
...dubbo/remoting/exchange/support/DefaultFuture.java	`75.51% <0%> (+2.04%)`	`0% <0%> (ø)`	⬇️
...apache/dubbo/common/config/ConfigurationUtils.java	`72% <0%> (+8%)`	`0% <0%> (ø)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78c7509...1947248. Read the comment docs.

ralf0131 · 2019-04-30T08:10:51Z

I suggest to add a check to ensure there is no incompatible license, below is what I used:

cat license-list.txt| grep -E 'GPL|General Public License' | grep -v CDDL | grep -v Apache | grep -v "Eclipse Public License" | grep -v "MIT" | grep -v "Common Development and Distribution License"

If everything is ok, there should be no output.

htynkn · 2019-04-30T09:12:12Z

I suggest to add a check to ensure there is no incompatible license, below is what I used:
cat license-list.txt| grep -E 'GPL|General Public License' | grep -v CDDL | grep -v Apache | grep -v "Eclipse Public License" | grep -v "MIT" | grep -v "Common Development and Distribution License"
If everything is ok, there should be no output.

Yeah, we can do this. but I'm wondering are we able to list all license which is not compatible with dubbo？ does GPL is the only one we need check?

ralf0131 · 2019-04-30T09:28:11Z

I suggest to add a check to ensure there is no incompatible license, below is what I used:
cat license-list.txt| grep -E 'GPL|General Public License' | grep -v CDDL | grep -v Apache | grep -v "Eclipse Public License" | grep -v "MIT" | grep -v "Common Development and Distribution License"
If everything is ok, there should be no output.
Yeah, we can do this. but I'm wondering are we able to list all license which is not compatible with dubbo？ does GPL is the only one we need check?

No. All the incompatible license are listed here: https://www.apache.org/legal/resolved.html#category-x

htynkn · 2019-05-01T13:22:55Z

@ralf0131 I update script and put all stuff into one script, please review again.

Below are dependencies which I'm not sure if need fix:

(JSR-000107 JCACHE 2.9 Public Review - Updated Specification License) JSR107 API and SPI (javax.cache:cache-api:1.0.0 - https://github.com/jsr107/jsr107spec)
(GNU LESSER GENERAL PUBLIC LICENSE, Version 2.1) SpotBugs Annotations (com.github.spotbugs:spotbugs-annotations:3.1.10 - https://spotbugs.github.io/)
(CC0 1.0 Universal License) Native Library Loader (org.scijava:native-lib-loader:2.0.2 - http://github.com/scijava/native-lib-loader)

ralf0131 · 2019-05-14T04:36:05Z

Below are dependencies which I'm not sure if need fix:

(JSR-000107 JCACHE 2.9 Public Review - Updated Specification License) JSR107 API and SPI (javax.cache:cache-api:1.0.0 - https://github.com/jsr107/jsr107spec)

Apache 2.0 License: OK

(GNU LESSER GENERAL PUBLIC LICENSE, Version 2.1) SpotBugs Annotations (com.github.spotbugs:spotbugs-annotations:3.1.10 - https://spotbugs.github.io/)

LGPL: which is imcomplatible with Apache, I am investigating it.

(CC0 1.0 Universal License) Native Library Loader (org.scijava:native-lib-loader:2.0.2 - http://github.com/scijava/native-lib-loader)

It is actually a 2-Clause BSD License , so I think it is ok.

I found some more reported by the script:

(BEA licensed) "Java Concurrency in Practice" book annotations (net.jcip:jcip-annotations:1.0 - http://jcip.net/)

BEA license is said to be incompatible with Apache, need to investigate it.

(Common Public License Version 1.0) JUnit (junit:junit:3.8.1 - http://junit.org)

CPL 1.0 is OK, but junit 3.8.1 should not be included, need to exclude it if possible.

(The BSD License) Stax2 API (org.codehaus.woodstox:stax2-api:3.1.4 - http://wiki.fasterxml.com/WoodstoxStax2)

Standard BSD license is compatible with Apache.

(CPL) WSDL4J (wsdl4j:wsdl4j:1.6.3 - http://sf.net/projects/wsdl4j)

CPL is compatible with Apache.

My suggestion to improve the script is to add the following to allowLicense:

Common Public License Version 1.0
The 2-Clause BSD License

ralf0131

Let merge it and improve it later.

htynkn · 2019-05-22T03:10:06Z

Let merge it and improve it later.

Sorry, I'm busy recently. will try to do some improvement when I get time. and also happy to see if anyone else can help on it

htynkn added 5 commits April 21, 2019 14:56

add plugin to do dependencies check

85f00a7

remove third-party properties file

867709c

Merge remote-tracking branch 'remotes/origin/master' into chore/add-d…

55336b1

…ependencies-check

polish config

49c4dcd

polish config

50e5613

ralf0131 reviewed Apr 30, 2019

View reviewed changes

Comment thread dubbo-compatible/src/license/THIRD-PARTY.properties Outdated

use missing info in bash

1947248

ralf0131 added this to the 2.7.2 milestone May 13, 2019

ralf0131 self-assigned this May 13, 2019

ralf0131 approved these changes May 22, 2019

View reviewed changes

ralf0131 merged commit 3fc6909 into apache:master May 22, 2019

htynkn deleted the chore/add-dependencies-check branch May 22, 2019 03:10

dubbo-bot mentioned this pull request May 24, 2019

[WeeklyReport] Weekly report for Dubbo 2019-5-17 to 2019-5-24 #4149

Closed

This was referenced Jul 12, 2022

[Question] <license and dependencies license check script> apache/shenyu#3696

Closed

[Task] <license and dependencies license check script> apache/shenyu#3697

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add dependencies check script#3941

Add dependencies check script#3941
ralf0131 merged 6 commits intoapache:masterfrom
htynkn:chore/add-dependencies-check

htynkn commented Apr 27, 2019

Uh oh!

htynkn commented Apr 27, 2019

Uh oh!

htynkn commented Apr 27, 2019 •

edited

Loading

Uh oh!

Uh oh!

codecov-io commented Apr 30, 2019 •

edited

Loading

Uh oh!

ralf0131 commented Apr 30, 2019

Uh oh!

htynkn commented Apr 30, 2019

Uh oh!

ralf0131 commented Apr 30, 2019

Uh oh!

htynkn commented May 1, 2019

Uh oh!

ralf0131 commented May 14, 2019

Uh oh!

ralf0131 left a comment

Uh oh!

htynkn commented May 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

htynkn commented Apr 27, 2019

What is the purpose of the change

Brief changelog

Verifying this change

Uh oh!

htynkn commented Apr 27, 2019

Uh oh!

htynkn commented Apr 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

codecov-io commented Apr 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ralf0131 commented Apr 30, 2019

Uh oh!

htynkn commented Apr 30, 2019

Uh oh!

ralf0131 commented Apr 30, 2019

Uh oh!

htynkn commented May 1, 2019

Uh oh!

ralf0131 commented May 14, 2019

Uh oh!

ralf0131 left a comment

Choose a reason for hiding this comment

Uh oh!

htynkn commented May 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

htynkn commented Apr 27, 2019 •

edited

Loading

codecov-io commented Apr 30, 2019 •

edited

Loading