Skip to content

analyze.md

Latest commit

 

History

History
306 lines (217 loc) · 20 KB

analyze.md

File metadata and controls

306 lines (217 loc) · 20 KB

fossa analyze

By default, the analyze command:

  • looks for projects in the current directory (and recursively in subdirectories)
  • extracts dependency graphs from those projects
  • infers a project name, branch, and revision for the project (from git or svn)
  • uploads the dependency graphs to FOSSA

For supported command-line flags, use fossa analyze --help

Specifying FOSSA project details

In addition to the usual FOSSA project flags supported by all commands, the analyze command supports the following FOSSA-project-related flags:

Name Short Description
--title 'some title' -t Set the title of the FOSSA project
--branch 'some branch' -b Override the detected FOSSA project branch
--project-url 'https://example.com' -P Add a URL to the FOSSA project
--jira-project-key 'some-key' -j Add a Jira project key to the FOSSA project
--link 'https://example.com' -L Attach a link to the current FOSSA build
--team 'some team' -T Specify a team within your FOSSA organization
--policy 'some policy' Assign a specific FOSSA policy to this project. Mutually excludes --policy-id.
--policy-id 'some policy id' Assign a specific FOSSA policy to this project by id. Mutually excludes --policy.
--project-label assign up to 5 labels to the project
--release-group-name 'MY_RG' add the project to this release group (also requires --release-group-release)
--release-group-release 'MY_RELEASE' add the project to this release version within the release group

Filtering Paths and Targets

The paths and targets filtering options allow you to specify the exact targets which be should be scanned.

Name Description
--only-target Only scan these targets. See targets.only in the fossa.yml spec.
--exclude-target Exclude these targets from scanning. See targets.exclude in the fossa.yml spec.
--only-path Only scan these paths. See paths.only in the fossa.yml spec.
--exclude-path Exclude these paths from scannig. See paths.exclude in the fossa.yml spec.
--include-unused-deps Include all deps found, instead of filtering non-production deps. Ignored by VSI.
--debug-no-discovery-exclusion Ignore these filters during discovery phase. This flag is for debugging only and may be removed without warning.
--without-default-filters Ignore default path filters. See default path filters

Printing FOSSA results

The --output flag can be used to print projects and dependency graph information to stdout, rather than uploading to FOSSA

fossa analyze --output

To print projects and dependency graph information to stdout in addition to uploading to FOSSA as normal, use the --tee-output flag.

fossa analyze --tee-output

Printing project metadata

The --json flag can be used to print project metadata after running fossa analyze successfully. This metadata can be used to reference your project when integrating with the FOSSA API.

fossa analyze --json
{"branch":"master", "id":"custom+<org-id>/new-project$123", "project":"<org-id>/new-project", "projectId":"custom+<org-id>/new-project", "revision":"123", "url":"https://app.fossa.com/projects/custom%2b+<org-id>$2fnew-project/refs/branch/master/123"}

Running a specific fossa-deps file

The --fossa-deps-file flag can be used to specify the fossa-deps file that you want to use. The name of the file is arbitrary.

See the fossa-deps documentation for configuration.

fossa analyze --fossa-deps-file /path/to/file

Vendored Dependencies

The Vendored Dependencies feature allows you to scan for licenses directly in your code. For more information, please see the Vendored Dependencies documentation.

Name Description
--force-vendored-dependency-scan-method Force the vendored dependency scan method. The options are 'CLILicenseScan' or 'ArchiveUpload'. 'CLILicenseScan' is usually the default unless your organization has overridden this.
--force-vendored-dependency-rescans Force vendored dependencies to be rescanned even if the revision has been previously analyzed by FOSSA. This currently only works for CLI-side license scans.

Custom License Searches

The "Custom License Searches" feature allows you to define text strings or regular expressions that are used to search through your codebase. If a match to the text string or regular expression is found, then a custom license is reported for that match.

An administrator of your organization can set up custom license searches that are always run when you analyze your codebase with fossa analyze. Using the --ignore-org-wide-custom-license-scan-configs flag will ignore those organization-wide custom license searches.

See the Custom License and Keyword Searches documentation for more details.

Running in a specific directory

fossa analyze path/to/directory

Scanning archive contents

With the --unpack-archives flag present, we unpack discovered archives to a temporary directory for dependency analysis. We recursively unpack archives-in-archives.

fossa analyze --unpack-archives

We support the following archive formats:

  • .zip
  • .tar
  • .tar.gz
  • .taz
  • .tgz
  • .tar.xz
  • .txz
  • .tar.bz2
  • .tbz
  • .tbz2
  • .tz2
  • .jar
  • .aar
  • .rpm, with...
    • gzip compression
    • lzma compression
    • xz compression
    • zstd compression

Enabling or disabling additional strategies

In addition to the standard flags, the analyze command supports the following additional strategy flags:

Name Description
--detect-vendored Enable the vendored source identification engine. For more information, see the C and C++ overview.
--detect-dynamic './some-binary Analyze the binary at the provided path for dynamically linked dependencies. For more information, see the C and C++ overview.
--static-only-analysis Do not use third-party tools when analyzing projects.
--strict Enforces strict analysis to ensure the most accurate results by rejecting fallbacks. When run with --static-only-analysis, the most optimal static strategy will be applied without fallbacks.

Snippet Scanning

Snippet scanning identifies potential open source code snippets within your first-party source code by comparing file fingerprints against FOSSA's knowledge base. This feature helps detect code that may have been copied from open source projects.

Enabling Snippet Scanning

Name Description
--snippet-scan Enable snippet scanning during analysis. This feature fingerprints your source files and checks them against FOSSA's snippet database.

Snippet Scanning must also be enabled for your organization, and is only available for enterprise customers. If you would like to enable it for your organization, please contact us.

More detail

For more detail about how Snippet Scanning works, how to use file filtering during Snippet Scanning, what information is sent to FOSSA's servers and a description of the Snippet Scan Summary, see the Snippet Scanning feature documentation.

Vendored Dependency Scanning with Vendetta

Vendetta is a feature that identifies the paths of potential open source code dependencies vendored in your project by comparing file hashes against FOSSA's knowledge base. This feature helps find dependencies that are included in your project directly as source.

Enabling Vendetta

Name Description
--x-vendetta Enable vendored dependency scanning during analysis. This experimental feature hashes your source files and checks them against FOSSA's open source component database.

More detail

For more detail about how Vendetta works, how to use file filtering during scanning, or what information is sent to FOSSA's servers, see the Vendetta feature documentation.

Experimental Options

Important: For support and other general information, refer to the experimental options overview before using experimental options.

In addition to the standard flags, the analyze command supports the following experimental flags:

Name Description
--experimental-enable-binary-discovery Enable reporting binary files as unlicensed dependencies. For more information, see the binary discovery overview.
--experimental-force-first-party-scans Force first party scans to run
--experimental-block-first-party-scans Force first party scans to not run. This can be used to forcibly turn off first-party scans if your organization defaults to first-party scans.
--experimental-analyze-path-dependencies License scan path dependencies, and include them in the final analysis. For more information, see the path dependency overview.

F.A.Q.

How do I add a project to a release group when it is analyzed?

To add the project you're analyzing to a release group, use fossa analyze --release-group-name 'MY_RG' --release-group-release 'MY_RELEASE_VERSION'

Note that the MY_RG release group must already exist, as well as MY_RELEASE_VERSION within it. You may use the release-group subcommand in advance to create these or do so within the FOSSA UI.

Why is the fossa-cli skipping my project?

fossa-cli may sometimes report a project of interest was skipped from the analysis. For example,

[ INFO] Scan Summary
[ INFO] ------------
[ INFO] 3 projects scanned;  2 skipped,  1 succeeded,  0 failed,  1 analysis warning
[ INFO]
[ INFO] * setuptools project in "sandbox/": succeeded with 1 warning
[ INFO] * setuptools project in "sandbox/example/": skipped (default path filters)
[ INFO] * setuptools project in "sandbox/external/": skipped (exclusion filters)

fossa-cli skips analysis, if and only if

  • (a) Target is excluded via fossa configuration file (this filtering is referred to as "exclusion filters").
  • (b) Target is excluded via default path filters (this filtering was previously referred to as "production path filtering").

What are the default filters?

Default filters are filters which fossa-cli applies by default. These filters, provide sensible non-production target exclusion. As fossa-cli relies on manifest and lock files provided in the project's directory, default filters, intentionally skip node_modules/ and such directories. If fossa-cli discovers and analyzes project found in node_modules/: fossa-cli will not be able to infer the dependency's scope (development or production) and may double count dependencies.

Specifically, fossa-cli by default skips any targets found within the following directories:

  • dist-newstyle
  • doc/
  • docs/
  • test/
  • tests/
  • example/
  • examples/
  • vendor/
  • node_modules/
  • .srclib-cache/
  • spec/
  • Godeps/
  • .git/
  • bower_components/
  • third_party/
  • third-party/
  • Carthage/
  • Checkouts/

To disable default filters, provide --without-default-filters flag when performing fossa analyze command. Currently, it is not possible to disable only a subset of default filters. If you would like to only apply a subset of default filters, you can use --without-default-filters in conjunction with exclusion filters. Refer to exclusion filters walkthough for example on how to apply path and target exclusion filters.

Can fossa-cli detect licensed/copyright content downloaded at runtime by dependencies?

Unfortunately, as of yet, fossa-cli cannot discover or analyze any licensed or copyrighted content retrieved at runtime by dependencies when it is not referenced in manifest or lock files.

For example, in python with datasets package,

from datasets import load_dataset
dataset = load_dataset("bigscience/P3", revision="f2cade2") # retrieved at runtime

In this case, fossa-cli would not be able to identify bigscience/P3 and its associated compliance obligation since they were not part of the manifest or lockfile - e.g. requirements.txt, setup.py, poetry.lock.

In this scenario, if the inclusion of bigscience/P3 is desired in FOSSA's reporting, we recommend fossa-deps.yml file to explicitly include bigscience/P3 in FOSSA's reporting. Likewise, you can also download huggingface.co/datasets/bigscience/P3 to disk and use vendor dependency scanning.

# Example fossa-deps.yml

custom-dependencies:
- name: huggingface.co/datasets/bigscience/P3
  version: "f2cade2"
  license: "Apache-2.0"

If you need more assistance, please contact FOSSA support.

How do I ensure fossa analyze does not exit fatally when no targets are discovered?

In some scenarios, you may want to configure the fossa analyze and fossa test CI workflow on an empty repository or directory with 0 targets. Unfortunately, fossa-cli does not have a configuration yet, which will allow for successful analysis (exit code of 0) when 0 targets are discovered.

For a workaround, create an empty reqs.txt file before running fossa analyze, which will force fossa-cli to presume an empty pip project (with 0 dependencies).

touch reqs.txt && fossa analyze && rm reqs.txt && fossa test

Common FOSSA Project Flags

All fossa commands support the following FOSSA-project-related flags:

Name Short Description
--project 'some project' -p Override the detected project name
--revision 'some revision' -r -Override the detected project revision
--fossa-api-key 'my-api-key' An alternative to using the FOSSA_API_KEY environment variable to specify a FOSSA API key
--endpoint 'https://example.com' -e Override the FOSSA API server base URL
--config /path/to/file -c Path to a configuration file including filename. By default we look for .fossa.yml in base working directory.
⚡