[baseline practices] .npmignore or package.json files

[jchip]

Do we have something for specifying the included files for publishing yet?

I just happen to look through some of the files in node_modules for one of my apps and this is what I found:

• coverage, .nyc_output
• .idea
• .eslintrc, .travis.yml, etc
• quite a few packages include their full test files.
• also for packages written in ts, some are including the TS source, is that needed?

Kind of related to #77

We should have some thing that suggest ensuring files exist in package.json and include essential files for publishing.

[ljharb]

First of all, "files" is super dangerous - forgetting to add a new file causes breakage. Nobody should ever use it.

.npmignore is fine, but also, imo npm explore foo && npm install && npm test should always work - i consider it a best practice to always ship full tests with every package.

[jchip]

I have no opinion about .npmignore or files except .npmignore should be in .npmignore itself. We should have some mention about being mindful to publish without extra files and npm offers two ways.

I approach this from practical stand point and prefer my node_modules smaller. I think publishing packages without extra files is one of the most overlooked thing and I've installed packages that publish demo images or temp files in the 10s of MB size. I think if there are a lot of tests it's better to leave them, but it's up to individual package owner. The practice I follow is prepublishOnly runs npm test.

[ljharb]

The cost of publishing an extra file is slower install times; the cost of omitting a needed file is breakage. There’s no semver contract around package sizes to break.

[jchip]

as a user, I prefer both: faster install and no breakage.

[ljharb]

In the better case, sure! But in the worse case, which would you prefer, a fast broken install, or a slow working one?

[jchip]

I'd like to clarify what you actually have issue with.

It's clear you are against files and that test should be published. I am sure we can word it to your satisfaction.

Do you actually have issue with the fact that we should have guideline about the broader topic I raised?

[ljharb]

In general, yes - there are very few files I think should ever be excluded from the published tarball, and I'd be concerned that we'd be unable to come up with guidance that wouldn't over-weight "package size" as a consideration.

In other words, I don't think that package authors should be solving this problem - I think npm should (and, with their "tink" project, they are).

[jchip]

OK, let me clarify. The primary motivation for bringing this up is because I don't think files like these should be published:

• .DS_Store, .nyc_output, coverage, .idea, .vscode

I don't think tink is solving that.

npm could default to ignore those files, but I imagine that's not something they want to do either.

I think having a guideline regarding be mindful of what files you published make sense.

[ljharb]

oh sure. But those should all be gitignored - which makes them ignored by npm by default.

[dominykas]

Depending on how you structure your project, you can have files be very safe. Basically, put stuff in lib/ and bin/ and you're grand. And docs/ as well.

Re tests - that's very debatable. I wish there was tooling to easily "switch to use this deep dep to be installed from their git" so that you can get tests, but shipping tests together with the package is an antipattern in my eyes, partly because it complicates security efforts as there is more code to review/analyse - and that is code that's never going to be used in production (yes, it's a tradeoff between dev needs and prod needs). Keep in mind, that the payload for event-stream was hidden inside tests... Either way, because it's debatable, I agree that we can't say that it's a "best practice".

Re other files - as a random one-off contributor I find it very uncomfortable when I need to add .idea to someone's .gitignore... and I know there's people who don't like it... so having that as a baseline would be very nice, but I'm not sure that's going to fly, because the list of common files is very large and it bloats the .gitignores. Instead, I'd really like it if npm actually respected a global .gitignore as well as the local one... There's an issue open for years.

[mcollina]

I think @styfle should join this discussion. Mainly because we sit on the opposite side of this fence: he wrote https://packagephobia.now.sh/ and I'm a strict believer that sources, docs, tests and examples are key part of a package. Specifically, I think I should be able to build/test that package without having to open up git. I'm a huge fan of "show source feature" of the 90s web.

Reducing the size of packages is important to reduce the size of the artifacts that we deploy. I think we should build a tool that traverse node_modules and remove not-needed files for production (tests, docs). This would be very nice to build as part of this group.

[styfle]

packages written in ts, some are including the TS source, is that needed?

No, it is not needed. You can publish a just the type declaration file (think header file if you use C++) which is perfect for anyone installing your package.

i consider it a best practice to always ship full tests with every package.

I disagree. I think npm publish should be the minimum build artifacts only. When I run npm install yourpackage, I can't even run your tests because npm only installs the production dependencies, not the devDependencies from yourpackage. And I can tell you from experience, that most people who publish tests to npm do so by accident, not on purpose. That's why I built Package Phobia.

I'm a huge fan of "show source feature" of the 90s web.

Met too! But that's not realistic because people who are publishing to npm are not necessarily writing JavaScript. They might be writing Rust and compiling to WASM and then deploying the WASM to npm. They don't need to deploy the Rust source code, nor do I want them too. Their tests are probably written in Rust too.

Probably a better approach to this "show the source feature" is to somehow enforce a build step when publishing to npm so that the source is guaranteed to match the build artifacts. Something like ZEIT Now Builders. But that's problem/topic for a different thread.

Depending on how you structure your project, you can have files be very safe.

I like files because I can just add files: ["dist"] and only the compiled output is going to be published. However, there is a known bug that is only affecting files but not .npmignore filtering. https://npm.community/t/ds-store-files-show-up-after-npm-publish/831

[ljharb]

npm’s tink project will allow you to publish gigabytes if you like, but consumers will only install what they need - so this “phobia” will go away soon anyways, without forcing everyone to hamstring offline usage.

Again, even if you want to restrict what shows up on npm, the “files” field is very dangerous, and i do not want this project to endorse its use in any way.

[jchip]

While a few things are debatable with opinions on either sides of the fence, I hope we can agree on the basic things that a guideline about having the proper setup to ensure some files are not published is a good practice.

The guideline should only refer to the methods npm provides to do this, and avoid having any preferences on files or .npmignore.

Maybe note some gotchas and tips:

• the existence of .npmignore makes npm ignore .gitignore, which is why some package is publishing .nyc_output and coverage, like this https://github.com/nodejitsu/node-http-proxy
• using .npmignore would publish any unintended files left over, for example temp files.
• files has a bug https://npm.community/t/ds-store-files-show-up-after-npm-publish/831
• files ensure no unintended files get publish accidentally, but it could be dangerous, but following a well defined directory structure makes that less of an issue
• publishing test is potentially a security attack vector, ie: incident in event-stream.

Other than that, I think we would leave the choice of whether to publish things like docs and test, or other files, to the individual package owner and their preferences.

[jchip]

also suggest to run a npm pack to inspect the tgz files content before doing a real publishing. I think there are tools out there that makes the publish process better. Maybe we can suggest one.