VIP: Add support for relative imports.

[jacqueswww]

Simple Summary

Allow using . and .. to specify relative imports of interfaces.

Abstract

Following a long discussion on #1361. It came to light that frameworks and programmers need more felxibility in terms of importing interfaces.

Motivation

The motivation is critical for VIPs that add or change Vyper's functionality. It should clearly explain why the existing Vyper functionality is inadequate to address the problem that the VIP solves as well as how the VIP is in line with Vyper's goals and design philosophy.

Specification

.. will specify parent directory of the current .vy being parsed.
. will specify current directory of the .vy being parsed.

e.g.

from .token_interface import TokenInterface

Will create an interface names TokenInterface from a file currently located at the same directory as the .vy file that is currently being parsed.

Environment variable:
Vyper shold be allowed to change it's lookup directory from CWD to one set in VYPER_PATHS.

Backwards Compatibility

Is forward compatible.

Dependencies

None.

Copyright

Copyright and related rights waived via CC0

[fubuloubu]

@mikeshultz We decided to take a different tactic to do what you were doing in #1361. We hope you might comment on it here :)

[mikeshultz]

For what it's worth, the toolkit I'm working on now invokes the vyper module directly and does import resolution itself instead of relying on invoking the CLI script.

That said, I still think this is a good idea. And if this resolution is not implemented in the compiler(but the CLI script with the package), this is the kind of thing that should be documented for other tools utilizing Vyper.

Generally, maybe this is the kind of thing that would be good to get some documented rules/best practices for how to resolve absolute paths as well. If you all think the current working directory is the best place to resolve from, that should be documented. Maybe absolute paths and the possibility of having a library PATH is a different discussion, however.

[fubuloubu]

Maybe the compiler API can be written such that if the compiler encounters any import statements, it references a "lookup" mapping of those packages to the specific source files they reference, which must be provided by the CLI script (and any other tools). This keeps the compiler functionally pure, and mitigates introducing the concept of the filesystem into the compiler.

[mikeshultz]

Isn't that what it does now? Well, except the file contents are provided instead of a path. The downside of that would be that whatever invokes the compiler will still need to know the location of the contract/package names to provide the compiler. In that case, may as well leave it the same.

Maybe a list in which each provided path is checked in order? That would be more inline with traditional PATH-like imports.

An example might be:

paths = [
    '~/my-project/contracts',
    '~/.local/lib/vyper/site-packages',
    '/usr/lib/vyper/site-packages',
]
vyper.compile_code(
    source_text,
    ['bytecode', 'abi'],
    import_paths=paths,
)

I guess for both relative and absolute imports it should be figured out if you want to keep the compiler filesystem agnostic, and leave that up CLI scripts. If so, some standards would still be good to keep everything inline so the user isn't guessing how it might work.

[fubuloubu]

It might need some sort of "project root" object for it to check against.

File directory:

root/
    contractA.vy
    more/
        contractB.vy

contractA.vy:

from more.contractB import InterfaceB
# ... stuff

contractB.vy:

from ..contractA import InterfaceA
# ... stuff

compiler:

paths = {
    'contractA': contractA_source,
    'more': {
        'contractB': contractB_source,
    }
}
vyper.compile_code(
    source_text,
    ['bytecode', 'abi'],
    import_paths=paths,
)

The thought is that the compiler would have paths to look at when determining what to map to when it encounters an import statement.

Probably a lot wrong with this approach though.

[pipermerriam]

I'd like to toss something out that may or may not be relevant which is about how we deal with namespacing of imports. This design will have long lasting influence on the language.

Python has dependency issues. It is really difficult to juggle dependencies since version conflicts are so easy to run into and the runtime uses a single namespace for dependency imports. Another way to say this is that you can only ever have one version of something installed and import web3 will import the same web3 module across all python files in a given runtime environment.

My understanding is that Node took a different approach, giving every package it's own namespace, thus allowing each package to install their own versions of each dependency and not sharing these dependencies across different packages. This fundamentally solves an entire class of dependency issues at the cost of a heavier runtime environment.

My inclination is to say that the Node approach is better in most/many of the ways that I think matter to Vyper. I'm curious to hear people's thoughts on this.

[mikeshultz]

This fundamentally solves an entire class of dependency issues at the cost of a heavier runtime environment.

For what it's worth, that cost can be pretty high. The worst part about dealing with node projects is dealing with dependencies. That's why there's a handful of competing package managers, and a ton of tools built up around just dealing with dependencies, and why the first solution to things "not working" is to rm -rf node_modules && npm i. Worked on a JS project recently that took about 30 minutes to install all the dependencies for their monorepo and it was a constant source of jokes and frustration.

That's not to say that Python is so much better when you have a project with mutual dependencies with your dependencies and projects that change their API like they change their underwear...

Figure I'd also mention that I've seen some projects that have taken to including version names in their contracts. For example, V01_MyContract. That allows them to not only keep a reference around for deployed contracts, but allow multiple versions to run side by side, and make it absolutely clear to the devs that versions of interfaces are different.

Not sure if that's the greatest solution, but thought I'd mention it since I've seen it in the wild.

[fubuloubu]

My inclination is to say that the Node approach is better in most/many of the ways that I think matter to Vyper.

Not to say the Node approach is at all better than the Python approach, but the fact of the matter is that this approach makes a lot of sense for smart contracts. It's also how ethPM works. Dependencies for smart contract should be orders of magnitude less than Javascript or Python (at least they are now), so I think this approach is reasonable and scaleable. We at least won't be constructing a "ethpm packages" directory denser than a neutron star.

---

That being said, I still think there is a requirement for a "local" context. Perhaps . can serve as that. For a flat contracts/ directory, that would be import .Contract for a file at Contract.{vy|json}. For things with multiple levels, this sort of limits it to only one level of depth in the relative import statement (e.g. import ..Contract) which I also believe is reasonable and scaleable.

The most complex smart contract systems I have seen have been on the order of 50 files, which fits fine in a 1-level deep folder structure. Vyper contract systems will be less file-heavy, due to the structure that Vyper contracts must fit within a single file (except for interfaces). I would argue anything that's larger than this level of complexity deserves to be composed into separate projects and handled through multiple deployed contract systems.