Skip to content

Manage packages#58

Open
muryanto1 wants to merge 4 commits intomasterfrom
manage_packages
Open

Manage packages#58
muryanto1 wants to merge 4 commits intomasterfrom
manage_packages

Conversation

@muryanto1
Copy link
Copy Markdown
Member

@muryanto1 muryanto1 commented Feb 26, 2019

Adding 'remove_anaconda_pkgs.py' script for removing anaconda packages that are older than certain time.

$ python ./remove_anaconda_pkgs.py -h
usage: remove_anaconda_pkgs.py [-h] [-c CHANNEL] [-p PACKAGES] [-R REGEX]
                               [-o OLDER_THAN] [-d]

remove your anaconda pkgs that are older than the specified constraint(s)

optional arguments:
  -h, --help            show this help message and exit
  -c CHANNEL, --channel CHANNEL
                        conda channel, ex: cdat/label/unstable (default: None)
  -p PACKAGES, --packages PACKAGES
                        conda packages, ex: 'thermo cdutil' (default: None)
  -R REGEX, --regex REGEX
                        regex to match with substring in the pkg version, ex:
                        '.*2016.*', '.*(2016|2017).*' (default: None)
  -o OLDER_THAN, --older_than OLDER_THAN
                        look for packages older than specified constraints:
                        days=<num>, hours=<num>, minutes=<num> or weeks=<num>
                        (default: None)
  -d, --dryrun          dry run do not actually do anything, just list out the
                        packages to be removed (default: False)


$ python ./remove_anaconda_pkgs.py -c cdat/label/linatest -o days=120 -d
CMD: conda search --override -c cdat/label/linatest
Loading channels: ...working... done
# Name                  Version           Build  Channel
cdutil          8.0.2018.07.30.15.26.ga076047          py36_0  cdat/label/linatest
cdutil          8.0.2018.08.03.08.35.g90c6425          py36_0  cdat/label/linatest
cdutil          8.0.2018.08.03.09.05.g2b08380          py36_0  cdat/label/linatest
cdutil          8.0.2018.08.03.09.36.g5f4fb99          py36_0  cdat/label/linatest
cdutil          8.0.2018.08.03.09.56.g8805ee1          py36_0  cdat/label/linatest
cdutil          8.0.2018.08.03.14.01.g298416a          py36_0  cdat/label/linatest
cdutil          8.0.2018.08.03.16.41.g3da5cf0          py36_0  cdat/label/linatest
cdutil          8.0.2018.08.03.17.03.gfdbaac2          py36_0  cdat/label/linatest
dv3d            8.0.2018.09.18.15.46.g0c84513          py27_0  cdat/label/linatest
dv3d            8.0.2018.09.18.15.53.g0c84513          py36_0  cdat/label/linatest
thermo          8.0.2019.02.15.22.33.g8fdc229            py_0  cdat/label/linatest

Going to remove: cdat/cdutil/8.0.2018.07.30.15.26.ga076047
Going to remove: cdat/cdutil/8.0.2018.08.03.08.35.g90c6425
Going to remove: cdat/cdutil/8.0.2018.08.03.09.05.g2b08380
Going to remove: cdat/cdutil/8.0.2018.08.03.09.36.g5f4fb99
Going to remove: cdat/cdutil/8.0.2018.08.03.09.56.g8805ee1
Going to remove: cdat/cdutil/8.0.2018.08.03.14.01.g298416a
Going to remove: cdat/cdutil/8.0.2018.08.03.16.41.g3da5cf0
Going to remove: cdat/cdutil/8.0.2018.08.03.17.03.gfdbaac2
Going to remove: cdat/dv3d/8.0.2018.09.18.15.46.g0c84513
Going to remove: cdat/dv3d/8.0.2018.09.18.15.53.g0c84513


$ python ./remove_anaconda_pkgs.py -R ".*2018.*" -c cdat/label/linatest -d
CMD: conda search --override -c cdat/label/linatest
Loading channels: ...working... done
# Name                  Version           Build  Channel
cdutil          8.0.2018.07.30.15.26.ga076047          py36_0  cdat/label/linatest
cdutil          8.0.2018.08.03.08.35.g90c6425          py36_0  cdat/label/linatest
cdutil          8.0.2018.08.03.09.05.g2b08380          py36_0  cdat/label/linatest
cdutil          8.0.2018.08.03.09.36.g5f4fb99          py36_0  cdat/label/linatest
cdutil          8.0.2018.08.03.09.56.g8805ee1          py36_0  cdat/label/linatest
cdutil          8.0.2018.08.03.14.01.g298416a          py36_0  cdat/label/linatest
cdutil          8.0.2018.08.03.16.41.g3da5cf0          py36_0  cdat/label/linatest
cdutil          8.0.2018.08.03.17.03.gfdbaac2          py36_0  cdat/label/linatest
dv3d            8.0.2018.09.18.15.46.g0c84513          py27_0  cdat/label/linatest
dv3d            8.0.2018.09.18.15.53.g0c84513          py36_0  cdat/label/linatest
thermo          8.0.2019.02.15.22.33.g8fdc229            py_0  cdat/label/linatest

Going to remove: cdat/cdutil/8.0.2018.07.30.15.26.ga076047
Going to remove: cdat/cdutil/8.0.2018.08.03.08.35.g90c6425
Going to remove: cdat/cdutil/8.0.2018.08.03.09.05.g2b08380
Going to remove: cdat/cdutil/8.0.2018.08.03.09.36.g5f4fb99
Going to remove: cdat/cdutil/8.0.2018.08.03.09.56.g8805ee1
Going to remove: cdat/cdutil/8.0.2018.08.03.14.01.g298416a
Going to remove: cdat/cdutil/8.0.2018.08.03.16.41.g3da5cf0
Going to remove: cdat/cdutil/8.0.2018.08.03.17.03.gfdbaac2
Going to remove: cdat/dv3d/8.0.2018.09.18.15.46.g0c84513
Going to remove: cdat/dv3d/8.0.2018.09.18.15.53.g0c84513

@muryanto1 muryanto1 requested a review from doutriaux1 February 26, 2019 00:23
@doutriaux1
Copy link
Copy Markdown
Contributor

doutriaux1 commented Feb 26, 2019

@muryanto1 thanks for this, few question, what about packages with multiple channels? e.g "main" and "nightly", is there a way for us to know this?

@doutriaux1
Copy link
Copy Markdown
Contributor

doutriaux1 commented Feb 26, 2019

if there's no way it's ok, let's make sure we do not have any nightly/main or nightly/v80 or this sort anywhere. Also we might want our default to ask before removing any file.

Muryanto added 2 commits February 26, 2019 09:11
adding a prompt before really removing
244d8f1
add support for specifying multiple channels, and by default it will …
376dc49 
…prompt user before really removing, unless '--no_prompt' option is specified
@muryanto1
Copy link
Copy Markdown
Member Author

muryanto1 commented Feb 26, 2019

I added support for specifying multiple channels, and by default it will prompt user before really removing, unless '--no_prompt' option is specified.

@xylar
Copy link
Copy Markdown
Collaborator

xylar commented Feb 26, 2019

I don't know if you guys are aware of this already, but some package versions have been removed that some E3SM software relies on (specifically cdp 1.4.1). I would recommend being very careful about removing packages from the anaconda channel.
cc @zshaheen @sterlingbaldwin

@zshaheen
Copy link
Copy Markdown
Contributor

zshaheen commented Feb 26, 2019

@xylar, yes I've noticed that. My plan is to release a version of e3sm_diags that's:

  • built with noarch, so it'll support Python 3.7, like how you've ask for here.
  • Use >= signs so it'll have future compatibility and work better in the unified env.

@muryanto1
Copy link
Copy Markdown
Member Author

muryanto1 commented Feb 26, 2019

I don't know if you guys are aware of this already, but some package versions have been removed that some E3SM software relies on (specifically cdp 1.4.1). I would recommend being very careful about removing packages from the anaconda channel.
cc @zshaheen @sterlingbaldwin

I have been testing the script that I added using my cdat/label/linatest channel...so I do not know what happened to those deleted cdp packages.

@xylar
Copy link
Copy Markdown
Collaborator

xylar commented Feb 26, 2019

I have been testing the script that I added using my cdat/label/linatest channel...so I do not know what happened to those deleted cdp packages.

Okay, no problem. It may be unrelated. I just wanted to be sure you were careful about which packages get deleted because obviously some over-zealous removal has happened recently.

@xylar
Copy link
Copy Markdown
Collaborator

xylar commented Feb 26, 2019

I opened #59 so it doesn't get confused with this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@doutriaux1 doutriaux1 Awaiting requested review from doutriaux1

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@muryanto1 @doutriaux1 @xylar @zshaheen
⚡