Update galaxy.tool_util.cwl for latest changes in CWL branch. by jmchilton · Pull Request #9175 · galaxyproject/galaxy

jmchilton · 2020-01-01T17:15:48Z

Needed for galaxyproject/planemo#978 but we may want to pull out just the changes to schema.py and backport them to 20.01.

jmchilton · 2020-01-03T15:25:28Z

Breaks something else in Planemo:

Exception [tool_proxy must be called with non-None uuid]

Tightened up how tools are loaded to try to ensure tools were loaded in consistent ways during the last CWL push and this broke non-internal Galaxy applications it seems.

nsoranzo · 2020-01-08T01:28:51Z

The failure of unit test test_validate_simple_config is due to the ruamel.yaml update to 0.16.0, whose changelog says: "duplicate keys are now an error when working with the old API as well".

jmchilton · 2020-01-08T14:50:03Z

common-workflow-language/cwltool@762532d

Looks like we could pin it to something in the 0.15 branch line for now? How does that sound?

nsoranzo · 2020-01-08T16:30:11Z

common-workflow-language/cwltool@762532d

Looks like we could pin it to something in the 0.15 branch line for now? How does that sound?

@jmchilton I think it would be better to workaround the duplicate-key weirdness of uWSGI's YAML files in _validate() in lib/galaxy/config/config_manage.py . I can try to implement that if you agree.

jmchilton · 2020-01-08T17:20:40Z

@nsoranzo sounds good to me if you have an idea how to do that.

which makes some tools fail by printing to stderr: ``` galaxy.tool_util.output_checker DEBUG 2020-01-07 02:59:07,720 job failed, detected state generic_error, standard error is - [/galaxy_venv/local/lib/python2.7/site-packages/cwltool/__init__.py:17: CWLToolDeprecationWarning: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as the Python 2.7 version of cwltool won't be maintained after that date. """, category=CWLToolDeprecationWarning) ] ```

xref. galaxyproject#8558 For other YAML files, `ordered_load()` will raise an exception when it finds a mapping with duplicate keys.

nsoranzo · 2020-01-12T02:11:31Z

All green! What do you guys think of how I've addressed the YAML issue?

mvdbeek · 2020-01-12T09:03:44Z

That looks very good to me, thanks @nsoranzo!

jmchilton · 2020-01-13T15:18:17Z

Thanks @nsoranzo - looks great!

jmchilton · 2020-01-13T15:31:34Z

Any appetite for backporting this to 20.01 for Planemo and such or should we just start cutting 20.5-dev packages?

mvdbeek · 2020-01-13T15:33:48Z

Yes, please, I'll open a PR

jmchilton · 2020-01-13T15:39:07Z

Awesome - that will bring us closer to a stable release of galaxy-tool-util and such. But I'll publish dev packages when this is merged into 20.01.

nsoranzo · 2020-01-13T16:58:11Z

Thanks both! I still had a couple of unsubmitted review comments, I'll move them to #9210.

jmchilton · 2020-01-13T17:37:28Z

Traceback (most recent call last):
  File "/Users/john/workspace/galaxy/lib/galaxy/tools/toolbox/base.py", line 663, in _load_tool_tag_set
    tool = self.load_tool(concrete_path, use_cached=False)
  File "/Users/john/workspace/galaxy/lib/galaxy/tools/toolbox/base.py", line 855, in load_tool
    tool = self.create_tool(config_file=config_file, tool_shed_repository=tool_shed_repository, guid=guid, **kwds)
  File "/Users/john/workspace/galaxy/lib/galaxy/tools/__init__.py", line 284, in create_tool
    raise e
  File "/Users/john/workspace/galaxy/lib/galaxy/tools/__init__.py", line 279, in create_tool
    tool_location_fetcher=self.tool_location_fetcher,
  File "/Users/john/workspace/galaxy/lib/galaxy/tool_util/parser/factory.py", line 39, in get_tool_source
    as_dict = ordered_load(f)
  File "/Users/john/workspace/galaxy/lib/galaxy/util/yaml_util.py", line 70, in ordered_load
    return yaml.load(stream, OrderedLoader)
  File "/Users/john/workspace/galaxy/.venv/lib/python3.6/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/Users/john/workspace/galaxy/.venv/lib/python3.6/site-packages/yaml/constructor.py", line 43, in get_single_data
    return self.construct_document(node)
  File "/Users/john/workspace/galaxy/.venv/lib/python3.6/site-packages/yaml/constructor.py", line 52, in construct_document
    for dummy in generator:
  File "/Users/john/workspace/galaxy/.venv/lib/python3.6/site-packages/yaml/constructor.py", line 400, in construct_yaml_seq
    data.extend(self.construct_sequence(node))
  File "/Users/john/workspace/galaxy/.venv/lib/python3.6/site-packages/yaml/constructor.py", line 122, in construct_sequence
    for child in node.value]
  File "/Users/john/workspace/galaxy/.venv/lib/python3.6/site-packages/yaml/constructor.py", line 122, in <listcomp>
    for child in node.value]
  File "/Users/john/workspace/galaxy/.venv/lib/python3.6/site-packages/yaml/constructor.py", line 92, in construct_object
    data = constructor(self, node)
  File "/Users/john/workspace/galaxy/lib/galaxy/util/yaml_util.py", line 50, in construct_mapping
    value = loader.construct_object(value_node, deep=deep)
  File "/Users/john/workspace/galaxy/.venv/lib/python3.6/site-packages/yaml/constructor.py", line 92, in construct_object
    data = constructor(self, node)
  File "/Users/john/workspace/galaxy/lib/galaxy/util/yaml_util.py", line 50, in construct_mapping
    value = loader.construct_object(value_node, deep=deep)
  File "/Users/john/workspace/galaxy/.venv/lib/python3.6/site-packages/yaml/constructor.py", line 92, in construct_object
    data = constructor(self, node)
  File "/Users/john/workspace/galaxy/lib/galaxy/util/yaml_util.py", line 50, in construct_mapping
    value = loader.construct_object(value_node, deep=deep)
  File "/Users/john/workspace/galaxy/.venv/lib/python3.6/site-packages/yaml/constructor.py", line 92, in construct_object
    data = constructor(self, node)
  File "/Users/john/workspace/galaxy/lib/galaxy/util/yaml_util.py", line 54, in construct_mapping
    "found duplicated key (%s)" % key, key_node.start_mark)
yaml.constructor.ConstructorError: while constructing a mapping
  in "/Users/john/workspace/galaxy/test/functional/tools/simple_constructs.yml", line 105, column 9
found duplicated key (has_line)
  in "/Users/john/workspace/galaxy/test/functional/tools/simple_constructs.yml", line 107, column 9

I guess my own fault really though...

jmchilton added kind/enhancement area/cwl labels Jan 1, 2020

galaxybot added the status/WIP label Jan 1, 2020

jmchilton removed the status/WIP label Jan 3, 2020

jmchilton changed the title ~~[WIP] Update galaxy.tool_util.cwl for latest changes in CWL branch.~~ Update galaxy.tool_util.cwl for latest changes in CWL branch. Jan 3, 2020

mvdbeek approved these changes Jan 3, 2020

View reviewed changes

galaxybot added this to the 20.05 milestone Jan 3, 2020

nsoranzo force-pushed the cwl_rev branch 2 times, most recently from 4b6db2c to 27c9982 Compare January 7, 2020 18:44

jmchilton and others added 6 commits January 10, 2020 11:03

Update galaxy.tool_util.cwl for latest changes in CWL branch.

6697904

Fix up bad import of galaxy.model stuff from tool_util.

d33d4fe

Fix schema salad requirement.

3a1db7b

More fixes for galaxy-tool-util from Planemo test suite.

1202663

Update cwltool dependencies

3bdb1ce

nsoranzo force-pushed the cwl_rev branch from f09858f to 3bdb1ce Compare January 10, 2020 11:07

nsoranzo added 3 commits January 10, 2020 11:08

Disable Python 2.7 mulled tests

0aa2199

Always use yaml.safe_load()

cb742b4

Merge duplicate keys before validating YAML config files

c498f45

xref. galaxyproject#8558 For other YAML files, `ordered_load()` will raise an exception when it finds a mapping with duplicate keys.

nsoranzo force-pushed the cwl_rev branch from ffec92f to c498f45 Compare January 11, 2020 22:16

jmchilton merged commit da637ab into galaxyproject:dev Jan 13, 2020

nsoranzo deleted the cwl_rev branch January 13, 2020 16:57

Conversation

jmchilton commented Jan 1, 2020

Uh oh!

jmchilton commented Jan 3, 2020

Uh oh!

nsoranzo commented Jan 8, 2020

Uh oh!

jmchilton commented Jan 8, 2020

Uh oh!

nsoranzo commented Jan 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmchilton commented Jan 8, 2020

Uh oh!

nsoranzo commented Jan 12, 2020

Uh oh!

mvdbeek commented Jan 12, 2020

Uh oh!

jmchilton commented Jan 13, 2020

Uh oh!

jmchilton commented Jan 13, 2020

Uh oh!

mvdbeek commented Jan 13, 2020

Uh oh!

jmchilton commented Jan 13, 2020

Uh oh!

nsoranzo commented Jan 13, 2020

Uh oh!

jmchilton commented Jan 13, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nsoranzo commented Jan 8, 2020 •

edited

Loading