Support binary download of spool content in ZOWE Explorer extender APIs

[VitGottwald]

Is your feature request related to a problem? Please describe.

We have a process that stores binary data in spool files and we would like to use ZOWE Explorer API to download it in a VSCode extension.

Describe the solution you'd like

Support binary download of spool content in ZOWE Explorer extender APIs (at least through z/OSMF).

Describe alternatives you've considered

We have a working mechanism to download the content through FTP (without ZOWE Explorer), but we would like to support the scenario with ZOSMF as well.

Additional context

The parameters of https://github.com/zowe/vscode-extension-for-zowe/blob/main/packages/zowe-explorer-api/src/profiles/ZoweExplorerZosmfApi.ts#L325 do not provide the ability to select binary/text download. It also looks like it is not implemented in the ZOWE SDK for jobs https://github.com/zowe/zowe-cli/blob/master/packages/zosjobs/src/DownloadJobs.ts#L52 . However, ZOSMF itself seems to support this. From the z/OSMF documentation the endpoint /zosmf/restjobs/jobs/{correlator}/files/{nnn}/records accepts a query parameter mode with possible values text/binary/record.
...

[github-actions]

Thank you for raising this issue.
The community has 90 days to upvote 👍 the issue.
If it receives 10 upvotes, we will move it to our backlog. If not, we will close it.

[zFernand0]

Hey @VitGottwald,
The implementation of the binary (and record) optional flags in the CLI (and Node.js SDK) was published in version v7.10.0.
Said enhancement was included in our Zowe Explorer API v2.7.0.
Here is a quick example on how to make use of it from a pure Zowe Explorer API perspective.

const zowe_explorer_api = require("@zowe/zowe-explorer-api");
const imperative = require("@zowe/imperative");
(async () => {
  const profile_info = new imperative.ProfileInfo("zowe");
  await profile_info.readProfilesFromDisk();
  const default_profile = profile_info.getDefaultProfile("zosmf");
  const loaded_profile = imperative.ProfileInfo.profAttrsToProfLoaded(default_profile);
  const merged_arguments = profile_info.mergeArgsForProfile(default_profile, { getSecureVals: true });
  const session = imperative.ProfileInfo.createSession(merged_arguments.knownArgs);
  const zosmf_jes_api = new zowe_explorer_api.ZosmfJesApi(loaded_profile);
  zosmf_jes_api.session = session;

  await zosmf_jes_api.downloadSpoolContent({
    jobid: "JOB01234",
    jobname: "JOBNAME",
    binary: true, // <-- cool feature : )
    outDir: __dirname
  });
})();

---

That being said, we are working on exposing this to end users somehow.
Here are our thoughts:

1. A new context-menu option.
   

2. Or automatically detecting that the downloaded spool content has non-printable characters.
   

   This approach is something similar to how VSCode complains depending the file tags on USS files.
   

---

Here is a test VSIX file for approach number 1 😋
zowe-explorer-2060.zip

Please let us know if that's sufficient (for now), or if you would like us to investigate a bit more what it would take to do the auto-detection.

[VitGottwald]

This is a great news, thank you for following up and integrating the new CLI capability to ZE @zFernand0!

Lets go through a typical scenario and for the sake of simplicity lets assume that my job only has one step. With multiple steps, the same logic would apply.

1. I submit my job
2. the step inside the job produces two outputs
   • the first output file (SYSPRINT) is a standard text report about the operation of the utility
     • summary of the requested operation
     • progress of the processing
     • encountered errors
     • final success or failure messages
   • the second output file (SYSDATA) contains binary data that is the actual result of the operation
3. I find the job in the ZE job view. I expand it and I see 5 outputs
   • (1) JESMSGLG (text)
   • (2) JESJCL (text)
   • (3) JESYSMSG (text)
   • (4) SYSPRINT (text)
   • (5) SYSDATA (binary)
4. I check JESMSGLG (text) for any allocation errors, abends, or other operating system messages. When everything is fine there ...
5. I check the SYSPRINT (text) to see the report about the requested operation. Everything looks good here as well.
6. Now I would like to inspect the binary output file SYSDATA

I believe that the two options you offer:

• downloading as text, detecting non-printable characters and offering a re-download in binary and
• right-click and select download in binary from context menu

are very nice will address the need from UI perspective.

The crucial piece is that these options are available on individual spool output files rather than on the whole job. What do you think?

[zFernand0]

I think it makes perfect sense. 😋

I felt a bit strange having to download the whole thing as binary because most DDs would just be garbled output except for the one (SYSDATA) that was actually meant to be binary.

I'll look into it further today.
Thanks for the concrete example!

[zFernand0]

Hey @VitGottwald,
Here is a quick update:

In the image above, you can see that Download and Download (Binary) have been added (with support for multi-select).
Hopefully this helps with downloading multiple spool files (from the same OR different jobs) at once 😋

And here is a the same implementation using the zowe-explorer-apis

const zowe_explorer_api = require("@zowe/zowe-explorer-api");
const imperative = require("@zowe/imperative");
const zowe_jobs_sdk = require("@zowe/zos-jobs-for-zowe-sdk");
(async () => {
  const profile_info = new imperative.ProfileInfo("zowe");
  await profile_info.readProfilesFromDisk();
  const default_profile = profile_info.getDefaultProfile("zosmf");
  const loaded_profile = imperative.ProfileInfo.profAttrsToProfLoaded(default_profile);
  const merged_arguments = profile_info.mergeArgsForProfile(default_profile, { getSecureVals: true });
  const session = imperative.ProfileInfo.createSession(merged_arguments.knownArgs);
  const zosmf_jes_api = new zowe_explorer_api.ZosmfJesApi(loaded_profile);
  zosmf_jes_api.session = session;

  const jobid = "JOB01234";
  const jobname = "JOBNAME";

  // The `downloadSpoolContent` downloads all spool files
  // await zosmf_jes_api.downloadSpoolContent({ jobid, jobname, binary: true, outDir: __dirname });
  
  const spools = (await zowe_jobs_sdk.GetJobs.getSpoolFiles(session, jobname, jobid));
  const spool = spools[spools.length - 1]; // select last spool file for testing purposes
  
  await zosmf_jes_api.downloadSingleSpool({ // using the zowe-explorer-api
  // await zowe_jobs_sdk.DownloadJobs.downloadSpoolContentCommon(session, {
  // You can uncomment the line above to use the jobs_sdk directly as well
    jobFile: spool,
    binary: true,
    outDir: __dirname,
  });
})();

[VitGottwald]

Thank you very much @zFernand0 it looks great!

As for the API, would it be possible to get a node stream rather than having to specify a folder and then read it back from it?

We do not plan to store it to the file system, and we do not have a folder to provide. We plan to process the data in memory as it keeps coming.

[zFernand0]

Thanks for sharing a little more info regarding the use case
After looking into this a little more, there seems to be no way to stream spool content without having to construct the zosmf endpoint yourself 😢

I'm looking into it though 😋

---

Update:
Here is a quick PR to update the SDK functionality

• Add stream capabilities to downloadSpoolContentCommon zowe-cli#1686

I'm afraid that this PR may not get included in 2.8.0 of the Zowe Explorer APIs.
However, you could experiment with this functionality (by using the @zowe/zos-jobs-for-zowe-sdk package) the day after the PR mentioned above is merged 😋