Rework Job Retrieval/Submission API

[tazend]

Branch is here

Classes

• Job class providing read-only access to all possible attributes of jobs (e.g. time of submission)
• Jobs class, acting as a custom collection (dict) to retrieve actual job-information, and holds all the Job instances which carry the information.
• JobStep class providing read-only access to all possible attributes of a single Job-step
• JobSteps class, acting as custom collection (dict) to hold all the 'JobStep' instances
• JobSubmission class, acting as a interface to create a job-submission records and to submit them
  • support parsing #SBATCH directives from the specified submit-script
    - will be pretty easy once all attributes also have the logic to accept sbatch-like input for these (e.g. parsing something more complex like --cpu-frequency.
  • support loading submit-description parameters from environment variables. Same as described above. There really needs to be a way to use env vars for setting params like SBATCH_xxx #130

General Functionality for a Job

• cancel (SIGKILL)
• send_signal (e.g send any signal like SIGTERM,SIGKILL,SIGUSR1 to the job)
• suspend/unsuspend
• hold/release
• modify (user must pass a JobSubmission instance with the changes to be made)
• notify
• Retrieve the batch-script and return it as a string

General Functionality for a JobStep

• cancel
• send_signal
• modify (only changing the TimeLimit is allowed for a job step

Other

• Documentation
• Write tests
• Write examples

Related to #11

[tazend]

Hi @giovtorres , I think I'm making some good progress with the job class(es) (job-submission, retrieval, etc..). Would it be possible to maybe create a seperate branch for this refactoring process? Maybe something like dev-refactor or just dev branch , where I could push the progress? (I think I'm not able to create branches here). I'd really like to bring the project forward in this regard.

What do you think?

[giovtorres]

Great to hear the progress! Go ahead and create a branch. You should be able to now.

[tazend]

I will continue to work on this again now.

Haven't done anything on this in a while, however last time I already got pretty far with the rework (except a few job-submission parameters that require a bit more extensive input-validation.)

[tazend]

Made a new branch here with current progress.

[tazend]

Pretty much 90% of the general functionality is now implemented/reworked, though it will still need some bug-fixing probably, polish and fixing some outstanding TODO's. And need to write tests for this.

[multimeric]

This change looks good (main...rework/jobs). This is the object oriented wrapper I was hoping pyslurm would have. Do you need any help getting this finished, e.g. code review?

The main thing I'm after is actually a wrapper around slurmdb_jobs, which you haven't covered in this PR. Is that something you've started elsewhere? If not, I can give it a go. Would it make more sense to branch off this (very large) branch, or main?

[tazend]

Hi @multimeric

Good timing :)
I do actually have something in the works already for a wrapper around the database Job API, and it is the next thing that I was going to put my focus on, as its also a part of Slurm that users interact with quite often.
I haven't pushed it yet, but I'll push a new branch based on this one and try to get it up later today when I'm back from work. I'll let you know and then I'd definitely appreciate feedback on it :)

For the rework regarding this issue - its pretty much done. Was just going to check for some last possible changes and some missing documentation/examples. But I do definitely want to get it finally merged and available for people in the next few days, and improvements/missing features can be applied continuously.

[multimeric]

Wonderful, let me know when you need help!