Add an argument to ipcluster plugin to specify the number of engines

[clearf]

• Add parameters MASTER_ENGINES and NODE_ENGINES for the ipcluster and ipclusterrestart plugins to allow for the specification of a certain number of engines on the master and nodes, respectively.
  • This is useful to run no calculations on the master (which can deplete resources and cause the master to hang), and
  • Use e.g., scikit-learn's native joblib support (e.g., using jobs=-1) to enable multiprocessing on the nodes and reduce the amount of message passing around machines in the cluster.

Also addresses #538

[cancan101]

Does this allow you to set master_engines to 0?

[cancan101]

I wrote some similar logic: develop...cancan101:develop#diff-4774c0a25748eaab7628c5b506730127 (sorry its intermingled with a couple other changes).

[clearf]

Yes, it allows you to set master engines to zero and set NODE_ENGINES as well.

I've had problems where my master has gotten hammered by calculations and/or memory (even with master_engines = num_processors-1 engines), and the whole cluster has timed out.

This PR is a pretty simple way of accomplishing that (and the parameters are respected for ipclusterrestart as well)

[cancan101]

if self.master_engines == 0 won't this branch be taken? i.e. using 0 or None to turn of the master will not work.

[clearf]

Hmm... this works in testing. Let me figure this out.

Oh, actually, at this point in the execution, this is the string '0', which evaluates to true. This is bad and confusing, though, so I will change to check for "None". Thanks.

[cancan101]

Also related to this PR: #538.