tests: better resilience for concurrent node starts

[prasannavl]

Summary

• Fixes most node start failures due to high concurrency during normal operation.
• Rationale: When concurrency levels are high, nodes end up failing to start due to the strict flaky way of checking of RPC start. While this is still not bulletproof, and if system is out of resources, there can still be other failures, however this adds significant resiliency for on-start check.
  • For most normal operation (eg: MAKE_JOBS=<no-of-logical-CPUs>), this provides a good default and avoids most false failures due to slightly delayed node starts.

Implications

• Storage

  • Database reindex required
  • Database reindex optional
  • Database reindex not required
  • None

• Consensus

  • Network upgrade required
  • Includes backward compatible changes
  • Includes consensus workarounds
  • Includes consensus refactors
  • None