limit nested extglob recursion, flatten extglobs

This flattens nested extglobs where that can be done correctly.

In other cases, the recursion into nested extglobs is limited to
2, in order to prevent runaway ReDOS patterns.

fix: GHSA-23c5-xmqv-rm74

Author: isaacs

README.md

@@ -412,6 +412,26 @@ match the pattern provided.
 That is, this is an intentional false negative, deemed an
 acceptable break in correctness for security and performance.
 
+### maxExtglobRecursion
+
+Max depth to traverse for nested extglobs like `*(a|b|c)`
+
+Default is 2, which is quite low, but any higher value swiftly
+results in punishing performance impacts. Note that this is _not_
+relevant when the globstar types can be safely coalesced into a
+single set.
+
+For example, `*(a|@(b|c)|d)` would be flattened into
+`*(a|b|c|d)`. Thus, many common extglobs will retain good
+performance and never hit this limit, even if they are
+excessively deep and complicated.
+
+If the limit is hit, then the extglob characters are simply not
+parsed, and the pattern effectively switches into `noextglob:
+true` mode for the contents of that nested sub-pattern. This will
+typically _not_ result in a match, but is considered a valid
+trade-off for security and performance.
+
 ## Comparisons to other fnmatch/glob implementations
 
 While strict compliance with the existing standards is a