Let GSK validate the CN and SAN domains of server certificates

[ChongZhou-Broadcom]

In httpClientSessionInitv2(), tls socket is initialized without CN/SAN validation.

GSK API is capable of validating the CN and SAN domains, but it's not a default behavior. The peer's hostname must be set to the socket handle by gsk_attribute_set_buffer(). The following code snippet is an example:

   ...
   rc = rc || gsk_attribute_set_callback(socket->socketHandle, GSK_IO_CALLBACK, &ioCallbacks);
+  if (!isServer && peerHost) {
+    rc = rc || gsk_attribute_set_buffer(socket->socketHandle, GSK_REFERENCE_ID_DNS, peerHost, 0);
+    rc = rc || gsk_attribute_set_buffer(socket->socketHandle, GSK_REFERENCE_ID_CN, peerHost, 0);
+    rc = rc || gsk_attribute_set_enum(socket->socketHandle, GSK_WILDCARD_VALIDATION_ENABLE, +GSK_WILDCARD_VALIDATION_ENABLE_ON);
+  }
   rc = rc || gsk_secure_socket_init(socket->socketHandle);
   ...

@ifakhrutdinov Irek said he will be doing some research for this. I created this draft PR for him as a reference.

I tried this change with ZSS. I used the following configuration to test if the hostname validation really takes effect:

  zss:
    enabled: true
    port: 7557
    crossMemoryServerName: ZWESIS_STD
    agent:
      jwt:
        fallback: true
      64bit: true
      mediationLayer:
        server:
          gatewayHostname: 1.2.3.4
          #gatewayHostname: my.real_domain.com
          gatewayPort: 7554

Without this change, ZSS worked with both gatewayHostname values. With this change, only the real host name worked, the IP address, which is not in my certificate, no longer works.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[ifakhrutdinov]

@ChongZhou-Broadcom , thanks for opening this; we'll discuss the changes.