Update AWS-S3 module to avoid AWS Config performance issues

[CarlosRS9]

Hello team,

As cloud team reported here there is a potential performance issue when using the AWS-S3 module to monitor AWS Config logs as wazuh-alerts index pattern fields may grow continuously due to the usage of instance ids, buckets and other variables as part of the field names. It can affect the Elasticsearch performance significantly.

Proposed solution

Several solution were analyzed and discarted in the related issue. After discussing this with the Cloud team we concluded the best solution we can apply is the removal of all fields that may cause this behaviour since its possible that an AWS Config alerts references more than one instance ID.

To clarify this solution, lets take a look to an example of a problematic alert:

configuration.AWS:InstanceInformation.Content.<i-AAAAAAAAAAAAA>.ComputerName": "name",
configuration.AWS:InstanceInformation.Content.<i-AAAAAAAAAAAAA>.InstanceId": "<i-AAAAAAAAAAAAA>",
configuration.AWS:InstanceInformation.Content.<i-AAAAAAAAAAAAA>.IpAddress": "<ip>",
configuration.AWS:InstanceInformation.Content.<i-AAAAAAAAAAAAA>.AgentType": "amazon-ssm-agent",
configuration.AWS:InstanceInformation.Content.<i-AAAAAAAAAAAAA>.ResourceType": "EC2Instance",
configuration.AWS:InstanceInformation.Content.<i-AAAAAAAAAAAAA>.AgentVersion": "<version>",
configuration.AWS:InstanceInformation.Content.<i-AAAAAAAAAAAAA>.PlatformVersion": "<version>",
configuration.AWS:InstanceInformation.Content.<i-AAAAAAAAAAAAA>.PlatformName": "<platform>",
configuration.AWS:InstanceInformation.Content.<i-AAAAAAAAAAAAA>.PlatformType": "Windows"

configuration.AWS:InstanceInformation.Content.<i-BBBBBBBBBBBBB>.ComputerName": "name",
configuration.AWS:InstanceInformation.Content.<i-BBBBBBBBBBBBB>.InstanceId": "<i-BBBBBBBBBBBBB>",
configuration.AWS:InstanceInformation.Content.<i-BBBBBBBBBBBBB>.IpAddress": "<ip>",
configuration.AWS:InstanceInformation.Content.<i-BBBBBBBBBBBBB>.AgentType": "amazon-ssm-agent",
configuration.AWS:InstanceInformation.Content.<i-BBBBBBBBBBBBB>.ResourceType": "EC2Instance",
configuration.AWS:InstanceInformation.Content.<i-BBBBBBBBBBBBB>.AgentVersion": "<version>",
configuration.AWS:InstanceInformation.Content.<i-BBBBBBBBBBBBB>.PlatformVersion": "<version>",
configuration.AWS:InstanceInformation.Content.<i-BBBBBBBBBBBBB>.PlatformName": "<platform>",
configuration.AWS:InstanceInformation.Content.<i-BBBBBBBBBBBBB>.PlatformType": "Windows"

In this example <i-AAAAAAAAAAAAA> and <i-BBBBBBBBBBBBB> correspond to instance IDs that are being used as a part of the field name. With the proposed solution every field from the ID onwards will be removed from the alert:

configuration.AWS:InstanceInformation.Content

This will reduce the number of available fields in the alerts but will ensure the compatibility while avoiding the performance and resources issue. The removed fields may be added again in the future if client request them.

Required changes

To apply this solution its necessary to update our AWS-S3 module so the AWS Config alerts contaning this hierarchy will be pre-pocessed to remove problematic fields before sending them to Analysisd.

Tasks

• Determine which configuration elements must be pre-processed.
• Apply the required changes to the AWS-S3 module to remove problematic AWS Config fields before sending the alert to Analysisd
• Obtain real AWS Config logs from Cloud team to properly test this solution.

[CarlosRS9]

Status Update

We have tried to determine which elements of the configuration need to be pre-processed. Since we do not have enough valid logs with this issue we have not been able to perform the required checks. Therefore, it has not been possible to determine exactly which elements should be pre-processed and which should not. However, we can conclude that all problematic cases had the 'configuration.<configuration_element>.Content.*' hierarchy. Because of that, we have implemented a function that pre-processes and removes the extra fields whenever that hierarchy is present.

Here is an example to clarify this. This is a fragment of a log with several fields to be removed:

{"configuration": {
    "AWS:InstanceInformation": {
        "Content": {
            "i-AAA": {
                "ComputerName": "name1",
                "InstanceId": "i-AAA",
                "IpAddress": "0.0.0.0",
                "AgentType": "amazon-ssm-agent",
                "ResourceType": "EC2Instance",
                "AgentVersion": "4.0",
                "PlatformVersion": "2",
                "PlatformName": "Amazon Linux",
                "PlatformType": "Linux"
            },
            "i-BBB": {
                "ComputerName": "name2",
                "InstanceId": "i-BBB",
                "IpAddress": "0.0.0.0",
                "AgentType": "amazon-ssm-agent",
                "ResourceType": "EC2Instance",
                "AgentVersion": "4.0",
                "PlatformVersion": "2",
                "PlatformName": "Amazon Linux",
                "PlatformType": "Linux"
            },
            "i-CCC": {
                "ComputerName": "name3",
                "InstanceId": "i-CCC",
                "IpAddress": "0.0.0.0",
                "AgentType": "amazon-ssm-agent",
                "ResourceType": "EC2Instance",
                "AgentVersion": "4.0",
                "PlatformVersion": "2",
                "PlatformName": "Amazon Linux",
                "PlatformType": "Linux"
            }
        },
        "SchemaVersion": "1.0"
    }
}}

With this solution, the values for i-AAA, i-BBB and i-CCC are removed and the value of the Content field correspond to the list of IDs (i-AAA, i-BBB and i-CCC). Here is the result after the pre-processing:

{
    'configuration': {
        'AWS:InstanceInformation': {
            'Content': ['i-AAA', 'i-BBB', 'i-CCC'],
            'SchemaVersion': '1.0'
        }
    }
}

The described function has been implemented. Tests have yet to be carried out to ensure this change does not break anything and verify the result obtained is the desired one. We must also obtain better logs to be able to perform the tests.

[CarlosRS9]

Status Update

The development is finished. Several tests have been performed to ensure everything works as expected. A more in-depth analysis and attempt to obtain better logs for the tests are required.