Final re-work of AWS wodle as per #510#1105
Merged
jesuslinares merged 25 commits into3.6from Aug 24, 2018
Merged
Conversation
- Scalability and functional enhancements for parsing of CloudTrail - Support for existing config params - Upgrade to a granular object key addressing to support multiple CloudTrails in S3 bucket - Support granular parsing by account id, region, prefix - Support only parsing logs after a given date - Support IAM credential profiles, IAM roles - Only look for new logs/objects since last iteration - Skip digest files altogether (only look at logs) - Move from downloading object and working with file on filesystem to byte stream - Inherit debug from modulesd - Add bounds checks for msg against socket buffer size; truncate fields if too big (#733) - Support multiple debug levels - Move connect error so not confused with general error - If fail to parse log, and skip_on_error, attempt to send me msg to wazuh - Support existing configurations by migrating data, inferring other required params Associated documentation updates covered in wazuh/wazuh-documentation#274
…avoid conflicts with Elasticsearch dynamic fields and legacy events - Reparse flag to support re-parsing of log files from s3 bucket - Use CloudTrail timestamp for ES timestamp - Various optimizations
- Refactor to make easier to read
The exit code 2 was used for both argparse errors and SIGINTs. Now, the SIGINT code is 12 and argparse errors are shown in ossec.log.
- Fix bug with aws_account_id being required (should not be), not parsing correctly when not included - Reduce msg debug chatter for reformatting - Rename script and integration (from aws to aws-cloudtrail) to support additional/other AWS integrations
All error codes higher than 2 are processed in the same way. The error message is extracted from the script output.
All socket errors were interpreted as "Wazuh must be running", invisibilizing any other socket error.
This was referenced Aug 14, 2018
* Refactor code into classes The code was very focused on processing cloudtrail logs, in order to process more aws logs an abstract class has been defined. * Fix compilation warning * Iterate over firehouse buckets * Modify alert structure depending on service Instead of aws.cloudtrail.event now it is aws.<service_name>.event. * Remove aws.macie.event.trigger field from generated alerts That field made the alert to be too big and, in consecuence, not being correctly processed by analysisd. * Read s3 files in text mode instead of binary mode * Fix bug getting creation date of files that don't match regex If a file doesnt have its creation date of its filename, get that information from the "LastModified" attribute. * Prevent non cloudtrail logs to be removed from db Only 100 logs were retained in DB for each region. Since firehouse logs don't have any region, only 100 logs were retained for all firehouse buckets at the same time. To prevent that, the bucket name is used as region name. * Remove overwriting wazuh's timestamp and add srcip for all aws services * Replace "cloudtrail" names with "s3" Since the integration now accepts more kinds of log services, cloudtrail name has been replaced for s3. * Import rules from wazuh-ruleset repository * Improvements on alert formatting Remove service name from the alert and reformat single element lists fields into dictionaries * Fix bug opening zipfiles * Update elasticsearch template with IP and aws field types * Show which bucket is being processed at info log level * Update changelog for #1131
Contributor
|
Pending:
|
added 4 commits
August 23, 2018 15:08
Contributor
|
Thanks for this great contribution @mgmacias95 @UranusBytes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hello team,
This PR adds the full rework of the AWS (#913) wodle made by @UranusBytes with some improvements:
AssumeRolefails.Tested with both python3 and python2.
Best regards,
Marta