Wazuh-logtest: Implement logtest as a thread of analysisd

[Lopuiz]

Wazuh version	Component	Install type	Install method	Platform
4.1.0	Analysisd	Manager	Packages/Sources	Linux

Hello team,

ossec-logtest is a useful tool for testing and verification of rules and decoders. Users can test their custom ruleset before adding it in a production environment.

Although it is a good tool, there are some issues in Logtest:

1. UI can't use it. The users must access the server where the wazuh-manager is installed and execute the binary in the console.
2. Can't test rules and decoders for eventchannel events, syscheck events, the events collected with different logcollector formats (mysql, postgresql, command, full_command), etc.

This development aims to deprecate the ossec-logtest in favor of wazuh-logtest.

This new tool will be an Analysisd thread which listens in AF_UNIX socket. Multiples clients can be connected. The server processes their requests and responds. The communication messages will be in JSON format.

Additionally, wazuh-logtest must allow testing more events than ossec-logtest.

Best regards,
Core team.

---

Working branch
feature/5337-logtest-enhancement

[JcabreraC]

Current status of ossec-logtest

Currently, logtest runs as a separate wazuh-manager program. This makes it easier to check rules, decoders and cdb lists that the manager does not have in memory, as each process has its own rule and decoder lists, its frequency, fts, timeframe, etc. This does not conflict with the state of Analysisd.

To differentiate the log flow of Analysisd and Logtest, the TESTRULE flag is used.

The logtest code can be found in the testrule.c file.

Logtest control flow

Program Initialization:

1. Read the configuration to load the rules.
2. Obtains the name of the server and removes the part of the domain if it is available.
3. Check that the user/group is valid.
4. Create an event list using the OS_CreateListEvent function.
5. Initialization of the decoders:
   a. Create and initialize the decoder list with OS_CreateOSDecoderList.
   b. If there is no decoder list in the global configuration file, read the decoders and reload them based on the ossec.conf file. Read the local decoder list.
   1. If there is a list of decoders in the global configuration file, it loads them from the ossec.conf file.
      c. Load the decoders with SetDecodeXML.
6. Initialization of the lists:
   a. Check the list file, if it does not give an error, load each list in a list struct with the function Lists_OP_LoadList.
7. Initialization of the rules:
   a. Create the rule list with Rules_OP_CreateRules.
   b. Read the rules.
   c. Find all rules that require list searches and attach the correct list structure to the rule with OS_ListLoadRules.
8. Initialize the analysisd queues with the function w_init_queues.
9. Sets the total number of enabled rules. Start with the first rule and calculate how many children/levels it can have to get the total of possible rules.
10. Creates a hash table of rules to read alerts from other servers.
11. Prepares the user.
12. Launches the startup message.

Log analysis

Log analysis has three phases:

1. Pre-processing:

• The message is received from the queue.
• The time at which the event was received is obtained.
• Blank lines and line feed are ignored.
• The default values for the event are configured.
• The OS_CleanMSG function divides the log in four fields: timestamp, hostname, program-name, log (message). It stores it in the event.

2. Decoders:

• The size of the log is obtained.
• Osdecoders decode the received event and obtain useful information using the function DecodeEvent.
• Data from events that share the same ID are grouped together.

3. Rules

• Get the first rule with OS_GetFirstRule and start a loop through all.
• If the event is an "alert" type, process the alert.
  • If not, check that the category matches the event type.
  • If not, check for each rule whether the current rule matches the event information.
• A pointer is created to the rule that has been generated.
• Level 0 is ignored.
• Check the ignore time. If the time the rule was ignored is less than the time it should be ignored, do not alert again.
• Check if the event should be ignored.
• Checks if it should be added to the ignore list.
• Shows the alert if it's configured.
• Check if the rule is if_matched_sid. If so, it copies the structure to its status memory.
• Checks if the rule is if_matched_group. If so, it copies the structure to its status memory.
• Adds the event to the list of previous events with OS_AddEvent.

4. The information is removed and the memory deleted only if the event was not added to the status memory.

[JcabreraC]

Design of Wazuh-logtest

Requirements

• It won't be a separate program anymore. Now it'll be a thread listening on a local socket (AF_UNIX).
• The input and output will be in JSON format.
• The input JSON will have a log or event field with the log in RAW.
• Multiple simultaneous and independent sessions should be allowed.
• The ruleset folder is common (several users compete for it), and each Logtest request will involve a ruleset parsing + log analysis.
• There will be a reload of rules.
• If the rules or decoders are not correct, the service reports the faults, but will not stop the execution.
• In a cluster, only the master can execute Logtest. The workers will send requests to the master.
• There must be a "Save" and an "Apply changes". "Apply changes" will cause the manager to restart
  "save" will only save.
• The number of threads can be chosen in the ossec.conf file. By default, just one thread.
• The maximum number of users running simultaneously logetes can be chosen in the ossec.conf file. 64 users by default.
• Wazuh-logtest will process at least the same logs as ossec-logtest.
• The time an active session will remain alive can be defined in the ossec.conf file. 15 minutes by the default.

Communication

Communication between wazuh-logtest and the API will not be with persistent connections. Each client's status and information will be saved, as they can make multiple requests. Messages in JSON format will contain a token identifying the client. The behavior would be as follows:

• A log processing request is received.
• The message is received and checked for session initiation for this message. If no session exists, one is initialized.
• The message is processed and sent a response.
• A logout request is received. The session corresponding to this user is searched and closed. All open sessions where the client has not connected to the server in more than 15 minutes are closed.

Configuration

We will have a new section in the ossec.conf file to configure some wazuh-logtest parameters.

By default, this cofiguration will be:

<rule_test>
  <enabled>yes</enabled>
  <threads>1</threads>
  <max_sessions>64</max_sessions>
  <session_timeout>15m</session_timeout>
</rule_test>

The labels are:

Option	Description	Default value	Values allowed
enable	determine if logtet is enabled or disabled	yes	yes/no
threads	number of wazuh-logtest threads	1	a number between 1 and 128, or auto to create one thread per CPU
max_sessions	number of users connected simultaneously	64	a number between 1 and 65534
session_timeout	time interval in which a client must remain offline to remove the resources associated with their session	15 minutes	a number to represent seconds, to represent the interval in minutes adds m, or h to represent the interval in hours.

Inputs

JSON input fields:

• token: Client ID
• event: Log to be processed
• log_format: Type of log. It can be: syslog, syscheck_event, eventchannel, eventlog, command, full_comand, mysql, etc.
• location: the origin of the log. User, agent, IP and file (if collected by Logcollector).

example:

{
 “token”: “kTy234Sdvtp7”,
 "event": "Jun 24 11:54:19 Master systemd[2099]: Started VTE child process 20118 launched by terminator process 17756.",
"log_format": "syslog",
"location": "master->/var/log/syslog"
}

Outputs

The fields of the output JSON will be the same as the JSON fields of an alert.

example:

Input:

{
"event": "Jun 24 11:28:09 master sudo: master : TTY=pts/0 ; PWD=/home/master ; USER=root ; COMMAND=/bin/su",
"log_format": "syslog",
"location": "master->/var/log/auth.log"
}

Output:

{
    "timestamp":"2020-06-24T11:28:09.308+0200",
    "rule":{
        "level": 3,
        "description":"Successful sudo to ROOT executed.",
        "id":"5402",
        "mitre":{
            "id":["T1169"],
            "tactics":["Privilege Escalation" ]
            },
        "firedtimes":1,
        "mail":false,
        "groups":[ "syslog", "sudo" ],
        "pci_dss":[ "10.2.5", "10.2.2" ],
        "gpg13":[
            "7.6",
            "7.8",
            "7.13"
        ],
        "gdpr":[ "IV_32.2"],
        "hipaa":[ "164.312.b"],
        "nist_800_53":[ "AU.14", "AC.7", "AC.6" ],
        "tsc":["CC6.8","CC7.2","CC7.3"]
    },
    "agent":{
        "id":"000",
        "name":"master"
    },
    "manager":{
            "name":"master"
    },
    "id":"1592990889.62076",
    "full_log":"Jun 24 11:28:09 master sudo: master : TTY=pts/0 ; PWD=/home/master ; USER=root ; COMMAND=/bin/su",
    "predecoder":{
        "program_name":"sudo",
        "timestamp":"Jun 24 11:28:09",
        "hostname":"master"
    },
    "decoder":{
        "parent":"sudo",
        "name":"sudo",
        "ftscomment":"First time user executed the sudo command"
    },
    "data":{
        "srcuser":"master",
        "dstuser":"root",
        "tty":"pts/0",
        "pwd":"/home/master",
        "command":"/bin/su"
    },
    "location":"/var/log/auth.log"
}