Test scripts for postgres

[vidyalakshmir]

We need a test script to test Postgres

• Either repurpose the existing test suite (usually Makefile will have a target test or check) to run within lind-wasm or just write a test script that tests basic functionality of postgres
• Each app should have its own shell script test.sh (lind-wasm-apps/postgres/test.sh, which outputs exit 0 on success, anything else will count as a fail. For now, check for binaries being installed in the correct location, and determine success/failure for that.
• Binaries are usually placed in lindfs/bin directory (preferably without any extension)
• Make sure all paths that the test uses are relative to lindfs and the corresponding files are actually copied to lindfs. To refer a file /home/lind/lind-wasm/lindfs/tests/postgres/test.txt, please use tests/postgres/test.txt
• Test scripts that are copied to lindfs can be placed in lindfs/tests/postgres folder
  Command to run an app within lind:
  lind_run $app
  For example, to run /home/lind/lind-wasm/lindfs/bin/bash with an argument /home/lind/lind-wasm/lindfs/tests/bash/test1.sh, please run

lind_run bin/bash tests/bash/test1.sh

[rishabhBudhouliya]

@vidyalakshmir as first steps, I am checking out PR #107 and building postgres and confirm that the binaries are copied at the right place. After that sanity step, I will write a test script that starts with binary existence checks, then do basic functionality tests.

[vidyalakshmir]

Sounds good. You could check bash, sed or curl test scripts for reference.

[rishabhBudhouliya]

Current status

What works
I've created a test script that does:

• check if postgres binaries exist in lindfs/bin
• validate if postgres --version works or not
• validate if postgres --help runs or not

I've manually tested the script and it seems to be working as stated.

What doesn't work:

1. initdb -D /tmp/pgdata --no-locale

LIND DEBUG PANIC: Unsupported mmap flag detected! Only MAP_FIXED,   
  MAP_SHARED, MAP_PRIVATE, MAP_POPULATE AND MAP_ANONYMOUS allowed

2. postgres --single -D /tmp/pgdata

FATAL: signalfd() failed

[rishabhBudhouliya]

@vidyalakshmir @rennergade
I've been trying to make postgres run and have been running into different issues

1. Unsupported mmap flags
   The initdb -D /tmp/pgdata --no-locale command fails with a panic

LIND DEBUG PANIC: Unsupported mmap flag detected! Only MAP_FIXED, MAP_SHARED, MAP_PRIVATE, MAP_POPULATE AND MAP_ANONYMOUS allowed

2. postgres --single -D /tmp/pgdata

FATAL: signalfd() failed

3. After fixing the mmap flags issue, we run initdb -D /tmp/pgdata --no-locale again and fail due to

  popen("postgres --boot ...") ->  glibc's _IO_new_proc_open() -> __posix_spawn() -> __spawni() -> clone(CLONE_VM | CLONE_VFORK) -> segfault

It seems like we don't have clone syscall implemented in lind-wasm. Furthermore, file_tests/deterministic/popen.c test is in skiplist so I am assuming it fails right now.

I am unsure of the blast radius of the changes that I'd make here, so should I fix these to move forward? Apparently, spawni() is a glibc-internal optimization that avoids fork + exec() and calls CLONE instead which is not implemented in lind-wasm.

[rennergade]

We definitely have clone implemented.

Theres a long standing PR to fix spawni: Lind-Project/lind-wasm#360

@qianxichen233 just needs to update the comments to merge it, can you do that qianxi?

[vidyalakshmir]

@rishabhBudhouliya I just talked to Qianxi. He mentioned the following:

• For signalfd() failure, first pls test if signalfd() is actually supported within lind. You could write a simple test for that. If it is not supported, one way is to try 'disabling signalfd' when postgres is built. While running configure, you can disable certain functions/sycalls. (Check this in python build). ./configure --help should give all options to enable/disable different options.
• For spawni, it would be better to first test locally - the changes in this PR merged with current main - to see if this fix still applies. A simple test that uses popen() should work. Since it requires bash, compiling bash would be required. (change bash to sh, as open() expects sh).

[rishabhBudhouliya]

@vidyalakshmir @rennergade
I have used Qianxi's PR and I'm able to get past the popen() limitations.

After applying certain changes to make initdb work first, I am current facing two problems, one:

1. sudo ./lind-boot /bin/initdb.wasm -D /tmp/pgdata --no-locale

thread 'lind-fork-3' panicked at sysdefs/src/logging.rs:6:5:                                                                                                                                 
  LIND DEBUG PANIC: Unsupported mmap flag detected! Only MAP_FIXED, MAP_SHARED, MAP_PRIVATE, MAP_POPULATE AND MAP_ANONYMOUS allowed

On adding further logs, I saw:

mmap: flags=0x54040021, unknown=0x54040000

0x54040000 is being passed to mmap but I am unsure how is this garbage value being passed (not a valid mmap flag)

2. If I remove the debug panic for unsupported flags and still run initdb, I see

    $ sudo /home/lind/lind-wasm/build/lind-boot /bin/initdb.wasm -D           
  /tmp/pgdata --no-locale                                                     
    selecting default "max_connections" ...                                   
    shell-init: error retrieving current directory: getcwd: cannot access     
  parent directories: Invalid argument                                                                                         
                                                                              
    thread 'lind-fork-2' panicked at lind-multi-process/src/lib.rs:952:50:    
    exec-ed module error: failed to run main module                           
                                                                              
    Caused by:                                                                
        0: failed to invoke command default                                   
        1: wasm trap: indirect call type mismatch                             
                                                  

There were couple of roadblocks I had to solve to reach here, all of those are documented in branch: https://github.com/Lind-Project/lind-wasm/tree/test/mkfifo and and https://github.com/Lind-Project/lind-wasm-apps/tree/compile-postgres-bkp

[vidyalakshmir]

Thanks Rishabh. This is a good progress.
@qianxichen233 Do you have any pointers to fix above errors?

[rishabhBudhouliya]

Currently, the postgres effort is blocked on: Lind-Project/lind-wasm#923

[rishabhBudhouliya]

Currently, we've moved to a different problem while running postgres in lind-wasm

ind@93213ae1641b:~/lind-wasm-apps$ sudo /home/lind/lind-wasm/build/lind-boot /bin/initdb.wasm -D /tmp/pgdata --no-locale
shell-init: error retrieving current directory: getcwd: cannot access parent directories: Invalid argument
The files belonging to this database system will be owned by user "root".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".

Data page checksums are enabled.

creating directory /tmp/pgdata ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... sysv
selecting default "max_connections" ... shell-init: error retrieving current directory: getcwd: cannot access parent directories: Invalid argument

thread 'lind-fork-4' (321165) panicked at /home/lind/lind-wasm/src/wasmtime/crates/sysdefs/src/logging.rs:6:5:
LIND DEBUG PANIC: Unsupported mmap flag detected! Only MAP_FIXED, MAP_SHARED, MAP_PRIVATE, MAP_POPULATE AND MAP_ANONYMOUS allowed
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread 'lind-fork-4' (321165) panicked at /rustc/7057231bd78d6c7893f905ea1832365d4c5efe17/library/core/src/panicking.rs:225:5:
panic in a function that cannot unwind
stack backtrace:
   0:     0x640245a04d2a - <unknown>
   1:     0x640245a1db3a - <unknown>

An unknown MMAP flag is being passed, I need to debug what that flag is and its significance.

[rishabhBudhouliya]

@rennergade @vidyalakshmir Currently, postgres's initdb panics because a garbage mmap flag arrives at RawPOSIX.
The corruption is happening between the C mmap() call (invoked by postgres) and the glibc syscall trampoline.

I am unsure what is corrupting the mmap flag. Any pointers on which direction I should debug this?