Skip to content

add static build CI tests#1073

Open
qianxichen233 wants to merge 2 commits intomainfrom
fix-1040
Open

add static build CI tests#1073
qianxichen233 wants to merge 2 commits intomainfrom
fix-1040

Conversation

@qianxichen233
Copy link
Copy Markdown
Contributor

@qianxichen233 qianxichen233 commented Apr 15, 2026

fix #1040

  • Add static_tests category under tests/unit-tests/ with deterministic tests for fork, threads, and TLS to cover static WASM builds
  • Add statictestreport harness that runs wasmtestreport with --static and -pthread flags; results are included in the combined report.html without generating a separate artifact file
  • Add --static flag to wasmtestreport.py (inserts --static before the source file in lind_compile invocations)
  • Refactor test_runner.py so harnesses can contribute HTML to the combined report.html purely in memory, without requiring a separate html_filename artifact

- Add static_tests category under tests/unit-tests/ with deterministic
  tests for fork, threads, and TLS to cover static WASM builds
- Add statictestreport harness that runs wasmtestreport with --static
  and -pthread flags; results are included in the combined report.html
  without generating a separate artifact file
- Add --static flag to wasmtestreport.py (inserts --static before the
  source file in lind_compile invocations)
- Refactor test_runner.py so harnesses can contribute HTML to the
  combined report.html purely in memory, without requiring a separate
  html_filename artifact
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we rename this file to dynamic related?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test runner invokes lind_compile with no additional argument, so this is essentially doing the test with default configuration in lind_compile. If default configuration in lind_compile is changed, then what this test runner is testing is also changed. Therefore, I'd like to keep the name

@github-actions
Copy link
Copy Markdown
Contributor

End-to-End Test Report

Test Preview

Unified Test Report

grate harness

MetricValue
Total11
Success11
Failures0
Compile Failures0
Runtime Failures0
Timeout Failures0
Missing Pair Failures0

Cases

TestStatusError TypeOutput
concurrent-request/geteuid_grate.cSuccess
STDOUT:
[Grate|geteuid] Registering geteuid handler for cage 2 in grate 1 with fn ptr addr: 2
[Cage | geteuid] PASS: 1000000 calls returned 10
[Grate|geteuid] PASS

STDERR:

interposing-calls/interpose-exec_grate.cSuccess
STDOUT:
[Grate|interpose-exec] Registering exec handler for cage 2 in grate 1 with fn ptr addr: 2
[Grate|interpose-exec] Handling function ptr: 2 from cage: 1
[Grate|interpose-exec] In exec_grate 1 handler for cage: 1
[Grate|interpose-exec] Handling function ptr: 2 from cage: 1
[Grate|interpose-exec] In exec_grate 1 handler for cage: 1
Exec successful, argv[1]: --execd
[Grate|interpose-exec] PASS

STDERR:

interposing-calls/interpose-exit_grate.cSuccess
STDOUT:
[Grate|interpose-exit] Registering exit handler for cage 2 in grate 1 with fn ptr addr: 2
Exiting...

[Grate|interpose-exit] PASS

STDERR:

interposing-calls/interpose-fork_grate.cSuccess
STDOUT:
[Grate|interpose-fork] Registering fork handler for cage 2 in grate 1 with fn ptr addr: 2
[Grate|interpose-fork] Handling function ptr: 2 from cage: 1
[Grate|interpose-fork] In fork_grate 1 handler for cage: 1
[Grate|interpose-fork] PASS

STDERR:

interposing-calls/interpose-mmap_grate.cSuccess
STDOUT:
[Grate|interpose-mmap] Registering mmap handler for cage 2 in grate 1 with fn ptr addr: 2
[Grate|interpose-mmap] Handling function ptr: 2 from cage: 1
[Grate|interpose-mmap] In mmap_grate 1 handler for cage: 1
[Grate|interpose-mmap] Handling function ptr: 2 from cage: 1
[Grate|interpose-mmap] In mmap_grate 1 handler for cage: 1
[Grate|interpose-mmap] Handling function ptr: 2 from cage: 1
[Grate|interpose-mmap] In mmap_grate 1 handler for cage: 1
[Grate|interpose-mmap] Handling function ptr: 2 from cage: 1
[Grate|interpose-mmap] In mmap_grate 1 handler for cage: 1
mmap test: PASS
[Grate|interpose-mmap] PASS

STDERR:

interposing-calls/interpose-register_grate.cSuccess
STDOUT:
[Grate|interpose-register] Registering register_handler for cage 2 in grate 1 with fn ptr addr: 3
[cage] registering 107. grateid: 2 cageid: 3
[Grate|interpose-register] Handling function ptr: 3 from cage: 1
[Grate|interpose-register] In register_grate 1 handler for cage: 1
[Grate|geteuid] Registering geteuid handler for cage 1 in grate 1 with fn ptr addr: 2
[Grate|interpose-register] Handling function ptr: 2 from cage: 1
[Grate|interpose-register] In register_grate 1 handler for cage: 1
[Grate|interpose-register] PASS

STDERR:

multi-register_grate.cSuccess
STDOUT:
[Grate|multi-register_grate] Registering geteuid handler for cage 2 in grate 1 with fn ptr addr: 2
[Grate|multi-register_grate] Registering getuid handler for cage 2 in grate 1 with fn ptr addr: 3
[Grate|multi-register_grate] Handling function ptr: 2 from cage: 1
[Grate|multi-register_grate] In multi-register_grate 1 handler for cage: 1
[Grate|multi-register_grate] Handling function ptr: 3 from cage: 1
[Grate|multi-register_grate] In multi-register_grate 1 handler for cage: 1
[Cage | multi-register] PASS: geteuid=10, getuid=20
[Grate|multi-register] PASS

STDERR:

race-test_grate.cSuccess
STDOUT:
pass

STDERR:

simple-tests/cpdata_grate.cSuccess
STDOUT:
[Grate|open] intercepts open call: thiscage=1, arg1cage=2
[Grate|open] copied pathname: random
[cage] fd=10

STDERR:

simple-tests/diff-cage-args_grate.cSuccess
STDOUT:
[Grate|diff-cage-args] Handling function ptr: 2 from cage: 1
[Grate|diff-cage-args] In open_grate 1 handler for cage: 1
Hello world. FD=-1
[Grate|diff-cage-args] Handling function ptr: 3 from cage: 1
Goodbye world! ret=4321 buf=helloworld
[Grate|diff-cage-args] PASS

STDERR:

simple-tests/geteuid_grate.cSuccess
STDOUT:
[Grate|geteuid] Registering geteuid handler for cage 2 in grate 1 with fn ptr addr: 2
[Grate|geteuid] Handling function ptr: 2 from cage: 1
[Grate|geteuid] In geteuid_grate 1 handler for cage: 1
[Cage | geteuid] PASS: geteuid ret = 10
[Grate|geteuid] PASS

STDERR:

static harness

Test Report

Deterministic Tests

Summary

MetricCount
Total Test Cases3
Number of Successes3
Number of Failures0
Number of Compilation Failure Native0
Number of Runtime Failure Native0
Number of Segmentation Fault Native0
Number of Timeout During Native0
Number of Lind Wasm Compile Failure0
Number of Lind Wasm Runtime Failure0
Number of Lind Wasm Segmentation Failure0
Number of Timeout During Lind Wasm run0
Number of Unknown Failure0
Number of C Compiler and Wasm Output mismatch0
Number of Fail Test: Native Succeeded (Should Fail)0
Number of Fail Test: Wasm Succeeded (Should Fail)0
Number of Fail Test: Both Native and Wasm Succeeded (Should Fail)0
Number of Fail Test: Native Compilation Failure (Should Succeed)0
Number of Fail Test: Wasm Compilation Failure (Should Succeed)0

Test Results by Category

Test CaseStatusError TypeNative TimeWasm TimeOutput
Static Tests
fork_simple.cSuccessNone0.052677s4.358553s
Success
thread.cSuccessNone0.050816s4.482568s
Success
tls_test.cSuccessNone0.055605s4.471051s
Success
Fail Tests

Summary

MetricCount
Total Test Cases0
Number of Successes0
Number of Failures0
Number of Compilation Failure Native0
Number of Runtime Failure Native0
Number of Segmentation Fault Native0
Number of Timeout During Native0
Number of Lind Wasm Compile Failure0
Number of Lind Wasm Runtime Failure0
Number of Lind Wasm Segmentation Failure0
Number of Timeout During Lind Wasm run0
Number of Unknown Failure0
Number of C Compiler and Wasm Output mismatch0
Number of Fail Test: Native Succeeded (Should Fail)0
Number of Fail Test: Wasm Succeeded (Should Fail)0
Number of Fail Test: Both Native and Wasm Succeeded (Should Fail)0
Number of Fail Test: Native Compilation Failure (Should Succeed)0
Number of Fail Test: Wasm Compilation Failure (Should Succeed)0

wasm harness

Test Report

Deterministic Tests

Summary

MetricCount
Total Test Cases202
Number of Successes202
Number of Failures0
Number of Compilation Failure Native0
Number of Runtime Failure Native0
Number of Segmentation Fault Native0
Number of Timeout During Native0
Number of Lind Wasm Compile Failure0
Number of Lind Wasm Runtime Failure0
Number of Lind Wasm Segmentation Failure0
Number of Timeout During Lind Wasm run0
Number of Unknown Failure0
Number of C Compiler and Wasm Output mismatch0
Number of Fail Test: Native Succeeded (Should Fail)0
Number of Fail Test: Wasm Succeeded (Should Fail)0
Number of Fail Test: Both Native and Wasm Succeeded (Should Fail)0
Number of Fail Test: Native Compilation Failure (Should Succeed)0
Number of Fail Test: Wasm Compilation Failure (Should Succeed)0

Test Results by Category

Test CaseStatusError TypeNative TimeWasm TimeOutput
File Tests
chartests.cSuccessNone0.047389s0.188210s
Success
chdir_getcwd.cSuccessNone0.048864s0.150088s
Success
chmod.cSuccessNone0.053089s0.156397s
Success
clock_gettime_highlevel.cSuccessNone0.119171s0.319593s
Success
clock_gettime_simple.cSuccessNone0.042059s0.131063s
Success
cloexec.cSuccessNone0.052903s0.220009s
Success
close.cSuccessNone0.059652s0.169667s
Success
creat_access.cSuccessNone0.050952s0.147564s
Success
doubleclose.cSuccessNone0.041998s0.126271s
Success
dup.cSuccessNone0.043681s0.149808s
Success
dup2.cSuccessNone0.049247s0.147312s
Success
dup3.cSuccessNone0.046639s0.150971s
Success
dupwrite.cSuccessNone0.049712s0.140898s
Success
etc_conf.cSuccessNone0.045772s0.153371s
Success
fchdir.cSuccessNone0.052852s0.162859s
Success
fchmod.cSuccessNone0.052741s0.158889s
Success
fcntl.cSuccessNone0.050433s0.153759s
Success
fdatasync.cSuccessNone0.048949s0.137247s
Success
filetest.cSuccessNone0.049120s0.143797s
Success
filetest1000.cSuccessNone0.058863s0.155460s
Success
flock.cSuccessNone0.059307s0.189345s
Success
fstat.cSuccessNone0.052301s0.157144s
Success
fstatfs.cSuccessNone0.044229s0.135279s
Success
fsync.cSuccessNone0.049973s0.140996s
Success
ftruncate.cSuccessNone0.057967s0.280190s
Success
getcwd.cSuccessNone0.047649s0.137616s
Success
getrandom.cSuccessNone0.050349s0.150030s
Success
ioctl.cSuccessNone0.053401s0.147321s
Success
link.cSuccessNone0.053440s0.207747s
Success
locale_test.cSuccessNone0.064092s0.643993s
Success
lseek.cSuccessNone0.054848s0.267082s
Success
mkdir_rmdir.cSuccessNone0.050395s0.149314s
Success
mkfifo_test.cSuccessNone0.056439s0.194918s
Success
mknod.cSuccessNone0.049425s0.156912s
Success
nocancel_io.cSuccessNone0.054297s0.175619s
Success
open.cSuccessNone0.046173s0.131722s
Success
openat.cSuccessNone0.045513s0.141701s
Success
path_conversion_safety.cSuccessNone0.054021s0.180776s
Success
pread_pwrite.cSuccessNone0.048339s0.158743s
Success
preadv_pwritev.cSuccessNone0.052512s0.170477s
Success
printf.cSuccessNone0.041214s0.123555s
Success
prlimit64.cSuccessNone0.043989s0.131525s
Success
read.cSuccessNone0.051744s0.149948s
Success
readbytes.cSuccessNone0.046965s0.135684s
Success
readdir_basic.cSuccessNone0.057658s0.181398s
Success
readlink.cSuccessNone0.050910s0.146345s
Success
readlinkat.cSuccessNone0.052420s0.159019s
Success
readv_writev_test.cSuccessNone0.051848s0.163796s
Success
rename.cSuccessNone0.052377s0.149715s
Success
sc-writev.cSuccessNone0.048073s0.142575s
Success
stat.cSuccessNone0.054012s0.151835s
Success
statfs.cSuccessNone0.045002s0.137469s
Success
sync_file_range.cSuccessNone0.047610s0.142396s
Success
timespec_time_t_compat.cSuccessNone0.048018s0.137508s
Success
truncate.cSuccessNone0.051460s0.163643s
Success
unlink.cSuccessNone0.052445s0.197818s
Success
unlinkat.cSuccessNone0.052628s0.166338s
Success
write.cSuccessNone0.043480s0.128614s
Success
writeloop.cSuccessNone0.054077s0.141291s
Success
writepartial.cSuccessNone0.050577s0.133191s
Success
writev.cSuccessNone0.053148s0.154701s
Success
Math Tests
math_link_smoke.cSuccessNone0.054100s0.138074s
Success
math_tests.cSuccessNone0.060065s0.169215s
Success
Memory Tests
brk.cSuccessNone0.047968s0.140787s
Success
fork_large_memory.cSuccessNone0.080244s0.419978s
Success
malloc.cSuccessNone0.046809s0.128826s
Success
malloc_large.cSuccessNone0.047392s0.133705s
Success
memcpy.cSuccessNone0.045616s0.133069s
Success
memory_error_test.cSuccessNone0.053371s0.173374s
Success
mmap.cSuccessNone0.042446s0.133453s
Success
mmap_aligned.cSuccessNone0.044409s0.147739s
Success
mmap_complicated.cSuccessNone0.051518s0.158931s
Success
mmap_file.cSuccessNone0.051508s0.152262s
Success
mmap_shared.cSuccessNone0.050701s0.155435s
Success
mmaptest.cSuccessNone0.047344s0.147350s
Success
mprotect.cSuccessNone0.044557s0.134528s
Success
mprotect_boundary.cSuccessNone0.044144s0.149527s
Success
mprotect_end_region.cSuccessNone0.043688s0.143356s
Success
mprotect_middle_region.cSuccessNone0.043806s0.144276s
Success
mprotect_multiple_times.cSuccessNone0.043755s0.144472s
Success
mprotect_same_value.cSuccessNone0.043690s0.142244s
Success
mprotect_spanning_regions.cSuccessNone0.045075s0.159019s
Success
sbrk.cSuccessNone0.047226s0.140336s
Success
segfault.cSuccessNone0.052538s0.171375s
Success
shm.cSuccessNone0.050012s0.158818s
Success
shmtest.cSuccessNone0.044405s0.137539s
Success
thread_malloc_sequential.cSuccessNone0.052379s0.172497s
Success
vtable.cSuccessNone0.056074s0.160282s
Success
Networking Tests
accept4.cSuccessNone0.056185s0.170892s
Success
dns_resolve_test.cSuccessNone0.049687s0.147025s
Success
dnstest.cSuccessNone0.051116s0.154000s
Success
epoll_edge_triggered.cSuccessNone0.208567s0.499376s
Success
epollcreate1.cSuccessNone0.052056s0.156368s
Success
error_handling_net.cSuccessNone0.058438s0.266774s
Success
getaddrinfo_test.cSuccessNone0.055903s0.192788s
Success
getaddrinfo_unspec.cSuccessNone0.056579s0.189779s
Success
gethostname.cSuccessNone0.045018s0.129539s
Success
getifaddrs.cSuccessNone0.052249s0.157331s
Success
getsockname.cSuccessNone0.056975s0.152674s
Success
getsockopt.cSuccessNone0.054781s0.204338s
Success
ipv6_basic.cSuccessNone0.057055s0.207202s
Success
makepipe.cSuccessNone0.044671s0.126454s
Success
nonblocking_eagain.cSuccessNone0.056269s0.214061s
Success
pipe.cSuccessNone0.055388s0.162514s
Success
pipe2.cSuccessNone0.052987s0.153026s
Success
pipeinput.cSuccessNone0.054712s0.172702s
Success
pipeinput2.cSuccessNone0.054233s0.172014s
Success
pipeonestring.cSuccessNone0.055069s0.173004s
Success
pipepong.cSuccessNone0.055799s0.176158s
Success
pipewrite.cSuccessNone0.047802s0.148871s
Success
poll.cSuccessNone0.051903s0.145781s
Success
recvfrom-sendto.cSuccessNone0.052571s0.162441s
Success
sendmsg_recvmsg_test.cSuccessNone0.053069s0.161372s
Success
serverclient.cSuccessNone0.053276s0.156472s
Success
shutdown.cSuccessNone0.055237s0.158268s
Success
shutdown_fork.cSuccessNone0.052174s0.162674s
Success
simple-select.cSuccessNone0.053665s0.172999s
Success
simple_epoll.cSuccessNone0.052103s0.154899s
Success
socket.cSuccessNone0.050007s0.142118s
Success
socket_cloexec.cSuccessNone0.050531s0.141294s
Success
socket_options_advanced.cSuccessNone0.057748s0.225049s
Success
socketepoll.cSuccessNone0.050552s0.148443s
Success
socketpair.cSuccessNone0.050794s0.156844s
Success
socketselect.cSuccessNone0.051241s0.150606s
Success
udp_send_recv.cSuccessNone0.158156s0.321057s
Success
uds-getsockname.cSuccessNone0.051844s0.151830s
Success
uds-nb-select.cSuccessNone2.062928s2.224261s
Success
uds-serverclient.cSuccessNone0.056413s0.187288s
Success
uds-socketselect.cSuccessNone0.052352s0.157888s
Success
writev_socket.cSuccessNone0.055442s0.194419s
Success
Process Tests
barrier_test.cSuccessNone0.050208s0.158492s
Success
chain_thread.cSuccessNone1.053877s1.158114s
Success
ctor_syscall_test.cSuccessNone0.040987s0.127967s
Success
cxa_atexit_test.cSuccessNone0.047463s0.132399s
Success
execve_shebang.cSuccessNone0.051376s0.149847s
Success
exit.cSuccessNone0.048230s0.132272s
Success
exit_failure.cSuccessNone0.051045s0.151600s
Success
exit_group_thread.cSuccessNone0.053310s0.157639s
Success
flockfile_test.cSuccessNone0.050749s0.169797s
Success
fork2malloc.cSuccessNone0.051992s0.153134s
Success
fork_select.cSuccessNone0.050105s0.160482s
Success
fork_simple.cSuccessNone0.048432s0.143622s
Success
fork_syscall.cSuccessNone0.055071s0.322452s
Success
fork_tls_ctype.cSuccessNone0.054137s0.177686s
Success
forkandopen.cSuccessNone0.053020s0.173679s
Success
forkdup.cSuccessNone0.055409s0.168296s
Success
forkexecuid.cSuccessNone0.050779s0.225548s
Success
forkexecv-arg.cSuccessNone0.051185s0.210580s
Success
forkexecv.cSuccessNone0.047979s0.199339s
Success
forkfiles.cSuccessNone0.052166s0.165813s
Success
forkmalloc.cSuccessNone0.052443s0.152231s
Success
forknodup.cSuccessNone0.052048s0.164068s
Success
function-ptr.cSuccessNone0.046201s0.134262s
Success
getegid_syscall.cSuccessNone0.051328s0.283341s
Success
getgid_syscall.cSuccessNone0.051544s0.293738s
Success
getpid.cSuccessNone0.044715s0.130208s
Success
getpid_syscall.cSuccessNone0.053266s0.306864s
Success
getppid.cSuccessNone0.054621s0.164247s
Success
getppid_syscall.cSuccessNone0.054102s0.255817s
Success
getuid.cSuccessNone0.051539s0.140738s
Success
getuid_syscall.cSuccessNone0.053415s0.196270s
Success
hello-arg.cSuccessNone0.041270s0.132897s
Success
hello.cSuccessNone0.041974s0.124980s
Success
longjmp.cSuccessNone0.042602s0.133849s
Success
mutex.cSuccessNone2.055002s2.161718s
Success
printf_deadlock_smoke.cSuccessNone0.060505s0.197302s
Success
printf_thread_test.cSuccessNone0.049883s0.165164s
Success
sem_forks.cSuccessNone0.057759s0.168447s
Success
setsid.cSuccessNone0.043815s0.130855s
Success
template.cSuccessNone0.050962s0.171352s
Success
test_exec_nofork.cSuccessNone0.049484s0.196322s
Success
test_unlink_open_file.cSuccessNone0.045632s0.130023s
Success
thread-test.cSuccessNone0.046174s0.140403s
Success
thread.cSuccessNone0.045896s0.138541s
Success
thread_cageid_race.cSuccessNone0.046753s0.198094s
Success
tls_test.cSuccessNone0.050321s0.166803s
Success
uname.cSuccessNone0.044792s0.129914s
Success
wait.cSuccessNone2.048445s2.154062s
Success
waitpid_anychild.cSuccessNone0.050659s0.151886s
Success
waitpid_syscall.cSuccessNone1.051631s1.192118s
Success
waitpid_wnohang.cSuccessNone0.052394s0.150949s
Success
Signal Tests
alarm.cSuccessNone7.050325s7.171142s
Success
eintr_fork_signal.cSuccessNone1.054700s1.174627s
Success
kill.cSuccessNone1.050541s1.152909s
Success
setitimer.cSuccessNone7.051204s7.179164s
Success
sigalrm.cSuccessNone2.051496s2.163621s
Success
sigaltstack.cSuccessNone0.054133s0.159421s
Success
sigchld.cSuccessNone1.054269s1.156043s
Success
signal-fork.cSuccessNone4.052773s4.158208s
Success
signal-simple.cSuccessNone0.052352s0.150115s
Success
signal_SIGCHLD.cSuccessNone0.049217s0.154793s
Success
signal_fork.cSuccessNone0.047610s0.160506s
Success
signal_int_ignored.cSuccessNone2.052320s2.161631s
Success
signal_kill_cleanup.cSuccessNone1.049240s1.152234s
Success
signal_procmask.cSuccessNone0.047335s0.146990s
Success
signal_recursive.cSuccessNone0.045710s0.144551s
Success
signal_sa_mask.cSuccessNone0.045605s0.137075s
Success
sigpipe.cSuccessNone1.053534s1.178902s
Success
sigprocmask.cSuccessNone1.050585s1.156893s
Success
Static Tests
fork_simple.cSuccessNone0.048852s0.143737s
Success
thread.cSuccessNone0.047791s0.140725s
Success
tls_test.cSuccessNone0.050217s0.163777s
Success
Fail Tests

Summary

MetricCount
Total Test Cases4
Number of Successes4
Number of Failures0
Number of Compilation Failure Native0
Number of Runtime Failure Native0
Number of Segmentation Fault Native0
Number of Timeout During Native0
Number of Lind Wasm Compile Failure0
Number of Lind Wasm Runtime Failure0
Number of Lind Wasm Segmentation Failure0
Number of Timeout During Lind Wasm run0
Number of Unknown Failure0
Number of C Compiler and Wasm Output mismatch0
Number of Fail Test: Native Succeeded (Should Fail)0
Number of Fail Test: Wasm Succeeded (Should Fail)0
Number of Fail Test: Both Native and Wasm Succeeded (Should Fail)0
Number of Fail Test: Native Compilation Failure (Should Succeed)0
Number of Fail Test: Wasm Compilation Failure (Should Succeed)0

Test Results by Category

Test CaseStatusError TypeNative TimeWasm TimeOutput
Dylink Tests
dlerror.cSuccessNone0.043666s0.131390s
Success
Memory Tests
mmap-negative1.cSuccessNone0.138832s0.132979s
Success
mmap-negative2.cSuccessNone0.112268s0.175671s
Success
Signal Tests
signal_resethand.cSuccessNone1.050369s1.157286s
Success

# --allow-pre-compiled : use .cwasm AOT binaries (consistent with dynamic harness)
# --compile-flags -pthread -lpthread : link pthread for thread/TLS tests
_STATIC_HARNESS_ARGS = [
"--run", "static_tests",
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this harness now owns static_tests, should we also exclude static_tests from the default wasmtestreport run? As written, I believe the normal wasm harness still discovers all tests/unit-tests/**/*.c, so these tests look like they’ll run once as dynamic-build tests and again here as static-build tests. Would it make sense to add a small guard near the existing if module_name == "wasmtestreport": block, perhaps appending a flag?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just resolved this, could you take a look again

Exclude static_tests from the wasmtestreport harness so they are not
run twice as dynamic-build tests (they are owned by statictestreport).

Also handle PermissionError in wasmtestreport's testfiles cleanup so a
permission issue on lindfs/testfiles does not crash the subprocess and
cause statictestreport to abort before writing the combined report.
@github-actions
Copy link
Copy Markdown
Contributor

End-to-End Test Report

Test Preview

Unified Test Report

grate harness

MetricValue
Total11
Success11
Failures0
Compile Failures0
Runtime Failures0
Timeout Failures0
Missing Pair Failures0

Cases

TestStatusError TypeOutput
concurrent-request/geteuid_grate.cSuccess
STDOUT:
[Grate|geteuid] Registering geteuid handler for cage 2 in grate 1 with fn ptr addr: 2
[Cage | geteuid] PASS: 1000000 calls returned 10
[Grate|geteuid] PASS

STDERR:

interposing-calls/interpose-exec_grate.cSuccess
STDOUT:
[Grate|interpose-exec] Registering exec handler for cage 2 in grate 1 with fn ptr addr: 2
[Grate|interpose-exec] Handling function ptr: 2 from cage: 1
[Grate|interpose-exec] In exec_grate 1 handler for cage: 1
[Grate|interpose-exec] Handling function ptr: 2 from cage: 1
[Grate|interpose-exec] In exec_grate 1 handler for cage: 1
Exec successful, argv[1]: --execd
[Grate|interpose-exec] PASS

STDERR:

interposing-calls/interpose-exit_grate.cSuccess
STDOUT:
[Grate|interpose-exit] Registering exit handler for cage 2 in grate 1 with fn ptr addr: 2
Exiting...

[Grate|interpose-exit] PASS

STDERR:

interposing-calls/interpose-fork_grate.cSuccess
STDOUT:
[Grate|interpose-fork] Registering fork handler for cage 2 in grate 1 with fn ptr addr: 2
[Grate|interpose-fork] Handling function ptr: 2 from cage: 1
[Grate|interpose-fork] In fork_grate 1 handler for cage: 1
[Grate|interpose-fork] PASS

STDERR:

interposing-calls/interpose-mmap_grate.cSuccess
STDOUT:
[Grate|interpose-mmap] Registering mmap handler for cage 2 in grate 1 with fn ptr addr: 2
[Grate|interpose-mmap] Handling function ptr: 2 from cage: 1
[Grate|interpose-mmap] In mmap_grate 1 handler for cage: 1
[Grate|interpose-mmap] Handling function ptr: 2 from cage: 1
[Grate|interpose-mmap] In mmap_grate 1 handler for cage: 1
[Grate|interpose-mmap] Handling function ptr: 2 from cage: 1
[Grate|interpose-mmap] In mmap_grate 1 handler for cage: 1
[Grate|interpose-mmap] Handling function ptr: 2 from cage: 1
[Grate|interpose-mmap] In mmap_grate 1 handler for cage: 1
[Grate|interpose-mmap] Handling function ptr: 2 from cage: 1
[Grate|interpose-mmap] In mmap_grate 1 handler for cage: 1
mmap test: PASS
[Grate|interpose-mmap] PASS

STDERR:

interposing-calls/interpose-register_grate.cSuccess
STDOUT:
[Grate|interpose-register] Registering register_handler for cage 2 in grate 1 with fn ptr addr: 3
[cage] registering 107. grateid: 2 cageid: 3
[Grate|interpose-register] Handling function ptr: 3 from cage: 1
[Grate|interpose-register] In register_grate 1 handler for cage: 1
[Grate|geteuid] Registering geteuid handler for cage 1 in grate 1 with fn ptr addr: 2
[Grate|interpose-register] Handling function ptr: 2 from cage: 1
[Grate|interpose-register] In register_grate 1 handler for cage: 1
[Grate|interpose-register] PASS

STDERR:

multi-register_grate.cSuccess
STDOUT:
[Grate|multi-register_grate] Registering geteuid handler for cage 2 in grate 1 with fn ptr addr: 2
[Grate|multi-register_grate] Registering getuid handler for cage 2 in grate 1 with fn ptr addr: 3
[Grate|multi-register_grate] Handling function ptr: 2 from cage: 1
[Grate|multi-register_grate] In multi-register_grate 1 handler for cage: 1
[Grate|multi-register_grate] Handling function ptr: 3 from cage: 1
[Grate|multi-register_grate] In multi-register_grate 1 handler for cage: 1
[Cage | multi-register] PASS: geteuid=10, getuid=20
[Grate|multi-register] PASS

STDERR:

race-test_grate.cSuccess
STDOUT:
pass

STDERR:

simple-tests/cpdata_grate.cSuccess
STDOUT:
[Grate|open] intercepts open call: thiscage=1, arg1cage=2
[Grate|open] copied pathname: random
[cage] fd=10

STDERR:

simple-tests/diff-cage-args_grate.cSuccess
STDOUT:
[Grate|diff-cage-args] Handling function ptr: 2 from cage: 1
[Grate|diff-cage-args] In open_grate 1 handler for cage: 1
Hello world. FD=-1
[Grate|diff-cage-args] Handling function ptr: 3 from cage: 1
Goodbye world! ret=4321 buf=helloworld
[Grate|diff-cage-args] PASS

STDERR:

simple-tests/geteuid_grate.cSuccess
STDOUT:
[Grate|geteuid] Registering geteuid handler for cage 2 in grate 1 with fn ptr addr: 2
[Grate|geteuid] Handling function ptr: 2 from cage: 1
[Grate|geteuid] In geteuid_grate 1 handler for cage: 1
[Cage | geteuid] PASS: geteuid ret = 10
[Grate|geteuid] PASS

STDERR:

static harness

Test Report

Deterministic Tests

Summary

MetricCount
Total Test Cases3
Number of Successes3
Number of Failures0
Number of Compilation Failure Native0
Number of Runtime Failure Native0
Number of Segmentation Fault Native0
Number of Timeout During Native0
Number of Lind Wasm Compile Failure0
Number of Lind Wasm Runtime Failure0
Number of Lind Wasm Segmentation Failure0
Number of Timeout During Lind Wasm run0
Number of Unknown Failure0
Number of C Compiler and Wasm Output mismatch0
Number of Fail Test: Native Succeeded (Should Fail)0
Number of Fail Test: Wasm Succeeded (Should Fail)0
Number of Fail Test: Both Native and Wasm Succeeded (Should Fail)0
Number of Fail Test: Native Compilation Failure (Should Succeed)0
Number of Fail Test: Wasm Compilation Failure (Should Succeed)0

Test Results by Category

Test CaseStatusError TypeNative TimeWasm TimeOutput
Static Tests
fork_simple.cSuccessNone0.052807s4.336295s
Success
thread.cSuccessNone0.045315s4.412763s
Success
tls_test.cSuccessNone0.059582s4.453918s
Success
Fail Tests

Summary

MetricCount
Total Test Cases0
Number of Successes0
Number of Failures0
Number of Compilation Failure Native0
Number of Runtime Failure Native0
Number of Segmentation Fault Native0
Number of Timeout During Native0
Number of Lind Wasm Compile Failure0
Number of Lind Wasm Runtime Failure0
Number of Lind Wasm Segmentation Failure0
Number of Timeout During Lind Wasm run0
Number of Unknown Failure0
Number of C Compiler and Wasm Output mismatch0
Number of Fail Test: Native Succeeded (Should Fail)0
Number of Fail Test: Wasm Succeeded (Should Fail)0
Number of Fail Test: Both Native and Wasm Succeeded (Should Fail)0
Number of Fail Test: Native Compilation Failure (Should Succeed)0
Number of Fail Test: Wasm Compilation Failure (Should Succeed)0

wasm harness

Test Report

Deterministic Tests

Summary

MetricCount
Total Test Cases199
Number of Successes199
Number of Failures0
Number of Compilation Failure Native0
Number of Runtime Failure Native0
Number of Segmentation Fault Native0
Number of Timeout During Native0
Number of Lind Wasm Compile Failure0
Number of Lind Wasm Runtime Failure0
Number of Lind Wasm Segmentation Failure0
Number of Timeout During Lind Wasm run0
Number of Unknown Failure0
Number of C Compiler and Wasm Output mismatch0
Number of Fail Test: Native Succeeded (Should Fail)0
Number of Fail Test: Wasm Succeeded (Should Fail)0
Number of Fail Test: Both Native and Wasm Succeeded (Should Fail)0
Number of Fail Test: Native Compilation Failure (Should Succeed)0
Number of Fail Test: Wasm Compilation Failure (Should Succeed)0

Test Results by Category

Test CaseStatusError TypeNative TimeWasm TimeOutput
File Tests
chartests.cSuccessNone0.045461s0.183815s
Success
chdir_getcwd.cSuccessNone0.046393s0.142102s
Success
chmod.cSuccessNone0.052506s0.153589s
Success
clock_gettime_highlevel.cSuccessNone0.117331s0.317370s
Success
clock_gettime_simple.cSuccessNone0.043337s0.126985s
Success
cloexec.cSuccessNone0.050111s0.211651s
Success
close.cSuccessNone0.057379s0.165049s
Success
creat_access.cSuccessNone0.049228s0.143191s
Success
doubleclose.cSuccessNone0.041940s0.123009s
Success
dup.cSuccessNone0.042309s0.140581s
Success
dup2.cSuccessNone0.047473s0.141061s
Success
dup3.cSuccessNone0.046376s0.145330s
Success
dupwrite.cSuccessNone0.049180s0.139426s
Success
etc_conf.cSuccessNone0.044225s0.143921s
Success
fchdir.cSuccessNone0.052024s0.154884s
Success
fchmod.cSuccessNone0.050714s0.154079s
Success
fcntl.cSuccessNone0.048592s0.146295s
Success
fdatasync.cSuccessNone0.047729s0.133481s
Success
filetest.cSuccessNone0.048101s0.135757s
Success
filetest1000.cSuccessNone0.056374s0.148803s
Success
flock.cSuccessNone0.057552s0.179873s
Success
fstat.cSuccessNone0.050391s0.149388s
Success
fstatfs.cSuccessNone0.043922s0.132071s
Success
fsync.cSuccessNone0.047709s0.133749s
Success
ftruncate.cSuccessNone0.053298s0.263001s
Success
getcwd.cSuccessNone0.045119s0.133170s
Success
getrandom.cSuccessNone0.048467s0.142122s
Success
ioctl.cSuccessNone0.054103s0.142864s
Success
link.cSuccessNone0.052188s0.201088s
Success
locale_test.cSuccessNone0.062602s0.626816s
Success
lseek.cSuccessNone0.053201s0.259709s
Success
mkdir_rmdir.cSuccessNone0.048773s0.143525s
Success
mkfifo_test.cSuccessNone0.054171s0.189589s
Success
mknod.cSuccessNone0.048296s0.150159s
Success
nocancel_io.cSuccessNone0.052611s0.172639s
Success
open.cSuccessNone0.044000s0.128264s
Success
openat.cSuccessNone0.044094s0.135909s
Success
path_conversion_safety.cSuccessNone0.052985s0.170331s
Success
pread_pwrite.cSuccessNone0.046229s0.148644s
Success
preadv_pwritev.cSuccessNone0.051421s0.156521s
Success
printf.cSuccessNone0.040702s0.117497s
Success
prlimit64.cSuccessNone0.042290s0.128281s
Success
read.cSuccessNone0.049544s0.146912s
Success
readbytes.cSuccessNone0.045718s0.131438s
Success
readdir_basic.cSuccessNone0.053266s0.166351s
Success
readlink.cSuccessNone0.049172s0.144092s
Success
readlinkat.cSuccessNone0.050897s0.150325s
Success
readv_writev_test.cSuccessNone0.050624s0.154317s
Success
rename.cSuccessNone0.051378s0.143221s
Success
sc-writev.cSuccessNone0.046920s0.140734s
Success
stat.cSuccessNone0.049901s0.146218s
Success
statfs.cSuccessNone0.043091s0.132882s
Success
sync_file_range.cSuccessNone0.047117s0.140472s
Success
timespec_time_t_compat.cSuccessNone0.046020s0.131079s
Success
truncate.cSuccessNone0.050746s0.159480s
Success
unlink.cSuccessNone0.051971s0.194442s
Success
unlinkat.cSuccessNone0.051305s0.164144s
Success
write.cSuccessNone0.042360s0.122294s
Success
writeloop.cSuccessNone0.051163s0.133821s
Success
writepartial.cSuccessNone0.049383s0.130320s
Success
writev.cSuccessNone0.050547s0.151445s
Success
Math Tests
math_link_smoke.cSuccessNone0.052737s0.132482s
Success
math_tests.cSuccessNone0.059295s0.165237s
Success
Memory Tests
brk.cSuccessNone0.046835s0.140089s
Success
fork_large_memory.cSuccessNone0.078430s0.387730s
Success
malloc.cSuccessNone0.045721s0.127035s
Success
malloc_large.cSuccessNone0.045987s0.131408s
Success
memcpy.cSuccessNone0.044687s0.126639s
Success
memory_error_test.cSuccessNone0.051469s0.170327s
Success
mmap.cSuccessNone0.041450s0.131118s
Success
mmap_aligned.cSuccessNone0.043706s0.145265s
Success
mmap_complicated.cSuccessNone0.050046s0.151552s
Success
mmap_file.cSuccessNone0.049567s0.143794s
Success
mmap_shared.cSuccessNone0.048531s0.144665s
Success
mmaptest.cSuccessNone0.044784s0.137540s
Success
mprotect.cSuccessNone0.043160s0.131341s
Success
mprotect_boundary.cSuccessNone0.043173s0.144913s
Success
mprotect_end_region.cSuccessNone0.042396s0.135765s
Success
mprotect_middle_region.cSuccessNone0.042889s0.140040s
Success
mprotect_multiple_times.cSuccessNone0.043214s0.140251s
Success
mprotect_same_value.cSuccessNone0.042539s0.135465s
Success
mprotect_spanning_regions.cSuccessNone0.043745s0.154314s
Success
sbrk.cSuccessNone0.044614s0.129650s
Success
segfault.cSuccessNone0.050697s0.161260s
Success
shm.cSuccessNone0.049062s0.151325s
Success
shmtest.cSuccessNone0.043982s0.135916s
Success
thread_malloc_sequential.cSuccessNone0.050928s0.161500s
Success
vtable.cSuccessNone0.054461s0.149337s
Success
Networking Tests
accept4.cSuccessNone0.054698s0.163953s
Success
dns_resolve_test.cSuccessNone0.048746s0.140873s
Success
dnstest.cSuccessNone0.048892s0.148979s
Success
epoll_edge_triggered.cSuccessNone0.206852s0.485711s
Success
epollcreate1.cSuccessNone0.050894s0.151229s
Success
error_handling_net.cSuccessNone0.057637s0.258300s
Success
getaddrinfo_test.cSuccessNone0.052964s0.185550s
Success
getaddrinfo_unspec.cSuccessNone0.085340s0.154085s
Success
gethostname.cSuccessNone0.042688s0.124247s
Success
getifaddrs.cSuccessNone0.051393s0.145976s
Success
getsockname.cSuccessNone0.057101s0.148093s
Success
getsockopt.cSuccessNone0.053390s0.198905s
Success
ipv6_basic.cSuccessNone0.054720s0.197881s
Success
makepipe.cSuccessNone0.042785s0.122154s
Success
nonblocking_eagain.cSuccessNone0.054268s0.210588s
Success
pipe.cSuccessNone0.052474s0.153271s
Success
pipe2.cSuccessNone0.051142s0.143298s
Success
pipeinput.cSuccessNone0.052696s0.165007s
Success
pipeinput2.cSuccessNone0.053364s0.166558s
Success
pipeonestring.cSuccessNone0.052881s0.163985s
Success
pipepong.cSuccessNone0.051822s0.166558s
Success
pipewrite.cSuccessNone0.045999s0.143359s
Success
poll.cSuccessNone0.050171s0.136555s
Success
recvfrom-sendto.cSuccessNone0.051099s0.157139s
Success
sendmsg_recvmsg_test.cSuccessNone0.050851s0.151942s
Success
serverclient.cSuccessNone0.050714s0.147923s
Success
shutdown.cSuccessNone0.053321s0.149856s
Success
shutdown_fork.cSuccessNone0.051140s0.152257s
Success
simple-select.cSuccessNone0.052090s0.166312s
Success
simple_epoll.cSuccessNone0.051748s0.151410s
Success
socket.cSuccessNone0.048952s0.136496s
Success
socket_cloexec.cSuccessNone0.049596s0.135922s
Success
socket_options_advanced.cSuccessNone0.055860s0.217846s
Success
socketepoll.cSuccessNone0.049224s0.143578s
Success
socketpair.cSuccessNone0.048551s0.148383s
Success
socketselect.cSuccessNone0.049854s0.145173s
Success
udp_send_recv.cSuccessNone0.157296s0.309832s
Success
uds-getsockname.cSuccessNone0.050587s0.143743s
Success
uds-nb-select.cSuccessNone2.057791s2.217365s
Success
uds-serverclient.cSuccessNone0.055388s0.180565s
Success
uds-socketselect.cSuccessNone0.051279s0.148190s
Success
writev_socket.cSuccessNone0.053705s0.183521s
Success
Process Tests
barrier_test.cSuccessNone0.048543s0.149860s
Success
chain_thread.cSuccessNone1.051102s1.151209s
Success
ctor_syscall_test.cSuccessNone0.041090s0.122094s
Success
cxa_atexit_test.cSuccessNone0.049158s0.134733s
Success
execve_shebang.cSuccessNone0.050295s0.145296s
Success
exit.cSuccessNone0.047736s0.127754s
Success
exit_failure.cSuccessNone0.049285s0.138617s
Success
exit_group_thread.cSuccessNone0.051088s0.149451s
Success
flockfile_test.cSuccessNone0.049862s0.162122s
Success
fork2malloc.cSuccessNone0.053103s0.148440s
Success
fork_select.cSuccessNone0.049111s0.152983s
Success
fork_simple.cSuccessNone0.047693s0.139287s
Success
fork_syscall.cSuccessNone0.054177s0.286077s
Success
fork_tls_ctype.cSuccessNone0.052246s0.171301s
Success
forkandopen.cSuccessNone0.051280s0.163054s
Success
forkdup.cSuccessNone0.056343s0.156543s
Success
forkexecuid.cSuccessNone0.049551s0.217732s
Success
forkexecv-arg.cSuccessNone0.048702s0.199640s
Success
forkexecv.cSuccessNone0.046848s0.193129s
Success
forkfiles.cSuccessNone0.050939s0.157818s
Success
forkmalloc.cSuccessNone0.051424s0.143092s
Success
forknodup.cSuccessNone0.051083s0.157676s
Success
function-ptr.cSuccessNone0.044654s0.130637s
Success
getegid_syscall.cSuccessNone0.050347s0.250849s
Success
getgid_syscall.cSuccessNone0.050899s0.256394s
Success
getpid.cSuccessNone0.043011s0.125714s
Success
getpid_syscall.cSuccessNone0.051641s0.266042s
Success
getppid.cSuccessNone0.050923s0.146168s
Success
getppid_syscall.cSuccessNone0.054190s0.227546s
Success
getuid.cSuccessNone0.050773s0.138597s
Success
getuid_syscall.cSuccessNone0.050066s0.182289s
Success
hello-arg.cSuccessNone0.041985s0.129017s
Success
hello.cSuccessNone0.041176s0.121613s
Success
longjmp.cSuccessNone0.042114s0.129132s
Success
mutex.cSuccessNone2.053320s2.158529s
Success
printf_deadlock_smoke.cSuccessNone0.059015s0.188428s
Success
printf_thread_test.cSuccessNone0.048678s0.154680s
Success
sem_forks.cSuccessNone0.053137s0.158844s
Success
setsid.cSuccessNone0.042692s0.125078s
Success
template.cSuccessNone0.049597s0.158285s
Success
test_exec_nofork.cSuccessNone0.048578s0.192215s
Success
test_unlink_open_file.cSuccessNone0.045591s0.128449s
Success
thread-test.cSuccessNone0.044917s0.133290s
Success
thread.cSuccessNone0.044600s0.133630s
Success
thread_cageid_race.cSuccessNone0.045520s0.167193s
Success
tls_test.cSuccessNone0.048725s0.147008s
Success
uname.cSuccessNone0.043548s0.128463s
Success
wait.cSuccessNone2.046996s2.143308s
Success
waitpid_anychild.cSuccessNone0.049621s0.143841s
Success
waitpid_syscall.cSuccessNone1.050189s1.179102s
Success
waitpid_wnohang.cSuccessNone0.050482s0.143102s
Success
Signal Tests
alarm.cSuccessNone7.049866s7.165128s
Success
eintr_fork_signal.cSuccessNone1.051423s1.163702s
Success
kill.cSuccessNone1.049538s1.151039s
Success
setitimer.cSuccessNone7.049926s7.177464s
Success
sigalrm.cSuccessNone2.050048s2.158661s
Success
sigaltstack.cSuccessNone0.051681s0.150946s
Success
sigchld.cSuccessNone1.051393s1.150557s
Success
signal-fork.cSuccessNone4.051930s4.154694s
Success
signal-simple.cSuccessNone0.051322s0.142976s
Success
signal_SIGCHLD.cSuccessNone0.050199s0.146471s
Success
signal_fork.cSuccessNone0.046977s0.155951s
Success
signal_int_ignored.cSuccessNone2.050861s2.153425s
Success
signal_kill_cleanup.cSuccessNone1.047687s1.146854s
Success
signal_procmask.cSuccessNone0.045166s0.144026s
Success
signal_recursive.cSuccessNone0.045461s0.142097s
Success
signal_sa_mask.cSuccessNone0.043961s0.134838s
Success
sigpipe.cSuccessNone1.051682s1.167046s
Success
sigprocmask.cSuccessNone1.049632s1.152342s
Success
Fail Tests

Summary

MetricCount
Total Test Cases4
Number of Successes4
Number of Failures0
Number of Compilation Failure Native0
Number of Runtime Failure Native0
Number of Segmentation Fault Native0
Number of Timeout During Native0
Number of Lind Wasm Compile Failure0
Number of Lind Wasm Runtime Failure0
Number of Lind Wasm Segmentation Failure0
Number of Timeout During Lind Wasm run0
Number of Unknown Failure0
Number of C Compiler and Wasm Output mismatch0
Number of Fail Test: Native Succeeded (Should Fail)0
Number of Fail Test: Wasm Succeeded (Should Fail)0
Number of Fail Test: Both Native and Wasm Succeeded (Should Fail)0
Number of Fail Test: Native Compilation Failure (Should Succeed)0
Number of Fail Test: Wasm Compilation Failure (Should Succeed)0

Test Results by Category

Test CaseStatusError TypeNative TimeWasm TimeOutput
Dylink Tests
dlerror.cSuccessNone0.042046s0.127243s
Success
Memory Tests
mmap-negative1.cSuccessNone0.131335s0.142744s
Success
mmap-negative2.cSuccessNone0.107188s0.138146s
Success
Signal Tests
signal_resethand.cSuccessNone1.049112s1.149015s
Success

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: add a minimal testsuite for static build wasm binary

3 participants