Cannot reduce tar-1.14

[michaelbrownuc]

When attempting to reduce tar, I get the error:

The specified oracle file ./test.sh cannot execute correctly.

[KihongHeo]

Chisel first checks whether a given test script runs correctly. It seems that test.sh does not preperly terminate with return code 0. Can you first try ./test.sh; echo $??

Some possible reasons are 1) tar-1.14.c is missing, 2) you missed running . setenv in ChiselBench/, or 3) timeout. For 3) can you increase the time limits in the script? In case of tar, we have two different variables TIMEOUT and TIMEOUT_LOW. You can increase both.

[michaelbrownuc]

1. the file is correctly generated from tar-1.14.c.origin.c
2. I did run the setenv command
3. The error occurs immediately, before the the first function reduction. I increased the timeout, this does not affect the failure.

The script test.sh returns 1 when running the command you specified. I did not alter the test script. I didn't see any immediate reason why it was not returning 0.

[KihongHeo]

Can you localize the problem using bash -x ./test.sh and tell me when the script stops?

[michaelbrownuc]

Here's the last 20 or so lines of the output from bash -x

• find . -not -path './optional*' -not -path './afl_crash*' -not -path './reducer-out*' -not -path './chisel-out*' -not -path './lib*' -not -path ./tar-1.14.c.chisel.c -not -path ./tar-1.14.c.reduced.c -not -path './bin' -not -path './status*' -not -path './json' -not -path './tests' -not -path './tar*' -not -path './test*' -not -path './afl_input*' -not -path ./Makefile -not -path ./setenv -not -path ./.config -delete
• return 0
• compile -fsanitize=address
• case $COV in
• CFLAGS='-w -fsanitize=address '
• clang /home/michael/ChiselBench/benchmark/tar-1.14/tar-1.14.c -w -fsanitize=address -o /home/michael/ChiselBench/benchmark/tar-1.14/tar-1.14.reduced
• return 0
• desired -fsanitize=address
• [[ -fsanitize=address == -\f\s\a\n\i\t\i\z\e=\l\e\a\k ]]
• cp /home/michael/ChiselBench/benchmark/tar-1.14/tar-1.14.reduced tar
  ++ ls tests/tar-archives-multiple-files tests/tar-complains-about-missing-file tests/tar-demands-at-least-one-ctx tests/tar-demands-at-most-one-ctx tests/tar-extracts-all-subdirs tests/tar-extracts-file tests/tar-extracts-from-standard-input tests/tar-extracts-multiple-files tests/tar-extracts-to-standard-output tests/tar-handles-cz-options tests/tar-handles-empty-include-and-non-empty-exclude-list tests/tar-handles-exclude-and-extract-lists tests/tar-handles-multiple-X-options tests/tar-handles-nested-exclude
• for file in $(ls tests/tar*)
• exit 1

[KihongHeo]

That is quite odd. test.sh does not rely on any other programs except the builtin tar in your os.
Do you use bash?
Can you take a look at log.txt?
Can you localize the problem more by printing the execution steps?

[michaelbrownuc]

It appears the issue is on line 40 of test.sh, execution of one of the files in tests/tar is tripping the exit statement. I'll try to narrow it down further.

[michaelbrownuc]

Looks like the issue is during the execution of tar-archives-multiple-files for the address sanitization option. No information in the log.txt file.

[michaelbrownuc]

I was able to get tar to run by adding a check similar to the one for the leak sanitizer to the desired() function. I'm not sure why this works on my end and not on yours, but this appears to be some progress.

[Mohannadcse]

How did you run the analysis? I'm using the docker-machine, then go to one of the examples (say tar-1.14), finally, I goto the merged directory where are the Makefile and test.sh. Here is the full path:
/chiselbench/benchmark/tar-1.14/merged

Then I run the make command make reduce but received the following error shown the the screenshot

[KihongHeo]

@Mohannadcse Can you try to replace "../target.mk" with "../../target.mk"?

[Mohannadcse]

Thanks. I changed it. There is another error

[KihongHeo]

Please adjust benchmark dir (BENCHMARK_DIR) in test.sh, e.g.,
benchmark/$BENCHMARK_NAME
to
benchmark/$BENCHMARK_NAME/merged

[Mohannadcse]

Thanks, the script works now.

[ternava]

Still, even with these two last issues fixed, I have an error when I try to run make reduce for tar and sort, in their /chiselbench/benchmark/tar-1.14/merged and /chiselbench/benchmark/sort-8.16/merged respective folders.

In both cases I receive the same error as mentioned before:

I looked in the log.txt of sort, this is the returned error.

chisel-bench/benchmark/sort-8.16/merged/test.sh

Line 102 in 490ae19

outputcheckerror "invalid number at field start: invalid count at start of \‘notexist\’" && exit 1

Whereas the log.txt of tar is empty.

Any idea how I can fix them?

[ternava]

Apparently, the error in sort is caused by the wrong use of characters inside comments in these two places:

chisel-bench/benchmark/sort-8.16/merged/test.sh

Line 102 in 490ae19

outputcheckerror "invalid number at field start: invalid count at start of \‘notexist\’" && exit 1

and

chisel-bench/benchmark/sort-8.16/merged/test.sh

Line 117 in 490ae19

outputcheckerror "multi-character tab \‘notexist\’" && exit 1

In my case, the error in sort is fixed by using the straight single quotation marks, i.e., instead of \‘notexist’\ it should be changed to 'notexist'.

[codingsoo]

Looks like the issue is during the execution of tar-archives-multiple-files for the address sanitization option. No information in the log.txt file.

I have the same problem.