Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add stdout and stderr logging for AV2. #104

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Add stdout and stderr logging for AV2. #104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ab52561
Add stdout and stderr logging for AV2.
tdaede Apr 9, 2021
35c7a6f
Add stdout and stderr logging for rav1e.
tdaede Apr 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions metrics_gather.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,7 @@ av2 | av2-ai | av2-ra | av2-ra-st | av2-ld | av2-as)
echo "#!/bin/bash" > /tmp/enc$$.sh
echo "TIMER='time -v --output='enctime$$-\$1.out" >> /tmp/enc$$.sh
echo "RUN='$AOMENC --codec=av1 --cq-level=$x --test-decode=fatal $CTC_PROFILE_OPTS -o $BASENAME-'\$1'.obu $EXTRA_OPTIONS --limit=130 --'\$1'=65 $FILE'" >> /tmp/enc$$.sh
echo "\$(\$TIMER \$RUN > $BASENAME$$-stdout.txt)" >> /tmp/enc$$.sh
echo "\$(\$TIMER \$RUN > $BASENAME$$-stdout.txt 2> $BASENAME$$-stderr.txt)" >> /tmp/enc$$.sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't $BASENAME$$-stderr.txt contain the PID of the executing process (for use with GOP parallelism)?

It looks like the files are copied back, but they will still contain the PID and not easily linkable from the AWCY frontend. Are these merged somewhere?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, they shouldn't contain the PID? However, they don't actually get merged properly. Both of those should probably be >> for mt.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The $$ in those file names is the PID. I don't think you want to use >> as that would interleave the output. Instead, after both have completed, you should merge these into a single file before copying.

chmod +x /tmp/enc$$.sh
for s in {limit,skip}; do printf "$s\0"; done | xargs -0 -n1 -P2 /tmp/enc$$.sh
$(cat $BASENAME-limit.obu $BASENAME-skip.obu > $BASENAME.obu)
Expand All @@ -260,7 +260,7 @@ av2 | av2-ai | av2-ra | av2-ra-st | av2-ld | av2-as)
rm -f /tmp/enc$$.sh enctime$$-limit.out enctime$$-skip.out $BASENAME-limit.obu $BASENAME-skip.obu
;;
*)
$($TIMER $AOMENC --codec=av1 --cq-level=$x --test-decode=fatal $CTC_PROFILE_OPTS -o $BASENAME.obu $EXTRA_OPTIONS $FILE > "$BASENAME-stdout.txt")
$($TIMER $AOMENC --codec=av1 --cq-level=$x --test-decode=fatal $CTC_PROFILE_OPTS -o $BASENAME.obu $EXTRA_OPTIONS $FILE > "$BASENAME-stdout.txt" 2> "$BASENAME-stderr.txt")
;;
esac
# decode the OBU to Y4M
Expand Down Expand Up @@ -299,7 +299,7 @@ thor-rt)
SIZE=$(stat -c %s $BASENAME.thor)
;;
rav1e)
$($TIMER $RAV1E $FILE --quantizer $x -o $BASENAME.ivf -r $BASENAME-rec.y4m --threads 1 $EXTRA_OPTIONS > $BASENAME-enc.out)
$($TIMER $RAV1E $FILE --quantizer $x -o $BASENAME.ivf -r $BASENAME-rec.y4m --threads 1 $EXTRA_OPTIONS > "$BASENAME-stdout.txt" 2> "$BASENAME-stderr.txt")
if hash dav1d 2>/dev/null; then
$($TIMERDEC dav1d -q -i $BASENAME.ivf -o $BASENAME.y4m) || (echo "Corrupt bitstream detected!"; exit 98)
elif hash aomdec 2>/dev/null; then
Expand Down
2 changes: 1 addition & 1 deletion work.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ class RDWork(Work):
def __init__(self):
super().__init__()
self.no_delete = False
self.copy_back_files = ['-stdout.txt']
self.copy_back_files = ['-stdout.txt', '-stderr.txt']
def parse(self, stdout, stderr):
self.raw = stdout
split = None
Expand Down