Build process needs to run twice to produce a working coverage report #111

hhirsch · 2017-03-10T11:29:18Z

I have fiddled with support for code coverage for our project GlPortal/RadixEngine#30

However it only started producing proper results after I ran the build process twice
GlPortal/RadixEngine@59a6968

Any idea what the problem is?

Here is a log of the build:
https://travis-ci.org/GlPortal/RadixEngine/builds/212253813

The text was updated successfully, but these errors were encountered:

chrisosaurus · 2018-07-06T03:11:20Z

Strange, from your travis log I see 2 interesting sections

The first time coveralls only analyses one files, the second time it analyses 'a lot'.

https://travis-ci.org/GlPortal/RadixEngine/builds/212253813#L506

[100%] Built target tests
Test project /data
    Start 1: XmlHelperTest
1/1 Test #1: XmlHelperTest ....................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 1
Total Test time (real) =   0.01 sec

CMakeCXXCompilerId.gcda:cannot open data file, assuming not executed
File 'CMakeCXXCompilerId.cpp'
Lines executed:0.00% of 7
Creating 'CMakeCXXCompilerId.gcno##CMakeCXXCompilerId.cpp.gcov'

# it then moves on to doing other things

https://travis-ci.org/GlPortal/RadixEngine/builds/212253813#L715

[100%] Built target tests
Test project /data
    Start 1: XmlHelperTest
1/1 Test #1: XmlHelperTest ....................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 1
Total Test time (real) =   0.01 sec

/data/external/json11/CMakeFiles/json11.dir/json11.cpp.gcda:cannot open data file, assuming not executed

File '/usr/include/c++/6.3.1/bits/stl_algobase.h'
Lines executed:0.00% of 23
Creating '#data#external#json11#CMakeFiles#json11.dir#json11.cpp.o###usr#include#c++#6.3.1#bits#stl_algobase.h.gcov'

File '/usr/include/c++/6.3.1/bits/predefined_ops.h'
Lines executed:0.00% of 1
Creating '#data#external#json11#CMakeFiles#json11.dir#json11.cpp.o###usr#include#c++#6.3.1#bits#predefined_ops.h.gcov'

[...]
# lots more files analysed

Very strange.

chrisosaurus · 2018-07-07T02:03:47Z

Attempted reproduction steps roughly following https://github.com/GlPortal/RadixEngine/blob/master/COMPILE.md
(I tried to use a build dir at first but it didn't seem to add anything, so tried without)

$ sudo pacman -S cmake make gcc pkg-config assimp libepoxy sdl2{,_mixer} bullet mesa tinyxml2 freeimage
$ sudo pip install cpp-coveralls
$ git clone https://github.com/GlPortal/RadixEngine.git
$ cd RadixEngine
$ git submodule init
$ git submodule update
$ cmake .
$ make
$ make test
$ coveralls --dump HERE
$ grep -o '"name":[^,]*,' HERE | wc -l
890
$ grep -o '"name":[^,]*,' HERE | head -10
"name": "include/radix/Camera.hpp",
"name": "include/radix/SimulationManager.hpp",
"name": "include/radix/EntityManager.hpp",
"name": "include/radix/Entity.hpp",
"name": "include/radix/Console.hpp",
"name": "include/radix/SoundManager.hpp",
"name": "include/radix/BaseGame.hpp",
"name": "include/radix/World.hpp",
"name": "include/radix/GameWorld.hpp",
"name": "include/radix/Viewport.hpp",

so it seems to be correctly generating coverage report sfor 890 source files after a single run.

Are you still able to produce this? if so please provide reproduction steps.

hhirsch · 2018-07-07T06:08:08Z

The test target does not exist so make test should just fail.
make should build the tests though.

iirc coveralls not running wasn't the problem but it reporting 0% coverage when there is 3% coverage.

Variations can be easily tested on a branch and I'll be happy to change the line accordingly:
https://github.com/GlPortal/RadixEngine/blob/master/.travis.yml

I tried to tackle this problem multiple times and I always had to revert to the run twice method.

chrisosaurus · 2018-07-08T01:03:52Z

make test seems to run the tests for RadixEngine

[chris@niflheim RadixEngine]$ git remote -vv
origin	https://github.com/GlPortal/RadixEngine.git (fetch)
origin	https://github.com/GlPortal/RadixEngine.git (push)

[chris@niflheim RadixEngine]$ make test
Running tests...
Test project /home/chris/devel/RadixEngine
    Start 1: XmlHelperTest
1/3 Test #1: XmlHelperTest ....................   Passed    0.01 sec
    Start 2: QuaternionTest
2/3 Test #2: QuaternionTest ...................   Passed    0.01 sec
    Start 3: MathTest
3/3 Test #3: MathTest .........................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 3

Total Test time (real) =   0.03 sec

Am I misunderstanding something?

I guess I can try find time to setup my own coveralls and travis for a clone of RadixEngine and see what I can get out of it

chrisosaurus · 2018-07-08T01:20:00Z

So in RadixEngine it looks like:

make tests will build the tests
make test and ctest are equialent - either one will run the tests

[chris@niflheim RadixEngine]$ make tests
[  3%] Built target RadixEntity
[  3%] Built target GLAD
[  4%] Built target json11
[ 11%] Built target VHACD_LIB
[ 33%] Built target angelscript
[ 36%] Built target easy_profiler
[ 88%] Built target RadixEngine
[ 93%] Built target MathTest
[ 96%] Built target XmlHelperTest
[100%] Built target QuaternionTest
[100%] Built target tests

[chris@niflheim RadixEngine]$ make test
Running tests...
Test project /home/chris/devel/RadixEngine
    Start 1: XmlHelperTest
1/3 Test #1: XmlHelperTest ....................   Passed    0.01 sec
    Start 2: QuaternionTest
2/3 Test #2: QuaternionTest ...................   Passed    0.01 sec
    Start 3: MathTest
3/3 Test #3: MathTest .........................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 3

Total Test time (real) =   0.03 sec

[chris@niflheim RadixEngine]$ ctest
Test project /home/chris/devel/RadixEngine
    Start 1: XmlHelperTest
1/3 Test #1: XmlHelperTest ....................   Passed    0.01 sec
    Start 2: QuaternionTest
2/3 Test #2: QuaternionTest ...................   Passed    0.01 sec
    Start 3: MathTest
3/3 Test #3: MathTest .........................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 3

Total Test time (real) =   0.03 sec

I've setup travis and coveralls for my clone, will play and around and then report results back, might take a while

chrisosaurus · 2018-07-08T01:33:32Z

As a side effect of investigating this I noticed an issue in that you have a public .coveralls.yml (only needed for private travis, not needed for open source.)

I created an issue on your repo to describe this
GlPortal/RadixEngine#173
and then sent a PR to fix this
GlPortal/RadixEngine#174

That should just work as-is unless I am misunderstanding something.

chrisosaurus · 2018-07-08T02:06:18Z

Okay, I now have coveralls running outside the docker container and I'm able to reproduce the 0% coveralls report on my clone.

chrisosaurus · 2018-07-08T02:48:13Z

tests and output:

Running:

ctest inside docker ONCE
cpp-coveralls outside docker ONCE

gives 0% coverage
Log: https://api.travis-ci.org/v3/job/401344880/log.txt

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'

Running:

ctest inside docker ONCE
cpp-coveralls outside docker TWICE

gives 0% coverage
Log: https://api.travis-ci.org/v3/job/401348185/log.txt

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls --dryrun -i include -i source -e tests --gcov-options \'-lp\'
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'

Running:

ctest inside docker TWICE
cpp-coveralls outside docker ONCE

gives 0% coverage
Log: https://api.travis-ci.org/v3/job/401348813/log.txt

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'

Running:

'everything' inside docker TWICE
cpp-coveralls outside docker ONCE

gives 0% coverage
Log: https://api.travis-ci.org/v3/job/401349052/log.txt

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest && cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'

Very strangely, only this last one produced a huge log.
All previous versions produced

1.24s$ coveralls -i include -i source -e tests --gcov-options \'-lp\'
CMakeCXXCompilerId.gcno:version '603*', prefer '408*'
CMakeCXXCompilerId.gcda:cannot open data file, assuming not executed
File 'CMakeCXXCompilerId.cpp'
No executable lines
Removing 'CMakeCXXCompilerId.gcno##CMakeCXXCompilerId.cpp.gcov'
{u'url': u'https://coveralls.io/jobs/38190001', u'message': u'Job #11.1'}

only the last one (running cmake/make/ctest twice, coveralls once) produces the output we expect

[0K$ coveralls -i include -i source -e tests --gcov-options \'-lp\'
CMakeCXXCompilerId.gcno:version '603*', prefer '408*'
CMakeCXXCompilerId.gcda:cannot open data file, assuming not executed
File 'CMakeCXXCompilerId.cpp'
No executable lines
Removing 'CMakeCXXCompilerId.gcno##CMakeCXXCompilerId.cpp.gcov'

/home/travis/build/mkfifo/RadixEngine/CMakeFiles/RadixEngine.dir/source/SoundManager.cpp.gcno:version '603*', prefer '408*'
/home/travis/build/mkfifo/RadixEngine/CMakeFiles/RadixEngine.dir/source/SoundManager.cpp.gcda:cannot open data file, assuming not executed
File '/usr/include/c++/6.3.1/bits/stl_tree.h'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#CMakeFiles#RadixEngine.dir#source#SoundManager.cpp.o###usr#include#c++#6.3.1#bits#stl_tree.h.gcov'

File '/usr/include/c++/6.3.1/bits/alloc_traits.h'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#CMakeFiles#RadixEngine.dir#source#SoundManager.cpp.o###usr#include#c++#6.3.1#bits#alloc_traits.h.gcov'

[...]

File '/usr/include/c++/6.3.1/atomic'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#external#easy_profiler#sample#CMakeFiles#profiler_sample.dir#main.cpp.o###usr#include#c++#6.3.1#atomic.gcov'

File '/usr/include/c++/6.3.1/bits/atomic_base.h'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#external#easy_profiler#sample#CMakeFiles#profiler_sample.dir#main.cpp.o###usr#include#c++#6.3.1#bits#atomic_base.h.gcov'

File '/data/external/easy_profiler/easy_profiler_core/include/easy/profiler.h'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#external#easy_profiler#sample#CMakeFiles#profiler_sample.dir#main.cpp.o###data#external#easy_profiler#easy_profiler_core#include#easy#profiler.h.gcov'

File '/usr/include/c++/6.3.1/bits/locale_facets.h'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#external#easy_profiler#sample#CMakeFiles#profiler_sample.dir#main.cpp.o###usr#include#c++#6.3.1#bits#locale_facets.h.gcov'

{u'url': u'https://coveralls.io/jobs/38190033', u'message': u'Job #13.1'}

but it still seemed to generate a coverage of 0% unless something else went wrong.

I think the problem is at least in part due to your make process not producing coverage data the first time it is run, possibly due to cmake not correctly settings that variable you pass in.

I'll keep poking around.

chrisosaurus · 2018-07-08T03:36:13Z

Confirmed that the single build isn't producing the profile data

        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l

Log: https://api.travis-ci.org/v3/job/401354549/log.txt

output

travis_time:end:014a2c6d:start=1531020384350424558,finish=1531020386367779703,duration=2017355145
�[0K
�[32;1mThe command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.�[0m
travis_time:start:12bf641b
�[0K$ find . -iname '*.gcda' | wc -l
0

travis_time:end:12bf641b:start=1531020386374362869,finish=1531020386390520461,duration=16157592
�[0K
�[32;1mThe command "find . -iname '*.gcda' | wc -l" exited with 0.�[0m
travis_time:start:03a20972
�[0K$ find . -iname '*.gcno' | wc -l
1

running just ctest twice

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l

Log: https://api.travis-ci.org/v3/job/401354666/log.txt

outputs

travis_time:end:0557cb6e:start=1531020363849677574,finish=1531020365103355512,duration=1253677938
�[0K
�[32;1mThe command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.�[0m
travis_time:start:13f0336e
�[0K$ find . -iname '*.gcda' | wc -l
0

travis_time:end:13f0336e:start=1531020365110018399,finish=1531020365127668255,duration=17649856
�[0K
�[32;1mThe command "find . -iname '*.gcda' | wc -l" exited with 0.�[0m
travis_time:start:01fb3190
�[0K$ find . -iname '*.gcno' | wc -l
1

running cmake/make/ctest twice

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest && cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l

Log: https://api.travis-ci.org/v3/job/401354576/log.txt

outputs

travis_time:end:0771da55:start=1531020637657257692,finish=1531020639721292131,duration=2064034439
�[0K
�[32;1mThe command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.�[0m
travis_time:start:1eb72f18
�[0K$ find . -iname '*.gcda' | wc -l
54

travis_time:end:1eb72f18:start=1531020639731115174,finish=1531020639750849314,duration=19734140
�[0K
�[32;1mThe command "find . -iname '*.gcda' | wc -l" exited with 0.�[0m
travis_time:start:1a6234f4
�[0K$ find . -iname '*.gcno' | wc -l
172

So the problem looks to be that your build tooling requires running cmake/make/ctest twice before it produces the gcno and gcda files.

Cpp-coveralls doesn't appear to be the problem here.

chrisosaurus · 2018-07-08T04:23:29Z

So poking around in CMakeFiles/ with grep looking for 'coverage' we get

single run of everything

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l
        - grep -Rni 'coverage' CMakeFiles/* | wc -l

Log: https://api.travis-ci.org/v3/job/401359763/log.txt

output

travis_time:end:1060f428:start=1531023482629119561,finish=1531023484216188718,duration=1587069157
�[0K
�[32;1mThe command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.�[0m
travis_time:start:076beae1
�[0K$ find . -iname '*.gcda' | wc -l
0

travis_time:end:076beae1:start=1531023484225581695,finish=1531023484243654484,duration=18072789
�[0K
�[32;1mThe command "find . -iname '*.gcda' | wc -l" exited with 0.�[0m
travis_time:start:015e15a6
�[0K$ find . -iname '*.gcno' | wc -l
1

travis_time:end:015e15a6:start=1531023484255969002,finish=1531023484273764591,duration=17795589
�[0K
�[32;1mThe command "find . -iname '*.gcno' | wc -l" exited with 0.�[0m
travis_time:start:308de1c4
�[0K$ grep -Rni 'coverage' CMakeFiles/* | wc -l
2

So no gcno, no gcda, and only 2 mentions of 'coverage'

running cmake twice

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON && cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l
        - grep -Rni 'coverage' CMakeFiles/* | wc -l

Log: https://api.travis-ci.org/v3/job/401359814/log.txt

output

travis_time:end:0ce384e0:start=1531023535870351025,finish=1531023537691723347,duration=1821372322
�[0K
�[32;1mThe command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.�[0m
travis_time:start:1e795360
�[0K$ find . -iname '*.gcda' | wc -l
54

travis_time:end:1e795360:start=1531023537698709357,finish=1531023537714924357,duration=16215000
�[0K
�[32;1mThe command "find . -iname '*.gcda' | wc -l" exited with 0.�[0m
travis_time:start:0f774228
�[0K$ find . -iname '*.gcno' | wc -l
172

travis_time:end:0f774228:start=1531023537721351873,finish=1531023537738222833,duration=16870960
�[0K
�[32;1mThe command "find . -iname '*.gcno' | wc -l" exited with 0.�[0m
travis_time:start:0dd3c07f
�[0K$ grep -Rni 'coverage' CMakeFiles/* | wc -l
3

3 mentions of coverage, 54 gcda files, 172 gcno files

running 'everything' twice inside docker

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest && cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l
        - grep -Rni 'coverage' CMakeFiles/* | wc -l

Log: https://api.travis-ci.org/v3/job/401359864/log.txt

travis_time:end:0599ce52:start=1531023759612471574,finish=1531023761327501590,duration=1715030016
�[0K
�[32;1mThe command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.�[0m
travis_time:start:1f4625f9
�[0K$ find . -iname '*.gcda' | wc -l
54

travis_time:end:1f4625f9:start=1531023761335501133,finish=1531023761353031077,duration=17529944
�[0K
�[32;1mThe command "find . -iname '*.gcda' | wc -l" exited with 0.�[0m
travis_time:start:134c59f5
�[0K$ find . -iname '*.gcno' | wc -l
172

travis_time:end:134c59f5:start=1531023761360645133,finish=1531023761376400031,duration=15754898
�[0K
�[32;1mThe command "find . -iname '*.gcno' | wc -l" exited with 0.�[0m
travis_time:start:2519013c
�[0K$ grep -Rni 'coverage' CMakeFiles/* | wc -l
3

Again 3 mentions of coverage, 54 gcda, 172 gcno files.

chrisosaurus · 2018-07-08T04:25:07Z

So at the very least, your cmake config is buggy, as these two commands produce different outputs:

cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest

cmake -DCOVERAGE:BOOL=ON && cmake -DCOVERAGE:BOOL=ON ./ && make && make tests && ctest

chrisosaurus · 2018-07-08T04:28:12Z

I have filed GlPortal/RadixEngine#175 for the cmake config issue

chrisosaurus · 2018-07-08T05:24:17Z

I'm still seeing a lot of noise in the genereated output, lots of mentions for system included files which will throw your coverage stats way off.

chrisosaurus mentioned this issue Jul 8, 2018

.coveralls.yml should not be public GlPortal/RadixEngine#173

Open

chrisosaurus mentioned this issue Jul 8, 2018

Buggy CMake config GlPortal/RadixEngine#175

Open

chrisosaurus mentioned this issue Jul 8, 2018

Runtime error in parse_gcov_file: 27-block 0 is in the gcov file, but appears to be unexpected by the parser #116

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build process needs to run twice to produce a working coverage report #111

Build process needs to run twice to produce a working coverage report #111

hhirsch commented Mar 10, 2017 •

edited

Loading

chrisosaurus commented Jul 6, 2018

chrisosaurus commented Jul 7, 2018 •

edited

Loading

hhirsch commented Jul 7, 2018 •

edited

Loading

chrisosaurus commented Jul 8, 2018

chrisosaurus commented Jul 8, 2018

chrisosaurus commented Jul 8, 2018

chrisosaurus commented Jul 8, 2018

chrisosaurus commented Jul 8, 2018 •

edited

Loading

chrisosaurus commented Jul 8, 2018 •

edited

Loading

chrisosaurus commented Jul 8, 2018

chrisosaurus commented Jul 8, 2018 •

edited

Loading

chrisosaurus commented Jul 8, 2018

chrisosaurus commented Jul 8, 2018

Build process needs to run twice to produce a working coverage report #111

Build process needs to run twice to produce a working coverage report #111

Comments

hhirsch commented Mar 10, 2017 • edited Loading

chrisosaurus commented Jul 6, 2018

chrisosaurus commented Jul 7, 2018 • edited Loading

hhirsch commented Jul 7, 2018 • edited Loading

chrisosaurus commented Jul 8, 2018

chrisosaurus commented Jul 8, 2018

chrisosaurus commented Jul 8, 2018

chrisosaurus commented Jul 8, 2018

chrisosaurus commented Jul 8, 2018 • edited Loading

chrisosaurus commented Jul 8, 2018 • edited Loading

chrisosaurus commented Jul 8, 2018

chrisosaurus commented Jul 8, 2018 • edited Loading

chrisosaurus commented Jul 8, 2018

chrisosaurus commented Jul 8, 2018

hhirsch commented Mar 10, 2017 •

edited

Loading

chrisosaurus commented Jul 7, 2018 •

edited

Loading

hhirsch commented Jul 7, 2018 •

edited

Loading

chrisosaurus commented Jul 8, 2018 •

edited

Loading

chrisosaurus commented Jul 8, 2018 •

edited

Loading

chrisosaurus commented Jul 8, 2018 •

edited

Loading