Add Python package sos-notebook. #7665

jdblischak · 2019-01-28T02:58:48Z

This PR adds a recipe for the Python package sos-notebook.

The build scripts install the jupyter kernel. The ipython and notebook packages are included in the host dependencies because they are used by the script that installs the kernel (source).

@BoPeng and @gaow, please confirm that you are willing to be co-maintainers of sos-notebook.

conda-forge-linter · 2019-01-28T02:58:57Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/sos-notebook) and found it was in an excellent condition.

BoPeng · 2019-01-28T03:09:32Z

Yes, I can join as a co-maintainer.

gaow · 2019-01-28T14:37:36Z

Yes I'm willing to be a co-maintainer. Thanks!

jdblischak · 2019-01-28T18:04:36Z

@BoPeng Are you sure that the SoS kernel doesn't need matplotlib?

Using the conda version of sos-notebook that I've uploaded to my personal channel:

conda create -n test-sos-notebook -c jdblischak -c conda-forge sos-notebook r-irkernel

The kernels for Python3 and R both work:

$ jupyter run <(echo '1+1')
$ jupyter run --kernel=ir  <(echo '1+1')

But the SoS kernel does not:

$ jupyter run --kernel=sos  <(echo '1+1')
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.7/site-packages/sos_notebook/kernel.py", line 1422, in <module>
    IPKernelApp.launch_instance(kernel_class=SoS_Kernel)
  File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "</opt/conda/lib/python3.7/site-packages/decorator.py:decorator-gen-791>", line 2, in initialize
  File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/ipykernel/kernelapp.py", line 484, in initialize
    self.init_kernel()
  File "/opt/conda/lib/python3.7/site-packages/ipykernel/kernelapp.py", line 389, in init_kernel
    user_ns=self.user_ns,
  File "/opt/conda/lib/python3.7/site-packages/traitlets/config/configurable.py", line 412, in instance
    inst = cls(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/sos_notebook/kernel.py", line 508, in __init__
    self.shell.enable_matplotlib('inline')
  File "/opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3343, in enable_matplotlib
    gui, backend = pt.find_gui_and_backend(gui, self.pylab_gui_select)
  File "/opt/conda/lib/python3.7/site-packages/IPython/core/pylabtools.py", line 276, in find_gui_and_backend
    import matplotlib
ModuleNotFoundError: No module named 'matplotlib'

After I install matplotlib, then the SoS kernel works.

BoPeng · 2019-01-28T18:16:34Z

As you can see from the error message, the function is inside ipykernel and I would expect that matplotlib be a dependency of ipykernel, not sos.

But if we need matplotlib to make the recipe work, please add it. I might add it back to setup.py in the future release of sos-notebook.

jdblischak · 2019-01-28T18:44:19Z

As you can see from the error message, the function is inside ipykernel and I would expect that matplotlib be a dependency of ipykernel, not sos.

Right. But the ipykernel works without matplotlib (at least for a simple calculation like 1+1). Thus the SoS kernel must be making a call to ipykernel that requires matplotlib even when no plot needs to be made. The ipykernel recipe only installs matplotlib to run its tests:

https://github.com/conda-forge/ipykernel-feedstock/blob/master/recipe/meta.yaml#L34

But if we need matplotlib to make the recipe work, please add it. I might add it back to setup.py in the future release of sos-notebook.

We have multiple options to consider:

Change the ipykernel recipe to have matplotlib as a dependency
Change the sos-notebook recipe to have matplotlib as a depenency
Change the SoS kernel to behave more like ipykernel in regards to plotting (i.e. only require matplotlib when a plot is being created). Would this be possible?

BoPeng · 2019-01-28T18:50:13Z

Please go with 2 and I will see what I can do about 3 later.

…orge/staged-recipes#7665

BoPeng · 2019-01-31T05:17:37Z

ok, I have stopped setting matplotlib mode to inline by default. Now the behavior of SoS is consistent with the Python kernel, namely the plot will not be shown by default and users have to use %matplotlib notebook explicitly to include the figures in notebook.

Although SoS setup.py will not have matplotlib by default, I would suggest that we keep the matplotlib dependency in conda to provide a more complete environment.

jdblischak · 2019-01-31T16:16:37Z

@conda-forge/staged-recipes This is ready for review.

I'd appreciate advice on my tests to ensure that the sos kernel is properly installed. Because the name of the pacakge, sos-notebook, is in the filepath, the tests that search for "sos" in the output from jupyter kernelspec list will pass even if it isn't installed (i.e. because sos appears in the path to the Python3 kernel).

    - jupyter kernelspec list | grep -w sos  # [unix]
    - jupyter kernelspec list | findstr sos  # [win]
    - jupyter run --kernel=sos <(echo "x=1")  # [unix]

ipykernel does not install it by default because it can run non-plotting code without matplotlib. However, the SoS kernel requires matplotlib to run any code.

https://github.com/jupyter/jupyter_kernel_test

jdblischak · 2019-03-29T17:35:49Z

@BoPeng Could you please advice on how to that the SoS kernel is functional after the conda package is installed? I've tried multiple different ideas, but I keep getting errors.

I tried testing the kernel using jupyter_kernel_test, but this failed with the following error:

AttributeError: module 'logging' has no attribute 'TRACE'

https://ci.appveyor.com/project/conda-forge/staged-recipes/builds/23437118#L1110
https://travis-ci.org/conda-forge/staged-recipes/builds/512705036#L1638
https://circleci.com/gh/conda-forge/staged-recipes/36688

So then I tried an even simpler test. Just try to run a script that only contained the line x=1. This fails with the same error:

https://travis-ci.org/conda-forge/staged-recipes/builds/513133132#L1515
https://circleci.com/gh/conda-forge/staged-recipes/36715

I've also tested sos-notebook 0.19.1 locally, and it is still failing. I'll push it here so that you can also view the results with the latest version.

marcelotrevisani · 2019-03-29T17:44:33Z

recipes/sos-notebook/meta.yaml

+  url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version }}.tar.gz
+  sha256: 9f3ae4371f2a9cf493c77190010e73270ddc658904618937889ce14c0ecebb09
+
+build:


Please, delete the build.sh and bld.bat. After that please add in your build section

script: - {{ PYTHON }} -m pip install . --no-deps --ignore-installed -vv - {{ PYTHON }} -m sos_notebook.install --sys-prefix

Otherwise, this recipe cannot be noarch

@marcelotrevisani Thanks! I just pushed this change.

BoPeng · 2019-03-29T17:49:39Z

@jdblischak sos notebook has its own test suite but some advanced usages requires other modules to test (e.g. some language modules). I did not check the details but your test might have failed due to the use of development version of SoS, which no longer has logging.TRACE. Anyway, to make sure everything is up to date, I have just released sos 0.19.3 and sos-notebook 0.19.2, and the TRACE error should go away if you use these two releases.

jdblischak · 2019-03-29T18:02:01Z

did not check the details but your test might have failed due to the use of development version of SoS, which no longer has logging.TRACE.

@BoPeng The version of SoS was the latest version available from conda-forge (0.19.2). All the tests I've tried are purposefully simple (i.e. basic arithmetic). They are just testing if it was installed properly.

Anyway, to make sure everything is up to date, I have just released sos 0.19.3 and sos-notebook 0.19.2, and the TRACE error should go away if you use these two releases.

Thanks! I'll update the recipe again.

jdblischak · 2019-03-29T18:14:53Z

Strange. It's working now that I specified the build steps in meta.yaml, as instructed by @marcelotrevisani to make it a proper noarch recipe. Maybe this was somehow causing the kernel to not be installed properly.

https://travis-ci.org/conda-forge/staged-recipes/builds/513149395#L1625

BoPeng · 2019-03-29T18:20:57Z

@jdblischak Then please do not update the recipe for sos notebook 0.19.2 because I noticed that the travis test for it failed.

jdblischak · 2019-03-29T18:42:36Z

@BoPeng The simple test passed on everything except Windows:

https://ci.appveyor.com/project/conda-forge/staged-recipes/builds/23461251#L1100

(%PREFIX%) %SRC_DIR%>jupyter run --kernel=sos ex.py 
'RootLogger' object has no attribute 'trace'
Traceback (most recent call last):
  File "C:\bld\sos-notebook_1553883966484\_test_env\Scripts\jupyter-run-script.py", line 9, in <module>
    sys.exit(main())
  File "C:\bld\sos-notebook_1553883966484\_test_env\lib\site-packages\jupyter_core\application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "C:\bld\sos-notebook_1553883966484\_test_env\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
    app.start()
  File "C:\bld\sos-notebook_1553883966484\_test_env\lib\site-packages\jupyter_client\runapp.py", line 115, in start
    raise Exception("jupyter-run error running '%s'" % filename)
Exception: jupyter-run error running 'ex.py'

BoPeng · 2019-03-29T18:56:34Z

I do not know what is going on but I will find a windows machine and test. Note that I have fixed tests of sos notebook and released sos notebook 0.19.3, so if you would like to upgrade, 0.19.3 is the version to try.

For testing if the basic features (namely kernel switch) of sos notebook work correctly, a basic unit test would be

import unittest

from ipykernel.tests.utils import execute, wait_for_idle, assemble_output
from sos_notebook.test_utils import sos_kernel

class TestSoSKernel(unittest.TestCase):
    def testKernel(self):
        with sos_kernel() as kc:
            execute(kc=kc, code='a = 1\n')
            wait_for_idle(kc)
            execute(kc=kc, code='%use Python3\n%get a\nb = a + 1')
            wait_for_idle(kc)
            execute(kc=kc, code='%use SoS\n%get b --from Python3\nprint(b)')
            stdout, stderr = assemble_output(kc.iopub_channel)
            self.assertEqual(stderr, '')
            self.assertEqual(stdout.strip(), '2')

if __name__ == '__main__':
    unittest.main()

so you can include this file and run python test.py.

jdblischak · 2019-04-03T17:04:14Z

I updated the recipe to 0.19.3. Fingers crossed!

BoPeng · 2019-04-03T17:41:43Z

The problem persists so this is a timing issue. Basically on a slow system the kernel will take a while to start so the test times out. Changing

from ipykernel.tests.utils import execute, wait_for_idle, assemble_output
from sos_notebook.test_utils import sos_kernel

to

from sos_notebook.test_utils import sos_kernel
from ipykernel.tests.utils import execute, wait_for_idle, assemble_output

will make the test use longer TIMEOUT defined in test_utils, and might help. If this still does not work, we can remove it for now.

BoPeng · 2019-04-03T19:03:19Z

I am out of ideas and can only suggest that you remove the with sos_kernel() test (for now).

jdblischak · 2019-04-03T21:42:33Z

I can't get the sos kernel to work inside of a conda environment. What happens when you run the code below on a Windows machine?

conda create -y -n test-win -c conda-forge ^
  ipykernel ipython markdown matplotlib 'nbconvert>=5.1.1' ^
  nbformat 'notebook>=5.0.0' pandas 'python>=3.6' ^
  'sos>=0.19.2' tabulate
conda activate test-win
pip install sos-notebook
python -m sos_notebook.install --sys-prefix
jupyter run --kernel=python3 ex.py
jupyter run --kernel=sos ex.py

When I run this, I get the following error (and it stalls, i.e. I have to press Enter to restore the command prompt):

jupyter run --kernel=sos ex.py
[IPKernelApp] WARNING | Parent appears to have exited, shutting down.

(test-win) C:\Users\john\repos\staged-recipes\recipes\sos-notebook>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\john\Miniconda3\envs\test-win\lib\multiprocessing\spawn.py", line 99, in spawn_main
    new_handle = reduction.steal_handle(parent_pid, pipe_handle)
  File "C:\Users\john\Miniconda3\envs\test-win\lib\multiprocessing\reduction.py", line 87, in steal_handle
    _winapi.DUPLICATE_SAME_ACCESS | _winapi.DUPLICATE_CLOSE_SOURCE)
PermissionError: [WinError 5] Access is denied

And one time I got the following error message:

jupyter run --kernel=sos ex.py
[IPKernelApp] WARNING | Parent appears to have exited, shutting down.

(test-win) C:\Users\john\repos\staged-recipes\recipes\sos-notebook>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\john\Miniconda3\envs\test-win\lib\multiprocessing\spawn.py", line 99, in spawn_main
    new_handle = reduction.steal_handle(parent_pid, pipe_handle)
  File "C:\Users\john\Miniconda3\envs\test-win\lib\multiprocessing\reduction.py", line 82, in steal_handle
    _winapi.PROCESS_DUP_HANDLE, False, source_pid)
OSError: [WinError 87] The parameter is incorrect

And to be fair, sometimes the python3 kernel also fails:

jupyter run --kernel=python3 ex.py
Traceback (most recent call last):
  File "C:\Users\john\Miniconda3\envs\test-win\Scripts\jupyter-run-script.py", line 9, in <module>
    sys.exit(main())
  File "C:\Users\john\Miniconda3\envs\test-win\lib\site-packages\jupyter_core\application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "C:\Users\john\Miniconda3\envs\test-win\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
    app.start()
  File "C:\Users\john\Miniconda3\envs\test-win\lib\site-packages\jupyter_client\runapp.py", line 112, in start
    reply = self.kernel_client.execute_interactive(code, timeout=OUTPUT_TIMEOUT)
  File "C:\Users\john\Miniconda3\envs\test-win\lib\site-packages\jupyter_client\blocking\client.py", line 325, in execute_interactive
    raise TimeoutError("Timeout waiting for output")
TimeoutError: Timeout waiting for output
[IPKernelApp] WARNING | Parent appears to have exited, shutting down.

Doing some searching, there are lots of threads about Windows security features affecting Jupyter in a conda environment, e.g. this SO question

BoPeng · 2019-04-03T22:10:31Z

With the same setup, sos notebook appears to be working all right (start with jupyter notebook). I observe the same error messages for python3 and sos kernels.

sos has more trace message for the "parent appears to have exited" thing.

Note that the sos_kernel() test passes

BoPeng · 2019-04-03T22:21:30Z

conda build sos-notebook almost succeeded, but

As you can see, the local conda sos kernel path is displayed. Perhaps you need to add absolute path to jupyter but the recipe should be ok on appveyor.

BoPeng · 2019-04-04T02:55:58Z

@jdblischak I used a very tedious way to check why the test failed. Basically, I created a minimal test case for sos-notebook for a branch of sos-notebook with a minimal appveyor config . Instead of using requirements.txt, this test uses pip's own system to download all dependencies. The test passed.

I then compared packages installed by this setup with the one that failed, and found that the following packages are missing:

prompt-toolkit from ipython
ptyprocess from pexpect
prometheus-client from notebook

Basically, ipython, pexpect and notebook are installed from conda-forge, but without these dependencies, so whereas pip install sos-notebook can install them, conda install sos-notebook will not. Could you please add these packages to the recipe and see if the test pass?

jdblischak · 2019-04-04T15:12:01Z

Could you please add these packages to the recipe and see if the test pass?

@BoPeng Great work! I added prompt_toolkit and prometheus_client from conda-forge.

I had noticed that ptyprocess was being installed via pip when I ran sos-notebook, despite already having installed the other dependencies. Unfortunately ptyprocess is not available for Windows from conda-forge, though this PR may change that: conda-forge/ptyprocess-feedstock#15

jdblischak · 2019-04-04T15:14:50Z

As you can see, the local conda sos kernel path is displayed. Perhaps you need to add absolute path to jupyter but the recipe should be ok on appveyor.

Running jupyter kernelspec list outside of conda build sos-notebook will not be informative for why the test failed. This is because conda build creates new environments for building and testing the recipe.

BoPeng · 2019-04-04T15:23:43Z

Still failed. Perhaps ptyprocess can be added since it is available through pip? Since the test did not wait for TIMEOUT seconds, the only explanation is the server (the kernel) failed to start due to missing dependency...

BoPeng · 2019-04-04T15:25:24Z

Another way to test is to remove ptyprocess etc from a local environment and check if it cripples the test.

jdblischak · 2019-04-04T15:29:00Z

@BoPeng Good ideas. I'll investigate more either later today or tomorrow.

BoPeng · 2019-04-04T15:31:02Z

Please just add ptyprocess and see if we are lucky. I dislike this kind of blind attempts to fix upstream problems but this is life.

jdblischak · 2019-04-04T15:41:03Z

Please just add ptyprocess and see if we are lucky. I dislike this kind of blind attempts to fix upstream problems but this is life.

It's not possible to include a pip dependency in a conda recipe. See conda/conda-build#548

The Windows build will fail early because it won't be able to find ptyprocess. I already ran this locally, which is how I originally discovered that ptyprocess was not available for Windows.

BoPeng · 2019-04-04T15:41:03Z

Never mind, ptyprocess is not needed for the test to pass. :sad:

jdblischak · 2019-04-04T15:42:01Z

Never mind, ptyprocess is not needed for the test to pass. :sad:

Shoot. That is sad. Thanks for investigating!

BoPeng · 2019-04-04T15:48:39Z

I just added conda list and pip list to the mini test, and you can see the complete package list here. Perhaps you can add similar commands before the test runs in the recipe... For later if you are busy though, because this has been a long battle.

And here is a list of pip/conda packages not found in the failed one

asn1crypto
cffi 
chardet
conda
conda-env
console-shortcut
cryptography
idna
menuinst
ptyprocess <- can reproduce if completely removed.
pycosat
pycparser 
pyopenssl
pysocks
pywin32
requests 
ruamel-yaml
urllib3
win-inet-pton

BoPeng · 2019-04-04T16:51:56Z

@jdblischak Sorry, I spoke too fast. After I completely removed ptyprocess (there is a directory and an info file), I can reproduce the problem, so I can conclude with some confidence that ptyprocess is the culprit of the test.

@isuruf It took us a lot of work to figure out why our test failed. Basically, ptyprocess is required by pexpect but it is not installed with the conda-forge version of pexpect, so whereas pip install sos-notebook will install it through pexpect, conda install sos-notebook -c conda-forge will not. Since we cannot add ptyprocess as a pip dependency, what can we do here?

Correction: ptyprocess is available through conda-forge, but it excludes win32. This is why the tests only fail under windows.

BoPeng · 2019-04-04T18:10:20Z

@jdblischak My suggestion is that we add

build:
  skip: True  # [win]

to the recipe and move on since this is an upstream problem that we cannot fix.

jdblischak · 2019-04-05T19:07:46Z

@BoPeng This is a disappointing end after all this troubleshooting, but I agree that without ptyprocess it won't be possible to use the SoS kernel on Windows. I did some more troubleshooting with Jupyter Notebook on Windows, and found the same thing that you did, namely that installing ptyprocess via pip fixed the problem.

jdblischak · 2019-04-05T19:25:07Z

@conda-forge/staged-recipes This Python recipe is ready for review

BoPeng · 2019-04-20T05:00:50Z

ping @conda-forge/staged-recipes

chrisburr · 2019-04-23T18:47:40Z

recipes/sos-notebook/meta.yaml

+    - ipython
+    - notebook >=5.0.0
+    - pip
+    - python >=3.6


If this can't be noarch you should add skip: True # [py<36] to the build section and specify remove the version constraint from Python:

Suggested change

- python >=3.6

- python

(same applies in the run section)

Having looked more closely it looks like this could be noarch: python?

chrisburr · 2019-04-23T18:51:28Z

recipes/sos-notebook/meta.yaml

+  sha256: 91fc849a9d8c0b5fb975ec82eb1f2b9be2603ff936b8241a928cc8d738a57e16
+
+build:
+  skip: True  # [win]


I think this can be removed now as ptyprocess is now noarch:

Suggested change

skip: True # [win]

Indeed. Maybe they overheard our discussion. :-)

@jdblischak Could you try to remove this line and see if the tests pass?

jdblischak force-pushed the sos-notebook branch from bfe18b1 to 5361e38 Compare January 30, 2019 14:40

BoPeng pushed a commit to vatlab/sos-notebook that referenced this pull request Jan 31, 2019

Stop setting matplotlib inline mode by default vatlab/sos#896 conda-f…

f8eb7f1

…orge/staged-recipes#7665

jdblischak force-pushed the sos-notebook branch from 5361e38 to 60b54b0 Compare February 7, 2019 16:43

jdblischak added 6 commits March 28, 2019 16:52

Add Python package sos-notebook.

e0a88eb

Add matplotlib as a dependency for sos-notebook.

5783369

ipykernel does not install it by default because it can run non-plotting code without matplotlib. However, the SoS kernel requires matplotlib to run any code.

Add a test to ensure that SoS kernel is functional.

351f3b5

Improve tests for SoS kernel installation.

fbe0a80

Fix sos-notebook test on Windows.

37affe9

Try testing kernel with jupyter_kernel_test.

e389f42

https://github.com/jupyter/jupyter_kernel_test

jdblischak force-pushed the sos-notebook branch from 3f0be22 to e389f42 Compare March 28, 2019 20:53

Try a simpler test.

dc1bb83

Update sos-notebook to 0.19.1.

d0b292e

marcelotrevisani reviewed Mar 29, 2019

View reviewed changes

Specify noarch build steps in build/script section.

5e29676

jdblischak added 2 commits April 3, 2019 14:05

Move setuptools from run to host.

44652e9

Use longer TIMEOUT defined in sos_notebook.test_utils.

0946379

Add prompt_toolkit and prometheus_client.

fe325bf

Skip Windows build since ptyprocess isn't available from conda-forge.

b77ff46

chrisburr reviewed Apr 23, 2019

View reviewed changes

Try setting noarch: python

27e99d6

chrisburr merged commit 90d84d6 into conda-forge:master Apr 23, 2019

Add Python package sos-notebook. #7665

Add Python package sos-notebook. #7665

Conversation

jdblischak commented Jan 28, 2019

conda-forge-linter commented Jan 28, 2019

BoPeng commented Jan 28, 2019

gaow commented Jan 28, 2019

jdblischak commented Jan 28, 2019

BoPeng commented Jan 28, 2019

jdblischak commented Jan 28, 2019

BoPeng commented Jan 28, 2019

BoPeng commented Jan 31, 2019

jdblischak commented Jan 31, 2019

jdblischak commented Mar 29, 2019

marcelotrevisani Mar 29, 2019

Choose a reason for hiding this comment

jdblischak Mar 29, 2019

Choose a reason for hiding this comment

BoPeng commented Mar 29, 2019

jdblischak commented Mar 29, 2019

jdblischak commented Mar 29, 2019

BoPeng commented Mar 29, 2019

jdblischak commented Mar 29, 2019

BoPeng commented Mar 29, 2019 • edited Loading

jdblischak commented Apr 3, 2019

BoPeng commented Apr 3, 2019

BoPeng commented Apr 3, 2019

jdblischak commented Apr 3, 2019

BoPeng commented Apr 3, 2019

BoPeng commented Apr 3, 2019

BoPeng commented Apr 4, 2019 • edited Loading

jdblischak commented Apr 4, 2019

jdblischak commented Apr 4, 2019

BoPeng commented Apr 4, 2019 • edited Loading

BoPeng commented Apr 4, 2019

jdblischak commented Apr 4, 2019

BoPeng commented Apr 4, 2019 • edited Loading

jdblischak commented Apr 4, 2019

BoPeng commented Apr 4, 2019

jdblischak commented Apr 4, 2019

BoPeng commented Apr 4, 2019 • edited Loading

BoPeng commented Apr 4, 2019 • edited Loading

BoPeng commented Apr 4, 2019 • edited Loading

jdblischak commented Apr 5, 2019

jdblischak commented Apr 5, 2019

BoPeng commented Apr 20, 2019

chrisburr Apr 23, 2019

Choose a reason for hiding this comment

chrisburr Apr 23, 2019

Choose a reason for hiding this comment

chrisburr Apr 23, 2019

Choose a reason for hiding this comment

BoPeng Apr 23, 2019

Choose a reason for hiding this comment

BoPeng commented Mar 29, 2019 •

edited

Loading

BoPeng commented Apr 4, 2019 •

edited

Loading

BoPeng commented Apr 4, 2019 •

edited

Loading

BoPeng commented Apr 4, 2019 •

edited

Loading

BoPeng commented Apr 4, 2019 •

edited

Loading

BoPeng commented Apr 4, 2019 •

edited

Loading

BoPeng commented Apr 4, 2019 •

edited

Loading