Merge pull request #6 from MIT-TESSE/0.1.1-SNAPSHOT

0.1.1 snapshot

.gitignore

@@ -1,3 +1,7 @@
 simulator/
 
 __pycache__
+
+*.pyc
+.vscode/
+logs/*

Dockerfile-ppo

@@ -0,0 +1,35 @@
+###################################################################################################
+# DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.
+#
+# This material is based upon work supported by the Under Secretary of Defense for Research and
+# Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions
+# or recommendations expressed in this material are those of the author(s) and do not necessarily
+# reflect the views of the Under Secretary of Defense for Research and Engineering.
+#
+# (c) 2020 Massachusetts Institute of Technology.
+#
+# MIT Proprietary, Subject to FAR52.227-11 Patent Rights - Ownership by the contractor (May 2014)
+#
+# The software/firmware is provided to you on an As-Is basis
+#
+# Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS Part 252.227-7013
+# or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this work
+# are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other
+# than as specifically authorized by the U.S. Government may violate any copyrights that exist in
+# this work.
+###################################################################################################
+
+from goseek-base:latest
+
+RUN apt-get update && \
+    apt-get install python-opencv -y && \
+    pip install stable-baselines && \
+    conda install tensorflow-gpu==1.13.1
+
+WORKDIR /goseek-challenge
+
+COPY baselines/agents.py baselines/agents.py
+
+COPY baselines/config/ppo-agent.yaml agent.yaml
+
+COPY ppo-weights.pkl ppo-weights.pkl

Instructions.md

@@ -46,32 +46,32 @@ Note that we recommend installing in [development mode](https://setuptools.readt
 See [below](#training) for further discussion.
 
 
-2. Clone this repository and install requirements.
+3. Clone this repository and install requirements.
 
 ```sh
 git clone https://github.com/MIT-TESSE/goseek-challenge.git
 cd goseek-challenge
 ```
 
 
-3. Next, you need to obtain GOSEEK simulator. Execute the following:
+4. Next, you need to obtain GOSEEK simulator. Execute the following:
 ```sh
 mkdir -p simulator
-wget https://github.com/MIT-TESSE/goseek-challenge/releases/download/0.1.0/goseek-v0.1.0.zip -P simulator
-unzip simulator/goseek-v0.1.0.zip -d simulator
-chmod +x simulator/goseek-v0.1.0.x86_64
+wget https://github.com/MIT-TESSE/goseek-challenge/releases/download/0.1.0/goseek-v0.1.3.zip -P simulator
+unzip simulator/goseek-v0.1.3.zip -d simulator
+chmod +x simulator/goseek-v0.1.3.x86_64
 ```
 
 This creates a new `simulator` folder, download and unzips the simulator to that folder, and makes the simulator executable.
-Note that if you choose to place the simulator in an alternative location, you will need to specify the location in a configuration file that overrides the default value such as in [config/ground-truth.yaml](config/ground-truth.yaml).
+Note that if you choose to place the simulator in an alternative location, you will need to specify the location in a configuration file that overrides the default value such as in [config/check-ground-truth.yaml](config/check-ground-truth.yaml).
 
-4. Test your installation by running a random agent. The agent receives observations and takes random actions:
+5. Test your installation by running a random agent. The agent receives observations and takes random actions:
 
 ```sh
-python eval.py --agent-config baselines/config/random-agent.yaml
+python eval.py --agent-config baselines/config/random-agent.yaml --episode-config config/check-ground-truth.yaml
 ```
 
-5. Next, build a docker image called `goseek-base`, which is needed to submit online solutions.
+6. Next, build a docker image called `goseek-base`, which is needed to submit online solutions.
 
 ```sh
 cd docker/goseek-base/
@@ -85,7 +85,29 @@ docker run --rm -it goseek-base /bin/bash -c "python eval.py --help"
 ```
 
 
-__NOTE__: In order to run the __Perception Pipeline__, you will need another docker image with [Kimera](https://github.com/MIT-SPARK/Kimera). Directions for building this image (named `goseek-kimera`) will be posted at a later time.
+7. In order to run the __Perception Pipeline__, you will need another docker image with [Kimera](https://github.com/MIT-SPARK/Kimera). Directions for building this image (named `goseek-kimera`) are as follows. 
+Please note that building this image is not required to submit a solution for the competition; it is only used for any local testing and/or training with the __Perception Pipeline__.
+We strongly encourage participants to review [further details](doc/perception-pipeline.md) on the perception pipeline, as well.
+
+    a. First, obtain [TensorRT](https://developer.nvidia.com/nvidia-tensorrt-download). 
+    Note that you'll need to create an Nvidia account and manually download it because TensorRT is proprietary.
+    Select `TensorRT-6.0.1.5.Ubuntu-18.04.x86_64-gnu.cuda-10.0.cudnn7.6.tar.gz` from the Nvidia TensorRT 
+    download page and **place the binary in `goseek-challenge/docker/goseek-kimera/`**.
+
+    b. Build the docker image.
+    Note that this takes quite a while.
+    (Go watch a movie? :man_shrugging:)
+
+    ```sh
+    docker build --rm -t goseek-kimera .
+    ```
+
+    c. And, optionally again, run the following to verify the docker image.
+    It should return the list of directory contents in [tesse_gym_bridge](https://www.github.com/MIT-TESSE/tesse-gym-bridge).
+
+    ```sh
+    docker run --rm -it goseek-kimera /bin/bash -c "source /catkin_ws/devel/setup.bash && rosls tesse_gym_bridge"
+    ```
 
 
 ## Usage
@@ -126,7 +148,7 @@ class Agent:
         (1, 240, 320, 3)
         >>> segmentation.shape
         (1, 240, 320, 3)
-        >>> rgb.shape
+        >>> depth.shape
         (1, 240, 320, 3)
         >>> pose.shape
         (1, 3)
@@ -214,17 +236,31 @@ optional arguments:
 
 For example, you can run the following to test against __Ground Truth__ data source:
 ```sh
-python test_locally.py -s simulator/goseek-v0.1.0.x86_64 -i submission -g
+python test_locally.py -s simulator/goseek-v0.1.3.x86_64 -i submission -g
 ```
 
+#### Submitting docker image 
+
+As mentioned in [competition overview](README.md), we are using the [EvalAI](https://evalai.cloudcv.org/) platform to host our submission server. To upload a solution, please follow the instructions below: 
 
+
+1. Install [EvalAI-CLI](https://evalai-cli.cloudcv.org/): `pip install evalai`.
+
+2. Create on account on EvalAI's [website](https://evalai.cloudcv.org/) and sign up for the [GOSEEK-Challenge](https://evalai.cloudcv.org/web/challenges/challenge-page/607/overview).
+
+3. Follow the instructions on the [submission](https://evalai.cloudcv.org/web/challenges/challenge-page/607/submission) tab to push your docker image.
+Note that we've provided four phases -- some to support development. 
+Only the leader of the **Competition Phase with Perception Pipeline** will be declared the competition winner.
 ## Examples
 
+
 See any of the following for additional information and examples.
 
 - [Baseline Proximal Policy Optimization (PPO)](doc/ppo-baseline.md)
 - [Additional problem details](doc/details.md)
-
+- [Instructions for running headless on a linux server](https://github.com/MIT-TESSE/tesse-core#running-tesse-headless)
+- [Generating configuration files](doc/generate.md)
+- [Details on the Perception Pipeline](doc/perception-pipeline.md)
 
 
 ## Disclaimer

README.md

@@ -69,8 +69,15 @@ The current timeline for the competition is as follows:
 Over the course of the competition, any important announcements or updates will be listed in this section.
 We recommend that you follow this repository to be alerted to these announcements.
 
-1. We've delayed the opening of submissions in light of [ICRA 2020](https://icra2020.org/)'s status, and will continue to track its status. 
-We do not anticipate that any future statements from the planning committee will change the current timeline of this challenge.
+1. We've posted version 0.1.3 of the challenge simulator.
+This provides better support for the __Perception Pipeline__ and addresses a minor bug.
+Please download this new simulator, if you were using 0.1.0 before.
+The link can be found in the [instructions](Instructions.md).
+2. We've also updated [tesse-gym](https://github.com/MIT-TESSE/tesse-gym) since our initial release to support the __Perception Pipeline__.
+You should update your clone of `tesse-gym` from the [instructions](Instructions.md).
+Please also rebuild the `goseek-base` docker image, as well as any of your submission images.
+3. We continue to track the status of [ICRA 2020](https://icra2020.org/). 
+We do not anticipate that any future statements from the planning committee will change the timeline of this challenge.
 
 ## Getting Started
 
@@ -102,14 +109,22 @@ Use `test_locally.py` for local testing.
 
 Assume you've named your docker image `submission` as above, then evaluate your agent with __Ground Truth__ data as follows.
 ```sh
-python test_locally.py -s simulator/goseek-v0.1.0.x86_64 -i submission -g
+python test_locally.py -s simulator/goseek-v0.1.3.x86_64 -i submission -g
 ```
 
-__NOTE__: Instructions for testing your agent with the __Perception Pipeline__ will be posted shortly.
+Similarly, evaluate your agent with __Perception Pipeline__ data as follows.
+```sh
+python test_locally.py -s simulator/goseek-v0.1.3.x86_64 -i submission -p
+```
+
+### Submit online
 
-### Submit online.
+1. Install [EvalAI-CLI](https://evalai-cli.cloudcv.org/): `pip install evalai`.
 
-__NOTE__: Instructions for submitting agents online will be available according to the competition timeline [above](#timeline).
+2. Create on account on EvalAI's [website](https://evalai.cloudcv.org/) and sign up for the [GOSEEK-Challenge](https://evalai.cloudcv.org/web/challenges/challenge-page/607/overview).
+3. Follow the instructions on the [submission](https://evalai.cloudcv.org/web/challenges/challenge-page/607/submission) tab to push your docker image.
+Note that we've provided four phases -- some to support development. 
+Only the leader of the **Competition Phase with Perception Pipeline** will be declared the competition winner.
 
 ## Acknowledgements
 

baselines/config/ppo-agent.yaml

@@ -1,2 +1,2 @@
 name: StableBaselinesPPO
-weights: ''  # MODEL WEIGHT PATH
+weights: 'ppo-weights.pkl'  # PUT WEIGHTS AT GOSEEK-CHALLENGE ROOT

config/check-ground-truth.yaml

@@ -0,0 +1,8 @@
+ENV:
+   sim_path: 'simulator/goseek-v0.1.3.x86_64'
+
+EPISODE:
+   scenes: [3, 5]
+   n_targets: [30, 30]
+   episode_length: [400, 400]
+   random_seeds: [10, 100]

config/ground-truth.yaml

@@ -5,4 +5,4 @@ EPISODE:
   scenes: [3, 5]
   n_targets: [30, 30]
   episode_length: [400, 400]
-  random_seeds: [10, 100]
+  random_seeds: [10, 100]

config/perception-pipeline.yaml

@@ -5,7 +5,7 @@ ENV:
   image_port: 9008
 
 EPISODE:
-  scenes: [3, 5]
-  n_targets: [30, 30]
-  episode_length: [400, 400]
-  random_seeds: [10, 100]
+  scenes: [3, 4, 5, 5]
+  n_targets: [30, 30, 30, 30]
+  episode_length: [400, 400, 400, 400]
+  random_seeds: [10, 1, 5, 6]

doc/details.md

@@ -23,7 +23,7 @@ They don't spawn in other rooms (e.g., restroom, breakroom).
 
 ### Semantics
 
-There are 10 semantic categories for objects as follows.
+There are 11 semantic categories for objects as follows.
 - 0: floor (0,171,143)
 - 1: ceiling (1,155,211)
 - 2: wall (2,222,110)
@@ -52,4 +52,4 @@ MIT Proprietary, Subject to FAR52.227-11 Patent Rights - Ownership by the contra
 
 The software/firmware is provided to you on an As-Is basis
 
-Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other than as specifically authorized by the U.S. Government may violate any copyrights that exist in this work.
+Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other than as specifically authorized by the U.S. Government may violate any copyrights that exist in this work.

doc/generate.md

@@ -0,0 +1,26 @@
+# Generating configuration files
+
+The follow example generates a configuration file with 10 episodes such as those in [../config](../config).
+If you find this useful, please adapt for your own use.
+
+```py
+import random
+import yaml
+
+k = 10  # number of episodes to generate
+l = 400  # episode length
+f = 30  # number of fruit
+scenes = [1, 2, 3, 4, 5]  # scenes (1 through 5 are provided)
+
+data = {}
+data['ENV'] = {'sim_path': ''}
+
+data['EPISODE'] = {'scenes': random.choices(scenes, k=k),
+                   'n_targets': [f for _ in range(k)],
+                   'episode_length': [l for _ in range(k)],
+                   'random_seeds': [random.randint(1, 1e6) for _ in range(k)]
+                   }
+
+with open('example.yaml', 'w') as outfile:
+    yaml.dump(data, outfile)
+```