No Detection or Tracking

[MuhammadAsadJaved]

Hi,
I was running your project in Xavier NX, after a few tries, I was able to run it successfully. But there are these problems.
1- There is no detection nor tracking. See attached image 1.
2- The process throws errors and stopped after processing a few frames. See attached image 2.
Note: I thought it might be NX environment problem, then I tested it in GTX 1080 Ti, but still have the same problems.
I also tested mainGoturnTracker.py as well. It is also not tracking the object. So seems like it is a problem in the tracker.

[MuhammadAsadJaved]

I just checked by saving output video, in the start of the video the detector detects the drone but only for one or two frames and then disappeared and no more detection or tracking.

Update: I have verified mainDetect.py and the detector is detecting drone very well. So I think the problem is only in the Detection+Tracking code. Can you please check the problem?

[berkantay]

Hey @MuhammadAsadJaved congrats you build and test the repo succesfully. Have you got any document for installation guideline or tutorial for the demo? For instance, jetpack version, opencv version, caffe version etc. and installation scripts for all of these.

Best Regards

[MuhammadAsadJaved]

Hi @berkantay

Here are the details:

• Device: Jetson Xavier NX (You can try TX2 or TX1)
• Jetpack version: 4.4 (I flashed new NX with SD card and it's default jetpack version) (Other versions should also work)
• OpenCV: 3.4.0 (First you can use default installed OpenCV version or install with pip3 if does not work then build from source using (https://jkjung-avt.github.io/opencv3-on-tx2/))
• Caffe: 1.1.0 build from source (https://jkjung-avt.github.io/caffe-on-tx2/)
• Pytorch: 1.0.0 (install pre-build wheel for Jetson TX2 ) Download version 1.0.0 according to your JetPack version from the given link (https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-6-0-now-available/72048)
  and Install using

sudo pip3 install   torch-1.0.0a0+bb15580-cp36-cp36m-linux_aarch64.whl

• Then when you run the project some small libraries may be missing, for the missing error message you can simply install those libraries with

sudo pip3 install   nameOfTheLibraray

If you got any problem during the installation process, feel free to post here.

Good luck.

[suatberkantgulen]

Hi @MuhammadAsadJaved

Thanks for information about installation steps,

Im getting error while build opencv from source: make[1]: *** [modules/cudev/CMakeFiles/opencv_cudev.dir/all] Error 2

Have you faced this problem?

I follow this document step by step: https://jkjung-avt.github.io/opencv3-on-tx2/
and im working with JetPack4.4

Thanks

[MuhammadAsadJaved]

@suatberkantgulen Unfortunately I did not found this problem during installation.

The error message says that make is not able to find the file '/usr/lib/aarch64-linux-gnu/libGL.so'. Check if there is a '/usr/lib/aarch64-linux-gnu/libGL.so.1.0.0' on your Jetson . If yes, you should be able to fix the problem by creating a symbolic link using

$ sudo ln -sf libGL.so.1.0.0 /usr/lib/aarch64-linux-gnu/libGL.so

You can change the libGL version accordingly.

[suatberkantgulen]

I see.

But /usr/lib/aarch64-linux-gnu/libGL.so already exist. But document reference for path under tegra:

cd /usr/lib/aarch64-linux-gnu/
sudo ln -sf tegra/libGL.so libGL.so

I have only one file about libGL which is libGLX_nvidia.so.0

So i use this command: sudo ln -sf tegra/libGLX_nvidia.so.0 libGL.so

It works but i am not sure that is a correct way.

Thank you

[suatberkantgulen]

Hi again :)

Firstly thanks for helps. i could build opencv 3.4 from source.

I get another error in the second step When i build caffe.
My reference : https://jkjung-avt.github.io/caffe-on-tx2/

Error:

Makefile:591: recipe for target '.build_release/src/caffe/layers/cudnn_conv_layer.o' failed

I think problem is about the cudnn version check. Because JetPack 4.4 does not include version info in cudnn.h. Its inside cudnn_version.h. I add version info from cudnn_version.h to cudnn.h. But stil getting same error.

By the way which cuda version are you using?

[MuhammadAsadJaved]

@suatberkantgulen Please do not skip any step from that installation method.

Now try this one and build again:

sudo apt-get install libboost-dev libboost-all-dev

• He also provides his Makefile copy in the tutorial, you can download and compare your with that one and verify if there is any mistake or not. (https://jkjung-avt.github.io/assets/2017-08-08-caffe-on-tx2/Makefile.config)

[suatberkantgulen]

Packs already installed. I compare again but i can not find any differences between 2 file

Here is my configuration of makefile:

## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers
# USE_OPENCV := 0
# USE_LEVELDB := 0
# USE_LMDB := 0
# This code is taken from https://github.com/sh1r0/caffe-android-lib
USE_HDF5 := 0

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
#	You should not set this flag if you will be reading LMDBs with any
#	possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3
OPENCV_VERSION := 3

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_62,code=sm_62 \
                -gencode arch=compute_62,code=compute_62

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas

# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
# PYTHON_INCLUDE := /usr/include/python2.7 \
		# /usr/lib/python2.7/dist-packages/numpy/core/include

# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
		# $(ANACONDA_HOME)/include/python2.7 \
		# $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include

# Uncomment to use Python 3 (default is Python 2)
PYTHON_LIBRARIES := boost_python-py36 python3.6m #boost_python3
PYTHON_INCLUDE := /usr/include/python3.6m \
                  /usr/local/lib/python3.6/dist-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/lib
# PYTHON_LIB := $(ANACONDA_HOME)/lib

# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)
WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/aarch64-linux-gnu /usr/lib/aarch64-linux-gnu/hdf5/serial

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib

# NCCL acceleration switch (uncomment to build with NCCL)
# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)
# USE_NCCL := 1

# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1

# N.B. both build and distribute dirs are cleared on `make clean`
BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
Q ?= @

[gzc-123456-gzc]

@MuhammadAsadJaved Is your video the one posted by the author on youtube？Has your main.GoturnTracker.py been verified successfully? Your main.detect can only detect for a while or there is no problem. Can you share it if the operation is successful? Thank you

[gzc-123456-gzc]

@MuhammadAsadJaved In detectandtrack, the first line videopath=none, I don’t know if you noticed

[MuhammadAsadJaved]

@gzc-123456-gzc No. I am using a different video, I remember I have verified the detector, it was working well but the tracker was not working. mainDetect.py detecting continuously without any failure if the drone is not too far.

[gzc-123456-gzc]

@MuhammadAsadJaved Thank you for the video link, I think it may be the problem of the target tracking weight file, you can try to retrain it

[gzc-123456-gzc]

@MuhammadAsadJaved Hello, is your problem solved? Is it a weight file issue? Or is it a code problem? If it is solved, can you tell us the reason for the inoperability?

[MuhammadAsadJaved]

@gzc-123456-gzc Not resolved yet. Have same problem. Are you able to use it without any problem? I think it's not a weight problem. it's something wrong with the tracking code.

[gzc-123456-gzc]

@MuhammadAsadJaved Not resolved yet. Maybe this question needs to be asked to the author, and the author should know the problem if it runs successfully. Also, I don't know why the .weights files in many demo files are used differently.

[berkantay]

could you please share your .weights file

[MuhammadAsadJaved]

@MuhammadAsadJaved Not resolved yet. Maybe this question needs to be asked to the author, and the author should know the problem if it runs successfully. Also, I don't know why the .weights files in many demo files are used differently.

Yes you are right. @gzc-123456-gzc

Which one? Yolo or GoTurn? @berkantay
Yolo weights are already available in the project in the yolo/model/ directory.
GoTurn weights file is available [here]

[gzc-123456-gzc]

@MuhammadAsadJaved
Have you run the file DetectAndTrack.py? You look at this code carefully

    if DetectFlag:
        ### If detection => initialize tracking and switch state to "Tracking"
        bbox = detecBBox[0]
        ### If you want to keep a list of all detection uncomment next line.
        ##self.detectionList.add(self.frameCount-1,bbox, confidence)
        ### Initialize tracker
        self.objTracker.init(frame, bbox, self.objRegressor)
        self.State_Flag ="Tracking"
        if confidence[0] < self.doubleCheckThres:
            ### If the confidence is smaller than "doubleCheckThres" => recheck next frame if it is a drone indeed
            self.time_trackingWV = self.VerificationInterval
        else:
            self.time_trackingWV = 0

    return DetectFlag, detecBBox, confidence

You can try add a comment on bbox = detecBBox[0] and delete comment on self.detectionList.add(self.frameCount-1,bbox, confidence)

[MuhammadAsadJaved]

@gzc-123456-gzc The DetectAndTrack.py is only for some pre-processing I think. It is not showing any output. I have tried. It's imported in main main_Detect_Track. I am only running 'sh runme.sh` .

I have updated pytorch version for another project. I will try it again.
Is it working well after your said changes?
If yes then can you post the procedure you followed?

Just change comment bbox = detecBBox[0] and uncomment self.detectionList.add(self.frameCount-1,bbox, confidence) and then run sh runme.sh ? or anything else?

[gzc-123456-gzc]

@MuhammadAsadJaved Hey, this is just my guess. I didn't run it successfully, and various environmental errors have caused me to be overwhelmed. But I don’t want to give up, I want to try。Also, which project is the other project you mentioned?

[MuhammadAsadJaved]

Those changes do not make any difference. Just checked again.
mainDetect.py is working well. The yolov3 is detecting very well. So I am sure there is something wrong with the tracker's code. We can try to review it. Once you run it successfully, You can also focus on it.

For GoTurn I do not know any other project. I also contact the coder of this project but he did not reply i also contacted his lab and teacher if they can update this project.

The second option is Yolov3+DeepSort
https://github.com/GeekAlexis/FastMOT

[MuhammadAsadJaved]

@ALL

Hi friends, Is there anyone able to track successfully? I have tried to contact the author on LinkedIn as well but he is not replying. seems like he is not managing this project anymore. I found one problem that the detector and tracker both working but when It go to the tracking part, the tracking is tracking but not drawing the bounding boxes on the right place.

These two pictures are from the same run. As you can see at the start the detector detects the object see the massage on window YOLO DETECTION and then passed to the tracker, the window also has massage GOTURN TRACKING and FPS information. So seems like the only problem is not drawing the bounding box after tracking. So we can try to figure it out if the author did not reply.