Issue with running torch2caffe/torch2caffe.lua

[vimalthilak]

Hi,

Thanks for providing an open torch2caffe conversion tool. I encountered the following error when I executed torch2caffe.lua:

th torch2caffe/torch2caffe.lua --help                                                                                                                       
/home/vimal/torch/install/bin/luajit: /home/vimal/torch/install/share/lua/5.1/trepl/init.lua:383: ./torch2caffe/lib.lua:19: Python error: import
ImportError: No module named torch2caffe.lib_py

stack traceback:
        [C]: in function 'error'
        /home/vimal/torch/install/share/lua/5.1/trepl/init.lua:383: in function 'require'
        torch2caffe/torch2caffe.lua:9: in main chunk
        [C]: in function 'dofile'
        ...imal/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x00406670

Any idea what I might be missing in my set-up? I tried to add an init.py but that's leading me to a completely different error

/home/vimal/torch/install/bin/luajit: /home/vimal/torch/install/share/lua/5.1/trepl/init.lua:383: ./torch2caffe/lib.lua:19: Python error: import
Traceback (most recent call last):
  File "/home/vimal/work/fb-caffe-exts/torch2caffe/lib_py.py", line 17, in <module>
    import caffe
  File "/home/vimal/work/gh-caffe/python/caffe/__init__.py", line 1, in <module>
    from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
  File "/home/vimal/work/gh-caffe/python/caffe/pycaffe.py", line 13, in <module>
    from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
ImportError: dlopen: cannot load any more object with static TLS

stack traceback:
        [C]: in function 'error'
        /home/vimal/torch/install/share/lua/5.1/trepl/init.lua:383: in function 'require'
        torch2caffe/torch2caffe.lua:9: in main chunk
        [C]: in function 'dofile'
        ...imal/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x00406670

Any help is greatly appreciated! Thanks.

[vimalthilak]

I worked around this issue by using LD_PRELOAD to load libcaffe shared object. I did encounter other issues such as unimplemented layers (LRN) which I was able to handle as well. I'm going to close this issue out and open a separate issue for other questions. Thanks!

[ajtulloch]

Hi Vimalthilak,

Sorry I missed this - yes, this uses fblualib's Python, so you need to ensure that pycaffe, libcaffe.so is on PYTHONPATH, etc.

For the additional layers - we're essentially adding them as-needed. I need to sync up and add a few new ones (cudnn.SpatialBatchNormalization, nn.CAddTable I think are the only additions). LRN would be cool - I assume you mean cudnn.SpatialCrossMapLRN?

[vimalthilak]

Hi @ajtulloch,

Thanks for the response!

• I assume you mean cudnn.SpatialCrossMapLRN?

It can be cudnn.SpatialCrossMapLRN or nn.SpatialCrossMapLRN. This layer is easy enough that I hacked around lib.lua and lib_py.py (IIRC) to get these to work.

By any chance, have you had a chance to port the model in https://github.com/soumith/inception.torch to caffe.

A port of inception-v3 would be nice as well but that requires batch normalization layer. Not sure if it's doable as-is.

I am happy to provide any help, if doable, if that helps with your effort to develop and add features to and/or test this library.

Thanks!

[fhdiaze]

Hi,

Where did you find torch2caffe.lib? I am getting the next error:

/usr/local/bin/luajit: torch2caffe.lua:9: module 'torch2caffe.lib' not found:No LuaRocks module found for torch2caffe.lib

Thanks

[vimalthilak]

@fhdiaze It's the lib module under torch2caffe. I think you need to ensure that lib under /torch2caffe is in the path somewhere.

[fhdiaze]

@vimalthilak thanks for the response, which file do you mean? all? I do not have so much experience with lua.

Thanks.

[ajtulloch]

It should work if you have the torch2caffe directory somewhere in package.path in Lua.

[fhdiaze]

Hi,

I am doing:

package.path = package.path .. ';/home/fhdiaze/Code/fb-caffe-exts/torch2caffe/?.lua

and after that

local t2c = require 'torch2caffe.lib'

But I am getting the next error:

[string "local t2c = require 'torch2caffe.lib'..."]:1: module 'torch2caffe.lib' not found:No LuaRocks module found for torch2caffe.lib

Do I have to install using luarocks?

Thanks

[ajtulloch]

Does something like http://stackoverflow.com/questions/18125775/lua-require-fails-to-find-submodule-but-searchpath-succeeds work?

[ngaloppo]

@ajtulloch We're trying to convert the pretrained Resnet Torch models here with torch2caffe but running into a problem where SpatialBatchNormalization layer is not supported. Do you have any update on adding support for that? How would we go about doing that?

[ajtulloch]

Sure, I need to resync with our internal branch. Give me a few hours.

[ajtulloch]

Sure, I need to resync with our internal branch. Give me a few hours.

[ngaloppo]

@ajtulloch 👍 thanks!

[ngaloppo]

@ajtulloch Do you have any update on the resync? Thanks!

[J6767]

I have got the test running now (penlight was causing problems), but all are failing:

Using `cudnn`   
>>>>>>>>> Test  
>>> Test:testAlexnet    

Failed  
>>> Test:testConvolution    

Failed  
>>> Test:testDropout    

Failed  
>>> Test:testFBThreshold    

Failed  
>>> Test:testGoogLeNet  

Failed  
>>> Test:testInception  

Failed  
>>> Test:testInceptionComponents    
checking model  
nn.Sequential {
  [input -> (1) -> (2) -> (3) -> output]
  (1): nn.SpatialConvolution(3 -> 1, 1x1)
  (2): nn.Dropout(0.500000)
  (3): nn.ReLU
}

Failed  
>>> Test:testLinear 

Failed  
>>> Test:testParallelModel  

Failed  
>>> Test:testReLU   

Failed  
>>> Test:testSequential 

Failed  
>>> Test:testSpatialConvolutionMM   

Failed  
>>> Test:testSpatialMaxPooling  

Failed  
>>> Test:testVggCombined    

Failed  
>>> Test:testVggConv    

Failed  
>>> Test:testVggLinear  

Failed  
>>> Test:testView   

Failed   

The error is due to:
t2c.run(opts, module) --attempt to call field 'run' (a nil value)

If anyone knows the root of this problem, please post. Thanks.

[KevinYuk]

HI @ajtulloch, have you resync with your internal branch? It seems that is still report "Unknown layer type: nn.SpatialBatchNormalization".
thanks.

[MFarooqAit]

Dear ajtulloch
i have the same error...can you help me in this regard

/usr/bin/luajit: /usr/share/lua/5.1/trepl/init.lua:384: /usr/share/lua/5.1/trepl/init.lua:384: module 'fb.python' not found:No LuaRocks module found for fb.python
no field package.preload['fb.python']
no file '/tmp/buildd/.luarocks/share/lua/5.1/fb/python.lua'
no file '/tmp/buildd/.luarocks/share/lua/5.1/fb/python/init.lua'
no file '/usr/share/lua/5.1/fb/python.lua'
no file '/usr/share/lua/5.1/fb/python/init.lua'
no file './fb/python.lua'
no file '/usr/share/luajit-2.1.0-beta1/fb/python.lua'
no file '/usr/local/share/lua/5.1/fb/python.lua'
no file '/usr/local/share/lua/5.1/fb/python/init.lua'
no file '/home/siraj/.luarocks/share/lua/5.1/fb/python.lua'
no file '/home/siraj/.luarocks/share/lua/5.1/fb/python/init.lua'
no file '/tmp/buildd/.luarocks/lib/lua/5.1/fb/python.so'
no file '/usr/lib/lua/5.1/fb/python.so'
no file './fb/python.so'
no file '/usr/local/lib/lua/5.1/fb/python.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
no file '/home/siraj/.luarocks/lib/lua/5.1/fb/python.so'
no file '/tmp/buildd/.luarocks/lib/lua/5.1/fb.so'
no file '/usr/lib/lua/5.1/fb.so'
no file './fb.so'
no file '/usr/local/lib/lua/5.1/fb.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
no file '/home/siraj/.luarocks/lib/lua/5.1/fb.so'
stack traceback:
[C]: in function 'error'
/usr/share/lua/5.1/trepl/init.lua:384: in function 'require'
main.lua:8: in main chunk
[C]: in function 'dofile'
/usr/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670
Error using run (line 38)
Failed to run Torch7 script.

[msiraj83]

@ajtulloch I have follow your steps... I compile Caffe with pycaffe and insert the path in .bashrc file. But now i got this error...

Can you help me in this regards..

Thanks