The instruction are for: Caffe3D : Ubuntu, Caffe (2D) : Windows.
The Caffe3D author did it in python 2.7, Ubuntu 14.04, CUDNN 5.1, CUDA 8.0
Read the errors at the end before you continue.
And always make the Makefile.config yourself don't just copy paste. check for the folders carefully.
Install Caffe 3D:
Download https://lmb.informatik.uni-freiburg.de/resources/opensource/caffe_unet_3D_v1.0.tar.gz
Extract.
Open README:
git clone https://github.com/BVLC/caffe.git
cd caffe
git checkout 8c66fa5f3c04e -b unet_patch
git cherry-pick 458928a # typo in installation.md
git cherry-pick b43c8e4 # CuDNN 5 support
git apply ../caffe_unet_3D_v1.0.patch # apply patch
git add .
git commit -m"U-Net 3D merged to BVLC/caffe 8c66fa5f3c04e"
Then open:
https://github.com/tbuikr/3D_DenseSeg
Edit different files as mentioned there.
Now need to compile for linux now:
https://github.com/adeelz92/Install-Caffe-on-Ubuntu-16.04-Python-3
Error: libcudnn.so.5 is not a symbolic link
https://askubuntu.com/questions/1025928/
After installing cuDNN, copying the extracted files to /usr/lib/cuda/lib64 and creating the symlinks, things may go wrong with the symlinks.
So go to /usr/local/cuda/lib64/ and run ls -lha libcudnn*.
You should see two symlinks (bold teal) and one single file. Something like this:
The exact version of libcudnn.so.5.1.5 maybe be a little different for you (maybe libcudnn.so.5.1.10). In that case, adapt the code accordingly
/usr/local/cuda/lib64$ ls -lha libcudnn*
lrwxrwxrwx 1 root root 13 Dez 25 23:56 libcudnn.so -> libcudnn.so.5
lrwxrwxrwx 1 root root 17 Dez 25 23:55 libcudnn.so.5 -> libcudnn.so.5.1.5
-rwxr-xr-x 1 root root 76M Dez 25 23:27 libcudnn.so.5.1.5
If libcudnn.so and libcudnn.so.5 are not symlinks then this is the reason why you got this error. If so, this is what you need to do:
/usr/local/cuda/lib64$ sudo rm libcudnn.so
/usr/local/cuda/lib64$ sudo rm libcudnn.so.5
/usr/local/cuda/lib64$ sudo ln libcudnn.so.5.1.5 libcudnn.so.5
/usr/local/cuda/lib64$ sudo ln libcudnn.so.5 libcudnn.so
Run sudo ldconfig again and there should be no errors
What I do:
cd /usr/local/cuda-8.0/lib64
ls -lha libcudnn*
sudo mv libcudnn.so libcudnn.so-bak
sudo mv libcudnn.so.5 libcudnn.so.5-bak
sudo ln libcudnn.so.5.1.10 libcudnn.so.5
sudo ln libcudnn.so.5 libcudnn.so
sudo ldconfig
ls -lha libcudnn*
sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install libatlas-base-dev
sudo apt-get install libopenblas-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
Makefile link for python3.6:
https://drive.google.com/open?id=12ke1KEZoPY5JUPXJ18WK-3pT2OYJznMM
**********************************************************************
Working steps for python3.6
conda create --name caffe36 python=3.6
conda activate caffe36
cd /media/rb/Omega/Linux/caffe
cd python
for req in $(cat requirements.txt); do pip install --no-cache-dir $req; done\
cd ..
export PYTHONPATH=/media/rb/Omega/Linux/caffe/python
export CPLUS_INCLUDE_PATH=/home/rb/anaconda3/envs/caffe36/include/python3.6m
python -m site
/home/rb/anaconda3/envs/caffe36/lib/python3.6/site-packages
Edit Makefile.config
ANACONDA_HOME := $(HOME)/anaconda3/envs/caffe36
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
$(ANACONDA_HOME)/include/python3.6m \
$(ANACONDA_HOME)/lib/python3.6/site-packages/numpy/core/include \\
PYTHON_LIBRARIES := boost_python3 python3.6m
Then do: # 6 because I have 6 cpu cores - so parallel processing and fast.
make clean
make all -j6
make test -j6
So updated cuDNN to 6 and no it didn't help.
Source: https://stackoverflow.com/questions/47342267/
**********************************************************************
The below steps are for python3.5
**********************************************************************
cd /media/rb/Omega/Linux/caffe
So things as before. The patch to 3D and everything, if already done no need to do again.
cd python
for req in $(cat requirements.txt); do pip install --no-cache-dir $req; done
(Wait for it to finish installation)
cd ..
export PYTHONPATH=/media/rb/Omega/Linux/caffe/python
export CPLUS_INCLUDE_PATH=/home/rb/anaconda3/envs/caffe/include/python3.5m
Then to find where the python is:
python -m site
USER_SITE = /home/rb/anaconda3/envs/caffe/lib/python3.5/site-packages
cp Makefile.config.example Makefile.config
Now edit the Makefile.config:
Even if we have opencv 4 we can set opencv: 3 to compile.
uncomment USE_CUDNN := 1
CUDA_DIR := /usr/local/cuda is fine
Comment:
PYTHON_INCLUDE := /usr/include/python2.7 \
/usr/lib/python2.7/dist-packages/numpy/core/include
Add:
ANACONDA_HOME := $(HOME)/anaconda3/envs/caffe
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
# $(ANACONDA_HOME)/include/python3.5m \
# $(ANACONDA_HOME)/lib/python3.5/site-packages/numpy/core/include \
Comment:
PYTHON_LIB := /usr/lib
Uncomment:
PYTHON_LIB := $(ANACONDA_HOME)/lib
Uncomment:
WITH_PYTHON_LAYER := 1
For other edits check:
https://drive.google.com/open?id=1tEcA5a0gMIp3HNjXCefovmai6XoGejTA
First see all the edits and do steps accordingly. The errors are at the end of this page.
Then do: # 6 because I have 6 cpu cores - so parallel processing and fast.
make clean
make all -j6
make test -j6
$CAFFE_ROOT/media/rb/Omega/Linux/caffe/build/tools
sudo vim ~/.bashrc
export PYTHONPATH=/media/rb/Omega/Linux/caffe/python:$PYTHONPATH
$ sudo nano ~/.bashrc
<remove those lines>
To save the changes, please press Ctrl + o, press enter to accept the changes and Ctrl + x to close nano. Then, to reload the .bashrc file with the changes made, please enter the following command to the Terminal:
$ source ~/.bashrc
Caffe (2D) Windows:
https://github.com/BVLC/caffe/tree/windows
https://www.youtube.com/watch?v=SeCE757egcE
Caffe only supports Python 2.7 or Python 3.5 (3.7 doesn't work)
We will be using 3.5 Anaconda.
You can install anaconda with python 3.7 and create a conda environment with python 3.5
I already have Anaconda so I won't be installing it now.
We will need to install Visual Studio First.
Installing Visual Studio 2015
This is one of the supported VS versions by Caffe so we'll install this.
Online installer (I used):
https://drive.google.com/open?id=1HiiP-tL3nmAZ6SMAGtnf8iHhhAn1OiCY
Source: https://stackoverflow.com/questions/44290672
Community edition is free.
Online Installer: [https://go.microsoft.com/fwlink/?LinkId=532606&clcid=0x409]
Offline Installer: [https://go.microsoft.com/fwlink/?LinkId=615448&clcid=0x409]
The "official" way to get the vs2015 is to go to
https://my.visualstudio.com/
join the " Visual Studio Dev Essentials" and then
search the relevant file to download
https://my.visualstudio.com/Downloads?q=Visual%20Studio%202015%20with%20Update%203
[My download link contains the above.]
Also, [This may not be necessary]
Source: https://stackoverflow.com/questions/14590947/
So I go to C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\cl.exe and
Right-Click->Properties on cl.exe in your VS install directory (the exact path appears in the CMake error);
Choose the Compatibility Tab;
Check "Run this program as administrator" in the "Privilege Level" box.
Installing NVIDIA Driver [Skip this if you have driver already]
Now we will need to have NVIDIA Driver.
Latest driver will work. I have RTX 2070 and driver version 430.64
http://us.download.nvidia.com/Windows/430.64/430.64-desktop-win10-64bit-international-whql.exe
Source: https://www.nvidia.com/Download/index.aspx?lang=en-us#
Installing CUDA 8.0
Since out Visual Studio version is 2015 (or ver 14.x), we will need to install CUDA 8.0 (not 7.5) as suggested by Caffe.
Goto CUDA Archive: https://developer.nvidia.com/cuda-toolkit-archive
https://developer.nvidia.com/cuda-80-ga2-download-archive
CUDA Toolkit 8.0 GA2 (Feb 2017),
OS: Windows
Arch: x86_64
Version: 10 (Win 10)
Installer type: Local exe
Download click cuda_8.0.61_win10.exe and the patch: cuda_8.0.61.2_windows.exe
Important:
Visual Studio must be installed. "A supported version of Microsoft Visual Studio"
The redistributables (alone) do not satisfy that requirement.
Otherwise you will get: No supported version of Visual Studio was found.
VS2015 has toolchain version numbering like 14.x
VS2017 has toolchain version numbering like 15.x
CUDA 8 did not support VS 2017, so toolchain versions like 15.x are not supported.
So, need to install VS2015.
The commercial version of VS is not required. You can use VS community edition.
Open the cuda_8.0.61_win10.exe, click extract to whatever location (default is fine).
System Check will say "Graphics Driver could not find compatible graphics hardware.)
The installer doesn't recognize my RTX 2070, because the installer was created before RTX 2070 was a defined product. So just click continue.
After you click continue, you will be able to select options to deselect the driver install.
Select Custom(Advanced) and deselect Driver components. Then next.
Then install the patch: cuda_8.0.61.2_windows.exe
Installing cuDNN:
As suggested by Caffe, cuDNN v5 is necessary.
We download:
Goto cuDNN archive: https://developer.nvidia.com/rdp/cudnn-archive
Download cuDNN v5.1 (Jan 20, 2017), for CUDA 8.0
cuDNN v5.1 Library for Windows 10
We get: cudnn-8.0-windows10-x64-v5.1.zip
Then Extract.
Copy the folders bin, include and lib to
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0
and merge.
Building Caffe:
Most of the operations are done in the C:\Projects folder. Since I have space in C driver I'll create a folder Projects there. Any other folder should work as well.
Install Git and Cmake.
https://cmake.org/download/
Windows win64-x64 Installer
Add to path is recommended.
Install Anaconda.
Before moving forward I recommend you to restart your PC once.
Open Anaconda Prompt (as Administrator recommended)
cd C:\Projects
Let's create a conda env first:
conda remove -n caffe --all
conda create --name caffe python=3.5
activate caffe
Version 3.5 is necessary.
cmake -help should show something if cmake is installed.
git clone https://github.com/BVLC/caffe.git
git checkout windows
Edit options inside build_win.cmd
Mine is here: https://drive.google.com/open?id=1wRgd6d-BcUn5m97iABULijBHLHBTAb7u
Set
WITH_NINJA=0 (we won't be using NINJA, we are using Visual Studio instead.)
CPU_ONLY=0 (we will compile for GPU support)
PYTHON_VERSION=3 (we have python 3.5)
BUILD_PYTHON=1
BUILD_MATLAB=0 (you can try building matlab as well, for me there was some build error and I didn't need matlab so I skipped matlab build)
set CONDA_ROOT=C:\Users\rb\Anaconda3 (where your anaconda is located) change for python 3
add
-DCUDNN_ROOT=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0 ^
to cmake option. [Line 172]
Since our cuDNN is located in that folder.
If there is a build error somewhere, you can delete the build folder inside caffe and start building from the beginning.
Also goto caffe\cmake folder and edit Cuda.cmake
Mine is here: https://drive.google.com/open?id=1oi2GWtjaUGEAUps1noz5halVhTEBeAxy
Source:
https://github.com/BVLC/caffe/issues/6408
https://www.cnblogs.com/zjutzz/p/9532483.html
Comment #set(Caffe_known_gpu_archs "20 21(20) 30 35 50 60 61") and add
set(Caffe_known_gpu_archs "30 35 50 60 61")
20,21,20 creates a lot of warnings.
To avoid the error: Nvcc fatal : Unsupported gpu architecture 'compute_75'
Comment
#if(__nvcc_res EQUAL 0)
# # nvcc outputs text containing line breaks when building with MSVC.
# # The line below prevents CMake from inserting a variable with line
# # breaks in the cache
# string(REGEX MATCH "([1-9].[0-9])" __nvcc_out "${__nvcc_out}")
# string(REPLACE "2.1" "2.1(2.0)" __nvcc_out "${__nvcc_out}")
# set(CUDA_gpu_detect_output ${__nvcc_out} CACHE INTERNAL "Returned GPU architetures from caffe_detect_gpus tool" FORCE)
#endif()
and add after #endif():
set(__nvcc_out "7.5")
To manually set nvcc architecture version to 7.5.
7.5 is compute capability for RTX 2070. Find yours here: https://developer.nvidia.com/cuda-gpus
Also sometimes you might get the error: The dependency target "pycaffe" of target "pytest" does not exist.
Do this in the caffe enviornment ('caffe' activated):
pip install numpy
python -c "import numpy; print(numpy.version.version)"
If there is a build error anywhere, you can delete the build folder inside caffe and start building from the beginning.
Finally run:
scripts\build_win.cmd
After the build completes while within the caffe environment do:
conda config --add channels conda-forge
conda config --add channels willyd
conda install --yes cmake ninja numpy scipy protobuf==3.1.0 six scikit-image pyyaml pydotplus graphviz
Then copy your build files, for me, inside C:\Projects\caffe\python
to
C:\Users\rb\Anaconda3\envs\caffe\Lib\site-packages
Copy: caffe, classify.py, CMakeLists.txt, etc to the site-packages folder.
Now caffe is installed you can try:
python -c "import caffe; print(caffe.__version__)"
You can also add C:\Projects\caffe\build\tools\Release to your system path for rb. See below.
Extra: Install Ninja:
Download ninja.exe (https://github.com/ninja-build/ninja/releases)
Add to path:
In Search, search for and then select: System (Control Panel)
Click the Advanced system settings link.
Click Environment Variables. In the section System Variables, find the PATH environment variable and select it. Click Edit. If the PATH environment variable does not exist, click New.
In the Edit System Variable (or New System Variable) window, specify the value of the PATH environment variable. Click OK. Close all remaining windows by clicking OK.
Reopen Command prompt window
Extra:
ERRORS while building:
fatal error: pyconfig.h: No such file or directory
make clean
export CPLUS_INCLUDE_PATH=/home/rb/anaconda3/envs/caffe/include/python3.5m
make all
/usr/local/cuda/include/host_config.h:119:2: error: #error -- unsupported GNU version! gcc versions later than 5 are not supported!
sudo ln -s /usr/bin/gcc-7 /usr/local/cuda/bin/gcc
sudo ln -s /usr/bin/g++-7 /usr/local/cuda/bin/g++
To remove or unlink you can either use unlink or rm
sudo unlink /usr/local/cuda/bin/gcc
sudo unlink /usr/local/cuda/bin/g++
This didn't slove either so:
https://gist.github.com/wangruohui/679b05fcd1466bb0937f
To solve this problem we have to "hack" the CUDA toolkit by editing /usr/local/cuda/include/host_config.h. Comment line 115.
--- #error -- unsupported GNU version! gcc versions later than 4.9 are not supported!
+++ //#error -- unsupported GNU version! gcc versions later than 4.9 are not supported!
but this caused so many error: expected a ";" errors so I reversed it.
This solved the issue:
rb@rbhost:~$ ls /usr/bin/gcc*
/usr/bin/gcc /usr/bin/gcc-ar /usr/bin/gcc-nm /usr/bin/gcc-ranlib
/usr/bin/gcc-7 /usr/bin/gcc-ar-7 /usr/bin/gcc-nm-7 /usr/bin/gcc-ranlib-7
sudo apt install gcc-5 g++-5
rb@rbhost:~$ ls /usr/bin/gcc*
/usr/bin/gcc /usr/bin/gcc-ar /usr/bin/gcc-nm /usr/bin/gcc-ranlib
/usr/bin/gcc-5 /usr/bin/gcc-ar-5 /usr/bin/gcc-nm-5 /usr/bin/gcc-ranlib-5
/usr/bin/gcc-7 /usr/bin/gcc-ar-7 /usr/bin/gcc-nm-7 /usr/bin/gcc-ranlib-7
sudo ln -s /usr/bin/gcc-5 /usr/local/cuda/bin/gcc
sudo ln -s /usr/bin/g++-5 /usr/local/cuda/bin/g++
There are a lot of warnings saying
nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
You can remove it by removing below from the makefile config
-gencode arch=compute_20,code=sm_20 \
-gencode arch=compute_20,code=sm_21 \
For me I'm fine with the warnings.
F0521 07:12:15.545190 27987 layer_factory.hpp:81] Check failed: registry.count(type) == 1 (0 vs. 1) Unknown layer type: Bias (known types: )
*** Check failure stack trace: ***
@ 0x7f7c0e1490cd google::LogMessage::Fail()
@ 0x7f7c0e14af33 google::LogMessage::SendToLog()
@ 0x7f7c0e148c28 google::LogMessage::Flush()
@ 0x7f7c0e14b999 google::LogMessageFatal::~LogMessageFatal()
@ 0x555ee74ec0ad caffe::LayerRegistry<>::CreateLayer()
@ 0x7f7c0b9b2d25 caffe::ScaleLayer<>::LayerSetUp()
@ 0x555ee728c63f caffe::Layer<>::SetUp()
@ 0x555ee768aba4 caffe::ScaleLayerTest_TestForwardBroadcastMiddleWithParamAndBias_Test<>::TestBody()
@ 0x555ee776dfb3 testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x555ee77675aa testing::Test::Run()
@ 0x555ee77676f8 testing::TestInfo::Run()
@ 0x555ee77677d5 testing::TestCase::Run()
@ 0x555ee7768aaf testing::internal::UnitTestImpl::RunAllTests()
@ 0x555ee7768dd3 testing::UnitTest::Run()
@ 0x555ee7279d9d main
@ 0x7f7c0ab46b97 __libc_start_main
@ 0x555ee7281f6a _start
Makefile:523: recipe for target 'runtest' failed
make: *** [runtest] Aborted (core dumped)
cleaning and making again solved this.
fatal error: numpy/arrayobject.h: No such file or directory
sudo apt-get install python-numpy
solved this.
from numpy.lib.arraypad import _validate_lengths
ImportError: cannot import name '_validate_lengths'
pip install --upgrade scikit-image
If Python Layer doesn't work,
pip install --upgrade protobuf==3.1.0.post1
# https://blog.csdn.net/feifei884431/article/details/52232381
# https://github.com/yihui-he/caffe-dice-loss-layer/blob/master/perClassLoss.py
# https://github.com/Lasagne/Recipes/issues/99
# Segmentation Error Solved by: pip install --upgrade protobuf==3.1.0.post1
# https://github.com/BVLC/caffe/issues/5357
import caffe
import numpy as np
class DiceLossClass(caffe.Layer):
def setup(self, bottom, top):
# Parameter assignment for each layer after initialization
# Here the bottom[0] and [1] contains all zeros only no data
if len(bottom) != 2:
raise Exception("Need two inputs to compute distance.")
params = params = eval(self.param_str)
self.nclass = params["nclass"]
def reshape(self, bottom, top):
# Define size of output top blob
self.diff=np.zeros_like(bottom[1].data,dtype=np.float32)
top[0].reshape(1)
def forward(self, bottom, top):
self.diff[...] = bottom[0].data
label = bottom[0].data.astype(np.uint8)
onehot_label = self.one_hot(label, self.nclass).astype(np.float32) #one hot
intersection = (onehot_label * bottom[1].data).sum()
self.sum = onehot_label.sum() + bottom[1].data.sum() + 1e-19
self.dice_coeff = 2. * intersection / self.sum
dice_loss = 1 - self.dice_coeff
top[0].data[...] = dice_loss
def backward(self, top, propagate_down, bottom):
#to calculate the partial derivative of the bottom layer when the current layer is backward propagating
if propagate_down[0]:
raise Exception("label not diff")
elif propagate_down[1]:
bottom[1].diff[...] = (-2.*self.diff+self.dice_coeff)/self.sum
else:
raise Exception("no diff")
def one_hot(self, x, c):
re_shape = [x.shape[0]] + list(x.shape[2:])+[c]
tr_index = [0, x.ndim-1] + list(np.arange(1, x.ndim-1))
x = np.eye(c)[x.flatten()]
x = x.reshape(re_shape)
return x.transpose(tr_index) #HW(Z)C to CHW(Z)
The Caffe3D author did it in python 2.7, Ubuntu 14.04, CUDNN 5.1, CUDA 8.0
Read the errors at the end before you continue.
And always make the Makefile.config yourself don't just copy paste. check for the folders carefully.
Install Caffe 3D:
Download https://lmb.informatik.uni-freiburg.de/resources/opensource/caffe_unet_3D_v1.0.tar.gz
Extract.
Open README:
git clone https://github.com/BVLC/caffe.git
cd caffe
git checkout 8c66fa5f3c04e -b unet_patch
git cherry-pick 458928a # typo in installation.md
git cherry-pick b43c8e4 # CuDNN 5 support
git apply ../caffe_unet_3D_v1.0.patch # apply patch
git add .
git commit -m"U-Net 3D merged to BVLC/caffe 8c66fa5f3c04e"
Then open:
https://github.com/tbuikr/3D_DenseSeg
Edit different files as mentioned there.
Now need to compile for linux now:
https://github.com/adeelz92/Install-Caffe-on-Ubuntu-16.04-Python-3
Error: libcudnn.so.5 is not a symbolic link
https://askubuntu.com/questions/1025928/
After installing cuDNN, copying the extracted files to /usr/lib/cuda/lib64 and creating the symlinks, things may go wrong with the symlinks.
So go to /usr/local/cuda/lib64/ and run ls -lha libcudnn*.
You should see two symlinks (bold teal) and one single file. Something like this:
The exact version of libcudnn.so.5.1.5 maybe be a little different for you (maybe libcudnn.so.5.1.10). In that case, adapt the code accordingly
/usr/local/cuda/lib64$ ls -lha libcudnn*
lrwxrwxrwx 1 root root 13 Dez 25 23:56 libcudnn.so -> libcudnn.so.5
lrwxrwxrwx 1 root root 17 Dez 25 23:55 libcudnn.so.5 -> libcudnn.so.5.1.5
-rwxr-xr-x 1 root root 76M Dez 25 23:27 libcudnn.so.5.1.5
If libcudnn.so and libcudnn.so.5 are not symlinks then this is the reason why you got this error. If so, this is what you need to do:
/usr/local/cuda/lib64$ sudo rm libcudnn.so
/usr/local/cuda/lib64$ sudo rm libcudnn.so.5
/usr/local/cuda/lib64$ sudo ln libcudnn.so.5.1.5 libcudnn.so.5
/usr/local/cuda/lib64$ sudo ln libcudnn.so.5 libcudnn.so
Run sudo ldconfig again and there should be no errors
What I do:
cd /usr/local/cuda-8.0/lib64
ls -lha libcudnn*
sudo mv libcudnn.so libcudnn.so-bak
sudo mv libcudnn.so.5 libcudnn.so.5-bak
sudo ln libcudnn.so.5.1.10 libcudnn.so.5
sudo ln libcudnn.so.5 libcudnn.so
sudo ldconfig
ls -lha libcudnn*
sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install libatlas-base-dev
sudo apt-get install libopenblas-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
Makefile link for python3.6:
https://drive.google.com/open?id=12ke1KEZoPY5JUPXJ18WK-3pT2OYJznMM
**********************************************************************
Working steps for python3.6
conda create --name caffe36 python=3.6
conda activate caffe36
cd /media/rb/Omega/Linux/caffe
cd python
for req in $(cat requirements.txt); do pip install --no-cache-dir $req; done\
cd ..
export PYTHONPATH=/media/rb/Omega/Linux/caffe/python
export CPLUS_INCLUDE_PATH=/home/rb/anaconda3/envs/caffe36/include/python3.6m
python -m site
/home/rb/anaconda3/envs/caffe36/lib/python3.6/site-packages
Edit Makefile.config
ANACONDA_HOME := $(HOME)/anaconda3/envs/caffe36
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
$(ANACONDA_HOME)/include/python3.6m \
$(ANACONDA_HOME)/lib/python3.6/site-packages/numpy/core/include \\
PYTHON_LIBRARIES := boost_python3 python3.6m
Then do: # 6 because I have 6 cpu cores - so parallel processing and fast.
make clean
make all -j6
make test -j6
make runtest -j6
make pycaffe -j6
After install add export CAFFE_ROOT='/media/rb/Omega/Linux/caffe' to bashrc
This also showed error Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM
After install add export CAFFE_ROOT='/media/rb/Omega/Linux/caffe' to bashrc
This also showed error Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM
Source: https://stackoverflow.com/questions/47342267/
**********************************************************************
The below steps are for python3.5
**********************************************************************
cd /media/rb/Omega/Linux/caffe
So things as before. The patch to 3D and everything, if already done no need to do again.
cd python
for req in $(cat requirements.txt); do pip install --no-cache-dir $req; done
(Wait for it to finish installation)
cd ..
export PYTHONPATH=/media/rb/Omega/Linux/caffe/python
export CPLUS_INCLUDE_PATH=/home/rb/anaconda3/envs/caffe/include/python3.5m
Then to find where the python is:
python -m site
USER_SITE = /home/rb/anaconda3/envs/caffe/lib/python3.5/site-packages
cp Makefile.config.example Makefile.config
Now edit the Makefile.config:
Even if we have opencv 4 we can set opencv: 3 to compile.
uncomment USE_CUDNN := 1
CUDA_DIR := /usr/local/cuda is fine
Comment:
PYTHON_INCLUDE := /usr/include/python2.7 \
/usr/lib/python2.7/dist-packages/numpy/core/include
Add:
ANACONDA_HOME := $(HOME)/anaconda3/envs/caffe
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
# $(ANACONDA_HOME)/include/python3.5m \
# $(ANACONDA_HOME)/lib/python3.5/site-packages/numpy/core/include \
Comment:
PYTHON_LIB := /usr/lib
Uncomment:
PYTHON_LIB := $(ANACONDA_HOME)/lib
Uncomment:
WITH_PYTHON_LAYER := 1
For other edits check:
https://drive.google.com/open?id=1tEcA5a0gMIp3HNjXCefovmai6XoGejTA
First see all the edits and do steps accordingly. The errors are at the end of this page.
Then do: # 6 because I have 6 cpu cores - so parallel processing and fast.
make clean
make all -j6
make test -j6
make runtest -j6
make pycaffe -j6
$CAFFE_ROOT/media/rb/Omega/Linux/caffe/build/tools
sudo vim ~/.bashrc
export PYTHONPATH=/media/rb/Omega/Linux/caffe/python:$PYTHONPATH
$ sudo nano ~/.bashrc
<remove those lines>
To save the changes, please press Ctrl + o, press enter to accept the changes and Ctrl + x to close nano. Then, to reload the .bashrc file with the changes made, please enter the following command to the Terminal:
$ source ~/.bashrc
https://github.com/BVLC/caffe/tree/windows
https://www.youtube.com/watch?v=SeCE757egcE
Caffe only supports Python 2.7 or Python 3.5 (3.7 doesn't work)
We will be using 3.5 Anaconda.
You can install anaconda with python 3.7 and create a conda environment with python 3.5
I already have Anaconda so I won't be installing it now.
We will need to install Visual Studio First.
Installing Visual Studio 2015
This is one of the supported VS versions by Caffe so we'll install this.
Online installer (I used):
https://drive.google.com/open?id=1HiiP-tL3nmAZ6SMAGtnf8iHhhAn1OiCY
Source: https://stackoverflow.com/questions/44290672
Community edition is free.
Online Installer: [https://go.microsoft.com/fwlink/?LinkId=532606&clcid=0x409]
Offline Installer: [https://go.microsoft.com/fwlink/?LinkId=615448&clcid=0x409]
The "official" way to get the vs2015 is to go to
https://my.visualstudio.com/
join the " Visual Studio Dev Essentials" and then
search the relevant file to download
https://my.visualstudio.com/Downloads?q=Visual%20Studio%202015%20with%20Update%203
[My download link contains the above.]
Also, [This may not be necessary]
Source: https://stackoverflow.com/questions/14590947/
So I go to C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\cl.exe and
Right-Click->Properties on cl.exe in your VS install directory (the exact path appears in the CMake error);
Choose the Compatibility Tab;
Check "Run this program as administrator" in the "Privilege Level" box.
Installing NVIDIA Driver [Skip this if you have driver already]
Now we will need to have NVIDIA Driver.
Latest driver will work. I have RTX 2070 and driver version 430.64
http://us.download.nvidia.com/Windows/430.64/430.64-desktop-win10-64bit-international-whql.exe
Source: https://www.nvidia.com/Download/index.aspx?lang=en-us#
Installing CUDA 8.0
Since out Visual Studio version is 2015 (or ver 14.x), we will need to install CUDA 8.0 (not 7.5) as suggested by Caffe.
Goto CUDA Archive: https://developer.nvidia.com/cuda-toolkit-archive
https://developer.nvidia.com/cuda-80-ga2-download-archive
CUDA Toolkit 8.0 GA2 (Feb 2017),
OS: Windows
Arch: x86_64
Version: 10 (Win 10)
Installer type: Local exe
Download click cuda_8.0.61_win10.exe and the patch: cuda_8.0.61.2_windows.exe
Important:
Visual Studio must be installed. "A supported version of Microsoft Visual Studio"
The redistributables (alone) do not satisfy that requirement.
Otherwise you will get: No supported version of Visual Studio was found.
VS2015 has toolchain version numbering like 14.x
VS2017 has toolchain version numbering like 15.x
CUDA 8 did not support VS 2017, so toolchain versions like 15.x are not supported.
So, need to install VS2015.
The commercial version of VS is not required. You can use VS community edition.
Open the cuda_8.0.61_win10.exe, click extract to whatever location (default is fine).
System Check will say "Graphics Driver could not find compatible graphics hardware.)
The installer doesn't recognize my RTX 2070, because the installer was created before RTX 2070 was a defined product. So just click continue.
After you click continue, you will be able to select options to deselect the driver install.
Select Custom(Advanced) and deselect Driver components. Then next.
Then install the patch: cuda_8.0.61.2_windows.exe
Installing cuDNN:
As suggested by Caffe, cuDNN v5 is necessary.
We download:
Goto cuDNN archive: https://developer.nvidia.com/rdp/cudnn-archive
Download cuDNN v5.1 (Jan 20, 2017), for CUDA 8.0
cuDNN v5.1 Library for Windows 10
We get: cudnn-8.0-windows10-x64-v5.1.zip
Then Extract.
Copy the folders bin, include and lib to
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0
and merge.
Building Caffe:
Most of the operations are done in the C:\Projects folder. Since I have space in C driver I'll create a folder Projects there. Any other folder should work as well.
Install Git and Cmake.
https://cmake.org/download/
Windows win64-x64 Installer
Add to path is recommended.
Install Anaconda.
Before moving forward I recommend you to restart your PC once.
Open Anaconda Prompt (as Administrator recommended)
cd C:\Projects
Let's create a conda env first:
conda remove -n caffe --all
conda create --name caffe python=3.5
activate caffe
Version 3.5 is necessary.
cmake -help should show something if cmake is installed.
git clone https://github.com/BVLC/caffe.git
git checkout windows
Edit options inside build_win.cmd
Mine is here: https://drive.google.com/open?id=1wRgd6d-BcUn5m97iABULijBHLHBTAb7u
Set
WITH_NINJA=0 (we won't be using NINJA, we are using Visual Studio instead.)
CPU_ONLY=0 (we will compile for GPU support)
PYTHON_VERSION=3 (we have python 3.5)
BUILD_PYTHON=1
BUILD_MATLAB=0 (you can try building matlab as well, for me there was some build error and I didn't need matlab so I skipped matlab build)
set CONDA_ROOT=C:\Users\rb\Anaconda3 (where your anaconda is located) change for python 3
add
-DCUDNN_ROOT=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0 ^
to cmake option. [Line 172]
Since our cuDNN is located in that folder.
If there is a build error somewhere, you can delete the build folder inside caffe and start building from the beginning.
Also goto caffe\cmake folder and edit Cuda.cmake
Mine is here: https://drive.google.com/open?id=1oi2GWtjaUGEAUps1noz5halVhTEBeAxy
Source:
https://github.com/BVLC/caffe/issues/6408
https://www.cnblogs.com/zjutzz/p/9532483.html
Comment #set(Caffe_known_gpu_archs "20 21(20) 30 35 50 60 61") and add
set(Caffe_known_gpu_archs "30 35 50 60 61")
20,21,20 creates a lot of warnings.
To avoid the error: Nvcc fatal : Unsupported gpu architecture 'compute_75'
Comment
#if(__nvcc_res EQUAL 0)
# # nvcc outputs text containing line breaks when building with MSVC.
# # The line below prevents CMake from inserting a variable with line
# # breaks in the cache
# string(REGEX MATCH "([1-9].[0-9])" __nvcc_out "${__nvcc_out}")
# string(REPLACE "2.1" "2.1(2.0)" __nvcc_out "${__nvcc_out}")
# set(CUDA_gpu_detect_output ${__nvcc_out} CACHE INTERNAL "Returned GPU architetures from caffe_detect_gpus tool" FORCE)
#endif()
and add after #endif():
set(__nvcc_out "7.5")
To manually set nvcc architecture version to 7.5.
7.5 is compute capability for RTX 2070. Find yours here: https://developer.nvidia.com/cuda-gpus
Also sometimes you might get the error: The dependency target "pycaffe" of target "pytest" does not exist.
Do this in the caffe enviornment ('caffe' activated):
pip install numpy
python -c "import numpy; print(numpy.version.version)"
If there is a build error anywhere, you can delete the build folder inside caffe and start building from the beginning.
Finally run:
scripts\build_win.cmd
After the build completes while within the caffe environment do:
conda config --add channels conda-forge
conda config --add channels willyd
conda install --yes cmake ninja numpy scipy protobuf==3.1.0 six scikit-image pyyaml pydotplus graphviz
Then copy your build files, for me, inside C:\Projects\caffe\python
to
C:\Users\rb\Anaconda3\envs\caffe\Lib\site-packages
Copy: caffe, classify.py, CMakeLists.txt, etc to the site-packages folder.
Now caffe is installed you can try:
python -c "import caffe; print(caffe.__version__)"
You can also add C:\Projects\caffe\build\tools\Release to your system path for rb. See below.
Extra: Install Ninja:
Download ninja.exe (https://github.com/ninja-build/ninja/releases)
Add to path:
In Search, search for and then select: System (Control Panel)
Click the Advanced system settings link.
Click Environment Variables. In the section System Variables, find the PATH environment variable and select it. Click Edit. If the PATH environment variable does not exist, click New.
In the Edit System Variable (or New System Variable) window, specify the value of the PATH environment variable. Click OK. Close all remaining windows by clicking OK.
Reopen Command prompt window
Extra:
ERRORS while building:
fatal error: pyconfig.h: No such file or directory
make clean
export CPLUS_INCLUDE_PATH=/home/rb/anaconda3/envs/caffe/include/python3.5m
make all
/usr/local/cuda/include/host_config.h:119:2: error: #error -- unsupported GNU version! gcc versions later than 5 are not supported!
sudo ln -s /usr/bin/gcc-7 /usr/local/cuda/bin/gcc
sudo ln -s /usr/bin/g++-7 /usr/local/cuda/bin/g++
To remove or unlink you can either use unlink or rm
sudo unlink /usr/local/cuda/bin/gcc
sudo unlink /usr/local/cuda/bin/g++
This didn't slove either so:
https://gist.github.com/wangruohui/679b05fcd1466bb0937f
To solve this problem we have to "hack" the CUDA toolkit by editing /usr/local/cuda/include/host_config.h. Comment line 115.
--- #error -- unsupported GNU version! gcc versions later than 4.9 are not supported!
+++ //#error -- unsupported GNU version! gcc versions later than 4.9 are not supported!
but this caused so many error: expected a ";" errors so I reversed it.
This solved the issue:
rb@rbhost:~$ ls /usr/bin/gcc*
/usr/bin/gcc /usr/bin/gcc-ar /usr/bin/gcc-nm /usr/bin/gcc-ranlib
/usr/bin/gcc-7 /usr/bin/gcc-ar-7 /usr/bin/gcc-nm-7 /usr/bin/gcc-ranlib-7
sudo apt install gcc-5 g++-5
rb@rbhost:~$ ls /usr/bin/gcc*
/usr/bin/gcc /usr/bin/gcc-ar /usr/bin/gcc-nm /usr/bin/gcc-ranlib
/usr/bin/gcc-5 /usr/bin/gcc-ar-5 /usr/bin/gcc-nm-5 /usr/bin/gcc-ranlib-5
/usr/bin/gcc-7 /usr/bin/gcc-ar-7 /usr/bin/gcc-nm-7 /usr/bin/gcc-ranlib-7
sudo ln -s /usr/bin/gcc-5 /usr/local/cuda/bin/gcc
sudo ln -s /usr/bin/g++-5 /usr/local/cuda/bin/g++
There are a lot of warnings saying
nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
You can remove it by removing below from the makefile config
-gencode arch=compute_20,code=sm_20 \
-gencode arch=compute_20,code=sm_21 \
For me I'm fine with the warnings.
F0521 07:12:15.545190 27987 layer_factory.hpp:81] Check failed: registry.count(type) == 1 (0 vs. 1) Unknown layer type: Bias (known types: )
*** Check failure stack trace: ***
@ 0x7f7c0e1490cd google::LogMessage::Fail()
@ 0x7f7c0e14af33 google::LogMessage::SendToLog()
@ 0x7f7c0e148c28 google::LogMessage::Flush()
@ 0x7f7c0e14b999 google::LogMessageFatal::~LogMessageFatal()
@ 0x555ee74ec0ad caffe::LayerRegistry<>::CreateLayer()
@ 0x7f7c0b9b2d25 caffe::ScaleLayer<>::LayerSetUp()
@ 0x555ee728c63f caffe::Layer<>::SetUp()
@ 0x555ee768aba4 caffe::ScaleLayerTest_TestForwardBroadcastMiddleWithParamAndBias_Test<>::TestBody()
@ 0x555ee776dfb3 testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x555ee77675aa testing::Test::Run()
@ 0x555ee77676f8 testing::TestInfo::Run()
@ 0x555ee77677d5 testing::TestCase::Run()
@ 0x555ee7768aaf testing::internal::UnitTestImpl::RunAllTests()
@ 0x555ee7768dd3 testing::UnitTest::Run()
@ 0x555ee7279d9d main
@ 0x7f7c0ab46b97 __libc_start_main
@ 0x555ee7281f6a _start
Makefile:523: recipe for target 'runtest' failed
make: *** [runtest] Aborted (core dumped)
cleaning and making again solved this.
fatal error: numpy/arrayobject.h: No such file or directory
sudo apt-get install python-numpy
solved this.
Undefined symbol error while... importing caffe
So one mistake I did was forgot to uncomment two lines in:
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
# $(ANACONDA_HOME)/include/python3.5m \
# $(ANACONDA_HOME)/lib/python3.5/site-packages/numpy/core/include \
This might have caused the error when I included
PYTHON_LIBRARIES := boost_python3 python3.5m
and also maybe the reason I had to install python-numpy
But again I got the errors so maybe I should create a new env using python 3.6 and try again because ,y /usr/lib/x86_64-linux-gnu had python3.6 maybe because ubuntu 18.04? check older versions of ubuntu. If you have to install ubuntu install 16.04
Also: I did this to get rid of the symbol error.
Symbol error was caused due to incompatibility between diff python versions.
no need to do ln -s below for python 3.6 and no need to install python-numpy for python3.6
cd /usr/lib/x86_64-linux-gnu
sudo rm -rf libboost_python.so
sudo ln -s libboost_python-py35.so libboost_python.so
sudo ldconfig
Uncommented PYTHON_LIBRARIES := boost_python3 python3.5m
cd /usr/lib/x86_64-linux-gnu
sudo mv libboost_python.so libboost_python.so-bak
sudo mv libboost_python3.so libboost_python3.so-bak
sudo ln -s libboost_python-py36.so libboost_python.so
sudo ln -s libboost_python-py36.so libboost_python3.so
sudo ln -s libpython3.6m.so.1.0 libpython3.5m.so
sudo mv libboost_python.so-bak libboost_python.so
sudo mv libboost_python3.so-bak libboost_python3.so
Check failed: error == cudaSuccess (30 vs. 0) unknown error
Maybe restart helped or maybe I did this and it helped to remove driver issues.
https://unix.stackexchange.com/questions/295241/
Run
dpkg --get-selections | awk '/i386/{print $1}'
And then if happy with them being removed, run
sudo apt-get remove --purge `dpkg --get-selections | awk '/i386/{print $1}'`
https://github.com/NVIDIA/DIGITS/issues/1663
I have solved this problem with
sudo ldconfig /usr/local/cuda/lib64
rb@rbhost:/media/rb/Omega/tensorflow/caffe/3D_DenseSeg$ sudo ldconfig /usr/local/cuda/lib64
/sbin/ldconfig.real: /usr/local/cuda/lib64/libcudnn.so.5 is not a symbolic link
rb@rbhost:/media/rb/Omega/tensorflow/caffe/3D_DenseSeg$ cd /usr/local/cuda/lib64/
rb@rbhost:/usr/local/cuda/lib64$ ls -lha libcudnn*
-rwxr-xr-x 1 root root 81M May 23 08:24 libcudnn.so
-rwxr-xr-x 1 root root 81M May 23 08:24 libcudnn.so.5
-rwxr-xr-x 1 root root 81M May 23 08:24 libcudnn.so.5.1.10
-rw-r--r-- 1 root root 68M May 23 08:24 libcudnn_static.a
rb@rbhost:/usr/local/cuda/lib64$ ^C
rb@rbhost:/usr/local/cuda/lib64$ sudo rm libcudnn.so
rb@rbhost:/usr/local/cuda/lib64$ sudo rm libcudnn.so.5
rb@rbhost:/usr/local/cuda/lib64$ ls -lha libcudnn*
-rwxr-xr-x 1 root root 81M May 23 08:24 libcudnn.so.5.1.10
-rw-r--r-- 1 root root 68M May 23 08:24 libcudnn_static.a
rb@rbhost:/usr/local/cuda/lib64$ sudo ln libcudnn.so.5.1.10 libcudnn.so.5
rb@rbhost:/usr/local/cuda/lib64$ sudo ln libcudnn.so.5 libcudnn.so
rb@rbhost:/usr/local/cuda/lib64$ sudo ldconfig
So one mistake I did was forgot to uncomment two lines in:
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
# $(ANACONDA_HOME)/include/python3.5m \
# $(ANACONDA_HOME)/lib/python3.5/site-packages/numpy/core/include \
This might have caused the error when I included
PYTHON_LIBRARIES := boost_python3 python3.5m
and also maybe the reason I had to install python-numpy
But again I got the errors so maybe I should create a new env using python 3.6 and try again because ,y /usr/lib/x86_64-linux-gnu had python3.6 maybe because ubuntu 18.04? check older versions of ubuntu. If you have to install ubuntu install 16.04
Also: I did this to get rid of the symbol error.
Symbol error was caused due to incompatibility between diff python versions.
no need to do ln -s below for python 3.6 and no need to install python-numpy for python3.6
cd /usr/lib/x86_64-linux-gnu
sudo rm -rf libboost_python.so
sudo ln -s libboost_python-py35.so libboost_python.so
sudo ldconfig
Uncommented PYTHON_LIBRARIES := boost_python3 python3.5m
cd /usr/lib/x86_64-linux-gnu
sudo mv libboost_python.so libboost_python.so-bak
sudo mv libboost_python3.so libboost_python3.so-bak
sudo ln -s libboost_python-py36.so libboost_python.so
sudo ln -s libboost_python-py36.so libboost_python3.so
sudo ln -s libpython3.6m.so.1.0 libpython3.5m.so
sudo mv libboost_python.so-bak libboost_python.so
sudo mv libboost_python3.so-bak libboost_python3.so
Check failed: error == cudaSuccess (30 vs. 0) unknown error
Maybe restart helped or maybe I did this and it helped to remove driver issues.
https://unix.stackexchange.com/questions/295241/
Run
dpkg --get-selections | awk '/i386/{print $1}'
And then if happy with them being removed, run
sudo apt-get remove --purge `dpkg --get-selections | awk '/i386/{print $1}'`
sudo dpkg --remove-architecture i386
Above worked.
This didn't fork for unknown error:https://github.com/NVIDIA/DIGITS/issues/1663
I have solved this problem with
sudo ldconfig /usr/local/cuda/lib64
rb@rbhost:/media/rb/Omega/tensorflow/caffe/3D_DenseSeg$ sudo ldconfig /usr/local/cuda/lib64
/sbin/ldconfig.real: /usr/local/cuda/lib64/libcudnn.so.5 is not a symbolic link
rb@rbhost:/media/rb/Omega/tensorflow/caffe/3D_DenseSeg$ cd /usr/local/cuda/lib64/
rb@rbhost:/usr/local/cuda/lib64$ ls -lha libcudnn*
-rwxr-xr-x 1 root root 81M May 23 08:24 libcudnn.so
-rwxr-xr-x 1 root root 81M May 23 08:24 libcudnn.so.5
-rwxr-xr-x 1 root root 81M May 23 08:24 libcudnn.so.5.1.10
-rw-r--r-- 1 root root 68M May 23 08:24 libcudnn_static.a
rb@rbhost:/usr/local/cuda/lib64$ ^C
rb@rbhost:/usr/local/cuda/lib64$ sudo rm libcudnn.so
rb@rbhost:/usr/local/cuda/lib64$ sudo rm libcudnn.so.5
rb@rbhost:/usr/local/cuda/lib64$ ls -lha libcudnn*
-rwxr-xr-x 1 root root 81M May 23 08:24 libcudnn.so.5.1.10
-rw-r--r-- 1 root root 68M May 23 08:24 libcudnn_static.a
rb@rbhost:/usr/local/cuda/lib64$ sudo ln libcudnn.so.5.1.10 libcudnn.so.5
rb@rbhost:/usr/local/cuda/lib64$ sudo ln libcudnn.so.5 libcudnn.so
rb@rbhost:/usr/local/cuda/lib64$ sudo ldconfig
from numpy.lib.arraypad import _validate_lengths
ImportError: cannot import name '_validate_lengths'
pip install --upgrade scikit-image
If Python Layer doesn't work,
pip install --upgrade protobuf==3.1.0.post1
# https://blog.csdn.net/feifei884431/article/details/52232381
# https://github.com/yihui-he/caffe-dice-loss-layer/blob/master/perClassLoss.py
# https://github.com/Lasagne/Recipes/issues/99
# Segmentation Error Solved by: pip install --upgrade protobuf==3.1.0.post1
# https://github.com/BVLC/caffe/issues/5357
import caffe
import numpy as np
class DiceLossClass(caffe.Layer):
def setup(self, bottom, top):
# Parameter assignment for each layer after initialization
# Here the bottom[0] and [1] contains all zeros only no data
if len(bottom) != 2:
raise Exception("Need two inputs to compute distance.")
params = params = eval(self.param_str)
self.nclass = params["nclass"]
def reshape(self, bottom, top):
# Define size of output top blob
self.diff=np.zeros_like(bottom[1].data,dtype=np.float32)
top[0].reshape(1)
def forward(self, bottom, top):
self.diff[...] = bottom[0].data
label = bottom[0].data.astype(np.uint8)
onehot_label = self.one_hot(label, self.nclass).astype(np.float32) #one hot
intersection = (onehot_label * bottom[1].data).sum()
self.sum = onehot_label.sum() + bottom[1].data.sum() + 1e-19
self.dice_coeff = 2. * intersection / self.sum
dice_loss = 1 - self.dice_coeff
top[0].data[...] = dice_loss
def backward(self, top, propagate_down, bottom):
#to calculate the partial derivative of the bottom layer when the current layer is backward propagating
if propagate_down[0]:
raise Exception("label not diff")
elif propagate_down[1]:
bottom[1].diff[...] = (-2.*self.diff+self.dice_coeff)/self.sum
else:
raise Exception("no diff")
def one_hot(self, x, c):
re_shape = [x.shape[0]] + list(x.shape[2:])+[c]
tr_index = [0, x.ndim-1] + list(np.arange(1, x.ndim-1))
x = np.eye(c)[x.flatten()]
x = x.reshape(re_shape)
return x.transpose(tr_index) #HW(Z)C to CHW(Z)