[Deep learning] Gradient Descent Optimizer

이미지
https://icim.nims.re.kr/post/easyMath/70 알기 쉬운 산업수학 알기 쉬운 산업수학 Gradient Descent Optimizer 2017년 12월 13일(수) | 김민중 URL  (1) 주어진 목적함수  f f 의 최솟값(minimum)을 찾아가는 알고리즘으로 다음과 같은 방식으로 최솟값을 찾아간다. 초기값  x 0 x 0 을 임의로 설정, 적당한 Learning rate  α α  설정 n ≥ 0 n ≥ 0 인 정수에 대해서  x n + 1 x n + 1 은 다음과 같이 정의한다. x n + 1 : = x n − α ⋅ ∇ f ( x n ) x n + 1 := x n − α ⋅ ∇ f ( x n ) 주의사항 함수 f f 의 모양이 convex가 아닌 경우 global minimum이 아닌 local minimum으로  x n x n 이 수렴할 가능성이 있다. Learning rate  α α  값이 큰 경우 최솟값으로  x n x n 이 수렴하는 것이 아니라 발산할 수 있다. Learning rate  α α  값이 작은 경우 수렴하는 속도가 지나치게 느릴 수 있다. 방정식  2 ⋅ x = 10 2 ⋅ x = 10  의 근을 Gradient Descent를 이용해서 찾아보자. 목적함수  f ( x ) : = ( 10 − 2 x ) 2 f ( x ) := ( 10 − 2 x ) 2 으로 설정하겠다( f f 의 최솟값인 0이 되게 하는  x x 값이 우리가 원하는 방정식의 근이다). 초기값  x 0 = 0 x 0 = 0 , Learning rate  α = 0.05 α = 0.05 으로 설정 f ′ ( x ) = 4 ( 2 x − 10 ) f ′ ( x ) = 4 ( 2 x − 10 ) 이므로  x 1 x 1 은 다음과 같이 구할 수 있다. x 1 = x 0 − α ⋅ f ′ ( x 0 ) = 0 − 0.05 ⋅ ( − 40 ) = 2 x

Nvidia graphic driver install for deep learning ( CUDA Toolkit and cuDNN )

This is my experienced guide line to install Nvidia driver. 
GPU: GM107M [Geforce GTX 950M] 
OS : Ubuntu 16.04
System : Lenovo ideated 700 151sk
Note : dual graphics card ( intel , nvidia )

There are three prerequisite conditions.
A> Root privilege is needed.
B> X-Server must be stopped.
C> Neuveau driver must be removed.
D> Secure boot to legacy[CSM] from Bios.



0. If you have installed nvidia graphics card driver then goto  number 1 else 2.

1. Remove old graphic card dirvers. 

$ sudo apt-get remove nvidia* && sudo apt-get autoremove


2. Disable neuveau kernel dirver for install nvidia driver.


$ sudo nano /etc/modprobe.d/blacklist-nouveau.conf

Add the following lines and save it.

blacklist nouveau
blacklist lbm-nouveau
options nouveau modeset=0
alias nouveau off
alias lbm-nouveau off


3.  Stop using neuveau driver from initial ram file system (Disable from kernel) and reboot system.


   $echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
    $update-initramfs -u

출처: http://iostream1029.tistory.com/1 [#include ]
$ echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
$ sudo update-initramfs -u
$ reboot
   $echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
    $update-initramfs -u

출처: http://iostream1029.tistory.com/1 [#include ]
$echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
    $update-initramfs -u

출처: http://iostream1029.tistory.com/1 [#include ]


4. Stop x-server (Display manager)

This command must run on virtual terminal ( Ctrl + Alt + F1 or  - F6 )
$ sudo service lightdm stop


 5. Install cuda_toolkit

The latest release of the CUDA Toolkit is v8.0.(5B)
I think that v7.0 is comparably stable version. (5A)

Before doing that someone should install required packages.

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install build-essential cmake git unzip pkg-config
$ sudo apt-get install libopenblas-dev liblapack-dev
$ sudo apt-get install linux-image-generic linux-image-extra-virtual linux-source linux-headers-generic

5A. Install cuda_toolkik v7.0


5A-1. Download CUDA Toolkit v7.0.




$ wget http://developer.download.nvidia.com/compute/cuda/7_0/Prod/local_installers/cuda_7.0.28_linux.run


5A-2.  Extract and install it.

$ chmod +x cuda_7.0.28_linux.run
$ mkdir cuda_toolkit
$ sudo ./cuda_7.0.28_linux.run -extract="this is your path"/cuda_toolkit
$ cd cuda_toolkit
$ sudo ./NVIDIA-Linux-x86_64-346.46.run --no-opengl-libs ( this is for dual graphics card )
$ modprobe nvidia
$ sudo ./cuda-linux64-rel-7.0.28-19326674.run
$ sudo ./cuda-samples-linux-7.0.28-19326674.run


5A-3. Setting up deep learning environment.

$ nano ~/.bashrc

Add the following lines and save it.

# CUDA Toolkit v7.0
export CUDA_HOME=/usr/local/cuda-7.0
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH
export PATH=${CUDA_HOME}/bin:${PATH}


5A-4. Let's test our installation.

$ source  ~/.bashrc
$ cd /usr/local/cuda/samples/1_Utilities/deviceQuery
$ sudo make
$ ./deviceQuery

After running deviceQuery, your output should have  "
Result = PASS" line.

5A-5. Clean up setup files.
$ cd "this is your cuda_toolkit path which has setup files"
$ cd ..
$ rm -rf cuda_toolkit cuda_7.0.28_linux.run


5B. Install cuda_toolkikv v8.0



5B-1. Download CUDA toolkit v8.0

($ sudo apt-get install -y wget )
$ wget  https://developer.nvidia.com/compute/cuda/8.0/prod/local_installers/cuda_8.0.44_linux-run

5B-2. Extract and install it

$ chmod +x cuda_8.0.44_linux-run
$ mkdir cuda_toolkit
$ sudo ./cuda_8.0.44_linux-run -extract="this is your path"/cuda_toolkit
$ cd cuda_toolkit
$ sudo ./NVIDIA-Linux-x86_64-367.48.run --no-opengl-files ( this is for dual graphics card , usually laptop use this chipset. )
$ modprobe nvidia
$ sudo ./cuda-linux64-rel-8.0.44-21122537.run
$ sudo ./cuda-samples-linux-8.0.44-21122537.run


5B-3. Settng up following lines and save it

$ nano ~/.bashrc

Add the following lines and save it.

# CUDA Toolkit v8.0
export CUDA_HOME=/usr/local/cuda-8.0
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH
export PATH=${CUDA_HOME}/bin:${PATH}


5B-4. Let's test our installation.

$ source  ~/.bashrc
$ cd /usr/local/cuda-8.0/samples/1_Utilities/deviceQuery
$ sudo make
$ ./deviceQuery

After running deviceQuery, your output should have  "
Result = PASS" line.

5B-5. Clean up setup files.
$ cd "this is your cuda_toolkit path which has setup files"
$ cd ..
$ rm -rf  cuda_8.0.44_linux.run


5C. About errors while installing.

5C-1. Secure boot problem

After run NVIDIA-Linux-x86_64-367.48.run you should see below message.

"The target kernel has CONFIG_MODULE_SIG set, which means that it supports cryptographic signatures on kernel modules. On some systems, the kernel may refuse to load modules without a valid signature from a trusted key. This system also has UEFI Secure Boot enabled; many distributions enforce module signature verification on UEFI systems when Secure Boot is enabled. Would you like to sign the NVIDIA kernel module?"

Then reboot and enter bios setup and then change your boot option.
Usually UEFI -> Legacy  but some bios UEFI -> CSM.

This blogger has the same problem. (http://askubuntu.com/questions/816168/nvidia-gt730-driver-problem-with-ubuntu-16-04/841377)


5C-2. login loop problem (Can't login after Nvidia driver install)

step1> Reinstall unity in ubuntu
$ sudo su
$ apt-get update
$ apt-get install --reinstall ubuntu-desktop
$ apt-get install unity
$ reboot
step2> Reinstall graphics card driver in ubuntu
$ sudo su 
$ apt-get update
$ apt-get install --reinstall ubuntu-desktop
$ apt-get install unity
$ apt-get remove --purge nvidia*
$ reboot
step3> Still do not work
$ sudo apt-get install nvidia-current
$ sudo reboot
step4> Uninstall old driver and install new one.
$ sudo apt-get remove --purge nvidia-* 
<for 32bit system>
$ cd && wget us.download.nvidia.com/XFree86/Linux-x86/331.38/NVIDIA-Linux-x86-331.38.run
<for 64bit system>
$ cd && wget us.download.nvidia.com/XFree86/Linux-x86_64/331.38/NVIDIA-Linux-x86_64-331.38.run
<finally>
$ chmod +x NVIDIA-Linux-*-331.38.run
$ sudo sh NVIDIA-Linux-*-331.38.run
$ sudo reboot
This is from here (Jiakai Zhang)


6. Install cuDNN


6-1. Register  accelerated computing developer program

6-2. Download cuDNN (Nvidia CUDA Deep Neural Network Library)
   
       We will use cuDNN v5.1 Library for Linux


6-3. Extract and copy ( replacing the filename and path as necessary )

$ cd ~
$ tar -zxf cudnn-8.0-linux-x64-v5.1-tgz
$ cd cuda
$ sudo cp lib64/* /usr/local/cuda/lib64/
$ sudo cp include/* /usr/local/cuda/include/

6-4. Clean up files

$ cd ~
$ rm -rf cuda
$ rm -f cudnn-8.0-linux-x64-v5.1-tgz




댓글

이 블로그의 인기 게시물

@import' when modules are disabled

MagicaVoxel black screen

[Iphone][build error] pngcrush caught libpng error: PNG unsigned integer out of range.