记一次Ubuntu18.04显卡驱动, CUDA, cuDNN的安装以及开源框架Yolov3的编译

发表于 2019-08-03 分类于折腾日志 Disqus：本文字数： 3.6k 阅读时长 ≈ 13 分钟

这段时间看到一个yolo的基于深度学习的开源的物体检测库，由于在笔记本上跑（集显）, 实时帧数只有0.1fps..小白的我才刚知道深度学习要N卡且安装cuda.

我那台都已经被我遗忘了有显卡存在的“集显”台式机终于可以派上用场了.

(English version translate by GPT-3.5)

准备

拥有Cuda和cuDNN支持的Nvidia的显卡, 一般GTX 9x, GTX 10x, RTX 20x都支持的.
我用Ubuntu 18.04 Desktop版本
Cuda + cuDNN安装时参考文档：真实机下 ubuntu 18.04 安装GPU +CUDA+cuDNN 以及其版本选择（亲测非常实用）
OpenCV安装参考文档：Build OpenCV 2.4.9 & Caffe with CUDA 9.0 on Ubuntu 16.04
Yolo 基于深度学习的物体检测, 使用v3版本, GitHub - AlexeyAB/darknet

说明

系统初始的安装, 包括apt更新, 这些就不阐述了

建议将源改为国内源（下面提供阿里云的源）

deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse

显卡驱动, Cuda, OpenCV安装

安装显卡驱动程序

这里我踩坑很久, 我翻了很多资料都说, 大多说需要什么卸载原有驱动, 然后禁用驱动, 但是新装的Ubuntu, 似乎并不需要这些步骤. 似乎ubuntu有非常方便的命令, 一行装驱动.

首先, 让Ubuntu检测下机器的显卡, 它会返回建议安装的驱动

1	ubuntu-drivers devices

会返回如下的内容

ruter@RuterPC:~$  ubuntu-drivers devices
== /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0 ==
modalias : pci:v000010DEd***************bc03sc00i00
vendor   : NVIDIA Corporation
model    : GP104 [GeForce GTX 1080]
driver   : nvidia-driver-430 - distro non-free recommended
driver   : nvidia-driver-390 - distro non-free
driver   : xserver-xorg-video-nouveau - distro free builtin

可以看到我这个显卡, 建议安装的驱动是nvidia-driver-430, 然后就是一键命令安装驱动, 基本傻瓜式.

安装图形驱动程序
1
sudo ubuntu-drivers autoinstall
这么做可以安装所有支持的驱动, 然后就等他好了.当然，也可以只安装上面推荐的430, 这里我选择执行上面的命令
1
sudo apt install nvidia-driver-430
安装完毕后建议重启下
1
sudo reboot

重启之后, 可以使用nvidia-smi命令来查看, 如果成功显示出来的话, 就表示安装成功了.

ruter@RuterPC:~$ nvidia-smi 
Fri Aug  2 19:16:21 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.26       Driver Version: 430.26       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:01:00.0  On |                  N/A |
|  0%   46C    P0    46W / 200W |    272MiB /  8116MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1109      G   /usr/lib/xorg/Xorg                           144MiB |
|    0      1246      G   /usr/bin/gnome-shell                         125MiB |
+-----------------------------------------------------------------------------+

同时系统设置里面 - Details也能看到显卡已经被识别出来了

用Ungine Heaven进行跑分, 看看显卡是否装上去了可以从这里下载 Heaven - UNIGINE Benchmarks

看跑分, 确定已经安装好了.

安装CUDA

CUDA(Compute Unified Device Architecture)是NVIDIA推出的用于自家GPU的并行计算框架, 可以通过该框架使用GPU来加速程序

CUDA® is a parallel computing platform and programming model developed by NVIDIA for general computing on graphical processing units (GPUs). With CUDA, developers are able to dramatically speed up computing applications by harnessing the power of GPUs.

从CUDA官网下载最新的包, 从官网中可以看到现在最新的是CUDA Toolkit 10.1 update1, 那就下这个好了, 这里建议使用Network的方式安装, 因为这么装相当于由apt完成整个安装过程, 会根据所需下载需要的包, 而不是下载2.5GB的文件.

点击Download 2.9KB`下载这个文件，放在Ubuntu的某个目录中, 然后按官网提示, 分别执行下面1，2，3，4 这4条命令(由于网络安装速度很慢, 可以考虑使用代理安装, 或者根据真实机下 ubuntu 18.04 安装GPU +CUDA+cuDNN 以及其版本选择（亲测非常实用）使用run文件安装)

sudo dpkg -i cuda-repo-ubuntu1804_10.1.168-1_amd64.deb
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
sudo apt-get update
sudo apt-get install cuda

慢慢下载, 可以看到文件很大（可以考虑给apt使用代理, 代理设置可以参考 Ubuntu的apt-get代理设置）

 libice-dev libnvidia-common-418 libnvidia-compute-418 libnvidia-decode-418
  libnvidia-encode-418 libnvidia-fbc1-418 libnvidia-gl-418 libnvidia-ifr1-418
  libopengl0 libpthread-stubs0-dev libsm-dev libx11-dev libx11-doc
  libx11-xcb-dev libxau-dev libxcb-dri2-0-dev libxcb-dri3-dev libxcb-glx0-dev
  libxcb-present-dev libxcb-randr0-dev libxcb-render0-dev libxcb-shape0-dev
  libxcb-sync-dev libxcb-xfixes0-dev libxcb1-dev libxdamage-dev libxdmcp-dev
  libxext-dev libxfixes-dev libxi-dev libxmu-dev libxmu-headers
  libxshmfence-dev libxt-dev libxxf86vm-dev mesa-common-dev
  nvidia-compute-utils-418 nvidia-dkms-418 nvidia-driver-418
  nvidia-kernel-common-418 nvidia-kernel-source-418 nvidia-modprobe
  nvidia-utils-418 openjdk-11-jre openjdk-11-jre-headless x11proto-core-dev
  x11proto-damage-dev x11proto-dev x11proto-fixes-dev x11proto-input-dev
  x11proto-xext-dev x11proto-xf86vidmode-dev xorg-sgml-doctools
  xserver-xorg-video-nvidia-418 xtrans-dev
The following packages will be upgraded:
  nvidia-settings
1 upgraded, 124 newly installed, 0 to remove and 1 not upgraded.
Need to get 1,671 MB of archives.  // 足足需要下载1.6GB, 但是比run的2.5GB要小多了, 如果没代理, 可能比下2.5G慢得多
After this operation, 4,245 MB of additional disk space will be used.
Do you want to continue? [Y/n]

耐心等待安装完毕

0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

done.
done.
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for fontconfig (2.12.6-0ubuntu2) ...
Processing triggers for mime-support (3.60ubuntu1) ...
Processing triggers for desktop-file-utils (0.23-1ubuntu3.18.04.2) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
W: Operation was interrupted before it could finish
ruter@RuterPC:~$

将cuda的lib添加到环境变量中

编辑 ~/.bashrc 添加以下内容

1 2	export PATH=/usr/local/cuda-10.1/bin${PATH:+:$PATH}} export LD_LIBRARY_PATH=/usr/local/cuda-10.1/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

然后保存

ruter@RuterPC:/usr/local$ vi ~/.bashrc 
ruter@RuterPC:/usr/local$ source ~/.bashrc 
添加上面的内容
ruter@RuterPC:/usr/local$ nvcc -V // 发现命令ok了
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Apr_24_19:10:27_PDT_2019
Cuda compilation tools, release 10.1, V10.1.168
ruter@RuterPC:/usr/local$

添加cuDNN

cuDNN全程是 CUDA Deep Neural Network library, 是Nvida CUDA的深度神经网络库, 用于加速深度神经网络GPU加速库

The NVIDIA CUDA® Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks.

从官网下载cuDNN（该步骤需要登陆, 没有nvidia账号的注册一个就行）
因为我装的是10.1的cuda, 因此这里下载需要下cuDNN 7.6.2版本的

下载好后进行解压

ruter@RuterPC:~/Downloads$ tar -zxvf cudnn-10.1-linux-x64-v7.6.2.24.tgz 
cuda/include/cudnn.h
cuda/NVIDIA_SLA_cuDNN_Support.txt
cuda/lib64/libcudnn.so
cuda/lib64/libcudnn.so.7
cuda/lib64/libcudnn.so.7.6.2
cuda/lib64/libcudnn_static.a
ruter@RuterPC:~/Downloads$

会解压出一堆so文件

执行以下命令（cuDNN解压出来的目录是cuda, 进入这个目录）

1
2
3

sudo cp include/cudnn.h /usr/local/cuda/include
sudo cp lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn.h   /usr/local/cuda/lib64/libcudnn*

至此, cuda和cudnn安装完毕

编译OpenCV

我这里选择的OpenCV版本是2.4.13, 因为上面文章中2.4.9我之前安装过有错误..啥错误忘了..
从OpenCV官网中下载 OpenCV – 2.4.13.6 - Sources 94.5MB.
然后根据Build OpenCV 2.4.9 & Caffe with CUDA 9.0 on Ubuntu 16.04的内容进行操作, 里面涉及到的文件也提供了下载 NCVPixelOperations.hpp可以从这里下载

解压OpenCV

1	unzip opencv-2.4.13.6.zip

控制台打印如下

ruter@RuterPC:~/Downloads$ unzip opencv-2.4.13.6.zip 
Archive:  opencv-2.4.13.6.zip
0354d01e79d870adb09eea8f6d95b4093c7b4c97
   creating: opencv-2.4.13.6/
  inflating: opencv-2.4.13.6/.tgitconfig  
   creating: opencv-2.4.13.6/3rdparty/
.....
/Package.appxmanifest  
  inflating: opencv-2.4.13.6/samples/winrt/OcvImageProcessing/OcvImageProcessing/pch.cpp  
  inflating: opencv-2.4.13.6/samples/winrt/OcvImageProcessing/OcvImageProcessing/pch.h  
ruter@RuterPC:~/Downloads$

安装所需库

sudo apt install build-essential

sudo apt install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev

sudo apt install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libdc1394-22-dev

sudo apt install libopenexr-dev yasm libopencore-amrnb-dev libtheora-dev libvorbis-dev libxvidcore-dev

sudo apt install python-tk libeigen3-dev libx264-dev libqt4-dev libqt4-opengl-dev sphinx-common texlive-latex-extra libv4l-dev default-jdk ant openjdk-11-jdk

进入opencv-2.4.13.6, 创建一个目录, 名字自己定义, 然后进入这个目录
1
2
mkdir build
cd build

编辑和修改以下文件

编辑文件**../cmake/FindCUDA.cmake**, 进行如下替换

1	vi ../cmake/FindCUDA.cmake

替换第1处, 共4处

找到

1	find_cuda_helper_libs(nppi)

替换成如下内容

find_cuda_helper_libs(nppial)
find_cuda_helper_libs(nppicc)
find_cuda_helper_libs(nppicom)
find_cuda_helper_libs(nppidei)
find_cuda_helper_libs(nppif)
find_cuda_helper_libs(nppig)
find_cuda_helper_libs(nppim)
find_cuda_helper_libs(nppist)
find_cuda_helper_libs(nppisu)
find_cuda_helper_libs(nppitc)

替换第2处, 共4处

找到

1	set(CUDA_npp_LIBRARY "${CUDA_nppc_LIBRARY};${CUDA_nppi_LIBRARY};${CUDA_npps_LIBRARY}")

替换成

set(CUDA_npp_LIBRARY "${CUDA_nppc_LIBRARY};${CUDA_nppial_LIBRARY};${CUDA_nppicc_LIBRARY};${CUDA_nppicom_LIBRARY};${CUDA_nppidei_LIBRARY};${CUDA_nppif_LIBRARY};${CUDA_nppig_LIBRARY};${CUDA_nppim_LIBRARY};${CUDA_nppist_LIBRARY};${CUDA_nppisu_LIBRARY};${CUDA_nppitc_LIBRARY};${CUDA_npps_LIBRARY}")

替换第3处, 共4处

找到

1	unset(CUDA_nppi_LIBRARY CACHE)

替换成

unset(CUDA_nppial_LIBRARY CACHE)
unset(CUDA_nppicc_LIBRARY CACHE)
unset(CUDA_nppicom_LIBRARY CACHE)
unset(CUDA_nppidei_LIBRARY CACHE)
unset(CUDA_nppif_LIBRARY CACHE)
unset(CUDA_nppig_LIBRARY CACHE)
unset(CUDA_nppim_LIBRARY CACHE)
unset(CUDA_nppist_LIBRARY CACHE)
unset(CUDA_nppisu_LIBRARY CACHE)
unset(CUDA_nppitc_LIBRARY CACHE)

替换第4处, 共4处

找到

1
set(nvcc_flags "")

替换成

1
set(nvcc_flags "--expt-relaxed-constexpr")

编辑文件 ../cmake/OpenCVDetectCUDA.cmake, 并进行如下替换

1	vim ../cmake/OpenCVDetectCUDA.cmake

替换第1处, 共2处

找到

1	set(_generations "Fermi")

替换出

1	set(_generations "Fermi" "Kepler" "Maxwell" "Pascal" "Volta")

替换第2处, 共2处

找到

set(__cuda_arch_ptx "")
if(CUDA_GENERATION STREQUAL "Fermi")
  set(__cuda_arch_bin "2.0 2.1(2.0)")
elseif(CUDA_GENERATION STREQUAL "Kepler")
  if(${CUDA_VERSION} VERSION_LESS "5.0")
    set(__cuda_arch_bin "3.0")
  else()
    set(__cuda_arch_bin "3.0 3.5")
  endif()
elseif(CUDA_GENERATION STREQUAL "Auto")

替换出

set(__cuda_arch_ptx "")
if(CUDA_GENERATION STREQUAL "Fermi")
  set(__cuda_arch_bin "2.0")
elseif(CUDA_GENERATION STREQUAL "Kepler")
  set(__cuda_arch_bin "3.0 3.5 3.7")
elseif(CUDA_GENERATION STREQUAL "Maxwell")
  set(__cuda_arch_bin "5.0 5.2")
elseif(CUDA_GENERATION STREQUAL "Pascal")
  set(__cuda_arch_bin "6.0 6.1")
elseif(CUDA_GENERATION STREQUAL "Volta")
  set(__cuda_arch_bin "7.0")
elseif(CUDA_GENERATION STREQUAL "Auto")

下载 NCVPixelOperations.hpp并替换modules/gpu/src/nvidia/core/NCVPixelOperations.hpp, 或者从原作者提供给Google Drive中下载
1
cp /tmp/1564763067059_NCVPixelOperations.hpp ../modules/gpu/src/nvidia/core/NCVPixelOperations.hpp

执行编译前的检测

cmake -D WITH_TBB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D WITH_V4L=ON -D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON -D WITH_QT=ON -D WITH_OPENGL=ON -D WITH_VTK=ON -D CUDA_GENERATION="Pascal" ..

控制台打印如下

CMake Warning at cmake/OpenCVPackaging.cmake:23 (message):
  CPACK_PACKAGE_VERSION does not match version provided by version.hpp
  header!
Call Stack (most recent call first):
  CMakeLists.txt:1103 (include)

-- Configuring done
-- Generating done
-- Build files have been written to: /home/ruter/Downloads/opencv-2.4.13.6/build
ruter@RuterPC:~/Downloads/opencv-2.4.13.6/build$

执行编译(make 后面-j表示使用N个线程, 后面不带数字表示使用所有线程, 单纯make表示单线程编译)

1	sudo make -j4

会在每个文件开头显示进度, 如下

c.o
[  5%] Building C object 3rdparty/libjasper/CMakeFiles/libjasper.dir/jas_tmr.c.o
[  5%] Building C object 3rdparty/libjasper/CMakeFiles/libjasper.dir/jas_tvp.c.o
[  5%] Building C object 3rdparty/libjasper/CMakeFiles/libjasper.dir/jas_version.c.o
......
Scanning dependencies of target opencv_calib3d_pch_dephelp
[  5%] Building CXX object modules/features2d/CMakeFiles/opencv_features2d_pch_dephelp.dir/opencv_features2d_pch_dephelp.cxx.o
[  7%] Building CXX object modules/calib3d/CMakeFiles/opencv_calib3d_pch_dephelp.dir/opencv_calib3d_pch_dephelp.cxx.o
[  7%] Generating opencv_video_pch_dephelp.cxx
Scanning dependencies of target opencv_video_pch_dephelp
[  7%] Building CXX object modules/video/CMakeFi

耐心等待编译完成… 时间不快的

    [mkdir] Created dir: /home/ruter/Downloads/opencv-2.4.13.6/build/modules/java/test/.build/build/jar
      [jar] Building jar: /home/ruter/Downloads/opencv-2.4.13.6/build/modules/java/test/.build/build/jar/opencv-test.jar

BUILD SUCCESSFUL
Total time: 2 seconds
[100%] Built target opencv_test_java
ruter@RuterPC:~/Downloads/opencv-2.4.13.6/build$

安装
1
sudo make install

验证

ruter@RuterPC:/usr/bin$ python
Python 3.6.8 (default, Jan 14 2019, 11:02:34) 
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'4.1.0'
>>>

如果要想导入cv2, 还需要安装opencv-python, 直接通过pip install opencv-python即可

至此, 所有软件已经安装完毕

Yolo v3的编译

Yolo (You only See once 是基于深度学习的物体检测, 用与检测和识别物体), 官网地址 YOLO: Real-Time Object Detection

yolo

这个也只是摸了一下, 还不是很深入.

从github中pull下代码

1	git clone https://github.com/AlexeyAB/darknet.git

进入 darknet, 编辑MakeFile, 头部的配置如下

GPU=1
CUDNN=1
CUDNN_HALF=0
OPENCV=1
AVX=0
OPENMP=0
LIBSO=1
ZED_CAMERA=0

解释
GPU=1  //使用CUDA GPU加速, 这个如果off的话, 基本用不了, 还有这个GPU专指N卡的GPU
CUDNN=1 // 启用CUDNN加速模型训练
CUDNN_HALF=0 // 构建Tesla来加速模型的检测和构建
 to build for Tensor Cores (on Titan V / Tesla V100 / DGX-2 and later) speedup Detection 3x, Training 2x
OPENCV=1 // 启用网络摄像头实时识别和视频文件的读取（支持4.x, 3.x, 2.x）
AVX=0
OPENMP=0 // 使用OpenMP来用多核CPU加速YOlo
  to build with OpenMP support to accelerate Yolo by using multi-core CPU
LIBSO=1 // 生产libdarknet.so来为其他应用提供接入支持，例如java jni调用so库, Yolo(Windows会生成dll文件)
ZED_CAMERA=0 // 开启ZED-3D摄像头支持

执行编译
1
make
编译好到使用, 还需要下载1个文件 yolov3.weights, 这是已经训练好的模型，可以识别桌子椅子和一些动物啥的, 当然可以自己去训练模型来识别张三李四的, 将这个文件放在刚刚github pull 过来的darknet里面

尝试执行Detect object, 得到如下的错误 libopencv_core.so.2.4: cannot open shared object file

1
2

ruter@RuterPC:~/Documents/github/darknet$ ./darknet detector test cfg/coco.data cfg/yolov3.cfg yolov3.weights data/dog.jpg
./darknet: error while loading shared libraries: libopencv_core.so.2.4: cannot open shared object file: No such file or directory

这是因为libopencv_core的so库是在/usr/local/lib/中, 可以通过这个命令看到它的目录

1	sudo updatedb && locate libopencv_core.so.2.4

得到

ruter@RuterPC:~/Documents/github/darknet$ sudo updatedb && locate libopencv_core.so.2.4
[sudo] password for ruter: 
/home/ruter/Downloads/opencv-2.4.13.6/build/lib/libopencv_core.so.2.4
/home/ruter/Downloads/opencv-2.4.13.6/build/lib/libopencv_core.so.2.4.13
/usr/local/lib/libopencv_core.so.2.4
/usr/local/lib/libopencv_core.so.2.4.13
ruter@RuterPC:~/Documents/github/darknet$

将/usr/local/lib/加入LD_LIBRARY_PATH即可

~/.bashrc中添加如下内容

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

source ~/.bashrc

再次执行Detect

ruter@RuterPC:~/Documents/github/darknet$ ./darknet detector test cfg/coco.data cfg/yolov3.cfg yolov3.weights data/dog.jpg
   layer   filters  size/strd(dil)      input                output
   0 conv     32       3 x 3/ 1    416 x 416 x   3 ->  416 x 416 x  32 0.299 BF
   1 conv     64       3 x 3/ 2    416 x 416 x  32 ->  208 x 208 x  64 1.595 BF
   2 conv     32       1 x 1/ 1    208 x 208 x  64 ->  208 x 208 x  32 0.177 BF
......
[yolo] params: iou loss: mse, iou_norm: 0.75, cls_norm: 1.00, scale_x_y: 1.00
Total BFLOPS 65.864 
 Allocate additional workspace_size = 52.43 MB 
Loading weights from yolov3.weights...
 seen 64 
Done!
data/dog.jpg: Predicted in 18.958000 milli-seconds.
bicycle: 99%
dog: 100%
truck: 94%
Gtk-Message: 02:15:14.299: Failed to load module "canberra-gtk-module"
init done 
opengl support available

detected

视频检测建议使用GPU加速, 否则就像下面一样(视频个位数帧数, 图片识别GPU提速百倍多)..

不开启GPU加速
Loading weights from yolov3.weights...
....

FPS:0.1 // 0.1的帧率, 基本用不了..
Objects:

pottedplant: 73% 
bottle: 57% 
bottle: 36% 

开启GPU加速
Loading weights from yolov3.weights...
...
FPS:33.1
Objects:

pottedplant: 83% 
chair: 93% 
bottle: 45% 
bottle: 43% 
bottle: 28%

其他的后面慢慢深入, 就这样了, 整个过程就完成了.