Just got access to a new UCS B200 M5 blade! My goal is to create a tensorflow lab on it. Let's get cracking!
It was installed with CentOS 7.5
cat /etc/redhat-release
cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
Let's make sure there is indeed a GPU:
lspci
...
d8:00.0 3D controller: NVIDIA Corporation GP104GL [Tesla P6] (rev a1)
Ok, now we need to get some drivers. We go to NVIDIA's page, fill out the form and get some drivers for RHEL7.
While waiting for downloads, we make it so we can sudo without a password. Run sudo visudo and edit these lines:
## Allows people in group wheel to run all commands
#%wheel ALL=(ALL) ALL
## Same thing without a password
%wheel ALL=(ALL) NOPASSWD: ALL
Now let's install the VNC server and some other packages we'll need. This will give us development tools and a remote desktop
sudo yum install tigervnc-server
sudo yum -y groupinstall "GNOME Desktop" "Development Tools"
sudo yum -y install kernel-devel
vncserver
Now let's attach to it... hmm. we can't. Is selinux running?
getenforce
Enforcing
yep. Let's turn that off for now. We don't need this. Use sudo to modify /etc/sysconfig/selinux
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
Have to reboot. But first let's install the driver:
edit /etc/default/grub to disable the nouvea driver.
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet rd.driver.blacklist=nouveau nouveau.modeset=0"
GRUB_DISABLE_RECOVERY="true"
load the new grub file
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
Add the nouvea driver to the blacklist by appending to (or creating in my case)
/etc/modprobe.d/blacklist.conf
blacklist nouveau
Back up the old stuff and make the new initrd
sudo mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r)-nouveau.img
sudo dracut /boot/initramfs-$(uname -r).img $(uname -r)
Now we reboot.
chmod 755 NVIDIA-Linux*
sudo ./NVIDIA-Linux-x86_64-396.44run
sudo shutdown -r now
I installed the the 32 bit compatible libraries because diskspace is cheap and time is short.
CUDA Libraries
We want tensorflow with the CUDA libraries. It makes tensorflow fast! We get it by navigating to their page. I downloaded the runtime one.
wget --no-check-certificate https://developer.nvidia.com/compute/cuda/9.2/Prod2/local_installers/cuda_9.2.148_396.37_linux
chmod 755 cuda_9.2.148_396.37_linux
sudo ./cuda_9.2.148_396.37_linux
I answer the questions as follows:
Do you accept the previously read EULA?
accept/decline/quit: accept
Install NVIDIA Accelerated Graphics Driver for Linux-x86_64 396.37?
(y)es/(n)o/(q)uit: n
Install the CUDA 9.2 Toolkit?
(y)es/(n)o/(q)uit: y
Enter Toolkit Location
[ default is /usr/local/cuda-9.2 ]:
Do you want to install a symbolic link at /usr/local/cuda?
(y)es/(n)o/(q)uit: y
Install the CUDA 9.2 Samples?
(y)es/(n)o/(q)uit: y
Enter CUDA Samples Location
[ default is /home/tsadministrator ]:
Since all went well you will see the following output
===========
= Summary =
===========
Driver: Not Selected
Toolkit: Installed in /usr/local/cuda-9.2
Samples: Installed in /home/tsadministrator, but missing recommended libraries
Please make sure that
- PATH includes /usr/local/cuda-9.2/bin
- LD_LIBRARY_PATH includes /usr/local/cuda-9.2/lib64, or, add /usr/local/cuda-9.2/lib64 to /etc/ld.so.conf and run ldconfig as root
To uninstall the CUDA Toolkit, run the uninstall script in /usr/local/cuda-9.2/bin
Please see CUDA_Installation_Guide_Linux.pdf in /usr/local/cuda-9.2/doc/pdf for detailed information on setting up CUDA.
***WARNING: Incomplete installation! This installation did not install the CUDA Driver. A driver of version at least 384.00 is required for CUDA 9.2 functionality to work.
To install the driver using this installer, run the following command, replacing <CudaInstaller> with the name of this run file:
sudo <CudaInstaller>.run -silent -driver
Logfile is /tmp/cuda_install_23222.log
(or at least something similar)
Now lets get the environment setup. Append to ~/.bash_profile
export PATH=/usr/local/cuda-9.2/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-9.2/lib64\${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
(I'm using 9.2 as this is the version of cuda I've installed, it may be different when you install so change as updates come available.)
cuDNN Library
To download the cuDNN libraries you need to have an NVIDIA developer account. You'll have to login to the
download site and download the Linux version
Here we are cuDNN v 7.2.1 with CUDA 9.2 as that is the library we used.
tar -zxvf cudnn-9.2-linux-x64-v7.2.1.38.tgz
sudo cp cuda/include/cudnn.h /usr/local/cuda/include
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*
Installing Tensorflow
Install pip
yum install epel-release
yum -y install gcc gcc-c++ python-pip python-devel atlas atlas-devel gcc-gfortran openssl-devel libffi-devel python-jinja2
Now we can install tensorflow:
yum remove python-enum34
pip install --upgrade pip
pip uninstall dnspython
pip instll dnspython==1.15.0
pip install tensorflow-gpu ipython
References
https://www.nvidia.com/en-us/data-center/gpu-accelerated-applications/tensorflow/
http://developer.download.nvidia.com/compute/cuda/6_5/rel/docs/CUDA_Getting_Started_Linux.pdf
https://blog.sicara.com/tensorflow-gpu-opencv-jupyter-docker-10705b6cd1d
https://gist.github.com/lyastro/26e0cd8245bcf64914857dd5e8445724
http://www.advancedclustering.com/act_kb/installing-nvidia-drivers-rhel-centos-7/