{"id":3641,"date":"2018-09-14T10:59:39","date_gmt":"2018-09-14T16:59:39","guid":{"rendered":"http:\/\/benincosa.com\/?p=3641"},"modified":"2018-09-17T13:45:41","modified_gmt":"2018-09-17T19:45:41","slug":"hands-on-with-nvidia-p6-and-ucs-b200-m5","status":"publish","type":"post","link":"https:\/\/benincosa.com\/?p=3641","title":{"rendered":"Hands on with NVIDIA P6 and UCS B200 M5"},"content":{"rendered":"

Just got access to a new UCS B200 M5 blade!\u00a0 My goal is to create a tensorflow lab on it.\u00a0 Let’s get cracking!<\/p>\n

It was installed with CentOS 7.5<\/p>\n

cat \/etc\/redhat-release<\/p>\n

cat \/etc\/redhat-release\r\nCentOS Linux release 7.5.1804 (Core)<\/pre>\n
Let’s make sure there is indeed a GPU:<\/p>\n
lspci\r\n...\r\nd8:00.0 3D controller: NVIDIA Corporation GP104GL [Tesla P6] (rev a1)<\/pre>\n
Ok, now we need to get some drivers. We go to NVIDIA’s page, fill out the form and get some drivers for RHEL7.<\/p>\n
\"\"<\/p>\n
While waiting for downloads, we make it so we can sudo without a password. Run sudo visudo and edit these lines:<\/p>\n
## Allows people in group wheel to run all commands\r\n#%wheel ALL=(ALL)       ALL\r\n\r\n## Same thing without a password\r\n%wheel  ALL=(ALL)       NOPASSWD: ALL<\/pre>\n
Now let’s install the VNC server and some other packages we’ll need.\u00a0 This will give us development tools and a remote desktop<\/p>\n
sudo yum install tigervnc-server\r\nsudo yum -y groupinstall \"GNOME Desktop\" \"Development Tools\"\r\nsudo yum -y install kernel-devel\r\nvncserver<\/pre>\n
Now let’s attach to it… hmm.\u00a0 we can’t.\u00a0 Is selinux running?<\/p>\n
getenforce\r\nEnforcing<\/pre>\n
yep.\u00a0 Let’s turn that off for now.\u00a0 We don’t need this.\u00a0 Use sudo to modify \/etc\/sysconfig\/selinux<\/p>\n
# This file controls the state of SELinux on the system.\r\n# SELINUX= can take one of these three values:\r\n#     enforcing - SELinux security policy is enforced.\r\n#     permissive - SELinux prints warnings instead of enforcing.\r\n#     disabled - No SELinux policy is loaded.\r\nSELINUX=disabled\r\n# SELINUXTYPE= can take one of three two values:\r\n#     targeted - Targeted processes are protected,\r\n#     minimum - Modification of targeted policy. Only selected processes are protected.\r\n#     mls - Multi Level Security protection.\r\nSELINUXTYPE=targeted<\/pre>\n
Have to reboot.\u00a0 But first let’s install the driver:<\/p>\n
edit \/etc\/default\/grub to disable the nouvea driver.<\/p>\n
GRUB_TIMEOUT=5\r\nGRUB_DISTRIBUTOR=\"$(sed 's, release .*$,,g' \/etc\/system-release)\"\r\nGRUB_DEFAULT=saved\r\nGRUB_DISABLE_SUBMENU=true\r\nGRUB_TERMINAL_OUTPUT=\"console\"\r\nGRUB_CMDLINE_LINUX=\"crashkernel=auto rd.lvm.lv=centos\/root rd.lvm.lv=centos\/swap rhgb quiet rd.driver.blacklist=nouveau nouveau.modeset=0\"\r\nGRUB_DISABLE_RECOVERY=\"true\"<\/pre>\n
load the new grub file<\/p>\n
sudo grub2-mkconfig -o \/boot\/grub2\/grub.cfg<\/pre>\n
Add the nouvea driver to the blacklist by appending to (or creating in my case)\u00a0\/etc\/modprobe.d\/blacklist.conf<\/code><\/p>\n
blacklist nouveau<\/pre>\n
Back up the old stuff and make the new initrd<\/p>\n
sudo mv \/boot\/initramfs-$(uname -r).img \/boot\/initramfs-$(uname -r)-nouveau.img\r\nsudo dracut \/boot\/initramfs-$(uname -r).img $(uname -r)\r\n<\/pre>\n
Now we reboot.<\/p>\n
chmod 755 NVIDIA-Linux*\r\nsudo .\/NVIDIA-Linux-x86_64-396.44run\r\nsudo shutdown -r now<\/pre>\n
I installed the the 32 bit compatible libraries because diskspace is cheap and time is short.<\/p>\n
CUDA Libraries<\/h2>\n
We want tensorflow with the CUDA libraries.\u00a0 It makes tensorflow fast!\u00a0 We get it by navigating to their page.\u00a0 I downloaded the runtime one.<\/p>\n
wget --no-check-certificate https:\/\/developer.nvidia.com\/compute\/cuda\/9.2\/Prod2\/local_installers\/cuda_9.2.148_396.37_linux<\/pre>\n
chmod 755 cuda_9.2.148_396.37_linux\r\nsudo .\/cuda_9.2.148_396.37_linux<\/pre>\n
I answer the questions as follows:<\/p>\n
Do you accept the previously read EULA?\r\naccept\/decline\/quit: accept\r\n\r\nInstall NVIDIA Accelerated Graphics Driver for Linux-x86_64 396.37?\r\n(y)es\/(n)o\/(q)uit: n\r\n\r\nInstall the CUDA 9.2 Toolkit?\r\n(y)es\/(n)o\/(q)uit: y\r\n\r\nEnter Toolkit Location\r\n [ default is \/usr\/local\/cuda-9.2 ]:\r\n\r\nDo you want to install a symbolic link at \/usr\/local\/cuda?\r\n(y)es\/(n)o\/(q)uit: y\r\n\r\nInstall the CUDA 9.2 Samples?\r\n(y)es\/(n)o\/(q)uit: y\r\n\r\nEnter CUDA Samples Location\r\n [ default is \/home\/tsadministrator ]:<\/pre>\n
Since all went well you will see the following output<\/p>\n
===========\r\n= Summary =\r\n===========\r\n\r\nDriver:   Not Selected\r\nToolkit:  Installed in \/usr\/local\/cuda-9.2\r\nSamples:  Installed in \/home\/tsadministrator, but missing recommended libraries\r\n\r\nPlease make sure that\r\n -   PATH includes \/usr\/local\/cuda-9.2\/bin\r\n -   LD_LIBRARY_PATH includes \/usr\/local\/cuda-9.2\/lib64, or, add \/usr\/local\/cuda-9.2\/lib64 to \/etc\/ld.so.conf and run ldconfig as root\r\n\r\nTo uninstall the CUDA Toolkit, run the uninstall script in \/usr\/local\/cuda-9.2\/bin\r\n\r\nPlease see CUDA_Installation_Guide_Linux.pdf in \/usr\/local\/cuda-9.2\/doc\/pdf for detailed information on setting up CUDA.\r\n\r\n***WARNING: Incomplete installation! This installation did not install the CUDA Driver. A driver of version at least 384.00 is required for CUDA 9.2 functionality to work.\r\nTo install the driver using this installer, run the following command, replacing <CudaInstaller> with the name of this run file:\r\n    sudo <CudaInstaller>.run -silent -driver\r\n\r\nLogfile is \/tmp\/cuda_install_23222.log<\/pre>\n
(or at least something similar)<\/p>\n
Now lets get the environment setup.\u00a0 Append to ~\/.bash_profile<\/p>\n
export PATH=\/usr\/local\/cuda-9.2\/bin${PATH:+:${PATH}}\r\nexport LD_LIBRARY_PATH=\/usr\/local\/cuda-9.2\/lib64\\${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}<\/pre>\n
(I’m using 9.2 as this is the version of cuda I’ve installed, it may be different when you install so change as updates come available.)<\/p>\n
cuDNN Library<\/h2>\n
To download the cuDNN libraries you need to have an NVIDIA developer account.\u00a0 You’ll have to login to the download site<\/a> and download the Linux version<\/p>\n
\"\"<\/p>\n
Here we are cuDNN v 7.2.1 with CUDA 9.2 as that is the library we used.<\/p>\n
tar -zxvf cudnn-9.2-linux-x64-v7.2.1.38.tgz\r\nsudo cp cuda\/include\/cudnn.h \/usr\/local\/cuda\/include\r\nsudo cp cuda\/lib64\/libcudnn* \/usr\/local\/cuda\/lib64\r\nsudo chmod a+r \/usr\/local\/cuda\/include\/cudnn.h \/usr\/local\/cuda\/lib64\/libcudnn*\r\n<\/pre>\n
 <\/p>\n
Installing Tensorflow<\/h2>\n
Install pip<\/p>\n
yum install epel-release<\/pre>\n
yum -y install gcc gcc-c++ python-pip python-devel atlas atlas-devel gcc-gfortran openssl-devel libffi-devel python-jinja2<\/pre>\n
Now we can install tensorflow:<\/p>\n
yum remove python-enum34\r\npip install --upgrade pip\r\npip uninstall dnspython\r\npip instll dnspython==1.15.0\r\npip install tensorflow-gpu ipython<\/pre>\n
 <\/p>\n
References<\/h2>\n
https:\/\/www.nvidia.com\/en-us\/data-center\/gpu-accelerated-applications\/tensorflow\/<\/a><\/p>\n
http:\/\/developer.download.nvidia.com\/compute\/cuda\/6_5\/rel\/docs\/CUDA_Getting_Started_Linux.pdf<\/a><\/p>\n
https:\/\/blog.sicara.com\/tensorflow-gpu-opencv-jupyter-docker-10705b6cd1d<\/a><\/p>\n
https:\/\/gist.github.com\/lyastro\/26e0cd8245bcf64914857dd5e8445724<\/a><\/p>\n
 <\/p>\n
Installing NVIDIA Drivers on RHEL or CentOS 7<\/a><\/p><\/blockquote>\n