1

My Ubuntu 16.10 server VM in MS Azure (NV6 series) suddenly had a hickup for unknown reasons (none of my doing), I had to restart it and when it came back online I was no longer able to use the GPU on the machine.

The nvidia-smi application freezes.

The command lspci yields

lspci: Cannot open /sys/bus/pci/devices/7ec1:00:00.0/resource: No such file or directory

And of course, that path (no longer?) exists. What does exist is,

$: ls /sys/bus/pci/devices/
0000:00:00.0/    0000:00:07.0/    0000:00:07.1/    0000:00:07.3/    0000:00:08.0/    b717ec1:00:00.0/

Some googling yielded a few similar questions like mine, many of which has been asked in the last 24 hours, like this one.

This might be due to Ubuntu or Azure, I have no idea which is the source of this problem or how to solve it.

Anyone have any ideas?

larslovlie
  • 13
  • I'm seeing the same problem right now on a newly deployed NV6 VM on Azure. It also does not detect the Tesla M60 GPU on that machine. It worked fine on a VM I deployed a few days ago. – RasmusW Apr 30 '17 at 19:38
  • In /sys/bus/pci/devices/ I have a device called 2f36c0b8:00:00.0. Except for the first 4 hex digits, that is the device ID that lspci complains about. I've tried deploying a new instance, and have found out that lspci stops working after apt-get dist-upgrade and reboot. Unfortunately, I have no solution - for this test VM I can skip apt-get dist-upgrade. – RasmusW Apr 30 '17 at 20:53
  • @RasmusW, check out Chris' answer below. This fixed it for me. – larslovlie May 02 '17 at 07:51

3 Answers3

1

I was having the same problem (using Azure NC24 instances) and after working at it for a few hours I found this post and decided to submit a support request to Microsoft. Here's what they told me:

Canonical appears to have recently released kernel 4.4.0-75 for Ubuntu 16.04 and this is having an adverse effect on Tesla GPUs on NC-series VMs. Installation of the 4.4.0-75 breaks the 8.0.61-1 version of the NVIDIA CUDA driver that’s currently recommended for use on these systems, resulting in nvidia-smi not showing the adapters and lspci returning an error similar to the following:

root@pd-nvtest2:~# lspci lspci: Cannot open /sys/bus/pci/devices/2baf:00:00.0/resource: No such file or directory

They suggest backing up the OS drive, running

apt-get remove linux-image-4.4.0-75-generic

and then

update-grub

Reboot and it should work! At the very least doing that fixed the lspci output for me, I still needed to fix some CUDA stuff but that's from earlier debugging attempts.

Chris Gorman
  • 26
  • 2
    Thank you very much, this worked for me, although I'm using Ubuntu 16.10 on Azure NV6. I did the following

    sudo apt-get remove linux-image-4.8.0-49-generic

    and this removed 4.8.0-49 and installed 4.8.0-51, which made nvidia-smi and lspci work again!

     – larslovlie May 02 '17 at 07:47
0

Maybe this is owing to you have stopped(deallocated) the Azure VM, and then started VM again. According to [1], the hardware IP(like gpu,cpu) has changed when you stop(deallocated) and then start VM again. But the Ubuntu system hasn't been updated for new hardware(like gpu, cpu) IP address. Hence, lspci will tell you cannot open some hardware ip address related folder.

[1]https://blogs.technet.microsoft.com/gbanin/2015/04/22/difference-between-the-states-of-azure-virtual-machines-stopped-and-stopped-deallocated/

Evan
  • 1
-1

On Azure VM this seems to be an issue with LIS on RedHat 7.5 Update Azure LIS for the VM and it should fix the issue.

wget https://aka.ms/lis
tar xvzf lis
cd LISISO
sudo ./install.sh
sudo reboot
cmak.fr
  • 8,696
Sham SV
  • 1