On Tue, Apr 28, 2020 at 09:57:24AM +0200, Takashi Iwai wrote:
On Mon, 27 Apr 2020 20:43:54 +0200, Alex Deucher wrote:
On Mon, Apr 27, 2020 at 2:39 PM Takashi Iwai tiwai@suse.de wrote:
On Mon, 27 Apr 2020 20:28:12 +0200, Alex Deucher wrote:
On Mon, Apr 27, 2020 at 2:07 PM Nicholas Johnson nicholas.johnson-opensource@outlook.com.au wrote:
On Mon, Apr 27, 2020 at 05:15:55PM +0200, Takashi Iwai wrote:
On Mon, 27 Apr 2020 16:22:21 +0200, Deucher, Alexander wrote: > > [AMD Public Use] > > > -----Original Message----- > > From: Nicholas Johnson nicholas.johnson-opensource@outlook.com.au > > Sent: Sunday, April 26, 2020 12:02 PM > > To: linux-kernel@vger.kernel.org > > Cc: Deucher, Alexander Alexander.Deucher@amd.com; Koenig, Christian > > Christian.Koenig@amd.com; Zhou, David(ChunMing) > > David1.Zhou@amd.com; Nicholas Johnson <nicholas.johnson- > > opensource@outlook.com.au> > > Subject: [PATCH 0/1] Fiji GPU audio register timeout when in BACO state > > > > Hi all, > > > > Since Linux v5.7-rc1 / commit 4fdda2e66de0 ("drm/amdgpu/runpm: enable > > runpm on baco capable VI+ asics"), my AMD R9 Nano has been using runpm / > > BACO. You can tell visually when it sleeps, because the fan on the graphics > > card is switched off to save power. It did not spin down the fan in v5.6.x. > > > > This is great (I love it), except that when it is sleeping, the PCIe audio function > > of the GPU has issues if anything tries to access it. You get dmesg errors such > > as these: > > > > snd_hda_intel 0000:08:00.1: spurious response 0x0:0x0, last cmd=0x170500 > > snd_hda_intel 0000:08:00.1: azx_get_response timeout, switching to polling > > mode: last cmd=0x001f0500 snd_hda_intel 0000:08:00.1: No response from > > codec, disabling MSI: last cmd=0x001f0500 snd_hda_intel 0000:08:00.1: No > > response from codec, resetting bus: last cmd=0x001f0500 > > snd_hda_codec_hdmi hdaudioC1D0: Unable to sync register 0x2f0d00. -11 > > > > The above is with the Fiji XT GPU at 0000:08:00.0 in a Thunderbolt enclosure > > (not that Thunderbolt should affect it, but I feel I should mention it just in > > case). I dropped a lot of duplicate dmesg lines, as some of them repeated a > > lot of times before the driver gave up. > > > > I offer this patch to disable runpm for Fiji while a fix is found, if you decide > > that is the best approach. Regardless, I will gladly test any patches you come > > up with instead and confirm that the above issue has been fixed. > > > > I cannot tell if any other GPUs are affected. The only other cards to which I > > have access are a couple of AMD R9 280X (Tahiti XT), which use radeon driver > > instead of amdgpu driver. > > Adding a few more people. Do you know what is accessing the audio? The audio should have a dependency on the GPU device. The GPU won't enter runtime pm until the audio has entered runtime pm and vice versa on resume. Please attach a copy of your dmesg output and lspci output.
pci 0000:08:00.1: D0 power state depends on 0000:08:00.0 The above must be the dependency of which you speak from dmesg.
Accessing the audio? I did not have a single method for triggering it. Sometimes it happened on shutdown. Sometimes when restarting gdm. Sometimes when playing with audio settings in Cinnamon Desktop. But most often when changing displays. It might have something to do with the audio device associated with a monitor being created when the monitor is found. If an audio device is created, then pulseaudio might touch it. Sorry, this is a very verbose "not quite sure".
To trigger the bug, this time I did the following:
Boot laptop without Fiji and log in
Attach Fiji via Thunderbolt (no displays attached to Fiji) and
approve Thunderbolt device
Log in again because the session gets killed when GPU is hot-added
Wait for Fiji to fall asleep (fan stops)
Open "dmesg -w" on laptop display
Attach display to DisplayPort on Fiji (it should still stay asleep)
Do WindowsKey+P to activate external display. The error appears in
dmesg window that instant.
Could it be a race condition when waking the card up?
I cannot get the graphics card fan to spin down if the Thunderbolt enclosure is attached at boot time. It only does it if hot-added.
If you think it will help, I can take out the Fiji and put it in a test rig and try to replicate the issue without Thunderbolt, but it looks like it will not spin the fan down if Fiji is attached at boot time.
Question, why would the fan not spin down if Fiji is attached at boot time, and how would one make the said fan turn off? Aside from being useful for pinning down the audio register issue, I would like to make sure the power savings are realised whenever the GPU is not being used.
Presumably something is using the device. Maybe a framebuffer console or X? Or maybe the something like tlp has disabled runtime pm on your device? You can see the current status by reading the files in /sys/class/drm/cardX/device/power/ . Replace cardX with card0, card1, etc. depending on which device is the radeon card.
I had card1 = Fiji stuck awake and card2 = Fiji asleep (both in separate Thunderbolt enclosures).
The sysfs values in /sys/class/drm/card{0,1}/device/power/ were the same.
The powertop utility did not help in "tunables" tab.
I compiled kernel without fbcon and it still did it.
But moving from Arch to Ubuntu changed the behaviour. I am still investigating.
FWIW, I have a fiji board in a desktop system and it worked fine when this code was enabled.
Is the new DC code used for Fiji boards? IIRC, the audio component binding from amdgpu is enabled only for DC, and without the audio component binding the runtime PM won't be linked up, hence you can't power up GPU from the audio side access automatically.
Yes, DC is enabled by default for all cards with runtime pm enabled.
OK, thanks, I found that amdgpu got bound via component in the dmesg output, too:
[ 21.294927] snd_hda_intel 0000:08:00.1: bound 0000:08:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu])
This is the place soon after amdgpu driver gets initialized. Then we see later another initialization phase:
[ 26.904127] rfkill: input handler enabled [ 37.264152] [drm] PCIE GART of 1024M enabled (table at 0x000000F400000000).
here shows 10 seconds between them. Then, it complained something:
[ 37.363287] [drm] UVD initialized successfully. [ 37.473340] [drm] VCE initialized successfully. [ 37.477942] amdgpu 0000:08:00.0: [drm] Cannot find any crtc or sizes
The above would be me hitting WindowsKey+P to change screens, but with no DisplayPort attached to Fiji, hence it unable to find crtc.
... and go further, and hitting HD-audio error:
That would be me having attached the DisplayPort and done WindowsKey+P again.
[ 38.936624] [drm] fb mappable at 0x4B0696000 [ 38.936626] [drm] vram apper at 0x4B0000000 [ 38.936626] [drm] size 33177600 [ 38.936627] [drm] fb depth is 24 [ 38.936627] [drm] pitch is 15360 [ 38.936673] amdgpu 0000:08:00.0: fb1: amdgpudrmfb frame buffer device [ 40.092223] snd_hda_intel 0000:08:00.1: azx_get_response timeout, switching to polling mode: last cmd=0x00170500
After this point, HD-audio communication was screwed up.
This lastcmd in the above message is AC_SET_POWER_STATE verb for the root node to D0, so the very first command to power up the codec. The rest commands are also about the power up of each node, so the whole error indicate that the power up at runtime resume failed.
So, this looks to me as if the device gets runtime-resumed at the bad moment?
It does. However, this is not going to be easy to pin down.
I moved from Arch to Ubuntu, and it behaves differently. I cannot trigger the bug in Ubuntu. Plus, it puts the GPUs asleep, even if attached at boot, unlike Arch. I will continue to try to trigger it. But even if this is a problem with the Linux distribution, it should not be able to trigger a kernel mode bug, so we should persist with finding it.
Regards, Nicholas
thanks,
Takashi