Re: [Bug 216859] New: PCI bridge to bus boot hang at enumeration
[+cc sound folks]
On Wed, Dec 28, 2022 at 06:02:48AM -0600, Bjorn Helgaas wrote:
On Wed, Dec 28, 2022 at 08:37:52AM +0000, bugzilla-daemon@kernel.org wrote:
Summary: PCI bridge to bus boot hang at enumeration Kernel Version: 6.1-rc1...
With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
Thank you very much for your report, Zeno!
v6.0 works, v6.1-rc1 fails. Would you mind booting v6.1-rc1 with the "ignore_loglevel initcall_debug" kernel parameters and taking a photo when it hangs?
How did you conclude that the hang is related to a PCI bridge? I see recent PCI messages in the photo, but it looks like the last message is from NFS, so I'm wondering if I'm missing some context. The v6.0 dmesg shows several other ntfs, fuse, JFS, etc messages before more PCI-related things. Anyway, the "initcall_debug" might help us narrow it down a bit.
Thanks very much for the bisection (complete log at [1])!
The bisection claims the first bad commit is:
833477fce7a1 ("Merge tag 'sound-6.1-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound")
with parents:
7e6739b9336e ("Merge tag 'drm-next-2022-10-05' of git://anongit.freedesktop.org/drm/drm") 86a4d29e7554 ("Merge tag 'asoc-v6.1' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound into for-linus")
Both 7e6739b9336e and 86a4d29e7554 tested "good" during the bisection.
There is a minor conflict when merging 86a4d29e7554 into the upstream, but I can't imagine that being resolved incorrectly.
Would you mind turning off CONFIG_SOUND in your .config and testing 833477fce7a1 again? I'm a little skeptical that the hang would be sound-related, but I guess it's a place to start.
Bjorn
Dear Bjorn
On Thu, Jan 12, 2023 at 9:08 PM Bjorn Helgaas helgaas@kernel.org wrote:
[+cc sound folks]
On Wed, Dec 28, 2022 at 06:02:48AM -0600, Bjorn Helgaas wrote:
On Wed, Dec 28, 2022 at 08:37:52AM +0000, bugzilla-daemon@kernel.org wrote:
Summary: PCI bridge to bus boot hang at enumeration Kernel Version: 6.1-rc1...
With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
Thank you very much for your report, Zeno!
v6.0 works, v6.1-rc1 fails. Would you mind booting v6.1-rc1 with the "ignore_loglevel initcall_debug" kernel parameters and taking a photo when it hangs?
How did you conclude that the hang is related to a PCI bridge? I see recent PCI messages in the photo, but it looks like the last message is from NFS, so I'm wondering if I'm missing some context. The v6.0 dmesg shows several other ntfs, fuse, JFS, etc messages before more PCI-related things. Anyway, the "initcall_debug" might help us narrow it down a bit.
Thanks very much for the bisection (complete log at [1])!
The bisection claims the first bad commit is:
833477fce7a1 ("Merge tag 'sound-6.1-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound")
with parents:
7e6739b9336e ("Merge tag 'drm-next-2022-10-05' of git://anongit.freedesktop.org/drm/drm") 86a4d29e7554 ("Merge tag 'asoc-v6.1' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound into for-linus")
Both 7e6739b9336e and 86a4d29e7554 tested "good" during the bisection.
There is a minor conflict when merging 86a4d29e7554 into the upstream, but I can't imagine that being resolved incorrectly.
Would you mind turning off CONFIG_SOUND in your .config and testing 833477fce7a1 again? I'm a little skeptical that the hang would be sound-related, but I guess it's a place to start.
Bjorn
Booting without sound into the commit 833477fce7a1 did not help. Same hang.
Best Zeno
[+cc folks from 145eed48de27 and framebuffer folks, regression list]
On Thu, Jan 12, 2023 at 02:08:19PM -0600, Bjorn Helgaas wrote:
On Wed, Dec 28, 2022 at 06:02:48AM -0600, Bjorn Helgaas wrote:
On Wed, Dec 28, 2022 at 08:37:52AM +0000, bugzilla-daemon@kernel.org wrote:
Summary: PCI bridge to bus boot hang at enumeration Kernel Version: 6.1-rc1...
With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
This is a regression between v6.0 and v6.1-rc1. Console output during boot freezes after nvidiafb deactivates the VGA console.
It was a lot of work for Zeno, but we finally isolated this console hang to 145eed48de27 ("fbdev: Remove conflicting devices on PCI bus").
The system actually does continue to boot and is accessible via ssh, but the console appears hung, at least for output. More details in the bugzilla starting at https://bugzilla.kernel.org/show_bug.cgi?id=216859#c47 .
Bjorn
[+cc Geert]
On Thu, Jan 26, 2023 at 06:11:24AM -0600, Bjorn Helgaas wrote:
On Thu, Jan 12, 2023 at 02:08:19PM -0600, Bjorn Helgaas wrote:
On Wed, Dec 28, 2022 at 06:02:48AM -0600, Bjorn Helgaas wrote:
On Wed, Dec 28, 2022 at 08:37:52AM +0000, bugzilla-daemon@kernel.org wrote:
Summary: PCI bridge to bus boot hang at enumeration Kernel Version: 6.1-rc1...
With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
This is a regression between v6.0 and v6.1-rc1. Console output during boot freezes after nvidiafb deactivates the VGA console.
It was a lot of work for Zeno, but we finally isolated this console hang to 145eed48de27 ("fbdev: Remove conflicting devices on PCI bus").
The system actually does continue to boot and is accessible via ssh, but the console appears hung, at least for output. More details in the bugzilla starting at https://bugzilla.kernel.org/show_bug.cgi?id=216859#c47 .
145eed48de27 ("fbdev: Remove conflicting devices on PCI bus") doesn't say what the benefit is, or what would break if we reverted it.
Does anybody have any clues? It would be nice to resolve this regression before v6.2, which will probably be released 2/12 or 2/19.
Bjorn
participants (2)
-
Bjorn Helgaas -
Zeno Davatz