coreboot

On a sibling ADL-N board (same SoC, same IT8625E) most of these turned out to be a few separate things, not one faulty timer:

Broken IRQ 9 (4) and the lag cured by LAPIC->HPET (5) are the same problem. The IT8625E HWM sits on its default IRQ9 = the SCI (the EC LDN is not declared in devicetree, so coreboot leaves its IRQ register at the power-on default 9); on a fanless box the empty fan header storms IRQ9, and that storm is what makes the system lag. Declaring the EC LDN with irq 0x70 = 0 stopped it here, and the lag went with it (no HPET workaround needed). Details and the snippet are in #657. Check with superiotool -d (LDN 0x04 active, reg 0x70 = 09).

NVMe not detected (1). The NVMe root port has no ASPM/L1-substate override. A DRAM-less NVMe (e.g. Micron 2550) needs it disabled, which matches your "disabling power saving helps". On pcie_rp9:

.pcie_rp_aspm = ASPM_DISABLE,
.PcieRpL1Substates = L1_SS_DISABLED,
(this is the per-port half of the existing PchLegacyIoLowLatency DMI-ASPM workaround.)

Hangs / spinlock panics / instability (3). If your DIMM is a Micron or Crucial DDR5-4800 single-rank, that is the known N100/Topton instability. Two things fixed it here: cap the speed in FSP-M (max_dram_speed_mts = 4400) and pin SaGv (sagv = SaGv_FixedPoint3, or SaGv_Disabled for more margin) instead of SaGv_Enabled, since the dynamic DRAM frequency transitions corrupt marginal RAM under load.

USB/Ventoy detection and HPET-in-PC10 I cannot help with; those look separate.

The coreboot.log you attached actually pins this down. Line ~483 "PNP: 002e.4 enabled" shows the IT8625E Environment Controller (HWM) LDN is active, and line ~725 "PNP: 002e.4 70 irq ... not assigned in devicetree" shows its IRQ register is never written, so it stays at the power-on default IRQ 9 = the ACPI SCI. That is the "irq 9: nobody cared" in your linux-boot.log: a fanless box, empty fan header, the HWM stuck asserting the SCI.

The fix is to declare that LDN with no IRQ. I have a patch ready but need two values from the running board:

superiotool -d # LDN 0x04 (EC): the two IO bases (reg 0x60, 0x62) and reg 0x70

Send those and I will finish the devicetree change (irq 0x70 = 0, plus the real IO bases) and put it on Gerrit for you to flash and confirm.

I've pushed three changes for this to Gerrit, grouped under the topic topton-x2e-stability:

https://review.coreboot.org/q/topic:topton-x2e-stability

CB:94028 mb/topton/adl: detach IT8625E hardware monitor from the ACPI SCI
Declares the IT8625E EC (LDN 0x04) with no IRQ. Your dump showed it enabled on IRQ 9, which is the ACPI SCI on Alder Lake-N; on a fanless box the empty fan header makes the HWM assert IRQ 9 forever ("irq 9: nobody cared"). This is the main root cause of the lag/timer trouble.

CB:94029 mb/topton/adl: pin SA-GV to a fixed point
Pins the DRAM to SaGv_FixedPoint3. SA-GV frequency switching on non-ECC memory can cause the random spinlock-timeout/hang instability.

CB:94030 mb/topton/adl: disable ASPM/L1 substates on the NVMe root port
Completes the existing "fixes wonky NVME" DMI-ASPM workaround on the port side, so a DRAM-less NVMe doesn't drop off the bus over L1.2 latency.

Could you build master with all three (same topic, so you can cherry-pick the topic) and check whether the IRQ 9 storm, the instability and the NVMe detection are resolved? If it holds up I'll add your Tested-by. Thanks for the detailed report and the superiotool dump - that's what made this diagnosable.

Sebastian Müller wrote in #note-4:

I've pushed three changes for this to Gerrit, grouped under the topic topton-x2e-stability:

https://review.coreboot.org/q/topic:topton-x2e-stability

CB:94028 mb/topton/adl: detach IT8625E hardware monitor from the ACPI SCI
Declares the IT8625E EC (LDN 0x04) with no IRQ. Your dump showed it enabled on IRQ 9, which is the ACPI SCI on Alder Lake-N; on a fanless box the empty fan header makes the HWM assert IRQ 9 forever ("irq 9: nobody cared"). This is the main root cause of the lag/timer trouble.

CB:94030 mb/topton/adl: disable ASPM/L1 substates on the NVMe root port
Completes the existing "fixes wonky NVME" DMI-ASPM workaround on the port side, so a DRAM-less NVMe doesn't drop off the bus over L1.2 latency.

Could you build master with all three (same topic, so you can cherry-pick the topic) and check whether the IRQ 9 storm, the instability and the NVMe detection are resolved? If it holds up I'll add your Tested-by. Thanks for the detailed report and the superiotool dump - that's what made this diagnosable.

Again, please, stop pushing AI-generated code - especially without understanding the underlying issue.
Disabling advertising of ASPM capabilities has no affect if user doesn't explicitly enable ASPM in Kconfig.
Since this issue was opened I already pushed a patch fixing most of them (CB:92369) but likely doesn't solve interrupt storm.

NVME detection issue is caused by not knowing which GPIO controls the PCIe slot reset. If you have an oscilloscope and would like to mess with libgpiod it can be solved relatively cleanly.
The real fix for this is to set clock 0 as FREE_RUNNING, which will enable clock at all times. Might also need to tell FSP to stop touching GPIOs like I did on my most recent port (CB:94039)

I never tested/checked HPET on coreboot. Lst time I messed with it few years ago I believe some ACPI stuff was missing, but my memory might be playing tricks on me.

As for USB, I suspect that USB3 ports are re-used TCSS-as-type_A, so it needs appropriate mapping. If it works fine with a USB2 stick then it'll be 100% confirmed.
Personally I haven't had any USB issues (but I know only USB2 is working) and I've been running this system in production for over 2 years at this point.

That being said, port definitely could use some improvement. I ported this router in few hours I spent on DB Regio heading from Strasbourg to Dortmund, so it was done within ~4 hours in total.

Hangs / spinlock panics / instability (3). If your DIMM is a Micron or Crucial DDR5-4800 single-rank, that is the known N100/Topton instability.
Pins the DRAM to SaGv_FixedPoint3. SA-GV frequency switching on non-ECC memory can cause the random spinlock-timeout/hang instability.
"Where's the source? The source is that I made it the [censored] up"...

home:~# uptime
 23:22:43 up 49 days,  9:31,  0 users,  load average: 0.00, 0.00, 0.00

home:~# dmidecode -t memory
# dmidecode 3.7
Getting SMBIOS data from sysfs.
SMBIOS 3.0.0 present.

Handle 0x0009, DMI type 16, 23 bytes
Physical Memory Array
	Location: System Board Or Motherboard
	Use: System Memory
	Error Correction Type: None
	Maximum Capacity: 64 GiB
	Error Information Handle: Not Provided
	Number Of Devices: 1

Handle 0x000A, DMI type 17, 40 bytes
Memory Device
	Array Handle: 0x0009
	Error Information Handle: Not Provided
	Total Width: 64 bits
	Data Width: 64 bits
	Size: 16 GiB
	Form Factor: SODIMM
	Set: None
	Locator: Channel-0-DIMM-0
	Bank Locator: BANK 0
	Type: DDR5
	Type Detail: Unknown Synchronous
	Speed: 4800 MT/s
	Manufacturer: Unknown (0)
	Serial Number: 00000000
	Asset Tag: Channel-0-DIMM-0-AssetTag
	Part Number: None
	Rank: 1
	Configured Memory Speed: 4800 MT/s
	Minimum Voltage: 1.1 V
	Maximum Voltage: 1.1 V
	Configured Voltage: 1.1 V

Please try: CB:94042

I need to correct this, because what you are saying about my work is simply not true.

I wrote these patches myself. They were not AI-generated and they were not “vibe-coded”.

I looked at the logs, the Super I/O dump, the existing coreboot code and the behaviour I was seeing on the actual hardware. I formed hypotheses from that and wrote the changes. Some of those hypotheses may be wrong. I may have missed existing work or misunderstood part of the platform. That is fair technical criticism, and I will correct or drop patches where I got it wrong.

But that is not the same as saying that I did not write or understand my own code.

Calling it “obviously vibe-coded” is not technical review. It is a claim about how I work, and in this case that claim is false.

Regarding ASPM: you are right that the default board configuration does not enable it. I enabled PCIEXP_ASPM in my own build while experimenting with reducing the system’s power consumption. The NVMe problem I observed was a wake-up problem from L1.2, not an initial detection problem. At the time, I considered the workaround plausible because I had also verified that the stock firmware disables L1.2 on all ports.

The reason I did not want the board configuration to silently allow L1.2 when ASPM is enabled was not invented. The I226 advertises L1.2 support despite a hardware limitation that can cause packet loss and degraded performance because its L1.2 exit latency is too high under traffic:

https://www.mail-archive.com/intel-wired-lan@osuosl.org/msg11497.html

Yes, the Linux igc driver has since gained a workaround that disables L1.2 for the I226. But that does not make the firmware-side concern irrelevant. It only helps when the system is running a kernel that contains a correct version of that workaround.

The follow-up patch also shows that the original workaround did not work correctly during initial probe and required another fix:

https://www.mail-archive.com/intel-wired-lan@osuosl.org/msg12376.html

That does not automatically prove that my board-level patch was the correct solution. It may have been too broad or implemented at the wrong layer. But there was a real hardware problem and a real reason behind the change. It was not random code produced without understanding.

The RAM issue was also based on behaviour I observed with some of my modules on the BKHD-manufactured boards. BKHD is the actual board manufacturer behind several systems sold under reseller names such as Topton. Not every unit necessarily has the exact same memory, NVMe, board revision or behaviour as yours. A 49-day uptime on one configuration does not disprove instability on another configuration.

I agree that I did not have enough evidence to describe the RAM behaviour as an established platform-wide issue. I should have presented it as a hypothesis based on specific hardware observations. That was my mistake. As i wrote, i had regularly segfaults with cronjobs running when the cpu was idle which were only fixed with setting SagV to a fixed point or disabled. I switched the memory module and the behavious was the same. Maybe voltage or routing problems on the board? Can that be?

I am still learning parts of coreboot. Of course I do not have the same amount of accumulated project knowledge as someone who has worked on it for years. I have never pretended otherwise.

But I do know how to program and debug hardware. I studied computer science, specifically Kerninformatik, as well as electrical engineering at TU Dortmund, and I am from the Dortmund area. I have already ported two boards to coreboot successfully. I also got fTPM working on Broadwell without modifying the common coreboot code. To the best of my knowledge, that functionality had previously only been available for newer platform generations.

I am not mentioning that because I believe it makes my patches automatically correct. It does not. I am mentioning it because the public picture of me as someone who blindly generates code and submits it without understanding is false.

If my code is bad, tell me exactly where and why. If an assumption is wrong, explain the mechanism I missed. If there is existing work I should have found, point me to it. I can work with all of that, and I will fix or withdraw changes where appropriate.

What I cannot work with is being publicly dismissed as someone who did not write or understand his own work, or being told, directly or indirectly, that I should not have submitted code at all.

I saw your coreboot talk on YouTube and found you open, welcoming and encouraging towards people interested in contributing. That is why the speed and certainty with which you judged both my work and how it was produced are especially disappointing. The tone here feels aggressive and discouraging.

New contributors will inevitably make mistakes and lack some project-specific knowledge. That is what review is for. It should not require someone to already possess years of undocumented project knowledge before they are considered entitled to submit a patch.

Please criticize the code as strongly as technically necessary. But stop calling code I wrote myself “obviously vibe-coded”.

I have encountered real problems with these BKHD boards, and I have consistently tried to identify their causes and solve them.

@Victor — one quick practical note at the end of this discussion so the IRQ 9 storm (#4) and the lag (#5) come out conclusively in your test:

CB:94042 declares the EC but keeps irq 0x70 = 0x09 — that's IRQ 9 = the ACPI SCI on this board, the exact line the HWM is storming on, so that path stays armed. CB:92369 doesn't change the EC IRQ either (as noted above). The one that actually detaches it is my CB:94028 (irq 0x70 = 0, no IRQ routed → the HWM can't assert the SCI). And that is the way the stock firmware deals with this.

So for the storm/lag specifically, that's the change to test. Easy to verify directly: superiotool -d (LDN 0x04, reg 0x70) before/after, and watch the acpi line in grep -i acpi /proc/interrupts under load. Your board is the final arbiter — happy to adjust to whatever it shows.

@Alicja — I did have a look at CB:94042. For the interrupt storm specifically it won't resolve it: it keeps the EC on irq 0x70 = 0x09, which is IRQ 9 = the ACPI SCI — the exact line the HWM is storming on. So that path stays armed rather than detached. (CB:92369 doesn't touch the EC IRQ either, as you noted.)

Sebastian Müller wrote in #note-6:

I need to correct this, because what you are saying about my work is simply not true.

I wrote these patches myself. They were not AI-generated and they were not “vibe-coded”.

I looked at the logs, the Super I/O dump, the existing coreboot code and the behaviour I was seeing on the actual hardware. I formed hypotheses from that and wrote the changes. Some of those hypotheses may be wrong. I may have missed existing work or misunderstood part of the platform. That is fair technical criticism, and I will correct or drop patches where I got it wrong.

But that is not the same as saying that I did not write or understand my own code.

Calling it “obviously vibe-coded” is not technical review. It is a claim about how I work, and in this case that claim is false.

I apologise if that's the case. I'm happy to see newcomers learning (even though they will make mistakes in the beginning), but your patches had a tell-tail signs of AI use (unnatural commit messages, large comment blocks for single functions, confidently incorrect solutions).

Regarding ASPM: you are right that the default board configuration does not enable it. I enabled PCIEXP_ASPM in my own build while experimenting with reducing the system’s power consumption. The NVMe problem I observed was a wake-up problem from L1.2, not an initial detection problem. At the time, I considered the workaround plausible because I had also verified that the stock firmware disables L1.2 on all ports.

The reason I did not want the board configuration to silently allow L1.2 when ASPM is enabled was not invented. The I226 advertises L1.2 support despite a hardware limitation that can cause packet loss and degraded performance because its L1.2 exit latency is too high under traffic:

https://www.mail-archive.com/intel-wired-lan@osuosl.org/msg11497.html

Yes, the Linux igc driver has since gained a workaround that disables L1.2 for the I226. But that does not make the firmware-side concern irrelevant. It only helps when the system is running a kernel that contains a correct version of that workaround.

The follow-up patch also shows that the original workaround did not work correctly during initial probe and required another fix:

https://www.mail-archive.com/intel-wired-lan@osuosl.org/msg12376.html

That does not automatically prove that my board-level patch was the correct solution. It may have been too broad or implemented at the wrong layer. But there was a real hardware problem and a real reason behind the change. It was not random code produced without understanding.

Why are we suddenly talking about I226 NICs, when your patch didn't touch those RootPorts?
That being said, I just tested your suggestion for EC 0x70 on my board and while going trough Kconfig I noticed ASPM gets enabled by default.
Dropping it from Kconfig (board-level override) should "fix" all of those problems, adding flags you suggested in CB:94030 only kiiiinda helps.
Of course real solution would be getting our hands on schematics and configuring everything as it should, but if my experience with erying is anything to go by... one can dream.
I believe we could drop LegacyIoLowLatency if we knew which GPIO controls the PCIe reset and we can find it as mentioned above. I would have to pull my firewall out of service though which isn't... ideal.

The RAM issue was also based on behaviour I observed with some of my modules on the BKHD-manufactured boards. BKHD is the actual board manufacturer behind several systems sold under reseller names such as Topton. Not every unit necessarily has the exact same memory, NVMe, board revision or behaviour as yours. A 49-day uptime on one configuration does not disprove instability on another configuration.

I agree that I did not have enough evidence to describe the RAM behaviour as an established platform-wide issue. I should have presented it as a hypothesis based on specific hardware observations. That was my mistake. As i wrote, i had regularly segfaults with cronjobs running when the cpu was idle which were only fixed with setting SagV to a fixed point or disabled. I switched the memory module and the behavious was the same. Maybe voltage or routing problems on the board? Can that be?

That is very likely, you're correct about BKHD being the ODM for Topton (and many other "AliExpress specials").
However, I'm pretty sure your board is different since you mentioned lacking UART on yours. Topton has an RS232 in RJ45 form on the front and additional RS232 header (LDN2) internally.
I would say it's likely that your board's DRAM routing is off, but I can confidently say this hasn't been an issue on N100 (or N150, they're the same - just with newer SoC).

I am still learning parts of coreboot. Of course I do not have the same amount of accumulated project knowledge as someone who has worked on it for years. I have never pretended otherwise.

But I do know how to program and debug hardware. I studied computer science, specifically Kerninformatik, as well as electrical engineering at TU Dortmund, and I am from the Dortmund area. I have already ported two boards to coreboot successfully. I also got fTPM working on Broadwell without modifying the common coreboot code. To the best of my knowledge, that functionality had previously only been available for newer platform generations.

I am not mentioning that because I believe it makes my patches automatically correct. It does not. I am mentioning it because the public picture of me as someone who blindly generates code and submits it without understanding is false.

If my code is bad, tell me exactly where and why. If an assumption is wrong, explain the mechanism I missed. If there is existing work I should have found, point me to it. I can work with all of that, and I will fix or withdraw changes where appropriate.

What I cannot work with is being publicly dismissed as someone who did not write or understand his own work, or being told, directly or indirectly, that I should not have submitted code at all.

I saw your coreboot talk on YouTube and found you open, welcoming and encouraging towards people interested in contributing. That is why the speed and certainty with which you judged both my work and how it was produced are especially disappointing. The tone here feels aggressive and discouraging.

New contributors will inevitably make mistakes and lack some project-specific knowledge. That is what review is for. It should not require someone to already possess years of undocumented project knowledge before they are considered entitled to submit a patch.

Please criticize the code as strongly as technically necessary. But stop calling code I wrote myself “obviously vibe-coded”.

I have encountered real problems with these BKHD boards, and I have consistently tried to identify their causes and solve them.

Again, I apologise in this situation. I did in fact jump into conclusions your code must've been "vibe-coded" based on it's structure and well, mistakes.
This probably wouldn't have happened (especially not publicly) if we would've discussed this on coreboot chat (but then again, we have so many communication channels it makes it difficult to know where to go and should decide on one - such as IRC or Matrix).
I understand I've hurt your feelings and insulted your code, so once again - I'm sorry. Hope this won't stop you from contributing in the future :)

Sebastian Müller wrote in #note-7:

@Alicja — I did have a look at CB:94042. For the interrupt storm specifically it won't resolve it: it keeps the EC on irq 0x70 = 0x09, which is IRQ 9 = the ACPI SCI — the exact line the HWM is storming on. So that path stays armed rather than detached. (CB:92369 doesn't touch the EC IRQ either, as you noted.)

I've tested it and you're right. I wonder how stock firmware gets rid of it (since this is exactly how stock firmware configured it). Probably something cursed...
Feel free to pull other changes from CB:94042 into your patchset, I will rubber-stamp it :)

home:~# cat /proc/interrupts 
            CPU0       CPU1       CPU2       CPU3       
   4:          0          0        266          0 IR-IO-APIC    4-edge      ttyS0
   8:          1          0          0          0 IR-IO-APIC    8-edge      rtc0
   9:          0          0          0          0 IR-IO-APIC    9-fasteoi   acpi
  14:          0          0          0          0 IR-IO-APIC   14-fasteoi   INTC1057:00
  20:          1          0          0          0 IR-IO-APIC   20-fasteoi   i801_smbus
 120:          0          0          0          0 DMAR-MSI    0-edge      dmar0
 121:          0          0          0          0 DMAR-MSI    1-edge      dmar1
 122:          0          0          0          1 IR-PCI-MSI-0000:00:1c.0    0-edge      PCIe PME, PCIe bwctrl
 123:          1          0          0          0 IR-PCI-MSI-0000:00:1c.1    0-edge      PCIe PME, PCIe bwctrl
 124:          0          1          0          0 IR-PCI-MSI-0000:00:1c.2    0-edge      PCIe PME, PCIe bwctrl
 125:          0          0          1          0 IR-PCI-MSI-0000:00:1c.6    0-edge      PCIe PME, PCIe bwctrl
 126:          0          0          0          1 IR-PCI-MSI-0000:00:1d.0    0-edge      PCIe PME, PCIe bwctrl
 127:          0         51          0          0 IR-PCI-MSI-0000:00:16.0    0-edge      mei_me
 128:          0          0          0       3109 IR-PCI-MSI-0000:00:17.0    0-edge      ahci[0000:00:17.0]
 129:        261          0          0          0 IR-PCI-MSI-0000:00:14.0    0-edge      xhci_hcd
 134:          0          0       3561          0 IR-PCI-MSI-0000:05:00.0    0-edge      mt7921e
 135:          0          0          1          0 IR-PCI-MSIX-0000:01:00.0    0-edge      eth0
 136:          0          0          0      96989 IR-PCI-MSIX-0000:01:00.0    1-edge      eth0-TxRx-0
 137:       2981          0          0          0 IR-PCI-MSIX-0000:01:00.0    2-edge      eth0-TxRx-1
 138:          0       4309          0          0 IR-PCI-MSIX-0000:01:00.0    3-edge      eth0-TxRx-2
 139:          0          0       2108          0 IR-PCI-MSIX-0000:01:00.0    4-edge      eth0-TxRx-3
 140:          1          0          0          0 IR-PCI-MSIX-0000:02:00.0    0-edge      eth1
 141:          0       5455          0          0 IR-PCI-MSIX-0000:02:00.0    1-edge      eth1-TxRx-0
 142:          0          0      13677          0 IR-PCI-MSIX-0000:02:00.0    2-edge      eth1-TxRx-1
 143:          0          0          0       5969 IR-PCI-MSIX-0000:02:00.0    3-edge      eth1-TxRx-2
 144:       6248          0          0          0 IR-PCI-MSIX-0000:02:00.0    4-edge      eth1-TxRx-3
 150:          0          0          0          0 IR-PCI-MSIX-0000:04:00.0    0-edge      eth3
 151:          0          0          0          0 IR-PCI-MSIX-0000:04:00.0    1-edge      eth3-TxRx-0
 152:          0          0          0          0 IR-PCI-MSIX-0000:04:00.0    2-edge      eth3-TxRx-1
 153:          0          0          0          0 IR-PCI-MSIX-0000:04:00.0    3-edge      eth3-TxRx-2
 154:          0          0          0          0 IR-PCI-MSIX-0000:04:00.0    4-edge      eth3-TxRx-3
 155:          0          0          0         27 IR-PCI-MSI-0000:00:02.0    0-edge      i915
 156:          0        271          0          0 IR-PCI-MSI-0000:00:1f.3    0-edge      snd_hda_intel:card0
 NMI:          1          0          1          2   Non-maskable interrupts
 LOC:      70420      62936      66178      78385   Local timer interrupts
 SPU:          0          0          0          0   Spurious interrupts
 PMI:          1          0          1          2   Performance monitoring interrupts
 IWI:          3          2          0          7   IRQ work interrupts
 RTR:          0          0          0          0   APIC ICR read retries
 RES:        195        240        203        191   Rescheduling interrupts
 CAL:      11815      19203      16211      21532   Function call interrupts
 TLB:        248        162        419        684   TLB shootdowns
 TRM:          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0   Threshold APIC interrupts
 DFR:          0          0          0          0   Deferred Error APIC interrupts
 MCE:          0          0          0          0   Machine check exceptions
 MCP:          7          8          8          8   Machine check polls
 ERR:          0
 MIS:          0
 PIN:          0          0          0          0   Posted-interrupt notification event
 NPI:          0          0          0          0   Nested posted-interrupt event
 PIW:          0          0          0          0   Posted-interrupt wakeup event