BCM MSI interrupt configuration

  

    BCM 交换芯片56960配置MSI中断方法

Describes how to configure and verify whether the switch is properly generating the MSI interrupt message using the BCM diagnostic shell.

 

Description

Message Signaled Interrupts (MSI) were introduced as an alternative to line triggered interrupts. Instead of requiring a dedicated pin, devices which support MSI can trigger an interrupt by writing a value to a physical location, which is often a memory-mapped control register inside an interrupt controller. All recent Broadcom switch devices support MSI. The following describes how to configure and verify whether the switch is properly generating the MSI interrupt message using the BCM diagnostic shell.

While currently, the reference Linux BDE in the SDK  does not enable MSI by default, several customers use MSI in their designs. As a result, whenever issues related to interrupts arise, customers raise questions as to whether the device is properly generating MSI or not.

MSI configuration is done by configuring the MSI capability in the PCI configuration space. The MSI capability registers are:

 

 31                 16 15       8| 7       0|

+---------------------+---------------------+

|Message Control Reg  |Ptr NextID|CapId05h  |

+-------------------------------------------+

|   LSB 32-bits Message Address Register    |

+-------------------------------------------+

|   MSB 32-bits Message Address Register    |

+-------------------------------------------+

                      |Message Data Register|

                      +---------------------+

 

The 16-bit Message Control Register has:

 

 15                   8 7 6   4 3    1 0

+---------------------------------------+

| Reserved             | |     |      | |

----------------------------------------

 

Where:

 

  7   : 64bit address capable

  6..4: Multiple message enabled

  3..1: Multiple message capable

  0   : MSI enabled

 

Examining the PCI configuration registers on a switch without MSI enable:

 

BCM.0> d pcic

0000: b85014e4  DeviceID=b850  VendorID=14e4

0004: 00100146  Status=0010  Command=0146

0008: 02000002  ClassCode=020000  RevisionID=02

000c: 00000008  BIST=00  HeaderType=00  LatencyTimer=00  CacheLineSize=08

0010: 60000004  BaseAddress0=60000004

0014: 00000000  BaseAddress1=00000000

0018: 00000000  BaseAddress2=00000000

001c: 00000000  BaseAddress3=00000000

0020: 00000000  BaseAddress4=00000000

0024: 00000000  BaseAddress5=00000000

0028: 00000000  CardbusCISPointer=00000000

002c: b85014e4  SubsystemID=b850  SubsystemVendorID=14e4

0030: 00000000  ExpansionROMBaseAddress=00000000

0034: 00000048  Reserved=000000  CapabilitiesPointer=48

0038: 00000000  Reserved=00000000

003c: 00000101  Max_Lat=00  Min_Gnt=00  InterruptPin=01  InterruptLine=01

0040: 00000000  Reserved=00  RetryTimeoutValue=00  TRDYTimeoutValue=00

0044: 00000000  PLLConf=0

0048: c8035001  -

0048: c8035001  CapabilityID=01 CapabilitiesPointer=50 PWR-MGMT

004c: 00002008  -

0050: 00005803  CapabilityID=03 CapabilitiesPointer=58 VPD

0054: 00000000  -

0058: 0080ac05  CapabilityID=05 CapabilitiesPointer=ac MSI

005c: 00000000  -

0060: 00000000  -

0064: 00000000  -

0068: 00000000  -

006c: 00000000  -

00ac: 00020010  CapabilityID=10 CapabilitiesPointer=00 PCIE

00b0: 003c8002  -

00b4: 00191c20  -

BCM.0>

 

Bit 0 of the MSI message control register in MSI capability register at 0058 is not set, MSI is not enabled. NOTE: This is bit 16 if reading the entire contents of 0058.

The following steps demonstrate how to enable MSI and verify MSI on a switch running the SDK in user-mode Linux.

Note of Caution: Be aware that this is just an exercise to verify whether the switch is properly generating MSI. After enabling MSI with the steps below, real interrupts will no longer be generated, so all interrupt-based services, e.g. MII access, packet DMA, hardware linkscan, etc., will not operate properly. Furthermore, since the steps below reconfigure the switch, restoring the system to normal behavior requires either explicitly disabling MSI, or power cycling the system.

Step 1. Prepare the system before configuring MSI

 

Before MSI can be configured, make sure to stop all interrupt based services.

 

BCM.0> link off

BCM.0> l2mode off

BCM.0> counter off

If memscan is running,

 

BCM.0> memscan off

MemSCAN: Stopped on unit 0

Step 2. Allocate memory for the MSI message

 

For this exercise, the message is written to a memory location which can be examined before and after the switch generates an MSI. The MSI configuration must know the physical write address for the message.

The SDK diagnostic shell has a convenient command 'dma' which allocates memory suitable for access by controllers other that the CPU. We allocate eight 32 words to use to test MSI.

 

BCM.0> dma alloc w 8 MSITest

Allocated 32 bytes of DMA-able memory at address 0x30640900

 

dma alloc sets $dma_laddr for convenience. Now, examine the contents of the newly allocated memory. $dma_laddr is the logical (virtual) memory address available to applications.

 

BCM.0> echo $dma_laddr

0x30640900

BCM.0> dump mw $dma_laddr 8

0x30640900: 00000000 00000000 00000000 00000000

0x30640910: 00000000 00000000 00000000 00000000

 

To configure MSI, the physical address that corresponds to the the logical address must be programmed into the MSI message address. To obtain the physical address,  map logical to physical address of the allocated buffer using dma l2p.

 

BCM.0> dma l2p  $dma_laddr

l2p(0x30640900) = 0x1d962900

 

The physical address is saved in $dma_paddr.

 

BCM.0> echo $dma_paddr

0x1dbf9600

Step 3. Configure MSI

Configuring MSI requires writing the MSI capability registers 0058..0064

 

BCM.0> d pcic

0000: b85014e4  DeviceID=b850  VendorID=14e4

0004: 00100146  Status=0010  Command=0146

0008: 02000002  ClassCode=020000  RevisionID=02

000c: 00000008  BIST=00  HeaderType=00  LatencyTimer=00  CacheLineSize=08

...

0058: 0080ac05  CapabilityID=05 CapabilitiesPointer=ac MSI

005c: 00000000  -

0060: 00000000  -

0064: 00000000  -

 

First write the lower 32 bits of the message address with the physical address obtained in step 2 above:

 

BCM.0> s pcic 0x5c $dma_paddr

             In case of SDK 6.4.X or later please use following command

             BCM.0> pcic write addr=0x5c data=<value of dma_paddr>

 

Next write the upper 32 bits of the message - in this case 0:

 

BCM.0> s pcic 0x60 0

             In case of SDK 6.4.X or later please use following command

             BCM.0> pcic write addr=0x60 data=0x0

Then write the message contents:

 

BCM.0> s pcic 0x64 0x0000cafe

             In case of SDK 6.4.X or later please use following command

             BCM.0> pcic write addr=0x64 data=0x0000cafe

Finally, enable MSI by setting bit 16 - actually bit 0 of the MSI control register:

 

BCM.0> s pcic 0x58 0x0081ac05

             In case of SDK 6.4.X or later please use following command            

             BCM.0> pcic write addr=0x58 data=0x0081ac05

Dump the pcic registers to confirm the MSI configuration:

 

BCM.0> d pcic

0000: b85014e4  DeviceID=b850  VendorID=14e4

0004: 00100146  Status=0010  Command=0146

0008: 02000002  ClassCode=020000  RevisionID=02

...

0058: 0081ac05  CapabilityID=05 CapabilitiesPointer=ac MSI

005c: 1d962900  -

0060: 00000000  -

0064: 0000cafe  -

 

Step 4. Verify MSI interrupts

 

Now that the switch is configured for MSI, to verify the switch is generating MSI, trigger an interrupt. This can be done several ways: by reading a PHY register, or doing a packet DMA, etc. Here the internal phy register 0x1f is read for xe0. Before triggering the switch to generate an interrupt, examine the message memory area:

 

BCM.0> dump mw $dma_laddr 1

0x30640900: 00000000

In case of SDK 6.4.X or later please use following command

BCM.0> xmem r <value of dma_paddr>

 

The phy register read should trigger the switch to generate a MSI as configured:

 

BCM.0> phy int xe0 0x1f

Port xe0 (PHY addr 0x81) Reg 31: MiimTimeOut:soc_miim_read, timeout (id=0x81 addr=0x1f)

 

ERROR: Port xe0: soc_miim_read failed: Operation timed out

 

Please notice the SDK still expects interrupts to arrive, but the test has simply configured the switch to write the MSI message to a memory area, just for the sake of testing. If the message area is examined, it shows the switch has written the message:

 

BCM.0> d mw $dma_laddr 1

0x30640900: feca0000

BCM.0>

             In case of SDK 6.4.X or later please use following command

BCM.0> xmem r <value of dma_paddr>

Note: if the CPU used to run this test does not support hardware cache coherency, there is a need to invalidate the data cache before the message is read. For example:

 

BCM.0> dma inval w $dma_laddr 1

 

As shown above,  the MSI message 0xcafe (byte-swapped) was written by the switch to configured MSI message area.

 

Step 5. Cleaning Up

 

To re-enable the switch to generate real interrupts, disable the MSI

 

BCM.0> s pcic 0x58 0x0080ac05

             In case of SDK 6.4.X or later please use following command

      BCM.0>pcic write addr=0x58 data=0x0080ac05

Interrupts should be working again:

 

BCM.0> phy int xe0 0x1f

Port xe0 (PHY addr 0x81) Reg 31: 0xffc0

BCM.0>