FLX-182 Review

Review summary meeting

Mandate

1. Compliance with functional and performance requirements (where this can be assessed at the design stage)
2. Mechanical, electrical and thermal stability (assuming 2 such cards populating a 2U server, alongside NICs)
3. Ease of manufacture for a third-party company based on design and provided documentation (PCB and board assembly)
4. Ease with which this design could evolve to one based on a Xilinx Versal Premium (VP-1552) FPGA with PCIe Gen 5, i.e. are there any aspects of the current design which could be modified to save work/cost later?

Materials

• Specification document
• Xpedition project
• Schematic (pdf)
• Board layout (pdf)
• Overview of FLX-181
• Overall system specification documents: see links in the January PDR agenda

Reviewers

• lorne.levinson@weizmannNOSPAMPLEASE.ac.il
• sorin.martoiu@cernNOSPAMPLEASE.ch
• basvdh@nikhefNOSPAMPLEASE.nl (Bas van der Heijden)
• charles.ietswaard@nikhefNOSPAMPLEASE.nl
• marius.wensing@cernNOSPAMPLEASE.ch
• f.schreuder@nikhefNOSPAMPLEASE.nl

Review strategy

• Divide the board's functional areas: PCIe, memory, links, board management, Power converter, Config and Flash, Busy
• Clock distribution and jitter cleaning
• For each requirement, walk thru' the schema to confirm that it is completely supported.
• Review tasks for the FLX-182 (google sheet)

Questions

• How can we evaluate if fulfilling requirements will be limited by FPGA resources?
• How do we configure flash? Is a special Linux required for the ARM processor?
• Which board simulations were done? Just the high-speed signals, or all? Cross-talk?
• Can they provide the test reports for the various tests that they have done?
• Does a thermal model exist?
• Resolve the FLX-712 issue of Bad Flash or boot timing

Remarks

• PCIe_PERST_B is connected to HDIO Bank 406 (K21), this works for the PCIe hard blocks, but not for CPM PCIe (FS, 21-06-2022)
  • Keep the connection to Bank 406 (FS, 21-06-2022)
  • Also connect PCIe_PERST_B to PMC_MIO 38 and PMC_MIO 39, so R337 must be placed and an additional connection to PMC_MIO 39 must be made. (FS, 21-06-2022)
• The part number XCVM1802-1MSEVSVA2197-ES9780 implies an engineering sample. I don't think the engineering sample will be used. (FS, 21-06-2022)
• U31 (LTI_REF_CLK) is specified at 40.08 MHz, but the minimum gtrefclk for the GTY is 60 MHz. I would therefore set it to 160.316 or 240.474MHz (FS, 24-06-2022)
• U53 (IP4856CX25) is not recommended for new designs (NRND) and cannot be ordered anymore (MW, 27-06-2022)
• PEX is in the BOM and in the Specification, but in fact, is not in the design. (FS)
• Missing AC coupling on feedback input to SI5345 for U45, U46. (LL)
• Ethernet chip dp83867isrgz:: none in stock, lead time: 938 Expected 05-May-23 (LL)
• 569BAAA000118BBG is used, but the title on schema is CVPD-922. (Sheet 45) (LL)
• Si5345: Why is there an option to put the two Si5345 in series, and not in parallel? (LL)
• Si5345: We need one output clock on a clock capable IO pin of the FPGA as a 320.000MHz TCLINK offset clock (FS)
  • Feed SA_REF_CLK_8 P/N to an FPGA (clock capable pin) (No need to route it to SI5345B)
  • Feed SB_REF_CLK_8 P/N to an FPGA (clock capable pin) (The SMP connectors J8/J9 could be connected through DNP 0R resistors)
  • Use the same input clocking scheme for SI5345A and SI5345B (IN0: FPGA pin, IN1: Si570, IN2: WR 125) (Allows using the same config file.)
• Layout remarks: FLX-182_Comments_Charles.xlsx (CI)
• from Jos:
  1. There is no BUSY output foreseen other than the BUSY output to the LTI. This is of course what is needed for Phase-2 running, but in lab tests, it may be useful to have a BUSY output as we have on the 712. In general, if realisable in a simple way, it may be useful to have access to several GPIO pins of the FPGA, ideally via the bracket or otherwise via a header.
  2. There is no legacy TTC input. This still may be useful in absence of an LTI in a lab or testbeam environment where several cards need to be clocked by the same reference clock. However, I realise that it will be problematic to implement this. Emulation of LTI functionality with a suitable COTS FPGA board probably can provide a solution, it would be good to investigate how this can be done. With an LTI emulator also BUSY could be handled, but for simple setups nevertheless, a 712-like BUSY output may be desirable.
  3. The effective bandwidth provided by the PCIe Gen4 interface is, assuming that it is twice the effective bandwidth of the Gen3 interface, about 24 GB/s, which appears to be fine for 24 FULL mode or 24 lpGBT links. However, only the data of the order of 10 25 Gb/s links running at full speed can be transferred via the PCIe interface, if half of the total amount of data is transferred via one PCIe endpoint and the other half via the other PCIe endpoint. Therefore: is it possible to transfer the data of 6 25 Gb/s links from one Firefly via one endpoint and the data from the other 6 links of that same Firefly via the other endpoint? If not two 25 Gb/s Firefly interfaces are needed, which I understood to cost considerably more than lower speed devices.
• from Marko.Mikuz@cernNOSPAMPLEASE.ch: The BCM' needs two electrical signals capable of 40MHz digital data transmission (Beam Permit, Injection Permit to ATLAS CIBU) to be routed out of the Felix Phase-II card.
• Schematic remarks from Bas: schematic_remarks_Bas.txt
• Power remarks: (SM)
  • DC/DC converters are not synchronized. This may lead to instantaneous high currents being drawn from the input.
  • VCCINT: LTM4700 delivers 50A max not 100A as specified in the schematic. Is it sufficient?
  • sense inputs configured to be used in a short local loop (DNP resistors) rather than long loop close to the consumer (preferable especially with high currents)
• Clock: - should a separate PLL for WR/Eth RX/TX loop be useful to free up other PLLs for various clocking scenarios in the future? (SM)
• Confirm that the MTP rows for TX and RX are the same as those used by the FLX-712. (LL)

BOM comments

designator	part	comment
U1	LTM4700IY	long lead-time (Aug2023 @mouser)
U39, U40	SN74LVC2G07DCKT	long lead-time (May2023 @mouser)
U41	TPS389001DSER	long lead-time (Nov2023 @mouser)
U44	TPS51200DRCT	long lead-time (Jul2023 @mouser)
U48, U50, U51	TCA9617A	long lead-time (May2023 @mouser)
U53	IP4856CX25	obsolete and not longer produced
U56	DP83867ISRGZ	long lead-time (May2023 @mouser, 77 weeks from manufacturer)
U60, U61	MT25QU02GC	long lead-time (Mar2023 @mouser)
U68, U69	ADM1066ACPZ	long lead-time (79 weeks @mouser)
U72-U78	TMP435ADGSR	long lead-time (digikey suggests MAX31730AUB + as in-stock replacement)
U8-U17	INA226AIDGSR	long lead-time (Jul2023 @mouser)
U95	TPSM5601R5SRDAR	long lead-time (Jul2023 @mouser)

Checks

• When a part has more than one package option, is the chosen option more likely to become unavailable or obsolete?
• Direct access to flash over PCIe through Programmable logic (PL)
• JTAG:
  • JTAG chain and options
  • Direct JTAG access to the Flash
  • USB connector to Digilent JTAG, but not micro-USB
• Heat sink mounting with screws to the PCB and fan quality
• Check power supply options set by pins
• Power distribution integrity, voltage drops
  • Are return currents in the grounds blocked or restricted in any way?
• Thermal issues
  • Is the airflow to any hot components blocked by other components?
  • Confirm no solder mask over copper flood under hot chips
• Check all high-speed lines for excessive vias, stubs, etc.
  • Differential pairs: has the layout tool guaranteed matching lengths?
  • Has the PCB manufacturer confirmed the geometry of all controlled impedance traces?
• Miscellaneous layout checks
  • matched length for high-speed buses
  • Review placement of bypass caps for the FPGA
  • Are there any test points connected to FPGA pins?
  • Are any parts infringing on the keep-out areas of the board edges?
  • are the mounting holes for the card bracket grounded? Are they the size and at the correct location?
  • If there are split ground or power planes, do any traces on adjacent signal planes cross the gap?
  • Do ground planes extend slightly closer to PCB edges than voltage planes (to reduce EMI)?
  • Sufficient distance of all signal traces from edges of planes and from edges of the board on outside layers
  • Are all power vias large enough?
  • Is there a place for board name and serial number?
  • clock lines: fewer vias, more isolation
  • For parts not used in a previous design, double-check that their footprints and pin numbers match the datasheet.
• Stack-up
  • copper thickness of power and return planes
  • potential board warping due to asymmetry of copper in the stack-up

BNL181 resource utilization for different firmware flavours

Remark for all device images in the table below: The BNL181 uses the VM1802-es1, which enumerates as a PCIe Gen1 device when built with Vivado 2022.1 (or > 2020.2), PCIe gen4 works with Vivado 2020.2, but routing fails when utilization is > ~50%

Flavour	Channels	LUTs	FF	BRAM	URAM	Power	Remarks	Device image
FULLMODE	24	58%	38%	38%	69%			FLX181_FULLMODE_24CH_CLKSELECT_GIT_phase2-FLX-1769_AddGBTForVCU128_rm-5.0_2576_220620_14_19.tar.gz
GBT	24	58%	45%	94%	73%		Semistatic, Build took 20 hours	FLX181_GBT_SEMISTATIC_24CH_CLKSELECT_GIT_phase2-FLX-1769_AddGBTForVCU128_rm-5.0_2576_220620_14_19.tar.gz
LPGBT	24	84%	39%	87%	69%	31.5 W	WNS -1.45ns	FLX181_LPGBT_24CH_CLKSELECT_GIT_phase2-FLX-1769_AddGBTForVCU128_rm-5.0_2579_220624_12_51.tar.gz
STRIP	24	137%	33%	100%	90%		Possible to reduce utilization? See link	FLX-1941
STRIP	24	101%	45%	67%	90%		Opt design after reduction of E-Links (FLX-1941)	FLX-1941
STRIP	24	65%	45%	76%	90%	30.4 W	See details: (FLX-1941)	FLX181_STRIPS_24CH_CLKSELECT_GIT_phase2-FLX-1769_AddGBTForVCU128_rm-5.0_2603_220705_10_01.tar.gz
PIXEL	24

-- LorneL - 2022-06-15