The Lpgbt: a Radiation Tolerant ASIC for Data, Timing, Trigger and Control Applications in HL-LHC

The lpGBT: a radiation tolerant ASIC for Data, Timing, Trigger and Control Applications in HL-LHC

Paulo Moreira, CERN On behalf of the lpGBT team

Topical Workshop on Electronics for Particle Physics 02 – 06 September 2019, Santiago de Compostela – Spain lpGBT Team

Design/Test • CERN David Porret, Daniel Hernandez, Julian Mendez, Jose Fonseca, Ken Wyllie, Nour Guettouche, Paulo Moreira, Pedro Leitao, Rui Francisco, Sophie Baron, Stefan Biereigel, Szymon Kulis • AGH UST Marek Idzik, Miroslaw Firlej, Jakub Moroń, Tomasz Fiutowski, Krzysztof Swientek • KU Leuven Bram Faes, Jeffrey Prinzie, Paul Leroux • UNL – FCT João Carvalho, Nuno Paulino • SMU Physics Datao Gong, Di Guo, Dongxu Yang, Jingbo Ye, Quan Sun, Tiankuan Liu, Wei Zhou • SMU Engineering Tao Zhang, Ping Gui

Building blocks • Czech Technical University Prague Miroslav Havranek, Tomas Benka • CERN Stefano Michelis, Iraklis Kremastiotis, Alessandro Caratelli, Kostas Kloukinas

2 lpGBT Link Architecture

High radiation doses No or small radiation doses

LHC: up to 100 Mrad (1014 1MeV n/cm2) HL – LHC: up to 1 Grad (1016 1MeV n/cm2) Short distance optical links: 50 to 300 m

FPGA Timing & Trigger Timing & Trigger TIA PD DAQ lpGBT DAQ LD LD Slow Control Slow Control

Custom ASICs On-Detector Off-Detector Radiation Hard Electronics Commercial Off-The-Shelf (COTS)

Electrical links to the frontend modules. Lengths: cm to few m Custom optoelectronics

3 More than a “Communications ASIC” • Capable of • 5.12 or 10.24 Gbps (for uplinks) • 2.56 Gbps (for downlinks) • Enables the implementation of RadTol links • DAQ • Trigger (constant and deterministic latency) • Experiment control [slow control] • Implements Control and Monitoring Functions • Three I2C Masters • 16 – bit General Purpose I/O port • Output reset pin • 10 – bit ADC (8 multiplexed inputs) Pin count: 289 (17 x 17) • 8 – bit voltage DAC Pitch: 0.5 mm • 8 – bit current DAC Size: 9 mm x 9 mm x 1.25 mm • Temperature sensor • Designed for radiation hardness • M. Firlej, et al., “An lpGBT sub-system for Total Ionizing Dose (TID): 200 Mrad environmental monitoring and control of • Extensive SEU protection (TMR, FEC) experiments” (Fri @ 09:50)

4 Communicates with • The counting room • Optical fibre links • The FE modules / ASICs • Electrical links (eLinks) • The Number and Bandwidth of eLinks is programmable • For Down eLinks • Bandwidth: 80/160/320 Mbps • Count: 16/8/4 • For Up eLinks

5 eLinks Special Features

• Ad hoc signalling “standard” (CLPS) • Differential for noise immunity and EMI • 100 W for good signal integrity • Low voltage swing for low power • eLink drivers (eTx) • Programmable amplitude • Programmable pre-emphasis • Data invert function • eLink Receivers (eRx) • Programmable equalization (3 settings) • Data invert function D. Guo, et al., “The eTx line driver and the eRx line receiver: two building blocks • Hi – Z or 100 W termination for data and clock transmission using the • External biasing for DC coupling CLPS standard” (Tue @ 17:20) • Internal biasing for AC Coupling • Phase – aligner mechanisms for error S. Kulis, et al. “A multi-channel multi-data free data reception rate circuit for phase alignment of data in • manual / semi-automatic / automatic the lpGBT” (Fri @ 11:30)

6 High – Speed links Special Features

• Transmitter RX Continuous Time Linear Equalizer • 5.12 / 10.24 Gbps • Programmable amplitude • Programmable pre-emphasis • Receiver • 2.56 Gbps • Programmable equalization • Reference and Reference-less locking • Built-In test features • HS – Loopback (2.56 Gbps) • PRBS generators / checkers • Eye Opening Monitor Q Sun, et al., “Downlink Equalization and • FEC error counter Eye Opening Monitor in the lpGBT” (Thu @ 10:00)

7 Timing Functions • Deterministic & fixed latency • For clocks • For data (up and down links) • Programmable frequency clock outputs • 29 outputs • 40 / 80 / 160 / 320 / 640 / 1280 MHz • Programmable phase/frequency clock outputs • 4 outputs • 50 ps phase-resolution • 40 / 80 / 160 / 320 / 640 / 1280 MHz S. Biereigel, et al., “The lpGBT PLL and • Low jitter CDR Architecture, Performance and SEE • RMS < 5 ps @ 40 MHz Robustness” (Fri @ 09:00)

8 Built In Test Features • Ring oscillators All curves normalized to CH1 at 1.2V • Four instances VDD = 1.08 / 1.2 / 1.32 V • One in each “corner” of the ASIC “Shades” correspond to the four channels • The chip can measure their oscillation frequency T = 10˚ C f ≈ 40 MHz @ 1.2 V • To estimate the “process corner” nom • To track the degradation with TID • Not a “dosimeter”! • SEU Monitoring Memory • 2500 Self Resetting Memory cells • Upset rate estimation! • TMR Testing • Enables/disables clock TMR • Verification of TMR during production

9 The lpGBT is Low-Power!

• CMS PS Module (10Gbs, FEC5) 410 mW (@ Vdd = 1.2V) • Uplink 14 x 640 Mbps • Downlink 2 x 320 Mbps • eClocks 2 x 320 MHz + 2 x 640 MHz • CMS 2S Module (5Gbs, FEC12) 330mW (@ Vdd = 1.2V) • Uplink 10 x 320 Mbps • Downlink 2 x 320 Mbps • eClocks 2 x 320 MHz • CMS Pixels (10Gbs, FEC5) 320mW (@ Vdd = 1.2V) • Uplink 7 x 1.28 Gbps • Downlink 7 x 160 Mbps • eClocks disabled • ATLAS ITK Strips (10Gbs, FEC5) 380 mW (@ Vdd = 1.2V) • Uplink 14 x 640 Mbps • Downlink 8 x 160 Mbps • eClocks 4 x 160MHz • Low speed system (10Gbs, FEC5) 490 mW (@ Vdd = 1.2V) • Uplink 28 x 320 Mbps • Downlink 16 x 80 Mbps • eClocks 16 x 40 MHz

10 lpGBT Prototyping • Tapeout • 25/07/2018 • 1st Prototypes • 300 Pieces • Foundry done: 12/10/2018 (die at CERN) • Packaging done: 21/11/2018 (packaged chips at CERN) • Evaluation testing • Chips proved “99.9% functional”! • A few “functions” need performance improvement! • 2nd Prototype batch • 300 chips ordered Dec 2018 • 1st + 2nd batch • 50 “naked” chips for SEU and TID testing •  150 ASICs distributed to the users • Currently  400 tested chips still available for distribution • 3rd Prototype batch • 400 chips ordered July 2019

11 Testing

• Test system Evaluation Testing @ CERN • Based on the Xilinx Virtex-7 FPGA VC707 Evaluation Kit Testing started November 2018 and a custom mezzanine board • Test cycle: • More than 1000 parameters measured at 7 supply SEU – Heavy Ions Heavy Ion Facility (HIF) voltages (including [rough] characterization of some March 2019 @ UCL Louvain, Belgium circuits: BERT, ADC, DAC,…) • All this in ≈ 220 s

SEU – Two Photon Testing Two Photon Absorption (TPA) facility May 2019 @ KU Leuven, Belgium

TID – X-Ray (up to 500 Mrad) X – Ray facility May / June 2019 @ CERN

TID – Annealing @ CERN July 2019

Temperature (-25 ˚C to 60 ˚C) @ CERN J. Mendez, et al., “LpGBT Tester: an FPGA S. Biereigel, et al., “Methods for Clock July 2019 based test system for the lpGBT ASIC” Signal Characterization using FPGA (Tue @ 17:20) Peripherals” (Thru @ 16:55)

12 Performance (1/…)

Function Circuit Functionality Performance TID SEU ASIC Control I2C Slave     (1) IC channel    N.A. EC port    N.A. Configuration memory    X eFuses   X N.A. Startup & WD     Power ON RST     Brown-Out Detector  X X N.A. (1) See note (7) Legend  – Functionality/performance good!  – Functionality/performance satisfactory but some aspects of it need improvement! X – Functionality/performance unsatisfactory must be reviewed! N.A. – Does Not Apply or wasn’t specifically tested. 13 Performance (2/…)

Function Circuit Functionality Performance TID SEU HS Transmitter PLL/CDR     (2) Tx @ 5.12 Gbps     Tx @ 10.24 Gbps     PRBS Generators    N.A. HS loopback   N.A. N.A. HS Receiver Equalizer     CDR     Frame Aligner   (3)   Rx @ 2.56 Gbps     Eye Opening Monitor   N.A. N.A. PRBS Checkers   N.A. N.A. (2) See: S. Biereigel, et al., “The lpGBT PLL and CDR Architecture, Performance and SEE Robustness” (Fri @ 09:00)

(3) Frame aligner fails to capture some of the unlock conditions making for a relatively long relocking time after an SEU 14 Performance (3/…)

Function Circuit Functionality Performance TID SEU ePortRx Deserializer    (4)  Phase-Aligner     PRBS Checkers   N.A. N.A. ePortTx Serializer     PRBS Generators   N.A. N.A. Mirror function     eTx     eTx + pre-emphasis     eRx     eRx + equalization    

(4) See: S. Kulis, et al. “A multi-channel multi-data rate circuit for phase alignment of data in the lpGBT” (Fri @ 11:30)

15 Performance (4/…)

Function Circuit Functionality Performance TID SEU Experiment control and monitoring I2C master (x3)    (5) N.A. GPIO     REST OUT     Reference voltage    N.A. Temperature sensor    N.A. 8[12]-bit Voltage DAC    N.A. 8-bit Current DAC    N.A. 10-bit ADC    N.A.

(5) The Yield of the I2C masters is low. Particularly, in about 43% of the ASICs at least one of the I2C masters fails. This is due to a timing problem (identified) in the logic. The yield of master “0” is the lowest. “Curiously”, some of the channels might “popup back into life” with Total Irradiation Dose!

16 Performance (5/…)

Function Circuit Functionality Performance TID SEU Clocks eClocks  (6)   Phase-Shifter     Process Monitors and Testing Ring oscillator (x4)    N.A. SEU Monitoring Memory   N.A.  TMR Testing  (7) N.A. N.A. N.A. Power Consumption    

(6) Measurements show the presence of systematic jitter (approaching ±10 ps P-P) in all the clocks except at 40 MHz. Additionally, significant duty cycle distortion is present in the high frequency clocks. This phenomena is not yet well understood, although the systematic jitter can be correlated with digital activity in the ASIC. (7) A logic mistake (identified) prevents accessing the ASIC through the I2C interface once one of the TMR clocks (40 MHz) is disabled!

17 TID and Power Consumption @ 10 Gbps & FEC 12

1.32 V

1.20 V

1.08 V

VDD (“All that is digital + I/O”)

VDDTX (HS TX)

VDDRX (HS RX + PLL + CLK GEN)

18 eFuses • At least for lpGBTs serving as masters, configuration data is read from the eFuses at Power-Up Reading the eFuses: Error Probabilities • This has to be done reliably so that communications can be established with the frontend modules and the counting room • However, X-ray irradiation tests have proven the fuse reading process to degrade with TID • At around 150 Mrad, “bit flips” can be observed during the read process! • Solutions to be implemented • Redesign the sense amplifier to avoid an offset buildup with TID • Add CRC to the configuration. If the computed CRC doesn’t match the stored one, the configuration will be reloaded until a match is found

19 Configuration Memory • Heavy Ion Tests proved the • It was decided to change to fully configuration memory to be synchronous loading of the susceptible to SEUs! configuration memory to avoid this • Since the ASIC operation relies problem and to allow the use of entirely on the values stored in the proven TMR design methodologies. configuration memory, this is a serious problem! • Two-Photon laser testing and RTL simulations allowed to pin-point the root cause • Values are loaded into the memory by asynchronous reset/set of the registers. • The reset/set signal originates from triplicated flip-flops but [unfortunately] their outputs are not voted! • If the RST signal is activated by an SEU, given its duration (up to 25 ns), the asynchronous self correcting mechanism is “overwritten”

20 Phase-Shifter

• Testing showed the Phase-Shifter not to be able to lock for the lowest specified supply voltage (VDD = 1.08V)! VDD = 1.08 V • Radiation only made things worse • VDD = 1.2 and 1.32 also affected! • What went wrong? VDD = 1.14 V • For this circuit, post-layout simulations were done with ASSURA extraction of parasitics. VDD = 1.20 V

• After the problem was found the circuit was re- VDD = 1.26 & 1.32 V simulated using CALIBRE to extract the parasitics • It was found that the delays estimated based on the ASSURA extraction were considerably lower than the ones obtained using CALIBRE, which seem to be closer to reality! • Notice that this is not a problem of the tools but rather that ELT modeling was never properly done for ASSURA • It was concluded that the architecture chosen for the delay-cell in the phase-shifter can’t achieve 48.8 ps (the target value) under all PVT corners. • A change of architecture in which inverting delay cells (instead on non-inverting) will be used is under study

21 Clock Performance • Random jitter component: • Jc-c< 2 ps RMS in PLL and CDR modes (all clocks) • However clocks 80 MHz and higher display a systematic jitter component: • Periodicity 25 ns (thus clearly correlated with the lowest clock frequency, 40 MHz, in the ASIC) • Can be as high as ±15 ps P-P • The P-P amplitude correlates well with chip digital activity • Coupling mechanism not yet understood!

Reference 1.6 ps RMS

22 Brown-Out Detector (1/2)

VDD

RST pulse VT To the RST manager VT extender

tripVoltageAdjust [2:0]

• Constantly monitors VDD • During Power Up • During operation

• During power up, holds the RST signal Active while VDD is below a specified Trip (VT) value [~ 90% VDD(min)]

• During operation, activates the RST if the supply voltage drops below VT • For SEU immunity 3 devices are instantiated in the lpGBT • The lpGBT can be configured to work without them

23 Brown-Out Detector (2/2) • Circuit functionally OK but • The three instances show a large dispersion in value of the trip voltage! • Radiation makes things even worse! • Circuit needs to be improved for matching to • Reduce the comparator offset dispersion

• Dispersion of the VT generator output voltage

24 Summary • The lpGBT is a highly versatile RadTol ASIC implementing a multitude of modes, bandwidths and many of the functions required to implement Experiment Control and Monitoring systems

• The 1st prototype of the lpGBT was fabricated in 2018

• Currently about 130 ASICs have been distributed to the users for prototyping of their systems • ~400 chips still available with another 400 available soon • The performance of the prototype [in almost all the cases] meets the specifications allowing the users to integrate the ASIC in their system prototypes

• We are currently working to solve the few “non-idealities” remaining. Tapeout of the final prototype will take place in 2020

25 Schedule • To keep updated with the project schedule please refer to: • https://ep-ese.web.cern.ch/content/lpgbt-vl-dcdc-schedule

2019 2020 2021 2022 2023 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2

Opto die Contract 1st QA Delivery 1st lot 2nd QA Delivery 2nd lot

LDQ10v5 Eng. run Wafer Test GBTIA Order Wafer Test

~360 ~500 Contract Pre prod. Pre prod QA Series production 2000 per month End of production (60000 units) VTRx+ samples samples

~800 Samples ~40 000 Total qty 2nd MPW Eng. run Production lpGBT samples available available available