Rapidio® the Embedded System Interconnect
Total Page：16
File Type：pdf, Size：1020Kb
RapidIO® The Embedded System Interconnect Sam Fuller RapidIO Trade Association, USA with contributions from Alan Gatherer CTO, Wireless Infrastructure, Texas, USA Charles Hill Motorola, Inc., Arizona, USA Victor Menasce Applied Micro Circuits Corporation, Ontario, Canada Brett Niver EMC Corporation, Hopkinton, USA Richard O’Connor Tundra Semiconductor Corporation, Ontario, Canada Peter Olanders Ericsson AB, Stockholm, Sweden Nupur Shah Xilinx Inc., California, USA David Wickliff Lucent Technologies, Illinois, USA RapidIO® The Embedded System Interconnect RapidIO® The Embedded System Interconnect Sam Fuller RapidIO Trade Association, USA with contributions from Alan Gatherer CTO, Wireless Infrastructure, Texas, USA Charles Hill Motorola, Inc., Arizona, USA Victor Menasce Applied Micro Circuits Corporation, Ontario, Canada Brett Niver EMC Corporation, Hopkinton, USA Richard O’Connor Tundra Semiconductor Corporation, Ontario, Canada Peter Olanders Ericsson AB, Stockholm, Sweden Nupur Shah Xilinx Inc., California, USA David Wickliff Lucent Technologies, Illinois, USA Copyright © 2005 John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England Telephone (+44) 1243 779777 Email (for orders and customer service enquiries): [email protected] Visit our Home Page on www.wileyeurope.com or www.wiley.com All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except under the terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP, UK, without the permission in writing of the Publisher. Requests to the Publisher should be addressed to the Permissions Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England, or emailed to [email protected], or faxed to (+44) 1243 770620. Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The Publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Other Wiley Editorial Offices John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA Jossey-Bass, 989 Market Street, San Francisco, CA 94103-1741, USA Wiley-VCH Verlag GmbH, Boschstr. 12, D-69469 Weinheim, Germany John Wiley & Sons Australia Ltd, 33 Park Road, Milton, Queensland 4064, Australia John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop #02-01, Jin Xing Distripark, Singapore 129809 John Wiley & Sons Canada Ltd, 22 Worcester Road, Etobicoke, Ontario, Canada M9W 1L1 Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN 0-470-09291-2 Typeset in 10/12pt Times by Graphicraft Limited, Hong Kong Printed and bound in Great Britain by TJ International, Padstow, Cornwall This book is printed on acid-free paper responsibly manufactured from sustainable forestry in which at least two trees are planted for each one used for paper production. Contents Preface xix 1 The Interconnect Problem 1 1.1 Processor Performance and Bandwidth Growth 1 1.2 Multiprocessing 2 1.3 System of Systems 3 1.4 Problems with Traditional Buses 4 1.4.1 Bus Loading 6 1.4.2 Signal Skew 6 1.4.3 Expense of Wider Buses 6 1.4.4 Problems with PCI 7 1.5 The Market Problem 7 1.6 RapidIO: A New Approach 8 1.6.1 Why RapidIO? 8 1.7 Where Will it be Used? 9 1.8 An Analogy 11 References 12 2 RapidIO Technology 13 2.1 Philosophy 13 2.2 The Specification Hierarchy 14 2.3 RapidIO Protocol Overview 15 2.3.1 Packets and Control Symbols 15 2.4 Packet Format 16 2.5 Transaction Formats and Types 17 2.6 Message Passing 17 2.7 Globally Shared Memory 18 2.8 Future Extensions 18 2.9 Flow Control 18 2.9.1 Link Level Flow Control 18 2.9.2 End-to-end Flow Control 19 vi CONTENTS 2.10 The Parallel Physical Layer 20 2.10.1 Parallel Electrical Interface 21 2.11 The Serial Physical Layer 21 2.11.1 PCS and PMA Layers 21 2.11.2 Electrical Interface 22 2.12 Link Protocol 22 2.13 Maintenance and Error Management 23 2.13.1 Maintenance 23 2.13.2 System Discovery 23 2.13.3 Error Coverage 23 2.13.4 Error Recovery 24 2.14 Performance 24 2.14.1 Packet Structures 24 2.14.2 Source Routing and Concurrency 25 2.14.3 Packet Overhead 25 2.15 Operation Latency 25 References 26 3 Devices, Switches, Transactions and Operations 27 3.1 Processing Element Models 27 3.1.1 Integrated Processor-memory Processing Element Model 28 3.1.2 Memory-only Processing Element Model 29 3.2 I/O Processing Element 29 3.3 Switch Processing Element 30 3.4 Operations and Transactions 30 3.4.1 Operation Ordering 30 3.4.2 Transaction Delivery 33 3.4.3 Ordered Delivery System Issues 33 3.4.4 Deadlock Considerations 33 4 I/O Logical Operations 35 4.1 Introduction 35 4.2 Request Class Transactions 35 4.2.1 Field Definitions for Request Class Transactions 36 4.3 Response Class Transactions 37 4.3.1 Field Definitions for Response Packet Formats 37 4.4 A Sample Read Operation 39 4.5 Write Operations 42 4.6 Streaming Writes 44 4.7 Atomic Operations 45 4.8 Maintenance Operations 45 4.9 Data Alignment 47 5 Messaging Operations 49 5.1 Introduction 49 5.2 Message Transactions 50 CONTENTS vii 5.2.1 Type 10 Packet Format (Doorbell Class) 50 5.2.2 Type 11 Packet Format (Message Class) 51 5.2.3 Response Transactions 54 5.3 Mailbox Structures 54 5.3.1 A Simple Inbox 55 5.3.2 An Extended Inbox 56 5.3.3 Receiving Messages 57 5.4 Outbound Mailbox Structures 57 5.4.1 A Simple Outbox 57 5.4.2 An Extended Outbox 58 5.4.3 Transmitting Messages 58 6 System Level Addressing in RapidIO Systems 61 6.1 System Topology 61 6.2 Switch-based Systems 62 6.3 System Packet Routing 64 6.4 Field Alignment and Definition 64 6.5 Routing Maintenance Packets 65 7 The Serial Physical Layer 67 7.1 Packets 68 7.1.1 Packet Format 68 7.1.2 Packet Protection 69 7.1.2.1 Packet CRC Operation 69 7.1.2.2 16-Bit Packet CRC Code 69 7.2 Control Symbols 70 7.2.1 Stype0 Control Symbol Definitions 71 7.2.1.1 Packet-accepted Control Symbol 72 7.2.1.2 Packet-retry Control Symbol 72 7.2.1.3 Packet-not-accepted Control Symbol 73 7.2.1.4 Status Control Symbol 73 7.2.1.5 Link-response Control Symbol 73 7.2.2 Stype1 Control Symbol Definitions 74 7.2.2.1 Start-of-packet Control Symbol 75 7.2.2.2 Stomp Control Symbol 75 7.2.2.3 End-of-packet Control Symbol 75 7.2.2.4 Restart-from-retry Control Symbol 75 7.2.2.5 Link-request Control Symbol 76 7.2.2.6 Multicast-event Control Symbol 76 7.2.3 Control Symbol Protection 77 7.2.3.1 CRC-5 Code 77 7.3 PCS and PMA Layers 77 7.3.1 PCS Layer Functions 78 7.3.2 PMA Layer Functions 79 7.3.3 Definitions 79 7.3.4 The 8B/10B Transmission Code 80 viii CONTENTS 7.3.5 Character and Code Group Notation 80 7.3.6 Running Disparity 81 7.3.6.1 Running Disparity Rules 81 7.3.7 8B/10B Encoding 82 7.3.8 Transmission Order 82 7.3.9 8B/10B Decoding 82 7.3.10 Special Code Groups and Characters 84 7.3.10.1 Packet Delimiter Control Symbol (/PD/) 84 7.3.10.2 Start of Control Symbol (/SC/) 84 7.3.10.3 Idle (/I/) 84 7.3.10.4 Sync (/K/) 84 7.3.10.5 Skip (/R/) 84 7.3.10.6 Align (/A/) 84 7.4 Using the Serial Physical Layer 84 7.4.1 Port Initialization Process 85 7.4.2 Packet Exchange Protocol 85 7.4.2.1 Control Symbols 85 7.4.2.2 Packets 86 7.4.3 Idle Sequence 87 7.4.4 Data Flow Across a 1x Serial RapidIO Link 87 7.4.5 Data Flow Across a 4x Serial RapidIO Link 87 7.4.6 Flow Control 89 7.4.6.1 Receiver-controlled Flow Control 89 7.4.6.2 Transmitter-controlled Flow Control 92 7.4.6.3 Input Retry-stopped Recovery Process 93 7.4.6.4 Output Retry-stopped Recovery Process 93 7.4.7 Link Maintenance Protocol 94 7.4.8 Canceling Packets 94 7.5 Transaction and Packet Delivery Ordering Rules 95 7.5.1 Deadlock Avoidance 96 7.6 Error Detection and Recovery 98 7.6.1 Lost Packet Detection 98 7.6.2 Link Behavior Under Error 98 7.6.3 Effect of Single-bit Code Group Errors 99 7.6.4 Recoverable Errors 100 7.6.4.1 Input Error-stopped Recovery Process 100 7.6.4.2 Output Error-stopped Recovery Process 100 7.6.4.3 Idle Sequence Errors 101 7.6.4.4 Control Symbol Errors 101 7.6.4.5 Reserved stype and cmd Field Encodings 101 7.6.4.6 Link Protocol Violations 101 7.6.4.7 Corrupted Control Symbols 102 7.6.4.8 Packet Errors 102 7.6.4.9 Link Time-out 102 7.7 Retimers and Repeaters 103 7.8 The Electrical Interface 103 CONTENTS ix 8 Parallel Physical Layer Protocol 107 8.1 Packet Formats 108 8.2 Control Symbol Formats 109 8.2.1 Acknowledgment Control Symbol Formats 110 8.2.1.1 Packet-accepted Control Symbol 110 8.2.1.2 Packet-retry Control Symbol 110 8.2.1.3 Packet-not-accepted Control Symbol 110 8.2.2 Packet Control Symbols 111 8.2.3 Link Maintenance Control Symbol Formats 112 8.2.3.1 Link Request Control Symbol 112 8.2.3.2 Link Response Control Symbol 113 8.3 Control Symbol Transmission Alignment 113 8.4 Packet Start and Control Symbol Delineation