Everything You Wanted To Know about Storage (But Were Too Proud To Ask) Part Mauve
J Metz, Cisco Dror Goldenberg, Mellanox Chad Hintz, Cisco Fred Knight, NetApp November 1, 2016 Today’s Presenters
J Metz Dror Goldenberg Chad Hintz Fred Knight SNIA Board VP Software SNIA-ESF Standards of Directors Architecture Board Technologist Cisco Mellanox Cisco NetApp
2 SNIA Legal Notice
The material contained in this presentation is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations and literature under the following conditions:
Any slide or slides used must be reproduced in their entirety without modification The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations. This presentation is a project of the SNIA. Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information.
NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.
3 About SNIA
4 Agenda
Channel vs. Bus Host Network Storage Control Plane vs. Data Plane
Fabric vs. Network Channel Busses Control Plane Data Plane
Fabric Network
5 Off We Go!
Channel vs. Bus Control Plane vs. Data Plane Fabric vs. Network
6 Bus vs. Channel
Bus Channel Transfers multiple signals Physical or logical data simultaneously communication line
7 Get On The Bus
Communication system that transfers multiple signals simultaneously between components
Control bus – send commands
Data bus – move data
Address bus – tell me where to find it
Power bus – shock it to me! The bus can be…
Shared or point to point
Parallel or serial
Internal or external 8 Busses in a Server
Memory Bus Cache Coherency Memory Bus DDR4 Bus DDR4
Peripheral Bus PCIe Storage Bus SATA/SAS PCI Chipset Switch Peripheral Bus SMBus/ PCIe i2c
NVMe 9 Get On The Bus
Common bus examples
Motherboard: PCIe (devices), DDR4 (memory), AGP (graphics)
Storage: SCSI, SATA, SAS 2 Peripherals: USB, Lightning, FireWire, Thunderbolt, i c/smbus What is not a bus?
Ethernet, Fibre Channel, InfiniBand
These are networks & fabrics
10 Faster Bus Performance
Faster data transfer on each lane
Higher frequencies (faster signal transfer rate)
More data per message (efficient encoding) More lanes in parallel
11 Channel Details
Path for transferring data
Physical—Wire, fiber or board trace
Logical—Digital TV channel; VPN link
Mainframe channel subsystem (CSS)
SSD/DRAM data transfer path Physical, electronic, or logical separation
Different wires/fibers
Different frequencies, multiplexing
12 More About Channels
Channel vs. bus?
Bus can be comprised of multiple physical channels (a.k.a. lanes)
Logical channels can operate over a bus or network Channel vs. lane?
Can be synonymous, i.e. “Channel” = “Lane”
Or one channel can have multiple lanes
13 Bus vs. Network
Bus Network
Inside box Outside the box
Short distances Longer distances
Shared / point-to-point links Usually point-to-point links
Not typically shared by multiple Shared by multiple systems systems
Byte/LBA addressing typically L2/3 addressing typically Both Bus and Network can use multiple channels! 14 Network – IB/Eth/FC
Processor Processor Node Node Network Consoles Switch Processor Switch Node
Switch Switch
Storage RAID Subsystem 15 Bus v Channel - Summary
A bus is a topology of interconnected devices A channel is a dedicated path for traffic that can sit upon a topology
16 Next Up…
Channel vs. Bus Control Plane vs. Data Plane Fabric vs. Network
17 Control Plane vs. Data Plane
Control Plane Data Plane The Plan The Movement
18 Network Centric
19 Where are we going? Control Plane- Network Centric
Networking (SAN or Data Networks) devices use the control plane to:
Learn which paths are available
Using static configuration (static routes-data networks) or via learning (routing protocols SAN or Data)
Figure out the best path(s) and tell the data plane
Any packet the device has to fully process (routing
protocols, bpdu, SSH/Telnet) 20 I know where to GO, Jump on! Data Plane
Networking devices use the data plane to:
Transmit or Move the traffic from one interface to another based on what we have learned from the Control Plane
21 Storage Centric
22 Storage Control/Data Plane
23 Traditional Storage
Control Plane and Data Plane together in one unit
24 Where are we going? Control Plane- Storage Centric
25 Hyper-Converged Centric
26 Hyper-convergence-Control Plane Centralized
Controller VM VM VM VM Centralized Control Plane Hypervisor Tight integration of x86 Local Physical Storage servers for compute and Controller VM VM VM storage, networking and VM virtualization in all-in-one Hypervisor appliance. Local Physical Storage
Controller VM VM VM VM
Hypervisor Local Physical Storage Data v. Control Planes - Summary
A Data Plane is the path by which data is moved from device to device A Control Plane is the path by which the management of that data plane is transmitted
28 Last But Not Least…
Channel vs. Bus Control Plane vs. Data Plane Fabric vs. Network
29 What is a Network
Definition
SNIA: An interconnect that enables communication among a collection of attached nodes, consisting of optical, electrical, or wireless transmission media, infrastructure in the form of hubs, switches, and/or routers, and protocols that make message sequences meaningful.
The Internet is the most famous Network.
30 Typical Network
31 Typical Network
LAN WAN
32 What is a Fabric
Definition:
A Fabric is specialized type of Network.
Wikipedia: Switched Fabric or switching fabric is a network topology in which network nodes interconnect via one or more network switches (including crossbar switches). Because a switched fabric network spreads network traffic across multiple physical links, it yields higher total throughput than broadcast networks, such as early Ethernet.
33 Typical Computer Fabric
It’s a fabric, what kind of fabric?
34 Typical Computer Fabric
This one is a tight, mesh fabric
35 Typical SAN Fabric
36 Wire vs. Protocol
Network
Ethernet (copper, optical, wireless) – IEEE 802.x family
Dialup / DSL / Broadband “wires” – ITU + others
TCP, IP, + lots more protocol - IETF Fabric
Fibre Channel wire (copper, optical)
InfiniBand wire (copper, optical)
Fibre Channel protocol – INCITS T11 family
InfiniBand protocol – IBTA 37 Wire vs. Protocol
Network / Fabric Protocols Media Governing Type Body
Ethernet TCP / IP (1) Copper, Optical, IEEE 802.x (2) Wireless IETF
Dialup / DSL / TCP / IP (1) Copper, Optical ITU + others FIOS / Broadband
Fibre Channel FC (FC-SW, FC-GS, etc) Copper, Optical INCITS T11 (2)
InfiniBand IB Copper, Optical IBTA (2)
(1) POP, HTTP, iSCSI, DHCP, DNS, etc (2) Optical Internetworking Forum (OIF) defines optical cable standards
38 Connected Fabrics
Network
Fabric Domain 1 Fabric Domain 2
39 Key thoughts
Generally speaking:
Networks are thought of as larger entities (WWW)
Fabrics are thought of as smaller more specialized entities LAN = Local Area Network (office/campus) WAN = Wide Area Network (internet) SAN = Storage Area Network (FC)
Specialized for storage iSAN = Storage Area Network (iSCSI)
40 Ethernet / FC differences
Generally speaking:
Ethernet: More loosely coupled - network switches have less knowledge Lots of end points that talk to each other Next hop routing (IS-IS, OSPF, BGP, EIGRP) Destination based flow control (backpressure) Higher layer corrects for packet loss
Fibre Channel: More tightly coupled - fabric switches have more knowledge Initiators talk to targets Distributed routing knowledge (FSPF) Source based flow control (credits) Virtually lossless 41 Fabric v. Network - Summary
Networks make up a series of interconnected devices and can consist of multiple topologies Fabrics are a type of network with distributed intelligence across all devices
42 Summary
Channel vs. Bus Host Network Storage Control Plane vs. Data Plane
Fabric vs. Network Channel Busses Control Plane Data Plane
Fabric Network
43 Summary
Channel vs. Bus A bus is a topology of interconnected devices A channel is a dedicated path for traffic that can sit upon a topology Control Plane vs. Data Plane A Data Plane is the path by which data is moved from device to device
A Control Plane is the path by which the management of that data plane is transmitted Fabric vs. Network Networks make up a series of interconnected devices and can consist of multiple topologies Fabrics are a type of network with distributed intelligence across all devices
44 Other Storage Terms Got Your Pride? This is a Series!
Check out – Chartreuse https://www.brighttalk.com/webcast/663/215131 The Basics: Initiator, Target, Storage Controller, RAID. Volume Manager and more Future Topics/Colors (in no particular order): Vermillion (What-If-Programming-and-Networking-Had-A-Baby Pod) Coherence/Cache Coherence, Storage APIs, Block, File, Object, Byte Addressable, Logical Block Addressing Teal (Buffering Pod) Ring Buffers, Queues/Queue Depth/Buffering/Flow Control Rosé (iSCSI Pod) iSCSI offload, TCP offload, Host-based iSCSI Turquoise (Where-Does-My-Data-Go Pod) Volatile v. Non-Volatile v Persistent Memory, NVDIMM v. RAM v. DRAM v. SLC v. MLC v. TLC v. NAND v. 3D NAND v. Flash v SSDs v. NVMe, NVMe (the protocol) More… Follow us @SNIAESF so you don’t miss any 45 After This Webcast
Please rate this Webcast and provide us with feedback This Webcast and a PDF of the slides will be posted to the SNIA Ethernet Storage Forum (ESF) website and available on-demand
www.snia.org/forums/esf/knowledge/webcasts A full Q&A from this webcast, including answers to questions we couldn't get to today, will be posted to the SNIA-ESF blog:
sniaesfblog.org Follow us on Twitter @SNIAESF Need help with all these terms? Download the 2016 SNIA Dictionary
http://www.snia.org/education/dictionary
46 Thank You
47