Lecture 1: Course Introduction + Review of Throughput HW Architecture Visual Computing Systems Stanford CS348K, Spring 2021 Hello from the course sta! Your instructor (me) Your CA Prof. Kayvon David Durst Stanford CS348K, Spring 2021 Visual computing applications have always demanded some of the world’s most advanced parallel computing systems Stanford CS348K, Spring 2021 Ivan Sutherland’s Sketchpad on MIT TX-2 (1962) Stanford CS348K, Spring 2021 The frame bu!er 16 2K shift registers (640 x 486 x 8 bits) Shoup’s SuperPaint (PARC 1972-73) Stanford CS348K, Spring 2021 The frame bu!er 16 2K shift registers (640 x 486 x 8 bits) Shoup’s SuperPaint (PARC 1972-73) Stanford CS348K, Spring 2021 Xerox Alto (1973) Bravo (WYSIWYG) TI 74181 ALU Stanford CS348K, Spring 2021 UNC Pixel Planes (1981), computation-enhanced frame bu!er Stanford CS348K, Spring 2021 Jim Clark’s Geometry Engine (1982) ASIC for geometric transforms used in real-time graphics Figure 2: Photograph of the Geometry Engine. Stanford CS348K, Spring 2021 NVIDIA Titan V Volta GPU (2017) ~ 12 TFLOPs fp32 Similar to ASCI Q (top US supercomputer circa 2002) Stanford CS348K, Spring 2021 Real-time rendering Screenshot: Red Dead Redemption Stanford CS348K, Spring 2021 Image analysis via deep learning Image Credit: Detectron2 Stanford CS348K, Spring 2021 Digital photography: major driver of compute capability of modern smartphones Portrait mode (simulate e!ects of large aperture DSLR lens) High dynamic range (HDR) photography Stanford CS348K, Spring 2021 Modern smartphones utilize multiple processing units to quickly generate high-quality images Apple A13 Bionic Image Credit: Anandtech / TechInsights Inc. Stanford CS348K, Spring 2021 Datacenter-scale applications Google TPU pods Image Credit: TechInsights Inc. Stanford CS348K, Spring 2021 Youtube Transcode, stream, analyze… Stanford CS348K, Spring 2021 On every vehicle: analyzing images for transportation Stanford CS348K, Spring 2021 Unique visual experiences Intel “FreeD”: 38 cameras in stadium used to reconstruct 3D, system renders new view from quarterback’s eyes Stanford CS348K, Spring 2021 What is this course about? Accelerator hardware architecture? Graphics/vision/digital photography algorithms? Programming systems? Stanford CS348K, Spring 2021 What we will be learning about Table 4. Results on NYUDv2. RGBD is early-fusion of the Table 5. Results on SIFT Flow9 with class segmentation RGB and depth channels at the input. HHA is the depth embed- (center) and geometric segmentation (right). Tighe [36] is ding of [15] as horizontal disparity, height above ground, and a non-parametric transfer method. Tighe 1 is an exemplar the angle of the local surface normal with the inferred gravity SVM while 2 is SVM + MRF. Farabet is a multi-scale con- direction. RGB-HHA is the jointly trained late fusion model vnet trained on class-balanced samples (1) or natural frequency that sums RGB and HHA predictions. samples (2). Pinheiro is a multi-scale, recurrent convnet, de- 3 noted RCNN3 ( ). The metric for geometry is pixel accuracy. Visualpixel mean Computingmean f.w. Workloads◦ acc. acc. IU IU Gupta et al.[Algorithms15] 60.3 - for 28.6 image/video 47.0 processing,pixel mean mean f.w. geom. FCN-32s RGB 60.0 42.2 29.2 43.9 acc. acc. IU IU acc. FCN-32s RGBDDNN61.5 evaluation, 42.4 30.5 45.5data compression,Liu et al.[25 etc.] 76.7 - - - - FCN-32s HHA 57.1 35.2 24.2 40.4 Tighe et al.[36] ----90.8 FCN-32s RGB-HHA 64.3 44.9 32.8 48.0 Tighe et al.[37]1 75.6 41.1 - - - Tighe et al.[37]2 78.6 39.2 - - - FCN-16s RGB-HHA 65.4 46.1 34.0 49.5 Farabet et al.[9]1 72.3 50.8 - - - NYUDv2 [33] is an RGB-D dataset collected using the Farabet et al.[9]2 78.5 29.6 - - - Microsoft Kinect. It has 1449 RGB-D images, with pixel- Pinheiro et al.[31] 77.7 29.8 - - - wise labels that have been coalesced into a 40 class seman- FCN-16s 85.2 51.7 39.5 76.1 94.3 tic segmentation task by Gupta et al.[14]. We report results If you don’t understand key on the standard split of 795 training images and 654 testing FCN-8s SDS [17] Ground Truth Image images. (Note: all model selection is performed on PAS- CAL 2011 val.) Table 4 gives the performance of our model workload characteristics, in several variations. First we train our unmodified coarse model (FCN-32s) on RGB images. To add depth informa- how can you design a “good” system? tion, we train on a model upgraded to take four-channel RGB-D input (early fusion). This provides little benefit, perhaps due to the difficultly of propagating meaningful gradients all the way through the model. Following the suc- cess of Gupta et al.[15], we try the three-dimensional HHA encoding of depth, training nets on just this information, as well as a “late fusion” of RGB and HHA where the predic- tions from both nets are summed at the final layer, and the resulting two-stream net is learned end-to-end. Finally we upgrade this late fusion net to a 16-stride version. SIFT Flow is a dataset of 2,688 images with pixel labels for 33 semantic categories (“bridge”, “mountain”, “sun”), as well as three geometric categories (“horizontal”, “verti- Figure 6. Fully convolutional segmentation nets produce state- cal”, and “sky”). An FCN can naturally learn a joint repre- of-the-art performance on PASCAL. The left column shows the sentation that simultaneously predicts both types of labels. output of our highest performing net, FCN-8s. The second shows Stanford CS348K, Spring 2021 We learn a two-headed version of FCN-16s with seman- the segmentations produced by the previous state-of-the-art system tic and geometric prediction layers and losses. The learned by Hariharan et al.[17]. Notice the fine structures recovered (first model performs as well on both tasks as two independently row), ability to separate closely interacting objects (second row), trained models, while learning and inference are essentially and robustness to occluders (third row). The fourth row shows a as fast as each independent model by itself. The results in failure case: the net sees lifejackets in a boat as people. Table 5, computed on the standard split into 2,488 training 6. Conclusion and 200 test images,9 show state-of-the-art performance on both tasks. Fully convolutional networks are a rich class of mod- els, of which modern classification convnets are a spe- cial case. Recognizing this, extending these classification nets to segmentation, and improving the architecture with 9Three of the SIFT Flow categories are not present in the test set. We multi-resolution layer combinations dramatically improves made predictions across all 33 categories, but only included categories ac- the state-of-the-art, while simultaneously simplifying and tually present in the test set in our evaluation. (An earlier version of this pa- per reported a lower mean IU, which included all categories either present speeding up learning and inference. or predicted in the evaluation.) Acknowledgements This work was supported in part What we will be learning about Modern Hardware Organization High-throughput hardware designs (parallel, heterogeneous, and specialized) fundamental constraints like area and power If you don’t understand key constraints of modern hardware, how can you design algorithms that are well suited to run on it e"ciently? Stanford CS348K, Spring 2021 What we will be learning about Programming Model Design Choice of programming abstractions, level of abstraction issues, domain-speci$c vs. general purpose, etc. Good programming abstractions enable productive development of applications, Halide while also providing system implementors #exibility to explore highly e"cient implementations Stanford CS348K, Spring 2021 This course is about architecting e"cient and scalable systems… It is about the process of understanding the fundamental structure of problems in the visual computing domain, and then leveraging that understanding to… To design more e"cient and more robust algorithms To build the most e"cient hardware to run these algorithms To design programming systems to make developing new applications simpler, more productive, and highly performant Stanford CS348K, Spring 2021 Course topics The digital camera photo processing pipeline in modern smartphones Basic algorithms (the workload) Programming abstractions for writing image processing apps Mapping these algorithms to parallel hardware Systems for creating fast and accurate deep learning models Designing e"cient DNN topologies, pruning, model search and AutoML Hardware for accelerating deep learning (why GPUs are not e"cient enough!) Algorithms for parallel training (async and sync) Raising level of abstraction when designing models System support for automating data labeling Processing and Transmitting Video E"cient DNN inference on video Trends in video compression (particularly relevant these days) How video conferencing systems work Recent advances in real-time (hardware accelerated) ray tracing Ray tracing workload Recent API and hardware support for real-time ray tracing How deep learning, combined with RT hardware, is making real time ray tracing possible Stanford CS348K, Spring 2021 Course Logistics Stanford CS348K, Spring 2021 Logistics of a all-virtual class ▪ Course web site: - http://cs348k.stanford.edu - My goal is to post lecture slides the night before class ▪ All announcements will go out via Piazza (not via Canvas) Stanford CS348K, Spring 2021 Expectations of you ▪ 40% participation - There will be ~1 assigned paper reading per class - You will submit a response to each reading by noon on class days - We will start