Introduction to Multi-Core Architecture (Textbook - Chapter 1)

Total Page:16

File Type:pdf, Size:1020Kb

Introduction to Multi-Core Architecture (Textbook - Chapter 1) The University of Texas at Arlington Lecture 3 CSE 5343/4342 Embedded Systems II Based heavily on slides by Dr. Roger Walker Introduction to Multi-Core Architecture (Textbook - Chapter 1) 2 1 Concurrency in Software • Most naïve users think of computer behavior as simple (e.g., just browsing the network) • However and meanwhile, the computer OS has many-many processes running, many of them multi-threaded (user interface in foreground processing in the back). • Concurrency allows multiple processes or threads (codes representing different subsystems ) to be in execution state 3 Parallel Computing • Allows simultaneous execution of threads or processes – not same thing as concurrent execution. • Concurrency once it is performed simultaneously is parallel computing. • Computer Architectures can be classified in two different dimensions, the number of instruction streams that can be processed at any given time, and the number of data streams that can be processed at any given time. 4 2 Flynn’s Taxonomy • Single Instruction Single Data (SISD) • Multiple Instruction Single Data (MISD) – not practical • Single Instruction Multiple Data (SIMD) • Multiple Instruction Multiple Data (MIMD) – most modern architectures 5 Moore’s Law The number of transistors on integrated circuits is doubling every (18) 24 months. 6 3 Parallel Computing in Microprocessors • Many people mistakenly think the law is a predictor of clock speed as it used to follow Moore’s law (0.1MHz to 3GHz). • Parallelism can more easily (than increasing clock rates) use more transistors: – Instruction Level Parallelization (ILP) – dynamic, out of order processing at the hardware level avoiding pipeline stalls – Superscalar processors can use non-overlapping components of the CPU to do ILP (multiple pipelines) • Multiple processes or threads are running on a processor today (at the software level): – concurrent thread processing (preemptive) – simultaneous thread processing (multiple processors, HT) which can easily make use of hardware parallelism or ILP 7 Threads • A Thread is a discrete sequence of related instructions that is executed independently of other instruction sequences. • Hardware Level Definition: A thread is an execution path that remains independent of other hardware execution paths. • OS maps software threads to hardware execution • A thread only needs the architecture space – registers, execution units, etc. • Logical processors can be created by duplicating the architecture space. 8 4 What Are Threads Good For? • Improving responsiveness of GUIs • Making programs easier to understand • Overlapping computation and I/O • Improving performance through parallel execution – if computer architecture allows 9 Thread Concurrency vs. Parallelism Concurrency: two or more threads are in progress at the same time: Thread 1 Thread 2 Parallelism: two or more threads are executing at the same time Thread 1 Thread 2 Multiple cores needed 10 5 Thread Level Parallelism • Time-sliced multi-threading – single processor • Simultaneous multithreading: with superscalar support (see next slide) • Multiple processors – multiple threads or processes run simultaneously on multiple processors • Physical processor – includes many resources including architecture state (registers, caches, execution units, etc.) 11 Simultaneous Multi-Threading • Simultaneous multi-threading or SMT - The actual execution units shared by the different logical processors. • Intel’s implementation called Hyper- threading or HT • To the OS (e.g., Linux or Windows) the computing unit appears as multiple physical processors and threads scheduled accordingly. 12 6 Intel’s HT Technology Implementation • Two logical processors: – Architecture state and On-DieOn-Die APIC1 duplicated CacheCache • Shared execution units: cache, branch ArchitectureArchitecture StateState ArchitectureArchitecture StateState predictors, control logic Adv.Adv. ProgrammableProgrammable Adv.Adv. ProgrammableProgrammable and buses InterruptInterrupt ControlControl InterruptInterrupt ControlControl ProcessorProcessor 1 APIC: Advanced Programmable ExecutionExecution Interrupt Controller; handles interrupts Resource sent to a specified logical processor. Resource System Bus 13 CPU Architectures From Text-book 14 7 HT, Single Processor, Multiple Processor Single Processor Thread 1 Idle Dual Processors Thread 2 HT Processor 15 Understanding Performance • Parallel processing speedup: Speed Up = Time_best_sequential / Time_parallel_implementation • Amdahl’s Law on speeding up a fraction of execution: Total Speed Up = 1 / [1-Fraction_Enhanced + (Fraction_Enchanced/Speedup )] 16 8 Amdahl Example • Amdahl’s Law on speeding up a fraction of execution: Total Speed Up = 1 / [1-Fraction_Enhanced + (Fraction_Enchanced/Speedup)] • Examples: Speedup half the program by 15% using parallel processing, then Total Speedup = 1/[(1-0.5)+(.5/1.15)] = 1.08 Thus the whole program speedup is 8 percent. 17 Amdahl to Parallel • Expressing in terms of the serial and parallel portions: Speedup = 1/[S + (1-S)/n] Where S is the time spent executing the serial portion of program and n is the number of execution cores • If n = 1, then there is no speedup • As n increases, the speedup approaches 1/S (e.g., if S is 10% then maximum speedup is 10 fold) 18 9 Parallel Code vs. Parallel Processors • For 2 cores and a 30% parallelized program: • Speedup with 2 cores: 1/(.7 + .3/2) ~ 85% of execution time • Speedup with 4 cores: 1/(.7+.3/4) ~ 77.5% of execution time • Doubling the parallel portion with two cores: 1/(0.4+ .6/2) ~ 70% of execution time • Thus only when the program is highly parallelized does adding more processors really help • (Calculations assume that the parallel portion is arbitrarily parallel) 19 Amdahl’s Law with Multi-core Threading Overhead • H(n) – actual OS system overhead + inter-thread activities (synchronization and other forms of communication). • Speedup = 1/(S+(1-S)/n + H(n)) – can be less than 1 if H(n) high • Keep H(n) due to threading small 20 10 Amdahl’s Law with HT • In HT, although multiple threads are running simultaneously each one of them runs slower. If each thread runs approximately k percent slower then: – Speedup=1/[S+(1-k)(1-S)/n+H(n)] • E.g., if threads run 30% slower then: – Speedup=1/[S+0.67(1-S)/n+H(n)] 21 Amdahl’s Law – Assumptions Made • Best performing serial algorithm is limited by available of CPU cycles (no I/O). However, a multi-core processor with separate caches, will reduce memory latency. • Serial algorithm is best possible solution. However, some problems may work better (larger speed up) with parallel algorithms. • As number of core increases, the problem stays the same. However, with more resources, problem grows to meet available resources 22 11 Gustafson’s Law • Gustafson’s achieved near linear speedup using 1,024 hypercube at Sandia in late 1980’s. • Gustafson’s Law: – Scaled speedup = N - (N-1)*s s is the non-parallelizable part of the process Addresses last assumption and fixed problem size in Amdahl’s law by assuming fixed time concept which leads to scaled speed up. 23 Summary • Concurrency refers to the notion of multiple threads in progress at the same time. • Parallelism refers to the concept of multiple threads executing simultaneously. • Modern software applications often consist of multiple processes or threads that can be executed in parallel. • Most modern computing platforms are multiple instruction, multiple data (MIMD) machines. These machines allow programmers to process multiple instruction and data streams simultaneously. 24 12 Summary • In practice, Amdahl's Law does not accurately reflect the benefit of increasing the number of processor cores on a given platform. • Linear speedup is achievable by expanding the problem size with the number of processor cores. 25 13.
Recommended publications
  • User Interface Design Issues in HCI
    IJCSNS International Journal of Computer Science and Network Security, VOL.18 No.8, August 2018 153 User Interface Design Issues In HCI Waleed Iftikhar, Muhammad Sheraz Arshad Malik, Shanza Tariq, Maha Anwar, Jawad Ahmad, M. Saad sultan Department of Computing, Riphah International University, Faisalabad, Pakistan Summary Command line is the interface that allows the user to This paper presents an important analysis on a literature review interact with the computer by directly using the commands. which has the findings in design issues from the year 1999 to But there is an issue that the commands cannot be changed, 2018. This study basically discusses about all the issues related they are fixed and computer only understands the exact to design and user interface, and also gives the solutions to make commands. the designs or user interface more attractive and understandable. This study is the guideline to solve the main issues of user Graphical user interface is the interface that allows the interface. user to interact with the system, because this is user There is important to secure the system for modern applications. friendly and easy to use. This includes the graphics, The use of internet is quickly growing from years. Because of pictures and also attractive for all type of users. The this fast travelling lifestyle, where they lets the user to attach with command line is black and white interface. This interface systems from everywhere. When user is ignoring the is also known as WIMPS because it uses windows, icons, functionalities in the system then the system is not secure but, in menus, pointers.
    [Show full text]
  • Designing for Increased Autonomy & Human Control
    IFIP Workshop on Intelligent Vehicle Dependability and Security (IVDS) January 29, 2021 Designing for Increased Autonomy & Human Control Ben Shneiderman @benbendc Founding Director (1983-2000), Human-Computer Interaction Lab Professor, Department of Computer Science Member, National Academy of Engineering Photo: BK Adams IFIP Workshop on Intelligent Vehicle Dependability and Security (IVDS) January 29, 2021 Designing for Increased Automation & Human Control Ben Shneiderman @benbendc Founding Director (1983-2000), Human-Computer Interaction Lab Professor, Department of Computer Science Member, National Academy of Engineering Photo: BK Adams What is Human-Centered AI? Human-Centered AI Amplify, Augment, Enhance & Empower People Human Responsibility Supertools and Active Appliances Visual Interfaces to Prevent/Reduce Explanations Audit Trails to Analyze Failures & Near Misses Independent Oversight à Reliable, Safe & Trustworthy Supertools Digital Camera Controls Navigation Choices Texting Autocompletion Spelling correction Active Appliances Coffee maker, Rice cooker, Blender Dishwasher, Clothes Washer/Dryer Implanted Cardiac Pacemakers NASA Mars Rovers are Tele-Operated DaVinci Tele-Operated Surgery “Robots don’t perform surgery. Your surgeon performs surgery with da Vinci by using instruments that he or she guides via a console.” https://www.davincisurgery.com/ Bloomberg Terminal A 2-D HCAI Framework Designing the User Interface Balancing automation & human control First Edition: 1986 Designing the User Interface Balancing automation &
    [Show full text]
  • 2.5 Classification of Parallel Computers
    52 // Architectures 2.5 Classification of Parallel Computers 2.5 Classification of Parallel Computers 2.5.1 Granularity In parallel computing, granularity means the amount of computation in relation to communication or synchronisation Periods of computation are typically separated from periods of communication by synchronization events. • fine level (same operations with different data) ◦ vector processors ◦ instruction level parallelism ◦ fine-grain parallelism: – Relatively small amounts of computational work are done between communication events – Low computation to communication ratio – Facilitates load balancing 53 // Architectures 2.5 Classification of Parallel Computers – Implies high communication overhead and less opportunity for per- formance enhancement – If granularity is too fine it is possible that the overhead required for communications and synchronization between tasks takes longer than the computation. • operation level (different operations simultaneously) • problem level (independent subtasks) ◦ coarse-grain parallelism: – Relatively large amounts of computational work are done between communication/synchronization events – High computation to communication ratio – Implies more opportunity for performance increase – Harder to load balance efficiently 54 // Architectures 2.5 Classification of Parallel Computers 2.5.2 Hardware: Pipelining (was used in supercomputers, e.g. Cray-1) In N elements in pipeline and for 8 element L clock cycles =) for calculation it would take L + N cycles; without pipeline L ∗ N cycles Example of good code for pipelineing: §doi =1 ,k ¤ z ( i ) =x ( i ) +y ( i ) end do ¦ 55 // Architectures 2.5 Classification of Parallel Computers Vector processors, fast vector operations (operations on arrays). Previous example good also for vector processor (vector addition) , but, e.g. recursion – hard to optimise for vector processors Example: IntelMMX – simple vector processor.
    [Show full text]
  • Computer Hardware Architecture Lecture 4
    Computer Hardware Architecture Lecture 4 Manfred Liebmann Technische Universit¨atM¨unchen Chair of Optimal Control Center for Mathematical Sciences, M17 [email protected] November 10, 2015 Manfred Liebmann November 10, 2015 Reading List • Pacheco - An Introduction to Parallel Programming (Chapter 1 - 2) { Introduction to computer hardware architecture from the parallel programming angle • Hennessy-Patterson - Computer Architecture - A Quantitative Approach { Reference book for computer hardware architecture All books are available on the Moodle platform! Computer Hardware Architecture 1 Manfred Liebmann November 10, 2015 UMA Architecture Figure 1: A uniform memory access (UMA) multicore system Access times to main memory is the same for all cores in the system! Computer Hardware Architecture 2 Manfred Liebmann November 10, 2015 NUMA Architecture Figure 2: A nonuniform memory access (UMA) multicore system Access times to main memory differs form core to core depending on the proximity of the main memory. This architecture is often used in dual and quad socket servers, due to improved memory bandwidth. Computer Hardware Architecture 3 Manfred Liebmann November 10, 2015 Cache Coherence Figure 3: A shared memory system with two cores and two caches What happens if the same data element z1 is manipulated in two different caches? The hardware enforces cache coherence, i.e. consistency between the caches. Expensive! Computer Hardware Architecture 4 Manfred Liebmann November 10, 2015 False Sharing The cache coherence protocol works on the granularity of a cache line. If two threads manipulate different element within a single cache line, the cache coherency protocol is activated to ensure consistency, even if every thread is only manipulating its own data.
    [Show full text]
  • Exploring the User Interface Affordances of File Sharing
    CHI 2006 Proceedings • Activity: Design Implications April 22-27, 2006 • Montréal, Québec, Canada Share and Share Alike: Exploring the User Interface Affordances of File Sharing Stephen Voida1, W. Keith Edwards1, Mark W. Newman2, Rebecca E. Grinter1, Nicolas Ducheneaut2 1GVU Center, College of Computing 2Palo Alto Research Center Georgia Institute of Technology 3333 Coyote Hill Road 85 5th Street NW, Atlanta, GA 30332–0760, USA Palo Alto, CA 94304, USA {svoida, keith, beki}@cc.gatech.edu {mnewman, nicolas}@parc.com ABSTRACT and folders shared with other users on a single computer, With the rapid growth of personal computer networks and often as a default system behavior; files shared with other the Internet, sharing files has become a central activity in computers over an intranet or home network; and files computer use. The ways in which users control the what, shared with other users around the world on web sites and how, and with whom of sharing are dictated by the tools FTP servers. Users also commonly exchange copies of they use for sharing; there are a wide range of sharing documents as email attachments, transfer files during practices, and hence a wide range of tools to support these instant messaging sessions, post digital photos to online practices. In practice, users’ requirements for certain photo album services, and swap music files using peer–to– sharing features may dictate their choice of tool, even peer file sharing applications. though the other affordances available through that tool Despite these numerous venues for and implementations of may not be an ideal match to the desired manner of sharing.
    [Show full text]
  • Massively Parallel Computing with CUDA
    Massively Parallel Computing with CUDA Antonino Tumeo Politecnico di Milano 1 GPUs have evolved to the point where many real world applications are easily implemented on them and run significantly faster than on multi-core systems. Future computing architectures will be hybrid systems with parallel-core GPUs working in tandem with multi-core CPUs. Jack Dongarra Professor, University of Tennessee; Author of “Linpack” Why Use the GPU? • The GPU has evolved into a very flexible and powerful processor: • It’s programmable using high-level languages • It supports 32-bit and 64-bit floating point IEEE-754 precision • It offers lots of GFLOPS: • GPU in every PC and workstation What is behind such an Evolution? • The GPU is specialized for compute-intensive, highly parallel computation (exactly what graphics rendering is about) • So, more transistors can be devoted to data processing rather than data caching and flow control ALU ALU Control ALU ALU Cache DRAM DRAM CPU GPU • The fast-growing video game industry exerts strong economic pressure that forces constant innovation GPUs • Each NVIDIA GPU has 240 parallel cores NVIDIA GPU • Within each core 1.4 Billion Transistors • Floating point unit • Logic unit (add, sub, mul, madd) • Move, compare unit • Branch unit • Cores managed by thread manager • Thread manager can spawn and manage 12,000+ threads per core 1 Teraflop of processing power • Zero overhead thread switching Heterogeneous Computing Domains Graphics Massive Data GPU Parallelism (Parallel Computing) Instruction CPU Level (Sequential
    [Show full text]
  • User Interface for Volume Rendering in Virtual Reality Environments
    User Interface for Volume Rendering in Virtual Reality Environments Jonathan Klein∗ Dennis Reuling† Jan Grimm‡ Andreas Pfau§ Damien Lefloch¶ Martin Lambersk Andreas Kolb∗∗ Computer Graphics Group University of Siegen ABSTRACT the gradient or the curvature at the voxel location into account and Volume Rendering applications require sophisticated user interac- require even more complex user interfaces. tion for the definition and refinement of transfer functions. Tradi- Since Virtual Environments are especially well suited to explore tional 2D desktop user interface elements have been developed to spatial properties of complex 3D data, bringing Volume Render- solve this task, but such concepts do not map well to the interaction ing applications into such environments is a natural step. However, devices available in Virtual Reality environments. defining new user interfaces suitable both for the Virtual Environ- ment and for the Volume Rendering application is difficult. Pre- In this paper, we propose an intuitive user interface for Volume vious approaches mainly focused on porting traditional 2D point- Rendering specifically designed for Virtual Reality environments. and-click concepts to the Virtual Environment [8, 5, 9]. This tends The proposed interface allows transfer function design and refine- to be unintuitive, to complicate the interaction, and to make only ment based on intuitive two-handed operation of Wand-like con- limited use of available interaction devices. trollers. Additional interaction modes such as navigation and clip In this paper, we propose an intuitive 3D user interface for Vol- plane manipulation are supported as well. ume Rendering based on interaction devices that are suitable for The system is implemented using the Sony PlayStation Move Virtual Reality environments.
    [Show full text]
  • Parallel Computer Architecture
    Parallel Computer Architecture Introduction to Parallel Computing CIS 410/510 Department of Computer and Information Science Lecture 2 – Parallel Architecture Outline q Parallel architecture types q Instruction-level parallelism q Vector processing q SIMD q Shared memory ❍ Memory organization: UMA, NUMA ❍ Coherency: CC-UMA, CC-NUMA q Interconnection networks q Distributed memory q Clusters q Clusters of SMPs q Heterogeneous clusters of SMPs Introduction to Parallel Computing, University of Oregon, IPCC Lecture 2 – Parallel Architecture 2 Parallel Architecture Types • Uniprocessor • Shared Memory – Scalar processor Multiprocessor (SMP) processor – Shared memory address space – Bus-based memory system memory processor … processor – Vector processor bus processor vector memory memory – Interconnection network – Single Instruction Multiple processor … processor Data (SIMD) network processor … … memory memory Introduction to Parallel Computing, University of Oregon, IPCC Lecture 2 – Parallel Architecture 3 Parallel Architecture Types (2) • Distributed Memory • Cluster of SMPs Multiprocessor – Shared memory addressing – Message passing within SMP node between nodes – Message passing between SMP memory memory nodes … M M processor processor … … P … P P P interconnec2on network network interface interconnec2on network processor processor … P … P P … P memory memory … M M – Massively Parallel Processor (MPP) – Can also be regarded as MPP if • Many, many processors processor number is large Introduction to Parallel Computing, University of Oregon,
    [Show full text]
  • Effective User Interface Design for Consumer Trust Two Case Studies
    2005:097 SHU MASTER'S THESIS Effective User Interface Design for Consumer Trust Two Case Studies XILING ZHOU XIANGCHUN LIU Luleå University of Technology MSc Programme in Electronic Commerce Department of Business Administration and Social Sciences Division of Industrial marketing and e-commerce 2005:097 SHU - ISSN: 1404-5508 - ISRN: LTU-SHU-EX--05/097--SE Luleå University of Technology E-Commerce ACKNOWLEDGEMENT This thesis is the result of half a year of work whereby we have been accompanied and supported by many people. It is a pleasant aspect that we could have this opportunity to express our gratitude to all of them. First, we are deeply indebted to our supervisor Prof. Lennart Persson who is from Division of Industrial Marketing at LTU. He helped us with stimulating suggestions and encouragement in all the time of research and writing of this thesis. Without his never-ending support during this process, we could not have done this thesis. Especially, we would like to express our gratitude to all of participants, who have spent their valuable time to response the interview questions and discuss with us. Finally, we would like to thank our family and friends. I, Zhou Xiling am very grateful for everyone who gave me support and encouragement during this process. Especially I felt a deep sense of gratitude to my father and mother who formed part of my vision and taught me the good things that really matter in the life. I also want to thank my friend Tang Yu for his never-ending support and good advices. I, Liu XiangChun am very grateful for my parents, for their endless love and support.
    [Show full text]
  • A Review of Multicore Processors with Parallel Programming
    International Journal of Engineering Technology, Management and Applied Sciences www.ijetmas.com September 2015, Volume 3, Issue 9, ISSN 2349-4476 A Review of Multicore Processors with Parallel Programming Anchal Thakur Ravinder Thakur Research Scholar, CSE Department Assistant Professor, CSE L.R Institute of Engineering and Department Technology, Solan , India. L.R Institute of Engineering and Technology, Solan, India ABSTRACT When the computers first introduced in the market, they came with single processors which limited the performance and efficiency of the computers. The classic way of overcoming the performance issue was to use bigger processors for executing the data with higher speed. Big processor did improve the performance to certain extent but these processors consumed a lot of power which started over heating the internal circuits. To achieve the efficiency and the speed simultaneously the CPU architectures developed multicore processors units in which two or more processors were used to execute the task. The multicore technology offered better response-time while running big applications, better power management and faster execution time. Multicore processors also gave developer an opportunity to parallel programming to execute the task in parallel. These days parallel programming is used to execute a task by distributing it in smaller instructions and executing them on different cores. By using parallel programming the complex tasks that are carried out in a multicore environment can be executed with higher efficiency and performance. Keywords: Multicore Processing, Multicore Utilization, Parallel Processing. INTRODUCTION From the day computers have been invented a great importance has been given to its efficiency for executing the task.
    [Show full text]
  • A High Level User Interface for Topology Controlled Volume Rendering
    Topological Galleries: A High Level User Interface for Topology Controlled Volume Rendering Brian MacCarthy, Hamish Carr and Gunther H. Weber Abstract Existing topological interfaces to volume rendering are limited by their re- liance on sophisticated knowledge of topology by the user. We extend previous work by describing topological galleries, an interface for novice users that is based on the design galleries approach. We report three contributions: an interface based on hier- archical thumbnail galleries to display the containment relationships between topo- logically identifiable features, the use of the pruning hierarchy instead of branch decomposition for contour tree simplification, and drag-and-drop transfer function assignment for individual components. Initial results suggest that this approach suf- fers from limitations due to rapid drop-off of feature size in the pruning hierarchy. We explore these limitations by providing statistics of feature size as function of depth in the pruning hierarchy of the contour tree. 1 Introduction The overall goal of scientific visualisation is to provide useful insight into existing data. As visualisation techniques have become more complex, so have the interfaces for controlling them. In cases where the interface has been simplified, it has often been at the cost of the functionality of the program. Thus, while expert users are capable of using state of the art technology, novice users who would otherwise have uses for this technology are restricted by unintuitive interfaces. Topology-based volume rendering, while powerful, is difficult to apply success- fully. This difficulty is due to the fact that the interface used to design transfer func- Brian MacCarthy University College Dublin, Belfield, Dublin 4, Ireland, e-mail: [email protected] Hamish Carr University of Leeds, Woodhouse Lane, Leeds LS2 9JT, England e-mail: [email protected] Gunther H.
    [Show full text]
  • Xfel Database User Interface P.D
    THPB039 Proceedings of SRF2015, Whistler, BC, Canada XFEL DATABASE USER INTERFACE P.D. Gall, V. Gubarev, D. Reschke, A. Sulimov, J.H. Thie, S. Yasar DESY, Notkestrasse 85, 22607 Hamburg, Germany Abstract Using this protection system we have divided all The XFEL database plays an important role for an customers into different groups: effective part of the quality control system for the whole ● RI group has permission to view production cavity production and preparation process for the results from RI only and open information European XFEL on a very detailed level. Database has the ● ZANON group has permission to view Graphical User Interface based on the web-technologies, production results from ZANON only and open and it can be accessed via low level Oracle SQL. information ● DESY group has permission to view all results INTRODUCTION from all companies Beginning from TTF a relational database for cavities ● Not authorised people have access to the open was developed at DESY using the ORACLE Relational information only Database Management System (RDBMS) [1]. To get authorised access to the database one have to The database is dynamically accessible from contact the responsible persons listed in the XFEL everywhere via a graphical WEB interface based on database GUI pages. ORACLE. At the moment we use the version Oracle GRAPHICAL USER INTERFACE Developer 10g Forms and Reports. The graphical tools are developed in Java. The XFEL database GUI was developed to meet the The database is created to store data for more than 840 requirements of experts involved. According to the people cavities coming from the serial production and about 100 needs the GUI applications can logically be divided into modules.
    [Show full text]