Direct3D ShaderX Vertex and Pixel Shader Tips and Tricks Edited by Wolfgang F. Engel Wordware Publishing, Inc. Team LRN Library of Congress Cataloging-in-Publication Data Direct3d ShaderX : vertex and pixel shader tips and tricks / edited by Wolfgang F. Engel. p. cm. Includes bibliographical references and index. ISBN 1-55622-041-3 1. Computer games--Programming. 2. Three-dimensional display systems. 3. Direct3D. I. Engel, Wolfgang F. QA76.76.C672 D57 2002 794.8'15265--dc21 2002006571 © 2002, Wordware Publishing, Inc. All Rights Reserved 2320 Los Rios Boulevard Plano, Texas 75074 No part of this book may be reproduced in any form or by any means without permission in writing from Wordware Publishing, Inc. Printed in the United States of America ISBN 1-55622-041-3 10987654321 0502 RenderMan is a registered trademark of Pixar Animation Studios. krass engine is a trademark of Massive Development. 3D Studio Max is a registered trademark of Autodesk/Discreet in the USA and/or other countries. GeForce is a trademark of NVIDIA Corporation in the United States and/or other countries. Some figures in this book are copyright ATI Technologies, Inc., and are used with permission. Other product names mentioned are used for identification purposes only and may be trademarks of their respective companies. All inquiries for volume purchases of this book should be addressed to Wordware Publishing, Inc., at the above address. Telephone inquiries may be made by calling: (972) 423-0090 Team LRN iii Contents Foreword .......................................xvi Acknowledgments..................................xvii Part 1: Introduction to Shader Programming Fundamentals of Vertex Shaders ...........................4 Wolfgang F. Engel What You Need to Know/Equipment .........................5 Vertex Shaders in the Pipeline .............................5 Why Use Vertex Shaders? ...............................7 Vertex Shader Tools ..................................8 NVIDIA Effects Browser 2/3 ..........................8 NVIDIA Shader Debugger ...........................9 Shader City ...................................10 Vertex Shader Assembler............................10 NVIDIA NVASM — Vertex and Pixel Shader Macro Assembler ....10 Microsoft Vertex Shader Assembler ...................11 Shader Studio ..................................11 NVLink 2.x ...................................12 NVIDIA Photoshop Plug-ins..........................13 Diffusion Cubemap Tool ............................14 DLL Detective with Direct3D Plug-in .....................15 3D Studio Max 4.x/gmax 1.1..........................16 Vertex Shader Architecture..............................16 High-Level View of Vertex Shader Programming ..................18 Check for Vertex Shader Support .......................19 Vertex Shader Declaration ...........................19 Set the Vertex Shader Constant Registers ...................22 Write and Compile a Vertex Shader ......................22 Application Hints .............................26 Complex Instructions in the Vertex Shader................27 Putting It All Together ..........................27 Swizzling and Masking ..........................31 Guidelines for Writing Vertex Shaders ..................32 Compiling a Vertex Shader ........................33 Create a Vertex Shader .............................34 Set a Vertex Shader ...............................34 Free Vertex Shader Resources .........................35 Summary ....................................35 What Happens Next? .................................35 Team LRN iv Contents Programming Vertex Shaders ............................38 Wolfgang F. Engel RacorX ........................................38 The Common Files Framework ........................40 Check for Vertex Shader Support.....................42 Vertex Shader Declaration ........................42 Set the Vertex Shader Constant Registers ................43 The Vertex Shader ............................43 Compile a Vertex Shader .........................45 Create a Vertex Shader ..........................46 Set a Vertex Shader ............................47 Free Vertex Shader Resources ......................47 Non-Shader Specific Code ........................47 Summary ....................................49 RacorX2........................................49 Create a Vertex Shader .............................51 Summary ....................................53 RacorX3........................................53 Vertex Shader Declaration ...........................54 Set the Vertex Shader Constant Registers ...................54 The Vertex Shader ...............................55 Directional Light .............................57 Diffuse Reflection ............................57 Summary ....................................59 RacorX4........................................59 Vertex Shader Declaration ...........................59 Set the Vertex Shader Constants ........................60 The Vertex Shader ...............................61 Specular Reflection ............................62 Non-Shader Specific Code ...........................65 Summary ....................................66 RacorX5........................................66 Point Light Source ...............................66 Light Attenuation for Point Lights .......................66 Set the Vertex Shader Constants ........................67 The Vertex Shader ...............................68 Summary ....................................70 What Happens Next? .................................70 Fundamentals of Pixel Shaders ...........................72 Wolfgang F. Engel Why Use Pixel Shaders? ...............................72 Pixel Shaders in the Pipeline .............................74 Pixel Shader Tools ..................................79 Microsoft Pixel Shader Assembler.......................79 MFC Pixel Shader ...............................80 Team LRN Contents v ATI ShadeLab..................................80 Pixel Shader Architecture ..............................81 Constant Registers (c0-c7) ...........................82 Output and Temporary Registers (ps.1.1-ps.1.3: r0+r1; ps.1.4: r0-r5) ....82 Texture Registers (ps.1.1-ps.1.3: t0-t3; ps.1.4: t0-t5) .............82 Color Registers (ps.1.1-ps.1.4: v0+v1) ....................83 Range ......................................83 High-Level View of Pixel Shader Programming ..................84 Check for Pixel Shader Support ........................84 Set Texture Operation Flags (D3DTSS_* flags)................85 Set Texture (with SetTexture()) ........................86 Define Constants (with SetPixelShaderConstant()/def ) ...........86 Pixel Shader Instructions ............................87 Texture Address Instructions .......................88 Arithmetic Instructions .........................101 Instruction Modifiers ..........................110 Instruction Modifiers ..........................116 Instruction Pairing ............................119 Assemble Pixel Shader ............................120 Create a Pixel Shader .............................120 Set Pixel Shader ................................121 Free Pixel Shader Resources .........................121 Summary.......................................121 What Happens Next? ................................122 Programming Pixel Shaders ............................125 Wolfgang F. Engel RacorX6 .......................................125 Check for Pixel Shader Support .......................126 Set Texture Operation Flags (with D3DTSS_* flags) ............126 Set Texture (with SetTexture()) ........................127 Define Constants (with SetPixelShaderConstant()/def) ...........127 Pixel Shader Instructions ...........................128 Per-Pixel Lighting ............................130 Assemble Pixel Shader ............................134 Create Pixel Shader ..............................134 Set Pixel Shader ................................136 Free Pixel Shader Resources .........................136 Non-Shader Specific Code ..........................136 Summary ...................................136 RacorX7 .......................................137 Define Constants (with SetPixelShaderConstant()/def) ...........137 Pixel Shader Instructions ...........................137 Summary ...................................140 RacorX8 .......................................140 Set Texture Operation Flags (with D3DTSS_* flags) ............140 Team LRN vi Contents Set Texture (with SetTexture()) ........................140 Pixel Shader Instructions ...........................142 Summary ...................................143 RacorX9 .......................................144 Summary ...................................147 Further Reading ...................................147 Basic Shader Development with Shader Studio ..................149 John Schwab Introduction .....................................149 What You Should Know ..............................149 Installation...................................149 Directories ...................................149 Coordinate Systems ..............................150 Features ....................................150 Limitations ..................................151 Default Light .................................152 Controls ....................................152 Create a Vertex Shader ...............................152 Step 1: Loading a Mesh ............................153 Step 2: Transformations............................153 Step 3: Starting a Shader ...........................154 Step 4: Editing