USOO7385612B1

(12) United States Patent (10) Patent No.: US 7,385,612 B1 Peterson (45) Date of Patent: Jun. 10, 2008

(54) DISTORTION OF RASTER AND VECTOR 6,232,987 B1 5, 2001 Choi et al. ARTWORK 6,271,864 B1 8/2001 Graham ...... 345,442 6,300,955 B1 10/2001 Zamir 6.459,439 B1 10/2002 Ahlquist, Jr. et al. (75) Inventor: John Peterson, Menlo Park, CA (US) 6,501.475 B1 12/2002 Cheng (73) Assignee: Adobe Systems Incorporated, San 6,522,328 B1 2/2003 Asente ...... 345,441 6,628,295 B2 * 9/2003 Wilensky ...... 345,594 Jose, CA (US) 6,633,300 B1 * 10/2003 Tomack et al...... 345,581 6,765,589 B1* 7/2004 Georgiev et al...... 345,646 (*) Notice: Subject to any disclaimer, the term of this 6,803,913 B1 * 10/2004 Fushiki et al...... 345/467 patent is extended or adjusted under 35 6,870,545 B1* 3/2005 Smith et al...... 345,619 U.S.C. 154(b) by 0 days. 6,911,980 B1* 6/2005 Newell et al...... 345,441 6,919,888 B1* 7/2005 Perani et al...... 345,420 (21) Appl. No.: 10/160,574 6,999,101 B1* 2/2006 Sanborn et al...... 345,619 (22) Filed: May 30, 2002 OTHER PUBLICATIONS Barry Fowler, Geometric manipulation of tensor product surfaces, (51) Int. Cl. Proceedings of the 1992 symposium on Interactive 3D graphics, p. G09G 5/00 (2006.01) 101-108, Jun. 1992, Cambridge, Massachusetts, United States.* (52) U.S. Cl...... 345/619; 34.5/581 (Continued) (58) Field of Classification Search ...... 345/442, 345/591-594,646–647, 441, 619, 581,474 Primary Examiner Jin-Cheng Wang See application file for complete search history. (74) Attorney, Agent, or Firm Fish & Richardson P.C. (56) References Cited (57) ABSTRACT U.S. PATENT DOCUMENTS Systems and techniques to apply an image distortion to two 5,428,728 A * 6/1995 Lung et al...... 345/469 image objects of different graphic types. In general, in one 5,481,659 A 1/1996 Nosaka et al. implementation, the technique includes: receiving an image 5,539,868 A 7/1996 Hosoya et al. distortion description to be applied to an image portion 5,598,182 A 1/1997 Berend et al. including a vector graphic and a raster graphic, the raster 5,731,819 A * 3/1998 Gagne et al...... 345,647 graphic being distortable separate from the vector graphic, 5,731,820 A 3/1998 Broekhuijsen applying the image distortion description to the vector 5,736,991 A 4, 1998 Tada ...... 345/474 graphic to produce a distorted vector graphic, and applying 5,754,183 A 5, 1998 Berend et al. the image distortion description to the raster graphic to 5,852,447 A 12/1998 Hosoya et al. 5,857,067 A 1/1999 Hassett et al. produce a distorted raster graphic, the distorted vector 5,920,324 A 7, 1999 Hasegawa et al. graphic and the distorted raster graphic together forming a 5,929,867 A 7, 1999 Herbstman et al. distorted image portion. 6,157.390 A 12/2000 Cheng 6,201,549 B1* 3/2001 Bronskill ...... 345,441 28 Claims, 14 Drawing Sheets

RECEWEARST NPTSIGNANGA foc RASTER GRACT BSC

RECEWASCON N. ESGNAGA 105 WCTOR GRAPCO BESTORE

RECEVEARD NUDEFINING AN fic MAGESRON ESCRIN

APPLY THE MAGE SCCNSCRON f 15 CTHE WECOR GRAC Osrl JCAISEOR WECTOR GRApkic

Y HE&E STORTON DESCRON TO THErASTER GRAPHC CPROCEA SORE ASER GRAC US 7,385,612 B1 Page 2

OTHER PUBLICATIONS CorelDraw 9 for Windows: Visual QuickStart Guide, Publisher: Peachpit Press, Jul. 6, 1999, Chapters 4, 14-16.* Venkat Krishnamurthy, Marc Levoy, Fitting smooth surfaces to CorelDraw 10 for Windows: Visual QuickStart Guide, Publisher: dense polygon meshes, Proceedings of the 23rd annual conference Peachpit Press, Nov. 6, 2001, Chapters 4, 14-16.* on Computer graphics and interactive techniques, p. 313-324, Aug. U.S. Appl. No. 09/458.917, filed Dec. 10, 1999, Newell et al. 1996. Peterson, John W., “IV.6 Tessellation of NURB Surfaces.” Copy Frederick W. B. Li, Rynson W. H. Lau, Real-time rendering of right 1994 by Academic Press, Inc. deformable parametric free-form surfaces, Proceedings of the ACM Watkins, M.A. and Worsey, A.J., “Degree reduction of Bezier Symposium on Virtual reality Software and technology, p. 131-138, curves.” CAD v20, #7, Sep. 1988. Dec. 20-22, 1999, London, United Kingdom.* Peterson, John, “Letters to the Editor Degree Reduction of Bezier Tony DeRose, Michael Kass, Tien Truong, Subdivision surfaces in curves.” Computer Aided Design vol. 23, #6 Jul/Aug. 1991. character animation, Proceedings of the 25th annual conference on DeRose, Tony D. et al., “Functional Composition Algorithms via Computer graphics and interactive techniques, p. 85-94, Jul. 1998.* Blossoming.” May 28, 1992. Tim Milliron, Robert J. Jensen, Ronen Barzel, Adam Finkelstein, Heckbert, Paul S., “Fundamentals of Texture Mapping and Image A framework for geometric warps and deformations, ACM Trans Warping.” Master's Thesis, Dept. of Electrical Engineering and actions on Graphics (TOG), v. 21 N. 1, p. 20-51, Jan. 2002.* Computer Science University of California, Berkeley, CA. Jun. 17. Steven Schkolne, Michael Pruett, Peter Schröder, Surface drawing: 1989. creating organic 3D shapes with the hand and tangible tools, Richard H. Bartels et al., “A Technique for the Direct Manipulation Proceedings of the SIGCHI conference on Human factors in com of Spline Curves'. Graphics Interface 1989, pp. 33-37. puting systems, p. 261-268, Mar. 2001, Seattle, Washington.* Barry Fowler, “Geometric Manipulation of Tensor Product Sur Karan Singh . Eugene Fiume, Wires: a geometric deformation faces”, ACM 0-89791-471-6/92/0003/0101, 1992, pp. 101-108. technique, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p. 405-414, Jul. 1998.* * cited by examiner U.S. Patent Jun. 10, 2008 Sheet 1 of 14 US 7,385,612 B1

RECEWEA FRS - 100 NPT ESGNANG A RASER GRAC O E. SORED

RECEW ASECN: iO5 N DESGNANG A / vs WCOR GRAPC. C. BE DSCRE:

RECEWEA RI) - 1 10 N ENNG AN MAGE SORON ESCRON

AY - VAGE DSTORTION DESCRIPTION - 115 C E WECOR GRAC / O ROCE A SORE WECOR GRAC

AY - VACGE DISTORTION DESCRIPTION - 120 O E KASER GRAC C. RCCE A SORE RASER GrAC

U.S. Patent Jun. 10, 2008 Sheet 2 of 14 US 7,385,612 B1

U.S. Patent Jun. 10, 2008 Sheet 3 of 14 US 7,385,612 B1

U.S. Patent Jun. 10, 2008 Sheet 4 of 14 US 7,385,612 B1

4OO Y

420 -

Y S w

Y msw i w & is is - werse s seas

s 1.

* X w s

42

U.S. Patent Jun. 10, 2008 Sheet S of 14 US 7,385,612 B1

55G Y.

U.S. Patent Jun. 10, 2008 Sheet 6 of 14 US 7,385,612 B1

M Ro(idrag) Rarag) R2(dra }

U.S. Patent Jun. 10, 2008 Sheet 7 of 14 US 7,385,612 B1

U.S. Patent Jun. 10, 2008 Sheet 8 of 14 US 7,385,612 B1

800

U.S. Patent Jun. 10, 2008 Sheet 9 of 14 US 7,385,612 B1

U.S. Patent Jun. 10, 2008 Sheet 10 of 14 US 7,385,612 B1

4. O8. 8

U.S. Patent Jun. 10, 2008 Sheet 11 of 14 US 7,385,612 B1

RECEIVEMAGE STORION too DESCRIPTION TO BE APPLIED / O AVECTOR GRAPC AND A RASER GRAC

CONVERT COORDINATES 110s OF THE VECTOR GRAPHC / INTO PARAMETERSOR ON REGON SPACE OF

COMPOSE THE IMAGE 7-1110 SOR ON DESCRON W - WECOR GRAC

if it 5 RECE ORDER OF RESULTING VECTOR GRAPHIC

SORON SBW) - REGON CARGER NYes - ini. Soon

THAN RASTER / ESCRON NGRAPHC / oro"'N

SE WAGE SORCN DESCRIPTIONASA CONTROL STRUCTURE GOVERN NG st RESANG O RASER GRAC U.S. Patent Jun. 10, 2008 Sheet 12 of 14 US 7,385,612 B1

S. SSNS U.S. Patent Jun. 10, 2008 Sheet 13 of 14 US 7,385,612 B1

;3 OC Y. v

1380 - 1365, f 382

FIG. 27

US 7,385,612 B1 1. 2 DISTORTION OF RASTER AND VECTOR through a single interface tool in a graphical user interface. ARTWORK The image portion can be a whole image or part of an image and can include a vector graphic and a raster graphic, where BACKGROUND OF THE INVENTION the raster graphic is distortable separate from the vector graphic (i.e., there need not be a predefined mapping asso The present application describes systems and techniques ciation between the raster graphic and the vector graphic). relating to distortion of raster and vector artwork. The image distortion description can be applied to the vector Many types of artwork are stored and manipulated in graphic to produce a distorted vector graphic, and the image computing machines, and Such artwork can be stored digi distortion description can be applied to the raster graphic to tally in multiple formats. A single artwork image can include 10 produce a distorted raster graphic. The distorted vector multiple distinct image objects, which can also be stored in graphic and the distorted raster graphic together form a different formats. For example, a raster graphic is an image distorted image portion. object stored as a set of pixel values corresponding to a The image distortion description can be applied to the defined frame (e.g., a rectangular grid). The pixel values can raster graphic independently from the image distortion include multiple values for each pixel, including multiple 15 description being applied to the vector graphic. The image color values, transparency/opacity values, and/or mask val distortion description can include a mathematical definition ues (e.g., an image clipping mask). A raster graphic repre of an image distortion in two dimensions. Distorting the sents an image as a set of discrete image samples. A vector raster graphic can involve using the image distortion graphic is an image object stored using a set of two description as a control structure governing resampling of dimensional coordinates connected by lines or curves (e.g., 20 the raster graphic. Distorting the raster graphic can involve cubic Bézier curves). Examples of vector graphics include tessellating the image distortion description, and resampling symbols, shapes, and text represented by outline fonts. the raster graphic based on the tessellated image distortion Interactive painting and illustration Software products, description. Distorting the vector graphic can involve com such as Illustrator(R), provided by Adobe Systems Incorpo posing the image distortion description with the vector rated of San Jose, Calif., provide tools to allow a user to 25 graphic to retain curve information as curve information and distort an image object. Such distortion functionality can be to maintain a resolution independent aspect of the vector useful for adding creative effects to an image object and/or graphic. to integrate an image object with other elements in a finished The raster graphic need not have a mapping association design (e.g., fitting artwork onto a circular label, or making with another image object in the image portion. The image artwork appear three-dimensional). Image distortions 30 portion can be part of a final format electronic document. include geometric and optical distortions, as well as image Receiving the image distortion description can involve warps and deformations generally (e.g., a deformation and a receiving a first input designating the raster graphic to be warp of a two-dimensional image are examples of an image distorted, receiving a second input designating the vector distortion). graphic to be distorted, and receiving a third input defining Example illustration software products that allow distor- 35 the image distortion description. Receiving the first and tions of raster images include GoovieR), provided by Meta second inputs designating the raster graphic and vector Creations Corporation of Carpinteria, Calif., Ampex Video graphic can involve receiving inputs defining a selection box Art, provided by Ampex Corporation of Redwood City, to cover a portion of the raster graphic and a portion of the Calif., and Photoshop(R), provided by Adobe Systems Incor vector graphic. Alternatively, receiving the first and second porated of San Jose, Calif. Example illustration software 40 inputs designating the raster graphic and vector graphic can products that allow distortions of vector artwork include involve receiving inputs generated by a macro operation. FreeHandR, provided by Macromedia Inc. of San Francisco, Receiving the third input defining the image distortion Calif., and CorelDraw, provided by Corel Corporation of description can involve receiving inputs specifying a con Ottawa, Ontario, Canada. figuration of an adjustable predefined distortion type, or 45 receiving inputs specifying movements of a cursor across a SUMMARY OF THE INVENTION distortion region. Distorting the vector graphic can involve converting coordinates of the vector graphic into a param In general, in one aspect, the invention features applica eter space of the distortion region. Distorting the raster tion of an image distortion to two image objects of different graphic can involve Subdividing the image distortion graphic types. According to an aspect, a single image 50 description before applying the image distortion description distortion description can be received, and this image dis to the raster graphic, if the distortion region is larger than the tortion description can be applied to two image objects of raster graphic. The image distortion description can include different graphic types to distort the image objects, where a tensor-product Bézier mesh, and the vector graphic can the image objects are separately distortable. The distortion include a cubic Bézier curve. of the two image objects can create a consistent distortion 55 A system can be provided to implement these features and across the image objects as defined by the image distortion techniques. The system can be part of a general purpose description, while retaining the distinct graphic characteris processing system, or the system can be a special purpose tics of the two image objects. Moreover, the distortion system designed for image processing and manipulation. description can be applied to the second image object Implementations of the systems and techniques described independently from the application to the first image object. 60 here may occur in hardware, firmware, Software or a com One image object need not be distorted through another bination of them and may include instructions for causing a image object, and the two image objects can be distorted by machine to perform the operations described. the same distortion description even if they are not distorted The invention can be implemented to realize one or more simultaneously. of the following advantages. Multiple types of image Advantageous implementations of the invention include 65 objects, such as vector graphics and raster graphics, can be one or more of the following features. An image distortion distorted simultaneously using a uniform control interface description to be applied to an image portion can be received while retaining the original formats. A graphical user inter US 7,385,612 B1 3 4 face can provide a single warping tool that allows both raster FIG. 28 illustrates a distorted electronic document having and vector artwork to be distorted together. A single distor multiple image objects of various types. tion description is applied to both vector and raster artwork Like reference symbols in the various drawings indicate to produce a consistent distortion across a distorted image, like elements. without converting the vector graphic into a raster graphic. Because vector artwork need not be converted into raster DETAILED DESCRIPTION form to accomplish the distortion, the advantages of vector artwork (e.g., device independent resolution) are retained. As used herein, the terms “artwork”. “image object' and The distortion description can be a free-form distortion or a 'graphic' are used interchangeably. Thus, a raster graphic is predefined type of distortion. A user of the distortion tool 10 also referred to as a raster image object and raster artwork does not need to consider format differences between the (with no intended suggestion that the graphic is in fact art). various image objects selected for distortion. A graphic is a set of data defining an image or image portion A mathematical control structure (e.g., a tensor product that can be displayed. Bézier mesh) can be used to specify an image distortion, and FIG. 1 is a flowchart showing a process of distorting a this same mathematical control structure can be applied to 15 group of image objects of different graphic types, including both vector graphics and raster graphics, thus allowing a a raster graphic and a vector graphic. A first input designat user of a painting and illustration program to distort both ing a raster graphic to be distorted is received at 100. A vector graphics and raster graphics at the same time with one second input designating a vector graphic to be distorted is image distortion description. The geometric result is the received at 105. The manner and order of the designations same on both types of artwork, and the process is transparent can be varied, but in general, a vector graphic and a raster to the user. graphic are separately designated by a user. Moreover, The details of one or more embodiments of the invention additional raster and vector graphics and/or portions of are set forth in the accompanying drawings and the descrip raster and vector graphics can be designated (e.g., selected, tion below. Other features and advantages of the invention or defined) for the same distortion to be applied. will become apparent from the description, the drawings, 25 Once a group of image objects have been designated, a and the claims. third input defining an image distortion description is received at 110. The image distortion description can BRIEF DESCRIPTION OF THE DRAWINGS describe a free-form distortion or a predefined type of distortion, such as a twist, Squeeze, twirl, wave, flag, arc, FIG. 1 is a flowchart showing a process of distorting a 30 Zig-Zag, ripple, spherify, pucker, bloat, or reflection distor group of image objects of different graphic types, including tion. The third input can be received from a distortion tool a raster graphic and a vector graphic. interface that can be a simple button or a more detailed FIG. 2 illustrates designation of a raster graphic and interface that allows a user to adjust distortion parameters multiple vector graphics to be distorted. before application and/or to paint a distortion directly onto FIG. 3 illustrates results of two predefined image distor 35 the designated raster and vector graphics. Detailed interfaces tions. can be provided for free-form distortion tools and/or pre FIG. 4 illustrates an image distortion description. defined-type distortion tools. FIG. 5 illustrates a technique for receiving an image The image distortion description is a mathematical defi distortion description. nition of an image distortion in two dimensions. This FIG. 6 illustrates direct curve manipulation. 40 mathematical definition can include higher dimensional FIG. 7 shows response curves. components, but in general, the mathematical definition FIG. 8 shows quadratic response curves. represents a mapping of a first two-dimensional image to a FIGS. 9 and 10 illustrate sampling to determine surface second two-dimension image. The image distortion descrip parameters. tion is applied to the vector graphic to produce a distorted FIG. 11 illustrates curve subdivision. 45 vector graphic at 115. Straight lines in the vector graphic can FIGS. 12 and 13 illustrate subdivision to determine Sur be converted into curves as needed, and curves in the vector face parameters. graphic can be distorted by modifying their control points FIG. 14 illustrates treatment of a surface drag as a series and/or Subdividing them into additional curves. of curve drags. The same image distortion description is applied to the FIG. 15 illustrates a direct manipulation of an “inflate' 50 raster graphic to produce a distorted raster graphic at 120. distortion. The frame of the raster graphic can be warped by the image FIG. 16 illustrates a wave distortion. distortion description into a distorted frame, and the image FIG. 17 shows linear response curves. distortion description (e.g., a distortion Surface Such as a FIG. 18 shows scaled response curves. Bézier tensor product mesh) can be converted into small FIG. 19 shows linear response curves. 55 polygons (e.g., triangles) that cover the distorted frame. For FIG. 20 shows scaled response curves. each polygon, the portion of the original raster graphic can FIG. 21 is a flowchart showing a process of distorting a be mapped onto the corresponding polygon. raster graphic and a vector graphic. The same distortion can thus be applied to both raster FIG. 22 illustrates a technique for distorting vector graph image objects and vector image objects at essentially the 1CS 60 same time. A user of an interactive painting and illustration FIGS. 23 and 24 illustrate a technique for distorting a program need not be aware of which objects in a piece of raster graphic. artwork are vector graphics and which are raster graphics. A FIG. 25 is a block diagram illustrating example subdivi uniform set of user interface controls for image distortion sion of a specified distortion and ordering of vertices. can be provided. Such uniform interface controls can allow FIG. 26 illustrates generated edge shards for handling 65 deformation and warping of selected artwork without chang image edges. ing the underlying fundamental nature of the artwork. Raster FIG. 27 illustrates subdividing a distortion region. artwork remains raster artwork, and vector artwork remains US 7,385,612 B1 5 6 vector artwork, but both are distorted in a visually consistent modified by a predefined flag distortion to create a flag manner across a distortion region as defined by the image image 300. A second distortion shows the artwork image 200 distortion description. as modified by a predefined arc distortion to create an arc FIG. 2 illustrates designation of a raster graphic and image 350. multiple vector graphics to be distorted. An image 200 is a piece of artwork that includes an image portion 205, which FIG. 4 illustrates an image distortion description. The itself is another image. The image 205 includes multiple image distortion description is a mathematical definition of image objects: a raster graphic 210, which is a picture of a an image distortion in two dimensions. This mathematical cat, an arrow vector graphic 215, a box vector graphic 220, definition can be parameterized using a grid of control a border vector graphic 225, and a text vector graphic 230. 10 points. When a distortion region is defined, a uniform The image objects have coordinates associated with the Surface (e.g., a rectangular Surface) can be created over the image 200 but not necessarily with each other. The image artwork in the distortion region. By distorting the shape of objects can be selected and/or portions of the image objects the uniform surface, the artwork in the distortion region can can be defined for distortion. Text objects that are not vector be distorted in a manner corresponding to the distorted graphics can be dynamically converted to vector graphics 15 Surface. (e.g., converted to outline fonts, or an alternative vector graphic format that includes text data, Such as Unicode or For example, a piece of artwork 400 has an associated ASCII (American Standard Code for Information Inter distortion surface 410, which can be a tensor product sur change) data) before being distorted. Such conversion of face. The distortion surface 410 has nine control points 420 text objects can be performed by a user or automatically by that define the shape of the Surface and thus the mapping of the distortion control interface. Additionally, a distortion the artwork 400 from a undistorted two-dimensional image operation can be applied directly using a graphical painting to a distorted two-dimensional image. Additional control tool. Such as a brush tool of variable shapes and sizes. points can be used, and the distortion surface can be a Bézier When a portion of a raster graphic is defined for distortion bicubic Surface, which has sixteen control points. (e.g., a selection box is drawn over a portion of the raster 25 These control points can be moved individually to reshape graphic), that portion can be separated and treated as a the distortion surface, or a more intuitive control interface distinct raster graphic for distortion. A distortion region of can be provided. Additional control handles can be provided the raster graphic and a slightly larger sampling region can that have predefined effects on the control points of the be defined. When a portion of a vector graphic is defined for distortion surface to create an adjustable predefined type of distortion, that portion can be carved off and treated as a 30 distinct vector graphic for distortion. The non-distortion distortion (e.g., symmetrical distortions such as those men portion of the vector graphic can either be separated from the tioned previously). Alternatively, a free-form type of distor distortion portion, or remain connected (e.g., with a straight tion control interface can be provided, such as a control line). Carving off the distortion portion can involve finding interface that draws an envelope around a distortion region a parametric point along a curve (e.g., the point at which the 35 and allows dragging of the envelope, or a control interface curve intersects an edge of the distortion region) and creat that causes movements of a cursor across the distortion ing two curves that intersect at this parametric point. region to have predefined effects on the distortion surface Image objects can be designated for distortion using a Such as allowing dragging of arbitrary points within the point-and-click-interface, a selection box drawing interface, distortion region. or other interfaces. For example, a selection box 240 can be 40 FIG. 5 illustrates a technique for receiving an image drawn over the image 205 to select the image objects and distortion description. A piece of artwork 450 has an asso define a distortion region. The box 240 can define a distor ciated distortion Surface with control points such as control tion region directly, in which case the box 240 designates a points 455. A cursor 460 can be used to select and drag an portion of the raster graphic 210, the entire arrow vector arbitrary target point on the distortion Surface and the graphic 215, a portion of the box vector graphic 220, a 45 portion of the border vector graphic 225, and a portion of the software determines how the control points, and thus the text vector graphic 230. In this case, the box 240 represents distortion surface, are distorted. The distortion surface the distortion region. behaves as a flexible or pliable sheet, which can be stretched Alternatively, the box 240 can define the distortion region and pulled using the cursor 460. Thus, movement of the indirectly by selecting all image objects that the box 240 50 cursor 460 causes movement of the control points, where the either contains or intersects. In this case, the box 240 defines amount of movement of a control point depends on that a distortion region 260 that includes all of the image objects control points relation to an arbitrary point on the distortion in the image 205. The selection box interface of this example Surface being dragged. The distortion Surface can have many provides the first and second inputs described above. Thus, control points creating a mesh. For example, the distortion drawing the box 240 to cover an upper left corner of the 55 surface can be a tensor product Bézier mesh. border vector graphic 225 represents the first input desig Manipulation of curves forms the basis of the flexible nating the vector graphic, and drawing the box 240 over an sheet method of distortion manipulation described above. upper left corner of the raster graphic 210 represents the Moreover, the notation for curves is simpler than for sur second input designating the raster graphic. Alternative faces, thus curve manipulation is described first. image object designation techniques are also possible, 60 including macro operations, such as pressing Ctrl-A to A Bézier curve is defined as: designate all image objects on a page or pressing a mouse button with a cursor within a predefined distance of two or more image objects, that cause the separate inputs designat ing the vector graphic and the raster graphic to be generated. 65 FIG. 3 illustrates results of two predefined image distor tions. A first distortion shows the artwork image 200 as US 7,385,612 B1 7 8 Where P, are the control points, and B,"(t) are the Bézier FIG. 7 shows how these constraints translate into adjust basis functions, defined as: ments of the canonical points on the Bézier curve in a graph of response curves 550. Each of the four response curves R(t) shows the extent of motion of one of the canonical dY. points (denoted 0, 1, 2, 3) as a percentage of the dragged Bio- i (1-of position change of any arbitrary drag point along the curve C, where the te value of the selected point on the curve is on the horizontal axis, and the vertical axis shows the d is the degree of the curve (e.g., for a cubic curve, d=3) and amount by which the end points and the section dividing the curve has d+1 control points. In order to manipulate the 10 points C(i/3), i=0, 1, ... , 3, on the curve (i.e., the canonical curve, we wish to drag a particular point on the curve points) are affected by dragging the curve at point C(t) to C(t) to a point P. Such that the curve is adjusted in a Para natural way. Doing this involves first determining tre Consider the case of dragging the curve at the first end which is the parameter of the curve that corresponds to the point, t 0. Because (according to the third constraint point on the curve to be moved to the new location. 15 above) this is the same as dragging Po, the response curve Ro One method for directly manipulating curves developed for curve point C(O) has a value of 1 at that point, which means that the point at the first end of the curve (which is the by Bartels and Beatty is based on the Householder equa same as the target point) moves as much as the target point tions. As shown in FIG. 6, a point on a curve 500 is dragged, is dragged. The response curve R for curve point C(/3) has and the curve 500 moves accordingly. The control points P, a value of $27 at t=0, because the Bézier basis function are modified to create a new set of points according to: for the Zeroth control point for a 3 degree curve is 9/27 evaluated at t 0. Thus, when the section dividing point A = Parag - Cidrag) that is /3 of the way along the curve is moved by a unit, the first end point of the curve moves by 9/27 of a unit. In a r B(t 25 similar way, the response curve R for C(2/3) has a value (of P = P + A- (idrag) the basis function) of /27 at the first end point of the curve, and the response curve R for C(1) has a value (of the basis function) of Zero at the first end point as required by the constraint above that, for t<2/3, C(1) does not move. The circumflex is added to curves or points modified by 30 When a target point C(t) is dragged to P. first a dragging operation. While the Bartels and Beatty method compute the new positions of the end points and the section produces a smooth change in the curve, it also produces dividing points of the new curve C by applying the response artifacts. The length of the curve is often preserved, making curve to the original curve at: it difficult to flatten a curved shape. The entire curve segment is adjusted, making local changes difficult. And either the 35 end points are left fixed, producing a large change in the Then, the new control points for the curve are determined by overall curve near the fixed point when a small change is writing the calculation of the Bézier curve points C(i/3), made, or the endpoints are allowed to move, making it i=0, . . . , 3 in matrix form: difficult to constrain changes. Based on a consideration of how the curve should behave 40 when a target point is dragged, the following criteria work where B is the basis coefficient matrix: well (in the case of a cubic curve) in defining how certain canonical points on the curve should move. An equivalent set can be written for, e.g., the quadratic case. B(0) B(O) B:(0) B(0) For t>/3, C(0) does not move. 45 For t<%, C(1) does not move B(1/3) B(1/3) B(1/3) B(1/3) Dragging at t-0 behaves just like moving the Po B(2/3) B(2/3) B:(2/3) B3(2/3) control point. B(1) B(1) B:(1) B(1) Dragging at t=1 behaves just like moving the Ps 50 control point. Dragging at t=/3 or t, % maximally effects that To find the new control points, C(i/t) is substituted for C(i?t) point on the curve. in the equation above, which is then solved for the new Given end points A and B (endpoints A and B are two of control points: the canonical points, and also happen, in this case, to be 55 control points) and intermediate division points X and Y, the three sections of the Bézier are denoted AX (section 1), XY Because the basis coefficient matrix is constant, it can be (section 2) and YB (section 3). When dragging a target point pre-computed. located in section 1, end point B will not move, and A, X, and Y will each move based on the position of the target 60 The new curve rendered from the control points will move point between A and X. When dragging a target point located towards P. but may not actually reach it. In an interactive in section 2, end points A and B will not move, and X and program, this is often not noticeable, because the constraint Y will each move based on the position of the target point will be satisfied on the next iterations of the mouse tracking between X and Y. When dragging a target point within loop. However, it is possible to avoid this behavior in a section 3, end point A will not move, and X, Y, and B will 65 manner described later. each move based on the position of the target point between The curve dragging method is extended to non-cubic Y and B. curves by creating new response curves R, for points on the US 7,385,612 B1 curve at C(i/d), i=0,..., d. For example, in FIG. 8, response curves 600 for a quadratic curve are shown. When the curve dragged is part of a multiple-segment Bézier curve, the curve continuity should be maintained by keeping the con trol points on the adjoining segments co-linear with the ones on the dragged segment, and at the same distance. Extending the methods described above to surfaces pro vides an intuitive, natural way to edit the distortion surfaces The two new curves are the same shape as the original, and described previously. A distortion surface of the kind shown 10 L and Ro have the same value. To split a Bézier surface, the in FIG. 5 can be described as: same technique can be applied to each row of control points to split along the u parameter or each column of control points to split along the V parameter. FIGS. 12 and 13 illustrate subdivision to determine Sur X V.B. (u)B (v) 15 face parameters. To find the parameters u and V correspond ing to a user-selected point on a surface 750, the surface can be recursively subdivided. A test can be made to see if the user-selected point is within the bounding box of the control where u, v are the parameters of the surface, V, is the (n+1) points of each subdivided surface half. If it is, the subdivi sion continues recursively on that half. At each Subdivision, X (m+1) mesh of control points, and B,"(u)B,"(v) are the the parametric bounds of the subsurface is recorded. This Bézier basis functions of degree m, n as described above. process continues until the size of the bounding box 755 The goal of the free-form manipulation is to drag an arbi becomes approximately the same size as a single pixel of the trary point on the surface S(u, v) to Pie in a similar display device. At this point, the center of the parametric fashion to the curve manipulation method described above. 25 bounds of the Subsurface provide a good approximation to the parametric coordinates of the selected point. To drag a point on the surface, the parameters u, V. Applying the drag method to the Surface proceeds in two of the selected point on the surface should be determined, steps as shown in FIG. 14. First the drag is applied to the given the selected target point P, so that S(u, v) is iso-curve 800 at V. Then, each of the control points of approximately equal to P. As shown in FIGS. 9 and 10, to 30 this iso-curve 800, such as a control point 805, is used as a accomplish this, a distortion surface 630 can be first coarsely P. for modifying the curves formed by the columns of sampled in uniform steps in u and V. A typical sampling rate control points in the surface. Because the surface drag is of 4(n+1)x4 (m+1) can be used across the entire surface 630. developed as a series of curve drags, it is useful to express By connecting points adjacent to each other in the parameter the curve manipulation method described above for drag space, the resulting sample points form a mesh of quadri 35 ging the point C(t) to P as a function that generates a new set of curve control points from the original control laterals covering the Surface. Each quadrilateral is checked points P: to see if it contains P. by checking P against the line equations of the four borders of the quadrilateral (the lines P-Drag(Pt.A) 40 are formed by proceeding around the quadrilateral in a where consistent direction, e.g., clockwise). If P is on the same side of all four lines, then it lies inside. If P is found to lie A Parag-C(tarag) inside one of the quadrilaterals 650, then the process is he iso-curve at V C'' is formed by control points P' repeated, except this time the sampling is done over the 45 computed via: parameter range for just that quadrilateral 655 rather than the entire Surface. The indices of the sample points containing Pe. are used to determine the parameters u, V. While this process can be repeated indefinitely to improve accu k =X V, B' (vaag), j = 0 ... m racy, in practice two iterations are sufficient for interactive 50 manipulation. An alternative technique for determining the parameters A new set of control points for this iso-curve is computed of a drag point corresponding to a user-selected point on the with: surface is illustrated in FIGS. 11-13. FIG. 11 illustrates curve 55 P=Drag(P, ldrag Fdrag-S(drag 'drag) subdivision. A Bézier curve C(t) 700 can be subdivided into two joined curves L(t) 710 and R(t) 715 by computing two The control points for this new curve are used to drag the new sets of control points from the original set using the columns of the control points by applying them to the following: Surface control points: 60 for j=0 . . . m. P."=V, i=0 ... n. P-Drag(P", V. P.'-P') l, r.) V-P.", i=0 ... n. 65 The methods described above are useful for free-form manipulation of the distortion Surface to produce an arbi trary shape. However, there are many cases where it is useful US 7,385,612 B1 11 12 to produce distortions with a symmetric shape, where the manipulation is confined to a single parameter. Consider an inflate pre-defined distortion as shown in FIG. 15. In this d = Pirag - Sy(u, v) case, as the parameter is changed, the edges of the distortion move in or out. The inflate distortion is a bi-quadratic. The 5 amount of distortion is controlled by a single parameter, which moves the control points on the center of each edge towards (or away from) the center of the mesh. While this manipulation can be done with a traditional user interface tool Such as a slider control, another approach is to provide 10 a handle 850 oh the surface that the user drags to the desired shape. Subject to the constraints of the symmetry. This Solving for y gives: closely matches the free-form dragging behavior described above, and makes the Surface editing more precise, easier to learn, and easier to use. 15 To provide this control, a method is needed to convert the movement of the mouse (in one dimension) into a corre sponding movement amount for the symmetric movement of the control points in the distortion. This generally involves modifying the Surface equation (see above) to incorporate the desired symmetric movement, and then Solving that equation for the amount to move the control points. Collecting terms and Substituting u=/4, v=/2 gives: Continuing with the inflate distortion example, a handle is placed at S(/2.0). This handle is constrained to move only 25 vertically and should track the top of the distortion. To 256d- 27(Voo -- 2V1.0 -- V2.0) 27(Vol -- V2.1) simplify the problem, consider just the top iso-curve of the 9(Vo2 + V2.2) - (Vols + 2 V3 + V23) surface (v-0, Voo). As the handle is dragged, the control (36) point Vo should move so that the point on the curve S(/2.0) tracks the mouse as it moves. To find out how far Vol should move so that the handle tracks the curve, consider the 30 This is the equation relating the mouse movement d to the formulation of the curve (with u=/2, and ignoring V for the control point movement y. moment): Other direct controls for pre-defined distortions can fol low the same general strategy of relating the mouse move 1 1 35 ment to a point on the Surface. Distortions can also have a Vol + Vo2 multiple handles (e.g., one for a vertical stretch and another for horizontal) with the implementation strategy being applied to each handle. With Voo and Voe fixed, we can see that Vo moves twice As mentioned, the method described above for free-form as far as the point on the curve S(/2.0). Thus, when the 40 manipulation of the distortion of the Surface using a point on handle at S(/2.0) is dragged, the four control points on the the Surface does not adjust a curve Sufficiently in a single edges are moved twice the distance the handle was moved, iteration to move a target point by an amount indicated by giving the illusion that the handle is fixed onto the edge of dragging. However, the method can be enhanced to move the distortion. the point by the required amount in a single step. Below we 45 discuss how this is done for the cubic case. The same As a second, more complex example, consider the wave approach can be applied to the quadratic case. distortion shown in FIG. 16. When the single parameter of In the basic method the desired displacements of four this distortion is changed, a left interior control point 900 canonical points on the curve, at t=0, /3, 2/3, and 1, are moves up, and a right interior control point 905 moves down specified by the response curves for four specific choices of in equal amounts. If y is the wave amount, then the distortion 50 the dragged point, t 0, 4, 24, and 1. The behavior of the is determined by: four on-curve canonical points for other values of t was V 1-c-ty derived by linear interpolation as shown by the linear segments of the response curves. Although linear functions are easy to compute, the result of the choice of linear 55 interpolation is that the dragged point may not be displaced where c is the vertical coordinate of the center of the by the correct amount. distortion's bounding box. (V, refers to just they coordi Among the possible methods for interpolating behavior nate V. because the X coordinates are not affected they may for intermediate values of t a set of scaled response be ignored). To give maximum control, the handle should curves, S derived in a manner explained below, works well. start out at S(4./2). As this handle is dragged up or down, 60 Starting with the response curves, R., of the basic method, we want it to track the surface, with V. moving in the same we find the resulting displacement of the Bézier control direction as the handle and V moving in the opposite points, and of the point, t. The point, t will have direction. To do this, we need to find the value correspond moved a fraction, f, of the required distance. We therefore ing to the difference between the original handle location at scale the displacements of the Bézier control points by the S(4./2) and P. If we call this difference d, and the 65 factor 1/f, The linear nature of the whole system ensures that corresponding distance the control points V and V, each this will result in the point, t being displaced by the move y, we find d is related to y Via the equation: COrrect amount. US 7,385,612 B1 13 14 By making different choices of the response curves we From equation 3, the change in position of the control can generate different corrected response curves. These points required to achieve a change in position of the four different choices will affect the relative amounts by which canonical points on the curve is the various control points move. The correction may be implemented by literally coding AP=ACV.A.M (EQ 5) the correction operations as described. Alternatively, the where appropriate correction can be derived algebraically, as explained below. We use the following scaled response curves notation 11 9 (there has been some reuse of variable names that were 10 - 2 1 1 1 1 previously used for other purposes in the earlier discussion): O 45 27 O 1 2 1 C(t)—Position of point on Bezier curve and t A = ° and MI - O 9 18 27 O O 1 1 AC(t)—Change in position of point on Bezier curve at t 2 - 3. ACV-ACoACACAC, Vector of curve points 9 9 O O O 1 at t=0, /3, 2/3, 1 15 O 1 - - P, i=0 . . .3—Position of Bezier control point i 2 2 AP, i=0 ... 3 Change in position of Bezier control point The changes in position of the canonical curve points, P=(PPPP) Row vector of AP, ACV, is determined from the response curves AP=(APAP APAP) Row vector of AP, R(t), i=0 . . . 3 Value of Response Curve att ACV-FAL.R(tdrag) (EQ 6) R(tie) -(Ro (tara)Ri (tie)R2 (tara)Rs (t))—Row WC Substituting into equation 5 tor of R,(t) S(t), i=0 . . .3—Value of Scaled Response Curve at AP=ALR(t).A.M. (EQ 7) drag 25 This is essentially the formulation used in the basic S(t.)-(So (tie)S 1(tarag)S2(tarag) S. (t))—Row WC method. Now we examine how a point, t, on the curve is tor of S,(t) affected. M Bezier coefficient matrix Combining equation 2 and equation 7 T(t)=(1 t tit) Row vector of powers oft (EQ 8) 30 T(t)' Transpose of T(t) This shows that the displacement, AC(t), of the point t on A Power matrix the curve is a scale factor, R(t). AT(t)' times the AL. Vector through which pointing Locator is dragged movement, AL, of the locator. In particular B' Vector of Bezier basis functions of degreen The position of a point, C(t), on a Bézier curve is given 35 by To ensure that the C(t) stays with the locator we require AC(t)=AL. We therefore need to modify the scale C(t)=PM. T(t) (EQ 1) factor by 1/(R(t).A.T(t)') This can be achieved by where Scaling the response curves to give the scaled response 40 curves, S(t), where 1 -3 3 - 1 S(taag)-R(taag) (R(taag).A.T(a)) (EQ 10) O 3 6 3 M = The revised expression for computing changes, AP, in the O O 3 -3 positions of the Bézier control points for a given displace O O O 1 45 ment, AL, of the locator at point to on the curve is, from equation 7: The change, AC(t), in position of a point for a given AP=AL.S(t).A.M. (EQ 11) change, AP, in the positions of the control points is The choice of R(t) used in the basic method is shown (EQ 2) in FIG. 17 as a graph of response curves 1000. The corre 50 sponding scaled response curves are shown in FIG. 18 as a Applying this to the four canonical points, t-0, /3, 2/3, 1 graph of response curves 1020. AC(O)AC(1/3)AC(2/3)AC(1)=ACV=APM.A (EQ 3) While the curves of FIG. 17 work well in an interactive environment, the discontinuity of slope in S (t) at where 55 ti-/3, and similarly in S (tie), may be considered undesirable. This can be fixed by making a different choice of R (t) and R2 (t). For example, FIG. 19 shows a 1 1 1 1 (EQ 4) graph of response curves 1040 for the case of a two-degree O 1 2 1 curve; response curves in which R(t) and R2 (t) are 3 3 60 2” degree polynomials interpolating the same boundary A = o 1 conditions: 9 9 O 1 8 1 27 27 R2(tarag) R1 (1-tarag) 65 is obtained by substituting the values t—0, /3, 2/3, 1 into the The corresponding scaled response curves are shown in FIG. column T(t)', expressed as a matrix. 20 as a graph of response curves 1060. US 7,385,612 B1 15 16 The treatment given here may be applied directly to be part of image artwork or an entire image itself. The raster Bézier curves of degree other than three. Differences include graphic is distortable separate from the vector graphic. The the size of the various vectors and matrices, the coefficients raster graphic need not have a predefined mapping associa in the Bézier coefficient matrix, M, and the boundary con tion with a vector graphic, as in the case of texture mapping ditions for the response curves. For degree 2 curves (qua in three-dimensional modeling. dratic), the boundary conditions require that for t drag-/2. The coordinates of the vector graphic are converted into the end of the curve, C(1), does not move, and similarly for a parameter space of the distortion region at 1105. The image t drag>/2 with respect to end C(O). See FIG. 8. distortion description is composed with the vector graphic at The linear case is the same as a linear (degree one) 1110. This composition of functions allows curve informa example proposed by Bartels and Beatty. 10 tion to be retained as curve information and maintains a When the end, t drag 0, of a curve is moved, the treatment resolution independent aspect of the vector graphic. The given here causes the curve to distort as if only the control result of this composition can be a function having an order point, Po, were moved—all other control points remain higher than acceptable. Thus, the resulting vector graphic fixed. This is not always desirable. For example, the curve can have its order reduced at 1115. Additional details regard may be part of a composite curve, consisting of many Bézier 15 ing techniques for distorting vector graphics are described segments connected end-to-end, with slope continuity below in connection with FIG. 22. between segments. In this case, moving the end of a curve Before the raster graphic is distorted, a check is made to segment should move not only the control point, Po, but also determine if the distortion region is larger than the raster the adjacent control point P. This will have the effect of graphic at 1120. This will be the case when one of the vector maintaining the slope of the end of the curve. If the same graphics to be distorted is not entirely contained in the raster treatment is given to the adjoining curve segment, whose graphic, and also when multiple raster graphics are to be end is also being dragged at the same time, then continuity distorted. If the distortion region is larger, the image distor of slope is maintained. This behavior can be incorporated tion description is subdivided at 1125. After subdivision of into the present scheme by modifying the boundary condi the image distortion description, or if the distortion region is tions on the response curves, as follows: 25 not larger than the raster graphic, the image distortion description is used as a control structure governing resam pling of the raster graphic at 1130. Additional details regard ing techniques for distorting raster graphics are described below in connection with FIGS. 23-26. This gives rise to the following response curves: 30 FIG. 22 illustrates a technique for distorting vector graph ics. In this example, an undistorted tensor product Bézier Ro(tar)—if (tig/3) then (1-3th) else (0) mesh 1200 is first placed over the distortion region (e.g., the R(t)=if (tg'/3) then (20/27+t 21/27) else bounding box) of the objects to be distorted. The coordinates (37/27-130/27) of each point defining the vector artwork (e.g., endpoints of 35 line segments, or control points of curves) are converted into R2 (tarag) R1 (1tarag) the local parameter space of the distortion region. As shown in FIG. 22, a line 1205 and a cubic Bézier curve 1210 are R2 (tarag) Ro(litarag) distorted. The distortion region has a parameter space from 0.1 in both directions, so the endpoints of the line 1205 can Corresponding scaled response curves can be derived, from 40 be specified for the distortion in coordinates relative to the these expressions using equation 10. overall size of the distortion (e.g., values between 0.0 and In any given implementation, both forms of boundary 1.0). conditions might be used. The original form may be appro A simple method for distorting the artwork is to sample priate for ends of composite curves, or for moving interior curve or line segments at many Small steps along their segment ends where slope continuity is not required. This 45 length. The resulting sample points are converted into the new set of response curves is more appropriate for interior parameter space of the distortion region, and used as param segment ends where slope continuity is required. The eters for the distortion. For example, if C(t) is a function remaining question concerns maintenance of slope continu representing the line or curve, and D(u,v) is a function ity between joined Bézier segments when t drag>0, because representing the distortion, then points on the distorted curve the adjacent segment will not normally be affected. Two 50 are found by evaluating D(C.(t),C(t)). This technique can strategies present themselves. Both involve repositioning the be simple to implement, but generally generates a good deal adjacent slope control point of the adjacent segment so that of data and also discretizes the curves, making them no it is co-linear with the end point and slope point of the longer resolution independent. current segment. One strategy would maintain the length of An alternative approach involves treating the definition of the adjacent slope vector. The other would maintain the ratio 55 a curve or line as a mathematical function C(t), and treating of the magnitudes of the slope Vectors at the adjoining ends. the image distortion description as another function D(u,v), Additional details regarding the above-described example as before. The two functions are then composed: H=DoC. techniques for receiving an image distortion description can Example techniques for composing Such functions are be found in U.S. Utility patent application Ser. No. 09/458, described in DeRose, Goldman, Hagen & Mann, “Func 917, filed Dec. 10, 1999 and entitled “MANIPULATION OF 60 tional Composition Algorthims via Blossoming ACM CURVES AND SURFACES', which is hereby incorporated Transactions on Graphics, Vol. 12 No. 2, April 1993, pp by reference. 113-135. FIG. 21 is a flowchart showing a process of distorting a The function H(t) resulting from the composition is a raster graphic and a vector graphic. An image distortion higher order curve that represents the distorted original. description is received at 1100. The image distortion 65 Since typical output device systems are currently unable to description is to be applied to an image portion including a render high order curves directly, additional control points vector graphic and a raster graphic. The image portion can can be created as needed to match the original shape, US 7,385,612 B1 17 18 creating new distorted vector graphics, such as a warped line parameter space, the vertices of each shard are arranged in 1215 (which can be cubic Bézier curve) and a warped cubic a clockwise order, as with a shard 1310. If the vertices Bézier curve 1220. Example techniques for reducing the appear in counter-clockwise order when they are finally order of such resulting functions are described in Watkins & rendered, like those of a shard 1315, then the shard is Worsey, “Degree Reduction of Bézier Curves”. Computer 5 backfacing. The orientation can be determined by examining Aided Design, Vol. 20 No. 7, September 1988, p. 398. the sign of the cross product of the vectors: (Po-p)x(p-pi), FIGS. 23 and 24 illustrate a technique for distorting a here p, refers to the distortion image coordinates of the shard raster graphic. Distortion of a raster graphic is essentially a vertices. resampling problem. Because the distorted image is no Before the shards are rendered, they can be first sorted so longer aligned with the pixel grid, the underlying abstract 10 that the folded over shards are displayed last, if it is desired image is sampled on the new pixel grid of the distorted to display the folded over portion, or first if the folded over image. One way to do this is to tessellate the image portion should be obscured. Additionally, the shards can be distortion description, and then resample the raster graphic sorted within the backfacing and frontfacing by their dis based on the tessellated image distortion description. tance in the original parameter space from the origin 0-u, A specified distortion 1240 of a raster graphic 1245 can be 15 v=0. This can help disambiguate situations where two cor recursively subdivided. This recursively splits the distortion ners fold over on each other. space into adjoining polygons (e.g., rectangles, triangles). When the shards are rendered, a resampling process can The Subdivision process continues until the polygons are be used to render the pixels from the original image through sufficiently small that they form a close approximation of the the specified distortion into the distorted image (e.g., the distorted shape of the frame of the original image (e.g., the device pixels). This can be done by interpolating the values polygons have less than a pixel units distortion). As the between the Source image pixels to compute the values of Subdivision progresses, the parametric coordinates of the the destination pixels. An example Such a resampling pro vertices of the polygons are recorded. These can be used to cess can be found in Williams, “Pyramidal Parametrics'. map the image pixels from the original shape onto the Proc. of SIGGRAPH '83. However, this process may not be distorted shape. 25 possible at the edge of the image, because there are no values The Subdivision can occur in two stages. First, the dis to interpolate to beyond the edge of the image. tortion 1240 can be subdivided into rectangles, until each of FIG. 26 illustrates generated edge shards for handling the rectangles holds less than a pixel units distortion. image edges. After initial shards 1340 are generated, a group Example techniques that can be used to recursively Subdi of extra shard 1345 can be created by identifying the initial vide the specified distortion 1240 are described in Peterson, 30 shards 1340 with edge boundaries (e.g., shards with edges “Tessellation of NURB Surfaces, in Graphics Gems IV. P. having parametric values u=0, u=1, v=0 or v=1). Device Heckbert (ed), 1994 Academic Press, p. 286. The techniques space (x,y) coordinates can be extended at least two pixels described in the Peterson paper can be simplified because away from the interior of the surface perpendicular to the the coordinates here are two-dimensional instead of three edge. The source parameter values of the extra shards 1345 dimensional, and because a simple Bézier mesh can be used 35 have vertices slightly beyond the 0.10.1 parameter range instead of a Non-Unform Rational B-Spline one. Once this of the image. As these edge shards 1345 are rendered, pixel Subdivision process is complete, the resulting polygons can values outside the 0.1 range can be assigned an empty be further split into triangles or shards. These shards are then value. The rendering process can automatically interpolate scan converted back into pixels. Various example techniques to this empty value, creating a Smooth edge for the distorted for Scan converting shards into pixels are described in 40 image. Heckbert, “Fundamentals of Texture Mapping and Image FIG. 27 illustrates subdividing a distortion region. As Warping, MS Thesis, UC Berkeley, UC Berkeley report discussed above, the image distortion description is applied UCB/CSD 89/516, and also in Heckbert, P., “Generic Con to the distortion region, which can be the bounding box for vex Polygon Scan Conversion and Clipping, in (A. Glass all the image objects to be distorted. When the raster artwork ner, Ed.) Graphics Gems, Academic Press, 1990, p. 84. 45 completely encloses the vector artwork, the tessellation The specified distortion of the raster graphic 1245 can process (e.g., recursive Subdivision, shard creation and Scan cause the image to fold over itself as shown in FIG. 24. This conversion) is applied to the raster artwork over the full type of distortion can be allowed as well. For example, the parameter range of the distortion area. However, if the raster triangles generated by the Subdivision described above can artwork does not fill the entire distortion area (e.g., it does be generated in a relatively arbitrary order (e.g., the depth 50 not completely contain the vector artwork, or there are first order of the subdivision process), which may not multiple raster images in the artwork to be distorted), then correspond to their layout in the distorted image. When the the parameter space of the tessellation for the raster is specified distortion causes the image to fold over itself an adjusted. analysis of the generated vertices can be used to allow for Adjusting the parameter space involves performing an this folding over. 55 initial subdivision of a distortion region 1360 to identify a FIG. 25 is a block diagram illustrating example subdivi corresponding Subdivision of the image distortion descrip sion of a specified distortion and ordering of vertices. A final tion. This initial subdivision of the image distortion descrip subdivided piece 1300 has corners with two sets of coordi tion is done to trim the parameter space before the tessel nates: (1) original image coordinates, typically referred to as lation described above is performed. For example, if a raster u and V, and (2) distortion image coordinates (e.g., the X and 60 graphic 1365 in the distortion region 1360 occupies a y coordinates of the display device). When the final subdi rectangle defined by points a 1370 and b 1380, the rectangle vided piece 1300 is split into shards 1305, the vertices of the coordinates can be converted to the local coordinates of the shards can be ordered in terms of the orientation in the distortion space (usually 0 . . . 1), and then four initial original image space, as shown in FIG. 25. A shard can be subdivisions can be performed. determined as either forward facing or backward facing (i.e., 65 The first subdivision splits the image distortion descrip folded over) by examining the orientation of the shard's tion vertically at a and discards the left portion 1372. The vertices as they appear in the final display. In the original second Subdivision splits the image distortion description US 7,385,612 B1 19 20 horizontally at a, and discards the top portion 1374. The tion). Generally, a computer will include one or more mass third Subdivision splits the image distortion description at b, storage devices for storing data files; such devices include and discards the right portion 1382. The last subdivision magnetic disks, such as internal hard disks and removable splits the image distortion description at b, and discards the disks; magneto-optical disks; and optical disks. Storage bottom portion 1384. Because the subdivisions are done in devices suitable for tangibly embodying software program the parameter space of the image distortion, the original instructions and data include all forms of non-volatile coordinates of b are modified to b because the parameter memory, including by way of example semiconductor space is shifted after the first two subdivisions. This can be memory devices, such as EPROM (electrically program accomplished with: mable read-only memory), EEPROM (electrically erasable 10 programmable read-only memory), and flash memory devices; magnetic disks Such as internal hard disks and 1 removable disks; magneto-optical disks; and CD-ROM it 1 - a disks. Any of the foregoing can be Supplemented by, or b, = mb + (1 - m) incorporated in, ASICs (application-specific integrated cir 15 cuits). To provide for interaction with a user, the invention can be implemented on a computer system having a display A similar computation can be performed to obtain b, device Such as a monitor or LCD (liquid crystal display) FIG. 28 illustrates a distorted page 1400 from an elec screen for displaying information to the user and a keyboard tronic document. An image portion that is distorted using the and a pointing device Such as a mouse or a trackball by techniques described above can be a full page image. Such which the user can provide input to the computer system. as the distorted page 1400, which includes multiple image The computer system can be programmed to provide a objects of various types such as raster, text and line artwork. graphical user interface through which computer programs The full page image can be stored in a final format electronic interact with users. document, which is an electronic document describing one 25 The invention has been described in terms of particular or more virtual pages having a predetermined final format. embodiments. Other embodiments are within the scope of The predetermined final format defines a specific visual the following claims. The logic flows depicted in FIGS. 1 appearance for the electronic document when displayed or and 21 do not require the particular order shown, or sequen printed. A final format document generally provides a tial order, to achieve desirable results. For example, receiv device-independent format for publishing and/or distribut 30 ing the image distortion description need not be an operation ing electronic documents. Thus, a final format document is performed separate from the application of the image dis conventionally a file that lies between a layout program and tortion description to vector and raster artwork. The image the typical raster image processors, which drive traditional distortion description can be applied to both types of artwork printing apparatus. as the distortion description is being received, or the image An example of a final format is the Portable Document 35 distortion description can be applied only partially as it Format (PDF) provided by Adobe Systems Incorporated of being received (e.g., vector graphics can be distorted on a San Jose, Calif. Example software for creating and reading screen as the distortion description is being input, while PDF documents is the Acrobat(R) software, also of Adobe raster graphics can be distorted once a complete distortion Systems Incorporated The Adobe R. Acrobat(R) software is description has been input; the general effect of the distor based on Adobe(R) PostScript(R) technology which describes 40 tion description on the raster graphic can be displayed as the formatted pages of a document in a device-independent input is received by using a generated graphic, which can fashion. overlay the raster graphic). In certain implementations, The invention can be implemented in digital electronic multitasking and parallel processing may be preferable. circuitry, or in computer hardware, firmware, Software, or in combinations of them. Apparatus of the invention can be 45 What is claimed is: implemented in a software product (e.g., a computer pro 1. A computer-implemented method comprising: gram product) tangibly embodied in a machine-readable receiving an image distortion description to be applied to storage device for execution by a programmable processor; an image portion including a vector graphic and a raster and processing operations of the invention can be performed graphic displayed on the screen of a display device, the by a programmable processor executing a program of 50 raster graphic and the vector graphic being separately instructions to perform functions of the invention by oper distortable and distinct image objects; ating on input data and generating output. The invention can applying the image distortion description to the vector be implemented advantageously in one or more software graphic to produce a distorted vector graphic and programs that are executable on a programmable system independently applying the image distortion descrip including at least one programmable processor coupled to 55 tion to the raster graphic to produce a distorted raster receive data and instructions from, and to transmit data and graphic, the applications being independent in the sense instructions to, a data storage system, at least one input that no result of either application is used as input to the device, and at least one output device. Each software pro other, the distorted vector graphic and the distorted gram can be implemented in a high-level procedural or raster graphic together forming a distorted image por object-oriented programming language, or in assembly or 60 tion; machine language if desired; and in any case, the language wherein independently applying the image distortion can be a compiled or interpreted language. Suitable proces description to the raster graphic comprises using the sors include, by way of example, both general and special image distortion description as a control structure gov purpose microprocessors. Generally, a processor will erning resampling of the raster graphic; receive instructions and data from a read-only memory, a 65 wherein receiving the image distortion description com random access memory and/or a machine-readable signal prises: receiving a first input designating the raster (e.g., a digital signal received through a network connec graphic to be distorted, receiving a second input des US 7,385,612 B1 21 22 ignating the vector graphic to be distorted, and receiv applying the image distortion description to the vector ing a third input defining the image distortion descrip graphic to produce a distorted vector graphic and tion; and independently applying the image distortion descrip wherein the designated raster graphic and the designated tion to the raster graphic to produce a distorted raster vector graphic define a distortion region, the method graphic, the applications being independent in the sense further comprising: that no result of either application is used as input to the converting coordinates of the vector graphic into a param other, the distorted vector graphic and the distorted eter space of the distortion region; and raster graphic together forming a distorted image por if the distortion region is larger than the raster graphic, tion; Subdividing the image distortion description with 10 wherein independently applying the image distortion respect to the raster graphic before applying the image description to the raster graphic comprises using the distortion description to the raster graphic; image distortion description as a control structure gov wherein independently applying the image distortion erning resampling of the raster graphic; description to the raster graphic further comprises: wherein receiving the image distortion description com tessellating the image distortion description into poly 15 prises: receiving a first input designating the raster gons; checking for any backward facing ones of the graphic to be distorted, receiving a second input des polygons; and resampling the raster graphic based on ignating the vector graphic to be distorted, and receiv the tessellated image distortion description. ing a third input defining the image distortion descrip 2. The method of claim 1, wherein applying the image tion; and distortion description to the vector graphic comprises com wherein the designated raster graphic and the designated posing the image distortion description with the vector vector graphic define a distortion region, the operations graphic to retain curve information as curve information and further comprising: to maintain a resolution independent aspect of the vector converting coordinates of the vector graphic into a param graphic; and wherein resampling the raster graphic com eter space of the distortion region; and prises extending edges of the raster graphic to provide data 25 if the distortion region is larger than the raster graphic, values for interpolation. Subdividing the image distortion description with 3. The method of claim 1, wherein the raster graphic does respect to the raster graphic before applying the image not have a mapping association with another image object in distortion description to the raster graphic; the image portion. wherein independently applying the image distortion 4. The method of claim 1, wherein the image portion 30 description to the raster graphic further comprises: comprises a full page image stored in a final format elec tessellating the image distortion description into poly tronic document. gons; checking for any backward facing ones of the 5. The method of claim 1, wherein receiving the third polygons; and resampling the raster graphic based on input defining the image distortion description comprises the tessellated image distortion description. receiving inputs specifying a configuration of an adjustable 35 11. The software product of claim 10, wherein applying predefined distortion type. the image distortion description to the vector graphic com 6. The method of claim 1, wherein receiving the first input prises composing the image distortion description with the designating the raster graphic to be distorted comprises vector graphic to retain curve information as curve infor receiving inputs defining a selection box to cover a portion mation and to maintain a resolution independent aspect of 40 the vector graphic; and wherein resampling the raster of the raster graphic, and wherein receiving the second input graphic comprises extending edges of the raster graphic to designating the vector graphic to be distorted comprises provide data values for interpolation. receiving inputs defining the selection box to cover a portion 12. The software product of claim 10, wherein receiving of the vector graphic. the first input designating the raster graphic to be distorted 7. The method of claim 1, wherein receiving the third 45 comprises receiving inputs defining a selection box to cover input defining the image distortion description comprises a portion of the raster graphic, and wherein receiving the receiving inputs specifying movements of a cursor across second input designating the vector graphic to be distorted the distortion region. comprises receiving inputs defining the selection box to 8. The method of claim 1, wherein the image distortion cover a portion of the vector graphic. description comprises a tensor-product Bézier mesh, and 50 13. The software product of claim 10, wherein receiving wherein the vector graphic comprises a cubic Bézier curve. the first input designating the raster graphic to be distorted 9. The method of claim 1, wherein receiving the first input comprises receiving a first input generated by a macro designating the raster graphic to be distorted comprises operation, and wherein receiving the second input designat receiving a first input generated by a macro operation, and ing the vector graphic to be distorted comprises receiving a wherein receiving the second input designating the vector 55 second input generated by the macro operation. graphic to be distorted comprises receiving a second input 14. The software product of claim 10, wherein receiving generated by the macro operation. the third input defining the image distortion description 10. A Software product, tangibly stored on a computer comprises receiving inputs specifying movements of a cur readable medium, the Software product comprising instruc sor across the distortion region. tions operable to cause a programmable processor to per 60 15. The software product of claim 10, wherein receiving form operations comprising: the third input defining the image distortion description receiving through a single interface tool in a graphical comprises receiving inputs specifying a configuration of an user interface an image distortion description to be adjustable predefined distortion type. applied to an image portion including a vector graphic 16. The software product of claim 10, wherein the image and a raster graphic displayed on the screen of a display 65 distortion description comprises a tensor-product Bézier device, the raster graphic and the vector graphic being mesh, and wherein the vector graphic comprises a cubic separately distortable and distinct image objects; Bézier curve. US 7,385,612 B1 23 24 17. The software product of claim 10, wherein the raster prising means for resampling the raster graphic based graphic does not have a mapping association with another on the tessellated image distortion description. image object in the image portion. 20. The system of claim 19, wherein the means for 18. The software product of claim 10, wherein the image applying the image distortion description to the vector portion comprises a full page image stored in a final format graphic comprises means for composing the mathematical electronic document. control structure with the vector graphic to retain curve 19. A system for illustrating two dimensional images, the information as curve information and to maintain a resolu system comprising: tion independent aspect of the vector graphic; and wherein means for receiving an image distortion description to be the means for resampling comprises means for extending applied to an image including a vector graphic and a 10 edges of the raster graphic to provide data values for raster graphic displayed on the screen of a display interpolation. device, the raster graphic and the vector graphic being 21. The system of claim 19, wherein the receiving means separately distortable and distinct image objects; and comprises free-form means for receiving arbitrary user means for applying the image distortion description to the designed image distortions. vector graphic to produce a distorted vector graphic and 15 22. The system of claim 19, wherein the raster graphic independently applying the image distortion descrip does not have a mapping association with another image tion to the raster graphic to produce a distorted raster object in the distorted image. graphic, the applications being independent in the sense 23. The system of claim 19, wherein the distorted image that no result of either application is used as input to the comprises a full page image stored in a final format elec other, wherein the distorted vector graphic and the tronic document. distorted raster graphic correspond to a consistent dis 24. The system of claim 19, wherein the means for tortion as defined by the image distortion description receiving the third input defining the image distortion across a distorted image including the distorted vector description comprises means for receiving inputs specifying graphic and the distorted raster graphic; a configuration of an adjustable predefined distortion type. wherein the means for applying the image distortion 25 25. The system of claim 19, wherein the means for description to the raster graphic comprises means for receiving the first input designating the raster graphic to be resampling the raster graphic using a mathematical distorted comprises means for receiving inputs defining a control structure; selection box to cover a portion of the raster graphic, and wherein the means for receiving the image distortion wherein the means for receiving the second input designat description comprises means for receiving a first input 30 ing the vector graphic to be distorted comprises means for designating the raster graphic to be distorted, a second receiving inputs defining the selection box to cover a portion input designating the vector graphic to be distorted, and of the vector graphic. a third input defining the image distortion description; 26. The system of claim 19, wherein the means for and receiving the first input designating the raster graphic to be wherein the designated raster graphic and the designated 35 distorted comprises means for receiving a first input gener vector graphic define a distortion region, the system ated by a macro operation, and wherein the means for further comprising: receiving the second input designating the vector graphic to means for converting coordinates of the vector graphic be distorted comprises means for receiving a second input into a parameter space of the distortion region; and generated by the macro operation. means for Subdividing the image distortion description 40 27. The system of claim 19, wherein the means for with respect to the raster graphic, if the distortion receiving the third input defining the image distortion region is larger than the raster graphic, before applying description comprises means for receiving inputs specifying the image distortion description to the raster graphic; movements of a cursor across the distortion region. wherein the means for applying the image distortion 28. The system of claim 19, wherein the image distortion description to the raster graphic comprises: means for 45 description comprises a tensor-product Bézier mesh, and tessellating the image distortion description into poly wherein the vector graphic comprises a cubic Bézier curve. gons; means for checking for any backward facing ones of the polygons; and the means for resampling com k k k k k UNITED STATES PATENT AND TRADEMARK OFFICE CERTIFICATE OF CORRECTION

PATENT NO. : 7,385,612 B1 Page 1 of 1 APPLICATION NO. : 10/160574 DATED : June 10, 2008 INVENTOR(S) : John W. Peterson

It is certified that error appears in the above-identified patent and that said Letters Patent is hereby corrected as shown below:

Claim 10: column 21, line 59: insert-storage--, after “readable'

Signed and Sealed this Fourteenth Day of April, 2009 Al (O-e-

JOHN DOLL Acting Director of the United States Patent and Trademark Office