A Computer Analysis of the Mirror in Hans Memling's Virgin and Child

Chapter 1

A computer analysis of the mirror in Hans Memling’s Virgin and Child and Maarten van Nieuwenhove

SILVIO SAVARESE Electrical and Computer Engineering University of Michigan Ann Arbor, MI Email: [email protected]

DAVID G.STORK Ricoh Innovations, 2882 Sand Hill Road Suite 115, Menlo Park CA USA and Stanford University, Stanford CA 94305 USA Email:[email protected]

ANDREY DEL POZO Computer Science Univesity of Illinois at Urbana-Champaign Urbana, IL Email:[email protected]

RON SPRONK Department of Art, Queen’s University, Kingston ON, K7L 3N6, Canada Email:[email protected] 2 Digital Imaging for Cultural Heritage Preservation

1.1 Introduction

Recently, a number of scholars trained in computer vision, pattern recognition, image processing, computer graphics and art history have developed methods for addressing problems in the history and interpretation of fine art [1, 2]. These methods, when paired with art historical and contextual knowledge, have shown modest but promising successes in authenticating works through statistical analysis of brush strokes [3] or patterns of dripped paint [4, 5], revealing the studio condition and working methods of realist artists [6], testing for artists’ use of optical aids, [7, 8] and de-warping of images depicted in convex mirrors to provide new views into artists’ studios [9]. These methods do not replace the expertise of the trained art scholar but rather are tools to enhance such expertise, much as a microscope expands the scholarly power of a biologist. In this book chapter we are aimed at using computer image analysis of depicted convex mirrors to shed light on a number of questions in the art history of the early Renaissance. Convex mirrors begin to appear prominently in the paintings of the early Northern Renaissance, as sym- bols of wealth, as metaphors for an all-seeing Deity, as objects of inherent visual interest and as showpieces for artists’ technical mastery. Prominent examples from the north and, somewhat later, Italy include: Jan van Eyck, Portrait of Giovanni Arnolfini and his wife (1434); Robert Campin, Heinrich von Werl and St. John the Baptist (1438); Petrus Christus, Saint Eligius (1449); Hieronymus Bosch, The seven deadly sins (c. 1480); Hans Memling, Vanity (c. 1485) and Maarten van Nieuwenhove diptych (1487); Quentin Metsys, The money lender and his wife (1514); Parmigianino, Self-portrait in a convex mirror (c. 1524); Laux Furtenagel, Hans Burgkmair and his wife (1527); Pieter Bruegel the Elder, Prudence (1559); Caravaggio, Martha and Mary Magdalene (c. 1598); Plane mirrors have appeared in European painting as well, of course, but there seem to be few if any clear depictions of concave mirrors around this time. Recent computer image analysis have provided tools for addressing a number of unresolved questions in the art history of the Renaissance. Criminisi and his colleagues modeled the optics of convex mirrors to dewarp the mirror images in Jan van Eyck’s Portrait of Giovanni Arnolfini and his wife and Robert Campin’s Heinrich von Werl and St. John the Baptist [9]. They could then, in software, perform the inverse optical transform to correct or “dewarp” the image depicted in the mirror. This inverse transform depends upon the single unknown parameter R, the radius of curvature or “bulginess” of the mirror (Figure 1.4). More accurately speaking, in their case the transform depends upon the unknown ratio of the facial radius to the radius of curvature. They adjusted the computer model mirror’s radius of curvature such that door jams, windows, and so on in the reconstructed image were as rectilinear as possible. In this way they revealed— for the first time in nearly 600 years—new views into the respective tableau rooms. These authors found, nevertheless, that even after best overall dewarping in Arnolfini, there remained slight distortions which could be attributed to the artist changing his viewing position, or that the mirror shape differed from the ideal, or that the artist’s hand was not uniformly true while copying the reflected image. They found the reconstructed space in Heinrich von Werl conformed sufficiently well to the laws of perspective that they could use rigorous visual metrology to estimate the relative heights of figures in the mirror space [10]. They also used the visual information recovered from the de-warped images and a variety of other clues to attribute again this painting to Robert Campin. Stork extended such analysis to address David Hockney and Charles Falco’s hypothesis that van Eyck used the very mirror he depicted within the painting to execute the rest of the painting by tracing a projected image [11]. That is, they hypothesized that van Eyck turned the convex mirror around and used it as a concave projection mirror, to project an image of the tableau onto the oak panel support, trace it, and then fill it in with paint. Stork estimated the overall size of the mirror in Arnolfini; then together with the relative mirror curvature provided by Criminisi et al., he computed the absolute radius of curvature of this convex mirror, R. He then computed this mirror’s focal length, fmir, which in the paraxial ray approximation is simply fmir = R/2 [12]. He also created a Computer Graphics model of the tableau to estimate the location and focal length of a putative optical projector for this painting, fproj. He found these two focal lengths differed significantly, and thus he rejected the suggestion that van Eyck might have used the depicted mirror to build a projector for executing this painting. Stork also estimated the smallest blur spot of such a A computer analysis of the mirror in Hans Memling’s Virgin and Child and Maarten van Nieuwenhove 3

Figure 1.1: Hans Memling, Virgin and Child and Maarten van Nieuwenhove (1487), detail, 22 × 44 cm. Notice that the mirror reflects the donor in the same room, silhouetted against the windows unifying the two panels. Image courtesy President and Fellows, Harvard College. mirror and showed that this spot was too large—that is, the image too blurry—for an artist to trace the fine detail such is found in this painting [13, 7]. Computer Graphics models of tableaus of paintings have been used to answer art historical questions. Johnson et al. [14] built a model of Vermeer’s Girl with a pearl earring, and then adjusted the position of the model of the lighting source until the rendered model matched the painting most closely. In this way, they estimated the direction of the illumination. Their main goal, though, was to integrate estimation from a number of disparate sources such as the cast shadows, lightness along occluding contours, and lightness throughout an approximate facial model [14]. Computer Graphics has also answered questions in art unrelated to convex mirrors. Stork and Furuichi created a Computer Graphics model of Georges de la Tour’s Christ in the carpenter’s studio, and compared the rendered images with the light in two locations: in place of the depicted candle, and “in place of the other figure” (i.e., in place of Christ when St. Joseph was painted and vice versa) [6]. In this way they found that for this painting—and for all others in de la Tour nocturne œuvre they tested—that the best fit position was at the candle. In this way they rebutted David Hockney’s optical projection claim, at least for Christ in the carpenter’s studio. Simple computer image analysis (elementary perspective analysis) has addressed another art historical claim about Memling and mirrors, but in this case concave mirrors. David Hockney claimed that Memling executed Flower still-life (c. 1490) by means of tracing an image projected by a concave mirror [11]. In support of his claim, Hockney pointed to the difference in location between the central vanishing point defined by the front half of the carpet and that defined by the back half of the carpet. He hypothesized that Memling refocussed a projector to overcome its limited depth of field and in doing so Memling tipped the projection mirror and thereby change the location of these central vanishing points. However, Stork did a full perspective analysis of the front half of the carpet, and of the back half of the carpet, which included tests for secondary vanishing points. This analysis showed that each portion of the carpet was not 4 Digital Imaging for Cultural Heritage Preservation

Figure 1.2: Hans Memling, Virgin and Child and Maarten van Nieuwenhove (1487). Wood panels, 52.5 × 41.5 cm each (including the original frames). Municipal Museums, Bruges, Hospitaalmuseum Sint- Janshospitaal; image courtesy President and Fellows, Harvard College. The two hinged panels appear to depict a single, unified space, since the Virgin’s red robe and the stone parapet with the carpet in the foreground continues over both panels. In the unified space of the full diptych, the donor sits to the Virgin’s left and is turned towards her. The convex mirror, on the other hand, reflects a different spatial organization. Here, the Virgin and the donor are seated at two perpendicular sides of the same table. In the mirror’s reflection, both figures are silhouetted against bright window-like features, and it has been suggested that these features must represent the beholder’s space, as seen from the depicted space back through be picture frames. In 2006 it was discovered that the top left corner of the Virgin and Child was intensively reworked by Memling. X-ray and infra-red imaging reveals that the initial composition showed a similar window as the one on the right, and that the stained glass window with Van Nieuwenhove’s coat of arms, the window shutters, and the convex mirror were all added later, painted over a continuing landscape.

in proper perspective, though it should be if executed under optical projections [8]. Such evidence thus rebuts Hockney’s optical projection claim.

In Section 1.2 we clarify the art historical question we address, one centered on the devotional practice associated with Memling’s diptych. As we shall see, its answer relies on an analysis of the image of the mirror depicted in its left panel. In Section 1.3 we describe our Computer Graphics modeling of the mirror and tableau. In Section 1.4 we introduce the mathematical tools that we employ in this analysis. In Section 1.5 we present our results, and in Section 1.6 we present our conclusions and implications for the understanding of this diptych. We conclude by speculating on other art historical problems that might be addressed by such computer methods. An early version of this manuscript appeared in [15]. A computer analysis of the mirror in Hans Memling’s Virgin and Child and Maarten van Nieuwenhove 5

1.2 Memling’s Diptych

Many devotional half-figured portrait diptychs, such as Memling’s Virgin and Child and Maarten van Nieuwenhove, apparently did not function while hanging on a wall or column, nor were they fully opened with their wings placed at a straight angle of 180◦ when used in private devotional practice. Instead, such objects were often used in a standing position with its hinged wings at an obtuse angle and it has been suggested that this was also the case with Hans Memling’s famed diptych in Bruges. However, Memling introduced some remarkable pictorial discrepancies between the wings of his diptych which point to the possibility that this object was actually hung on a wall or column. In such a situation, the left wing with the Virgin and Child would have been secured to the wall in a stationary position, while the wing with the portrait of the Virgin and Child and Maarten van Nieuwenhove could be opened and closed or otherwise manipulated. The two hinged wings appear to depict a single, unified space, since the stone parapet in the foreground, decked with a carpet, continues over both panels. The donor’s prayer book is placed on a fold of the Virgin’s red robe that spills onto the right wing. In this unified space, the donor sits to the Virgin’s left and is turned towards her. He appears to be situated in the same plane as the Virgin, immediately behind the parapet. The reflection in the convex mirror belies such a spatial organization, however (Figures. 1.1 and 1.13, below). Here, the Virgin and the donor are seated at perpendicular sides of a table rather than a parapet. Memling also used remarkably different perspective systems for the two wings of his diptych. On the right wing, the wall behind the Virgin and Child and Maarten van Nieuwenhove is depicted in a relatively steep perspective, and its vanishing point is actually positioned on the left wing. The composition of the Virgin and Child, on the other hand, is in full frontal perspective, and the painter’s vantage point is located directly in front of the Virgin, on the panel’s vertical center line. Hence, the ideal viewing point of the entire diptych in its fully opened position is not located in front of the center of the diptych, but in front of the center of the left wing with the Virgin and Child. De Vos and others have suggested that the perspective system for Memling’s full diptych ‘falls into place’ when the beholder stands in front of the Virgin and Child while the right wing is placed at a more or less right angle (i.e., around 90◦) from the left wing [16]. We test to see whether the final reflected image, as it appears in the painting, does or does not conform to a simple Computer Graphics model of the tableau. Our technical question, then, is whether the mirror in the left panel of the diptych was painted with the subjects present, or instead added afterwards. This, in turn, comes down to the question of whether the image is highly consistent with the presence of a model, or inconsistent with the presence of a model. In the former case, the figural reference would likely give correspondence; in the latter case, the artist likely worked from memory or imagination. We note in passing that a similar question applies to van Eyck’s Arnolfini portrait, specifically whether the room was fictive (in the artist’s imagination) or actually present. Despite some inconsistencies between the direct and de-warped reflected images,[9] the overall consistency between the full room, as drawn directly, and the reflection of the room, warped in the convex mirror was high [7]. It would have been a pioneering artistic and visual accomplishment of the highest order if van Eyck could paint a fictive wedding room, and paint the distorted image in the mirror of a fictive room with the spatial consistency we find through the computer reconstruc- tions. As such, it seems far more likely that van Eyck work from a real, physical room as referent than that he painted an imaginary or fictive room. Recent technical examination of the diptych by means of infrared reflectography and X-radiography of the Memling diptych found further evidence that the reflections in the mirror do not depict an actual situation since the entire window behind the proper right shoulder of the Virgin was dramatically altered from its initial design. The stained glass window with Van Nieuwenhove’s coat of arms, the window shutters, and the convex mirror were all added later by Memling, painted on top of an earlier depicted window with mullions with a view on a landscape, which echoed the present window behind the Virgin’s proper left shoulder. The analogous question of praxis is more problematic for the mirror behind the Virgin in the left panel of Memling’s Virgin and Child and Maarten van Nieuwenhove (1487). Notice that the reflected image does not display the warping that is most prominent around the perimeter of convex mirrors such as the van 6 Digital Imaging for Cultural Heritage Preservation

Eyck Arnolﬁni, the Metsys Money lender, and the Christus, Saint Eligius mirrors; nor does the dark gray color of the reﬂected image correspond to the Virgin’s red gown as it otherwise should [17]. In short, there are strong indications that the mirror was not part of the original design, but an afterthought by the artist.

1.3 Computer Vision Analysis

We have created a three-dimensional virtual environment in Maya (a three-dimensional commercial modeling software [18]) where the Mary is located (Figure 1.3). We assume that the background of the painting (windows, wooden wall, etc.) is located on the left side wall of the virtual room, the background wall in the painting. This wall also contains the convex mirror. Furthermore, we assume that Mary is located some- where between the background wall and the camera (observer). The relative positions of camera, Mary and background wall are constrained since the scene viewed from the camera must match the one in the diptych; in other words, their relative positions must be compatible with what we see in the diptych. Moreover, the amount of curvature in the (reﬂected) image of straight objects is another constraint. Figure 1.3 shows the virtual setup for our Computer Graphics analysis. The overall scale is irrelevant to our investigations, but there are several other unknown sizes and positions, such as the position of the artist (“camera”), position and size of Mary, and facial size of the convex mirror. Moreover, we modeled the mirror as a section of a sphere, whose radius of curvature—its bulginess—is a priori unknown.

Mary Mirror Background wall Painter

Figure 1.3: We modeled the full virtual environment for the left panel of the diptych as a ﬂat back wall, a planar Mary, and a protruding convex mirror whose radius of curvature is estimated so as to conform with other image data. The artist’s viewing position is indicated by the camera at the right.

These unknown sizes and positions are constrained and such constraints allow us to infer their relative values. For instance, Figure 1.4 shows the effect of the radius of curvature, R on the angle of view of the scene, and hence the relative sizes of objects. The theoretical analysis presented in [19] provides the tools for estimating a range of possible values of curvature given the measurements on the observer image plane (that is, the painting), location and orientation of the mirror and position of the reflected scene (see Section 1.4 for details). Measured quantities can be, for instance, the location of feature points such as corner points in the image plane. There is another parameter we can account for. Not only does the mirror curvature R affect the scale of the reflected scene, but it also induces a deformation on the reflected scene: straight lines are reflected as curves and the shorter the radius of curvature the more severe the deviation. This is, then, another constraint on our configuration. The analytical expressions in [19] allow predicting values for R given a geometrical A computer analysis of the mirror in Hans Memling’s Virgin and Child and Maarten van Nieuwenhove 7 on θ on i R θi eflecti R θ eflecti r θr g in r g in r arin arin Scene appe Scene Scene appe Scene Memlingʼs eye Memlingʼs eye Memlingʼs

Figure 1.4: Left: A small radius of curvature, R, leads to a wide angle of view, that is, a wide angle of view in reflection. Right: A large radius of curvature, leads to a small angle of view, that is a narrow angle of view in reflection. Of course, the angle of incidence equals the angle of reflection (i.e., θi = θr). configuration (observer view direction, distance of the mirror from the observer, distance of the reflected scene from the observer). The best value for R may be then inferred by minimizing the deviation of the predicted position of feature points (e.g., corner points) from their measurements in the painting. Figure 1.5 shows some of the sample configurations we explored. We found that the one that best matched the image data (the distortion in the convex mirror and the relative sizes of objects and positions) is configuration (d), at the lower right. We adjusted the radius of curvature curvature of the mirror to make the de-warped or rectified image as rectilinear as possible (Figure 1.6). This provides the relative size of the (back) of Mary’s head which in turn depends upon her distance from the mirror. Notice in the detail of the original (left of Figure 1.12) that the reflected image does not display the warping that is most prominent around the perimeter of convex mirrors such as the van Eyck Arnolfini, the Metsys Money lender, and the Christus Saint Eligius mirrors; nor does the dark gray color of the reflected image correspond to the Virgin’s red gown as it otherwise should. In short, there are strong indications that the mirror was not part of the original design, but an afterthought by the artist.

1.4 Modeling Reﬂections off a Mirror Surface

In this section we summarize our main theoretical tools for studying the relationship between mirror’s location and shape, and corresponding observations. For more details see [19]. Estimating the 3D shape of physical objects is one of most useful functions of vision. Texture, shading, contour, stereoscopy, motion parallax and active projection of structured lighting are the most frequently studied cues for recovering 3D shape. These cues, however, are often inadequate for recovering the shape of shiny reflective objects, such as a silver plate, a glass goblet or a well-washed automobile, for it is not possible to observe their surfaces directly, rather only what they reflect. Yet, the ability of recovering the shape of specular or highly reflective surfaces is valuable in many in Computer Vision applications such as industrial metrology of polished parts, medical imaging of moist or gelatinous tissues, digital archival of artworks and heritage objects, remote sensing of liquid surfaces, and diagnostic of space metallic structures, among others. Although specular surfaces are difficult to measure with traditional techniques, specular reflections present an additional cue that potentially may be exploited for shape recovery. A curved mirror produces distorted images of the surrounding world (Figure 1.7). For example, the image of a straight line reflected by a curved mirror is, in general, a curve. It is clear that such distortions are systematically related to the 8 Digital Imaging for Cultural Heritage Preservation

(a) (b)

Figure 1.5: We adjusted the relative positions and sizes of objects to create different conﬁgurations of the scene; four are shown here. The distance values are: 53.7, 17.9, 11.4 and 9 for panel a, b, c and d respectively. The curvature 1/c values are: 16, 8.75, 6.85 and 6 for panel a, b, c and d respectively. Numerical values of distance and curvature are represented in Maya standard units. Notice that all these quantities can be expressed only up to an unknown scale. We found that the one that best matched the image data (the distortion in the convex mirror and the relative sizes of objects and positions) is conﬁguration (d), at the lower right.

shape of the surface. Is it possible to invert this map, and recover the shape of the mirror from its reﬂected images? The general inverse mirror problem is clearly under-constrained: by opportunely manipulating the surrounding world, we may produce a great variety of images from any curved mirror surface, as illustrated by the anamorphic images that were popular during the Renaissance (Figure 1.7(b)) or as shown in Fig- ure 1.5. This inverse problem may become tractable under the assumption that some knowledge about the structure of the scene is available.

In [19] we presented the analytical relationship linking the local shape of a curved mirror surface to the distortions produced on a scene it reflects. We call this step direct map. Also, we showed that, by taking advantage of this analytical relationship, local information about the geometry of the surface may be recovered by measuring the local deformation of the reflected images of a planar scene pattern. We call this step inverse map or 3D recovery step. In the inverse map, we assume that the surface is unknown, the camera is calibrated and the geometry and position of the scene pattern is known in the camera reference system (Figure 1.8). These relationships and constraints are used in the analysis presented in Section 1.3. In this section we analyze in further details such relationships, and demonstrate that these can be used to recover location geometry of a reflective surface. A computer analysis of the mirror in Hans Memling’s Virgin and Child and Maarten van Nieuwenhove 9

–

Figure 1.6: The de-warped or rectiﬁed view, found through adjusting the radius of curvature of the virtual model mirror and additional photo-editing. The ﬁgure shows the back of the Mary.

1.4.1 Direct map We start from the observation that the mapping from a scene line to its reflected curve in the camera image plane (due to mirror reflection) not only changes orientation of the scene line, but also stretches its length, modifying the local scale of the scene line. Such a deforming mapping can be easily illustrated by the grid points (1, 2, 3) and their corresponding reflected points (10, 20, 30) on the mirror surface (Figure 1.9). As shown in [19], from this direct map it is possible to derive the analytical expressions for the local geometry in the image (namely, orientation and local scale of reflected image curve at a given point - (e.g. 10)) as a function of the position, orientation and curvature of the mirror surface. This result can be generalized to the case of a scene patch of arbitrary texture, and obtain the direct mapping between the scene patch (around a given point po) to the corresponding reflected patch in the image plane (around qo) as function of the local parameters (location, orientation and curvature) of the mirror surface (around ro). Here the point ro is the reflection of po and qo is its image (Figure 1.10).

1.4.2 Inverse map If local shape of the mirror surface is unknown, we can use a number of measurements in the image to estimate shape properties. Typical measurements are local position, orientation and local scale of a reflected scene patch in the image plane. For instance, such local image measurements may be computed at a point 0 0 0 0 0 (e.g. 1 in Figure 1.9, or qo in Figure 1.10) from its four neighboring reflected points (e.g., 2 , 3 , 4 , 5 ). As shown in [19], by comparing these measurements with their corresponding analytical expressions, we obtain a set of constraints, which lead to the solution for the surface position ro. Once ro is found, normal, curvature and third-order local parameters of the surface around ro can be found in closed-form solution. Such relationships are critical for predicting values for R (i.e., Memling’s mirror curvature radius) given a geometrical configuration (observer’s view direction, distance of the mirror from the observer, distance 10 Digital Imaging for Cultural Heritage Preservation

Figure 1.7: A curved mirror produces distorted images of the surrounding world. We explore the geometry linking the shape of a curved mirror surface to the distortions produced on a scene it reﬂects.

Camera

Mirror Surface

Scene

Figure 1.8: Setup: A (calibrated) camera is observing the planar scene reﬂected on the surface of the smooth mirror surface. of the reﬂected scene from the observer). As explained in Section 1.3, the best value for R may be then inferred by minimizing the deviation of the predicted position of feature points (e.g., corner points) from their measurements in the painting.

1.4.3 Experimental Validation In this section we show that the theoretical results presented in the previous section can be successfully used to estimate shape properties of mirror surfaces from just one single image. We compared reconstruction results with ground truth 3D models. As we shall see in Section 1.5, these results confirm that the analysis introduced in Section 1.3 can be indeed potentially employed to recover shape properties of the Memling’s mirror. Theoretical results in [19] are validated by recovering local surface parameters of real mirror objects. A Kodak DC290 digital camera with 1792×1200 resolution was used to capture an image of a mirror surface reflecting a planar grid. The corners of the grid served as mark points and were used to measure position, orientation and local scale of reflected grid. Notice the grid can be replaced by any pattern as long as local A computer analysis of the mirror in Hans Memling’s Virgin and Child and Maarten van Nieuwenhove 11

Figure 1.9: Mapping between scene points 1, 2, 3, 4, 5 and their corresponding reﬂections 10, 20, 30, 40, 50.

Table 1.1: Quantitative Reconstruction Results: First column shows the geometrical quantity whose reconstruction accuracy was evaluated. Subsequent columns show the corresponding ground truth value, the mean error and its standard deviation, respectively. The last one shows the number of points reconstructed. position, orientation and scale can be extracted from the reﬂected image. We validated our 3D reconstruction technique with three simple shapes (plane, sphere and cylinder) where ground truth measurements were available (Figure 1.11, Table 1.1). In summary, the analysis in [19] gives the ability to:

• Describe the mapping from an arbitrary planar patch in the scene to the corresponding specular re- ﬂection in the image. The mapping is function of the local geometry of the mirror surface.

• Recover local information about the geometry of the surface (i.e. position, curvature and higher order shape information - up to 3rd order) by measuring the local deformation of the reﬂected images of a planar scene pattern.

Moreover, the analysis in [19] is appealing in that:

• A static monocular camera can be used (as shown in Figure 1.11 and Table 1.1) with only one image we can calibrate the camera, the scene and recover the shape of an unknown mirror surface.

• No initial conditions about the surface position and shape are required.

• Local shape can be recovered in closed from solution. 12 Digital Imaging for Cultural Heritage Preservation

Planar scene Mirror surface

Image plane

Figure 1.10: The direct mapping: The analytical expression describing the mapping from an arbitrary planar patch in the scene (around a generic point po) to the corresponding specular reﬂection in the image. The mapping is function of the local geometry of the mirror surface around the reﬂection point ro.

For these reasons, we view our analysis [19] as a powerful tool for studying the mirror surface in the Memling’s diptych. The case of a full uncalibrated scene and camera introduces additional degrees of freedom. Partial lack of calibration can be accommodated by integrating additional cues and some form of prior knowledge on the likely statistics of the scene geometry (as discussed in Section 1.3).

1.5 Results

Once we had the configuration that was most commensurate with the visual information in the painting and the rules of optics, we studied the image of Mary reflected in the convex mirror and the image we would expect had Memling accurately portrayed such a tableau. First, we could rectify the image in the mirror, that is, give the virtual view from the position of the mirror back into the space of the tableau. Figure 1.12 shows such a rectified view. Visual inspection demonstrates that the reflection painted by the artist shows a more dramatic bending of the linear structure on the left side as the right column of Figure 1.13 shows. This bend is not compatible with the best mirror curvature predicted by our analysis as the left column of Figure 1.13 shows. It appears as the artist exaggerated the curvature in order to make the bulging effect of the spherical mirror more compelling. For instance, see the line feature highlighted in red in the lower panels of Figure 1.13. Other minor incompatibilities can be noticed as well. For instance, notice the amount of fore-shortening of the reflected figures-the Mary and the van Nieuewenhove on the right: the best mirror curvature predicted by our analysis would induce a higher amount of fore-shortening in the two figures than those measured in the original painting. Specifically, the rectified reflected image of the Mary is not compatible with the foreground figure of the Mary. Figure 1.14 shows the rectified reflected image of the Mary. If we overlay it with the foreground image of the Mary, we notice that the two figures do not overlap perfectly; the red region in the figure is the difference. Although we do not assume that the artist would have been photographically accurate throughout a painting, the “error” in the reflected image seems too large to have arisen had Memling had the model as referent. A computer analysis of the mirror in Hans Memling’s Virgin and Child and Maarten van Nieuwenhove 13

(b) (a) Estimate tangent planes

Estimate tangent planes

Scene Pattern Scene Pattern

Figure 1.11: Qualitative Reconstruction Results: (a) A cylindrical mirror is placed over the grid pattern. We reconstructed 20 surface points and normals. The reconstructed points are highlighted with white circles. (c) Top view of the reconstruction. Each reconstructed surface point is depicted by its tangent plane and normal. (b) A spherical mirror placed over the grid pattern. (d) Top view of the reconstruction. .

Figure 1.12: Left: Mirror in the original painting. Right: Original painting with rectiﬁed mirror image. Image courtesy President and Fellows, Harvard College. 14 Digital Imaging for Cultural Heritage Preservation

Figure 1.13: Left column: Image of Mary reflected in the convex mirror we would expect had Memling accurately portrayed such a tableau. Curvature and scene configuration are those of panel (c) in Figure 1.5. Right column: Image of Mary reflected in the convex mirror in Memling’s original painting. Visual inspection shows that reflection painted by the artist exhibits a more dramatic bending of left side of the wall (e.g., notice the linear structure highlighted in red in the right-bottom panel). This bend is not compatible with the best mirror curvature predicted by our analysis. Notice the lower curvature of the same linear structure highlighted in red in the left-bottom panel.

1.6 Conclusions

In conclusion, we have found two major inconsistencies or incompatibilities between the image depicted in the convex mirror and the image we would expect if the room and Mary were accurately rendered: • curvature of the mirror and amount of distortion that reflected lines should have in order to be compatible with that curvature. • the shape of the Mary’s figure and its reflected counterpart. These findings are in line with the idea that the mirror was indeed an afterthought by Memling (or the patron), and that the painted reflection was most probably not painted following an actual model. We hope that our findings will aid further art historical research into the actual usage of this and other diptychs in devotional practice. Our work extends that of Criminisi and his colleagues who opened new vistas through dewarping the reflections in convex mirrors in Renaissance art [9]. While that previous work highlighted differences between northern and Italian approaches to realism (the northern based somewhat more on close observation [20], the Italian based somewhat more on reasoned construction [21]), our work sheds light on religious A computer analysis of the mirror in Hans Memling’s Virgin and Child and Maarten van Nieuwenhove 15

Figure 1.14: Each figure shows the overlap of the rectified (and left-right reversed) reflection of Mary and the scaled direct view. The right panel highlights the difference in shape between these two images - region of non-overlapping that seems too large to have arisen if Memling had worked from an actual sitter as referent. praxis in the early Renaissance. More broadly, though, our work further demonstrates that new computer methods can shed light on problems in the history of art, and are most likely to profit from collaboration between computer vision experts, deeply familiar with both the power and limitations of image algorithms, and humanistic art scholars, who understand the art historical context and questions addressed [7].

1.7 Acknowledgments

We thank the Municipal Museums, Bruges, Hospitaalmuseum Sint-Janshospitaal for permission to repro- duce the Memling diptych. Dr. Savarese and Andrey DelPozo were supported, in part, by grant 0413312 from the National Science Foundation. We also proﬁted from the comments from several reviewers. 16 Digital Imaging for Cultural Heritage Preservation Bibliography

[1] D. G. Stork and J. Coddington, eds., Computer image analysis in the study of art, vol. 6810. Belling- ham, WA: SPIE/IS&T, 2008.

[2] D. G. Stork, J. Coddington, and A. Bentkowska-Kafel, eds., Computer vision and image analysis in the study of art, vol. 7531. Bellingham, WA: SPIE/IS&T, 2010.

[3] C. R. Johnson, E. Hendriks, I. J. Berezhnoy, E. Brevdo, S. M. Hughes, I. Daubechies, J. Li, E. Postma, and J. Z. Wang, “Image processing for artist identiﬁcation,” IEEE Signal Processing magazine, vol. 25, no. 4, pp. 37–48, 2008.

[4] R. P. Taylor, “Order in Pollock’s chaos,” Scientiﬁc American, vol. 287, no. 66, pp. 116–121, 2002.

[5] M. Irfan and D. G. Stork, “Multiple visual features for the computer authentication of Jackson Pol- lock’s drip paintings: Beyond box-counting and fractals,” in SPIE Electronic Imaging: Image processing: Machine vision applications II, vol. 7251, pp. 72510Q1–11, SPIE/IS&T, 2009.

[6] D. G. Stork and Y. Furuichi, “Image analysis of paintings by computer graphics synthesis: An investigation of the illumination in Georges de la Tour’s Christ in the carpenter’s studio,” in Computer image analysis in the study of art, vol. 6810, pp. 68100J1–12, SPIE/IS&T, 2008.

[7] D. G. Stork, “Optics and realism in Renaissance art,” Scientiﬁc American, vol. 291, no. 6, pp. 76–84, 2004.

[8] D. G. Stork, “Did Hans Memling employ optical projections when painting Flower still-life?,” Leonardo, vol. 38, no. 2, pp. 57–62, 2005.

[9] A. Criminisi, M. Kemp, and S.-B. Kang, “Reﬂections of reality in Jan van Eyck and Robert Campin,” Historical methods, vol. 3, no. 37, pp. 109–121, 2004.

[10] A. Criminisi, Acccurate visual metrology from single and multiple uncalibrated images. ACM Distin- guished Dissertation Series, London: Springer-Verlag, 2001.

[11] D. Hockney, Secret knowledge: Rediscovering the lost techniques of the old masters. New York, NY: Viking Studio, 2001.

[12] D. Falk, D. Brill, and D. Stork, Seeing the Light: Optics in nature, photography, color, vision and holography. New York, NY: Wiley, 1986.

[13] D. G. Stork, “Optics and the old masters revisited,” Optics and Photonics News, vol. 15, no. 3, pp. 30– 37, 2004.

[14] M. Johnson, D. G. Stork, S. Biswas, and Y. Furuichi, “Inferring illumination direction estimated from disparate sources in paintings: An investigation into Jan Vermeer’s Girl with a pearl earring,” in Computer image analysis in the study of art, vol. 6810, p. 68100I, SPIE/IS&T, 2008.

17 18 Digital Imaging for Cultural Heritage Preservation

[15] S. Savarese, R. Spronk, D. G. Stork, and A. DelPozo, “Reﬂections on praxis and facture in a devotional portrait diptych: A computer analysis of the mirror in hans memlings virgin and child and maarten van nieuwenhove,” in 20th Annual Symposium Electronic Imaging (SPIE, ed.), 2008.

[16] D. D. Vos, Hans Memling: The complete works. New York, NY: Harry N. Abrams, 1994.

[17] J. O. Hand, C. A. Metzger, and R. Spronk, Prayers and Portraits: Unfolding the Netherlandish diptych. New Haven, CT: Yale U. Press, 2006.

[18] http://usa.autodesk.com

[19] S. Savarese, M. Chen, and P. Perona, “Local shape from mirror reﬂections,” International Journal of Computer Vision, vol. 64, no. 1, pp. 31–67, 2005.

[20] S. Alpers, The art of describing: Dutch art in the seventeenth century. Chicago, IL: University of Chicago Press, 1984.

[21] L. S. Adams, Italian Renaissance Art. Boulder, CO: Westview Press, 2001.