C Copyright by Jonathan C. Mckinney, 2004 BLACK HOLE ACCRETION SYSTEMS

°c Copyright by Jonathan C. McKinney, 2004 BLACK HOLE ACCRETION SYSTEMS

JONATHAN C. MCKINNEY

B.S., Texas A&M University of College Station, 1996 M.S., University of Illinois at Urbana-Champaign, 1999

DISSERTATION

Submitted in partial fulﬁllment of the requirements for the degree of Doctor of Philosophy in Physics in the Graduate College of the University of Illinois at Urbana-Champaign, 2004

Urbana, Illinois Abstract

Accretion onto a black hole is the most efficient known process to convert gravitational energy into radiation. Some gamma-ray bursts (GRBs), some X-ray binaries, and all active galactic nuclei (AGN) are likely powered by accretion onto a rotating black hole. I model the global, time- dependent accretion flow around a black hole using the nonradiative viscous hydrodynamic (VHD), Newtonian magnetohydrodynamic (MHD), and general relativistic MHD (GRMHD) equations of motion, which are integrated numerically. I wrote the VHD and nonrelativistic MHD numerical codes, and coauthored the GRMHD code (Gammie et al., 2003). I first studied VHD accretion disk models, such as those studied by Igumenshchev et al. (1999, 2000) and Stone et al. (1999), hereafter IA and SPB. IA and SPB use the VHD model of accretion, but they gave qualitatively different measurements of the energy per baryon accreted, angular momentum per baryon accreted, and the radial scaling law for various quantities. While there was concern in the community that the different results were due to numerical error, I found that the different results could be reproduced using my single VHD code to model the accretion flow (McKinney and Gammie, 2002). The differences in their results were due to differences in their experimental designs. Seemingly small changes to the VHD model introduced nonnegligible changes to the results. This suggests that a self-consistent MHD, rather than phenomenological VHD, model for turbulence is required to study accretion flow. First discovered during the VHD study described above, I found that VHD, nonrelativistic MHD, and GRMHD numerical accretion disk models can produce significant numerical artifacts unless the flow near the inner radial boundary condition, at rin, is out of causal contact with the

flow at r > rin. For the VHD model, this corresponds to setting rin so the flow there is always ingoing at supersonic speeds. For the MHD models, this corresponds to setting rin so the flow there is always ingoing at superfast speeds. For the GRMHD model, this is easily constructed by using

Kerr-Schild (horizon-penetrating) coordinates. In this case, rin is chosen to be inside the horizon, where all waves are ingoing. I next studied MHD and GRMHD numerical accretion disk models to test the results of previously studied phenomenological disk models. These models are based on the Shakura & Sunyaev α-disk model and suggest the disk should terminate at the innermost stable circular orbit (ISCO) of a black hole. Simplified GRMHD models predict super-efficient accretion due to energy extraction from a rotating black hole. I found that MHD and GRMHD numerical models show that the disk does not terminate at the ISCO, and magnetic fields continue to exert a torque on the disk inside the ISCO. The disk will likely continue to emit radiation inside the ISCO, altering the predicted spectra of accretion disks. GRMHD numerical models of thick and thin disks show that the energy

iii per baryon accreted closely follows the thin disk efficiency, so super-efficient accretion does not seem to be a generic property of thick magnetized relativistic disks (McKinney and Gammie, 2004). The Blandford-Znajek (BZ) effect, describing the extraction of spin energy of a rotating black hole by the magnetosphere, plausibly powers the jet in some GRBs, some microquasars, and all AGN. I found that GRMHD numerical models of thick disks around a rotating black hole show that an evacuated, nearly force-free magnetosphere develops as predicted by BZ (McKinney and Gammie, 2004). The BZ solution for the energy extracted is remarkably accurate in this region for a black hole with a/M . 0.5 and qualitatively accurate for all a/M, where a is the Kerr spin parameter and M is the mass of the black hole. GRMHD numerical models with a/M & 0.5 show a mildly relativistic (Lorentz factor Γ ∼ 1.5 − 3) collimated Poynting jet around the polar axis. Currently, no self-consistent MHD model of the accretion flow around a black hole shows a jet with Γ & 3. Additional physics is likely required to obtain Γ ∼ 100 as models predict in GRBs, and to obtain Γ ∼ 3 − 10 as seen in some microquasars and AGN. I studied the VHD, MHD, and GRMHD accretion models by performing numerical simulations on our group’s Beowulf computer clusters, which I designed and constructed. For about $50,000, one can buy a private cluster of computers that will provide as much computing power as today’s typical time-shared “supercomputer.” I give an account of the procedures necessary to design, build, and test a Beowulf cluster. The main conclusion is obvious: test one’s code on test nodes before purchasing the entire cluster in order to confirm the performance and reliability of the chosen components (CPUs, motherboard, network, etc.).

iv To my mom.

v Acknowledgments

I strongly express my gratitude to my supervisor, Prof. Charles Forbes Gammie, whose expertise, understanding, and patience added considerably to my graduate experience. Charles demonstrates a broad-minded, razor-sharp knowledge of astrophysics, and his example has driven me to become a better scientist. Charles has tirelessly reviewed many of my written works, and with his help, I have gradually improved my skills as a writer. I express a special thanks to Scott Noble and Ruben Krasnopolsky, who both proofread the unpublished parts of my thesis. They gave excellent comments that greatly improved the focus, clarity, and readability of the thesis. Throughout their careers as postdocs for Charles, they have provided me with many stimulating conversations and uncountable helpful pointers. I thank Paul Ricker for excellent comments about the Beowulf cluster appendix. I thank the members of my thesis committee: Charles Gammie, Stu Shapiro, Susan Lamb, and Jen-Chieh Peng. Their time is precious and I am grateful for all the comments on my written thesis and oral defense. I particularly thank Stu Shapiro for being a great source of inspiration and guidance in the study of relativity and compact objects. I thank Charles Gammie, Stu Shapiro, Bill Watson, David Campbell, and Peter Anninos for writing recommendation letters for my future career as a postdoc at Harvard with Ramesh Narayan, who I thank for hiring me and being patient while I finish my thesis. I thank and love my family for the support they provided me through my entire life and in particular, I must acknowledge my fiancéand best friend, Elena, without whose love, encouragement, and editing assistance I would not have finished this thesis. My financial support was largely provided by a NASA GSRP Fellowship Grant NGT5-50343 (S01-GSRP-044) and partially supported by a GE fellowship. During my thesis work, Charles was supported by an NCSA Faculty Fellowship, the UIUC Research Board, NSF ITR grant PHY 02-05155, and NSF PECASE grant AST 00-93091. Computations were done in part under NCSA grants AST010012N and AST010009N using the Origin 2000 and Posic Linux cluster at NCSA. Some computations were performed on Platinum.

vi Table of Contents

List of Figures ...... x

List of Tables ...... xiv

List of Abbreviations and Acronyms ...... xv

Summary ...... xix

1 Observations, Theory, and Models ...... 1 1.1 Summary of Introduction ...... 1 1.2 Introduction to Black Hole Systems ...... 4 1.2.1 Formation of Black Holes ...... 5 1.2.2 Gamma-Ray Bursts ...... 7 1.2.3 Black Hole X-ray Binaries ...... 9 1.2.4 Normal and Active Galactic Nuclei and Quasars ...... 12 1.3 Basic Accretion Disk Theory ...... 15 1.3.1 Accretion Luminosity and Mass of the Compact Object ...... 15 1.3.2 Some Accretion-Based Arguments ...... 19 1.4 Models of Accretion Disks and GRBs ...... 24 1.4.1 Angular Momentum Transport Models ...... 25 1.4.2 Radiative Disk Models ...... 29 1.4.3 Gamma-Ray Bursts Models ...... 33 1.5 Characteristic Quantities and Model Validity Estimates ...... 38 1.5.1 Estimated State and Structure of Accretion Flow ...... 38 1.5.2 Validity of the Fluid, MHD, and ideal MHD Approximations ...... 46 1.6 Summary of Motivation for a GRMHD Model and Open Questions ...... 49 1.7 Summary of Dissertation Results ...... 50 1.7.1 Viscous Hydrodynamics Summary ...... 52 1.7.2 Global 2D/3D MHD Summary ...... 54 1.7.3 HARM / GRMHD Summary ...... 56 1.7.4 BZ Eﬀect Summary ...... 57 1.7.5 BZ/Inﬂow Solution Comparison Summary ...... 58

2 Numerical Models of Viscous Accretion Flows Near Black Holes ...... 60 2.1 Summary of Chapter ...... 60 2.2 Introduction ...... 60 2.3 Model ...... 62 2.4 Numerical Methods ...... 65 2.4.1 Numerical Treatment of Low Density Regions ...... 66 2.4.2 Diagnostics ...... 66

vii 2.4.3 Code Tests ...... 67 2.5 Results ...... 68 2.5.1 Fiducial Model Evolution ...... 69 2.5.2 Dependence on Inner Boundary Location and Gravitational Potential . . . . 74 2.5.3 Comparison of Torus and Injection Models ...... 76 2.5.4 Other Parameters ...... 77 2.6 VHD Summary ...... 78 2.7 Global 2D MHD Simulations ...... 79 2.8 Global 3D MHD Simulation ...... 81

3 HARM: A Numerical Scheme for General Relativistic Magnetohydrodynamics 82 3.1 Summary of Chapter ...... 82 3.2 Introduction ...... 82 3.3 A GRMHD Primer ...... 84 3.4 Numerical Scheme ...... 86 3.4.1 Constrained Transport ...... 88 3.4.2 Wave Speeds ...... 89 3.4.3 Implementation Notes ...... 90 3.5 Code Veriﬁcation ...... 92 3.5.1 Linear Modes ...... 92 3.5.2 Nonlinear Waves ...... 96 3.5.3 Transport ...... 100 3.5.4 Orszag-Tang Vortex ...... 100 3.5.5 Bondi Flow in Schwarzschild Geometry ...... 105 3.5.6 Magnetized Bondi Flow ...... 105 3.5.7 Magnetized Equatorial Inﬂow in Kerr Geometry ...... 107 3.5.8 Equilibrium Torus ...... 111 3.6 Magnetized Torus Near Rotating Black Hole ...... 111 3.7 Conclusion ...... 114

4 A Measurement of the Electromagnetic Luminosity of a Kerr Black Hole . . . 116 4.1 Summary of Chapter ...... 116 4.2 Introduction ...... 116 4.3 Review of Analytic Models ...... 119 4.3.1 Coordinates ...... 119 4.3.2 Governing Equations ...... 120 4.3.3 Blandford-Znajek Model ...... 121 4.3.4 Equatorial MHD Inﬂow ...... 126 4.4 Numerical Experiments ...... 128 4.4.1 Fiducial Model ...... 129 4.4.2 Comparison with BZ ...... 136 4.4.3 Comparison to Inﬂow Solution ...... 141 4.5 Parameter Study ...... 144 4.5.1 Black Hole Spin ...... 144 4.5.2 Field Geometry and Strength ...... 148 4.5.3 Numerical Parameters ...... 149 4.6 Discussion ...... 151

viii 5 Future Studies ...... 155 5.1 Thin Disks ...... 155 5.2 Relativistic Jet-Disk Connection ...... 156 5.3 Gamma-Ray Bursts: GRMHD Collapsar Model ...... 156

A Model of Plasmas ...... 159 A.1 Equation of State ...... 160 A.1.1 EOS for GRBs ...... 160 A.1.2 EOS for X-ray binaries and AGN ...... 162 A.2 Validity of the Fluid Approximation ...... 162 A.3 Validity of the Ideal Fluid Approximation ...... 164 A.4 Validity of the Plasma and MHD Approximations ...... 165 A.5 Validity of the Ideal MHD Approximation ...... 167

B Beowulf cluster ...... 172 B.1 History of Design Decisions for our Clusters ...... 172 B.2 Cluster Performance and Advanced Network Drivers ...... 179 B.3 Cluster access to Internet ...... 183 B.4 Hardware ...... 184 B.5 Software ...... 189 B.5.1 Choices for OS ...... 189 B.5.2 OS Installation ...... 190 B.5.3 Software Installation and Usage Notes ...... 191 B.5.4 MPI Implementation in Fluid Codes ...... 194 B.5.5 Running MPI Jobs ...... 196 B.6 Testing Cluster Reliability and Performance ...... 197 B.6.1 Reliability and Performance Issues ...... 197 B.6.2 Bandwidth and Latency of Network ...... 199 B.6.3 Code Performance ...... 200 B.7 Beowulf Cluster Summary ...... 209

References ...... 211

Curriculum Vitae ...... 250

ix List of Figures

1.1 Cartoon plot of one hemisphere of the axisymmetric accretion disk, corona, funnel/wind, and plunging regions. The outer radius of the GRMHD simulation is 40GM/c2. The ﬁducial radius (r = 12L, where L = GM/c2) is where all “state” quantities are evaluated. These state quantities are then used to estimate the validity of the ﬂuid, MHD, and ideal MHD approximation at this location...... 39

2 2.1 Time-averaged spatial structure of fiducial run (Run A; α = 0.1, rin = 2.7GM/c , 2 and rout = 600GM/c ). Shown are the density (upper left), Bernoulli parame- 2 2 ter (Be = (1/2)v + cs/(γ − 1) + Ψ) (upper right; dotted line is a negative con- 2 tour), scaled mass flux r sin θ(ρ0v) (lower left), and scaled angular momentum flux 3 2 ˆ r sin θ(ρ0vvφ + Π · φ) (lower right). The flow is not symmetric about the equator because the flow exhibits long timescale antisymmetric variations. Convective bubbles form at the interface between positive and negative Bernoulli parameter (i.e. unbound and bound matter)...... 70 2 2.2 The evolution of M/˙ M˙ inj, e = E/˙ (Mc˙ ), and l = L˙ c/(GMM˙ ) in the fiducial run (Run A). The dotted line indicates the thin disk value. The run has clearly entered a quasi-steady state. The evolution is relatively smooth with a small variation on a timescale τ ≈ 4×104. This is the timescale for convective bubble formation (the low point in rest-mass accretion rate is when bubble forms). For this model the bubble forms at alternate poles. A full cycle requires of order one rotation period at the injection radius...... 72 2.3 The radial run of θ and time averaged quantities from the fiducial run (Run A). Shown are the density (upper left), squared sound speed (upper right), radial velocity (lower left), specific angular momentum (lower right; solid line), and circular orbit specific angular momentum (lower right; dashed line). Crudely speaking, the inner −0.6 flow is consistent with a radial power law. The best fits to a power law are: ρ0 ∝ r , −0.5 −2 −0.8 cs ∝ r , |vr| ∝ r , and vφ ∝ r . The plots are averaged over θ = π/2 ± π/6. . 73 2.4 The effect of moving the inner boundary on the accretion rates of mass, angular momentum, and energy (Run A vs. Run B). The top panel shows M/˙ M˙ inj, the middle panel E/˙ (Mc˙ 2), and the bottom panel L˙ c/(GMM˙ ). The solid curve is Run 2 2 A, which has rin = 2.7GM/c . The dashed curve is Run B, which has rin = 6GM/c . Evidently Run B has a different variability structure and different time averaged values for the accretion rates. The relatively rapid and high-amplitude variations in Run B are due to nonphysical interactions with the inner radial boundary. Only by ensuring a supersonic flow (as in Run A) can one avoid these nonphysical effects. . 75 2.5 Field line snapshot after 8 orbits (at the t = 0 torus density maximum at r = 9.4GM/c2) for a global 2D MHD simulation. Full non-linear turbulence drives the accretion process. Note that the polar field is essentially radial while the accretion disk is dominated by turbulence generated by the MRI...... 80

x 3.1 The L1 norm of the error in u for a slow wave as a function of Nx for both the monotonized central (MC) and minmod limiter. The straight lines show the slope expected for second order convergence...... 93 3.2 The L1 norm of the error in the single nonzero component of the velocity for an Alfvénwave as a function of Nx for both the monotonized central (MC) and minmod limiter. The straight lines show the slope expected for second order convergence. . . 94 3.3 The L1 norm of the error in u for a fast wave as a function of Nx for both the monotonized central (MC) and minmod limiter. The straight lines show the slope expected for second order convergence...... 95 3.4 The run of density in the Komissarov nonlinear wave tests...... 97 3.5 The run of ux in the Komissarov nonlinear wave tests...... 98 3.6 Snapshot of the final state in HARM’s integration of Ryu & Jones test 5A (a version of the Brio & Wu shock tube) but with c = 100. The figure shows primitive variable values at t = 0.15. Quantitative agreement is found to within ≈ 1%, as expected. . 99 3.7 Snapshot of the final state in HARM’s integration of Ryu & Jones test 2A, with c = 100...... 101 3.8 Convergence results for the transport test...... 102 3.9 A cut through the density in the nonrelativistic Orszag-Tang vortex solution from HARM (solid line, with c = 100), from VAC (dashed line), and 4× the difference (lower solid line) at a resolution of 6402...... 103 3.10 Comparison of results from HARM and the nonrelativistic MHD code VAC for the Orszag-Tang vortex. The plot shows the L1 norm of the difference between the two results as a function of resolution for the primitive variables ρ0 (squares) and u (triangles). The straight line shows the slope expected for first order convergence. The errors are large because they are an integral over an area of (2π)2...... 104 3.11 Convergence results for the unmagnetized Bondi accretion test onto a Schwarzschild black hole. The straight line shows the slope expected for second order convergence. 106 3.12 Convergence results for the magnetized Bondi accretion test onto a Schwarzschild black hole. The straight line shows the slope expected for second order convergence. 108 3.13 The equatorial inflow solution in the Kerr metric for a/M = 0.5 and magnetization parameter Fθφ = 0.5. The panels show density, radial component of the four-velocity in Boyer-Lindquist coordinates (with the square showing the location of the fast point), the φ component of the four-velocity, and the toroidal magnetic field Bφ = F φt...... 109 3.14 Convergence results for the magnetized inflow solution in a Kerr metric with a/M = 0.5. Parameters for the initial, quasi-analytic solution are given in the text. The straight line shows the slope expected for second order convergence. The L1 error norm for each of the nontrivial variables are shown. The small deviation from second order convergence at high resolution is due to numerical errors in the quasi-analytic solution used to initialize the solution...... 110 3.15 Convergence results for the Fishbone and Moncrief equilibrium disk around an a/M = 0.95 black hole...... 112 3.16 Density field, for a magnetized torus around a Kerr black hole with a/M = 0.5 at t = 0 (left) and at t = 2000M (right). The color is mapped from the logarithm of the density; black is low and dark red is high. The resolution is 3002...... 113

xi 3.17 Evolution of the rest-mass accretion rate (top), the speciﬁc energy of the accreted matter (middle), and the speciﬁc angular momentum of the accreted matter (bottom) for a black hole with a/M = 0.5...... 115

4.1 Initial (left) and final (right) distribution of log ρ0 in the fiducial model on the r sin θ− −7 r cos θ plane. At t = 0 black corresponds to ρ0 ≈ 4 × 10 and dark red corresponds −7 to ρ0 = 1. For t = 2000, black corresponds to ρ0 ≈ 4×10 and dark red corresponds to ρ0 = 0.57. The black half circle at the left edge is the black hole...... 130 2 4.2 (a) The distribution of β, b /ρ0, and ut in the fiducial run, based on time and hemi- spherically averaged data. Starting from the axis and moving toward the equator: 2 (1) ut = −1 contour shown as a solid black line; (2) b /ρ0 = 1 contour shown as a red line; (3) β = 1 contour shown as a magenta line that nearly matches part of the ut = −1 contour line; and (4) β = 3 contour is shown as cyan line. (b) Motivated by the left panel, the right panel indicates the location of the five main subregions of the black hole magnetosphere. They are (1) the disk: a matter dominated region 2 where b /ρ0 ¿ 1; (2) the funnel: a magnetically dominated region around the poles 2 where b /ρ0 À 1 where the magnetic field is collimated and twists around and up the axis into an outflow; (3) the corona: a region in the relatively low density upper layers of the disk with weak time-averaged poloidal field; (4) the plunging region; and (5) the wind, which straddles the corona-funnel boundary. See Section 4.4.1 for a discussion...... 131 4.3 Initial (left) and final (right) distribution of Aφ. Level surfaces coincide with magnetic field lines and field line density corresponds to poloidal field strength. In the initial state field lines follow density contours if ρ0 > 0.2ρ0,max...... 133 4.4 Contour plot of the time and hemispheric average of Aφ. Level surfaces coincide with magnetic field lines and field line density corresponds to poloidal field strength. 134 4.5 Evolution of rest-mass, energy, and angular momentum accretion rate for our fiducial run of a weakly magnetized tori around a black hole with spin a = 0.938. For 500 < t < 2000 the time average of these values is M˙ 0 ' 0.35, E/˙ M˙ 0 ' 0.87, and L/˙ M˙ 0 ' 1.46 as shown by the dashed lines. The dotted lines show the classical thin disk values (E/˙ M˙ 0 ' 0.82 and L/˙ M˙ 0 ' 1.95). See Section 4.4.1 for a discussion. . . . 135 (EM) 4.6 Electromagnetic energy flux density FE (θ) on the horizon for the fiducial run, based on time and hemisphere averaged data. The mean electromagnetic energy flux is directed outward. See Section 4.4.1 for a discussion...... 137 4.7 The run of the force-free parameter ζ for the a = 0.5 run; when ζ ¿ 1 the field is approximately force-free. The parameter has been time and hemisphere averaged. The contours show (beginning from the pole and moving toward the equator) ζ = 10−3, 10−2, 10−1. The small closed contours at large radius and close to the axis have ζ = 10−2. The small closed contours from the equator to θ ∼ π/4 have ζ = 10−1. See Section 4.4.2 for a discussion...... 138 4.8 Left panel: Magnetic field angular frequency on the horizon relative to black hole rotation ω(θ)/ΩH . The solid line indicates time and hemisphere averaged data from our a = 0.5 MHD integration. The middle dotted line is the prediction of the BZ model (ω/ΩH = 1/2). The dashed line (top) is the value predicted by the inflow model. Right panel: the run of field rotation frequency ω with radius along a single field line that intersects the horizon at θ = 0.2. ω is constant to within 3%, as expected for a steady flow. See sections 4.4.2 and 4.4.3 for a discussion...... 139

xii 4.9 (a) Square of radial field ((Br(θ))2) on the horizon in the a = 0.5 MHD integration, from time and hemisphere averaged data. Solid line is the field for our numerical model. The dotted line shows the Blandford and Znajek (1977) perturbed monopole solution with the field strength normalized to the numerical solution at the pole. (EM) The dashed line is the inflow solution. (b) Electromagnetic energy flux FE (θ) on the horizon in the a = 0.5 MHD integration, from time and hemisphere averaged data. The solid line shows the numerical model, the dotted line shows BZ’s spun-up monopole solution, and the dashed line shows the inflow solution. See sections 4.4.2 and 4.4.3 for a discussion...... 140 4.10 A comparison of the time-averaged fiducial model near the equator (within θ = π/2 ± 0.3) with the inflow solution of Gammie (1999). In the right two panels the black dotted line is the thin disk value. In all cases the red vertical line is the location of the ISCO. The black line for the upper left panel is the numerical result. For the other three panels, the particle term is shown in cyan, the internal energy term is shown in magenta, and the electromagnetic term is shown in green. The blue line in each plot represents the inflow model result. Notice that the run of density with radius shows no feature at the ISCO. See the Section 4.4.3 for discussion...... 142 4.11 The ratio of electromagnetic to matter energy flux on the horizon. The solid line indicates numerical data while the dotted line indicates a best fit of E˙ (EM)/E˙ (MA) = 2 −0.068(2 − r+) . See Section 4.5.1 for a discussion...... 145

B.1 Astronomy Building computer room electrical layout as of Feb, 2004. BH is located north and center. This excludes the addition of another new node for BH and the collaborating group’s cluster of 12 nodes...... 178 B.2 BH Beowulf cluster primary elements...... 180 B.3 Digital photograph of BH cluster...... 181 B.4 Cluster block diagram of BH cluster. Figure shows bandwidth between elements on motherboard and switch...... 185 B.5 Bandwidth (left) and latency (right) for gigabit Ethernet connection on BH cluster. 200

xiii List of Tables

1.1 Fiducial Black Hole Accretion Systems 1 ...... 5 1.2 Fiducial Black Hole Accretion Systems 2 ...... 18 1.3 Fiducial Black Hole Accretion Systems 3 ...... 24 1.4 Accretion Flow Regions ...... 40 1.5 Accretion Disk State ...... 43 1.6 Validity of Fluid Approximation ...... 46 1.7 Validity of MHD Approximation ...... 47 1.8 Validity of Ideal MHD Approximation ...... 48

2.1 Parameter List ...... 69 2.2 Results List ...... 69

4.1 Commonly used symbols ...... 118 4.2 Black Hole Spin Study ...... 146 4.3 Field Strength and Geometry Study ...... 148 4.4 Resolution Study ...... 150

B.1 Single CPU performance in ZCPS ...... 202 B.2 Origin 2000 MPI performance in kZCPS for 2D MHD code ...... 203 B.3 NCSA Platinum MPI performance in kZCPS for 2D & 3D MHD code ...... 204 B.4 BH Xeon Cluster MPI performance in kZCPS for ZEUS-based 2D MHD code . . . . 205 B.5 BH Xeon Cluster MPI performance in kZCPS for 3D MHD code ...... 206 B.6 BH Xeon BH Cluster MPI performance in kZCPS for 2D HARM code ...... 207

xiv List of Abbreviations and Acronyms

ADAF Advection Dominated Accretion Flow.

AGN Active Galactic Nuclei.

ASCA Advanced Satellite for Cosmology and Astrophysics.

BATSE Burst And Transient Source Experiment.

BeppoSAX Beppo Satellite per Astronomia X, in honor of Giuseppe “Beppo” Occhialini.

BC Boundary Condition.

BH Black Hole.

BHC Black Hole Candidate.

BL Lac BL Lacertae.

BTU British Thermal Unit.

BZ Blandford & Znajek (1977).

CDM Cold Dark Matter.

CPU Central Processing Unit.

CV Cataclysmic Variable.

EM ElectroMagnetic.

EMF Electromotive Force.

EOS Equation Of State.

GM Glenn’s Messages (Myrinet message protocol).

GR General Relativity.

GRB Gamma-Ray Burst.

GRMHD General Relativistic MagnetoHydroDynamics.

GRS Galactic Radio Source.

xv HARM High Accuracy Relativistic Magnetohydrodynamics.

HD HydroDynamics.

HDE Henry Draper Extension (catalogue).

HETE High Energy Transient Explorer

HLL Harten-Lax-van Leer

HMXB High-Mass X-ray Binary.

HST Hubble Space Telescope.

IA Igumenshchev & Abramowicz (2000,2001).

IC Index Catalogue (of nebulae).

IMBH Intermediate-Mass Black Hole.

IMF Initial Mass Function.

IR InfraRed.

ISCO Inner-most Stable Circular Orbit.

KS Kerr-Schild.

LF Lax-Friedrich.

LIGO Laser Interferometer Gravitational Wave Observatory.

LISA Laser Interferometer Space Antenna.

LMC Large Magellanic Cloud.

LMXB Low-Mass X-ray Binary.

MC Monotonized Central.

MGC Millennium Galaxy Catalogue.

MHD MagnetoHydroDyanmics.

MKS Modiﬁed Kerr-Schild.

MPI Message Passing Interface.

MRI MagnetoRotational Instability.

MTW Misner, Thorne, & Wheeler (1973).

xvi NGC New General Catalogue.

NS Neutron Star.

NSE Nuclear Statistical Equilibrium.

OS Operating System.

PWF Popham, Woosley, & Fryer (1999).

QED Quantum ElectroDynamics.

QPO Quasi-Periodic Oscillation.

RXTE Rossi X-ray Timing Explorer.

RJ Ryu & Jones (1995).

RMR Relativistic MagnetoRotators.

ROSAT R¨ontgen Satellite.

SGI Silicon Graphics Incorporated.

SLE Shapiro, Lightman, & Eardley (1976).

SDSS Sloan Digital Sky Survey.

SMBH Super-Massive Black Hole.

SN Supernova.

SPB Stone, Pringle, & Begelman (1999).

SRMHD Special Relativistic MagnetoHydroDynamics.

SS/SS73 Shakura & Sunyaev (1973)

SXT Soft X-ray Transient.

ULX UltraLuminous X-ray (object).

URCA The URCA process involves scattering of a neutrino, proton, neutron, and electron. Coined by Gamow & Schoenberg in 1941 in humorous reference to the money drain they experienced gambling in Urca, Brazil.

UV UltraViolet.

VHD Viscous HydroDynamics.

xvii VLA Very Large Array.

VLBI Very Long Base Interferometry.

VLBA Very Long Baseline Array.

XMM-Newton X-ray Multi-Mirror - Newton.

XSPEC X-Ray Spectral ﬁtting package.

YSO Young Stellar Object.

xviii Summary

About twice per day, gamma-ray observatories detect a gamma-ray burst (GRB) with a typical duration of seconds (for a review see Piran 2004). Some GRBs have an associated lower-energy afterglow. Redshift measurements of the afterglow, together with the fireball model of GRBs, suggest GRBs have a luminosity of 1045 − 1053 erg/s. A high luminosity GRB is likely the most powerful event in the universe. The known duration of GRBs allows one to estimate that most GRBs have a total energy of ∼ 1051 erg, which is comparable to the energy of a supernova. The so-called fireball model suggests a GRB comes from an outflow with a Lorentz factor of Γ ∼ 100, so they have one of the fastest bulk flow velocities in the universe. A GRB likely originates from the collapse of a massive star into a neutron star or black hole. The duration of a GRB is much longer than the radial free-fall time for the stellar envelope, which suggests the stellar matter has a nonnegligible angular momentum and an accretion disk likely forms. The GRB is thought to be generated by a jet from the accretion of stellar matter onto a rapidly spinning neutron star or black hole. The jet is suspected to penetrate the stellar surface and, according to the standard model, the observed gamma-rays are produced in relativistic shocks within the jet. About half of all star systems have only one star. Some single-star systems contain a massive star that collapses into a neutron star or black hole, which eventually accretes all the stellar matter that could form a disk or material that falls back from the supernova. Some of these may continue to accrete interstellar matter or roam the universe undetected except by gravitational lensing and faint X-ray emission. However, about half of all star systems are binaries, and the neutron star or black hole can continue to accrete by capturing matter from the companion star. Such systems have a high X-ray luminosity, so are called X-ray binaries (for a review see Lewin et al. 1995). With a persistent power output up to 1036 − 1038 erg/s, X-ray binaries are the brightest persistent sources of X-rays in the sky. By comparison, the Crab pulsar, otherwise the brightest X-ray source in the sky, has an X-ray luminosity of ∼ 1037 erg/s. Many X-ray binaries, called microquasars, have a relativistic jet with Γ ∼ 3 − 10. Some neutron star X-ray binaries unstably burn hydrogen and helium leading to thermal X-ray bursts with a power output of 1036 −1039 erg/s, and some unstably burn carbon leading to thermal X-ray bursts with a power output of 1043 erg/s. The compact object in some X-ray binaries has a mass M ∼ 3 − 20 M¯, which suggests that they cannot be neutron stars, so they are dubbed black hole candidates (BHCs). None of these BHC X-ray binaries exhibit thermal X-ray bursts. This suggests they each have no surface and so likely each contain a black hole with an event horizon. An example black hole X-ray binary is Cygnus X-1, which harbors a compact object with M ∼ 7 − 13 M¯ (Webster and Murdin, 1972; Bolton, 1972; Gies and Bolton, 1986; Herrero et al., 1995).

xix Observations of a galaxy’s stellar dynamics, gas kinematics, or the galactic mass to light ratio 5 9 suggest that in most galaxies the center harbors a supermassive (M ∼ 10 − 10 M¯) black hole (for a review see Krolik 1999a). Such supermassive black holes likely formed from accretion onto, 2 4 or merging of, intermediate-mass black holes with M ∼ 10 − 10 M¯. The motions of stars near 6 our galactic nucleus, SgrA*, provide the best evidence for a black hole with M ∼ 2.6 × 10 M¯ (Eckart and Genzel, 1997; Ghez et al., 1998; Schödelet al., 2002). An active galaxy is a galaxy from which a significant fraction of the energy output is emitted by the nucleus rather than by the stars, dust, and interstellar gas. Such nuclei are called active galactic nuclei (AGN). AGN have a luminosity of 1042 − 1048 erg/s, which has long been considered to be generated by accretion onto the supermassive black hole. Observations of the active galaxy MCG-6-30-15 show a highly redshifted Fe Kα line emission profile with significantly different strengths for the red and blue wing of the profile. These features of the Fe Kα line profile are consistent with emission from a rapidly rotating v/c ∼ 0.2 accretion disk within a few gravitational radii (GM/c2) of a (possibly rotating) black hole (Pariev and Bromley, 1998; Tanaka et al., 1995). Accretion onto a black hole is the most efficient known process to convert gravitational energy into radiation. Some GRBs, some X-ray binaries, and all AGN are likely powered by accretion of plasma onto a rotating black hole. Models of the accreting plasma typically use the hydrodynamic (HD) approximation, or when including a magnetic field, the magnetohydrodynamic (MHD) approximation. Since these systems involve relativistic flow around a rotating black hole, the MHD approximation is solved using general relativity (GR) theory, which gives the GRMHD equations of motion. This thesis mostly involves the study of nonradiative accretion disks using the GRMHD approximation. Despite some useful time-independent and simplified HD, MHD, and GRMHD analytic solutions, a numerical model is likely required to study the global, time-dependent accretion flow. Typically, the time-dependent solution of accretion flow is only a quasi-steady state, where flow quantities can have large deviations from the time-averaged value. I developed a viscous HD code, a nonrelativistic MHD code, and a GRMHD code. The thin disk solution of Shakura & Sunyaev (SS) uses an unmagnetized viscous model for angular momentum transport, which is likely due to MHD turbulence (Balbus and Hawley, 1991). The SS model predicts that the accretion disk terminates at the innermost stable circular orbit (ISCO) of a black hole. The SS solution is often used to interpret observations of spectra from accretion disks, and the termination of the disk leads to a significant spectral feature. Some studies assume the SS solution to be accurate, and properties such as black hole spin, disk orientation, and the location of the disk inner-edge are derived. I used the nonrelativistic MHD and GRMHD codes to study magnetized accretion flow. Like others at the time, I found that the disk does not terminate at the ISCO, and magnetic fields continue to exert a torque on the disk inside the ISCO. The disk will continue to emit radiation inside the ISCO, altering the predicted spectra of accretion disks and altering those derived quantities (Reynolds and Begelman, 1997). For X-ray binaries and AGN, a more accurate spectral fit may suggest that the Fe Kα line emission is from farther out in the disk and that the black hole is not necessarily rotating.

xx Blandford and Znajek (1977) (BZ) argued that accretion flow around a black hole leads to evacuation of the polar regions and the development of a force-free magnetosphere. They found a solution, called the BZ-effect, which describes the extraction of spin energy of a rotating black hole by the magnetosphere. I used the GRMHD code to numerically study thick disks around a rotating black hole and found that an evacuated, nearly force-free magnetosphere develops as predicted by BZ. The BZ solution is remarkably accurate in this region for a black hole with a/M . 0.5, where a is the Kerr spin parameter. For models with a/M & 0.5, I find a mildly relativistic (Γ ∼ 1.5 − 3) collimated Poynting jet around the polar axis. However, additional physics is likely required to obtain Γ ∼ 100 as models predict in GRBs, and to obtain Γ ∼ 3 − 10 as seen in some microquasars and AGN. The Gammie (1999) inflow solution describes a cold, thin accretion flow around a rotating black hole. The Gammie solution predicts the radial dependence of comoving mass density, comoving magnetic energy density, and the accretion rate of energy and angular momentum per baryon. The Gammie solution also predicts accretion could occur super-efficiently (i.e. more energy is released than rest mass accreted), and that likely the efficiency is greater than that in a thin disk due to the magnetic field in the plunging region. I found the numerical GRMHD solution for the radial dependence of those quantities to be marginally consistent with the Gammie inflow solution. Discrepancies are likely due to the numerical solution consisting of hot disk, while the Gammie solution is for a cold disk. None of the numerical models I studied showed super-efficient accretion. In fact, for most thick and thin disks, the energy per baryon accreted closely followed the thin disk efficiency found by Bardeen (1970). I designed and constructed a Beowulf cluster of computers to help develop the VHD, MHD, and GRMHD codes and use these codes to perform numerical simulations of accretion flow. I describe how to design, build, and test a Beowulf cluster. The main conclusion is that one should test one’s own code on test nodes and construct a test cluster before purchasing the entire cluster. Performance and reliability vary greatly between personal computers (PCs) and server computers (SCs). The main performance difference is due to the PCI bus, on which both add-on and built- in network chips operate. The network chip and the PCI bus are typically the bottleneck in the performance of parallel calculations. Today’s PCs are all composed of 32-bit 33Mhz-based motherboards, while SCs are composed of 64-bit 66/133Mhz-based motherboards. A PC is about 1/2 the price of an SC, so PCs are an attractive option. However, for a cluster with more than only about 4 nodes, a cluster of PCs costs more per unit performance than a cluster of SCs. Since some or all black hole accretion systems could harbor a thin disk, I plan to study thin disks and to determine how the BZ luminosity depends on disk thickness. A few groups have studied GRMHD models of jets, and all fail to show Γ factors as high as in observed jets. I plan to study other mechanisms for jet acceleration and collimation, and I plan to study the connection between the disk and jet. I also plan to include the relevant microphysics to the GRMHD model to study the accretion disk that likely forms during a GRB.

xxi 1 Observations, Theory, and Models

This chapter summarizes the observations, theory, and models of black hole accretion disk systems. Section 1.1 outlines this introduction.

1.1 Summary of Introduction

Against the force of gravity, a spheroidal star is mostly supported by pressure forces, while a thin disk is mostly supported by centrifugal, or even magnetic, forces. The formation of a disk typically occurs because radiative energy is released faster than angular momentum is transported. The disk can be described as a result of the conservation of mass, energy, and momentum in the gravitational potential. As the disk matter falls in the gravitational potential, collisions allow the gravitational potential energy to be converted into radiative energy (instead of kinetic energy). Such a disk is known as an accretion disk. The entire structure of the disk and surrounding dynamically-coupled medium is called the accretion flow. The gravitating body in some accretion disk systems is a planet (held up by gas or matter pressure), young stellar object (YSO) (held up by gas or radiation pressure), white dwarf (star held up by electron degeneracy pressure), or neutron star (NS) (star held up by neutron degeneracy pressure). This thesis is primarily concerned with those accretion disk systems that contain a black hole. For a review, see Shapiro and Teukolsky (1983). Accretion disk models often use the fluid approximation to describe the underlying microphysics of ionized media. The fluid approximation assumes that the smallest region of interest contains a large number of particles and that particle collisions are frequent enough to sustain a statistical equilibrium state. A magnetized fluid is often modeled by the magnetohydrodynamic (MHD) approximation. The typical MHD approximation assumes the fluid is ionized, can be treated as a so-called weakly-coupled plasma, and particles have negligible gyration radii. A weakly-coupled plasma is one that contains both positive and negative charges, but is effectively neutral over a distance that spans many particles. A simplified form of the MHD approximation is the ideal MHD approximation, which assumes that the plasma is perfectly conducting, currents arise instantaneously from fields, the Hall effect is negligible, electronic pressure gradients are negligible, and the pressure is isotropic (see, e.g., Krall and Trivelpiece 1973). A further simplification is the single-component ideal MHD approximation, which assumes all species are in statistical equilibrium within a fluid element and that there is a well-defined average density, velocity, etc. for the fluid element. For a gas of protons and electrons, the single-component approximation involves neglecting the electron inertia. For some fluids, some of these approximations are coupled to the same underlying approximation. For example, if the electron inertia is negligible, then currents

1 arise instantaneously from fields for a gas of protons and electrons. Throughout the thesis, the accretion flow is assumed to be well-described by the single-component ideal MHD approximation, but this will be tested near the end of this chapter. This thesis considers both the nonrelativistic and general relativistic ideal MHD approximations to describe the black hole accretion flow. To provide context for the accretion disk theory to be presented later, Section 1.2 gives a summary of some astrophysical objects that probably contain an accreting black hole, such as gamma-ray bursts (GRBs), X-ray binaries, a source in the Galactic nucleus called SgrA*, and active galactic nuclei (AGN). That section introduces the basic observed and implied features of these objects and also introduces 6 fiducial black hole accretion disk systems: 2 GRBs, 1 X-ray binary, SgrA*, and 2 AGN. The purpose of the discussion is to provide context for the accretion disk theory discussion that follows and to motivate general relativistic magnetohydrodynamic (GRMHD) models of these systems by summarizing the evidence for black holes and magnetized relativistic motion. Later in this chapter, accretion disk models are developed for these 6 fiducial systems. These models are used to determine some accretion flow properties, such as the mass density and temperature of the disk. Also later, the validity of the fluid, MHD, and ideal MHD approximations as a model for the accretion flow in these 6 fiducial systems is tested. Since the inner region of the accretion disk is typically not resolved by current telescopes, the theory of this part of the accretion disk often rests on a spatially-unresolved flux of photons. This flux can be decomposed into a luminosity, time variability, and spectra of the disk system. Sec- tion 1.3 outlines the basic theory of disks around compact stars (such as black holes) by discussing the luminosity, time variability, and some spectral features of accretion disks. The section shows how the luminosity and time variability independently provide an estimate for the mass of the accreting compact object. The section discusses how some spectral features, such as the Fe Kα line profile, may even provide details about the spatial structure of the inner disk in some X-ray binaries and some AGN (Karas et al., 1992; Tanaka et al., 1995; Pariev and Bromley, 1998; Miller et al., 2004). The purpose of this section is to provide accretion theory-based arguments that suggest some of the 6 fiducial objects, and those like them, are likely powered by an accretion disk that extends close to a black hole. An element of a disk accretes only by losing angular momentum. Since angular momentum is conserved, other elements must gain angular momentum and so move outward. Thus, for a disk to accrete, angular momentum must be transported outward. The first models of accretion disks assumed some form of magnetic or hydrodynamic turbulence drives angular momentum transport (Shakura and Sunyaev, 1973). The turbulence is often phenomenologically modeled as an effective shear viscosity that causes differentially-rotating layers to exchange angular momentum (Novikov and Thorne, 1973; Eardley and Lightman, 1975; Pringle, 1981). While a hydrodynamic instability may generate turbulence in a disk, no known hydrodynamic instability drives sufficiently vigorous angular momentum transport to account for observations of astrophysical accretion disks (for a review, see Balbus and Hawley 1998). Later, it was realized that magnetized accretion disks are naturally unstable, and that the so-called magnetorotational instability (MRI) could drive angular

2 momentum transport and account for accretion in astrophysical disks (Balbus and Hawley, 1991). The first part of Section 1.4 summarizes the effective viscosity and MRI models for the transport of angular momentum in a highly ionized accretion disk. The purpose of that section is to motivate the use of a magnetic, rather than viscous, model of the angular momentum transport. Radiative processes are self-consistently included in accretion disk models that seek to account for the observed spectra and luminosity of accretion disk systems. Some radiative processes, such as Compton scattering or bremsstrahlung, are involved in determining the structure of the disk and in producing a significant observational feature. Other radiative processes, such as Fe Kα line fluorescence, are weakly involved in determining the disk structure, but are important observational features from disks. Some radiative processes, such as synchrotron radiation from non-thermal particles, require physics beyond the thermodynamic equilibrium approximation. The second part of Section 1.4 discusses radiative accretion disk models that self-consistently determine disk structure. The purpose of that section is to provide a summary of previous work and to provide estimates of the rest-mass accretion rate for the 6 fiducial systems. Ultimately, radiative models are required to compare with observations, however a nonradiative GRMHD model may be sufficient to describe the mechanism of jets in black hole accretion systems. This thesis focuses on testing the plausibility of the Blandford-Znajek (BZ) effect (Blandford and Znajek, 1977) as a mechanism to generate relativistic jets, rather than focusing on the mechanisms involved in determining observational spectra from the disk. The last part of Section 1.4 discusses GRB models and the likelihood of the BZ-effect powering a GRB. GRBs are discussed more ex- tensively than X-ray binaries and AGN since the nonradiative GRMHD models, as developed in this thesis, more readily apply to the accretion disk that forms in the so-called “collapsar” model of a GRB (Woosley, 1993; Paczyński,1998; MacFadyen and Woosley, 1999). That is, it is later shown that the collapsar GRB model likely does not need a detailed radiative transport model for photons and neutrinos. Section 1.5 estimates various quantities, such as mass density, temperature, velocity, magnetic field, and other quantities for the 6 fiducial systems. No radiative GRMHD model, with a realistic equation of state, has been developed or studied numerically. Therefore, a combination of models is used and checked for consistency. Section 1.5 uses the results of nonradiative GRMHD numerical models (as in Chapter 4), unmagnetized radiative neutrino-cooled analytic models (as in Popham et al. 1999), and unmagnetized photon-cooled radiative analytic models (as described by Shakura and Sunyaev 1973; Novikov and Thorne 1973) of accretion disks to make all the estimates. GRB systems are approximated by a radiative ideal Fermi gas equation of state, while X-ray binary and AGN systems are approximated by an ideal gas + photon radiation pressure equation of state. As a consistency check of the estimates for the 2 GRB systems, the results of the nonradiative, ideal gas GRMHD numerical model are compared to the radiative, ideal Fermi gas model. A summary of the radiative Fermi model is provided in the first part of Appendix A. The purpose of the section is to provide estimates of the various quantities that help to provide an intuition about these systems. Section 1.5 also estimates the validity of the fluid, MHD, and ideal MHD approximation for the

3 6 ﬁducial systems. These estimates are as discussed in any plasma physics book, and summarized in Appendix A. The purpose of the section is to show that the single-component ideal MHD approximation is an excellent model1 for the 6 ﬁducial (and likely all) black hole accretion systems. Section 1.6 summarizes the properties of GRBs, X-ray binaries, SgrA*, and AGN that suggest a GRMHD model is required to study the accretion disk near the black hole in these systems. Some open questions are posed about the source of the luminosity and jets in these systems, which may be resolved by a GRMHD model. Section 1.7 reviews the main results of chapters 2-5 in this thesis.

1.2 Introduction to Black Hole Systems

Einstein (1916) formulated his general relativity (GR) theory, which has stationary solutions called black holes (for a discussion see, e.g., Shapiro and Teukolsky 1983, Chapt. 12). Einstein’s GR theory today remains the standard and most fundamental theory of gravity (see, e.g., Misner et al. 1973, and for a historical perspective see Thorne 1994). Schwarzschild (1916) derived an exact, static, and spherically symmetric solution to Einstein’s equations. A generalization of the Schwarzschild solution for a stationary, charged2, rotating, space-time is the Kerr-Newman metric (see, e.g., Shapiro and Teukolsky 1983, Chapt. 12, for a discussion). The Schwarzschild and Kerr-Newman solutions to Einstein’s equations define the properties of stationary, classical black holes as studied in this thesis. This thesis focuses on black holes as the object at the center of the accretion disk. The first discussion in this section summarizes the plausible methods for forming a black hole as a remnant of the collapse of a massive gas cloud or a massive star, critical mass build-up on a degenerate star due to accretion, or the collision between two compact objects. The types of astrophysical systems involved in such black hole formation scenarios include 1) gamma-ray bursts, which each likely contain a black hole with M ∼ 3 M¯; 2) black hole X-ray binaries, which each likely contain a black hole with M ∼ 3 − 20 M¯ ; and 3) galaxies, some of which contain one or 6 9 33 more super-massive black holes (SMBHs) with M ∼ 10 − 10 M¯, where M¯ = 1.989 × 10 g is the mass of the Sun. The second discussion describes the properties of gamma-ray bursts, X-ray binaries, and active galactic nuclei, and introduces 6 fiducial black hole accretion disk systems: 2 gamma-ray bursts, 1 X-ray binary, a radio source at the center of our galaxy called SgrA*, and 2 active galactic nuclei. These 6 fiducial systems are used as examples of black hole accretion systems for several calculations, such as those in Section 1.5. Table 1.1 summarizes the estimated distance to, black hole mass of, and bolometric luminosity of the 6 fiducial objects described in this section. Throughout this thesis, similar tables are presented that cumulate additional quantities as relevant to the discussion in each section. A summary description of these objects/events is given below.

1This is shown to be true everywhere except in current sheets. 2Charged black holes are likely not astrophysically relevant. Any buildup of charge is quickly neutralized by accretion and pair production.

4 Table 1.1. Fiducial Black Hole Accretion Systems 1

Parameter NS-BH GRB 030329 LMC X-3 SgrA* NGC4258 M87

dL ? 804 Mpc 55 kpc 8 kpc 7 Mpc 18 Mpc 6 7 9 M[M¯] 3 3 10 2.6 × 10 4 × 10 3 × 10 53 52 38 37 43 42 Lbol[erg/s] 10 3 × 10 10 10 3 × 10 2.3 × 10

Note. — dL is the luminosity distance, M is the mass in solar mass units, and Lbol is the bolometric luminosity. These quantities are shown for the 5 ﬁducial black hole accretion disk systems. See text for references.

Some of the objects discussed in this section have their distance from the Sun determined by the observed redshift (z = λobs/λemit − 1, where λ is the photon wavelength) of spectral lines. For these objects, the distance is found by integrating the Friedman-Robertson-Walker (FRW) metric to determine the comoving distance (dC ) that gives a luminosity distance of dL = (1+z)dC . The power law ΛCDM model and WMAP (and related) data give a Hubble constant h ∼ 0.71 km s−1 Mpc−1,

Ωm ∼ 0.27, and ΩΛ ∼ 0.73 for a ﬂat (k = 0) FRW metric model of the cosmos (Spergel et al., 2003).

1.2.1 Formation of Black Holes

Subrahmanyan Chandrasekhar found that if a white dwarf (an accepted object at the time) is suﬃciently massive, then the degenerate electrons will become highly relativistic and can drive the star to collapse (Chandrasekhar, 1931b,a). He predicted that a white dwarf would undergo collapse if the mass reaches Mch ∼ 1.4 M¯. The collapse of a white dwarf can lead to a neutron star supported by neutron degeneracy pressure. Eddington (1935) realized that if Chandrasekhar 3 was correct, then a star with a mass much larger than 1.4 M¯ should collapse to a black hole .

Modern estimates of the upper limit to the mass of a neutron star give M ∼ 1.8−2.2 M¯ (Akmal et al., 1998). If one only requires GR and causality to hold, then the upper limit is M ∼ 3.4 M¯ (Rhoades and Ruffini, 1974; Hartle, 1978), but see (Yuan et al., 2004; Abramowicz et al., 2002). Regardless of the equation of state, once an object is contained within a light trapping surface, GR theory predicts that the result must be a black hole (Hawking and Ellis, 1973). A classical black hole is a simple, conservative model, while alternatives invoke exotic neutron star physics (Bahcall et al., 1990) or modifications to GR in the strong-field limit (see, e.g., Babak and Grishchuk 2003; DeDeo and Psaltis 2003). This thesis considers a compact object with a mass greater than about

M ∼ 3.4 M¯ to be a black hole candidate (BHC).

3The term black hole was coined later by Wheeler in 1968.

5 There are two likely ways of creating a “stellar mass black hole” with M ∼ 3−20 M¯: 1) stellar core-collapse; and 2) compact object collisions. Collapse of the iron core of a massive star can lead to a black hole or neutron star remnant and the generation of a supernova (for a review see Woosley et al. 1993; Wheeler et al. 2000). Black hole formation may also occur during the collision of two neutron stars (Eichler et al., 1989), or during other collisions involving a compact object (Popham et al., 1999). Isolated solar-mass black holes have been indirectly detected by micro-lensing events (Bennett et al., 2002; Agol et al., 2002; Agol and Kamionkowski, 2002). Black holes are found with normal (non-compact) stellar companions in binary systems (for a review see, e.g., Lewin et al. 1995; McClintock and Remillard 2003), which are called black hole X-ray binaries due to their powerful X-ray emission. The black hole in the binary likely results from the collapse of a massive star, while the other star remains intact. The initial mass function (IMF) ξ(M) speciﬁes the distribution in mass of newly formed stellar populations, and ξ(M) can be used to estimate the mass and number of black holes compared to other stars. From the observed dependence of bolometric luminosity on star mass, Salpeter (1955) suggested that ξ(M) ∝ M −(1+x), where x = 1.35 and the distribution extends from a lower bound of M1 ∼ 0.1 M¯ to an upper bound of M2 ∼ 125 M¯ (see also Miller and Scalo 1979; Scalo 1986).

The present-day stellar IMF is now thought to extend up to M ∼ 200 M¯ (Larson, 2002) and studies of population III (low-metallicity) stars suggest the IMF is probably top-heavy (Schneider et al., 2002). Stars with zero metallicity have little mass loss, so massive low-metallicity stars may collapse to massive black holes. Numerical simulations predict there should exist stars with M > 100 M¯

(Abel et al., 2000, 2002). Other studies suggest the first stars with M < 140 M¯ should have mass loss like normal metallicity stars (Fryer, 1999). For stars with 140 M¯ < M < 260 M¯, an electron-positron pair instability during oxygen burning leaves no remnant. For M > 260 M¯, the star collapses directly to a black hole with a remnant mass about half the original stellar mass. For 5 M ∼ 10 M¯, no stable hydrogen burning occurs and the protostar collapses directly to a black hole (Baumgarte and Shapiro, 1999; Shibata and Shapiro, 2002). 6 Cold dark matter (CDM) models suggest the first generation baryonic clouds have M ∼ 10 M¯ (Bromm et al., 2002). Such clouds are suggested to fragment into stars. However, a small fraction may suffer direct collapse to a black hole. If one thousandth of the initial cloud mass collapses directly to a black hole, then there would exist intermediate-mass black holes (IMBHs) with masses 2 4 M ∼ 10 − 10 M¯ (Balberg and Shapiro, 2002; Shapiro and Shibata, 2002). This is consistent with the observed correlation between the mass of the central SMBH and the mass of the bulge of −3 a galaxy, which is MBH /Mbuldge ∼ 10 (Kormendy and Gebhardt, 2001). Hierarchical growth of galaxies through accretion of satellite galaxies leads to IMBHs in off-center positions, as confirmed by numerical simulations of dense star clusters (Portegies Zwart et al., 1999; Portegies Zwart and McMillan, 2002; McCrady et al., 2003). There are so-called ultraluminous X-ray sources (ULXs) that are observationally similar to black hole X-ray binaries, but ULXs have a higher luminosity of L ∼ 1039 − 1041 erg/s. ROSAT, ASCA, and Chandra observations show that about 50% of spiral galaxies have ULXs that are

6 more compact and variable than AGN (Fabbiano et al., 2003), off-center from the galactic center, and associated with star-forming regions. ULXs are also observed in globular clusters in elliptical galaxies (Kaaret et al., 2001). A ULX is found in, for example, the center of irregular galaxy M82 (Cigar galaxy) (Matsumoto et al., 2001). ULXs could correspond to accreting IMBHs. Simple 2 3 luminosity constraints suggest M & 10 − 10 M¯, dynamical friction constraints imply M < 6 3 4 10 M¯, and the velocity cusps in globular clusters imply M ∼ 10 − 10 M¯ (Van Der Marel, 2004). However, if the emission is beamed or time-dependent, these ULX sources could be stellar- mass black holes (King et al., 2001). ULXs may be explained as IMBHs confined to the disk of the host galaxy that accrete interstellar matter (Krolik, 2004), although see Rappaport et al. (2004). Whether ULXs are actually accreting IMBHs or stellar mass black holes is an active area of research. The formation of a SMBH is a likely result of star formation (Rees, 1984; Genzel et al., 1997). The formation of SMBHs could be due to one or more IMBHs that fall to the galaxy center due to dynamical friction (Haiman and Loeb, 2001). A scenario for creating a SMBH involving the collision of multiple IMBHs may be unable to account for all SMBHs today (Islam et al., 2003). This scenario would result in little black hole spin to power radio jets (Hughes and Blandford, 2003; Gammie et al., 2004), so if the scenario is accurate then black hole spin would not power jets. A single IMBH seed could be driven to a supermassive mass by accretion of gas. This accretion scenario generates the observed correlations (Kormendy and Gebhardt, 2001) between black hole mass and bulge mass or bulge velocity dispersion (Van Der Marel, 2004). However, recent evidence of a SMBH binary in radio galaxy 3C66B is consistent with the hierarchical growth of galaxies (Sudou et al., 2003). Thus, both hierarchical (merger dominated) and anti-hierarchical (gas accretion dominated) scenarios may have occurred. The preceding discussion summarized how black holes form with different masses, ranging from 6 9 stellar mass black holes with M ∼ 3 − 20 M¯ to SMBHs with M ∼ 10 − 10 M¯, and some basic observational evidence was given for the existence of these black holes. The following discussion summarizes the evidence for the existence of black holes and accretion disks in specific events/objects. The purpose of the following discussion is to suggest that some gamma-ray bursts, some X-ray binaries, the centers of some normal galaxies, active galactic nuclei, and quasars are likely powered by an accreting black hole.

1.2.2 Gamma-Ray Bursts

Gamma-ray bursts (GRBs) are brief bursts of X-rays or gamma-rays that are observed a couple times a day, last from seconds to hours, appear randomly distributed in the sky, and do not repeat (for a review see Piran 1999, 2004). The Vela satellites detected the ﬁrst GRB in 1967. Not until 1973 were reports of GRBs declassiﬁed and published (Klebesadel et al., 1973). In 1990, the Burst and Transient Source Experiment (BATSE), onboard the Compton Gamma-Ray Observatory, localized several hundreds of bursts. Their random distribution on the sky suggested a

7 cosmological origin (Paczyński,1986; Meegan et al., 1992). However, not until 1997 did BeppoSAX detect GRB970508 with an optical counterpart (afterglow) with emission lines showing a redshift of z = 0.835 (dL = 5300 Mpc) (Metzger et al., 1997; Frail et al., 1997), demonstrating a cosmological origin. Approximately two GRB detections occur per day. Possibly related events called X-ray flashes (XRFs) are detected several times a year and exhibit no gamma-ray emission (Heise et al., 2001; Yamazaki et al., 2002). GRB durations form a bimodal distribution of short-duration GRBs lasting an average of δT ∼ 0.3 s and of long-duration GRBs lasting on average of δT ∼ 35 s (Kouveliotou et al., 1993). The shortest GRB currently ever observed lasted for δT ∼ 6ms (Bhat et al., 1992) and had 200µ s temporal structure, while the longest GRB (actually an XRF) currently ever observed lasted for 2550 s (0.7 hr) (see review by Zand et al. 2003). Short-duration GRBs are observed to have a harder spectrum compared to long-duration GRBs (Paciesas et al., 1999). Some long-duration GRBs have an associated X-ray to radio afterglow emission that contains spectral lines that allow a redshift measurement, which indicates these GRBs occur at cosmological distances. The typical redshift of a GRB is z ≈ 1 (dL = 6600 Mpc) (Mészáros,2001) and has been measured up to z ∼ 4.5

(dL = 43000 Mpc) (Andersen et al., 2000). No afterglow, and hence redshift, has been observed for short-duration GRBs. The GRB emission is likely relativistically beamed, but the afterglow redshift can be used to 2 −1 estimate the distance and isotropic rest-frame energy Eiso(γ) = 4πFγdL(1 + z) and isotropic 2 rest-frame luminosity Liso(γ) = Eiso(1 + z)/δT = 4πFγdL, where Fγ is the observed fluence of gamma-rays (see, e.g., Frail et al. 2001). For example, GRB 990510 had a 50 − 300 keV fluence −5 2 of Fγ = 3 × 10 erg/ cm over a period of δT ∼ 33 s (Kippen, 1999) and an associated optical afterglow with FeII and MgII absorption lines with redshift z = 1.619 (Vreeswijk et al., 1999). This 52 gives dL ∼ 7600 Mpc. Assuming an isotropic emission, the total energy is Eiso(γ) ∼ 5 × 10 erg 51 and total luminosity is Liso(γ) ∼ 4.2 × 10 erg/s. “Cosmic GRBs” at redshift z & 0.5 are possibly 51 standard candles with an actual total energy production of about Eγ ∼ 10 erg (Sari et al., 1999; Frail et al., 2001; Panaitescu and Kumar, 2001; Bloom et al., 2003), although there are significant 48 50 outliers with Eγ ∼ 10 − 10 erg (Sazonov et al., 2004; Soderberg et al., 2004). In most GRB models, the GRB is suspected to be produced from a relativistic jet that was emitted, by some mechanism involving accretion, near the polar axis of a black hole accretion disk system. The “collapsar” model scenario, used to explain long-duration GRBs, entails the collapse of a massive star and formation of an accretion disk around a newly-formed black hole of mass M ∼ 3 M¯ (Woosley, 1993; Paczyński,1998; MacFadyen and Woosley, 1999). Short-duration GRBs may be generated by the collision of a neutron star and black hole, which subsequently forms an accretion disk around the rotating black hole (Narayan et al., 1991, 1992). This introduction later estimates the properties of a disk that could form during a neutron star - black hole (NS-BH) collision and during the collapse of a massive star, such as what likely occurred in GRB 030329.

8 NS-BH Collision Model

There are no afterglows, and so no redshift measurements, associated with short-duration GRBs. Thus, these events have no estimated isotropic luminosity that could provide a clue about the nature of the progenitor. One model for short-duration GRBs is a NS-BH collision, which is estimated to have an isotropic luminosity of about 1053 erg/s and last less than a second (Narayan et al., 1992; Popham et al., 1999).

GRB 030329

The prompt gamma-ray emission from GRB 030329 lasted for about δT = 25 s (Vanderspek et al., 2003), and within hours was followed by an afterglow observed from X-ray, UV, optical, infrared, and radio emissions. Observations of a supernova light curve (SN2003dh) in the afterglow of GRB 030329 confirmed that the collapse of a massive star is spatially and temporally connected to the GRB (Stanek et al., 2003; Kawabata et al., 2003; Uemura et al., 2003; Mészáros,2003;

Hjorth et al., 2003). Emission lines in the afterglow have a redshift of z = 0.169 (dL = 804 Mpc) −4 −2 (Greiner et al., 2003b). The GRB ﬂuence within 30 − 400 keV was Fγ ∼ 10 erg cm , which 51 50 gives Eiso(γ) ∼ 6 × 10 erg and Liso(γ) ∼ 3 × 10 erg/s. The likely total beamed energy is 49 ◦ Eγ ∼ 2 × 10 erg in a beam with angular width ∼ 3.5 (see, e.g., Greiner et al. 2003a), which is an outlier in a GRB Eγ histogram.

1.2.3 Black Hole X-ray Binaries

Observations indicate that 50-80% of stars are in multiple star systems, with most of these being binary stellar systems (for a review see Evans 1999; Tohline 2002). About half of all stars are in binary systems, and some contain an accreting compact object (likely a neutron star or black hole) and have strong emissions of X-rays that likely originates from an accretion disk (for a review see, e.g., Lewin et al. 1995; McClintock and Remillard 2003). X-rays are emitted mostly from the inner region of the accretion disk near the collapsed star (see, e.g., Shapiro and Teukolsky 1983, Chapt. 13 for discussion). X-ray binaries have a persistent X-ray luminosity in the range of L ∼ 1032 − 1038 erg/s, where the companion star has a bolometric luminosity in the range of 1032 − 1038 erg/s. An X-ray binary is a true binary system, which is resolved as a binary by observing the motion of the normal star due to the gravitational inﬂuence of the compact object. The motion is detected by 1) observing the wobble motion of the normal star on the celestial sphere ; 2) spectroscopic observations of the light-of-sight velocity ; or 3) the eclipsing of one object by another. The mass of the compact object in X-ray binaries is often determined by the mass function

3 3 Mc sin i Porbvn f(Mc,Mn, i) = 2 = , (1.1) (1 + Mn/Mc) 2πG where Mc is the mass of the compact object, Mn is the mass of the normal star, i is the inclination

9 of the orbit with respect to the line of sight, Porb is the orbital period of the binary, and vn is the line of sight velocity. A lower limit to Mc can be found by setting Mn = 0 and i = π/2, while often other secondary measurements allow higher lower limits on the mass of the compact object. The distribution of the compact object’s estimated mass in X-ray binaries shows a clear bimodal distribution with about 22 objects with mass 1.1 − 2 M¯ and a distribution of about 18 objects with masses from 3 − 16 M¯ with a mean and peak of 10 M¯ (see, e.g., Zand et al. 2004; Postnov and Cherepashchuk 2004, ﬁgure 1, and see McClintock and Remillard 2003).

Classes of X-ray Binaries

There are approximately 240 known X-ray binaries that are classified according to their supposed underlying physics (Zand et al. 2004 and references therein). These classifications include high-mass X-ray binaries (HMXBs), low-mass X-ray binaries (LMXBs), Be X-ray binaries, X-ray bursters, X-ray pulsars, soft X-ray transients (SXTs), and microquasars. A neutron star is the compact object in some HMXBs, some LMXBs, all Be X-ray binaries, all X-ray bursters, all X-ray pulsars, some SXTs, and some microquasars. A black hole is the compact object in some HMXBs, some LMXBs, some SXTs, and some microquasars. A single object can be in multiple classes. X-ray binaries are likely powered by the accretion of matter from the normal star onto the compact star. The formation of the accretion disk can be described by two limiting scenarios associated with either HMXBs or LMXBs. HMXBs are X-ray binaries with a massive M ∼ 10 − 30 M¯ stellar companion, such as a Be (B-type with prominent emission lines) star or blue supergiant. An accretion disk likely forms from the companion’s stellar wind material. LMXBs are X-ray binaries with a low-mass M . 5 M¯ stellar companion, which is typically a main sequence solar mass star. In some cases the companion star is degenerate or evolved (subgiant or red giant). An accretion disk forms as the companion overfills its Roche lobe (Lagrange gravitational equipotential surface between the compact and normal star) and matter plunges onto the compact object. For a catalogue of LMXBs see Liu et al. (2001). The spatial distribution in our galaxy of X-ray binaries shows about 90 LMXBs with a slight concentration in the center, while HMXBs are more evenly distributed (Grimm et al., 2002). SXTs are LMXBs that are only discovered after undergoing a so-called accretion outburst (also called an X-ray nova) (Tanaka and Shibazaki, 1996). Typical SXTs are usually faint or unobservable in X-rays during a “quiescent” state. Typical SXTs contain a K-type subgiant or dwarf that is transferring mass to a black hole through an accretion disk. It is thought that during “quiescence,” mass is accumulating in an accretion disk. When an outburst occurs, most of disk falls into the compact object. The accretion outburst likely occurs due to some type of temperature dependent disk instability. A similar mechanism operates in dwarf novae (Cataclysmic Variables (CVs)) (Warner, 1995). Galactic microquasars (Mirabel et al., 1992) are X-ray binaries that are probably powered by a black hole accretion disk system or an exotic mechanism for energy emission from a neutron star. Galactic microquasars are named for their strikingly similar features to quasars or AGN

10 (Mirabel and Rodr´ıguez,1994, 1999; Fender and Belloni, 2004), such as apparently superluminal (v/c ∼ 3 − 10) jets, which plausibly originate from a relativistic system with a well-deﬁned axis of rotation. Microquasars have a higher X-ray luminosity than other X-ray binaries. Unlike other X-ray binaries, microquasars might also be sources of gamma-rays (Romero, 2004). AGN and microquasars are likely more than simply morphologically similar (Rees, 1998), since they may both contain an accreting rotating black hole. This thesis later estimates the properties of the disk around the black hole in LMC X-3 described below. A description of the classic galactic microquasar GRS1915+105 and the classic black hole X-ray binary Cygnus X-1 is also given below.

LMC X-3

For a review of LMC X-3 see Cowley (1992). Large Magellanic Cloud (LMC) X-3 is in the LMC about 55 kpc from the Sun. LMC X-3 likely contains a B5 subgiant with mass 4− 8 M¯ in a 1.7day orbit around a compact object, so LMC X-3 is often classified as a HMXB. The normal companion star’s shape is severely distorted by the compact object, which is likely a ∼ 10 M¯ mass black hole 38 (Cowley et al., 1983). The X-ray luminosity of LMC X-3 is Lx ∼ 10 erg/s (White and Marshall, 1984; Treves et al., 1988). Despite being classified as a HMXB, recent XMM-Newton observations suggest that the disk is likely formed by Roche-lobe overflow (Soria et al., 2001). As with many X-ray binaries, the observed disk flux of hard or soft X-rays shows periodic behavior (Cowley et al., 1991; Ebisawa et al., 1993) and transitions between hard and soft emission (Wilms et al., 2001a).

GRS1915+105

A classic galactic microquasar is GRS1915+105 (V1487 Aql), which is about 12.5 kpc away from the Sun. The companion star is a late-type giant with M ∼ 0.8 ± 0.5 M¯, so is also classiﬁed as a LMXB (Greiner et al., 2001b; Harlaftis and Greiner, 2004). GRS 1915+105 is the most luminous of all known X-ray binaries (Done et al., 2004), with a highly variably X-ray output of ∼ 1038 − 1040 erg/s and shows emission from apparently superluminal jets (Mirabel and Rodr´ıguez, 1994, 1999; Fender and Belloni, 2004). Radio synchrotron emission from the jet shows plasma blobs moving at apparently superluminal speeds with an estimated true speed of v/c ∼ 0.92. The black hole mass is suspected to be about 14 M¯ (Greiner et al., 2001a).

Cygnus X-1

Cygnus X-1 (Cyg X-1) is a source about 2.6 kpc away in the Cygnus constellation. The optical counterpart is a supergiant called HDE 226868 which is an O9-B0 supergiant with a surface tem- 4 perature of 3.1 × 10 K and a mass of 20 − 30 M¯, so Cygnus X-1 is classiﬁed as a HMXB. Cyg X-1 is the ﬁrst object that was broadly agreed to require the existence of a black hole. The compact object is believed to be a black hole since it has a mass of 7 − 13 M¯ (Webster and Murdin, 1972; Bolton, 1972; Gies and Bolton, 1986; Herrero et al., 1995).

11 1.2.4 Normal and Active Galactic Nuclei and Quasars

All the stars from a normal (non-active) galaxy together have a typical luminosity of 1044 erg/s. Normal galaxies derive their luminosity from an ensemble of stars and so are, to zeroth order, related to a black body spectrum with a small range of strong emission. An active galaxy is a galaxy from which a significant fraction of the energy output is emitted by the nucleus rather than by the stars, dust, and interstellar gas. The luminosity of such nuclei is typically between 1042 − 1048 erg/s. Some nuclei outshine their host galaxy by 4 orders of magnitude. The nuclei in such active galaxies are called active galactic nuclei (AGN) (see review by Krolik 1999a). The distance from the Sun to a typical active galaxy is less than a Gpc (z < 0.2), which is close enough to resolve the stellar distribution and sometimes the individual stars. AGN come in many different types, which are largely a result of viewing angle (for a discussion of AGN unification models see, e.g. Krolik 1999a). Unlike normal galaxies, most AGN emit in a broad-band spectrum. For example, NGC 4151 is observed to have a flat spectrum from the mid-infrared (1013Hz, 0.04 eV) to the hardest X-rays observed (1019Hz, 41 keV). The active galaxy Markarian 421, known as a blazar, shows dramatic variability with flares reaching into the TeV range (Punch et al., 1992). AGN are observed to have spectra with emission lines such as the Lyα, Balmer lines, and the X-ray Fe Kα line near 6.4 keV. These lines are Doppler-broadened from relatively broad lines (several thousand km s−1) to relatively narrow lines (few hundred km s−1). For example, MCG-6-30-15, known as a Seyfert I, exhibits Fe Kα line fluorescence, whose broad, blue-shifted profile likely indicates emission from deep within the gravitational potential of a supermassive black hole (Tanaka et al., 1995). See Krolik (1999a) for a discussion of AGN spectra features and their likely origins. Within AGN, there are many intermediate absorbers and emitters between our line of sight and the accretion disk that significantly affect the observations. For example, the broad-line and narrow-line regions in AGN are an interesting study alone (see, e.g., Krolik 1999a; Elvis 2000, and references therein). Some of the first observations of AGN were of so-called radio galaxies with jets emerging from a point-like source. A typical jet has lobes that extend a few hundred kiloparsecs, a distance that is comparable to or larger than that of the host galaxy size (see, e.g., the classic review by Bridle and Perley 1984). AGN jets are observed to be relativistic with some having variable structure moving at an apparent speed of v/c ∼ 3 − 10 (see, e.g., West et al. 1998). AGN jets are sources of synchrotron radiation (Baade, 1956), indicating the existence of a strong magnetic field. The radio synchrotron radiation is likely due to internal shocks that produce high-energy electrons in the jet magnetic field. Recent results also find that X-rays can be produced as the AGN jet collides with external matter and produces knots within the jet (Hardcastle et al., 2003; Kraft et al., 2003). These structures are often tracked to measure the apparent velocity, which is typically superluminal. Jet motion in quasar 3C279 has been followed over a period of several years (Wehrle et al., 2001), and the jet has apparent velocities of 4.8 − 7.5c. The quasar 3C273 has an apparently superluminal

12 v/c ∼ 10. HST images of the jet in M87 show well-defined knotty structures (Bridle et al., 1994), which move at an apparently superluminal v/c ∼ 6. 6 9 AGN are likely powered by accreting SMBHs of mass M ∼ 10 − 10 M¯, as estimated from 1) the AGN’s compactness and luminosity ; 2) kinematic models of stellar motion (rotation profile and velocity dispersion) near nuclei ; 3) observations of Doppler shifts or maser emission from disk (see review by Kormendy and Richstone 1995; Richstone et al. 1998) ; 4) reverberation mapping (Blandford and McKee, 1982; Peterson, 1993; Peterson et al., 2004) ; and 5) relativistic effects for some AGN (Fabian and Vaughan, 2003). Using Hubble Space Telescope (HST) photometry and ground-based observations of kinematics, Magorrian et al. (1998) apply dynamical models to 36 nearby galaxies. They find that 97% of early-type galaxies have SMBHs, and this suggests SMBHs could be common in (active) galactic nuclei. Similar results are found by van der Marel (1999); Tremaine et al. (2002). The galactic material near the nucleus provides a source of matter to form an accretion disk. Observations of NGC 4261 show a radio source (3C 720) and jet that likely originate from nearby 8 a black hole, which is estimated to have M ∼ 5 × 10 M¯ (Ferrarese et al., 1996). Around the radio 6 2 5 source is a dusty torus with a radius 240 pc (∼ 10 GM/c ) and mass of Mdisk ∼ 5 × 10 M¯. The mass is determined by assuming NGC 4261 has the same ratio of surface density of hydrogen to color excess E(B-V) as our Galaxy, as provided by Bohlin et al. (1978). The torus mass distribution is perpendicular to the orientation of the jet (Jaffe et al., 1993), which suggests the disk is feeding the black hole and producing the jet. The centers of the galaxy, nucleus, and disk do not coincide. This offset between the disk and galaxy suggests that the torus may have formed from an interacting dwarf galaxy, but it must have been captured > 107 years ago (Ferrarese et al., 1996). The offset between the nucleus and galaxy could be accounted for by recoil from a time-dependent jet, as suggested by Shklovski (1982); Rudnick and Edgar (1984). Despite recent advances in observational technology, only instruments currently in the concept phase will have sufficient angular resolution to spatially resolve the inner (R . 100GM/c2) accretion disk in AGN (Rees, 2001). Therefore, there remain fundamental questions that can only be answered by folding observations through models of AGN disk structure. A quasi-stellar object (QSO, or quasar) is a distant unresolved luminous object, which is likely the nucleus of an active galaxy with an unresolved distribution of stars or protostellar gas. The redshift of quasars ranges from z = 0.06−6.4 (dL = 265 Mpc−64000 Mpc), where most quasars are at z ∼ 1.6 (dL = 12000 Mpc) (see Spinard review article in Mason 2004 and see the Sloan Digital Sky Survey (SDSS) derived quasar-redshift histogram in Oguri et al. 2004). The luminosity of quasars and high-luminosity AGN are comparable. Estimates for the luminosity and compactness 6 9 of quasars suggests they are likely powered by accretion onto a SMBH with M ∼ 10 − 10 M¯ (Zeldovich, 1964; Salpeter, 1964). High redshift quasars are likely newly forming galaxies, where all that can be seen is the bright nucleus. Quasars likely evolve into AGN (Lynden-Bell, 1969). For the purposes of this thesis, it is sufficient for AGN and quasars to be referred to collectively as AGN.

13 This thesis later focuses on estimating disk properties within the nucleus of three galaxies that contain a SMBH: 1) the normal galaxy SgrA* ; 2) the active galaxy NCG4258 ; and 3) the active galaxy M87.

SgrA*

The Milky Way is a spiral galaxy about 31 kpc across and has a total luminosity of about 1042 erg/s (from about 400 billion stars) (Mihalas and Binney, 1981). The Galactic nucleus contains a compact variable radio source called Sgr A* that has a bolometric luminosity of 4 × 1037 erg/s (for reviews see Genzel and Townes 1987; Blitz et al. 1993; Morris and Serabyn 1996; Melia and Falcke 2001) and is about 8 kpc away from the Sun (Reid, 1993). SgrA* may be an X-ray source (Baganoff et al., 2003) and a source for high-energy (∼TeV) gamma-rays (see, e.g., Aharonian and Neronov 2004). As similarly observed in the nucleus of NGC 4261, within 3 pc of SgrA* there is a circumnuclear 7 −3 5 disk, with T ∼ 7000 K, ρ ∼ 10 cm , and M ∼ 3 × 10 M¯ (Shukla et al., 2004), where T is the temperature and ρ is the rest-mass density. This disk is the likely source of matter for the inner accretion disk. The strongest case for a SMBH is in our own galaxy. Infra-red imaging of stellar orbits (e.g. S2 with pericenter of 6 × 10−4 pc, 124AU, 2100GM/c2) in the central region of our galaxy suggest 6 a dark object lies near the center with M ∼ 2.6 × 10 M¯ (Eckart and Genzel, 1997; Ghez et al., 1998; Schödelet al., 2002). A study of the accelerations and velocities of stars in the central region shows the central mass must be extremely compact (Ghez et al., 2000; Eckart et al., 2002). The only known object that could remain stable for a sufficient amount of time is a SMBH. Alternatives, such as a compact cluster of neutron stars, will collapse on short times scales compared to the age of a nucleus (Genzel et al., 1997; Maoz, 1998).

NGC4258

The spiral galaxy NGC4258 (M106) is about 7 Mpc (Greenhill et al., 1995b; Herrnstein et al., 1999) away in the constellation Canes Venatici, has a bulge (galaxy without spiral disk) bolometric luminosity of 5 × 1042 erg/s, has a nucleus with a 2 − 10 keV X-ray luminosity of 4 × 1040 erg/s (Makishima et al., 1994), and has an estimated nuclear bolometric luminosity of 3 × 1043 erg/s (see, e.g., Wilkes et al. 1995; Herrnstein et al. 1998; Yuan et al. 2002; Woo and Urry 2002). VLBI observations of the nucleus show water maser emission from a rotating disk between 0.13 pc−0.26 pc (Greenhill et al., 1995a). The acceleration of the maser spots or the proper motion of the spots allows a geometric distance measurement (Greenhill et al., 1995a; Herrnstein et al., 1999). The 7 kinematics of the rotation suggest the nucleus likely harbors a 3.9×10 M¯ mass black hole (Watson and Wallin, 1994; Miyoshi et al., 1995).

14 M87

The elliptical galaxy M87 (NGC4486,3C 274) is about 30 kpc across, 18 Mpc away in the Virgo cluster (Whitmore et al., 1995), has a total luminosity of 2×1044 erg/s, and has a nuclear bolometric luminosity 2.3×1042 erg/s (Ho, 1999). Within 18 pc of the central region of the nucleus is an ionized 4 9 (T ∼ 10 K) accretion disk whose velocity proﬁle suggests it harbors a 3 × 10 M¯ mass black hole (Sargent et al., 1978; Young et al., 1978; Ford et al., 1994; Harms et al., 1994; Macchetto et al., 1997). M87 also shows a well-collimated jet at large distances (Biretta et al., 1991, 1999) and a broad jet near an accretion disk (Junor et al., 1999) with “jet formation” at about .015 pc (3100AU, 100GM/c2) from the black hole. HST images of the jet in M87 show a well-deﬁned knotty structure (Bridle et al., 1994), which moves at an apparent speed of v/c ∼ 6.

1.3 Basic Accretion Disk Theory

The previous section introduced GRBs, X-ray binaries, and AGN, and introduced 6 fiducial objects that are a reference point for some following discussions. As was discussed, many of these systems likely harbor a black hole. A black hole by itself possesses no astrophysically significant means to generate energy. However, a black hole coupled with an accretion disk is possibly the most efficient producer of large amounts of energy in the universe. Some GRBs, some X-ray binaries, and all AGN are likely powered by gas accretion onto a black hole. Through dissipative processes, gas tends to bind gravitationally to a compact object. As for a spiral galaxy, the rings of Saturn, protoplanetary/protostar disks, and similar objects, this matter eventually forms a disk if the energy of the matter is lost faster than its angular momentum. In the case of a highly ionized plasma, the angular momentum is then transported outward by magnetic instabilities and matter is accreted onto the compact object (for a review see Balbus and Hawley 1998). For typical astrophysical systems, the disk is centrifugally supported and rotates many times before finally plunging into (or onto) the compact object. During this process a significant amount of energy can be released per baryon rest-mass. The basic theory for how this works is described in this section. Below is a description of some basic theoretical concepts used to estimate the mass of the compact object and the luminosity of the accretion disk. This basic theory is then applied to argue that most AGN likely harbor SMBHs, GRBs likely originate from stellar objects rather than an AGN or quasars, and many X-ray binaries likely each harbor a black hole.

1.3.1 Accretion Luminosity and Mass of the Compact Object

A black hole is the most compact object known to exist, and this allows black hole accretion disk systems to release large amounts of gravitational binding energy per baryon accreted. The compactness of a star can be estimated as C = GM/Rc2, where R is the surface radius, M is the mass of the black hole, and G is Newton’s constant. The nonrelativistic gravitational binding

15 energy is proportional to Epot ∼ GMm/R, where m is the baryon mass. The nonrelativistic 2 efficiency (energy released per baryon rest-mass energy) can be estimated as η ∼ Epot/(mc ) = C, which is largest for a black hole at η ∼ 0.5−1 for R at the horizon. A general relativistic calculation defines this efficiency more precisely.

Nominal Accretion Luminosity

A reasonable estimate for an upper limit to the luminosity of an accretion disk can be found for a disk that is always in Keplerian motion. The matter is assumed to accrete in a tightly wound spiral around the compact object. The accreting matter slowly drifts inward due to angular momentum exchange between successive rings of matter. The gravitational binding energy is assumed to be released as radiation. The ﬁnal inner radial ring is at the innermost stable circular orbit (ISCO) for a black hole and the surface for other stars (Bardeen, 1970). For the case of a nonrotating black hole surrounded by a thin disk terminating at the ISCO, the energy per baryon released is η ∼ 6%, while for a maximally rotating black hole η ∼ 42%. Some disk models suggest η . 10−4 near the black hole (see, e.g., Narayan and Yi 1995), while others suggest η & 1 (see, e.g., Gammie 1999).

These efficiencies set an upper limit Lacc, referred to as the nominal accretion luminosity, to the actual luminosity L, with 2 L . Lacc ≡ ηM˙ 0c , (1.2) where M˙ 0 is the accretion rate of rest-mass. For a star with surface radius R, all the remaining ˙ 2 ˙ kinetic energy is released as radiation at the surface, where L = 1/2M0vff = GM0M/R is released by a rest-mass with free fall velocity vff and accretion rate M˙ 0 at the surface. The surface efficiency 2 2 is thus η = L/M˙ 0c = GM/Rc = C, which is η ∼ 0.01% for a white dwarf and η ∼ 10% for a neutron star. These efficiencies can be compared to η ∼ 0.7% for hydrogen burning in stars.

Eddington Luminosity and Minimum Mass

Another upper bound to the luminosity of an accretion disk can be estimated by considering spherical accretion onto a compact object (Bondi, 1952). Assume that the gas ﬂow is composed of mostly ionized hydrogen, spherical, in steady-state, nonrelativistic, and optically thin (τγ ¿ 1) to photons. If the gas has a thickness H, then the optical depth for a uniform mass density is

τγ ∼ κρ0H, (1.3) where κ is the opacity for the appropriate density and temperature regime (Bell and Lin, 1994). For a typical temperature and density of the accretion ﬂow, electron scattering dominates and 8π 2 2 2 then κ ∼ σT Ye/mp, where Ye is the number of electrons per baryon and σT = 3 (e /mec ) ∼ 6.65 × 10−25 cm2 is the Thomson scattering cross section. For such an accreting gas, the radial force balance between radiative scattering of electrons and the gravitational force on protons gives an estimate for the maximum luminosity of the system and the minimum mass of the gravitating

16 object. First, the force of radiation on the gas is found. The radiative energy ﬂux is F = L/(4πr2), where L is the luminosity of a radially-directed photons from a compact region. From a photon 2 momentum of p = E/c, the radiative momentum ﬂux is then Prad = L/4πr c. This is the pressure exerted by photons on a completely absorbing surface. The radiation force on a gas depends on 2 the opacity, where the outward force on a single free electron is Frad = σT Prad = LσT /4πr c for a gas of completely ionized hydrogen. Second, the inward force of nonrelativistic gravity on the gas 2 2 is Fgrav = GM(mp + me)/r ≈ GMmp/r . The protons and electrons are assumed to maintain the neutrality of the plasma by electrostatic coupling. In this spherical, optically thin, nonrelativistic approximation, the force balance between radiation and gravity results in a maximum luminosity, called the Eddington luminosity LE, where

4πGMc L . L ≡ = 1.3 × 1038(M/ M ) erg/s = 3.2 × 104(M/ M )L , (1.4) E κ ¯ ¯ ¯

33 where L¯ = 3.89 × 10 erg/s is the luminosity of the Sun. A system with mass M with L > LE would stop accreting due to outward radiative forces. Setting equation 1.2 equal to equation 1.4 gives the Eddington rest-mass accretion rate of

2 M˙ 0,E ≡ LE/ηc . (1.5)

The Eddington argument can be inverted to give the minimum mass (Eddington mass)

−5 ∼ M & ME = 3 × 10 (L/ L¯)M¯ = (L/L39)(10 M¯) (1.6) for the gravitating body in a system that radiates at the Eddington luminosity, where L39 ≡ 39 10 erg/s. So an object with L = L39 must have a mass M > 10 M¯ to sustain such a luminosity.

The estimate for maximum luminosity LE and minimum mass ME for a system undergoing steady spherical uniform accretion of an optically thin medium works well to explain the upper limit to several non-spherical accretion systems (see, e.g., Margon and Ostriker 1973). However, there are magnetized super-Eddington atmospheres that are unstable, but that are quasi-steady in a time-averaged (statistical) sense (Begelman, 2001). Also, the Eddington luminosity does not generally apply to all optically thick mediums. As shown later, the disk in GRB systems is very optically thick to photons, so the Eddington luminosity is not expected to be an upper limit. The luminosity is so high in GRB systems that if the disk were optically thin, then outward radiative forces would essentially instantaneously halt accretion. Table 1.2 includes the data from Table 1.1 with the addition of the ratio of luminosity to Eddington luminosity for the 6 ﬁducial objects.

Time Variability: Maximum Size and Reverberation Mapping

The maximum size of the emitting region can be estimated from the ﬂuctuations in the frequency integrated luminosity with time scale δT . Assume that the variations in the luminosity of background

17 Table 1.2. Fiducial Black Hole Accretion Systems 2

Parameter NS-BH GRB 030329 LMC X-3 SgrA* NGC4258 M87

dL ? 804 Mpc 55 kpc 8 kpc 7 Mpc 18 Mpc 6 7 9 M[M¯] 3 3 10 2.6 × 10 4 × 10 3 × 10 53 52 38 37 43 42 Lbol[erg/s] 10 3 × 10 10 10 3 × 10 2.3 × 10 14 13 −8 −6 Lbol/LE 3 × 10 8 × 10 0.08 3 × 10 0.006 6 × 10

objects can be accounted for, and that the variations are from the system of interest. Suppose that the entire emitting region is contained within a single point of observation, then fluctuations in the luminosity from causally connected regions must be within R . cδT . There could be historically inevitable or coincidental fluctuations over larger scales, but R is the largest causally connected emitting size due to the limited speed of light. More information can be gathered from the time-variability of spectral features than the variability of the frequency integrated luminosity. Close to AGN there are many intermediate optically thick emitting and absorbing clouds (see, e.g., Krolik 1999a; Elvis 2000, and references therein). One can time the emission response of so-called broad-line regions (BLRs) in comparison to the continuum emission. It was realized that the time delay between the emission-line variations and continuum are due to light travel-time effects within the BLR. The emission lines echo or “rever- berate” the changes in the continuum from the accretion disk. This delay can be used to constrain the proper motion and size of the BLRs, and then the BLR kinematics can be used to estimate the mass of the black hole (Blandford and McKee, 1982; Peterson, 1993; Peterson et al., 2004).

Fe Kα line Proﬁle

An accretion disk is sometimes modeled as having a hot corona (see, e.g., Ostriker 1976), which is suspected to be generated similarly as the Sun’s. In such corona models, the disk is often treated as a cold slab of material. The corona of an accretion disk can generate X-rays, which either Compton scatter free electrons in the accretion disk or get photoelectrically absorbed by a neutral atom. Photons above the threshold energy for transition cause the ejection of an electron from a neutral atom in the disk. The largest cross section for cosmic abundances is associated with the excitation of a K-shell (n=1) electron into the L-shell (n=2). The de-excitation leads either to fluorescence of a K-α photon or to the electron exciting another electron (autoionization) that carries the energy away. The resulting spectrum is a combination of the fluorescent lines, that result from cosmic abundances, and the incident flux that is typically modeled as an X-ray power law (George and Fabian, 1991). The fluorescence also includes the Compton reflection bump at higher energies (E ∼ 30 keV) due to reprocessing (Compton scattering or reflection) of the X-ray power law in the optically thick slab. The observed intensity of this feature in AGN is consistent with the presence

18 of an accretion disk that subtends a solid angle of 2π (see, e.g., Nandra et al. 1991). The Kα line from iron has the largest fluorescent emission and has a rest-frame energy of E = 6.40 keV. AGN observations show that this feature has a much wider profile than expected from simply absorption along the line of sight (Makishima, 1986). The true X-ray reflection processes in an accretion disk are typically more complicated than that described above (Ballantyne et al., 2001).

Often reported is the equivalent width (EE.W.) of the emission line. This is equal to the energy at which the cumulative flux at E < EE.W. in the continuum emission has the same energy as that contained within the line emission. When viewing the incident plus reflected spectrum, the Fe Kα line has an equivalent width of E ∼ 180 eV. Other lines have much smaller equivalent widths (Matt et al., 1997). In summary, in the corona model where the disk is a cold slab, the Fe Kα line emission is produced by fluorescence when hard X-rays from a hot corona illuminate the cold, optically thick component of an accretion disk (George and Fabian, 1991). However, this model has come under fire since observations show the X-ray continuum and Fe Kα line profile are not temporally correlated in MCG-6-30-15 (see, e.g., Matsumoto et al. 2003). This suggests the basic model may be wrong or that a detailed model is required. An important question is whether the properties of the disk and black hole can be measured without directly resolving the accretion disk. Fe Kα line profiles are important since they may allow a reconstruction of disk properties such as disk radii, the spin of a black hole, inclination, and emissivity (Chen and Halpern, 1989; Fabian et al., 1989; Laor, 1991; Pariev and Bromley, 1998). Observations of the Fe Kα line profile from black hole accretion system are discussed below.

1.3.2 Some Accretion-Based Arguments

The above basic theory shows that an accretion disk could be an eﬃcient producer of large amounts of radiation. It is interesting to give rough arguments that invoke an accretion disk to explain the luminosity and some spectral features in astrophysical systems. The following discussion argues that most quasars likely have supermassive black holes, that some X-ray binaries likely each contain a black hole, that GRBs are likely related to stars rather than AGN or quasars, that the extraction of rotational energy from a rotating black hole could lead to a large fraction of the luminosity from GRBs, X-ray binaries, and AGN (or quasars), and that Fe Kα line emission from AGN and X-ray binaries probes the inner radial region of an accretion disk around a (likely rotating) black hole.

Soltan Argument for Rotating Super-Massive Black Holes in Quasars

Black hole accretion has long been considered the most likely power source of quasars (Zeldovich, 1964; Salpeter, 1964). The ratio of quasar radiative energy density to supermassive black hole mass density is ∼ 0.2 (Yu and Tremaine, 2002; Elvis et al., 2002). If the accretion proceeds as a thin disk (Bardeen, 1970) with radiative eﬃciency η(a) around a black hole with spin parameter a, then η & 0.2. This suggests that the average black hole spin is a ∼ 0.96. Even an eﬃciency of η & 0.1

19 gives a & 0.67. This result also applies to nonradiative magnetized thick disk models, where the thick disk efficiency is found to be comparable to the thin disk efficiency (McKinney and Gammie, 2004). Assume that accretion proceeds with a radiative efficiency of ∼ 10%, where the black hole is the spent fuel from the accretion process. One can estimate the remnant black hole mass (Soltan, 1982; Rees, 1984; Cavaliere and Padovani, 1988) from the efficiency η ∼ 0.1, average quasar luminosity L ∼ 1045 erg/s, timescale of quasar activity (age of universe: T ≈ 4.3 × 1017 sec), typical distance 28 to a quasar (redshift z ∼ 1.6 → dC ≈ 1.4 × 10 cm), observed number of quasars per square degree −2 6 6 −3 (nQ ∼ 100 deg → NQ ≈ 4×10 ), and average number density of L? galaxies (n? ∼ 2.4×10 Gpc , where Gpc = 109 pc) that are capable of harboring a SMBH (Krolik, 1999a). The average accreted mass per L? galaxy is then µ ¶ Ã ! N LT M ∼ Q ¡ ¢ ∼ 107M . (1.7) 2 4 3 ¯ n? ηc 3 πdC

The left fraction is the number of quasars per galaxy for a given volume of space. The right fraction is the mass accreted per quasar per unit volume of space. The product is the average mass accreted into the nucleus per galaxy. This suggests that the typical remnant object from accretion should be a SMBH, and the mass is in basic agreement with observations. For a more recent version of this argument, see Merloni (2004).

Black Hole Mass of GRBs and AGN from Variability

As mentioned above, the shortest GRB currently ever observed lasted for 6ms (Bhat et al., 1992) with 200µ s temporal structure. This gives a maximum size of 60 km and a maximum black hole mass of 40 M¯ for the emitting source, so likely this GRB is (and likely others are) related to a compact star rather than to AGN or quasars. Intra-day variability is typically observed in all quasars and AGNs, while in MGC-6-30-15 the X- ray luminosity shows rapid variability on order of 100 s (Reynolds et al., 1995; Yaqoob et al., 1997). MCG-6-30-15 in an active galaxy in the constellation of Centaurus with a redshift of z = 0.0078

(dL = 37 Mpc). For MCG-6-30-15, R . cδT ∼ 170 astronomical units (AU). The compactness 7 43 gives M . 2 × 10 M¯ for a black hole. The luminosity of 4 × 10 erg/s (Reynolds et al., 1997) is limited by the Eddington luminosity, and so M & 3.2 × 105 for a black hole. More advanced 6 time-series based arguments can be made that estimate the black hole mass to be M ∼ 10 M¯ (Reynolds, 2000; Lee et al., 2000; Vaughan et al., 2003). Within AGN there must exist an object capable of generating power that is comparable to or even larger than that of an entire galaxy of stars, yet it must be compact enough to ﬁt within ∼ 100AU. The current standard model is that the central engine of AGN is an accreting SMBH (for an introductory review see, e.g., Armitage 2004).

20 Dimensional Argument for Blandford-Znajek Eﬀect

The mechanism of jet production is an area of active research. One mechanism that may generate a jet is the extraction of rotational energy from a rotating compact star by the surrounding magnetosphere or magnetized disk. Compact stars have the largest magnetic fields due to the approximate advection of magnetic flux during collapse. Compact stars, such as black holes and neutron stars, can hold significant rotational energy that can be released through magnetic braking between the star and the magnetized part of the accretion flow (such as a magnetosphere) (Goldreich and Ju- lian 1969; Blandford and Znajek 1977; Kim et al. 2004 and references therein). The process of magnetically extracting rotational energy from a rotating star is called the Blandford-Znajek (BZ) effect. For a black hole, the power generated by the BZ-effect can be estimated dimensionally from 2 3 the local energy density of the magnetic field (∼ B ), the volume of the black hole (∼ r+), and the light crossing time (∼ r+/c), giving

µ ¶2 µ ¶2 µ ¶2 µ ¶ 2 2 45 B M B M LBZ ∼ B r+c ∼ 10 5 7 erg/s ∼ LE 5 7 (1.8) 10 G 10 M¯ 10 G 10 M¯ ¡ ¢ ³ √ ´ 2 2 where B is the magnetic field strength near the black hole, r+ ≡ GM/c 1 + 1 − a is the radius of the event horizon, a ≡ J/M 2 , and J is the angular momentum of the black hole. This dimensional estimate does not behave properly for arbitrary a, but it is accurate within an order of magnitude for a & 0.5 − 1. A more accurate estimate is obtained by invoking the black hole spin 2 2 4 2 angular frequency ΩH = ac/2r+ (see MTW §33.4), which gives LBZ ∼ B (GM/c ) ΩH /c that has the correct a dependence (R. Krasnopolsky, private communication). The density scale can be set by assuming the accretion rate is at the Eddington rate M˙ 0,E (e.g., with efficiency, η = 10%) or by estimating the rest-mass accretion rate for a particular object from observations of the luminosity and independent estimates for the mass of the compact object. The magnetic field near the black hole can then be estimated from MHD simulations or arguments for equipartition between the gas 2 and magnetic pressure near the black hole (i.e. β = pg/(b /2) ∼ 1), where the pressure is estimated from some model of accretion. For a typical AGN or quasars, MHD simulations give B ∼ 105G and 45 LBZ ∼ 10 erg/s, which is comparable to the luminosity of jets and radio lobes in the strongest of AGN. Renormalizing the BZ luminosity for GRBs and X-ray binaries shows that the jets in these objects may also be powered by the BZ-effect. More advanced analytic estimates suggest that the BZ process is too inefficient to account for a large fraction of power output except in a handful of AGN with low accretion rates and relatively low efficiency (see, e.g. Ghosh and Abramowicz 1997; Armitage and Natarajan 1999; Livio et al. 1999). However, recently the BZ effect has been determined self-consistently using numerical models of force-free models (Komissarov, 2001) and, as performed in this thesis, GRMHD models of disks (McKinney and Gammie, 2004). It is found that the BZ luminosity is indeed lower than nominal accretion luminosity, but that the luminosity due to the BZ effect is focused in a collimated jet at

21 the axis of the black hole accretion disk system. Thus, the BZ eﬀect remains a plausible source of jet energy. A similar total luminosity is estimated for electromagnetic disk winds (Blandford and Payne, 1982; Krasnopolsky et al., 1999, 2003) and for super-Eddington radiation-driven winds (Lucy and Solomon, 1970; Castor et al., 1975; Abbott, 1982; Vitello and Shlosman, 1988; Warner, 1995; Murray et al., 1995; Proga et al., 1998). Likely all these eﬀects are important to some degree for some systems.

X-ray Binary QPOs as Probe of Accretion Disk and Space-Time

Quasi-periodic oscillations (QPOs) with kHz frequencies are observed in the time series of the X-ray emission from some neutron star accretion systems and some black hole accretion systems, such as GRS1915+105. These kHz QPOs provide a plausible means to probe the inner part of an accretion disk and map the space-time in the vicinity of a black hole. For those black hole systems that have kHz QPOs, the kHz QPOs exhibit a scaling of frequency with black hole mass (f ∝ 1/M) that is consistent with the kHz QPO being produced in an accretion disk near the black hole (see, e.g., Abramowicz et al. 2004, and references therein). For a discussion see di Matteo and Psaltis (1999); Strohmayer (2001); Abramowicz et al. (2004); Kato (2004) and references therein.

Event Horizon Argument for Compact Objects in X-ray Binaries

Of the ∼ 240 known X-ray binaries, ∼ 100 show time-dependent phenomena such as pulsations and strong X-ray bursts that do not correspond to accretion outbursts. X-ray bursters have frequent X-ray bursts with quiescent (lower, persistent) X-ray flux between. The quiescent luminosity is order 1032 − 1034 erg/s (the companion star has a bolometric luminosity 1032 − 1039 erg/s). X-ray bursts are classified as Type I and II, where only Type I bursts are discussed since only they are relevant to this discussion. Type I X-ray bursts are observed as a burst of thermal emission with a luminosity of 1036 − 1039 erg/s that last for seconds and recurs in hours to days. The burst energy is found to be proportional to the duration of the preceding inactivity period. There are about 80 known Type-I X-ray bursts. There are about 8 known so-called superbursts with luminosity 1043 erg/s that lasts for hours and recurs in years (Zand et al., 2004). Type-I bursts are thought to occur due to unstable nuclear burning of hydrogen or helium, and in the case of superbursts, of carbon (see, e.g., Woosley et al. 2004, and references therein). Type I bursts are believed to be due to unstable nuclear burning on the surface of neutron stars, while the persistent background flux from such objects comes from conversion of gravitational energy into radiation. For standard Type-I X-ray bursts, the unstable burning comes from 4H → 4He, 3 4He → 12C, and 5 4He + 84 H → 104Pd, which releases 6.7 MeV, 0.6 MeV, and 6.9 MeV per baryon, respectively. The latter two processes are the “triple alpha” and the rapid proton (rp) process. The persistent release of gravitational energy is E ∼ GMm/R ∼ 200 MeV per baryon. Thus, the ratio of gravitational to thermonuclear energy is about 30 − 40.

22 Evidence for the thermonuclear origin for Type I bursts includes that the ratio of persistent energy to burst energy is 30 to 40 and one observes Type I behavior: the longer the preceding fuel accumulation, the more intense the burst. The burning site is argued to be a neutron star since 2 4 only one normal star is seen in the optical, the Stefan-Boltzmann law L = σ4πR Teﬀ gives a typical neutron star radius, and the maximum luminosity is consistent with the Eddington luminosity for a neutron star. Type I X-ray bursts may provide a useful tool to study X-ray binaries with BHCs. No BHC has ever been observed to have a Type I X-ray burst, which is consistent with the fact that such bursts can only occur on a surface. Assuming this is correct, observations of Type I bursts on BHCs would reject the event horizon hypothesis and the existence of classical black holes. Thus reﬁning models of Type I X-ray bursts is important in order to predict their occurrence (Narayan and Heyl, 2002, 2003; Woosley et al., 2004). One must be able to predict not only why some neutron stars have Type I bursts, but also why some neutron stars do not have Type I bursts. Preliminary studies suggest that there is evidence for event horizons (Narayan, 2003a,b; Cornelisse et al., 2003; Tournear et al., 2003; McClintock et al., 2004), but see Abramowicz et al. (2002).

Black Hole Mass/Rotation from Fe Kα line Proﬁles

Observations of line emission from AGN and X-ray binaries show a broadened profile that may be due to relativistic effects in the accretion disk near a rotating black hole. One can generate line profiles as observed on Earth by mapping photon trajectories in Kerr space-time either analytically (Karas et al., 1992) or numerically (Bromley et al., 1997; Cunningham, 1975). Evidence of an accretion disk in AGN includes the ASCA observations of the Fe Kα line profile from MCG-6- 30-15. The Fe Kα line profile in MCG-6-30-15 is shifted to the red, likely demonstrating general relativistic redshifting within a few GM/c2 of the black hole. The red and blue wings of the Fe Kα line profile have different intensities, which are consistent with the emission coming from parts of the accretion disk that move towards and away from the point of observation. The red and blue wing intensities suggest the accretion disk is moving relativistically with speeds approaching 0.2c (Pariev and Bromley, 1998; Tanaka et al., 1995; Fabian et al., 2002; Vaughan and Fabian, 2004). The emission profile is consistent with line emission from within a few Schwarzschild radii (2GM/c2) of the black hole (Tanaka et al., 1995; Fabian et al., 2002; Vaughan and Fabian, 2004). X-ray binary Fe Kα line profiles have also been observed and studied (Miller et al., 2004), and the profiles are consistent with the Fe Kα line being emitted within a few Schwarzschild radii of the black hole. Thus, the Fe Kα line apparently allows a probe of the innermost part of the accretion disk. With Chandra’s and XMM-Newton’s high resolution X-ray imaging, it may be possible to use Fe Kα line profiles to test specific aspects of general relativity such as frame dragging (Bromley et al., 1997). Models of the Fe Kα line from MCG-6-30-15 suggest the emission is ≈ 3 times more likely to originate from a disk around a rotating black hole than a non-rotating black hole (Bromley et al., 1997; Dabrowski et al., 1997). Also, observations of the Fe Kα line are consistent with the

23 Table 1.3. Fiducial Black Hole Accretion Systems 3

Parameter NS-BH GRB 030329 LMC X-3 SgrA* NGC4258 M87

dL ? 804 Mpc 55 kpc 8 kpc 7 Mpc 18 Mpc 6 7 9 M[M¯] 3 3 10 2.6 × 10 4 × 10 3 × 10 53 52 38 37 43 42 Lbol[erg/s] 10 3 × 10 10 10 3 × 10 2.3 × 10 14 13 −8 −6 Lbol/LE 3 × 10 8 × 10 0.08 3 × 10 0.006 6 × 10 −8 −5 −2 −2 M˙ 0 5 M¯/ s 0.1 M¯/ s 10 M¯/ yr 10 M¯/ yr 10 M¯/ yr 10 M¯/ yr

presence of energy extraction from a rotating black hole by the BZ-effect (Wilms et al., 2001b; Miller et al., 2002; Maraschi and Tavecchio, 2003). Despite these suggestions about black hole rotation and energy extraction, the typical Fe Kα line profile calculation, such as performed by XSPEC (Speith et al., 1995), assumes the accretion disk is unmagnetized, viscous, and thin. As is discussed in the next section, such thin disk models typically assume that the accretion disk terminates at the ISCO, which marks a sharp transition where matter plunges into the black hole. This sharp transition would lead to specific spectral features in the observations. As discussed below, magnetic fields can torque the disk within the plunging region and keep the matter from simply falling into the black hole at the ISCO. Indeed, numerical GRMHD studies of accretion disks (see, e.g., McKinney and Gammie 2004), suggest that no sharp transition occurs at the ISCO. Inside the ISCO, the disk should produce weak Fe Kα line emission that the typical thin disk model might be incorrectly interpreting as evidence for black hole rotation and energy extraction (Reynolds and Begelman, 1997).

1.4 Models of Accretion Disks and GRBs

The previous section discussed the most basic principles of accretion disk theory and gave some accretion-based estimates for the properties of these systems. This section goes into more detail by discussing mechanisms for angular momentum transport, some radiative models of accretion, and models for GRBs. First, a discussion of angular momentum transport summarizes why a self- consistent magnetic model is required to study accretion disks. Next, a summary of the thin α-disk model (Shakura and Sunyaev, 1973) derivation is given as an example of a radiative viscous HD model. The purpose of presenting the derivation summary is to show how such disk solutions are found and to show the basic structure of the equations. Next, there is a summary of other radiative disk models. Finally, the section ends with a discussion of models of GRBs. The purpose of the discussion of GRB models is to motivate a GRMHD model for the study of GRBs and core-collapse supernovae in general. The GRB and radiative disk models discussed in this section have been used by others to estimate the rest-mass accretion rate of the 6 ﬁducial systems. Table 1.3 includes the data from

24 Table 1.2 and adds an estimate for the rest-mass accretion rate. The rest-mass accretion rates for the NS-BH GRB and Collapsar GRB (GRB 030329) disks come from simulations and estimates in MacFadyen and Woosley (1999); Popham et al. (1999). Estimates for the rest-mass accretion rate for LMC X-3 are discussed in Ebisawa et al. (1993); Shimura and Takahara (1995); Wilms et al. (2001a); Brocksopp et al. (2001). The rest-mass accretion rate for SgrA* comes from Quataert et al. (1999). The rest-mass accretion rate for NGC4258 comes from Gammie et al. (1999). The rest-mass accretion rate for M87 is from Reynolds et al. (1996); Ho (1999).

1.4.1 Angular Momentum Transport Models

The purpose of this discussion is to summarize why an MHD, rather than HD or viscous HD (VHD), model is required to study ionized accretion disks. First, there is a short discussion of turbulence, which has long believed to be the driver of angular momentum transport in accretion disks. Next, the α-disk model is summarized. Finally, the MHD model is discussed. The discussion of the MHD model includes a summary of the magnetorotational instability (MRI) that likely dominates hydrodynamic instabilities in the generation of turbulence and angular momentum transport in accretion disks (for a review see Balbus and Hawley 1998). Now, a general discussion of turbulence is presented. Turbulent flow can be qualitatively defined as a complex pattern of flow in which microscopic perturbations are enhanced to macroscopic scales as an expression of the flow’s internal degrees of freedom (Kadomtsev, 1965; Tennekes and Lumley, 1972; Eckman, 1981; She and Leveque, 1994; Frisch, 1995). The small-scale pattern of turbulent flow is typically sensitive to the initial conditions (the flow is chaotic), but the average pattern of turbulent flow is insensitive to the initial conditions (the flow is ergodic). Turbulence is 1) stationary if the average values do not vary with time; 2) uniform if the average values do not depend on position; and 3) isotropic if the average values at a specific point in space do depend on the direction. For a finite system undergoing fully developed turbulence, there is a cascade of energy from large to small scales, at which point the energy is dissipated by kinematic viscosity or resistivity into heat (Tatarski, 1961; Kolmogorov, 1941b,a). Kolmogorov found that an incompressible fluid demonstrating stationary, uniform, isotropic turbu- −5/3 lence has an “inertial range” of cascadingq energy, with a wave energy Wk = k , where k is the 2 2 2 magnitude of the wave vector k = kx + ky + kz . The inertial range is between the largest dimensions of a physical system and the dissipation scale. A magnetized, anisotropic, compressible fluid might not be expected to be described by this simple Kolmogorov spectrum, but paradoxically, astrophysical measurements are consistent with the Kolmogorov spectra. Two example systems that follows the Kolmogorov spectra are the magnetized solar wind (Belcher and Davis, 1971; Goldstein et al., 1995; Leamon et al., 1998) and the interstellar electron density spectrum on small scales (108 cm − 1015 cm) (Armstrong et al., 1995) and large scales ( pc) (Lazarian et al., 2001). To study turbulence in accretion disks, one must first identify the source of turbulence and then determine the mechanism of dissipation. Since both of these effects have not been well understood

25 for an accretion disk, models of turbulence have been developed. Generally, all black hole accretion models assume the approximation of hydrodynamics (HD), or when including a magnetic ﬁeld, of MHD. Many accretion models have been developed since the pioneering work of Bondi who described a model for inviscid (perfect, ideal) HD spherical accretion (Bondi, 1952). A summary of some of the early models by Shakura and Sunyaev (1973); Novikov and Thorne (1973); Pringle and Rees (1972); Pringle (1981); Frank et al. (1992) and others can be found in chapter 14 of Shapiro and Teukolsky (1983). Magnetically-driven winds oﬀ the disk are also likely important for angular momentum transport (Blandford and Payne, 1982; Krasnopolsky et al., 1999, 2003) (and see references therein). However, this discussion only considers the process of angular momentum transport in the interior of the disk.

Anomalous Viscosity

See Section 2.3 for the viscous HD (VHD) equations of motion. The so-called α-disk model (Shakura and Sunyaev, 1973) introduces into the (spherical polar) HD equations of motion an r-φ stress (Πrφ) that is proportional to αP , where α ≡ Πrφ/P is a dimensionless constant and P is the pressure. The addition of a shear stress allows for angular momentum transport by driving an exchange of momentum between differentially rotating layers of the disk. This anomalous viscosity is an attempt to model the effect that turbulence generates locally within the disk. While there are global HD instabilities that could initiate the required turbulence, nonlinear studies show that known HD instabilities saturate at low levels or do not apply to an accretion disk near a black hole. No local HD linear or nonlinear instabilities are known to exist in Keplerian disks (Balbus and Hawley, 1998), but this is not a settled issue (see, e.g., Bisnovatyi-Kogan 2004; Kuznetsov et al. 2004). The α-based shear viscosity model can be considered as a zeroth-order approximation to what is mostly likely a magnetic effect, which is described in the next section. The thin disk model of black hole accretion based upon viscosity requires that the inner edge of the accretion disk be at the ISCO. Fluid flows through the ISCO and plunges into the black hole without any effect on the accretion disk or the polar region, as proposed by Page and Thorne (1974). They also realized early on, however, that a magnetic field could significantly alter this picture of the plunging region. The magnetic field within the plunging region may torque the accretion disk outside the ISCO (Krolik, 1999b; Gammie, 1999; Agol and Krolik, 2000). Before discussing magnetic torques inside the ISCO, a summary of the MHD equations of motion is presented.

MHD Equations of Motion

The governing equations for nonrelativistic ideal MHD are the same as those for viscous HD without viscous heating, except for the addition of the Lorentz force to the momentum equation:

Dv ρ = −∇P − ρ ∇φ + (∇ × B) × B (1.9) 0 Dt 0

26 and the addition of the induction equation:

∂B = ∇ × (v × B) (1.10) ∂t √ where B is the magnetic field vector (a factor of 4π is absorbed into the definition of B, as in Heaviside-Lorentz units). The general dispersion relation for linear waves in a uniform background is: 2 2 ˆ 2 4 2 ˆ 2 (vp)(vp − (k · va) )(vp − (cmvp) + (cs(k · va)) ) = 0 (1.11) where vp = ω/k is the phase velocity, k is the wave vector, kˆ = k/k is the wave unit vector, ω is the √ p frequency of oscillation, v = B/ ρ is the Alfvénvelocity, c = c2 + v2 is the magnetosonic p a 0 m s a speed, cs = ∂P/∂ρ0|S is the sound speed, and S is the entropy. There are clearly 8 wave solutions. The first term gives two modes: the entropy mode that corresponds to entropy perturbations at constant pressure ; and the monopole mode that is removed by the solenoidal condition of ∇ · B = 0. The second term gives left- and right-going Alfvénwaves, which are transverse and move at the speed va cos θ, where θ is the angle between k and va. The third term gives left- and right-going fast and slow magnetosonic waves. For this term, if va · k = 0 the fast wave is a purely compressive,q longitudinal wave with phase speed cm. For waves of arbitrary va · k the 1 2 1 4 2 2 2 1/2 phase velocity is 2 cm ± 2 [cm − 4csva cos θ] , where the +(−) corresponds to the fast(slow) magnetosonic wave. Clearly, the limitation of the above nonrelativistic MHD model is the inability to model relativistic flows. While a pseudo-Newtonian potential (Paczyńskiand Wiita, 1980) reproduces the positions of the ISCO, the marginally stable circular orbit, and approximately the binding energy of the last stable orbit, the potential is unable to model a rotating black hole. See Chapters 3 and 4 for the general relativistic coordinates, governing equations, and the characteristics of the general relativistic MHD (GRMHD) equations. Both nonrelativistic and relativistic MHD models find similar results that deviate from expectations built on the α-disk model.

MHD Turbulence

Early accretion disk theory considered magnetic stresses as insignificant (Shakura, 1972). Later accretion disk studies realized that a magnetic field may provide a type of viscosity (Novikov and Thorne, 1973; Eardley and Lightman, 1975; Pringle, 1981). Indeed, previous experimental (Velikhov, 1959) and theoretical (Chandrasekhar, 1961) work found instabilities in differentially rotating magnetized fluids. Balbus and Hawley (1991) rediscovered this magneto-rotational instability (MRI) (or Balbus-Hawley instability) by performing a local stability analysis of a magnetic fluid in a differentially rotating Newtonian disk. The MRI operates similarly near a Kerr black hole (Gammie, 2004). The MRI generates angular momentum transport even when the field is weak. A local stability analysis shows that the MRI grows on a dynamical time of Ω−1 and that the wavelength of the fastest growing mode is λc ∼ va/Ω (Balbus and Hawley, 1991), where Ω is the

27 angular frequency of rotation. Including a magnetic field (and thus the MRI) solves the question of “what is α” in α-disk models by self-consistently generating the turbulence from the magnetic field’s interaction with the fluid. It is still useful, however, to use α as a dimensionless measure of the stress that transports angular momentum. In MHD, α ≡ −δvrδvφ/P and αmag ≡ −δBrδBφ/P are parameters one can measure anywhere in space (δ is the difference from the mean flow, so α measures the turbulent part of the flow). Development of the MRI can be understood qualitatively by considering the stability of a magnetized fluid element in a differentially rotating disk. If a fluid element at radius r0 is threaded by a field line that is parallel to the axis of rotation, then a radial disturbance in the fluid can cause the field line to bend. The initial disturbance causes the fluid element at r < r0 to gain rotational speed by conservation of angular momentum. The fluid element at r > r0 loses speed for the same reason. The field line simply flexes back and resists the separation if the field strength is high. However, if the field strength is relatively low, the field line pulls slightly back on the fluid elements and reduces the angular momentum of the lower fluid element at r < r0 and raises the angular momentum of the fluid element at r > r0 (i.e. the field transfers angular momentum). This in turn increases their speed and thus separation from the original position resulting in an instability (Balbus and Hawley, 2002). In general, a magnetized accretion disk is unstable to the MRI when dΩ2/dr < 0, where Ω is the frequency of rotation. This criterion is easily satisfied for an accretion disk around a compact object. By comparison, the Rayleigh (hydrodynamic) criterion is dl/dr < 0 for instability, where l = r2Ω is the specific angular momentum. While no accretion disk model satisfied the Rayleigh criterion for instability, both theoretical and observed accretion disks have an angular velocity that decreases outwards. Numerical simulations of the MRI confirm the linear stability analysis (Balbus and Hawley, 1991). Subsequent numerical work showed that the MRI is capable of sustaining turbulence in the nonlinear regime (Hawley and Balbus, 1991). The MRI was shown to generate sufficient angular momentum transport to sustain accretion, and the MRI develops a dynamo in the nonlinear regime. The rediscovery of the MRI renewed hope that accretion disks could be understood by a direct calculation, rather than by using the α-disk model. One simplified form of the HD or MHD equations of motion is called the local shearing box approximation, which is a result of an expansion of the equations at a specific radius with a box size δr. The inner and outer “radial” boundaries are treated as periodic with an additional shearing term that describes the differential rotation. Simulations of accretion disks have been performed in the local shearing box approximation (Hawley and Balbus, 1991; Hawley et al., 1995; Stone et al., 1996), which determined the relevance of the MRI instability to, and the vertical magnetized structure of, accretion disks. Accretion disk simulations with 2D axisymmetric or cylindrical symmetries, and full 3D simulations, have been performed to study the global structure of the accretion disk near the black hole (Armitage, 1998; Hawley, 2000, 2001). For an initially 2D symmetric accretion disk, 3D geometry appears to be required only to sustain a magnetic dynamo. All these simulations have shown that the MRI self-consistently accounts for the transport of angular in ionized accretion

28 disks.

MHD Eﬀects Inside ISCO and Ergosphere

The previous discussion showed how the MHD model goes beyond the predictive capability of the α-disk model by self-consistently describing the generation of turbulence and angular momentum transport. Another new result is that the MHD model shows that the disk is magnetically torqued inside the ISCO, as seen in both pseudo-Newtonian numerical models (Hawley, 2000; Hawley and Krolik, 2001, 2002) and GRMHD numerical models (De Villiers et al., 2003a; De Villiers and Hawley, 2003b; De Villiers et al., 2003b; McKinney and Gammie, 2004). Although, how applicable these simulations are to, say, CVs, YSOs, and protoplanetary disks is not certain (Armitage et al., 2001; Reynolds and Armitage, 2001). The α-disk model assumes that the accretion disk terminates at the ISCO, which marks a sharp transition where matter plunges into the black hole. This sharp transition leads to specific spectral features in the observations. As mentioned above in the discussion of the Fe Kα line profile, the presence of a magnetic torques inside the ISCO can affect the conclusions about evidence for black hole rotation and energy extraction. Models of the Fe Kα line profile using an α-disk model seemed to be consistent with the presence of an accretion disk around a rotating black hole (Bromley et al., 1997; Dabrowski et al., 1997; Wilms et al., 2001b; Miller et al., 2002; Maraschi and Tavecchio, 2003). However, inside the ISCO the disk should produce weak Fe Kα line emission that the α-disk model might be incorrectly interpreting as evidence for black hole rotation and energy extraction (Reynolds and Begelman, 1997). The magnetic field may also extract angular momentum and energy from a rotating black hole into the accretion disk through an MHD version of the BZ effect (Blandford and Znajek, 1977; McKinney and Gammie, 2004). Relativistic magnetized steady state accretion disks postulated that a magnetic field might allow a higher accretion efficiency than the thin disk efficiency (including super-efficient η > 1) due to magnetic torques in the plunging region (Gammie, 1999). Remarkably, GRMHD simulations of both thick and thin disks show an efficiency similar to that of a thin disk (McKinney and Gammie, 2004).

1.4.2 Radiative Disk Models

The previous section established that magnetic ﬁelds are required to self-consistently describe the generation of turbulence and process of angular momentum transport within an accretion disk. MHD models also showed that the disk is magnetically torqued inside the ISCO, and that energy can be magnetically extracted from a rotating black hole. Radiative processes in an accretion disk are also dynamically important because they cool the disk, introduce radiative instabilities, and introduce a radiative pressure. Ultimately, of course, several diﬀerent radiative processes may be involved in determining observations, but the focus of this section is on those radiative processes that determine disk structure.

29 Shakura and Sunyaev (1973) constructed an accretion disk solution (SS73 model) that describes the structure of a disk that steadily accretes due to a local anomalous viscous transport of angular momentum. The anomalous viscosity is parameterized by the α model that was described in Section 1.4.1. By approximating the thin disk by a surface density using a height-integrated approximation, including optically thick thermal bremsstrahlung absorption and optically thick electron scattering, they find a complete solution to the accretion flow and spectrum emitted from the disk surface. This model is a useful starting point for estimating the properties of accretion disks. A relativistic version of the thin disk model is given by Novikov and Thorne (1973); Eard- ley and Lightman (1975), but a pseudo-Newtonian approximation of the gravitational potential Φ = GM/(r − 2GM/c2) (Paczyńskiand Wiita, 1980) is sufficient to grasp the salient aspects of the solution. A short derivation of this SS73 model is provided in Shapiro and Teukolsky (1983), §14.5, where below is a summary of the main points. Following the SS73 derivation summary is a discussion of more advanced radiative disk models. The SS73 model assumes that 1) the disk scale height is much less than the radius (H/R ¿ 1) ;

2) an anomalous ﬂuid viscosity introduces a shear stress (Πrφ ∼ αP ) ; 3) rest-mass is conserved ; 4) angular momentum is conserved, where there is a balance between the angular momentum accreted and angular momentum removed by the viscous stress ; 5) vertical momentum is conserved and vertical equilibrium is established, where H/R ≈ cs/vK ; 6) energy is conserved, where there is a balance between kinetic energy, viscous heat generation, and cooling by photon radiation that diﬀuses through the optically thick disk or cools directly in an optically thin disk. The pressure is assumed to be due to radiation and the gas. In the innermost radial region where the temperature is highest, radiation pressure typically dominates gas pressure. In this thesis, disks in X-ray binaries and AGN are modeled with this photon-cooling model. For such disks, the opacity is typically dominated by electron scattering, and this is assumed here. These assumptions allow one to derive the complete state of the disk, where the midplane rest-mass density, midplane temperature, and dimensionless disk height to radius ratio are found to be

−5 −3 2 −1 −2 3/2 −2 ρ0 = (1.5 × 10 g cm )η (αM) m˙ r Ξ T = (5 × 107 K)(αM)−1/4r−3/8 (1.12) h/r = 0.95η−1mr˙ −1Ξ,

1/2 where Ξ ≡ 1 − (6/r) ,m ˙ ≡ M˙ 0/M˙ 0,E, M˙ 0,E is the Eddington mass accretion rate given in 2 Equation 1.5, M is measured in units of M¯, and r, h are both measured in units of GM/c (i.e. h/r is dimensionless). Notice that the assumption that the disk is thin (h/r ¿ 1) is not valid in the inner-radial region when the luminosity is near the Eddington limit. The standard SS73 disk model predicts a disk midplane temperature of T ∝ M −1/4, and thus AGN would be expected to have much lower temperatures than X-ray binaries. The accretion disk in AGN and X-ray binaries, as described by the SS73 model of accretion, produces a soft quasi- thermal spectrum dominated by UV and optical emission in AGN, and dominated by soft X-rays

30 in black hole X-ray binaries. However, there is greater uniformity in the observed X-ray spectra in these systems, which have hard X-ray emissions in a power law out to E ∼ 0.3 MeV. Also, the SS73 model is unstable in the radiation-dominated region of the inner accretion disk, thus violating the steady-state assumption (Lightman and Eardley, 1974; Piran, 1978; Abramowicz, 1981). Since the SS73 model is unstable and does not account for hard X-ray emission, other models have been invoked to explain the X-ray emission. These models often introduce a temperature difference between the ions and electrons in the flow, known as a two-temperature model. This is plausible since the average energy released per accreted particle is µ ¶ E˙ GM/c2 E˜ ≡ ∼ ηc2 , (1.13) M˙ 0 R where E˙ is the total energy accretion rate. Since R ∝ GM/c2, E˜ is independent of the mass of the compact object. For η ∼ 10%, this available energy is up to ∼ 51 keV (∼ 6 × 109 K) for electrons and ∼ 94 MeV (1012 K) for protons. Thus, it is plausible that the electrons and protons could have different temperatures, with inverse Comptonization cooling the electrons and generating hard X- rays (Thorne and Price, 1975; Sunyaev and Truemper, 1979). Depending upon how thermally coupled the electrons and protons are, there may be more or less energy available to the electrons to power inverse Comptonization. This process has been considered as a plausible mechanism for generating hot coronae in accretion disks (Stern et al., 1995). A discussion of Comptonization can be found in Rybicki and Lightman (1979); Pozdniakov et al. (1983) and is summarized below.

Compton scattering involves a photon with energy Eγ and an electron with mass me and gas 2 temperature Te. If mec À Eγ À 4kbTe, then the kinetic energy of the electron is negligible and the photon loses energy. However, if Eγ ¿ 4kbTe, then it can be shown that in the lab frame the 2 photon gains an energy proportional to Γe, where Γe is the electron’s relativistic Lorentz factor. As long as Eγ ¿ 4kbTe, the photon gains energy until the photon reaches thermal equilibrium with the electrons. For low enough photon energy and high enough electron energy the photon can be highly energized, a process referred to as unsaturated inverse Comptonization. If there is little ampliﬁcation the process is referred to as saturated inverse Comptonization. The “hot disk” SLE model (Shapiro et al., 1976) is a two temperature model. In this model, in the inner-radial accretion disk, ions reach T ∼ 1011 − 1012 K, while electrons reach T ∼ 109 K. Hard X-rays are produced by inverse Comptonization of soft photons from electrons in the cooler parts of the disk. The two-temperature models suggest that far away from the compact object, where the disk is relatively cold, the disk follows the SS73 model. The disk there is thin, generating the UV and optical spectra observed. In the inner radial regions a transition to a two-temperature thick disk may occur, and then the ions remain hot and heat electrons, which continuously cool to generate the hard X-rays observed. The transition radius is inversely proportional to the rest-mass accretion rate. At low mass accretion rates, the disk is in the “quiescent” state with much of the inner radial region of the disk forming a thick disk generating low-hard (low luminosity, hard X-ray) emission. At the highest rest-mass accretion rates, the thin disk reaches close to the compact object

31 and disk generates high-soft (high luminosity, soft X-ray or UV) emission. While the SLE model is unstable (Pringle, 1976; Piran, 1978), the two basic considerations may still be viable. First, accretion flow may be two-temperature. Second, Comptonization of photons may provide the hard X-rays even for a single temperature flow. For example, magnetic dissipation has long been considered a mechanism for heating a hot coronal region (Galeev et al., 1979), and this is seen in most global simulations of accretion disks. This is consistent with the so-called “hot corona” model (see, e.g., Ostriker 1976), which also models hard X-rays production as due to Comptonization. A possibly stable version of the SLE model is the advection dominated accretion flow (ADAF) model. As with the SLE model, the ions remain hot, are unable to cool efficiently, and hold most of the accretion energy (Ichimaru, 1977; Rees et al., 1982). The fiducial nonrelativistic ADAF model (Narayan and Yi, 1995) includes synchrotron and bremsstraulung cooling, and Comptonization and synchrotron self-absorption, where a relativistic version is given by Gammie and Popham (1998); Popham and Gammie (1998). The key result is that the flow is highly inefficient (η ∼ 10−4). This is due to the hot ions holding most of the energy down to the neutron star or black hole, while electrons cool and generate the observed spectra. A black hole absorbs the ADAF with little radiation from the accretion disk (Narayan and Yi, 1994, 1995; Abramowicz et al., 1995). Such models have been applied fairly successfully to X-ray binaries (Ichimaru, 1977), to AGN (Rees et al., 1982), and to our galactic nucleus (Narayan et al., 1995, 1998; Quataert et al., 1999). ADAF models have been used to study whether the compact object in an X-ray binary is a neutron star or black hole. The primary difference between these objects is that the neutron star has a surface, while a black hole does not. A possible method to determine the existence of the event horizon (or the lack of a surface on a BHC) is by comparing the quiescent luminosity of X-ray binaries with neutron stars to X-ray binaries with BHCs. It is found that X-ray binaries with BHCs are substantially underluminous compared to their neutron star counterparts per unit Eddington luminosity, which suggests that the excess luminosity in neutron stars is due to emission from the surface and that the BHCs have no surface (Narayan et al., 1997; Menou et al., 1999; Garcia et al., 2001; McClintock et al., 2004). In order to confirm this hypothesis one needs to accurately model the accretion disk. The radiative models of Narayan et al. basically agree with observations by assuming that the accretion disk takes on either a highly radiative, thin disk state or a nonradiative, thick disk state. In either case, the matter releases its energy upon impacting the surface of the neutron star. If BHCs had a surface, then they too would exhibit this surface luminosity. Since they do not, this suggests they have no surface, and given their large mass suggests they are likely classical black holes. The stability of the ADAF model is not certain, and the flow could be unstable to developing outflows near the black hole (Blandford and Begelman, 1999) or be convectively unstable (Quataert and Gruzinov, 2000). The success of the two-temperatures or hot-corona models is that they predict the X-ray spectrum better than the SS73 model due to the inclusion of electronic cooling by Comptonization.

32 All of these radiative models assume a viscous model to describe the transport of angular momentum. The existence of the Balbus & Hawley instability (Balbus and Hawley, 1991) suggests one should study magnetized accretion disks rather than viscous accretion disks. Only simplified analytic models have appeared that include a magnetic field (Gammie, 1999; Merloni, 2003; Li, 2004). Radiative disks have been studied using time-dependent, radiative MHD numerical models of accretion flow (see, e.g., Agol et al. 2001; Turner and Stone 2001; Turner et al. 2002; Turner 2004). These are effectively SS73 models with magnetic fields. Unlike the SS73 model, these simulated disks have a layered vertical structure and are less radiation-dominated for otherwise similar parameters as the SS73 model. No numerical or analytic models have been constructed that include a magnetic field with otherwise similar physics to, say, the two-temperature model with ADAF-type radiative processes. No radiative GRMHD models of accretion disks have yet been studied numerically.

1.4.3 Gamma-Ray Bursts Models

The standard model for the GRB engine is a hyper-accreting black hole or neutron star (for a review see, e.g., Piran 1999; Mészáros2002; Piran 2004). The accretion flow is expected to produce large amounts of energy from annihilation of neutrinos or from spin energy extracted from a rotating magnetized compact object. Respectively, these two processes are believed to produce an ultrarelativistically moving pair plasma that internally shocks to produce gamma-rays, or to produce a Poynting flux jet that develops hydromagnetic instabilities that shocks internally to produce gamma-rays. Eventually, this outflow can interact with the surrounding media to produce so-called external shocks. This section first discusses the prompt gamma-ray and afterglow emission (for a review see Piran 1999, 2004). The purpose of this discussion is to suggest that the GRB progenitor must be relativistic, compact, and likely sometimes occurs during a supernova. Next, this section discusses the “collapsar” model of the accretion system for long-duration GRBs (Woosley, 1993; Paczyński, 1998; MacFadyen and Woosley, 1999). The purpose of this discussion is to suggest that a GRMHD model is required to study the collapsar model. Later in Section 1.5, it is shown that the nonradiative GRMHD models used in this thesis (in Chapters 3 and 4) are a good model for the study of collapsars.

Models for GRB Emission and Afterglow Emission

The prompt gamma-ray emission from a GRB is found to be non-thermal and highly variable in time. The temporal variability of GRB light curves suggests the GRB progenitor has a size R . 60 km for some short-duration bursts, and R . 300 km for some long-duration bursts. However, 53 3 an isotropic estimate for the number density of gamma-rays nγ ≈ (10 erg)/((500 keV)R ) is much 2 larger than what is required to generate electron-positron pairs (Ee−e+ ∼ 2mec ). This implies the emitting media would have an optical depth of τ ∼ 1015, and suggests that the emission should

33 be thermal. The prompt gamma-rays are actually non-thermal, and this paradox is called the “compactness problem” (Piran, 1999, 2004). One resolution to the compactness problem is to assume that the flow is moving ultrarelativistically. By assuming the observed radiation is from a source moving toward us with Lorentz factor Γ, blue-shifting of photons and modifications to the probability of photon collisions require Γ & 100 to obtain τ . 1. However, there is no well- understood mechanism for jet collimation or large Lorentz factors. If geometric or relativistic beaming occurs, then an estimate of the actual luminosity can be obtained by appropriately reducing the estimate for the isotropic luminosity. The value of the actual luminosity can be useful to determine the nature of the progenitor. If beaming occurs geometrically, then the isotropic luminosity estimate must be reduced by a beaming fraction of 2 ∼ θj /2 for θj . 1, where θj is the opening angle of one side of, an assumed, bipolar jet (Rhoads, 1999; Sari, 1999). If beamed relativistically, a local observer sees an enhancement of Γ2, so one must reduce the estimate by 1/Γ2. One expects 3 possible phases if the initial Lorentz factor is large, and these phases lead to approximately 3 different domains in the light curve (Sari et al., 2 1999): 1) early relativistic beaming with ∼ (θjΓ) causally disconnected patches; 2) a slowdown leads to Γ ∼ θj and thus rapid sideways expansion that leads to a break in the light curve; and 3) at late time Γ ∼ 1 and one observes an exponentially decaying emission. In this ultrarelativistic fireball model, internal shocks are produced because the outflow has a range of velocities. These relativistic internal shocks produce gamma rays via synchrotron or synchrotron self-Comptonization emission. Such a fireball model has a “baryonic contamination” problem (see, e.g., Piran 2004). A flow contaminated with baryons can absorb the radiative energy into bulk kinetic energy, producing too few photons compared to observations. Internal shocks allow this kinetic energy to be converted back into radiation, but too much conversion too quickly leads to sub-relativistic bulk motion leading again to a compactness problem. Another solution to the compactness and baryonic contamination problems is to assume the jet carries nonradiative energy in the form of a Poynting flux, and the flow only creates gamma- rays at sufficiently large distances. For example, a Poynting flux could be converted into radiation far from the collapsing star by internal dissipation (see, e.g., Thompson 1994; Mészárosand Rees 1997; Spruit et al. 2001; Drenkhahn 2002; Drenkhahn and Spruit 2002; Sikora et al. 2003; Lyutikov et al. 2003) such as magnetic reconnection. Preliminary (controversial) evidence for a magnetic dominated outflow has been found in GRB 021206 (Coburn and Boggs, 2003), consistent with a magnetic outflow directly from the inner engine (Lyutikov et al., 2003). As discussed below, a Poynting flux jet may be primarily responsible for GRBs. The GRB afterglow is suspected to be generated by interaction with the interstellar medium in external shocks (see, e.g., Piran 2004). Optical, radio, and X-ray afterglows have energies of 1050 − 1052 erg, generally about 1/10th of the energy in gamma-rays. GRBs, such as GRB 990123 and GRB 990510, show breaks in the optical light curve, which is consistent with an external shock model. An analysis of the light breaks using an adiabatic synchrotron model for cooling (see, e.g. Piran 1999, 2004), similar to the Blandford-McKee self-similar solution (Blandford and McKee,

34 1976), shows the breaks are consistent with Γ ∼ 100 − 500. There is no temporal scaling between the GRB and afterglow lightcurves, and this lack of scaling is consistent with the GRB as generated in internal shocks and the afterglow as generated in external shocks. A GRB likely develops from a compact central engine (Piran, 1999; Mészáros,2002). If one assumes that the GRB emitting region is causally connected by light propagation effects, then the variability from GRB light curves gives an estimate of the maximum emitting region (R ∼ cδT ). The typical GRB variability is 1 ms − 1 s, giving a maximum emitting region of R ∼ 300 km. The shortest GRB currently ever observed lasted for 6ms (Bhat et al., 1992) with 200µ s temporal structure. This gives a maximum size of R = 60 km for the emitting source. Energetic and beaming considerations suggest a GRB is a collimated jet with an energy of ∼ 1051 erg and a Lorentz factor of 102 − 103 (Sari et al., 1999; Frail et al., 2001; Panaitescu and Kumar, 2001; Bloom et al., 2003). Such supernova-like energies suggest a subclass of massive stars undergoing core-collapse could generate GRBs. There is no clear evidence for whether the GRB is a “standard candle” with standard energy output and jet beaming characteristics (e.g. see references in Liang et al. 2004). Recent observations suggest that at least some GRBs are not standard candles. For example, GRB 031203 had a very low luminosity from an otherwise normally beamed GRB (Sazonov et al., 2004; Soderberg et al., 2004). The estimated GRB event rate of ∼ 1/(107 yr) per galaxy, or ∼ 1/(105 yr) if there is beaming, also suggests a GRB-SN connection. HST observations show an association between optical afterglows and host galaxies (Sahu et al., 1997; Akerlof et al., 1999), and show a correlation between GRBs and star-forming regions in galaxies (Fruchter et al., 1999). Core-collapse supernovae models have suggested a GRB-SN connection since GRB 980425 and SN 1998bw at z = 0.008 (Galama et al., 1998; Iwamoto et al., 1998). The afterglow from long-duration GRB 030329 and supernova SN2003dh at redshift z = 0.169 occurred essentially simultaneous in time and overlap in space confirming a supernova connection to long duration GRBs (Uemura et al., 2003; Mészáros,2003; Hjorth et al., 2003; Kawabata et al., 2003; Stanek et al., 2003). Chandra observations of supernova remnant W49B in our own Milky-Way show the likely remnants of a GRB (Keohane et al., 2004), so a forensic study may reveal details of the GRB event.

Models for the Engine of Long/Short Duration GRBs

The temporal variability of GRB light curves suggests the GRB progenitor is a system with a size R . 60 km, and M . 40 M¯ if a compact object, for some short-duration bursts; and with a size R . 300 km, and M . 200 M¯ if a compact object, for some long-duration bursts. The progenitor must release the binding energy of a solar mass object over an extended period (many light crossing times), rather than in a single explosion. This suggests an accretion disk origin is plausible. Accretion in other systems is known to produce relativistic jets. The two likely scenarios for generating a GRB are 1) accretion of a 0.001 − 1 M¯ disk around a few solar mass black hole ; or 2) gamma-ray dipole or spin-down emission from a highly magnetized neutron star called a magnetar.

35 The matter that forms the accretion disk around the solar-mass black hole or neutron star could be from core-collapse material, supernova fallback material, a neutron star, a white dwarf, or a Wolf-Rayet star. GRB events have a duration that follows a bimodal distribution, which could result from an accretion disk system formed from these diﬀerent types of secondary material and their associated angular momentum (and thus circularization radius and accretion time scale) (Popham et al., 1999). The current leading explanation for a long-duration GRB is the collapse of a massive star

(M ∼ 25 − 40 M¯). The most attractive model for long-duration GRBs is the collapsar model (Woosley, 1993; Paczyński,1998; MacFadyen and Woosley, 1999) of which there are two types. A Type I collapsar results from a collapsing massive star that has too weak a shock wave to generate a supernova. The collapse of the iron core leads to a neutron star, which then quickly leads to a black hole and accretion disk from the fall back of the envelope. A Type II collapsar is a successful supernova of a massive star that blows away all helium and some heavy elements outside a neutron star core. Some material fails to reach escape velocity, falls onto the neutron star, and slowly forms a black hole and accretion disk. The collapsar model as developed by Woosley (1993); MacFadyen and Woosley (1999) includes several pieces of microphysics, such as 1) a realistic EOS (Lattimer and Swesty, 1991; Lattimer, 1996; Blinnikov et al., 1996; Lattimer and Prakash, 2000); 2) the URCA process for nuclear burning (Bodenheimer and Woosley, 1983); and 3) neutrino cooling and annihilation in the optically thin parts of the flow (Itoh et al., 1989, 1996; MacFadyen and Woosley, 1999). For a discussion of collapsar-like GRB accretion disk models, see for example Popham et al. (1999); Narayan et al. (2001); Kohri and Mineshige (2002); Di Matteo et al. (2002). For a discussion of neutrino-driven explosions, see for example Fryer and Mészáros(2003). For a discussion of optically thick neutrino accretion disks, see for example Lee et al. (2004). The collapsar model has been studied numerically, including 1) viscous HD models with the above microphysics, following the evolution of a pre-supernova star to collapse, accretion disk formation, and jet formation (MacFadyen and Woosley, 1999); 2) special relativistic MHD models of the jet (MacFadyen et al., 2001); 3) nonrelativistic MHD models with the above microphysics, following the formation of a disk and jet (Proga et al., 2003); and 4) stationary space-time general relativistic models, with no microphysics, of a black hole with a pseudo-post supernova star (Mizuno et al., 2004a,b). Numerical simulations show that the penetration of a jet through a stellar surface and evolution of the jet through a surrounding pressure gradient can lead to large Lorentz factors of Γ ∼ 44, close to that required by the standard fireball model (see, e.g., Aloy et al. 2000; Zhang et al. 2003). Numerical models have yet to include numerical relativity and MHD to study the collapse of a rotating magnetized massive star to a black hole (see review by Stergioulas 2003, §4.3). Despite extremely detailed neutrino physics, typically the magnetic field is neglected in core- collapse supernovae calculations that seek to follow the entire collapse of a massive star (for the latest typical code see, e.g., Liebendörferet al. 2004). In unraveling the mechanism by which core- collapse supernovae explode, the simulation of a successful supernova has been found to be sensitive

36 to the accuracy of the neutrino transport (Messer et al., 1998; Yamada et al., 1999; Burrows et al., 2000; Rampp and Janka, 2000; Liebendörferet al., 2001; Mezzacappa et al., 2001). For some time this has been regarded as implying one requires highly accurate neutrino physics to model core- collapse supernovae (Liebendoerfer, 2004), however this could also be interpreted as suggesting other physics (say, a magnetic field) is required to model core-collapse supernovae. The GRB is suspected to be due to either (or both) 1) neutrino and anti-neutrino annihilation that develops into an electron-positron pair fireball at the polar axis of the disk or black hole and eventually produces gamma-rays; or 2) an electromagnetic Poynting flux from the disk or black hole following field lines at the poles and eventually produces gamma-rays. As discussed above, Poynting flux generating GRB models solve both the compactness and baryon-contamination problems. The energy extracted from a rotating black hole by a magnetic field can easily exceed the energy for neutrino-mediated energy transport (Mészárosand Rees, 1997). Indeed, all core- collapse events may be powered by MHD processes rather than neutrino processes (Leblanc and Wilson, 1970; Bisnovatyi-Kogan and Ruzmaikin, 1974; Bisnovatyi-Kogan et al., 1976; Symbalisty, 1984; Woosley and Weaver, 1986; Duncan and Thompson, 1992; Khokhlov et al., 1999; Akiyama et al., 2003; Thompson et al., 2004a). Core collapse explosions are observed to be significantly polarized, asymmetric, and often bi-polar, indicating a strong role of rotation and a magnetic field (Wang and Wheeler, 1996; Wheeler et al., 2000; Wang et al., 2001, 2002, 2003). Core-collapse also involves shearing subject to the Balbus-Hawley instability as in accretion disks (Akiyama et al., 2003). Thus, there is sufficient evidence that long-duration GRB models should include a magnetic field. There are fewer observational phenomena associated with short-duration GRBs, and so it is more difficult to model short-duration GRBs than to model long-duration GRBs. Without an afterglow counterpart to the GRB, a redshift measurement is not possible, and correlating a short GRB with a host galaxy is difficult. The short-duration GRBs are observed to have harder gamma- rays than long-duration bursts. This is believed to be due to a lack of baryons obstructing gamma- ray emission. Short-duration GRBs could be due to binary NS mergers (Narayan et al., 1992) or NS-BH mergers (Narayan et al., 1991). The NS-NS or NS-BH collision processes are currently the best model for short-duration GRBs, since these systems produce harder gamma-rays as seen in observations. However, short-duration GRBs could be produced by a relative of the collapsar model, which has no supernova and little baryon-loading that also leads to harder emission. During NS-NS or NS-BH collisions, gravitational wave emission is likely significant, and if the GRB is close enough to Earth, gravity waves may be detectable by LIGOII/LISA (van Putten et al., 2004). Thus, gravitational waves may help determine the origin of short-duration GRBs. Some GRBs may also be due to spin energy extraction from a highly magnetized neutron star called a magnetar (Wheeler et al., 2000; Rees and Mészáros,2000; Thompson et al., 2004b). In the so-called cannon-ball model, the asymmetric stellar collapse leads to a magnetar moving at ∼ 1000 km s−1 (Dado et al., 2002; Huang et al., 2003), where a pair plasma fireball is ejected

37 highly anisotropically (∼ 3 − 10◦), but otherwise as described by Usov (1992).

1.5 Characteristic Quantities and Model Validity Estimates

In Section 1.2, 6 fiducial black hole accretion systems were introduced to establish their distance from the Sun, black hole mass, and bolometric luminosity. It was argued that these objects likely have accreting black holes. In Section 1.3, the basic theory of accretion was introduced, and allowed a calculation of the ratio of the bolometric to Eddington luminosity. It was found that the luminosity, variability, and spectra could be used to deduce basic and precise properties of black hole accretion disk systems. In Section 1.4, magnetic and radiative models of accretion disks were introduced, which have allowed researchers to estimate the rest-mass accretion rate of the 6 fiducial systems. It was found that a magnetic model was required for self-consistent generation of turbulence to drive angular momentum transport, and that radiation is crucial to determine the thickness and structure of an accretion disk. It was also discussed how GRMHD models are likely required to study GRBs and core-collapse supernovae. This section uses this cumulative knowledge to estimate the state of the accretion disk in the 6 fiducial systems by using an approximate radiative GRMHD model of the accretion flow. First, the “state” of the accretion flow is determined. The “state” refers to quantities such as the density, temperature, and magnetic field strength. In principle, the state should be determined by a completely self-consistent numerical or analytic solution. However, no such solution has been derived or numerically constructed for a radiating magnetized disk in general relativity. A compromise is made by using radiative unmagnetized analytic models together with nonradiative GRMHD numerical solutions of accretion disks as studied in chapter 4. A consistency check is performed to verify that, indeed, this is a reasonable treatment. Once the complete magnetized state of the disk is determined, the validity of the fluid, MHD, and ideal MHD approximations is tested for the 6 fiducial systems. Even though the accretion disk may extend to R & 1000GM/c2, the focus of this study is on the state of the accretion flow, and the validity of the approximations, near the black hole within R . 40GM/c2 where relativistic effects are important. The density and temperature vary widely within and between systems. While the conditions for the MHD approximation are plausibly satisfied in the bulk of the disk, they may not be satisfied, for example, in the more vacuous corona (Novikov and Thorne, 1973). While I have tested the validity of the approximations for the entire global accretion flow, only the state of the disk and the validity of the approximations in the disk are shown. The disk results shown are representative for the non-disk part of the global solution, and any deviations are discussed.

1.5.1 Estimated State and Structure of Accretion Flow

In order to estimate the state of the GRB disk, and the magnetic ﬁeld in the other disks, we use results of nonradiative GRMHD numerical models of accretion ﬂow described in chapter 4. For

38 FUNNEL P 0 L E CORONA

WIND

BLACKHOLE

DISK

PLUNGING REGION r=12L EQUATOR r=40L

Figure 1.1 Cartoon plot of one hemisphere of the axisymmetric accretion disk, corona, funnel/wind, and plunging regions. The outer radius of the GRMHD simulation is 40GM/c2. The fiducial radius (r = 12L, where L = GM/c2) is where all “state” quantities are evaluated. These state quantities are then used to estimate the validity of the fluid, MHD, and ideal MHD approximation at this location. those simulations, the mass density, pressure, and magnetic field are determined self-consistently for a thick disk with scale height to radius ratio of H/R ∼ 0.26. In that study, it was found that the global accretion flow can be understood basically as having different regions based upon the ratio of 2 2 magnetic energy to mass energy (B /ρ0c ) and the ratio of gas to magnetic pressure (β ≡ pgas/pb, 2 where pb = B /2). Figure 1.1 shows the regions of the accretion flow. The disk is defined to be where β < 1/3, the plunging region where β > 1/3 within a disk scale height, the corona region 2 where 1/3 < β < 1, and the funnel region where B /ρ0 > 1. The figure shows the fiducial radius 2 2 r = rfid = 12GM/c and the outer radius r = rout = 40GM/c used in the simulation. The fiducial radius rfid is the location where all “state” quantities are evaluated and all “validity parameters” are evaluated that test the validity of the fluid, MHD, and ideal MHD approximations. These regions can be treated approximately based upon their average properties. We take a spatiotemporal average of quantities during the turbulent period of a global GRMHD simulation. The time average is performed over t = 1000GM/c3 to t = 2000GM/c3, and a volume integral is performed over each region for each quantity in the table. Table 1.4 shows the average value of the dimensionless rest-mass density ρ0/ρ?, where ρ? is some fiducial rest-mass density; the value 2 of B /ρ0 ; and the value of β for each of the 4 regions. I have determined the state (density, temperature, magnetic field, optical depth, etc.) for all of these regions for each of the 6 fiducial objects. I have also determined the validity of the fluid, MHD, and ideal MHD approximations for all these regions for each of the 6 fiducial systems. For brevity, this section only presents the details

39 Table 1.4. Accretion Flow Regions

2 Region ρ0/ρ? B /ρ0 β = pgas/pb

Disk 0.072 0.0004 9.5 Plunge 0.18 0.018 1.6 Corona 0.0068 0.003 1.5 Funnel/Jet 0.00023 7.4 0.0047

Note. — Dimensionless quantities for regions as described in Figure 4.2 for H/R = 0.26 GRMHD simulation. See Section 1.5 for a detailed description.

for the accretion disk itself. From Table 1.4 one can obtain the dimensionless ideal gas temperature

β/2 pgas θ ≡ 2 ≡ 2 , (1.14) B /ρ0 ρ0c which can be converted to kelvin for an ideal gas of protons (where pgas = ρ0kbTp[K, ideal]/mp) to obtain µ 2 ¶ ∼ mpc Tp[K, ideal] = θ . (1.15) kb The GRMHD simulation of a thick disk, with an initially ﬁxed H/R = 0.26, has a rest-mass accretion rate of Ã µ ¶ ! GM 2 M˙ = A ρ c , (1.16) 0,numerical M ? c2 where AM = 0.2 from the GRMHD numerical simulation. By setting the numerical mass accretion rate equal to the true rest-mass accretion rate (M˙ 0,numerical = M˙ 0), the rest-mass density scale (ρ?) and then rest-mass density (ρ0) can be determined from Table 1.4. The true mass accretion rate is determined from Table 1.3. Our GRMHD model is a nonradiative model with an initial (and approximately always) height to radius ratio of H/R ∼ 0.26. By using the ideal gas law relating density, temperature, and

40 √ pressure, H/R ∼ cs/vK , and B ∝ ρ0c, one can derive the following proportionalities µ ¶ H −1 ρ ∝ 0 R µ ¶ H P ∝ R µ ¶ H 2 T [K] ∝ (1.17) R µ ¶ H −1/2 B ∝ , R and from numerical simulations one can estimate that µ ¶ H −3/5 M˙ ∝ . (1.18) 0 R

These proportionalities are used to scale the simulated solution with H/R = 0.26 to another solution with arbitrary H/R. For the sake of consistency, the true rest-mass accretion rate is assumed to vary with H/R similarly as the numerical rest-mass accretion rate. It is assumed that the physical rest-mass accretion rates are no diﬀerent for thin or thick disks. The rest-mass accretion rate is 2/5 determined by a few GRMHD simulations that show M˙ 0/ρ0 ∝ (H/R) . No detailed GRMHD studies of thin disks (H/R . 0.01) have been performed that generally verify these scaling laws, but they should be roughly accurate. The state parameters are estimated for a characteristic length, time, and rest-mass density

GM L = 0 c2 GM T = (1.19) 0 c3 Ã ! µ ¶ M˙ 0 1 ρ? = 2 , AM cL0 where L0 and T0 are the natural length and time scales for a black hole, and ρ? is found from M˙ 0 =

M˙ 0,numerical. The characteristic value of any other quantity can be found from these deﬁnitions of mass, length, and time.

GRB Disk State Determination

Since the GRMHD simulations are nonradiative, the value of H/R is estimated from the results of an unmagnetized, viscous, neutrino-dominated accretion disk model (Popham et al., 1999) (hereafter PWF). PWF determine the balance between the energy loss due to radiation of neutrinos and energy gain due to turbulent dissipation. The PWF model provides the disk scale height per unit radius (H/R) at a particular radius for a given rest-mass accretion rate. This value of H/R is used

41 to scale, as described above, the state resulting from the nonradiative GRMHD simulation with H/R ∼ 0.26 to the PWF solution for H/R.

X-ray Binary and AGN Disks State Determination

For the X-ray binary and AGN disks, a thin disk state is self-consistently found by solving the equations of mass, angular momentum, and energy conservation for a radiative disk as in the so- called α-disk model (Shakura and Sunyaev, 1973). This typically results in H/R . 0.05. A thick disk state (studied, but not shown below) is assumed to be an ADAF with H/R ∼ 1 (Narayan and Yi, 1995). The only additional component required for these unmagnetized models is the ﬁeld strength, which is estimated using the H/R scaling law above. The non-disk regions are assumed to follow Table 1.4 for both thin and thick disks.

The State of the Accretion Disk

The “state” of the accretion disk for the 6 fiducial black hole accretion systems is shown in Table 1.5. The columns of the table correspond to one of the 6 fiducial systems, and the rows give estimates for state parameters such as density and field strength. Only the state of the disk region is shown. The first 5 rows of Table 1.5 are from Table 1.3 and shown here for convenience. State parameters in Table 1.5 are

1. rfid: Characteristic ﬁducial radius in centimeters in a typical numerical study. Used to determine accretion and rotation time scales.

2. rres: Characteristic smallest radius in centimeters in a typical numerical study. Used to determine characteristic smallest scale L.

3. L: Characteristic smallest length scale in centimeters in a typical numerical study. The radial

range is resolved logarithmically dr/r ∝ const. and the height range as H/rres ∝ const..

For Nr radial zones dr = rres log (rout/rin)/Nr, where rin and rout are the inner and outer 2 radial ranges of interest. For this study typically rin ∼ GM/c , near the event horizon, and 2 rout ∼ 40 − 400GM/c .

4. T : Characteristic smallest time scale in seconds in a typical numerical study (T = L/c).

−3 5. ρ0: Characteristic rest-mass density in g cm .

6. B: Characteristic magnetic ﬁeld in Gaussian units.

7. B/Bcrit,e: Fraction of QED electronic critical field for which the magnetic field exhibits 2 3 ∼ 13 particle modes and photons behave nonlinearly. Bcrit,e ≡ mec /(e~) = 4 × 10 G. See for example Heyl and Hernquist (1999). Only GRB disk has a field around the critical value.

8. Tp: Temperature of protons (ionized hydrogen) and other particles in kelvin.

42 Table 1.5. Accretion Disk State

Parameter NS-BH GRB 030329 LMC X-3 SgrA* NGC4258 M87

6 6 7 12 13 15 rfid[cm] 5.3 × 10 5.3 × 10 1.8 × 10 4.6 × 10 7.1 × 10 5.3 × 10 5 5 6 11 12 14 rres[cm] 4.4 × 10 4.4 × 10 1.5 × 10 3.8 × 10 5.9 × 10 4.4 × 10 L[cm] 5.9 × 103 5.9 × 103 2 × 104 5.1 × 109 7.9 × 1010 5.9 × 1012 T [s] 2 × 10−7 2 × 10−7 6.6 × 10−7 0.17 2.6 2 × 102 −3 11 10 −6 −7 −8 ρ0[g cm ] 6.1 × 10 1.6 × 10 0.0072 1.5 × 10 2.4 × 10 1.3 × 10 B[G] 1.7 × 1015 2.7 × 1014 1.1 × 106 1.4 × 102 2.8 × 102 3.7 B/Bcrit,e 38 6.1 − − − − 10 10 7 5 5 4 Tp[K] 2.1 × 10 1.2 × 10 1.8 × 10 3.5 × 10 4.8 × 10 7.1 × 10 −6 −8 −8 −9 xp 0.0019 0.0011 1.7 × 10 3.2 × 10 4.4 × 10 6.5 × 10 −5 −5 −5 xe 3.5 2 0.0031 6 × 10 8.2 × 10 1.2 × 10 nn/ndegen,n 0.32 0.0095 − − − − 2 ne/ndegen,e 25 1.6 × 10 − − − − Ye 0.01 0.5 0.5 0.5 0.5 0.5 Xnuc 0.17 0.17 − − − − 4 T/tNSE 1.3 × 10 48 − − − − Xion,H 0.043 0.026 1 1 1 1 2 2 5 tacc 0.091 s 0.15 s 32 s 1.4 × 10 yr 2.7 × 10 yr 8.4 × 10 yr trot 3.9 ms 3.9 ms 13 ms 0.93 hr 0.6 day 0.12 yr H/rfid 0.26 0.2 0.025 0.0011 0.0031 0.00048 prad/pgas 0.0031 0.0057 18 0.62 9.7 0.57 pgas/pgas,0 0.39 1.6 − − − − 15 15 2 3 4 3 τγ 3.4 × 10 3.3 × 10 6.3 × 10 1.5 × 10 1.1 × 10 6.6 × 10 τν 1.2 0.0084 − − − −

Note. — First 4 parameters are previously discussed and are based upon astrophysical observations. The value of M˙ 0 is based upon both observations and models of the accretion disk (see Section 1.4). The remaining parameters describe the state of the plasma in the accretion disk for various astrophysical 2 systems at a radius of r = rfid = 12GM/c , close to the black hole. The GRB-type disks (ﬁrst 2 columns) are modeled using a radiative VHD PWF model of the disk thickness (Popham et al., 1999) and nonradiative GRMHD simulations to set all other quantities. The other disks (remaining 4 columns) are determined by the SS73 model of a thin accretion disk (Shakura and Sunyaev, 1973) for all quantities except the magnetic ﬁeld strength, which is determined by scaling the GRMHD solution to the appropriate H/R.

43 2 9. xp: Proton relativity parameter, xp = kbTp/(mpc ). Protons are relativistic if greater than 1, otherwise nonrelativistic.

2 10. xe: Electron relativity parameter, xe = kbTe/(mec ). Electrons are relativistic if greater than

1, otherwise nonrelativistic. It is assumed that Te ∼ Tp.

11. nn/ndegen,n: Ratio of actual number density of neutrons to number density of neutrons if degenerate. Neutrons are degenerate if greater than 1, otherwise nondegenerate. See Kohri and Mineshige (2002). Only the GRB disk has nonnegligible neutron/proton degeneracy.

12. ne/ndegen,e: Ratio of actual number density of electrons to number density of electrons if degenerate. Electrons are degenerate if greater than 1, otherwise nondegenerate. See Kohri and Mineshige (2002). Only the GRB disk has nonnegligible electron degeneracy.

13. Ye: Electron fraction: Ye = ne/(np + nn). Assumes charge neutrality and that the positron number density is small (true for all the cases studied here). See Kohri and Mineshige (2002).

14. Xnuc: Mass fraction in free nucleons. Rest are mostly α particles. Assumes nuclear statistical equilibrium. See Woosley and Baron (1992). Not relevant for SS73 model of disk for X-ray binary and AGN. A hot thick disk model may require this.

15. T/τnse: Ratio of the characteristic smallest time to the time to achieve nuclear statistical equilibrium. See Khokhlov (1989). Not relevant for SS73 model of disk for X-ray binary and AGN. A hot thick disk model may require this.

16. Xion,H : Ionization fraction of electrons in a hydrogen gas. This assumes thermal statistical equilibrium, and that pressure and radiative ionization are negligible. In GRB disks, the interparticle spacing ∼ the electron orbit radius, so pressure ionization may increase the ionization fraction. See Equation A.10 in Appendix A. q 3 2 17. τacc: Accretion time scale: τacc ∼ (1/α) rfid/(GM)/(H/rfid) . Assumes α ∼ 0.1, and the α model approximately holds.

3/2 √ 18. τrot: Rotation time scale: τrot ∼ 2π(rfid + a) GM. Assumes a ∼ 0.9375, although at rfid this is not a signiﬁcant eﬀect.

19. H/rfid: Scale height of the disk per unit radius at the ﬁducial radius. Determined self- consistently without a magnetic ﬁeld for the thin disk state. The SS73 solution is used for AGN and X-ray binaries, and the PWF solution is used for GRBs.

1 4 20. prad/pgas: Ratio of radiation to gas pressure, where prad = 3 aT for an optically thick medium 5 4 3 2 (τγ À 1), where a = 4σ/c and σ = 2π kb /(15h c ) is the Stefan-Boltzmann constant. The gas pressure for X-ray binary or AGN disks is pgas = (kb/µ)ρ0T , where µ is the mean molecular weight per free particle. For a hot fully ionized plasma of cosmic abundances µ =∼ 0.62, while

44 for a cold cloud of molecular hydrogen µ =∼ 2.4. The GRB disk pressure is determined as in Kohri and Mineshige (2002).

21. pgas/pgas,0: Ratio of GRMHD simulation estimated gas pressure to actual gas pressure as described above. Error results if radiation pressure, degeneracy pressure, or other non-ideal gas pressures exceeds the ideal gas pressure used in the GRMHD simulation. Not applicable to X-ray binary and AGN thin disk solution, which is constructed self-consistently as in SS73.

22. τγ: Disk optical depth to photons τγ ∼ κρ0H, where κ is the mean opacity for the given temperature and density (see Bell and Lin 1994 for photon scattering and absorption opacities).

23. τν: Disk optical depth to neutrinos. Since this is negligible for accretion ﬂow near X-ray binaries and AGN, this is only shown for GRB disks.

GRB Disk State

See Appendix A for a discussion of GRB equations of state. The field in GRB accretion disks is slightly above the critical magnetic field where QED effects, such as photon splitting γ → γγ and electron/positron/photon pair generation γ → e+e−γ, occur (see, e.g., Erber 1966). Notice that the thin GRB disks have sub-relativistic protons and neutrons but have marginally relativistic electrons. The NS-BH and collapsar disks have degenerate electrons, but marginally non-degenerate protons and neutrons. For the NS-BH disk, the degeneracy of neutrons leads to a small electron fraction (i.e. high neutronization). The GRB disks are marginally in nuclear statistical equilibrium. At this particular radius, alpha particles dominate free nucleons. The PWF solution at slightly larger radii gives a disk that is hotter (thicker) and so has more free nucleons. Gas pressure dominates the thin GRB disks. The ratio of actual gas pressure to simulation gas pressure shows the NS-BH and collapsar disks are well approximated by the simulation gas pressure. Thus, these estimates are self-consistent. Since the GRB disks are very optically thick, the radiation in the disk can be approximated by a radiation pressure term. The collapsar disk has a low neutrino optical depth, so the optically thin approximation is valid. Thus, a nonradiative GRMHD model can be easily modified to include optically thin neutrino emission in order to study radiative GRMHD collapsar models of long-duration GRBs. The NS-BH disk is estimated to have an optical depth of τ ∼ 1, but the effects of electron degeneracy may lower this value (Kohri and Mineshige, 2002). The states of the plunging and coronal regions are similar to the state of the disk. The funnel region within the NS-BH GRB disk has a magnetic field strength ∼ 300× that of the critical field strength, while the collapsar disk field strength is ∼ 40× the critical field strength. This indicates a need to account for QED effects in the funnel region.

45 Table 1.6. Validity of Fluid Approximation

Parameter NS-BH GRB 030329 LMC X-3 SgrA* NGC4258 M87

11 12 6 11 11 13 L/λmfp,e−p 4.5 × 10 1.7 × 10 1.1 × 10 1.6 × 10 2.1 × 10 4 × 10 10 10 3 7 7 9 T νc,e−p 1.9 × 10 8.2 × 10 2.7 × 10 6.3 × 10 8.7 × 10 7.1 × 10 Re 1.5 × 1014 2.5 × 1014 6.9 × 106 7.2 × 1012 8.3 × 1012 4 × 1015

Note. — If a system has all values much greater than unity, then the ideal fluid approximation is valid. If all but Re are much greater than one, then the viscous fluid approximation is valid. The ideal fluid approximation is valid for all these systems in the thin disk state shown. In a thick disk state (not shown) the X-ray binary and AGN systems do not explicitly behave as a fluid, but may be forced to behave like a fluid due to plasma instabilities, as occurs in the solar wind.

X-ray binary and AGN State

In the thin disk state, the X-ray binary and AGN have low relativity parameters and essentially no degeneracy. Treating the fluid as a nonrelativistic ideal gas is a good approximation as long as either 1) the gas pressure dominates over radiation pressure in the optically thin regime or 2) the disk is optically thick. Notice that all the disks are optically thick at the fiducial radius, so a radiative diffusion treatment can be used to study these disks at this location. Indeed, treating the radiation as simply a pressure may be sufficient. The thick disk in the X-ray binary or AGN is estimated to be H/R ∼ 1. These disks are either marginally optically thick or completely optically thin. The thick disk (ADAF) models of LMC X-3, Sgr A*, and NGC4258 likely require radiative transport to understand their radiative properties at the fiducial radius. However, the thick disk (ADAF) model of M87 is very optically thin and the radiation flux may be modeled by an optically thin emissivity. The states of the (thick) plunging and (thick) coronal regions are similar to the state of the thick disk.

1.5.2 Validity of the Fluid, MHD, and ideal MHD Approximations

Table 1.6 shows the estimates that test the validity of the fluid approximation. See Section A.2 for a discussion of the role of these dimensionless parameters. The value of L/λmfp,A−B is the number of mean free paths, within a distance L, for particle A hitting particle B. The value of T νc,A−B is the number of collisions in a characteristic time. The conditions L/λmfp À 1 and T νc À 1 were tested for all colliding species, but for brevity only the electron-proton values are shown. The value of Re is the Reynolds number. If a system has all values much greater than unity, then the ideal fluid approximation is valid. If all but Re are much greater than one, then the viscous fluid approximation is valid.

46 Table 1.7. Validity of MHD Approximation

Parameter NS-BH GRB 030329 LMC X-3 SgrA* NGC4258 M87

14 14 9 12 13 14 T ωp 2.1 × 10 3.1 × 10 1.7 × 10 6.4 × 10 4 × 10 6.9 × 10 13 13 9 14 14 16 L/λD 5.3 × 10 8.2 × 10 5.4 × 10 1.5 × 10 7.8 × 10 3.5 × 10 Λ 3.7 × 105 1.3 × 105 7.6 × 106 1.4 × 106 5.6 × 106 1.4 × 106

Note. — If all rows are much greater than unity, then the plasma approximation is valid for that system. The plasma approximation is valid for all these systems.

All species are highly coupled in the GRB accretion disks. This includes the photons, which cannot escape the disk. For the hotter regions, the temperature reaches the point where nuclear interactions dominate over Coulomb interactions. The values shown for electron-proton collisions are within a few orders of magnitude of the values for the proton-proton and neutron-proton collisions. The weakest coupling is between protons and photons, but even these together act as a fluid. For the X-ray binary and AGN, the values are shown for the thin disk. The thin X-ray binary and AGN disks have photons well-coupled to electrons, making the disk optically thick. All species are well-coupled except photons and protons. For the X-ray binary and AGN, thick (ADAF) disk models of the disks may not be treatable by the fluid approximation without further consideration. The most coupled pair in these systems is the weak coupling between protons and electrons. Despite the possible necessity of Boltzmann transport in AGN and X-ray binaries with thick disks, likely plasma instabilities operate as in the solar wind and the plasma can be treated effectively as a fluid. The solar wind too cannot be explicitly treated as a fluid, yet the MHD approximation is a reasonable approximation (see, e.g., Usmanov et al. 2000). Plasma instabilities likely force the solar wind to behave as a fluid (Feldman and Marsch, 1997), and similar phenomena may occur in thick disks. In any event, the single- component fluid approximation serves as an interesting starting point for more realistic plasma calculations. The plunging, corona, and funnel region regimes are similar. Table 1.7 shows the estimates that test the validity of the plasma/MHD approximation. See Section A.4 for a discussion of the role of these dimensionless parameters. From top to bottom, these parameters are associated with 1) charge separation and a finite current rise time ; 2) charge separation; and 3) the coupling of plasma (> 1 implies weakly coupled). If all rows are much greater than unity, then the plasma approximation is valid for that system. The plasma approximation is valid for all the states and regions of all objects. Thus, quasi-neutrality holds. Table 1.8 shows the estimates that test the validity of the single-component ideal MHD approximation. See Section A.5 for a discussion of the role of these dimensionless parameters. In summary,

47 Table 1.8. Validity of Ideal MHD Approximation

Parameter NS-BH GRB 030329 LMC X-3 SgrA* NGC4258 M87

12 11 3 5 6 6 T ωg,p 3.1 × 10 5 × 10 7 × 10 2.2 × 10 7 × 10 7 × 10 13 13 6 8 10 10 L/λg,p 5.1 × 10 1.1 × 10 3.8 × 10 8.7 × 10 2.4 × 10 6.2 × 10 2 13 14 11 17 17 19 T ωp/ωg,e 3.4 × 10 3.1 × 10 2.3 × 10 1 × 10 1.2 × 10 3.7 × 10 2 2 10 11 8 14 14 16 Lωpdv/(ωg,ec ) 6.5 × 10 6 × 10 4.4 × 10 1.9 × 10 2.4 × 10 7.1 × 10 2 12 11 6 9 11 12 Ldvωg,e/vtherm,e 2.6 × 10 6.6 × 10 4 × 10 6.6 × 10 1.5 × 10 1 × 10 13 13 6 8 10 10 Lωg,p/vtherm,p 5.1 × 10 1.1 × 10 3.8 × 10 8.7 × 10 2.4 × 10 6.2 × 10 2 14 14 9 12 13 14 (L/λg,p) /(T ωg,p) 8.2 × 10 2.3 × 10 2.1 × 10 3.4 × 10 7.9 × 10 5.4 × 10 13 14 11 14 16 16 RM 6.7 × 10 1.1 × 10 7.6 × 10 5.3 × 10 1.3 × 10 5.4 × 10 5 10 9 12 dv/vdrift,p−n - - 2.3 × 10 2.4 × 10 2.7 × 10 1.2 × 10 5 17 16 21 dv/vdrift,e−n - - 1.9 × 10 1.1 × 10 1.9 × 10 7.3 × 10 11 12 8 14 14 16 dv/vdrift,B 1.5 × 10 1.2 × 10 4.4 × 10 1.9 × 10 2.4 × 10 7.1 × 10

Note. — If the term is much greater than unity, then the associated eﬀect is negligible and further motivates the ideal-MHD approximation. If all terms are much greater than unity, then the single- component ideal-MHD approximation is valid for that system. The single-component ideal-MHD approximation is valid for all these systems.

in order from top to bottom, the parameters correspond to the importance of 1) the gyration time scale ; 2) gyration length scale ; 3) charge separation ; 4) the Hall effect (J × B) ; 5) the electron pressure effect ; 6) an anisotropic pressure ; 7) finite gyration radii (FLR vs. MHD order, where > 1 implies MHD ordering) ; 8) Ohmic dissipation (RM ≡magnetic Reynolds number) ; 9) ambipolar diffusion drift between protons and neutrons ; 10) ambipolar diffusion drift between electrons and neutrons ; and 11) the drift between electrons and protons that generates the current to sustain the magnetic field. If the term is much greater than unity, then the associated effect is negligible and further motivates the ideal-MHD approximation. If all terms are much greater than unity, then the ideal-MHD approximation is valid for that system. The GRB disks (disk thickness set by PWF solution) and both the thin (SS73) and thick (H/R ∼ 1) disk solution for X-ray binary or AGN disks suggest the ideal MHD approximation is valid. The plunging, corona, and funnel regions are qualitatively similar. In summary, the fluid approximation is valid for the GRB disks and thin X-ray binary and AGN disks. However, to maintain fluidity in thick disks in the X-ray binary or AGN, plasma instabilities are required to maintain fluidity. This is likely to occur, as in the solar wind (see, e.g., Usmanov et al. 2000; Feldman and Marsch 1997). The plasma approximation is valid for all disks in all regions. If fluidity is sustained, then the plasma and single-component ideal-MHD approximations are valid for all systems in all regions.

48 1.6 Summary of Motivation for a GRMHD Model and Open Questions

HST, Chandra, and ground-based observations of GRB counterparts/afterglows, such as associated with GRB 020813 and GRB 030329, suggest a supernova connection to GRBs (see, e.g., Uemura et al. 2003; Mészáros2003). These results support the black hole accretion disk model for generating GRB jets (see, e.g. Piran 1999; MacFadyen and Woosley 1999; van Putten and Ostriker 2001; Butler et al. 2003; Mészáros2003). Observations of GRBs will always be folded through models, since the accretion disk will not be resolved in the foreseeable future. The current most attractive model for long duration GRBs is the collapsar model (Woosley, 1993; Paczyński,1998; MacFadyen and Woosley, 1999). In order to determine the flow’s Lorentz factors and the importance of a magnetic field to jet collimation (Cameron, 2001; Wheeler et al., 2002), a GRMHD model is likely required. There are currently about 22 known X-ray binaries with a BHC (McClintock and Remillard, 2003; Zand et al., 2004), where about 18 of those are confirmed (dynamical) black hole X-ray binaries. Estimates of the mass of the black hole come from observations of oscillations in the motion of the stellar companion. For example, the X-ray binary system Cygnus X-1 was found to contain a compact object with a mass of M ∼ 7 − 13 M¯, which is in excess of any possible neutron star (Webster and Murdin, 1972; Bolton, 1972; Gies and Bolton, 1986; Herrero et al., 1995). Some of these X-ray binaries, called microquasars since comparable to quasars and related AGN (Mirabel and Rodr´ıguez,1994, 1999; Fender and Belloni, 2004), exhibit apparently superluminal (v/c ∼ 3 − 10) jets, which likely originate from a relativistic system with a well-defined axis of rotation. Microquasars likely form similarly to the jets that form in GRBs, and also likely require a GRMHD model. Astronomers have long thought that quasars are likely powered by the accretion of matter onto a SMBH (Zeldovich, 1964; Salpeter, 1964; Blandford, 1984), and AGN have long been considered to be related to quasars (Lynden-Bell, 1969). RXTE X-ray observations of AGN show rapid variability on the order of days. Observations of MCG-6-15-30 show variability on timescales of order 100 s (Reynolds et al., 1995; Yaqoob et al., 1997), suggesting that the very inner region of an accretion disk is being probed (Edelson and Vaughan, 2000). Radio interferometry has been able to probe within a few hundred gravitational radii of a central black hole within Sgr A* and M87 (see, e.g., Lo et al. 1998; Junor et al. 1999; Doeleman et al. 2001). Fe Kα line emission shows different intensities for the red and blue wings, which suggests the accretion disk is moving relativistically with speeds approaching 0.2c (Pariev and Bromley, 1998; Tanaka et al., 1995). Thus, a general relativistic model of the accretion disk is required. Some of the first observations of AGN were of so-called radio galaxies with jets extending a few hundred kiloparsecs out of a point-like source. AGN jets are observed to be relativistic with some jets having variable structure moving at an apparent speed of 10 times the speed of light, (see, e.g., West et al. 1998). AGN jets are sources of synchrotron radiation (Baade, 1956), indicating the existence of a strong magnetic field. HST, VLA, and VLBI all provide high-resolution images

49 of jets, including M87’s jet showing detailed large and small scale structure with apparent speeds of v/c ∼ 6 (see, e.g., Biretta et al. 1999). However, only future observatories have any hope of resolving the inner accretion disk at the source of the jet in AGN (Rees, 2001). AGN jets are likely produced similarly to jets in GRBs and microquasars, and also likely require a GRMHD model. Some GRBs, some X-ray binaries, and all AGN, are likely powered by a black hole accretion disk system. The single-component ideal-MHD approximation applies to the disk and surrounding regions near the black hole for all these objects. A GRMHD model is likely required to study this general relativistic, highly magnetized plasma ﬂow. Since the GRMHD equations are non-trivial to study analytically, it is necessary to perform numerical studies of these systems. In particular, self- consistent, time-dependent studies of the global solution (i.e. disk-corona or disk-jet connection) require a numerical model. It has yet to be determined what fraction of the observed luminosity from AGN or X-ray binaries is due to (a) a radiative accretion disk, (b) black hole spin energy extraction, or (c) a disk wind. For GRBs, photon production due to neutrino annihilation should also be considered as a component of the luminosity. A radiative GRMHD model is required to determine the disk contribution to the luminosity due to photons or neutrinos, and has yet to be studied self-consistently for the global accretion ﬂow. To determine the contribution of energy from black hole spin energy extraction or the disk near the black hole, one requires a GRMHD model. Unanswered questions about AGN jets, microquasar jets, and GRB jets include: 1) what is the energy source for jets? ; 2) what is the connection between the disk and jet? ; 3) what is the mechanism for jet production, collimation, and variability? ; and 4) is the jet composed of ion/electron or positron/electron pair plasma?

1.7 Summary of Dissertation Results

The introduction summarized the evidence for black hole accretion disk systems and discussed how black hole accretion disk systems exhibit relativistic magnetized flow near a (likely rotating) black hole. It was discussed how a magnetic field self-consistently generates turbulence and angular momentum transport in a disk, while unmagnetized models using an α-viscosity model introduce ad hoc parameters. The observational evidence for relativistic magnetized flow, and the lack of self- consistency in unmagnetized models, demonstrates that a GRMHD model is required. A discussion was presented of radiative accretion disk models, which find the structure of an accretion disk by solving the equations of mass, energy, and momentum conservation. At the end of the introduction, the state of the accretion disk in 6 fiducial systems was discussed, and the single-component ideal MHD approximation was found to be valid for these 6 fiducial black hole accretion disk systems. It was shown that an optically thin radiative GRMHD collapsar model can accurately and self- consistently model some long-duration GRBs. Finally, a summary was provided for why a GRMHD numerical model is required to study accretion flow in GRBs, X-ray binaries, and AGN. Some isolated regions of an accretion flow may be well-described by simplified analytic solutions,

50 such as the Blandford-Znajek (BZ) solution in the nearly force-free funnel region around a rotating black hole (Blandford and Znajek, 1977) and the Gammie inflow solution for the high-density plasma flow in the equatorial plunging region (Gammie, 1999). Typically, in deriving analytic solutions, the simplifying assumptions do not allow an understanding of the connection between, and a self-consistent solution for, the black hole, disk, corona, and jet. This is referred to as describing the “global” accretion flow, rather than the “local” accretion flow. Despite some successful simplified analytic solutions, the VHD, MHD, and GRMHD equations are sufficiently complicated that numerical studies would likely yield results not easily (or possibly) derived analytically. Numerical studies are likely required to understand the global, time-dependent structure of accretion flows around rapidly rotating black holes. In order to study black hole accretion disk systems, I first developed a code that integrates the 2D global Newtonian VHD equations of motion with a pseudo-Newtonian potential (McKinney and Gammie, 2002); then I developed a code that integrates the 2D and 3D global nonrelativistic MHD equations of motion with a partially relativistic treatment of Alfvénand sound waves and an artificial resistivity to capture heating in current sheets (unpublished); and finally I helped develop a code that integrates the GRMHD equations of motion in a stationary space-time (Gammie et al., 2003). The main results of this thesis are that 1) the inner radial boundary condition must stay out of causal contact with the rest of the flow in order to avoid nonphysical outflows and other numerically-induced artifacts; and the viscosity models are compromised by the arbitrariness of model parameters (McKinney and Gammie, 2002); 2) the BZ-effect is likely an important source of energy due to the development of a strong magnetic field in the funnel region; 3) the BZ model accurately predicts the electromagnetic luminosity in the nearly force-free funnel region near black holes with spin parameter a . 0.5 ; 4) the BZ model is qualitatively accurate for all black hole spins; 5) the BZ-effect contributes significantly to the energy content of jets in systems with a thick disk around a black hole with a spin parameter of a & 0.5; 6) for thick or thin disks, the nominal accretion efficiency is typically close to the classical thin disk value and generally less than unity (McKinney and Gammie, 2004), while the Gammie inflow model suggested the efficiency may be larger than thin disk (including super-efficient) due to magnetic torques in the plunging region (Gammie, 1999) ; and 7) for thick and thin disks, the angular momentum per baryon accreted is consistent with the predictions of the Gammie inflow model, which suggested that the angular momentum per baryon accreted should be less than predicted by viscous, unmagnetized thin disk models. The following discussion summarizes the results of this thesis to be described in detail in chapters 2 to chapters 4. Following these chapters, in chapter 5, is a summary of possible future studies.

51 1.7.1 Viscous Hydrodynamics Summary

The turbulent transport of angular momentum has long been suspected as the cause of accretion in disks around stars. Viscosity is a natural model to capture the effective friction due to the mixing action of turbulent flow (Shakura and Sunyaev, 1973). This viscosity is called an anomalous viscosity, since the true molecular viscosity is too small to drive angular momentum transport. Typically, the anomalous viscosity is estimated as generating a stress proportional to αP , where α is a constant and P is the pressure. Such a model is referred to as an α-disk model. The actual mechanism of angular momentum transport was unknown until the rediscovery of the MRI (Balbus and Hawley, 1991). The MRI likely dominates the generation of turbulence in ionized accretion disks (Balbus and Hawley, 1998). I decided to develop a viscous hydrodynamics (VHD) code to study black hole accretion. The VHD code uses a numerical method based on ZEUS-2D (Stone and Norman, 1992) with the addition of an explicit scheme for the viscosity. ZEUS uses an operator-split, finite-difference algorithm on a staggered mesh that uses an “artificial viscosity” to capture shocks. This artificial viscosity is only activated in shocked regions. I was particularly interested in measuring the mass, energy, and angular momentum flux through the inner radial boundary. These diagnostics are useful in a discussion of accretion disk luminosity and variability. I used the VHD code to study black hole accretion models, which were similar to previously studied models using various experimental designs (Igumenshchev and Abramowicz 1999, 2000; Igumenshchev et al. 2000; all hereafter IA), (Stone et al. 1999, hereafter SPB). IA and SPB used different models for the accretion disk, anomalous viscosity, and gravitational potential. Unfortu- nately, these authors found quite different results using otherwise similar methods. The primary differences between IA’s and SPB’s results are in the energy per baryon and angular momentum per baryon accreted. IA and SPB also found different results for the radial scaling power laws of density, pressure, energy per baryon accreted, and angular momentum per baryon accreted. It was uncertain if these differences were due to the experimental design or due to bugs in the numerical code. Our goal was to form a bridge between these different results by using an identical code to study both types of experiments. IA chose to form an accretion disk by injecting marginally bound matter (Bernoulli parameter <

0) far from the hole at the rate M˙ 0,inj. SPB choose to start with a torus in hydrostatic equilibrium. IA include all components of the (spherical polar) Navier-Stokes viscous stress tensor, while SPB only include the r − φ and θ − φ components. SPB only include the toroidal shear components. They suggest the differential rotation leading to the MRI causes negligible poloidal shear compared to the toroidal shear. Also, IA and SPB use different forms for the viscosity coefficient ν (units of p length2/time). IA use ν ∝ c2/Ω , where c = ∂P/∂ρ | is the sound speed, S is the entropy, p s k s 0 S 3 and Ωk = GM/r is the Keplerian angular velocity. SPB typically use ν ∝ ρ0, where ρ0 is the rest-mass density. IA’s viscosity prescription is chosen to concentrate the stress where the disk is hot, while SPB’s choice is meant to confine the stress to the bulk of the mass. Any dimensionally

52 reasonable form of the viscosity coefficient can be used, but the goal is to choose a form that mimics the MRI driven angular momentum transport. I used a single VHD code to study both IA’s and SPB’s experimental designs, and I was able to reproduce the results of both IA and SPB. This confirms that the differences in their results were not due to a bug, but due to differences in their experimental design. Model choices that introduced the most significant differences include the choice of viscosity prescription and how to model the source of matter. However, I also found that both IA’s and SPB’s choices for other model parameters can lead to numerical artifacts. The model parameters in question are the radius of the inner boundary condition (rin) and the form of the black hole’s gravitational potential. The gravitational potential can be either purely Newtonian (φ = GM/r) or pseudo-Newtonian (φ = GM/(r − 2GM/c2)) (Paczyńskiand Wiita, 1980). There are two competing factors in deciding where to place the inner boundary: 1) the desire to produce a simulation free of numerical artifacts; and 2) computational cost. The inner boundary’s location, rin, must be chosen so the boundary condition applied there does not affect the flow at 2 r > rin. In modeling a black hole, one must choose rin so that as rin → 2GM/c there are negligible 2 changes in the solution. Thus, one must convergence test a solution as rin → 2GM/c . Of course, a Newtonian potential has no intrinsic length scale. However, as in Bondi flow (Bondi,

1952), the sonic point (the radius for supersonic flow at which −vr/cs = 1) can determine a length scale. For inviscid flow, the sonic point determines the physical point at which information cannot travel upstream. In more complicated flows, this supersonic boundary may be time-dependent, but one can define a minimum distance away from rin that always contains a supersonic flow. Indeed, near a black hole there must exist a surface that is always supersonic (Shapiro and Teukolsky, 1983). 2 2 Moving rin → 2GM/c in a nonrelativistic code is expensive (time step, dt ∝ (r − 2GM/c )). Thus, the inner boundary is usually placed close to the horizon, but far enough away, so the simulation can be completed in a reasonable time. However, if rin is chosen such that the flow is subsonic at the inner boundary, information can travel back into the flow. Clearly, this violates the physical model of a black hole and may lead to spurious measurements for the accretion rate of mass, angular momentum, and energy.

For the VHD numerical models with a pseudo-Newtonian potential, only by choosing rin . 2 2.7GM/c was the flow supersonic at rin. In comparison, a numerical model with a pseudo- 2 Newtonian potential with rin = 6GM/c has subsonic flow at rin. As studied by IA, a numerical model with a Newtonian potential has subsonic flow at all rin. I found that the accretion of energy and angular momentum per baryon are affected by numerical artifacts for models with subsonic flow at rin. Models with subsonic flow at rin show up to 5 times lower rest-mass accretion rates, and such models show spurious high-frequency oscillations in the accretion rates of mass, energy, and angular momentum. Models with a Newtonian potential never develop consistent supersonic flow at rin, and the angular momentum per baryon accreted oscillates around zero instead of taking a definite value. Models with subsonic flow at rin showed

53 more complicated outflows of matter and energy in the polar regions. Thus, significant numerical artifacts can be avoided by 1) using a pseudo-Newtonian potential; and 2) choosing a suitable rin, such that the flow is supersonic there. These VHD simulations are limited by the 2D axisymmetric condition, the use of viscosity as a model of turbulence (as opposed to direction simulations of MHD turbulence), and use of the pseudo-Newtonian potential with no relativistic corrections.

1.7.2 Global 2D/3D MHD Summary

The α-disk viscosity model continues, to this day, to be used as a model for the turbulent transport of angular momentum in accretion disks. The thin α-disk model of accretion, based upon this turbulent viscosity, predicts that the disk plunges radially into the black hole in a sharp transition at the innermost stable circular orbit (ISCO) (Bardeen, 1970; Shakura and Sunyaev, 1973; Page and Thorne, 1974; Thorne, 1974; Abramowicz et al., 1978). However, the true kinematic viscosity is much lower than the anomalous viscosity used in α-disk models. A magnetic field was discovered to develop a magneto-rotational instability (MRI) in accretion disks with a high ionization fraction (Balbus and Hawley, 1991; Hawley and Balbus, 1991). The MRI dominates kinematic viscosity as a mechanism for the generation of turbulence and angular momentum transport in accretion disks (Balbus and Hawley, 1998). The magnetic field may be dynamically important inside the ISCO and generate a non-negligible torque on the disk, and the magnetic field could allow for a super-efficient nominal accretion luminosity (Gammie, 1999; Krolik, 1999b). A magnetic field also likely plays a central role in the generation and collimation of a jet near a compact object (see, e.g., Blandford and Znajek 1977; Blandford and Payne 1982; Begelman et al. 1984). In particular, the BZ effect may be responsible for producing a jet in gamma-ray bursts (GRB), microquasars, and active galactic nuclei (AGN) (Blandford and Znajek, 1977). The magnetic field could generate and heat corona (see, e.g., Stern et al. 1995). I was interested in solving the 2D or 3D MHD equations of motion as applied to black hole accretion systems to determine 1) whether the torque on the disk diminishes near the ISCO ; 2) whether a magnetic field can launch and collimate jets; and 3) whether the polar field is strong enough to make the BZ-effect comparable in magnitude to the accretion disk luminosity. I was also interested in whether the polar BZ-effect provides the energy source for jets, but a direct measurement of the BZ-effect requires a general relativistic calculation. I developed a ZEUS-type code (Stone and Norman, 1992) that evolves the equations of nonrelativistic MHD. This MHD code uses an artificial resistivity (Stone and Pringle, 2001) to avoid the generation of a class of numerical artifacts called “point shocks” and poor energy conservation in the dissipation of current sheets. The MHD code also includes relativistic corrections that limit the Alfvénand sound speed to the speed of light (Miller and Stone, 2000), as described below. Typically, MHD numerical studies of a black hole accretion system show the development of an evacuated funnel region around the poles. In unpublished simulations, I found that this funnel

54 √ region had Alfvénspeed va = B/ ρ0 ∼ 100 − 1000c. Since the stability criteria on the time step is dt < dx/va (dx being some grid size), a large Alfvénspeed sharply reduces the allowed time step. I modified the equations of motion to limit the speed of Alfvénand sound waves to the speed of light ; this is referred to as a wave speed limiter. The method involves including a partial correction for the p 2 2 displacement current. The modified equations of motion have Alfvénspeed va = B/ ρ0 + B /c , so as ρ0 → 0, va → c. The sound speed is similarly limited. The axisymmetric approximation may lead to unrealistic accretion flow geometries. Global 3D simulations of nonaxisymmetric accretion flow can give rise to a self-sustained dynamo, while axisymmetric flow cannot (Cowling, 1934). A numerical method that uses 3D spherical polar coordinates, rather than 3D Cartesian coordinates, allows more accurate (less diffusive) simulations of spherical or cylindrical accretion flow. However, the coordinate singularity in such a coordinate system presents at least two problems. First, the coordinate singularity makes it difficult to accurately model non-radial flow near the axis. Second, the Alfvénwave speed stability criteria on the time step near the polar axis in 3D spherical polar coordinates gives dt ∝ r sin θ/va for motion in the φ-direction. For a standard rectangular-based grid, the flow is always over-resolved in the φ-direction near the outer radial region of the polar axis, and so simulations require many more time steps to complete an evolution. The MHD code was extended to include a type of 3D Cartesian grid in order to avoid these two problems. In this case, there is no coordinate singularity. In order to maintain decent spherical symmetry and avoid introducing a strong m = 4 mode due to the Cartesian grid, I model the inner and outer boundaries of the grid as pseudo-spherical (i.e. Cartesian grid approximation of an inner and outer sphere). Using this MHD code, I reproduced work by Hawley (2000) and Stone and Pringle (2001) who numerically model a thick disk of magnetized fluid using the MHD approximation. I found that there is no sharp transition at the ISCO in any fluid quantities, and the torque does not vanish near the ISCO. Thin disks studies are required to compare with the thin disk model, but resolving a thin disk requires much more computer power and has not yet been attempted. Nonrelativistic or pseudo-Newtonian (Paczyńskiand Wiita, 1980) MHD models cannot model black hole rotation, hence cannot show black hole energy extraction by the BZ-effect. However, 2 2 dimensional analysis allows one to estimate the BZ luminosity as LBZ ∼ B r+c, where B is the magnetic field and r+ is the radius of the horizon. Since jets form around the polar axis of an accretion disk near the black hole, I determine this dimensionally estimated BZ luminosity in the polar region. The numerical model of an accretion disk thus provides the ratio of the BZ luminosity per unit rest-mass accretion rate. If I assume an Eddington rest-mass accretion rate, then I find that the BZ luminosity is comparable to the energy generation rate observed in jets and radio lobes in the most luminous AGN. Simulations also show a collimated Poynting flux dominated jet in the funnel region along the polar axis. However, while the flow appears to be relativistic, there is no way to determine this accurately with a (mostly) nonrelativistic numerical model. The modified Newtonian code was

55 unable to properly evolve the relativistic flow in the funnel, since the Alfvénand sound speeds reached near the speed of light. For example, I found it impossible to remove the inner radial boundary condition from causal contact with the rest of the flow (i.e. one cannot keep the inner radial boundary inside the fast point in MHD). Also, while I found a large Poynting flux from the horizon, the Newtonian model should not be capable of producing such an effect. Most of the Poynting flux energy could be artificially generated by contact between the inner radial boundary and the flow at larger radii. Thus, confidence was lost in the numerical evolution of the funnel region. The Lorentz factor of the jet and the dependence of the jet and BZ luminosity on black hole spin remain undetermined until a general relativistic calculation is performed. Ultimately, these problems with the (unpublished) nonrelativistic MHD results pushed us to develop a general relativistic code.

1.7.3 HARM / GRMHD Summary

In collaboration with Charles Gammie and Gábor Tóth,I helped developed the HARM (High Accuracy Relativistic Magnetohydrodynamics) code. HARM directly solves the equations of ideal MHD in conservative form using a Godunov method that solves for the conserved fluxes using a simplified Lax-Friedrich (LF) or Harten-Lax-van Leer (HLL) scheme. Conservative methods can accurately resolve shocks. However, if the various types of energy (rest-mass, internal energy, kinetic energy, and magnetic energy) in the problem become disparate in magnitude, then due to the form of the conservative equations of motion (the energy scales are sometimes summed together) truncation error in one type of energy can lead to large errors in other energies. HARM uses 8 primitive quantities: 1) comoving mass density; 2) comoving internal energy density ; 3,4,5) three velocity components; and 6,7,8) three magnetic field components. HARM uses the primitive quantities to compute 8 conserved quantities: 1) rest-mass density or, equivalently, particle number ; 2) total energy; 3,4,5) three momentums (one is angular momentum); and 6,7,8) three magnetic fluxes. The physically conserved quantities are total rest-mass (or total particle number), energy, angular momentum, and magnetic flux. HARM computes the conserved quantities from primitive quantities ; evolves the conservative quantities ; and inverts the conserved quantities to primitive quantities using a multidimensional Newton-Raphson method. At the same time as HARM was being written, De Villiers and Hawley (2003a) developed a fully relativistic code using a different numerical method. Their code solves the nonconservative form of the equations of motion where primitive variables are directly evolved. This has the advantage of being faster than HARM, since it avoids the inversion of conserved to primitive variables. It is also less diffusive compared to LF or HLL methods. However, their method has problems with accuracy in strong shocks and it is uncertain whether loss of conservation of particle number, energy, and angular momentum significantly affects the solution. In practice, HARM and the method by De Villiers and Hawley (2003a) give comparable results when applied in astrophysical contexts. Indeed, their method is better able to handle regions with

56 2 b /ρ0 À 1 (magnetic energy density to mass energy density ratio much greater than one), such as the funnel region. However, both codes have difficulty in this funnel region when evolving an accretion disk around a rapidly spinning black hole. Their current method uses Boyer-Lindquist coordinates that requires a high-resolution grid near the event horizon for an accurate measurement of the accretion rate of energy and angular momentum per unit mass. HARM uses Kerr-Schild (horizon-penetrating) coordinates, and this allows HARM to find reasonable accretion rates of energy and angular momentum at lower resolution. 2 HARM could be improved to handle flows with larger b /ρ0 by using methods that reduce the truncation error. HARM is limited by the exclusion of dynamic space-time, radiation, and other microphysics. Some additional physics will likely continue to be modeled phenomenologically. One useful phenomenological model is a cooling function that maintains a constant disk scale height to radius ratio (H/R). This can be used to study disks with different H/R independent of the origin of the scale height, which would naturally be determined by balancing radiative and gas pressure effects. Other physical processes that do not dynamically couple to the accretion flow, such as the synchrotron cooling in a jet, can be accurately modeled without significant modification to HARM.

1.7.4 BZ Eﬀect Summary

Some GRBs, microquasars, and AGN may be powered by the electromagnetic braking of a rapidly rotating black hole. The BZ-effect (here, broadly defined as any electromagnetic means of extracting energy from a rotating black hole) is the most likely astrophysical means of extracting energy from a rapidly rotating black hole. Estimates for the nominal black hole spin in astrophysical environments give a rapid black hole spin of about a ∼ 0.92 (Gammie et al., 2004). Phenomenological estimates determined that the BZ luminosity is likely small compared to the disk luminosity (Ghosh and Abramowicz, 1997; Armitage and Natarajan, 1999; Livio et al., 1999). However, phenomenological models have only been worked out for simple field geometries. Given the complicated nature of the accretion flow and its connection to the jet, their estimates may not apply to physical accretion flow or to a global solution of the disk + jet (McKinney and Gammie, 2004). I investigated the BZ-effect using HARM via axisymmetric numerical simulations of a rapidly rotating black hole surrounded by a magnetized plasma. The plasma is described by the equations of GRMHD, and the effects of radiation are neglected. The evolution is followed for 2000GM/c3, and the computational domain extends from inside the event horizon to typically 40GM/c2. The initial conditions are similar to the nonrelativistic MHD study. The numerical model starts with a plasma torus, in hydrostatic equilibrium, with an overlaid weak magnetic field. The torus is at an initially fixed H/R ∼ 0.26, and typically remains at that H/R since no radiation was included. I found that all models with a & 0.5 have an outward Poynting flux on the horizon in the Kerr- Schild frame, which means energy is extracted from the black hole. None of the models have net outward energy flux on the horizon. One model, with a net magnetic flux through the disk, shows a net outward angular momentum flux on the horizon. The model with a net magnetic flux in the

57 disk also has the largest black hole energy extraction rate per unit nominal accretion luminosity of 80%. Thus, the BZ luminosity can be comparable to the disk luminosity. The limitations of the numerical models presented include the assumption of axisymmetry, an ideal gas equation of state, and a nonradiative gas. The assumption of axisymmetry is likely not problematic since I find quantitative agreement with 3D results for the energy and angular momentum per baryon accreted through the horizon (De Villiers and Hawley, 2003a; De Villiers et al., 2003a) and the ratio of electromagnetic energy flux to matter energy flux on the horizon (Krolik, private communication). Models in axisymmetry may overestimate the BZ effect by, for example, allowing the presence of axisymmetric sheets of magnetic field. In 3D, these sheets would be disrupted by MHD turbulence. Also, a compressed or tangled axisymmetrically-forced magnetic field is less likely to diffuse than the same magnetic field in 3D. Radiative effects are crucial for comparing with observations, but are only easy to implement in limited form, e.g. weak synchrotron cooling in jets. Radiative effects are likely dynamically important in accretion disks in X-ray binaries and AGN, such as demonstrated by the photon bubble instability (Gammie, 1998).

1.7.5 BZ/Inﬂow Solution Comparison Summary

I compared the GRMHD numerical results of black hole accretion to two analytic steady state models: 1) the force-free magnetosphere solution of Blandford & Znajek (Blandford and Znajek, 1977) and 2) the equatorial inflow solution of Gammie (Gammie, 1999). The BZ solution applies to the funnel region near the polar axis of the rotating black hole, while the Gammie inflow solution applies to the equatorial region inside the ISCO. I presented a self-contained rederivation of the Blandford-Znajek model in Kerr-Schild (horizon penetrating) coordinates. Unlike the original BZ derivation, this solution does not require the use of a physical observer to determine the boundary condition on the coordinate singularity at the horizon, as required in Boyer-Lindquist coordinates. The solution for the far-field radial dependence of the energy flux only relies on the solution’s separability, rather than requiring the solution to match Michel’s (1973) solution, as done by BZ. I used HARM to evolve an accretion disk around a black hole with a spin parameter of a = 0.5. A nearly force-free region developed in an evacuated funnel region around the poles of the black hole. Thus, in this region the BZ solution could be compared to the solution from the numerical model. In this force-free region, on the horizon, I measured 1) the radial magnetic field (Br), 2) the electromagnetic rotation frequency (ω), and 3) the electromagnetic energy extracted (E˙ (EM)). The analytic BZ solution of these quantities was found to be in excellent agreement with the GRMHD numerical model solution. Since the BZ solution is a perturbative solution, valid for a ¿ 1, rapidly rotating black holes are not expected to follow the BZ solution. However, all black hole spins (−0.938 ≤ a ≤ 0.969) and field geometries tested show a nearly maximally efficient BZ-process with ω ≈ ΩH /2 on the horizon, as expected (Thorne and MacDonald, 1982), where ΩH ≡ a/(2r+) is the rotation frequency of the black hole. Models with arbitrarily large black hole spin show a

58 nearly force-free evacuated funnel region with a BZ-luminosity that is qualitatively similar to the BZ solution. As in the BZ solution, the numerical solution contains field lines that extend to large radius in the nearly force-free funnel region. As predicted by the BZ solution, along such field lines the value of ω is nearly constant. The Gammie inflow solution determines the fully relativistic MHD flow quantities for a stationary, cold (H/R = 0) disk. The solution is completely determined by the rest-mass accretion rate and a parameter that sets the magnetization of the flow. The Gammie inflow solution does fairly well at predicting the space-time average of many flow quantities. In particular, the Gammie inflow agrees fairly with the numerical solution for the accretion rate of particle energy vs. radius, accretion rate of electromagnetic energy vs. radius, and comoving magnetic energy vs. radius. The radial velocity and mass density vs. radius do not fit well. This is partially because the Gammie inflow solution does not model hot flow, while the numerical solution is of a hot disk. Future analytic studies may investigate a hot inflow solution, and relax other assumptions (see, e.g., Li 2004), to compare with numerical results. The Gammie inflow model suggests that magnetic torques in the plunging region may increase the nominal accretion efficiency η (even beyond unity). I found all models have η < 1. Remarkably, the efficiency closely follows the thin disk efficiency of Bardeen (1970).

59 Numerical Models of Viscous 2 Accretion Flows Near Black Holes

2.1 Summary of Chapter

We report on a numerical study of viscous fluid accretion onto a black hole. The flow is axisymmetric and uses a pseudo-Newtonian potential to model relativistic effects near the event horizon. The numerical method is a variant of the ZEUS code. As a test of our numerical scheme, we are able to reproduce results from earlier, similar work by Igumenshchev and Abramowicz and Stone et al. We consider models in which mass is injected onto the grid as well as models in which an initial equilibrium torus is accreted. In each model we measure three “eigenvalues” of the flow: the accretion rate of mass, angular momentum, and energy. We find that the eigenvalues are sensitive

to rin, the location of the inner radial boundary. Only when the ﬂow is always supersonic on

the inner boundary are the eigenvalues insensitive to small changes in rin. We also report on the sensitivity of the results to other numerical parameters.1

2.2 Introduction

Black hole accretion flows are the most likely central engine for quasars and active galactic nuclei (AGN) (Zeldovich, 1964; Salpeter, 1964). As such they are the subject of intense astrophysical interest and speculation. Recent observations from XMM-Newton, Chandra, Hubble, VLBA, and other ground- and space-based observatories have expanded our understanding of the time variability, spectra, and spatial structure of AGN. Radio interferometry, in particular, has been able to probe within a few hundred gravitational radii (GM/c2) of the central black hole, e.g. Lo et al. (1998); Junor et al. (1999); Doeleman et al. (2001). Despite these observational advances, only instruments now in the concept phase will have sufficient angular resolution to spatially resolve the inner accretion disk (Rees, 2001). And so there remain fundamental questions that we can only answer by folding observations through models of AGN structure. All black hole accretion flow models require that angular momentum be removed from the flow in some way so that material can flow inwards. In one group of models, angular momentum is removed directly from the inflow by, e.g., a magneto-centrifugal wind (Blandford and Payne, 1982). Here we will focus on the other group of models in which angular momentum is diffused outward

1Published in ApJ Volume 573, Issue 2, pp. 728-737. Reproduction for this thesis is authorized by the copyright holder.

60 through the accretion flow. It has long been suspected that the diffusion of angular momentum through an accretion flow is driven by turbulence. The α model (Shakura and Sunyaev, 1973) introduced a phenomenological shear stress into the equations of motion to model the effects of this turbulence. This shear stress is proportional to αP , where α is a dimensionless constant and P is the (gas or gas + radiation) pressure. This shear stress permits an exchange of angular momentum between neighboring, differentially rotating layers in an accretion disk. In this sense it is analogous to a viscosity (see also Lynden-Bell and Pringle (1974)) and is often referred to as the “anomalous viscosity.” The α model artfully avoids the question of the origin and nature of turbulence in accretion disks. This allows useful estimates to be made absent the solution to a difficult, perhaps in- tractable, problem. Recently, however, significant progress has been made in understanding the origin of turbulence in accretion flows. It is now known that, in the magnetohydrodynamic (MHD) approximation, an accreting, differentially rotating plasma is destabilized by a weak magnetic field (Balbus and Hawley, 1991; Hawley and Balbus, 1991). This magneto-rotational instability (MRI) generates angular momentum transport under a broad range of conditions. Numerical work has shown that in a plasma that is fully ionized, which is likely the case for the inner regions of most black hole accretion flows, the MRI is capable of sustaining turbulence in the nonlinear regime (Hawley and Balbus, 1991; Hawley et al., 1995; Hawley, 2000; Hawley and Krolik, 2001). Studies of unmagnetized disks have greatly reduced the probability that a linear or nonlinear hydrodynamic instability drives disk turbulence. While there are known global hydrodynamic instabilities that could in principle initiate turbulence, these have turned out to saturate at low levels or require conditions that are not relevant to an accretion disk near a black hole. As of this writing, no local, linear or nonlinear hydrodynamic instabilities that transport angular momentum outwards are known to exist in Keplerian disks (Balbus and Hawley, 1998). Work on magnetized disks has now turned to global numerical models. These are possible thanks to advances in computer hardware and algorithms. Recent work by Hawley (2000); Hawley and Krolik (2001), Stone and Pringle (2001), and Hawley and Krolik (2001) considers the evolution of inviscid, nonrelativistic MHD accretion flows in two or three dimensions. Some of this work uses a pseudo-Newtonian, or Paczyńskiand Wiita (1980), potential as a model for the effects of strong-field gravity near the event horizon. Other work on global models has considered the equations of viscous, compressible fluid dynamics as a model for the accreting plasma (Igumenshchev and Abramowicz, 1999; Stone et al., 1999; Igumenshchev and Abramowicz, 2000; Igumenshchev et al., 2000). The viscosity is meant to mock up the effect of small scale turbulence, presumably generated by magnetic fields, on the large scale flow. In light of work on numerical MHD models, this may seem like a step backwards. The MHD models, however, are computationally expensive and introduce new problems with respect to initial and boundary conditions. It therefore seems reasonable to investigate the less expensive α based viscosity models. In this paper we investigate axisymmetric, numerical, viscous inflow models. This work was motivated by the earlier work of Igumenshchev and Abramowicz (1999, 2000) and

61 Stone et al. (1999), hereafter referred to as IA99, IA00, and SPB99, respectively (IA99 and IA00 are collectively referred to as IA, in which case SPB99 is simply referred to as SPB). These authors studied similar viscous inflow models yet found different radial scaling laws for radial velocity, density, and angular momentum. They also found different values for the accretion rate of mass and angular momentum. They used different experimental designs, however. We set out to discover whether the results from these authors differed due to numerical methods or model parameters. Along the way, we took a systematic approach to studying numerical parameters and boundary conditions. One particular point of concern, which will be described in greater detail below, is the inner boundary condition. This lies in the energetically dominant portion of the flow, so errors there can potentially corrupt the entire model. In this paper we show that aspects of results presented by other researchers are sensitive to model and numerical parameters. These results should be useful to others contemplating large-scale numerical models of black hole accretion. The paper is organized as follows. In § 2.3 we discuss our models. In § 2.4 we discuss numerical methods. In § 2.5 we discuss a fiducial solution and results from a survey of other solutions. In § 2.6 we summarize our results.

2.3 Model

We are interested in modeling the plasma within a few hundred GM/c2 of a black hole. We will consider only axisymmetric models (the work of Igumenshchev et al. (2000) suggests that 2D and 3D viscous models give similar results). Throughout we use standard spherical polar coordinates r, θ, and φ. We solve numerically the axisymmetric, nonrelativistic equations of compressible hydrodynamics in the presence of an anomalous stress Π, which is meant to model the eﬀects of small-scale turbulence on the mean ﬂow. The governing equations then express the conservation of mass

Dρ 0 + ρ (∇ · v) = 0, (2.1) Dt 0 momentum, Dv ρ = −∇P − ρ ∇Ψ − ∇ · Π, (2.2) 0 Dt 0 and energy, Du = −(P + u)(∇ · v) + Φ. (2.3) Dt

Here, as usual, D/Dt ≡ ∂/∂t + v · ∇ is the Lagrangian time derivative, ρ0 is the rest-mass density, v is the velocity, u is the internal energy density, P is the pressure, and Ψ is the gravitational potential. The dissipation function Φ is given by the product of the anomalous stress tensor Π with the rate-of-strain tensor e ij Φ = Πije , (2.4)

62 (sum over indices) where the anomalous stress tensor is the term-by-term product

Πij = −2ρ0νeijSij, (2.5)

(no sum over indices) where Sij is a symmetric matrix ﬁlled with 0 or 1 that serves as a switch for each component of the anomalous stress. The rate of strain tensor e is a symmetric tensor that in spherical polar coordinates has ∂v 1 err = r − (∇ · v), (2.6) ∂r 3 1 ∂v v 1 eθθ = θ + r − (∇ · v), (2.7) r ∂θ r 3 v v 1 1 ∂v eφφ = r + θ cot θ − (∇ · v) + φ , (2.8) r r 3 r sin θ ∂φ 1 ∂ v 1 ∂v erθ = (r ( θ ) + r ), (2.9) 2 ∂r r r ∂θ 1 ∂ v 1 ∂v erφ = (r ( φ ) + r ), (2.10) 2 ∂r r r sin θ ∂φ and 1 sin θ ∂ v 1 ∂v eθφ = ( ( φ ) + θ ). (2.11) 2 r ∂θ sin θ r sin θ ∂φ The equation of state is P = (γ − 1)u. (2.12)

For the gravitational potential we use the pseudo-Newtonian potential of Paczyńskiand Wiita 2 (1980): Ψ = −GM/(r − rg) (here rg ≡ 2GM/c ). This potential reproduces features of the orbital structure of a Schwarzschild spacetime, including an innermost stable circular orbit (ISCO) located at r = 6GM/c2. In a few cases we use the Newtonian potential Ψ = −GM/r for comparison with others’ work. IA and SPB describe results exclusively from a Newtonian potential, although IA00 report in a footnote that otherwise identical experiments in a pseudo-Newtonian potential show significant changes in the flow structure for r ∼ 2GM/c2. We must now make some choices for the anomalous stress tensor. One might argue on very general grounds for a Navier-Stokes prescription, and indeed IA and we use a prescription where all elements of S are 1 (the “IA prescription”). SPB, on the other hand, use the Navier-Stokes prescription with all components zero except Srφ, Sθφ, Sφr, and Sφθ (the “SPB prescription”). SPB justify this choice by arguing that it more appropriately models MHD turbulence; this was later supported by results presented in Stone and Pringle (2001). We must also choose a viscosity coefficient. We consider three different prescriptions: one similar to those chosen by IA; a second viscosity coefficient similar to that chosen by SPB; and a third, similar form that vanishes rapidly near the poles. Explicitly,

2 ν = α(cs/ΩK ), (2.13)

63 2 ν = α(ρ0/ρ?)Ω0r0, (2.14)

2 3/2 ν = α(cs/ΩK )sin (θ), (2.15) p are the IA, SPB, and MG prescriptions, respectively, where cs = γP/ρ0 is the sound speed, and

2 1 ∂Ψ GM ΩK ≡ = 2 (2.16) r ∂r r(r − rg) is the “Keplerian” angular velocity. Here ρ? and Ω0 are values of ρ0 and Ω at a fiducial radius r0. The choice of viscosity coefficient for IA and MG is based, as usual, on dimensional arguments. SPB99’s choice focuses the viscosity where most of the matter is, a numerical convenience. Our MG prescription is a small modification of the IA prescription to concentrate the viscosity toward the equator. These choices are to a large extent arbitrary, although one might attempt to motivate the choice by comparison with MHD simulations, as do Stone and Pringle (2001). Nevertheless, some dynamical properties of MHD turbulence, such as the elastic properties that produce magnetic tension and hence Alfven waves, can never be modeled with a viscosity.

The model has boundaries at θ = ±π/2 and at r = rin, rout. At the θ boundaries we use the usual polar axis boundary conditions. At the radial boundaries we use “outflow” boundary conditions; ideally these boundary conditions should be completely transparent to outgoing waves. Fuel for the accretion flow must be provided either in the initial conditions or continuously over the model evolution. Some global numerical accretion flow models have started with an equilibrium torus. Examples include Hawley (1991), Hawley et al. (1995), Hawley (2000), Hawley and Krolik (2001), Hawley et al. (2001), and Hawley and Krolik (2001). Others have started with an initial configuration of matter that is not in equilibrium. For example, matter may be placed in orbit about the black hole, but with sub-Keplerian angular momentum, so that once the simulation commences it immediately falls toward the hole. Examples of this approach include Hawley et al. (1984), Koide et al. (1999), Koide et al. (2000), and Meier et al. (2001). This approach may enhance transients associated with the choice of initial conditions, although it can also be physically well motivated, as in studies of core-collapse supernovae. One can also inject fluid continuously onto the computational grid over the course of the evolution. Examples of this include IA99 and IA00. One might also use an inflow boundary condition at the outer radial edge of the grid, as in Blondin et al. (2001). The main advantage of injection models is that they allow one to achieve a steady, or statistically steady, state. In this paper we will consider only the equilibrium tori and on-grid injection models. The equilibrium tori (Papaloizou and Pringle, 1984; Fishbone and Moncrief, 1976; Jaroszynski et al., 1980) are steady-state solutions to the equations of inviscid hydrodynamics. They assume a γ polytropic distribution of mass and internal energy, P = Kρ0 , and a power-law rotation profile, Ω ∝ (r sin θ)−q. The radial and meridional components of the velocity vanish. There are 5 parameters that describe the torus: (1) the location of the torus pressure maximum r0 ; (2) the location of the innermost edge of the torus, rt,in < r0; (3) the maximum value of the density, ρ? = ρ0(r0);

64 (4) the angular velocity gradient q = d ln Ω/d ln R; and (5) the entropy constant K. On-grid injection adds matter to the model at a constant rate. The matter is injected with a non-zero speciﬁc angular momentum and with zero radial or meridional momentum in a steady patternρ ˙0(r, θ) which is typically symmetric about the equator. Parameters for this scheme include:

(1) a characteristic radius for injection rinj; (2) the rate of mass injection M˙ inj; (3) the speciﬁc angular momentum of the injected ﬂuid, vφ = f1rΩK . We usually set f1 = 0.95, so that the

ﬂuid circularizes near rinj. This restricts transients associated with circularization to the outer portions of the computational domain; (4) the internal energy of the injected ﬂuid, u = f2ρ0Ψ.

We always set f2 = 0.2 so that the fluid is marginally bound, i.e. has Bernoulli parameter Be ≡ 2 2 (1/2)v + cs/(γ − 1) + Ψ < 0. One must also choose the injection patternρ ˙0(r, θ). IA99 choose a radially narrow region, but do not explicitly giveρ ˙0(r, θ). We use a Gaussian, but found that none of the results are sensitive to the precise profile. The accretion rate of mass, energy, and angular momentum are insensitive to large changes in the size of the injection region except for the extreme cases of filling the entire θ width or injecting in 2 locations. These extreme cases are sufficiently different to be referred to as a completely different model; they lead to a qualitative change in the flow. For example, a full range θ injection region has matter that will collide with any outflow at the poles. A bipolar injection leads to an equatorial outflow. Our models have radial width σr = 0.05(rin + rout)/2 and

σθ = π/8.

2.4 Numerical Methods

Our numerical method is based on ZEUS-2D (Stone and Norman, 1992) with the addition of an explicit scheme for the viscosity. ZEUS is an operator-split, finite-difference algorithm on a staggered mesh that uses an artificial viscosity to capture shocks (in addition to the anomalous viscosity in equations [2.2]). This algorithm guarantees that momentum and mass are conserved to machine precision. Total energy is conserved only to truncation error, so total energy conservation is useful in assessing the accuracy of the evolution. The inner and outer radial boundary conditions are implemented by copying primitive variable values (ρ0, u, and v) from the last zone on the grid into a set of “ghost zones” immediately outside the grid. Inflow from outside the grid is forbidden; we set vr(rin) = 0 if vr(rin) > 0 and vr(rout) = 0 if vr(rout) < 0. Since we expect inflow on the inner boundary, this switch should seldom be activated. We have found that frequent activation of the switch is usually an indication of a numerical problem.

We use a radial grid uniform in log(r − rg). We require that dr(r)/(rin − rg) ≤ 1/4 so that the structure of the pseudo-Newtonian potential is well resolved. The grid is uniform in θ. The grid has Nr × Nθ zones.

65 2.4.1 Numerical Treatment of Low Density Regions

Like many schemes for numerical hydrodynamics, ZEUS can tolerate only a limited dynamic range in density. It is therefore necessary to impose a density minimum ρ0fl to avoid small or negative densities. Our procedure for imposing the floor is equivalent to adding a small amount of mass to the grid every time the floor is invoked. Mass is added in such a way that momentum is conserved. To monitor the effect of the density floor, we track the rate of change of total mass and total energy

(from kinetic energy change) due to this procedure, M˙ fl and E˙ fl. −10 ˙ 3 2 −5 We set ρ0fl = 10 Minjc /(GM) for injection runs and ρ0fl = 10 ρ? for torus runs. Lower values for ρ0fl do not lead to a significant change in the solution. Larger values of ρ0fl give M˙ fl ∼ M˙ , the accretion rate through the inner boundary. The atmosphere also becomes more massive and begins to affect torus stability– vertical oscillations are excited in the inner disk by a Kelvin-Helmholtz like instability. This should be avoided. We must also surround the torus in a low density atmosphere in the initial conditions. The density of the atmosphere is ρ0fl and the internal energy density is u = Uoρ0Ψ, where Uo is a constant fraction of order unity (e.g. IA and we choose Uo = 0.2). The addition of the atmosphere has no effect on the solution since the mass source’s evolution eventually dominates the flow everywhere. SPB99 choose a different method of constructing the initial atmosphere but obtain late-time results that are similar to ours.

It is also necessary to impose a ﬂoor ufl on the internal energy density. This we take to be the minimum value of u in the initial atmosphere. As for the mass, we track the rate of change of total energy due to the internal energy ﬂoor, that along with the kinetic energy is included in E˙ fl.

2.4.2 Diagnostics

Global numerical simulations of accretion flows are complicated; it is possible to measure many quantities associated with the flow. Some are astrophysically relevant, and some are not. In our view particular interest attaches to the time-averaged flux of mass, energy, and angular momentum through the inner boundary. Physically, these are directly related to the luminosity of the accretion flow and the rate of change of mass and angular momentum of the central black hole. As described by Narayan and Popham (1993), these are in a sense the nonlinear “eigenvalues” of the model. The rest-mass accretion rate is Z M˙ = ρ0v · dS, (2.17) S where S is the inner radial surface of the computational domain. The total energy accretion rate is Z 1 E˙ = (( v2 + h + Ψ)ρ v + Π · v) · dS, (2.18) 2 0 S where h = (u + P )/ρ0 = γu/ρ0 is the specific enthalpy with our equation of state. The angular

66 momentum accretion rate is Z ˆ L˙ = r sin θ(ρ0vvφ + Π · φ) · dS. (2.19) S

It is also sometimes useful to focus on the reduced eigenvalues l = L/˙ M˙ and e = E/˙ M˙ . These value of mass, energy, and angular momentum are recorded at about 2 grid zones away from rin. This avoids any error that may occur when evaluating directly on the boundary where the inflow boundary condition is applied. We also track volume-integrated quantities, the flux of mass, energy, and angular momentum across all boundaries, and floor added quantities in order to evaluate the consistency of the results. Mass and angular momentum are conserved to machine precision, although “machine precision” implies a surprisingly large random walk in the integrated quantities over the full integration because the calculation requires millions of timesteps. Total energy is conserved to truncation error, not machine precision, and thus is a useful check on the quality of the simulation. Total energy conservation implies

E˙ err = E˙ vol + E˙ + E˙ out − E˙ fl, (2.20) where E˙ vol is the rate of change of the volume integrated total energy, E˙ out is the flux of total energy through the outer radial boundary, and E˙ fl is the rate of total energy added due to the kinetic energy change (because of the mass density floor) and internal energy density floor. Ideally,

E˙ err = 0. Truncation errors can (and do) lead to cumulative, rather than random, changes in the total energy. A useful gauge of the magnitude of these errors is E˙ err/E. For all runs we performed the error rate is within 10% of 10−5c3/GM for a torus run and within 10% of 10−4c3/GM for a viscous injection run.

2.4.3 Code Tests

Our version of ZEUS reproduces all hydrodynamic test results from Stone and Norman (1992), including their spherical advection and Sod shock tests. We also find excellent agreement with steady spherical accretion solutions, i.e. Bondi flow (Bondi, 1952). An inviscid equilibrium torus run also persists for many dynamical times with insignificant deviations from the initial conditions. We have parallelized our code using the MPI message passing library. On the Origin 2000 at NCSA we are able to achieve about 2.5×107 zone updates per second using 240 CPUs, or about 35 GFLOPs. This is 159 times faster than the single CPU speed, which represents a parallel efficiency of 66%.

67 2.5 Results

The initial motivation for undertaking this calculation was to understand differences between results reported in IA and SPB. Using our code, which is based on the same algorithm used in SPB’s calculations, we ran a series of tests attempting to reproduce SPB’s results. These test calculations used all of SPB’s model choices, including SPB’s viscosity prescription, a Newtonian potential, and a torus for the mass source. We were able to reproduce most quantitative results reported in SPB99’s torus calculations. This includes their radial scaling laws. For example, in a model that ˙ 0 2 −1 −1/2 −1 is identical to SPB99’s Run B, we find M ∝ r, ρ0 ∝ r , cs ∝ r , vφ ∝ r , and |vr| ∝ r . These power law slopes are identical to those reported by SPB99. As another example, we found M˙ = 1.23 × 10−3 torus masses per torus orbit at the pressure maximum for a model identical to SPB’s Model A (their fiducial model); SPB report M˙ = 1.0 × 10−3 in the same units. Given the fluctuations in rest-mass accretion rate, our value and SPB’s value are fully consistent. We were even able to reproduce certain numerical artifacts associated with the inner radial boundary, such as a density drop and temperature spike near the inner boundary. Recall that SPB evolve an initial torus and allow it to accrete; IA use a different experimental design in which matter is steadily injected onto the grid. They also use a different viscosity prescription. We ran a second series of test calculations attempting to reproduce IA’s results. These test calculations used all of IA’s model choices, including viscosity prescription, Newtonian potential, etc. We were able to reproduce all of IA99’s calculations except those that include thermal conduction (which we did not attempt to reproduce). In each case we found that the qualitative nature of the flow is similar to that described in IA99. In particular, we agree on which models are stable and unstable and which models exhibit outflows. We also find qualitative agreement with their contour plots of, e.g., density pressure, mass flux, and Mach number. We also find qualitative agreement with their radial run of cs/VK and specific angular momentum. Our results do not agree precisely, but this is likely due to small differences in mass injection scheme (because IA99 do not give theirρ ˙0(r, θ)). Finally, we can also reproduce the radial scalings given in IA00 for their model A. The most significant difference between the results of IA and SPB was due to the choice of anomalous stress prescription, as might have been anticipated. Qualitatively, the stress components that are included in IA and not SPB tend to smooth the flow and suppress turbulence. Thus SPB99’s simulations result in more vigorous convection than simulations performed by IA. The choice of mass supply (torus vs. injection) also leads to a significant difference between IA and SPB’s results; this is discussed in more detail below. The fact that we can reproduce both SPB’s and IA’s results using a single code is consistent with the hypothesis that differences between their reported results (e.g. the lower degree of convection reported in IA than SPB, and the differences in radial scaling laws) is due to differences in experimental design and viscosity prescription rather than numerical methods. While we cannot completely rule out the possibility that SPB, IA, and we have made identical experimental errors,

68 Table 2.1. Parameter List

3 Run Nr Nθ Visc. Potential rin/rg rout/rg Rinj /rg γ α tf (c /GM) A 108 50 IA PN 1.35 300 248 3/2 0.1 7.3 × 105 B 80 50 IA PN 3 300 248 3/2 0.1 4.4 × 105 C 80 50 IA Newt. 3 300 248 3/2 0.1 7.3 × 105 D 64 40 MG PN 1.2 76 62 3/2 1.0 3.3 × 103 E 128 80 MG PN 1.4 21 17 5/3 0.01 3.0 × 104 F 128 80 MG PN 1.4 81 21 5/3 0.01 6.8 × 104

Note. — IA and MG are viscosity prescription described in equations 7-9. PN is the pseudo-Newtonian 2 potential of Paczy´nski& Wiita (1980). Run B uses Run C as initial conditions. Here rg = 2GM/c , and in run F Rinj is the position of the torus density peak ρ0.

Table 2.2. Results List

3 2 −2 Run Steady State Time (GM/c ) Max. Mach at rin M˙ 0 −(E/˙ (M˙ 0c )) × 10 L˙ c/(GMM˙ 0) A 4.3 × 105 -1.4 5.96 × 10−2 2.06 1.75 B 2.4 × 105 +0.0 1.95 × 10−2 3.72 1.29 C 2.4 × 105 +0.0 9.46 × 10−3 6.77 .0746 D ≥ 3.3 × 103 -0.4 ≥ 3.59 × 10−2 4.64 −0.167 E 5.5 × 103 -3.0 3.48 × 10−2 3.01 3.41 F 2.0 × 104 -3.4 5.03 × 10−1 3.11 3.35

Note. — Runs A-E are injection runs with rest-mass accretion rate units in M˙ 0,inj and Run F is a torus run with rest-mass 2 3 accretion rate unit in ρ0(GM) /c . Run C’s angular momentum ﬂuctuations are 10 times the average value shown.

this seems unlikely. This comparison thus lends credibility to SPB, IA, and our numerical results.

2.5.1 Fiducial Model Evolution

We now turn from reproducing earlier viscosity models to considering new aspects of our own models. First, consider the evolution of a “ﬁducial” model (Run A in Table 2.1 and Table 2.2). 2 2 2 The ﬁducial model has rin = 2.7GM/c , rout = 600GM/c , rinj = 495GM/c , γ = 3/2, α = 0.1,

Nr = 108, Nθ = 50. It uses a pseudo-Newtonian potential, mass is supplied by injection, and the viscosity prescription follows IA. It was run from t = 0 to t = 7.3 × 105GM/c3. Run A is similar to IA99’s “Model 5”, except that it uses a pseudo-Newtonian potential. In a statistically steady state the flow is characterized by a quasi-periodic outflow. Hot bubbles form 2 2 at the interface between bound (Bernoulli parameter Be = (1/2)v + cs/(γ − 1) + Ψ < 0) and unbound (Be > 0) material. These hot bubbles are buoyant and move away from the black hole. This appears to be a low-frequency, low wavenumber convective mode (IA refer to it as a “unipolar outflow”). Higher wavenumber convective modes are evidently suppressed by the viscosity. Figure 2.1 shows time-averaged plots of various quantities in the fiducial run. The time average

69 2 Figure 2.1 Time-averaged spatial structure of fiducial run (Run A; α = 0.1, rin = 2.7GM/c , and 2 2 2 rout = 600GM/c ). Shown are the density (upper left), Bernoulli parameter (Be = (1/2)v +cs/(γ− 2 1) + Ψ) (upper right; dotted line is a negative contour), scaled mass flux r sin θ(ρ0v) (lower left), 3 2 ˆ and scaled angular momentum flux r sin θ(ρ0vvφ +Π·φ) (lower right). The flow is not symmetric about the equator because the flow exhibits long timescale antisymmetric variations. Convective bubbles form at the interface between positive and negative Bernoulli parameter (i.e. unbound and bound matter).

70 is performed from 200 equally spaced data dumps from t = 4.3×105GM/c3 to t = 7.3×105GM/c3. 2 We show only the region rin < r < 30GM/c , whereas the computational domain is much larger: 2 rin < r < 600GM/c . The injection point is located far outside the plotted domain at r = 496GM/c2. Because of the strong time-dependence of the flow in the fiducial run, the flow at any instant may look very different from these time averaged plots. The upper left panel in Figure 1 shows the average density; notice that the density is not symmetric about the equator. This is because the flow involves long-timescale quasi-periodic variations which are not quite averaged out over the course of the run. The upper right corner shows the Bernoulli parameter Be. Dotted lines are negative; notice that there is a substantial amount of fluid near the equator that is unbound in the sense that Be > 0. Nevertheless, this material is still flowing inward in a nearly laminar fashion. Near the poles, the time-averaged Be < 0, but this region experiences large fluctuations. Polar outflows are typically associated with positive fluctua- 2 tions in Be. The lower left panel shows the scaled mass flux r sin θ(ρ0v). Notice that much of the mass flux is along the surfaces of the inflow rather than along the equator. The lower right panel 3 2 ˆ shows the scaled angular momentum flux r sin θ(ρ0vvφ + Π · φ). As for the mass flux, most of the activity is along the surface of the flow. 2 Figure 2.2 shows the time series of the reduced eigenvalues: M/˙ M˙ inj, e = E/˙ (Mc˙ ), and l = Lc/˙ (GMM˙ ). Also shown as dashed lines are the thin disk values for e and l. These assume a thin, cold disk terminating at r = 6GM/c2. The low value of l is due to two effects. First, the disk is already sub-Keplerian by the time the flow reaches the innermost stable circular orbit. In addition, there are residual viscous torques in the plunging region that lower the specific angular momentum of the accreted material (see Figure 2.3, below). Notice that Figure 2.2 shows a smooth evolution that varies on a timescale τ ≈ 4 × 104 at late time. The largest variations in rest-mass accretion rate are related to the appearance of large convective bubbles. Figure 2.3 shows the θ and time averaged run of several quantities with radius. The averages are taken over 4.3 × 105GM/c3 < t < 7.3 × 105GM/c3 and |θ − π/2| < π/6 2. The upper left plot shows the run of density. Notice that here, as for the other quantities, there is a spike near rinj, an intermediate region, and then an inner, roughly power-law region. The upper right plot shows 2 (cs/c) ; the lower left shows |vr|/c. Notice that the radial velocity exceeds the speed of light at the inner boundary. Similarly the azimuthal velocity vφ/c shown in the lower right panel approaches the speed of light. Also shown in that panel is the circular velocity (dashed line). Evidently the flow is slightly sub-Keplerian at most radii. The radial run of flow quantities in the inner regions can be fit by power laws, as done by IA and SPB. Our best fit power laws for the fiducial model (Run A) over 2.7GM/c2 < r < 20GM/c2 −0.6 −0.5 −2 −0.8 2 2 are ρ0 ∝ r , cs ∝ r , |vr| ∝ r , and vφ ∝ r . Between 2.7GM/c < r < 6GM/c , vφ is −0.9 best fit by vφ ∝ r , which is nearly, but not exactly, consistent with conservation of fluid specific −1 2 angular momentum (vφ ∝ r ). Angular momentum is not exactly conserved at r < 6GM/c because of viscous torques.

2Averaging over |θ − π| < π/36 produces nearly identical results, but we have chosen to use IA’s range in θ.

71 2 Figure 2.2 The evolution of M/˙ M˙ inj, e = E/˙ (Mc˙ ), and l = L˙ c/(GMM˙ ) in the ﬁducial run (Run A). The dotted line indicates the thin disk value. The run has clearly entered a quasi-steady state. The evolution is relatively smooth with a small variation on a timescale τ ≈ 4 × 104. This is the timescale for convective bubble formation (the low point in rest-mass accretion rate is when bubble forms). For this model the bubble forms at alternate poles. A full cycle requires of order one rotation period at the injection radius.

72 Figure 2.3 The radial run of θ and time averaged quantities from the fiducial run (Run A). Shown are the density (upper left), squared sound speed (upper right), radial velocity (lower left), specific angular momentum (lower right; solid line), and circular orbit specific angular momentum (lower right; dashed line). Crudely speaking, the inner flow is consistent with a radial power law. The −0.6 −0.5 −2 −0.8 best fits to a power law are: ρ0 ∝ r , cs ∝ r , |vr| ∝ r , and vφ ∝ r . The plots are averaged over θ = π/2 ± π/6.

73 The careful reader may notice that the power law slopes quoted in the last paragraph are not consistent with a constant rest-mass accretion rate. This is because the power laws are derived from averages over |θ − π/2| < π/6, following SPB. If one averages over all θ (following IA) and time, then the resulting proﬁles are consistent with constant mass, energy, and angular momentum accretion rates, as they must be for a ﬂow that is steady when averaged over large times.

2.5.2 Dependence on Inner Boundary Location and Gravitational Potential

Having established that the differences between SPB and IA’s models are due to model choices rather than numerics, we were also interested in studying whether any features of global viscous accretion models are strongly dependent on numerical parameters. The first parameter we considered was the location of the inner boundary. In models that use a Newtonian gravitational potential (such as IA and SPB) the location of the inner boundary is not an interesting parameter in the sense that there is no physical lengthscale that one can compare rin to: it is simply a scaling parameter. In models that use a pseudo-Newtonian potential, however, there is a feature (a “pit”) in the potential on a lengthscale GM/c2. Starting with our fiducial model, then, what is the effect of shifting rin? 2 Our fiducial run has rin = 2.7GM/c . This may be compared with Run B, which has rin = 6GM/c2. Figure 2.4 compares the accretion rates in the two runs. Evidently there are two changes in the solution. First, the time-averaged accretion rates differ by a large factor. The mean rest- mass accretion rate is factor of 3 lower in Run B than Run A. The reduced eigenvalues e and l also differ by about 50% (see Table 2). Second, the time variation of the accretion rates differs, with Run B showing far more short-timescale variations. The short-timescale variations are due to the interaction of unstable convective modes with the boundary conditions. Inspection of the runs reveals an enhancement of convection and turbulence near rin in Run B. The differences between Run B and Run A are caused by the boundary location. Gradual variation of rin (in models not discussed in detail here) reveals that if the flow on the inner boundary is everywhere and always supersonic, then the solution is similar to Run A. If the flow is subsonic, then the solution exhibits artifacts like those seen in Run B. Evidently forcing the flow to be supersonic on the inner boundary causally disconnects the flow from the boundary. 3 This eliminates nonphysical reflection of linear and nonlinear waves from the boundary and renders the precise implementation of the numerical boundary conditions irrelevant. We do not want the reader to think that this problem arises because we happened to choose the wrong numerical implementation of the boundary conditions. Our implementation is the standard ZEUS outflow boundary condition, and it is widely used in astrophysical problems. While it may be possible to implement more transparent boundary conditions in the context of other numerical schemes, a survey of the numerical literature shows that in multiple dimensions this is an area of

3Although the viscous fluid equations of motion are not hyperbolic, and the flow in the supersonic region is in principle in causal contact with the rest of the flow, the coupling is exponentially weak.

74 Figure 2.4 The effect of moving the inner boundary on the accretion rates of mass, angular momen- 2 tum, and energy (Run A vs. Run B). The top panel shows M/˙ M˙ inj, the middle panel E/˙ (Mc˙ ), 2 and the bottom panel L˙ c/(GMM˙ ). The solid curve is Run A, which has rin = 2.7GM/c . The 2 dashed curve is Run B, which has rin = 6GM/c . Evidently Run B has a different variability structure and different time averaged values for the accretion rates. The relatively rapid and high- amplitude variations in Run B are due to nonphysical interactions with the inner radial boundary. Only by ensuring a supersonic flow (as in Run A) can one avoid these nonphysical effects.

75 active research (Roe, 1989; Karni, 1991; Dedner et al., 2001; Bruneau and Creus, 2001), and that no general solution to the problem has been found. Furthermore, a simple example shows that no local extrapolation scheme can work for all accretion problems. Consider a numerical model of a steady spherical inflow (Bondi flow) in a gravitational potential Ψ(r). Let us suppose that we are primarily interested in accurately measuring M˙ . We know from the analytic solution of the problem that M˙ depends on the shape of the potential everywhere outside the sonic point. If we place the inner boundary rin outside the sonic point and use a local extrapolation scheme, we won’t always get the correct answer because the local extrapolation doesn’t have any information about the shape of the potential between rin and the sonic point. Put differently, one can’t determine a global solution from local extrapolation at the boundary. The key point is that, while aspects of the solution may be accurate, M˙ (and L˙ and E˙ ) are sensitive to the boundary conditions. It is worth noting that the outer boundary is always in causal contact with the flow, but does not cause the same type of artifacts as the inner boundary. Experiments show that the flow is qualitatively insensitive to the location and implementation of the outer boundary condition. The time averaged M˙ , however, is sensitive to both rout and rinj/rout. The time averaged l and e scale out this mass dependence and so are qualitatively and quantitatively insensitive to both rout and rinj/rout. It is also worth noting the effects of changing the gravitational potential. Run C (identical to IA99 Model 5) is identical to Run B except that the potential is now Newtonian. It is qualitatively similar to Run B, but M˙ is now a factor of 5 lower than Run A. Run C also has the property that l oscillates about 0.0. This is a problem if the focus of the simulation is measuring M˙ or L˙ . To summarize: the location of the inner radial boundary can determine the character of the flow. If the flow is everywhere and always supersonic (or super-fast-magnetosonic in MHD) on the inner boundary then boundary-related corruption of the flow is impossible. Since it is computationally expensive to place the inner boundary very deep in the potential (for our model, the time step 2 dt ∼ (rin − 2GM/c )), the optimal location for the inner boundary is just inside the radius where the radial Mach number always exceeds 1. The results of IA and SPB do not focus on the time-dependence of the accretion values, so much of their discussion is unaffected by their treatment of the inner radial boundary. As discussed below, there are small changes related to the appearance of outflows.

2.5.3 Comparison of Torus and Injection Models

The torus and injection methods represent sharply different approaches to studying accretion flows. The equilibrium torus presents a physically well-posed problem, but the accretion flow is transient: no steady state can be achieved. The injection method reaches a quasi-steady state, but much of the computational domain is wasted on evolving the injection region, which has no astrophysical analog: it is nonphysical. It is natural to ask whether these two widely used schemes for supplying

76 mass can be made comparable or used to measure any of the same quantities. We selected two runs, E (torus) and F (injection), that had similar mass distributions in an evolved state. The torus run was studied at a time when M˙ was close to its maximum. Run F’s M˙ is a factor of 10 larger than Run E’s. This difference might have been anticipated from the sensitivity of the injection run to rout and rinj/rout: runs in which mass is concentrated closer to the outer boundary tend to have lower accretion rates because more of the mass escapes through the outer boundary. The time averaged rest-mass accretion rate is therefore strongly dependent on the method of mass supply. The energy and angular momentum accretion rates per unit mass are, however, insensitive to the experimental design. We find that e and l differ by less than 3% in Runs E and F (see Table 2.1 and Table 2.2). These quantities are apparently set by conditions near the inner boundary (the ISCO for the Pseudo-Newtonian potential), and can be measured in either type of experiment.

2.5.4 Other Parameters

We have varied rinj and rout/rinj, and as reported above, these strongly affect the time-averaged value of M˙ . The sense of the effect is that a simulation with a larger rout/rinj loses less matter through the outer boundary, and this results in more matter streaming back into the black hole (by up to a factor of 10). The qualitative nature of the flow, however, is roughly independent of rout/rinj in that, e.g., the temporal power spectrum of M˙ is similar. The qualitative nature of the

flow is dependent on rinj. If one fixes rout/rinj and all other parameters, the range in α where unipolar outflows are observed tends to become smaller and disappears altogether for rinj as small as 40GM/c2. The dependence of accretion models similar to ours on α has already been investigated by IA. They find that the flow changes from turbulent to laminar as α is increased and the higher viscosity damps modes of increasing lengthscale. IA find that α . 0.03 the flow is turbulent, and for α & 0.3 the flow is laminar. For 0.03 < α < 0.3 the flow exhibits a “unipolar” outflow. Our results are in agreement with IA. However, our models with a pseudo-Newtonian potential and super-sonic flow at the inner radial boundary show a slight shift in the values of α that exhibit unipolar outflows.

We did find a critical value of α ≈ 0.5 above which no supersonic flow at rin could be achieved 2 due to viscous heating, at least for γ = 3/2 and γ = 5/3 and for rin ≥ 2.1GM/c . Smaller values of rin were not computationally practical. This high α is typically associated with a bipolar outflow, 2 2 as seen by IA. Even in this case, however, a choice of rin = 2.1GM/c instead of rin = 6GM/c 2 leads to a qualitatively different profile for the flow. The flow with smaller rin = 2.1GM/c has a bipolar outflow starting at larger radius (10GM/c2) rather than immediately on the boundary as 2 with rin = 6GM/c , and the rest-mass accretion rate increases by a factor of 3.

Finally, we studied the dependence of the results on numerical resolution. We ﬁnd that Nr ×

Nθ = 108 × 50 is suﬃcient at α = 0.1 to resolve the shortest wavelength convective mode. Also, we chose our value of r0 to agree with SPB99’s torus models. We experimented with varying r0 and

77 ﬁnd that, all things being equal, smaller r0 gives more laminar ﬂow.

2.6 VHD Summary

Work in this field will shortly focus on global MHD models in pseudo-Newtonian potentials and in full general relativity. In our view it is useful to understand the solution space for physically and numerically simpler viscous models before turning to MHD. It is even possible, as Stone and Pringle (2001) have claimed, that viscous hydrodynamics provides a crude approximation to the MHD results. In any event, this investigation provides a preview of some of the experimental issues that will play a role in most future global numerical investigations of accretion flows. This investigation was initially motivated by a desire to understand whether the differences between earlier global viscous hydrodynamics simulations performed by IA and SPB were caused by differences in experimental design or numerical method. IA and SPB reported different degrees of convective turbulence in their models and found different radial scalings for vertically averaged quantities such as temperature and density. Using a single code, we were able to reproduce both sets of results. We conclude that the differences are due to experimental design. We also found, while reproducing IA and SPB’s results, that some aspects of our solutions were sensitive to the numerical treatment of the region close to the inner boundary in models that use a pseudo-Newtonian potential. In particular, l and e, the specific angular momentum and energy of accreted material, are strongly dependent on rin, the location of the inner boundary. When the

flow is supersonic at rin the location of the boundary does not affect l and e. But when the flow is subsonic at rin the flow interacts strongly with the numerical boundary condition. This produces spurious outflow events and makes l and e dependent on rin. Evidently for accurate measurement of these quantities it is necessary to isolate the numerical boundary condition behind a sonic transition that is located within the computational domain; one must place the inner boundary condition inside a “sound horizon”. We are not saying that all models that lack a sonic transition in the computational domain are fatally flawed. Whether the treatment of the inner boundary condition is problematic or not depends on what is being measured. For the nonlinear eigenvalues L,˙ E˙ , and M˙ that we have focused on here, however, the treatment of the inner boundary condition is crucial. Furthermore, the only guarantee that the inner boundary condition is not governing the solution is to isolate it behind a sonic transition; this is the only completely safe choice. The location of the inner boundary may prove even more crucial in MHD models. Much of the character of the flow is determined in the turbulent, energetically important region of the flow just outside the fast magnetosonic transition, just as the region immediately outside the sonic transition determines the nature of the viscous flows described in this paper. We have performed some preliminary numerical tests and find that, as in the viscous flow, l and e for an MHD flow are sensitive to the treatment of the inner boundary. For various reasons it may prove difficult to achieve a fast magnetosonic transition in the computational domain; accurate treatment of the

78 inner boundary condition may require fully (general) relativistic MHD. We have also compared two commonly used experimental designs for black hole accretion flow studies: models that begin with an equilibrium torus, and models that continuously inject fluid onto the grid. The choice between these models is to some degree a matter of taste. We find the equilibrium torus slightly easier to initialize and analyze. Remarkably, the two different approaches produce indistinguishable measurements for l and e, the specific angular momentum and energy of the accreted material. A parallel, viscous, axisymmetric hydrodynamics code based on that used in this paper can be found at http://kerr.physics.uiuc.edu.

2.7 Global 2D MHD Simulations

Prior analytic calculations find that the magnetic stress −BrBφ → 0 near the innermost stable circular orbit (ISCO) of a black hole. This has bearing on whether magnetic stresses can exert a torque between the hole and disk. Thus, at least two interesting questions regarding a magnetic disk are: 1) does the magnetic stress approach 0 near the ISCO? and 2) is the polar field strong enough for the BZ effect to be relevant? With a global 2D MHD code we have been able to reproduce prior work by Hawley (2000) (hereafter H00) and Stone and Pringle (2001) (hereafter SP01). This work involves an equilibrium torus (in HD) threaded by a field where the vector potential component Aφ ∝ ρ, so that the field resides totally within the torus and is axisymmetric. H00, SP01, and we all find there is nothing special about the ISCO, contrary to most prior analytic calculations (except in, for example, Gammie (1999)). The magnetic stress continues to grows as r → 2GM/c2 directly through the

ISCO, and is relatively large compared to the Reynolds stress −vrvφ there as well. To discover whether the polar field is substantial enough to make the BZ effect a significant source of energy, we need to simulate both the pole and the disk. This is not possible for H00 since his computational grid excises the polar region, and SP01 do not report on this. We find that the polar field surrounding the black hole is relatively strong compared to that in the disk. This field is supported by the disk and hole interaction as the disk continually pumps and compresses field into the black hole. Figure 3 shows the field lines of a 2D axisymmetric, spherical polar coordinate simulation after about 8 orbits (at the t = 0 torus density maximum) once full non-linear MRI turbulence has developed. We measure the polar magnetic field strength to be 2-4× that of the average disk field at the inner radial edge, contrary to estimates by Ghosh and Abramowicz (1997); Livio et al. (1999). With such a strong polar field, it is plausible that the BZ effect generates a significant fraction of AGN power output. One can compute the power output of the BZ effect using equation 1.8. I estimate the field strength B by assuming my computer simulation’s mass accretion rate M˙ is equal to the Eddington ¡ ¢ value (M˙ ∼ 1.3×1038 ηc2 −1 ( M ) erg s−1), where η is order 10%. In the simulation the accretion M¯

79 Figure 2.5 Field line snapshot after 8 orbits (at the t = 0 torus density maximum at r = 9.4GM/c2) for a global 2D MHD simulation. Full non-linear turbulence drives the accretion process. Note that the polar ﬁeld is essentially radial while the accretion disk is dominated by turbulence generated by the MRI.

80 3 p ˙ ρ0(rh/2) 2 rate scales like M ∝ (GM/c3) and the field strength scales like B ∝ ρ0c , where ρ0 is the typical density of the initial mass source. Thus, M˙ can be used to solve for ρ0 in cgs units which is then used to solve for B in cgs units. This gives P ∼ 4 × 1045 erg s−1. By limiting the calculation to 2D, the field is forced to avoid twisting around the pole or diffusing around the pole, so we may be overestimating the polar field. Also, as stated by the anti-dynamo theorem (Cowling, 1934), an isolated 2D axisymmetric system cannot sustain a magnetic dynamo. Thus, the next step in modeling the black hole environment is to perform a global 3D calculation.

2.8 Global 3D MHD Simulation

A global 3D calculation involves about twice the number of computational operations per grid zone, and requires N 3 zones instead of N 2, so a 3D calculation can be quite expensive. One would prefer to use spherical polar coordinates since the black hole is spherical, and because in these coordinates the ZEUS algorithm conserves angular momentum well. We performed global 3D MHD calculations in spherical polar coordinates and found that the singularity at the poles leads to several problems. The most significant problem is that grid zones near the polar axis become smaller since zones have a side with arc length of r sin θdφ. In order to guarantee a stable algorithm, the Courant condition requires dt ∝ dx/v for a grid size dx and fluid speed v. Since the grid size is dx ∝ θ near the θ = 0 pole, and the polar region is nearly a vacuum with potentially strong fields (hence large va), this leads to a requirement that dt ∝ θ/va and severely limits the time step. We therefore decided to use a Cartesian grid. This presents two problems: 1) angular momentum is no longer well conserved by the algorithm ; and 2) The grid geometry forces an m = 4 mode (spherical mode for the φ direction) in the solution. We try to diminish both of these problems by cutting an approximation to a sphere out of the Cartesian mesh at the outer boundary and around the black hole. Preliminary results show that the time step is well behaved and the m = 4 mode is minimal. Angular momentum conservation appears acceptable, but quantitative analysis is pending. Questions we will attempt to answer with this code are: 1) does an accretion flow generate a magnetized outflow? ; 2) how does Bremsstrahlung cooling affect a magnetized accretion disk? and 3) what is the power output due to the BZ effect?

81 HARM: A Numerical Scheme for 3 General Relativistic Magnetohydrodynamics

3.1 Summary of Chapter

We describe a conservative, shock-capturing scheme for evolving the equations of general relativistic magnetohydrodynamics. The fluxes are calculated using the Harten, Lax, and van Leer scheme. A variant of constrained transport, proposed earlier by Tóth,is used to maintain a divergence free magnetic field. Only the covariant form of the metric in a coordinate basis is required to specify the geometry. We describe code performance on a full suite of test problems in both special and general relativity. On smooth flows we show that it converges at second order. We conclude by showing some results from the evolution of a magnetized torus near a rotating black hole.1

3.2 Introduction

Quasars, active galactic nuclei (AGN), X-ray binaries, gamma-ray bursts, and core-collapse supernovae are all likely powered by a central engine subject to strong gravity, strong electromagnetic fields, and rotation. For convenience, we will refer to this class of objects as relativistic magnetorotators (RMRs). RMRs are among the most luminous objects in the universe and are therefore the center of considerable theoretical attention. Unfortunately the governing physical laws for RMRs, while well known, are nonlinear, time-dependent, and intrinsically multidimensional. This has stymied development of a first-principles theory for their evolution and observational appearance and strongly motivates a numerical approach. To fully understand RMR structure, one must be able to follow the interaction of a non- Maxwellian plasma with a relativistic gravitational field, a strong electromagnetic field, and possibly a strong radiation field as well. This general problem remains beyond the reach of today’s algorithms and computers. A useful first step, however, might be to study these objects in a nonradiating magnetohydrodynamic (MHD) model. In this case the plasma can be treated as a fluid, greatly reducing the number of degrees of freedom, and the radiation field can be ignored. The relevance of this approximation must be evaluated in astrophysical context and will not be considered here.

1Published in ApJ Volume 589, Issue 1, pp. 444-457. Reproduction for this thesis is authorized by the copyright holder. Initial HARM work as in paper primarily performed by CFG, later eﬀorts primarily by JCM, and GT provided numerical advice.

82 We were motivated, therefore, to develop a method for integrating the equations of ideal, general relativistic MHD (GRMHD), and that method is described in this paper. This is in a sense well- trodden ground: many schemes have already been developed for relativistic fluid dynamics. What has not existed until recently is a scheme that: (1) includes magnetic fields; (2) has been fully verified and convergence tested; (3) is stable and capable of integrating a flow over many dynamical times. A pioneering GRMHD code has been developed by Koide and collaborators (see, e.g. Koide et al. 2002, Koide et al. 1999, Koide et al. 2000, Meier et al. 2001). Our code differs in that we have subjected it to a fuller series of tests (described in this paper), we can perform longer integrations than the rather brief simulations described in the published work of Koide’s group, and our code explicitly maintains the divergence-free constraint on the magnetic field. A GRMHD Godunov scheme based on a Roe-type approximate Riemann solver has been developed by Komissarov and described in a conference proceeding (Komissarov, 2001). A rather complete review of numerical approaches to relativistic fluid dynamics is given by Mart´ıand Müller(2003) and Font (2003). The first numerical GRMHD scheme that we are aware of is by Wilson (1977), who integrated the GRMHD equations in axisymmetry near a Kerr black hole. While it was recognized throughout the 1970s that relativistic MHD would be relevant to problems related to black hole accretion (particularly following the seminal work of Blandford and Znajek (1977) and Phinney (1983); see also Punsly (2001)), no further work appeared until Yokosawa (1993). The discovery by Balbus and Hawley (1991) that magnetic fields play a crucial role in regulating accretion disk evolution (reviewed in Balbus and Hawley (1998)) and the absence of purely hydrodynamic means for driving accretion in disks (Balbus and Hawley, 1998) further motivated the development of relativistic MHD schemes. More recently there have been several efforts to develop GRMHD codes, including the already-mentioned work by Koide and collaborators and by Komissarov. A ZEUS-like scheme for GRMHD has also been developed and is described in a companion paper (De Villiers and Hawley, 2002). Special relativistic MHD (SRMHD) is the foundation for any GRMHD scheme, although there are nontrivial problems in making the transition to full general relativity. SRMHD schemes have been developed by vanPutten (1993); Balsara (2001); Koldoba et al. (2002); Komissarov (1999) and Del Zanna et al. (2003). We were particularly influenced by the clear development of the fundamental equations in Komissarov (1999) for his Godunov SRMHD scheme based on a Roe- type approximate Riemann solver, and by the work of Del Zanna and Bucciantini (2002) and Del Zanna et al. (2003) who chose to use the simple approximate Riemann solver of Harten et al. (1983) in their special relativistic hydrodynamics and SRMHD schemes, respectively. Our numerical scheme is called HARM, for High Accuracy Relativistic Magnetohydrodynamics.2 In the next section we develop the basic equations in the form used for numerical integration in HARM (§2). In §3 we describe the basic algorithm. In §4 we describe the performance of the code on a series of test problems. In §5 we describe a sample evolution of a magnetized torus near a rotating black hole.

2Also named in honor of R. H¨arm,who with M. Schwarzschild was a pioneer of numerical astrophysics.

83 3.3 A GRMHD Primer

The equations of general relativistic MHD are well known, but for clarity we will develop them here in the same form used in numerical integration. Unless otherwise noted c = 1 and we follow the notational conventions of Misner et al. (1973), hereafter MTW. The reader may also ﬁnd it useful to consult Anile (1989). The ﬁrst governing equation describes the conservation of particle number:

µ (nu );µ = 0. (3.1)

Here n is the particle number density and uµ is the four-velocity. For numerical purposes we rewrite this in a coordinate basis, replacing n with the “rest-mass density” ρ0 = mn, where m is the mean rest-mass per particle: 1 √ √ ∂ ( −g ρ uµ) = 0. (3.2) −g µ 0

Here g ≡ Det(gµν). The next four equations express conservation of energy-momentum:

µ T ν;µ = 0, (3.3)

µ where T ν is the stress-energy tensor. In a coordinate basis,

¡√ t ¢ ¡√ i ¢ √ κ λ ∂t −g T ν = −∂i −g T ν + −g T λ Γ νκ, (3.4)

λ where i denotes a spatial index and Γ νκ is the connection. The energy-momentum equations have been written with the free index down for a reason. Symmetries of the metric give rise to conserved currents. In the Kerr metric, for example, the axisymmetry and stationary nature of the metric give rise to conserved angular momentum and energy currents. In general, for metrics with an ignorable coordinate xµ the source term on the t right hand side of the evolution equation for Tµ vanish. These source terms do not vanish when the equation is written with both indices up. The stress-energy tensor for a system containing only a perfect fluid and an electromagnetic field is the sum of a fluid part,

µν µ ν µν Tﬂuid = (ρ0 + u + p)u u + pg , (3.5)

(here u ≡ internal energy and p ≡ pressure) and an electromagnetic part,

1 T µν = F µαF ν − gµνF F αβ. (3.6) EM α 4 αβ

Here F µν is the electromagnetic ﬁeld tensor (MTW: “Faraday”), and for convenience we have

84 √ absorbed a factor of 4π into the definition of F . The electromagnetic portion of the stress-energy tensor simplifies if we adopt the ideal MHD approximation, in which the electric field vanishes in the fluid rest frame due to the high conductivity of the plasma (“E + v × B = 0”). Equivalently the Lorentz force on a charged particle vanishes in the fluid frame: µν uµF = 0. (3.7)

It is convenient to deﬁne the magnetic ﬁeld four-vector

1 bµ ≡ ²µνκλu F , (3.8) 2 ν λκ

µνλδ √1 where ² is the Levi-Civita tensor. Recall that (following the notation of MTW) ² = − −g [µνλδ], where [µνλδ] is the completely antisymmetric symbol and = 0, 1, or −1. These can be combined (with the aid of identity 3.50h of MTW):

µν µνκλ F = ² uκbλ. (3.9)

µ Substitution and some manipulation (using identities 3.50 of MTW and bµu = 0; the latter follows from the deﬁnition of bµ and the antisymmetry of F ) yields

1 T µν = b2uµuν + b2gµν − bµbν. (3.10) EM 2

Notice that the last two terms are nearly identical to the nonrelativistic MHD stress tensor, while the ﬁrst term is higher order in v/c. To sum up,

1 T µν = (ρ + u + p + b2)uµuν + (p + b2)gµν − bµbν (3.11) MHD 0 2 is the MHD stress-energy tensor. The electromagnetic ﬁeld evolution is given by the source-free part of Maxwell’s equations

Fµν,λ + Fλµ,ν + Fνλ,µ = 0. (3.12)

The rest of Maxwell’s equations determine the current

µ µν J = F ;ν, (3.13) and are not needed for the evolution, as in nonrelativistic MHD. Maxwell’s equations can be written in conservative form by taking the dual of eq.(3.12):

∗µν F ;ν = 0. (3.14)

∗ 1 κλ Here Fµν = 2 ²µνκλF is the dual of the electromagnetic ﬁeld tensor (MTW: “Maxwell”). In ideal

85 MHD F ∗µν = bµuν − bνuµ, (3.15) which can be proved by taking the dual of eq.(3.9). µ µ The components of b are not independent, since b uµ = 0. Following, e.g., Komissarov (1999), it is useful to deﬁne the magnetic ﬁeld three-vector Bi = F ∗it. In terms of Bi,

t i µ b = B u giµ, (3.16)

bi = (Bi + btui)/ut. (3.17)

The space components of the induction equation then reduce to

√ i √ j i i j ∂t( −gB ) = −∂j( −g (b u − b u )) (3.18) and the time component reduces to

1 √ √ ∂ ( −g Bi) = 0, (3.19) −g i which is the no-monopoles constraint. The appearance of Bi in these last two equations are what motivates the introduction of the ﬁeld three-vector in the ﬁrst place. To sum up, the fundamental equations as used in HARM are: the particle number conservation equation (3.2); the four energy-momentum equations (3.4), written in a coordinate basis and using the MHD stress-energy tensor of equation (3.11); and the induction equation (3.18), subject to the constraint (3.19). These hyperbolic 3 equations are written in conservation form, and so can be solved numerically by well-known techniques.

3.4 Numerical Scheme

There are many possible ways to numerically integrate the GRMHD equations. A first, zeroth- order choice is between conservative and nonconservative schemes. Nonconservative schemes such as ZEUS (Stone and Norman, 1992) have enjoyed wide use in numerical astrophysics. They permit the integration of an internal energy equation rather than a total energy equation. This can be advantageous in regions of a flow where the internal energy is small compared to the total energy (highly supersonic flows), which is a common situation in astrophysics. A nonconservative scheme for GRMHD following a ZEUS-like approach has been developed and is described in a companion paper (De Villiers and Hawley, 2002). We have decided to write a conservative scheme. One advantage of this choice is that in one dimension, total variation stable schemes are guaranteed to converge to a weak solution of the equations by the Lax-Wendroff theorem (Lax and Wendroff, 1960) and by a theorem due to LeVeque

3The GRMHD equations exhibit the same degeneracies as the nonrelativistic MHD equations.

86 (1998). While no such guarantee is available for multidimensional ﬂows, this is a reassuring starting point. Furthermore, one is guaranteed that a conservative scheme in any number of dimensions will satisfy the jump conditions at discontinuities. This is not true in artiﬁcial viscosity based nonconservative schemes, which are also known to have trouble in relativistic shocks (Norman and Winkler, 1986). Conservative schemes for GRMHD have also been developed by Komissarov (2001) and by Koide et al. (1999). A conservative scheme updates a set of “conserved” variables at each timestep. Our vector of conserved variables is √ t t t i U ≡ −g(ρ0u ,Tt ,Ti ,B ). (3.20)

These are updated using ﬂuxes F. We must also choose a set of “primitive” variables, which are interpolated to model the ﬂow within zones. We use variables with a simple physical interpretation:

i i P = (ρ0, u, v ,B ). (3.21)

Here vi = ui/ut is the 3-velocity.4 The functions U(P) and F(P) are analytic, but the inverse operations (so far as we can determine) are not. 5 There is also no simple expression for F(U). To evaluate U(P) and F(P) one must find ut and bµ from vi and Bi. To find ut, solve the µ ν µ quadratic equation gµνu u = −1. Next use equations (3.16) and (3.17) to find b ; these require only multiplications and additions. The remainder of the calculation of U(P) and F(P) requires raising and lowering of indices followed by direct substitution in equation (3.11) to find the components of the MHD stress-energy tensor. Since we update U rather than P, we must solve for P(U) at the end of each timestep. We use a multidimensional Newton-Raphson routine with the value of P from the last timestep as an initial guess. Since Bi can be obtained analytically, only 5 equations need to be solved. The Newton-Raphson method requires an expensive evaluation of the Jacobian ∂U/∂P. In practice we evaluate the Jacobian analytically. It is possible to evaluate the Jacobian by numerical derivatives, but this is both expensive and a source of numerical noise. The evaluation of P(U) is at the heart of our numerical scheme; the procedure must be robust. We have found that it is crucial that the errors (differences between the current and target values of U) used to evaluate convergence in the Newton-Raphson scheme be properly normalized. We √ t normalized the errors with −gρ0u . To evaluate F we use a MUSCL type scheme with “HLL” fluxes (Harten et al., 1983). The fluxes are defined at zone faces. A slope-limited linear extrapolation from the zone center gives

PR and PL, the primitive variables at the right and left side of each zone interface. We have implemented the monotonized central (“Woodward”, or “MC”) limiter, the van Leer limiter, and the minmod limiter; unless otherwise stated, the tests described here use the MC limiter, which is

4We initially used ui as primitive variables, but the inversion ut(ui) is not always single-valued, e.g. inside the ergosphere of a black hole. That is, there are physical ﬂows with the same value of ui but diﬀerent values of ut. 5Del Zanna et al. (2003) have found that the inversion P(U) can be reduced analytically to the solution of a single, nonlinear equation.

87 the least diﬀusive of the three.

From PR, PL, calculate the maximum left and right-going wave speeds c±,R, c±,L, and the ﬂuxes

FR = F(PR) and FL = F(PL). Let cmax ≡ MAX(0, c+,R, c+,L) and cmin ≡ −MIN(0, c−,R, c−,L), then the HLL ﬂux is c F + c F − c c (U − U ) F = min R max L max min R L (3.22) cmax + cmin

If cmax = cmin, the HLL ﬂux becomes the so-called local Lax-Friedrichs ﬂux.

3.4.1 Constrained Transport

The pure HLL scheme will not preserve any numerical representation of ∇·B = 0. An incomplete list of options for handling this constraint numerically includes: (1) ignore the production of monopoles by truncation error and hope for the best (in our experience this causes the scheme to fail in any complex flow); (2) introduce a divergence-cleaning step (this entails solving an elliptic equation at each timestep); (3) use an Evans and Hawley (1988) type constrained transport scheme (this requires a staggered mesh, so that the magnetic field components are zone face centered); (4) introduce a diffusion term that causes numerically generated monopoles to diffuse away (Marder (1987); this typically leaves a monopole field with rms value somewhat larger than the truncation error). We have chosen a fifth option, a version of constrained transport that can be used with a zone-centered scheme. This idea was introduced by one of us in Toth (2000), where it is called the flux-interpolated constrained transport (or “flux-CT”) scheme 6. It preserves a numerical representation of ∇ · B = 0 by smoothing the fluxes with a special operator. The disadvantage of this method is that it is more diffusive than the “bare”, unconstrained scheme. The advantage is that it is extremely simple. To clarify how zone-centered constrained transport works we now give a specific example for a special relativistic problem in Cartesian coordinates t, x, y. To fix notation, write the induction equation as i j ∂tB = −∂jFi , (3.23) √ j where we have used −g = 1 and the fluxes Fi are

x Fx = 0 F y = 0 y (3.24) y y x x y Fx = b u − b u x x y y x y Fy = b u − b u = −Fx

Notice that the fluxes are centered at different locations on the grid: F x fluxes live on the x face of each zone at grid location i − 1/2, j (we use i, j to denote the center of each zone), while F y fluxes

6We have also experimented with Tóth’s“flux-CD”, which preserves a different representation of ∇ · B = 0. This appears to be slightly less robust. It also has a larger effective stencil.

88 j live on the y face at i, j − 1/2. The smoothing operator replaces the numerical (HLL-derived) Fi ˜j with Fi , deﬁned by

˜x Fx (i − 1/2, j) = 0 ˜y Fy (i, j − 1/2) = 0h ˜y 1 y Fx (i, j − 1/2) = 8 2Fx (i, j − 1/2) y y +Fx (i + 1, j − 1/2) + Fx (i − 1, j − 1/2) x x −Fy (i − 1/2, j) − Fy (i + 1/2, j) i (3.25) −F x(i − 1/2, j − 1) − F x(i + 1/2, j − 1) h y y ˜x 1 x Fy (i − 1/2, j) = 8 2Fy (i − 1/2, j) x x +Fy (i − 1/2, j + 1) + Fy (i − 1/2, j − 1) y y −Fx (i, j − 1/2) − Fx (i, j + 1/2) i y y −Fx (i − 1, j − 1/2) − Fx (i − 1, j + 1/2)

It is a straightforward but tedious exercise to verify that this preserves the following corner-centered numerical representation of ∇ · B:

∇ · B = (Bx(i, j) + Bx(i, j − 1) − Bx(i − 1, j) − Bx(i − 1, j − 1)) /(2∆x) (3.26) + (By(i, j) + By(i − 1, j) − By(i, j − 1) − By(i − 1, j − 1)) /(2∆y), where ∆x and ∆y are the grid spacing.

3.4.2 Wave Speeds

The HLL approximate Riemann solver does not require eigenvectors of the characteristic matrix (as would a Roe-type scheme), but it does require the maximum and minimum wave speed (eigenvalues). These wave speeds are also required to fix the timestep via the Courant conditions. The relevant speed is the phase speed “ω/k” of the wave, and it turns out that only speeds for waves with wavevectors aligned along coordinate axes are required. Suppose, for example, that one needs to know how rapidly signals propagate in the fluid along the x1 direction. First, find a wavevector kµ = (−ω, k1, 0, 0), that satisfies the dispersion relation for the mode in question: D(kµ) = 0. Then the wave speed is simply ω/k1.

The dispersion relation D(kµ) = 0 for MHD waves has a simple form in a comoving frame. In terms of the relativistic sound speed c2 = (∂(ρ + u)/∂p)−1 = γp/w, (the last holds only if s 0 √ s 2 p = (γ −1)u) and the relativistic Alfv´envelocity vA = B/ E, where E = b +w and w ≡ ρ0 +u+p, the dispersion relation is ¡ ¢ ω ω2 − (k · v )2 × ¡ ¡ A ¢ ¢ (3.27) 4 2 2 2 2 2 2 2 2 2 2 2 2 ω − ω k (vA + cs(1 − vA/c )) + cs(k · vA) /c + k cs(k · vA) = 0,

Here c is the (temporarily reintroduced) speed of light. The ﬁrst term is the zero frequency entropy

89 mode, the second is the Alfvénmode, and the third contains the fast and slow modes. The eighth mode is eliminated by the no-monopoles constraint. √ √ The relativistic sound speed asymptotes to c γ − 1 = c/ 3 for γ = 4/3, and the Alfvén 2 speed asymptotes to c. In the limit that B /ρ0 À 1 and p/ρ0 6À 1, the GRMHD equations are a superset of the time-dependent, force-free electrodynamics equations recently discussed by Komissarov (2002c); these contain fast modes and Alfvénmodes that move with the speed of light. They are indistinguishable from vacuum electromagnetic modes only when their wavevector is oriented along the magnetic field. To find the maximum wave speeds we need to evaluate the comoving-frame dispersion relation for the fast wave branch from coordinate frame quantities. This is straightforward because the µ 2 µ dispersion relation depends on scalars, which can be evaluated in any frame: ω = kµu ; k = KµK , where K = (g +u u )kν is the part of the wavevector normal to the fluid 4-velocity; v2 = b bµ/E; µ µν √ µ ν A µ µ (k · vA) = kµb / E. The relevant portion of the dispersion relation (for fast and slow modes) is thus a fourth order polynomial in the components of kµ. This can be solved either analytically or by standard numerical methods. The two fast mode speeds are then used as cmax and cmin in the HLL fluxes. We have found it convenient to replace the full dispersion relation by an approximation:

2 2 2 2 2 2 ω = (vA + cs(1 − vA/c ))k . (3.28)

This overestimates the maximum wavespeed by a factor ≤ 2 in the comoving frame. The maximum error occurs for k k vA, vA/cs = 1, and vA ¿ c, and it is usually much less, particularly if the ﬂuid is moving super-Alfv´enicallywith respect to the grid. This approximation is convenient because it is quadratic in kµ, and so can be solved more easily.

3.4.3 Implementation Notes

For completeness we now give some details of the implementation of the algorithm. Time Stepping. Our scheme is made second order in time by taking a half-step from tn to tn+1/2, evaluating F(P(tn+1/2)), and using that to update U(tn) to U(tn+1). Modiﬁcation of Energy Equation. A direct implementation of the energy equation can be inaccurate because the magnetic and internal energy density can be orders of magnitude smaller than the rest mass density. To avoid this we subtract the particle number conservation equation from the energy equation, i.e., we evolve

√ t t √ i i √ κ λ ∂t( −g(Tt + ρ0u )) = −∂i( −g(Tt + ρ0u )) + −gTλ Γtκ. (3.29)

In the nonrelativistic limit, this procedure subtracts the rest mass energy density from the total energy density.

90 µν Speciﬁcation of Geometric Quantities. In two dimensions we need to evaluate gµν, g , √ µ and −g at four points in every grid zone (the center, two faces, and one corner) and Γνλ at the zone center. It would be diﬃcult to accurately encode analytic expressions for all these quantities.

HARM is coded so that an analytic expression need only be provided for gµν; all other geometric quantities are calculated numerically. The connection, for example, is obtained to sufficient accuracy by numerical differentiation of the metric. This minimizes the risk of coding errors in specifying the geometry. It also minimizes coordinate dependent code, making it relatively easy to change coordinate systems. Minimal coordinate dependence, besides following the spirit of general relativity, enables one to perform a sort of fixed mesh refinement by adapting the coordinates to the problem at hand. For example, near a Kerr black hole we use log(r) as the radial coordinate instead of the usual Boyer-Lindquist r, and this concentrates numerical resolution toward the horizon, where it is needed. Density and Internal Energy Floors. Negative densities and internal energies are forbidden by the GRMHD equations, but numerically nothing prevents their appearance. In fact, negative internal energies are common in numerical integrations with large density or pressure contrast. Following common practice, we prevent this by introducing “floor” values for the density and internal energy. These floors are enforced after the half-step and the full step. They preserve velocity but do not conserve rest mass or energy-momentum. Outflow Boundary Conditions. In the rotating black hole calculations described below we use outflow boundary conditions at the inner and outer radial boundaries. The usual implementation of outflow boundary conditions is to simply copy the primitive variables from the boundary zones into the ghost zones. This can result in unphysical values of the primitive variables in the ghost zones– for example, velocities that lie outside the light cone– because of variations in the metric between the boundary and ghost zones. We have experimented with a variety of schemes for projecting variables into the ghost zones in the context of black hole accretion flow calculations (described in §5). We find that some are more robust than others. The most robust extrapolates the density, internal energy, and radial magnetic field according to √ √ P (ghost) = P (boundary) −g(boundary)/ −g(ghost), (3.30) the θ and φ components of the velocity and magnetic field according to

P (ghost) = P (boundary)(1. − ∆r/r), (3.31) and the radial velocity according to

P (ghost) = P (boundary)(1. + ∆r/r). (3.32)

The extrapolation of θ and φ components of magnetic ﬁeld and velocity results in weak damping

91 of these components near the boundary. Slightly diﬀerent choices of the extrapolation coeﬃcients (i.e. (1. − 2∆r/r)) are much less robust. Performance. We have implemented both serial and parallel versions of the code. In serial mode the code integrates the black hole accretion problem (described in §5) at ≈ 54, 000 zone cycles per second on a 2.4 GHz Intel Pentium 4, when compiled using the Intel C compiler. The parallel code was implemented using MPI.

3.5 Code Veriﬁcation

Here we present a test suite for verifying a GRMHD code. The tests are nonrelativistic, special and general relativistic, and one and two dimensional. The list of problems for which there are known, exact solution is short, since exact solutions of multidimensional GRMHD problems are algebraically complicated. This list of test problems was developed in collaboration with J. Hawley and J.-P. de Villiers. Unless otherwise stated we set γ = 4/3 and c = 1.

3.5.1 Linear Modes

This first test considers the evolution of a small amplitude wave in two dimensions. The unperturbed i y z x x state is ρ0 = 1, p = 1, u = 0, B = B = 0, B = B0 . The basic state is parametrized x 2 2 by α = (B0 ) /(ρ0c ); our fiducial test runs have α = 1. Onto this basic state we introduce a perturbation of the form exp(ik · x − iω(k)t), where (kx, ky) = (2π, 2π), and the amplitude is fixed y −4 x by δB = 10 B0 . The computational domain is x, y ∈ [0, 1), [0, 1), and the boundary conditions are periodic. The wave is either slow, Alfvénic,or fast. This test exercises almost all terms in the governing equations. The numerical resolution is

(Nx,Ny) ≡ N(5, 4) zones, and the integration runs for a single wave period 2π/ω, so that a perfect scheme would return the simulation to its original state. We measure the L1 norm of the difference between the final state and the initial state for each primitive variable. For example, we measure Z L1(δρ0) = dxdy|ρ0(t = 0) − ρ0(t = 2π/ω)|. (3.33) for the density. All primitive variables exhibit similar convergence properties (as they must, since with the exception of the magnetic field, they are tightly coupled together). In Figures 3.1, 3.2, and 3.3 we present the L1 norm of the error for runs using the monotonized central limiter and a Courant number of 0.8, in addition to the results for the minmod limiter. These runs have α = 1. Figure 3.1 shows the results for the slow wave, Figure 3.2 for the Alfvénwave, and Figure 3.3 for the fast wave. Evidently the convergence rate asymptotes to second order, although more slowly for the minmod limiter. The code performs similarly well over a range of α, provided only that δB2/2 ¿ p, which is x necessary for the wave to be in the linear regime. We have been unable to find a value of B0 where

92 Figure 3.1 The L1 norm of the error in u for a slow wave as a function of Nx for both the monotonized central (MC) and minmod limiter. The straight lines show the slope expected for second order convergence.

93 Figure 3.2 The L1 norm of the error in the single nonzero component of the velocity for an Alfv´en wave as a function of Nx for both the monotonized central (MC) and minmod limiter. The straight lines show the slope expected for second order convergence.

94 Figure 3.3 The L1 norm of the error in u for a fast wave as a function of Nx for both the monotonized central (MC) and minmod limiter. The straight lines show the slope expected for second order convergence.

95 x the code fails completely for a linear amplitude disturbance, although for very large values of B0 the evolution becomes inaccurate because of numerical noise in the evaluation of P(U).

3.5.2 Nonlinear Waves

Komissarov (1999) has proposed a suite of one-dimensional nonlinear tests for special relativistic MHD. Komissarov presents a total of 9 tests (see his Table 1). The nonlinear Alfv´enwave (test 5), and the compound wave (test 6) cannot be reconstructed without a separate derivation of the exact analytic solution, and we will not provide that here. For the remaining tests Komissarov’s Table 1 contains several misprints that are corrected in Komissarov (2002b). Our code is able to integrate each of Komissarov’s remaining 7 tests, although in some cases we must reduce the Courant number (usually 0.8) or resort to the slightly more robust van Leer slope limiter. Tests that required special treatment are: fast shock (Courant number = 0.5); shock tube 1 (Courant number = 0.3, van Leer limiter); shock tube 2 (Courant number = 0.5); collision (Courant number = 0.3; van Leer limiter). x Figures 3.4 and 3.5 show the run of ρ0 and u , respectively, for all 7 tests. These may be compared with Komissarov’s ﬁgures. Notice that, unlike Komissarov, we have in all cases set

Nx = 400 and x ∈ (−2, 2). We have not obtained the exact solutions used by Komissarov, but the solutions can still be checked quantitatively. For example, the slow shock wave speed is 0.5; since the calculation ends at t = 2 the slow shock front should be, and is, located at x ≈ 1.0. The fast shock speed is 0.2, so at t = 2.5 the fast shock wave front should be, and is, located at x ≈ 0.5. There are artifacts evident in the ﬁgures. In particular there is ringing near the base of the switch-on and switch-oﬀ rarefaction waves. This is common and is seen in Komissarov’s results as well. In addition the narrow, Lorentz-contracted shell of material behind the shock in shock tube

1 is poorly resolved; the correct shell density is ≈ 0.88 but a resolution of Nx > 800 is required to find this result to an accuracy of a few percent. There is also a transient associated with the fast shock that propagates off the grid and so is not visible in Figures 3.4 and 3.5. A complete set of nonlinear wave tests for one dimensional nonrelativistic MHD was developed by Ryu and Jones (1995) (hereafter RJ). We can run these under HARM by rescaling the speed of light to c = 102 in code units, where all velocities in the tests are O(1). This should lead to results that agree with RJ to O(v/c) ≈ 1%. The results can be checked quantitatively by comparison to the tables provided by RJ. Figure 3.6 shows our results for RJ test 5A, which is a version of the familiar Brio and Wu (1988) magnetized shock tube test. Like RJ we use 512 zones between x = 0 and x = 1, and we measure the results at t = 0.15. To compare to RJ quantitatively, consider ux in the region behind the fast rarefaction wave, near x = 0.7. RJ report ux = −0.277 here, while we measure ux = −0.273, which differs by 1%, as expected. Similar agreement is found for the other variables. The most unsatisfactory feature of the solution is the visible post-shock oscillations. The amplitude of these features varies, depending on the Courant number (here 0.9) and the choice of slope limiter

96 Figure 3.4 The run of density in the Komissarov nonlinear wave tests.

97 Figure 3.5 The run of ux in the Komissarov nonlinear wave tests.

98 Figure 3.6 Snapshot of the ﬁnal state in HARM’s integration of Ryu & Jones test 5A (a version of the Brio & Wu shock tube) but with c = 100. The ﬁgure shows primitive variable values at t = 0.15. Quantitative agreement is found to within ≈ 1%, as expected.

99 (here MC).

Figure 3.7 shows our results for RJ test 2A. In the region near x = 0.6, RJ report that By =

1.4126, and we find essentially exact agreement (By = 1.41262) after averaging over a small region near x = 0.6. This test has two pairs of closely spaced slow shocks and rotational discontinuities that are difficult to resolve, and our scheme barely obtains the correct peak values of uy and By, even though there are about 20 zones inside the “horns” visible in the By panel of the figure.

3.5.3 Transport

This special relativistic test evolves a disk of enhanced density moving at an angle to the grid until it returns to its original position. The computation is carried out in a domain x, y ∈ [−0.5, 0.5), [−0.5, 0.5) and the boundary conditions are periodic. The initial state has vx = vy = 0.7, x y t or u = u ≈ 4.95, corresponding to u ≈ 7.07. The initial density ρ0 = 1 except in a disk at r < rs = 0.45, where ρ0 = 3/2 + cos(2πr/rs). The initial pressure p = 1, and the initial magnetic ﬁeld is zero. The test is run until t = 10/7, when the system should return exactly to its initial state. Numerically, we use the monotonized central limiter and set the Courant number to 0.8. The resolution is ﬁxed so that Nx = 5Ny/4. Figure 3.8 shows the L1 norm of the error in ρ0 as a function of x resolution. The convergence rate asymptotes to second order.

3.5.4 Orszag-Tang Vortex

The Orszag-Tang vortex (OTV) is a classic nonlinear MHD problem (Orszag and Tang, 1979). Here we compare our code, with the speed of light set to 100 so that that it is eﬀectively nonrelativistic, to the output of VAC (Toth and Odstrcil, 1996), an independent nonrelativistic code developed by one of us. The version of VAC used here is TVD-MUSCL using the monotonized central limiter. It is dimensionally unsplit and uses a scheme similar to HARM to control ∇ · B. The problem is integrated in the periodic domain x ∈ (−π, π], y ∈ (−π, π] from t = 0 to t = π. Our version of the OTV has γ = 4/3, but is otherwise identical to the standard problem.

Results are shown in Figure 3.9, which shows ρ0 along a cut through the model at y = π/2 and t = π. The resolution is 6402. The solid line shows the results from HARM; the dashed line shows the results from VAC. The lower solid line shows the difference between the two multiplied by 4. Evidently our code behaves similarly to VAC on this problem. We can quantify this by asking how the difference between the HARM and VAC solutions changes as a function of resolution. Figure 3.10 shows the variation in the L1 norm of the difference between the two solutions. Thus the line marked ρ0 shows Z 2 2 dxdy|ρ0(HARM; N ) − ρ0(VAC; N )| (3.34) evaluated at t = π. The codes converge to one another approximately linearly, as expected for a

100 Figure 3.7 Snapshot of the ﬁnal state in HARM’s integration of Ryu & Jones test 2A, with c = 100.

101 Figure 3.8 Convergence results for the transport test.

102 Figure 3.9 A cut through the density in the nonrelativistic Orszag-Tang vortex solution from HARM (solid line, with c = 100), from VAC (dashed line), and 4× the diﬀerence (lower solid line) at a resolution of 6402.

103 Figure 3.10 Comparison of results from HARM and the nonrelativistic MHD code VAC for the Orszag-Tang vortex. The plot shows the L1 norm of the diﬀerence between the two results as a function of resolution for the primitive variables ρ0 (squares) and u (triangles). The straight line shows the slope expected for ﬁrst order convergence. The errors are large because they are an integral over an area of (2π)2.

104 flow containing discontinuities. If this study were extended to higher resolution convergence would eventually cease because the HARM solution would differ from the VAC solution due to finite relativistic corrections.

3.5.5 Bondi Flow in Schwarzschild Geometry

Spherically symmetric accretion (Bondi ﬂow) in the Schwarzschild geometry has an analytic solution (see, e.g., Shapiro and Teukolsky 1983) that can be compared with the output of our code. This appears to be a one-dimensional test, but for HARM it is actually two dimensional. Although the pressure is independent of the Boyer-Lindquist coordinate θ, the θ acceleration does not vanish identically. This is because pressure enters the momentum equations through a ﬂux (−∂θ(p sin θ) in the Newtonian limit) and a source term (p cos θ in the Newtonian limit). Analytically these terms cancel; numerically they produce an acceleration that is of order the truncation error.

Our test problem follows that set out in Hawley et al. (1984): we ﬁx the sonic point rs = 2 2 r 8GM/c , M˙ = 4πr ρ0u = −1, and γ = 4/3. The problem is integrated in the domain r ∈ (1.9, 20)GM/c2 for ∆t = 100GM/c3. We use coordinates based on the Kerr-Schild system, whose line element is

ds2 = −(1 − 2r/ρ 2)dt2 + (4r/ρ 2)drdt + (1 + 2r/ρ 2)dr2 + ρ 2dθ2+ ¡ 0 0 ¢ 0 0 2 2 2 2 2 2 sin θ ρ0 + a (1 + 2r/ρ0 ) sin θ dφ (3.35) 2 2 2 2 −(4ar sin θ/ρ0 )dtdφ − 2a(1 + 2r/ρ0 ) sin θdrdφ,

2 2 2 2 where we have set GM = c = 1. In (3.35) only ρ0 = r +a cos (θ); elsewhere ρ0 is density. In this test, a = 0. We modify these coordinates by replacing r by x1 = log(r). The new coordinates are implemented by changing the metric rather than changing the spacing of grid zones. We measure the L1 norm of the difference between the initial conditions (exact analytic solution) and the final state. The difference is taken over the inner 3/4 of the grid in each direction, thus excluding boundary zones where errors may scale differently. This test exercises many terms in the code because in Kerr-Schild coordinates only three of the ten independent components of the metric are zero.

The L1 norm of the error in internal energy for the Bondi test is shown in Figure 3.11. Similar results obtain for the other independent variables. The solution converges at second order.

3.5.6 Magnetized Bondi Flow

The next test considers a Bondi flow containing a spherically symmetric, radial magnetic field. The solution to this problem is identical to the Bondi flow described above because the flow is along the magnetic field, so all magnetic forces cancel exactly. This is a difficult test, however, because numerically the magnetic terms cancel only to truncation error. This causes problems at high magnetic field strength. We use the same Bondi solution as in the last subsection and parameterize the magnetic field

105 Figure 3.11 Convergence results for the unmagnetized Bondi accretion test onto a Schwarzschild black hole. The straight line shows the slope expected for second order convergence.

106 2 2 2 strength by b /ρ0 at the inner boundary. Our ﬁducial run has (b /ρ0)(r = 1.9GM/c ) = 10.56.

The L1 norm of the error in the internal energy is shown in Figure 3.12. Similar results obtain for the other independent variables. The solution converges at second order. 2 2 We have considered models with a range of (b /ρ0)(rin). Lowering (b /ρ0)(rin) produces results 2 similar to those in our fiducial test run. Raising (b /ρ0)(rin) first causes the code to produce inaccurate results (at ∼ 103, where the radial velocity profile is smoothly distorted from the true solution) and then to fail (at ∼ 104). This is an example of a general problem with conservative schemes when the basic energy density scales (rest mass, magnetic, and internal) differ by many orders of magnitude.

3.5.7 Magnetized Equatorial Inﬂow in Kerr Geometry

This test considers the steady-state, magnetized inflow solutions found by Takahashi et al. (1990), as specialized to the case of inflow inside the marginally stable orbit by Gammie (1999). This solution exercises many of the important terms in the governing equations, in particular the interaction of the magnetized fluid with the Kerr geometry. We use Boyer-Lindquist coordinates to specify this problem, but the solution is integrated in the modified Kerr-Schild coordinates described above. The flow is assumed to lie in the neighborhood of the black hole’s equatorial plane and is thus one dimensional, much like the Weber-Davis model for the solar wind. As above, we set GM = c = 1. The particular inflow solution we consider is for a black hole with spin parameter a/M = 0.5. 2 r The model has an accretion rate FM = −1 = 2πρ0r u (adopting the notation of Gammie 1999). 2 r The magnetization parameter Fθφ = r B = 0.5. The flow is constrained to match to a circular orbit at the marginally stable orbit. This is enough to uniquely specify the flow. It follows that

(see Gammie 1999) Ftθ = ΩFθφ, where Ω is the orbital frequency at the marginally stable orbit. For a/M = 0.5, Ω ≈ 0.10859. The solution that is regular at the fast point has angular momentum 2 r r 2 r r flux FL = 2πr (u uφ − b bφ) ≈ −2.8153 and energy flux FE = 2πr (u ut − b bt) ≈ −0.90838. The fast point is located at r ≈ 3.6167, and the radial component of the four-velocity there is ur = −0.040547. Figure 3.13 shows the radial run of the solution. We initialize the flow with a numerical solution that is subject to roundoff error. The near- equatorial nature of the solution is mimicked by using a single zone in the θ direction centered on

θ = π/2. The computational domain runs from 1.02× the horizon radius rh to 0.98× the radius of the marginally stable orbit rmso. For a/M = 0.5, rh = 1.866, and rmso = 4.233. The analytic ﬂow model is cold (zero temperature) but we set the initial internal energy in the code equal to a small value instead. The model is run for ∆t = 15. r φ φ Figure 3.14 shows the L1 norm of the error in ρ0, u , u , and B and a function of the total number of radial gridpoints N. The straight line shows the slope expected for second order convergence. The small deviation from second order convergence at high N in several of the variables is due to numerical errors in the initial solution, which relies on numerical derivatives (Gammie,

107 Figure 3.12 Convergence results for the magnetized Bondi accretion test onto a Schwarzschild black hole. The straight line shows the slope expected for second order convergence.

108 Figure 3.13 The equatorial inﬂow solution in the Kerr metric for a/M = 0.5 and magnetization parameter Fθφ = 0.5. The panels show density, radial component of the four-velocity in Boyer- Lindquist coordinates (with the square showing the location of the fast point), the φ component of the four-velocity, and the toroidal magnetic ﬁeld Bφ = F φt.

109 Figure 3.14 Convergence results for the magnetized inﬂow solution in a Kerr metric with a/M = 0.5. Parameters for the initial, quasi-analytic solution are given in the text. The straight line shows the slope expected for second order convergence. The L1 error norm for each of the nontrivial variables are shown. The small deviation from second order convergence at high resolution is due to numerical errors in the quasi-analytic solution used to initialize the solution.

110 1999).

3.5.8 Equilibrium Torus

Our next test concerns an equilibrium torus. This class of equilibria, found originally by Fishbone and Moncrief (1976) and Abramowicz et al. (1978), consist of a “donut” of plasma surrounding a black hole. The donut is supported by both centrifugal forces and pressure and is embedded in a vacuum. Here we consider a particular instance of the Fishbone & Moncrief solution. A practical problem with this test is that HARM abhors a vacuum. We have therefore introduced floors on the density and internal energy that limit how small these quantities can be. The −4 −3/2 −6 −5/2 floors are dependent on radius, with ρ0min = 10 (r/rin) and umin = 10 (r/rin) . This means that the torus is surrounded by an insubstantial, but dynamic, accreting atmosphere that interacts with the torus surface. To minimize the influence of the atmosphere on our convergence test, we take the L1 norm of the change in variables only over that region where ρ0 > 0.02ρ0max. The problem is integrated in modified Kerr-Schild coordinates. The Kerr-Schild radius r has been replaced by the logarithmic radial coordinate x1 = ln(r), and the Kerr-Schild latitude θ has been replaced by x2 such that θ = πx2 + (1/2)(1 − h) sin(2πx2). Clearly 0 ≤ x2 ≤ 1 maps to 0 ≤ θ ≤ π. This coordinate transformation has a single adjustable parameter h; for h = 1 we recover the original coordinate system (the θ coordinate is simply rescaled by π). As h → 0 numerical resolution is concentrated near the midplane. We have integrated a Fishbone-Moncrief disk around a black hole with a/M = 0.95, to maximize t general relativistic effects. We set u uφ = const. = 3.85 (this is the defining feature of the Fishbone-

Moncrief equilibria) and rin = 3.7. The grid extends radially from rin = 0.98rh = to rout = 20. The coordinate parameter h described in the last paragraph is set to 0.2. The numerical resolution is N × N, where N = 8, 16, 32,..., 512, and the solution is integrated for ∆t = 10. Figure 3.15 shows the L1 norm of the error for each variable as a function of N. Second order convergence is obtained. The sum of the evidence presented in this section strongly suggests that we are solving the equations of GRMHD without signiﬁcant, compromising errors.

3.6 Magnetized Torus Near Rotating Black Hole

Finally we oﬀer an example of how HARM can be applied to a real astrophysical problem: the evolution of a magnetized torus near a rotating black hole. Again we set GM = c = 1.

The initial conditions contain a Fishbone-Moncrief torus with a/M = 0.5, r(pmax) = 12, and rin = 6. Superposed on this equilibrium is a purely poloidal magnetic ﬁeld with vector potential

Aφ ∝ MAX(ρ0/ρ0max −0.2, 0), where ρ0max is the peak density in the torus. The ﬁeld is normalized 2 so that the minimum value of pgas/pmag = 10 . The orbital period at the pressure maximum (r = 12), is 264 as measured by an observer at inﬁnity.

111 Figure 3.15 Convergence results for the Fishbone and Moncrief equilibrium disk around an a/M = 0.95 black hole.

112 Figure 3.16 Density ﬁeld, for a magnetized torus around a Kerr black hole with a/M = 0.5 at t = 0 (left) and at t = 2000M (right). The color is mapped from the logarithm of the density; black is low and dark red is high. The resolution is 3002.

113 The integration extends for ∆t = 2000, or about 7.6 orbital periods at the pressure maximum. Figure 3.16 shows the initial and final density states projected on the R = r sin(θ), Z = r cos(θ) plane. Color represents log(ρ0). The coordinate parameter h, which concentrates zones toward the midplane, is set to 0.2. The torus atmosphere is set to the floor values (see above), and the MC limiter is used. The numerical resolution is 3002. The flux of mass, energy, and angular momentum through the inner boundary are described in Figure 3.17. Initially the fluxes are small because the initial conditions are near an (unstable) equilibrium. The magnetorotational instability (Balbus and Hawley, 1991) e-folds for just over an orbital period, after which the magnetic field has reached sufficient strength to distort the original torus and drop material into the black hole. Later, the torus is turbulent and accretion occurs at a more or less steady rate.

3.7 Conclusion

Like all hydrodynamics codes, HARM has failure modes. We will discuss one that is likely to 2 2 be relevant to future astrophysical simulations. When B /ρ0 À 1 and B À u, the magnetic energy is the dominant term in the total energy equation. Because the fields are evolved separately, truncation error in the field evolution can lead to large fractional errors in the velocity and internal energy. An example of this was discussed in §3.5.6, where the magnetized Bondi flow test fails for 2 large values of B /ρ0. Another example can be found in the strong cylindrical explosion problem of Komissarov (1999), where an overpressured region embedded in a uniform magnetic field produces a relativistic blast wave. HARM fails on the strong-field version of this problem unless we turn the Courant number down to 0.1, use the minmod limiter, and sharply increase the accuracy parameter used in the 2 4 P(U) inverter. This is a particularly difficult problem, with B /ρ0 as large as 10 . The problems caused by magnetically dominated regions appears to be generic to conservative relativistic MHD schemes, where small errors in magnetic energy density lead to fractionally large errors in other components of the total energy. At present this is unavoidable, and has motivated the development of schemes for the evolution of the electromagnetic field in the force-free limit (Komissarov, 2002c). Finally, to sum up: we have described and tested a code that evolves the equations of general relativistic magnetohydrodynamics. This code, together with the code described in a companion paper by De Villiers and Hawley (2002), are the first that stably evolve a relativistic plasma in a Kerr spacetime for many light crossing times. The advent of practical, stable GRMHD codes opens the door for the study of many problems in the theory of RMRs. For example, it may be possible to directly evaluate the importance of magnetic energy extraction from rotating black holes and the importance of black hole spin in determining jet parameters. It may also be possible to couple these schemes to numerical relativity codes and use them to study dynamical spacetimes with electromagnetic sources.

114 Figure 3.17 Evolution of the rest-mass accretion rate (top), the speciﬁc energy of the accreted matter (middle), and the speciﬁc angular momentum of the accreted matter (bottom) for a black hole with a/M = 0.5.

115 A Measurement of the 4 Electromagnetic Luminosity of a Kerr Black Hole

4.1 Summary of Chapter

Some active galactic nuclei, microquasars, and gamma ray bursts may be powered by the electromagnetic braking of a rapidly rotating black hole. We investigate this possibility via axisymmetric numerical simulations of a black hole surrounded by a magnetized plasma. The plasma is described by the equations of general relativistic magnetohydrodynamics, and the effects of radiation are neglected. The evolution is followed for 2000GM/c3, and the computational domain extends from inside the event horizon to typically 40GM/c2. We compare our results to two analytic steady state models, including the force-free magnetosphere of Blandford & Znajek. Along the way we present a self-contained rederivation of the Blandford-Znajek model in Kerr-Schild (horizon penetrating) coordinates. We find that (1) low density polar regions of the numerical models agree well with the Blandford-Znajek model; (2) many of our models have an outward Poynting flux on the horizon in the Kerr-Schild frame; (3) none of our models have a net outward energy flux on the horizon; and (4) one of our models, in which the initial disk has net magnetic flux, shows a net outward angular momentum flux on the horizon. We conclude with a discussion of the limitations of our model, astrophysical implications, and problems to be addressed by future numerical experiments.1

4.2 Introduction

A black hole of mass M and angular momentum J = aGM/c, 0 ≤ a/M < 1 has a free energy associated with its angular momentum (or “spin”). This energy can, in principle, be tapped by manipulating particle orbits so that negative energy particles are accreted (Penrose, 1969). Spin energy can also be tapped by superradiant scattering of vacuum electromagnetic waves (Press and Teukolsky, 1972), gravity waves (Hawking and Hartle, 1972; Teukolsky and Press, 1974), or magnetohydrodynamic (MHD) waves (Uchida, 1997). It can also be tapped through the action of force-free electromagnetic ﬁelds (Blandford and Znajek, 1977). The Blandford-Znajek (BZ) eﬀect– broadly used here to mean the extraction of energy from rotating holes via a magnetized plasma– appears to be the most astrophysically plausible exploita-

1Published in ApJ 20 August 2004 issue. Reproduction for this thesis is authorized by the copyright holder.

116 tion of black hole spin energy. Relativistic jets in active galactic nuclei, galactic microquasars, and gamma-ray bursts (GRBs) may well be powered by the BZ eﬀect. Despite some hints (see, e.g., Wilms et al. 2001b, Miller et al. 2002, Maraschi and Tavecchio 2003) and the general consistency of this idea with the data, however, there is no direct observational evidence for black hole energy extraction. In this paper we take an experimental approach and study the BZ eﬀect through direct numerical simulation of a magnetized plasma accreting onto a black hole.

The energy stored in black hole spin is potentially large. If Mirr is the “irreducible mass” of the black hole where, in units such that G = c = 1,

1 M 2 = Mr , (4.1) irr 2 + p 2 and r+ = M(1 + 1 − (a/M) ) is the horizon radius, then the free energy is µ ¶ 61 M Espin = M − Mirr < 5.3 × 10 8 erg. (4.2) 10 M¯ or ≈ 30% of the gravitational mass of a maximally rotating hole. This corresponds to a luminosity 10 8 of . 4 × 10 (M/10 M¯)L¯ if released over a Hubble time. Estimates suggest that black hole accretion is surprisingly efficient, in the sense that the ratio of quasar radiative energy density to supermassive black hole mass density is ∼ 0.2 (Yu and Tremaine, 2002; Elvis et al., 2002). During the accretion process some mass-energy is radiated away and the rest is incorporated into the black hole. Through electromagnetic spindown this energy gets a second chance to escape. A combination of efficient thin disk accretion (in which all radiation is somehow permitted to escape) followed by the Penrose process can in principle extract up to √ (1 − 1/ 6)c2 = 0.59c2 per gram of accreted rest-mass. In practice, of course, much less energy is likely to be available. One goal of our investigation is to discover how much less. Part of the answer may lie with the calculations already described in Gammie et al. (2004): if black hole spins are limited by the equilibrium value found there (a/M ≈ 0.92) then the nominal thin disk efficiency of the accretion phase is about ≈ 17%, much less than the 42% expected at a/M = 1. In this paper we consider the self-consistent evolution of a weakly magnetized torus surrounding a rotating black hole. The evolution is carried out numerically in the axisymmetric ideal MHD approximation. As the evolution progresses the computational domain develops matter dominated regions near the equator and electromagnetic field dominated regions near the poles. To fix expectations for the structure of these regions we review two analytic models for the interaction of a magnetized plasma with a black hole in § 4.3. Along the way we develop the relevant notation and coordinate systems. In § 4.4 we describe our numerical model and give a summary of numerical results for a high resolution fiducial model. In § 4.5 we consider the dependence of our results on model parameters. A discussion and summary may be found in § 4.6. From here on we adopt units such that GM = c = 1. Table 4.1 gives a list of commonly used symbols.

117 Table 4.1. Commonly used symbols

Symbol Fiducial Value Description

Model Parameters 2 a 0.938 black hole spin (J/M ) √ 2 r+ 1.347 radius of the event horizon (r+ = 1 + 1 − a ) risco 2.044 radius of the ISCO (innermost stable circular orbit) redge 6 radius of inner edge of torus rmax 12 radius of the pressure maximum ΩH ≈ 0.3477 spin frequency of zero angular momentum observer at r+ Rin 0.98r+ inner radial grid location Rout 40 outer radial grid location β 100 ratio of gas to magnetic pressure (initially pgas,max ) pmag,max γ 4/3 pgas = (γ − 1)u

Diagnostics M˙ 0 see sections 4.3.2 & 4.4 rest-mass flux into the black hole E˙ see sections 4.3.2 & 4.4 energy flux into the black hole E˙ (EM) see sections 4.3.2 & 4.4 electromagnetic energy flux E˙ (MA) see sections 4.3.2 & 4.4 matter energy flux L˙ see sections 4.3.2 & 4.4 angular momentum flux into the black hole L˙ (EM) see sections 4.3.2 & 4.4 electromagnetic angular momentum flux L˙ (MA) see sections 4.3.2 & 4.4 matter angular momentum flux (EM) L˜ see sections 4.4.1 & 4.5 L˜ = E˙ /(−²M˙ 0); ² = 1 − E/˙ M˙ 0

Variables b2/2 see section 4.4.3 electromagnetic energy density in the fluid frame ∗ it Br,Bθ,Bφ see section 4.3.2 magnetic field components. Bi = F Aφ see section 4.4 azimuthal component of electromagnetic vector potential v˜r see section 4.4.1 asymptotic radial velocity (i.e. vr at r = ∞) ω see sections 4.4.2 & 4.4.3 spin frequency of electromagnetic field Ω see section 4.4.3 spin frequency of fluid (Ω = uφ/ut)

118 4.3 Review of Analytic Models

In this section we review two quasi-analytic, steady state models for the interaction of a black hole with the surrounding plasma. The purpose of this review is to introduce our coordinate system and notation and to describe the models in a form suitable for later comparison with numerical results. Along the way, we give a self-contained derivation of the BZ eﬀect in Kerr-Schild (horizon penetrating) coordinates. To the extent that the analytic and numerical models agree, the comparison also builds conﬁdence in the numerical models.

4.3.1 Coordinates

Before proceeding it is useful to deﬁne three coordinate bases for the Kerr metric. Boyer-Lindquist (BL) coordinates. These are the most familiar coordinates for the Kerr metric. In BL coordinates t, r, θ, φ µ ¶ 2 r Σ A sin2 θ 4 a r sin2 θ ds2 = − 1 − dt2 + dr2 + Σ dθ2 + dφ2 − dφ dt (4.3) Σ ∆ Σ Σ where Σ ≡ r2 + a2 cos2 θ, ∆ ≡ r2 − 2r + a2 and A ≡ (r2 + a2)2 − a2∆ sin2 θ. The determinant of the 2 2 metric g ≡ Det(gµν) = −Σ sin θ. In BL coordinates the metric is singular on the event horizon at r = r+ where ∆ = 0. Kerr-Schild (KS) coordinates. The Kerr-Schild coordinates t, r, θ, φ are regular on the horizon. They are closely related to BL coordinates: r[KS] = r[BL] and θ[KS] = θ[BL]. The line element is µ ¶ µ ¶ µ ¶ 2 r 4 r 2 r ds2 = − 1 − dt2 + dr dt + 1 + dr2 + Σ dθ2 Σ Σ Σ µ µ ¶ ¶ 2 r +sin2 θ Σ + a2 1 + sin2 θ dφ2 Σ µ ¶ µ ¶ 4 a r sin2 θ 2 r − dφ dt − 2 a 1 + sin2 θ dφ dr, (4.4) Σ Σ and g = −Σ2 sin2 θ. The transformation matrix from BL to KS is

∂t[KS] 2r = , (4.5) ∂r[BL] ∆ and ∂φ[KS] a = ; (4.6) ∂r[BL] ∆ all other off-diagonal components are 0 and all diagonal components are 1. The inverse transformation matrix is identical, with the signs of the off-diagonal components reversed. Modified Kerr-Schild (MKS) coordinates. Our numerical integrations are carried out in a mod-

119 iﬁed KS coordinates x0, x1, x2, x3, where x0 = t[KS], x3 = φ[KS], and

r = ex1 , (4.7)

1 θ = πx + (1 − h) sin(2πx ). (4.8) 2 2 2 Here h is an adjustable parameter that can be used to concentrate grid zones toward the equator as h is decreased from 1 to 0. The transformation matrix from KS to MKS is diagonal and trivially constructed from the explicit expressions for r and θ in equations 4.7 and 4.8.

4.3.2 Governing Equations

For a magnetized plasma the equations of motion are

µν ¡ µν µν ¢ T ;ν = TMA + TEM ;ν = 0. (4.9) where T µν is the stress-energy tensor, which can be split into a matter (MA) and electromagnetic (EM) part. In the ﬂuid approximation

µν µ ν µν TMA = (ρ0 + ε + p)u u + pg , (4.10)

µ where ρ0 ≡ rest-mass density, ε ≡ internal energy, p ≡ pressure, u is the ﬂuid four-velocity, and we assume throughout an ideal gas equation of state

p = (γ − 1)ε. (4.11)

In terms of F µν, the Faraday (or electromagnetic ﬁeld) tensor,

1 T µν = F µγF ν − gµνF αβF , (4.12) EM γ 4 αβ √ where we have absorbed a factor of 4π into the deﬁnition of F µν. We assume that particle number is conserved: µ (ρ0u );µ = 0. (4.13)

The evolution of the electromagnetic ﬁeld is given by the space components of the source-free Maxwell equations ∗ µν F ;ν = 0, (4.14) where ∗F is the dual of the Faraday, and the time component gives the no-monopoles constraint. The inhomogeneous Maxwell equations

µ µν J = F ;ν (4.15)

120 define the current density J µ but are otherwise not required here. We adopt the ideal MHD approximation, where µν uµF = 0, (4.16) which implies that the electric field vanishes in the rest frame of the fluid. In our numerical models the fundamental (or “primitive”) variables that describe the state of i ∗ it the plasma are ρ0, ε, B ≡ F , plus three variables which describe the motion of the plasma. In Gammie et al. (2003) we used the plasma three-velocity. Here we use

γβi u˜i ≡ ui + , (4.17) α p 2 2 i j i ti 2 2 tt where γ ≡ 1 + q , q ≡ giju˜ u˜ , β ≡ g α is the shift, and α = −1/g is the lapse. We made this change to improve numerical stability. Because the three velocity components have a finite range, truncation error can move the plasma velocity outside the light cone. The variablesu ˜i have the important property that they range over −∞ to ∞, and this makes it impossible for the plasma to step outside the light cone. To write the electromagnetic quantities in terms of the primitive variables, define the four-vector µ t i µ i i i t t b with b ≡ giµB u and b ≡ (B + u b )/u . With some manipulation one finds

b2 T µν = b2uµuν + gµν − bµbν, (4.18) EM 2 and ∗ µν F = bµuν − bνuµ. (4.19)

The no-monopoles constraint becomes

√ i ( −gB ),i = 0. (4.20)

A more complete account of the relativistic MHD equations can be found in Gammie et al. (2003) or Anile (1989).

4.3.3 Blandford-Znajek Model

BZ studied a rotating black hole surrounded by a stationary, axisymmetric, force-free, magnetized plasma. They obtain an expression for the energy flux through the event horizon and, given a solution for the field geometry when a = 0, find a perturbative solution when a ¿ 1. Here we present a self-contained rederivation, which will be compared to numerical models in Section 4.4.2. Those not interested in the derivation may find a summary set of equations in 4.3.3. A comparison of the analytic BZ model to our numerical models can be found in Section 4.4.2. µν We follow an approach that differs slightly from BZ. We solve T ;ν = 0 directly rather than µν using JµF = 0, which is equivalent in the force-free approximation. Also, because our solution

121 is developed in KS coordinates, which are regular on the horizon, we obtain the BZ solution by applying a regularity condition on the horizon and at large radius, rather than the physically equivalent approach of applying a regularity condition on the horizon in the Carter tetrad (Znajek, 1977) and then applying the result as a boundary condition in BL coordinates. Finally, if we assume separability of the solution then we do not need to require that the solution match the ﬂat-space force-free solution of Michel (1973).

Derivation in KS coordinates

Over the poles of the black hole it is reasonable to expect that the density is low, but the ﬁeld strength is comparable to that at the equator. In the limit that

2 b À ρ0 + ε + p, (4.21) where b2 is the ﬁeld strength in the ﬂuid frame, one may assume that the matter contribution to the stress energy tensor can be ignored and

µν µν T ≈ TEM. (4.22)

This is the force-free limit. µ The ideal MHD condition u Fµν = 0 implies that the electric field vanishes in the rest frame of ∗ µν the fluid. Therefore the invariant E · B = 0, or in covariant form F Fµν = 0. The electromagnetic field is then said to be degenerate. In the force-free limit the governing equations are then

µν TEM;ν = 0 (4.23) and ∗ µν F ;ν = 0. (4.24)

As BZ point out, the same basic set of equations can be derived without assuming that the plasma obeys the fluid equations. We now specialize to KS coordinates and write down the Faraday tensor in terms of a vector potential Aµ, Fµν = Aν,µ −Aµ,ν. We assume that the field is axisymmetric (∂φ → 0) and stationary ∗ µν (∂t → 0). Evaluating the condition F Fµν = 0, one finds

Aφ,θAt,r − At,θAφ,r = 0. (4.25)

It follows that one may write A A t,θ = t,r ≡ −ω(r, θ) (4.26) Aφ,θ Aφ,r where ω(r, θ) is an as-yet-unspeciﬁed function. It is usually interpreted as the “rotation frequency”

122 of the electromagnetic ﬁeld (this is Ferraro’s law of isorotation; see e.g. Frank et al. 2002, §9.7 in a φ nonrelativistic context). This yields Fµν in terms of the free functions ω, Aφ, and B , the toroidal magnetic ﬁeld:

Ftr = −Frt = ωAφ,r (4.27)

Ftθ = −Fθt = ωAφ,θ (4.28)

√ φ Frθ = −Fθr = −gB (4.29)

Frφ = −Fφr = Aφ,r (4.30)

Fθφ = −Fφθ = Aφ,θ (4.31) with all other components zero. Written in this form, the electromagnetic field automatically √ r √ θ satisfies the source-free Maxwell equations. Notice that Aφ,θ = −gB and Aφ,r = − −gB . We want to evaluate the radial energy flux Z π √ E˙ ≡ 2π dθ −gFE (4.32) 0

r (MA) (EM) where FE ≡ −Tt . This can be subdivided into a matter FE and electromagnetic FE part, although in the force-free limit the matter part vanishes. Similar expressions can be written for the angular momentum flux L˙ and angular momentum flux density FL, and for the mass flux M˙ 0 and mass flux density FM . In the limit of a steady flow these conserved quantities correspond to the radial flux measured by a stationary observer at large distance from the black hole. Using the definition of the electromagnetic stress-energy tensor (4.12) and the relations (4.27)- (4.31), it is a straightforward exercise to evaluate

a F (EM) = −2(Br)2ωr(ω − ) sin2 θ − BrBφω∆ sin2 θ. (4.33) E 2r

The radial angular momentum ﬂux density is F (EM) = F (EM)/ω. One can verify by direct trans- L E √ 2 formation that FE[KS] = FE[BL] and FL[KS] = FL[BL]. On the horizon r = r+ = 1 + 1 − a and ∆ = 0, so the horizon energy ﬂux is

(EM) r 2 2 FE |r=r+ = 2(B ) ωr+(ΩH − ω) sin θ (4.34) where ΩH ≡ a/(2r+) is the rotation frequency of the black hole (see MTW §33.4). This result, r 2 which is identical to BZ’s result, implies that if 0 < ω < ΩH and (B ) > 0 then there is an outward directed energy flux at the horizon. Because the flux was evaluated in KS coordinates the horizon did not require special treatment as in Znajek (1977). (EM) φ To finish evaluating E˙ we need to find Aφ, ω, and B . This requires solving the equations µ of motion (4.9). They can be evaluated directly or in the reduced form J Fµν = 0 (as in BZ), in which case one must also evaluate the currents using Maxwell’s equations. In either form this is a

123 difficult, nonlinear problem which probably cannot be solved in any general way. To make progress, BZ find solutions to the equations of motion when a = 0, then perturb them by allowing the black hole to spin slowly with a ¿ 1. If we assume that the initial field has ω = Bφ = 0, then we may expand the vector potential

(0) 2 (2) 4 Aφ = Aφ (r, θ) + a Aφ (r, θ) + O(a ), (4.35)

(1) where Aφ = 0 by symmetry (Aφ should be even in a). The ﬁeld rotation frequency vanishes in the unperturbed solution, and ω(2) = 0 because ω should be odd in a, so

ω = aω(1)(r, θ) + O(a3) (4.36) and similarly for the toroidal ﬁeld

Bφ = aBφ(1)(r, θ) + O(a3). (4.37)

(2) (1) φ(1) (0) We are now in a position to find the free functions Aφ , ω , and B , given an initial field Aφ that satisfies the basic equations when a = 0. (0) BZ consider two forms for Aφ : a monopole field and a paraboloidal field. Here we review only (0) the (possibly split) monopole, where Aφ = −C cos θ and C is an arbitrary constant. One may obtain the perturbed solution by making the following sequence of deductions. (1) The t and φ components of equation (4.9), expanded to lowest nontrivial order in a, require (EM) (EM) that FL and FE be independent of radius. Therefore they are functions of θ alone. Since

(EM) (1) (EM) FE = aω FL , (4.38) we conclude that ω(1) is a function of θ alone. (2) The r component of equation (4.9), together with the requirement that Bφ(1) be finite on the horizon (all components of Fµν are well-behaved on the horizon in KS coordinates), yields a single nontrivial solution: µ ¶ C 2 Bφ(1) = − 1 − 4ω(1) + (4.39) 4r2 r This solution is well behaved at the horizon and at large radius as long as ω(1) is finite on the horizon and grows less rapidly than r2 at large r. (3) The θ component of equation (4.9), which is the trans-field force balance equation, can now (2) (1) (2) be reduced to an equation involving Aφ and ω . If we require that Aφ = Cf(r)g(θ), then one (1) (1) 2 may deduce that (a) ∂θω = 0, i.e. ω = const.; (b) g(θ) = cos θ sin θ. Then f(r) must satisfy Ã ! 2f 0 6f r + 2 (ω(1) − 1/8)(r2 + 2r + 4) f 00 + − + − = 0 (4.40) r(r − 2) r(r − 2) r3(r − 2) r(r − 2)

124 which is equivalent to BZ’s equation (6.7). This has an exact solution with two constants of integration. One of the constants of integration is set by requiring that the solution be ﬁnite on the horizon. Part of the solution can be regularized at large r by ﬁxing the other constant of integration, but the remaining divergence can only be zeroed by setting ω(1) = 1/8; this is already suggested by the form of the preceding equation. For r > 2 the regular solution is µ ¶ 2 2 r r2(2r − 3) 1 + 3r − 6r2 r 11 1 r r2 f(r) = Li ( ) − ln(1 − ) ln + ln + + + − , (4.41) 2 r r 2 8 12 2 72 3r 2 2 where Li2 is the second polylogarithm function: Z 1 ln(1 − tx) Li2(x) = − dt . (4.42) 0 t

For r < 2 the solution is given by the real part of equation (4.41). In the limit of large r µ ¶ 1 ln r f(r) ∼ + O , (4.43) 4r r2 which agrees with BZ. (2) To sum up, using only the assumption of separability of Aφ and the regularity of physical quantities in Kerr-Schild coordinates on the horizon and at inﬁnity, we ﬁnd

1 ω(1) = (4.44) 8 C 4 Bφ(1) = − (1 + ) (4.45) 8r2 r and (2) 2 Aφ = Cf(r) cos θ sin θ. (4.46) with f(r) given by equation (4.41). Our solution is identical to BZ’s after transforming to Boyer- φ Lindquist coordinates and transforming from our B to BZ’s BT , although BZ’s expression for f(r) contains some unclosed parentheses.

BZ Derivation Summary

In Kerr-Schild coordinates, then, the magnetic ﬁeld components are

C C ¡ ¢ Br = + a2 −2 cos θ + r2(1 + 3 cos 2θ)f(r) , (4.47) r2 2r4 C Bθ = −a2 cos θ sin θf 0, (4.48) r2

125 both accurate through second order in a, and

C 4 Bφ = −a (1 + ), (4.49) 8r2 r accurate through ﬁrst order in a. In Boyer-Lindquist coordinates,

Br[BL] = Br[KS], (4.50)

Bθ[BL] = Bθ[KS], (4.51) (a − 2rω) Bφ[BL] = Bφ[KS] − Br[KS] , (4.52) ∆ and BZ’s toroidal ﬁeld 2 φ BT = ∆ sin θB [BL] (4.53)

(which is different from BZ’s Bφ). There has been some concern about causality in the application of the force-free approximation (see, e.g., Punsly 2003, see also Komissarov 2002a, 2004a). The MHD equations are hyperbolic and causal (as are the equations of force-free electrodynamics). Below we show that a numerical 2 evolution of the MHD equations agrees well with the BZ solution in those regions where b /ρ0 À 1. This is either a remarkable coincidence or else the BZ solution is an accurate representation of the strong-field limit of ideal MHD. For comparison with computational models, the most relevant aspects of the BZ theory are that: (1) the field is force-free; (2) the field rotation frequency ω = a/8+O(a3) in the monopole geometry case and ω = a/8 + O(a3) at the poles (θ = 0, π/2) in the paraboloidal field case considered by BZ;2 (3) if the field geometry is nearly monopolar and a is small enough that the expansion to lowest order in a is accurate, then Br(θ) is given by equation (4.47); and (4) if the field geometry 2 is monopolar and a is small, then the energy flux density FE ∝ sin θ on the horizon. We compare this analytic BZ model to our numerical models in Section 4.4.2.

4.3.4 Equatorial MHD Inﬂow

Gammie (1999) considered a stationary, axisymmetric MHD inflow in the “plunging region”, between the innermost stable circular orbit (ISCO) and the event horizon. The flow was assumed to be cold (zero pressure), nearly equatorial, and to proceed along lines of constant latitude θ. The latter assumption ignores the requirement of cross-field force-balance. This model is analogous to the Weber and Davis (1967) model for the solar wind, only turned inside out so that the wind flows from the disk into the black hole. The model builds on earlier work by Takahashi et al. (1990), Phinney (1983), and Camenzind (1986). The analytic model derived here will be used to compare to numerical models in Section 4.4.3. 2According to the numerical results of Komissarov (2001) and the argument of MacDonald and Thorne (1982), ω adjusts to ≈ ΩH /2 hole even at large a.

126 The MHD inﬂow model is stationary (∂t → 0), axisymmetric (∂φ → 0) and nearly equatorial

(θ ≈ π/2) so ∂θ → 0 by symmetry. In addition flow proceeds along lines of constant θ. As a result the model is one dimensional with a single independent variable r. The nontrivial dependent variables are the radial and azimuthal four-velocity ur and uφ, the radial and azimuthal magnetic r φ field B and B , and the rest-mass density ρ0. With these assumptions the equations of general relativistic MHD can be integrated completely. The constancy of energy flux √ r − −gT t = const., (4.54) and angular momentum flux √ r −gT φ = const., (4.55)

µ follow from T ν;µ = 0. The source-free Maxwell equations imply √ −gBr = const., (4.56) which expresses the constraint ∇ · B = 0, and the relativistic “isorotation law”,

√ ∗ rφ √ −g F = −g(urbφ − uφbr) = const. (4.57) where bµ is the magnetic ﬁeld four-vector (deﬁned above). Finally, conservation of particle number implies √ r −gρ0u = const. (4.58)

These five constants yield five constraints on the five nontrivial fundamental variables ur, uφ, Br, φ B , and ρ0. Given the constants, and using the constitutive relations that relate the constants and fundamental variables, one can solve the resulting set of nonlinear equations for the fundamental variables at each radius. The next step is to determine the constants. The radial magnetic flux and the rest-mass flux are determined by conditions in the disk and can be left as free parameters. The remaining three degrees of freedom are fixed by imposing boundary conditions. Gammie (1999) imposed the following conditions: (1) the flow is regular at the fast point (the flow is automatically regular at the Alfvénpoint– see Phinney (1983) for a discussion– and the slow point is absent because the flow is cold) ; and (2,3) the four-velocity components ur and uφ match onto a cold disk at the ISCO. Energy can be extracted from the black hole if the Alfvénpoint lies inside the ergosphere (Takahashi et al., 1990). Gammie (1999) calculated E˙ and L˙ as a function of a and Br and showed that for even modest magnetic field strength these were modified from the values anticipated in classical thin disk theory. The implications of these modified fluxes for the structure– particularly the surface brightness– of a thin disk were explored by Agol and Krolik (2000). For comparison with numerical models, the key predictions of the inflow model are: (1) the

127 constancy of the conserved quantities with radius; (2) matching of the flow velocity to circular orbits at the ISCO; (3) modification of the angular momentum and energy fluxes from their thin disk values; and (4) the run of all the fluid variables with radius in the plunging region.

4.4 Numerical Experiments

All our experiments evolve a weakly magnetized torus around a Kerr black hole in axisymmetry. The focus of our numerical investigation is to study a high resolution model (4.4.1), compare with the BZ model (4.4.2), and compare to the Gammie inflow model (4.4.3). In § 4.5 we investigate how various parameters affect the results. Any dimensional quantity can be recovered from a numerical quantity since we set GM = c = ρ? = 1, where ρ? is some rest mass density set to unity in a simulation. The initial conditions consist of an equilibrium torus (Fishbone and Moncrief 1976 ; Abramowicz et al. 1978) which is a “donut” of plasma with a black hole at the center. The donut is supported against gravity by centrifugal and pressure forces, and is embedded in a vacuum. We consider a particular instance of the Fishbone and Moncrief (1976) solutions, which are defined by the t condition u uφ = const. We normalize the peak density ρ0,max = ρ? to 1 and fix the inner edge of 3 the torus at redge = 6. We also set γ = 4/3. Absent a magnetic field, the initial torus is a stable equilibrium.4 Into the initial torus we introduce a purely poloidal magnetic field. The field can be described using a vector potential with a single nonzero component Aφ ∝ MAX(ρ0/ρ0,max − 0.2, 0) The field is therefore restricted to regions with ρ0/ρ0,max > 0.2. The field is normalized so that the minimum ratio of gas to magnetic pressure is 100. The equilibrium is therefore only weakly perturbed by the magnetic field. It is, however, no longer stable (Balbus and Hawley, 1991; Gammie, 2004). Our numerical scheme is HARM (Gammie et al., 2003), a conservative, shock-capturing scheme for evolving the equations of general relativistic MHD. HARM uses constrained transport to maintain a divergence-free magnetic field (Evans and Hawley, 1988; Toth, 2000). The inversion of conserved quantities to primitive variables is performed by solving a single non-linear equation (Del Zanna and Bucciantini, 2002) or by a slower but more robust multi-dimensional Newton-Raphson method. Unless otherwise stated we use modified Kerr-Schild (MKS) coordinates with h = 0.3.

The computational domain is axisymmetric, with a grid that typically extends from rin = 0.98r+ to rout = 40, and from θ = 0 to θ = π/2. HARM is unable to evolve a vacuum, so we are forced to introduce “floors” on the density and internal energy. When the density or internal energy drop below these values they are immediately reset. This sacrifices exact conservation of energy, particle number, and angular momentum, although it is reasonable to assume that when the floors are small enough the true solution is −4 −3/2 −6 −5/2 recovered. The floors are position dependent, with ρ0,min = 10 r and εmin = 10 r . We

3We have run a limited number of γ = 5/3 models and ﬁnd results essentially identical to those discussed below. 4In axisymmetry. The torus is unstable to global nonaxisymmetric modes (Papaloizou and Pringle, 1983).

128 discuss the effect of varying the floor in Section 4.5.3. At the outer boundary we use an “outflow” boundary condition. This means we project all primitive variables into the ghost zones while forbidding inflow. The inner boundary condition is identical except that, because the boundary is inside the event horizon, we never need to worry about backflow into the computational domain. At the poles we use a reflection boundary condition where we impose appropriate symmetries for each variable across the axis.

4.4.1 Fiducial Model

First consider the evolution of a high resolution fiducial model with a = 0.938. This is close to the spin equilibrium value (where d(a/M)/dt = 0) found by Gammie et al. (2004) for a series of similar Fishbone-Moncrief tori. t The fiducial model has u uφ = 4.281, the pressure maximum is located at rmax = 12, the inner edge at (r, θ) = (6, π/2), and the outer edge at (r, θ) = (42, π/2). The orbital period at the pressure 3/2 maximum 2π(a + rmax) ' 267, as measured by an observer at infinity. The numerical resolution of the fiducial model is 4562. The zones are equally spaced in modified

Kerr-Schild coordinates x1 and x2, with coordinate parameters h = 0.3. Small perturbations are introduced in the velocity field, and the model is run for ∆t = 2000, or about 7.6 orbital periods at the pressure maximum. The initial state is Balbus-Hawley unstable. The inner edge of the disk quickly makes a transition to turbulence. Transport of angular momentum by the magnetic field causes material to plunge from the inner edge of the disk into the black hole. The turbulent region gradually expands outward to involve the entire disk. The disk relaxes toward a “Keplerian” velocity profile, meaning that the orbital frequency along the equator is close to the circular orbit frequency. The disk enters a long, quasi-steady phase in which the accretion rates of rest-mass, angular momentum, and energy onto the black hole fluctuate around a well-defined mean. Figure 4.1 shows the initial and final density states projected on the (R, z = r sin θ, r cos θ)- −7 plane. Color represents log(ρ0). The initial density maximum is 1 and the minimum is ≈ 4 × 10 . The final state contains shocks driven by the interaction with the magnetic field, outflows near the surface of the disk, and an evacuated “funnel” region near the poles. The left panel in Figure 4.2 indicates the relative densities of internal, magnetic, and rest- mass energy. The magenta and cyan contours show the ratio of the average pressure to average magnetic pressure, β ≡ 2¯p/b¯2. The overbar indicates an average taken over 1000 < t < 2000 and over both hemispheres. The cyan contour indicates β = 3 and encircles most of the high density, approximately Keplerian disk. The magenta contour indicates β = 1. The red contour indicates ¯2 where b /ρ¯0 = 1. Between the pole and this contour the magnetic energy density exceeds the internal and rest-mass energy density. The black contour surrounds a region, extending to large radius, where −ut > 1 and the flow is directed outward (at large radius −ut asymptotes to the Lorentz factor). That is, the particle energy-at-infinity is larger than the rest-mass density: so

129 Figure 4.1 Initial (left) and ﬁnal (right) distribution of log ρ0 in the ﬁducial model on the r sin θ − −7 r cos θ plane. At t = 0 black corresponds to ρ0 ≈ 4 × 10 and dark red corresponds to ρ0 = 1. For −7 t = 2000, black corresponds to ρ0 ≈ 4 × 10 and dark red corresponds to ρ0 = 0.57. The black half circle at the left edge is the black hole.

130 FUNNEL

CORONA

WIND

BLACKHOLE

DISK

PLUNGING REGION

2 Figure 4.2 (a) The distribution of β, b /ρ0, and ut in the fiducial run, based on time and hemi- spherically averaged data. Starting from the axis and moving toward the equator: (1) ut = −1 2 contour shown as a solid black line; (2) b /ρ0 = 1 contour shown as a red line; (3) β = 1 contour shown as a magenta line that nearly matches part of the ut = −1 contour line; and (4) β = 3 contour is shown as cyan line. (b) Motivated by the left panel, the right panel indicates the location of the five main subregions of the black hole magnetosphere. They are (1) the disk: a matter 2 dominated region where b /ρ0 ¿ 1; (2) the funnel: a magnetically dominated region around the 2 poles where b /ρ0 À 1 where the magnetic field is collimated and twists around and up the axis into an outflow; (3) the corona: a region in the relatively low density upper layers of the disk with weak time-averaged poloidal field; (4) the plunging region; and (5) the wind, which straddles the corona-funnel boundary. See Section 4.4.1 for a discussion.

131 the fluid is in a sense, unbound. We use the value of ut to estimate the radial component of the 3-velocity at infinity (˜vr), which is independent of the coordinate system. The right panel in Figure 4.2 defines some useful terminology inspired by the left panel, following De Villiers and Hawley (2003a) and Hirose et al. (2003). Moving from the axis to the equator, the 2 “funnel” is the nearly evacuated, strongly magnetized region (b À ρ0 + ε + p), that develops over the poles. The “wind” consists of a cone of material near the edge of the funnel that is flowing outward with an asymptotic radial velocity ofv ˜r ∼ 0.75c. Near the outer edge of our computational domain the wind becomes marginally superfast. The “corona” lies between the funnel and the disk and has b2/2 ∼ p except in strongly magnetized filaments. In the “disk” b2/2 < p and the plasma follows nearly Keplerian orbits. Finally, the “plunging” region, which lies between the disk and the event horizon, contains accreting material moving on magnetic field and pressure modified geodesics. Figure 4.3 shows the evolution of the poloidal magnetic field. The panels show contours of constant Aφ, so the density of contours is directly related to the poloidal field strength, and the contours follow magnetic field lines. The contours are projected on the (R = r sin θ, z = r cos θ)- plane, and show the initial and final state. The initial field is confined to a region much smaller than the torus as a whole because field is introduced only in those portions of the disk that have

ρ0 > 0.2ρ0,max. Notice that by the end of the simulation the field has mixed in to the funnel region and has a regular geometry there. In the disk and at the surface of the disk the field is curved on the scale of the disk scale height. The field strengths and geometries we see are consistent with Hirose et al. (2003). This includes the absence of disk to disk field loops, and that the funnel field collimates instead of connecting back into the disk (thus providing a means for the outflow to escape to large radii).

Figure 4.4 shows contours of time and hemisphere averaged Aφ. The time averaged field is even more regular in the funnel than the snapshot in Figure 4.3. Time averaging tends to sharply reduce the field strength in the corona and disk because the field fluctuates in magnitude and direction there.

Figure 4.5 shows the accretion rate of rest-mass (M˙ 0), energy per unit rest-mass (E/˙ M˙ 0), and angular momentum per unit rest-mass (E/˙ M˙ 0) evaluated inside the horizon at the inner boundary of the computational domain. For 500 < t < 2000 the time average values are M˙ 0 ≈ 0.35,

E/˙ M˙ 0 ≈ 0.87, and L/˙ M˙ 0 ≈ 1.46. These average values are shown as dashed lines. The dotted lines show the classical thin disk values E/˙ M˙ 0 ≈ 0.82 and L/˙ M˙ 0 ≈ 1.95 obtained by setting these ratios equal to respectively the specific energy and angular momentum of particles on the ISCO. The energy per baryon is therefore slightly above the thin disk value, but the angular momentum per baryon is significantly below the thin disk value. It may be useful to recast the energy flux in terms of a nominal “radiative efficiency”5 ² =

1 − E/˙ M˙ 0. For the fiducial run ² = 13%, which is slightly lower than the thin disk with ² = 18%. This is likely due to the high temperature of the flow. On the horizon about 20% of the energy flux

5Our evolution is nonradiative, so the true radiative eﬃciency is zero.

132 Figure 4.3 Initial (left) and final (right) distribution of Aφ. Level surfaces coincide with magnetic field lines and field line density corresponds to poloidal field strength. In the initial state field lines follow density contours if ρ0 > 0.2ρ0,max.

133 Figure 4.4 Contour plot of the time and hemispheric average of Aφ. Level surfaces coincide with magnetic field lines and field line density corresponds to poloidal field strength.

134 Figure 4.5 Evolution of rest-mass, energy, and angular momentum accretion rate for our ﬁducial run of a weakly magnetized tori around a black hole with spin a = 0.938. For 500 < t < 2000 the time average of these values is M˙ 0 ' 0.35, E/˙ M˙ 0 ' 0.87, and L/˙ M˙ 0 ' 1.46 as shown by the dashed lines. The dotted lines show the classical thin disk values (E/˙ M˙ 0 ' 0.82 and L/˙ M˙ 0 ' 1.95). See Section 4.4.1 for a discussion.

135 would vanish if we set the internal energy to zero. The corresponding zero-temperature eﬃciency

(1 + ut) would be 32%. The chief object of our study is to measure the electromagnetic luminosity of the hole. The time and hemisphere averaged electromagnetic energy flux on the horizon is shown in Figure 4.6. In the funnel region the energy flux density is outward, as predicted by the force-free model of BZ. We compute other interesting quantities by integrating over the horizon and taking a time average (for technical reasons we are using a less resolved time sampling here than used to make Figure 4.5, but the time averages have fractional differences of only 10%). We find E˙ (EM)/E˙ (MA) = −2.3%, where (EM) (MA) the energies per baryon are E˙ /M˙ 0 = −0.018 and E˙ /M˙ 0 = 0.77. It is useful to define the (EM) ratio of electromagnetic luminosity to nominal accretion luminosity L˜ = E˙ /(−²M˙ 0). We find L˜ = 16%. Thus while the electromagnetic energy flux is outward, it is a small fraction of the inward material energy flux and the BZ luminosity is small compared to the nominal accretion luminosity. The BZ luminosity from the polar regions of the horizon provides about 1/2 of the radial electromagnetic energy flux (E˙ (EM)) in the wind, while another 1/2 comes from the surface of the corona but is mostly dissipated into kinetic energy of the wind. Therefore, the BZ-effect is an important source of Poynting flux in the wind at large distances since it flows freely along highly ordered poloidal field lines connected to unbound plasma. A control calculation at a = 0 and a resolution of 2562 gives E˙ (EM)/E˙ (MA) = 0.33% and (EM) (MA) L˜ = −6.5%, where the energies per baryon are E˙ /M˙ 0 = 0.0032 and E˙ /M˙ 0 = 0.95. (EM) E˙ /M˙ 0 > 0 and L˜ < 0 are as expected, since the outward energy flux must vanish for a nonrotating hole (i.e. the BZ effect is not operating). For our sequence of models the BZ effect does not operate for a . 0.5 (see Section 4.5.1). The matter energy flux ratio may be compared to (MA) the thin disk value of E˙ /M˙ 0 = 0.94.

4.4.2 Comparison with BZ

The BZ solution was reviewed in Section 4.3.3. BZ were able to find steady force-free field solutions in the limit that a ¿ 1. Since the fiducial run has a = 0.938, we ran a special a = 0.5 model for comparison with BZ. The BZ solution was found in the force-free limit, so the first question one might ask is whether any region of the model is force-free. To measure this we recall that in the force-free limit

µν µν T ;ν = F Jν = 0. (4.59)

So when the ﬁeld is force-free the parameter ¯ ¯ ¯ µν κ ¯ ¯ F JνFµκJ ¯ ζ = ¯ µ κλ ¯ (4.60) JµJ FκλF is small compared to 1. Figure 4.7 shows the time and hemispherical averaged ζ(r, θ) from t = 1000 to t = 2000 for

136 (EM) Figure 4.6 Electromagnetic energy flux density FE (θ) on the horizon for the fiducial run, based on time and hemisphere averaged data. The mean electromagnetic energy flux is directed outward. See Section 4.4.1 for a discussion.

137 Figure 4.7 The run of the force-free parameter ζ for the a = 0.5 run; when ζ ¿ 1 the ﬁeld is approximately force-free. The parameter has been time and hemisphere averaged. The contours show (beginning from the pole and moving toward the equator) ζ = 10−3, 10−2, 10−1. The small closed contours at large radius and close to the axis have ζ = 10−2. The small closed contours from the equator to θ ∼ π/4 have ζ = 10−1. See Section 4.4.2 for a discussion.

138 Figure 4.8 Left panel: Magnetic field angular frequency on the horizon relative to black hole rotation ω(θ)/ΩH . The solid line indicates time and hemisphere averaged data from our a = 0.5 MHD integration. The middle dotted line is the prediction of the BZ model (ω/ΩH = 1/2). The dashed line (top) is the value predicted by the inflow model. Right panel: the run of field rotation frequency ω with radius along a single field line that intersects the horizon at θ = 0.2. ω is constant to within 3%, as expected for a steady flow. See sections 4.4.2 and 4.4.3 for a discussion. the a = 0.5 model. The contours show (beginning from the pole and moving toward the equator) ζ = 10−3, 10−2, 10−1. The entire funnel region has ζ < 10−2 and is therefore effectively force-free. This is true in both a time-averaged and instantaneous sense in the funnel for all our runs. This opens the possibility that the BZ solution describes the funnel.

A key feature of the BZ model is that the ﬁeld rotation frequency ω ≈ ΩH /2 for a ¿ 1 if the

field has a monopole geometry. Figure 4.8a shows the ratio ω/ΩH on the horizon. Within the force-free region, which runs from 0 < θ < 0.4 on the horizon, the average ω/ΩH ≈ 0.45. The small difference from the BZ could be due to higher order terms in the expansion in a, but Komissarov (2001) has integrated the equations of force-free electrodynamics for a monopolar field geometry and at a = 0.5 finds that ω rises from ≈ 0.495ΩH at the pole to ≈ 0.51ΩH at the equator, so this seems unlikely. The difference is more likely due to small deviations from force-free behavior (mass loading of field lines by the numerical “floor” on the density). In an axisymmetric steady state both the force-free equations and the MHD equations predict that the rotation frequency ω (and other quantities) are constant along field lines. Figure 4.8b shows the variation of ω with radius along a field line that intersects the horizon at θ = 0.33. As expected ω ≈ const., with a variation of less than 3% from maximum to minimum. r BZ’s spun-up monopole model makes definite predictions about the variation of B and FE on

139 Figure 4.9 (a) Square of radial field ((Br(θ))2) on the horizon in the a = 0.5 MHD integration, from time and hemisphere averaged data. Solid line is the field for our numerical model. The dotted line shows the Blandford and Znajek (1977) perturbed monopole solution with the field strength normalized to the numerical solution at the pole. The dashed line is the inflow solution. (b) (EM) Electromagnetic energy flux FE (θ) on the horizon in the a = 0.5 MHD integration, from time and hemisphere averaged data. The solid line shows the numerical model, the dotted line shows BZ’s spun-up monopole solution, and the dashed line shows the inflow solution. See sections 4.4.2 and 4.4.3 for a discussion.

140 the horizon. Figure 4.9a shows the variation in time and hemisphere averaged (Br)2 and compares to BZ’s monopole field calculation. The single adjustable parameter of the model normalizes the field strength. We have set this normalization by requiring that (Br)2 match at the pole. Evidently the variation matches the BZ prediction closely even well outside the force-free region at θ ≈ 1.1. Figure 4.9b shows the variation in radial energy flux on the horizon as predicted by the BZ model using the pole-normalized field. Here the match is quite close out to θ ≈ π/4. It is slightly surprising that the BZ solution does so well even in regions that are not force-free. This is likely a result of trans-field force balance and geometry controlling the distribution of field on the horizon and hence the radial energy flux. To summarize: in our low spin numerical experiment the funnel is approximately force-free within the funnel. It is approximately in a steady state and hence ω is approximately constant along field lines. Furthermore, ω, Br, and the radial electromagnetic energy flux are all in good agreement with the spun-up monopole force-free model on the horizon. We have not compared the entire funnel region with the monopole model because the field is collimated there and not well-described by the monopole solution.

4.4.3 Comparison to Inﬂow Solution

The inflow solution of Gammie (1999) considers a near-equatorial stationary MHD inflow in the plunging region, reviewed in Section 4.3.4. Here we compare the inflow models with the fiducial model. Unlike the funnel, the plunging region is rapidly fluctuating, so we expect the inflow model to match only the time-averaged data from the simulation. The inflow model has two free parameters: the field strength and the accretion rate. The field strength we match by finding the parameter that gives the best fit to the mean magnetic energy density between the ISCO and the event horizon. The rest-mass flux is chosen to agree with the time-averaged data from the simulation. The ratio of the field strength to the square root of the accretion rate is a dimensionless parameter that controls the solution; in the units of Gammie r√ (1999), where 2πρ0u −g = −1, we use Fθφ = 1.09 for the comparison model. r 2 Figure 4.10 shows a comparison of u , L/˙ M˙ 0, comoving energy densities (ρ0, b /2, and ε), and energy fluxes (E/˙ M˙ 0) in the inflow solution. The comparison data from the fiducial run has been averaged over |θ − π/2| < 0.3 and 500 < t < 2000. Each panel in the figure contains a vertical line at the ISCO. The upper left panel compares the radial component of the four-velocity (in KS and BL coordinates) in the inflow and numerical solutions. The substantial differences are due to the finite temperature of the flow; the inflow solution is cold by assumption. Radial pressure gradients in the numerical model (which are absent in the inflow solution) begin to accelerate material inward outside the ISCO, and the flow becomes supersonic near the ISCO. The upper right and lower right panels show components of the energy and angular momentum flux from the simulation and inflow solutions. The dashed horizontal line in each case shows the

141 Figure 4.10 A comparison of the time-averaged fiducial model near the equator (within θ = π/2 ± 0.3) with the inflow solution of Gammie (1999). In the right two panels the black dotted line is the thin disk value. In all cases the red vertical line is the location of the ISCO. The black line for the upper left panel is the numerical result. For the other three panels, the particle term is shown in cyan, the internal energy term is shown in magenta, and the electromagnetic term is shown in green. The blue line in each plot represents the inflow model result. Notice that the run of density with radius shows no feature at the ISCO. See the Section 4.4.3 for discussion.

142 values expected for a thin disk; the inﬂow solution is constrained to match the thin disk at the

ISCO. The cyan lines show uφ (upper panel) and ut (lower panel) from the simulation, while the blue lines show the prediction from the inflow model. The energy flux matches rather well (although notice that this is only a small fraction of the energy flux), while the angular momentum is overestimated; in the simulation the plasma has sub-Keplerian angular momentum by the time it reaches the ISCO. 2 The electromagnetic components of the per unit mass flux of angular momentum flux (b uφ/ρ0− r r 2 r r b bφ/(ρ0u )) and energy flux (b ut/ρ0 − b bt/(ρ0u )) are also shown in the upper and lower right panels of Figure 4.10 (green line ≡ simulation, blue line ≡ inflow solution). The inflow solution matches well, although it tends to overestimate the magnitude of the outward directed energy flux. The magenta lines in the upper and lower right panels show the internal energy component of the normalized angular momentum flux ((ε + p)uφ/ρ0) and energy flux (−(ε + p)ut/ρ0). This component of the fluxes is zero by assumption in the inflow solution, and it is evidently an important component of the fluxes in our thick disk simulations. This leads to large corrections to the angular momentum and energy fluxes; the total normalized angular momentum flux is significantly smaller than the thin disk prediction, while the energy flux is, seemingly by conspiracy, very close to the thin disk. The lower left panel shows the rest-mass density from the inflow solution (upper blue line) and from the simulation (cyan line). The mass flux in the inflow solution is normalized so that it matches the simulation mass flux. Since mass flux is approximately constant with radius, the run of density is directly related to the run of ur. What is remarkable here is that there is no feature in the simulation ρ0 near the ISCO. In fact it is nearly constant from well outside the ISCO in to the event horizon. The surface density varies smoothly as well. This confirms the point made by Krolik and Hawley (2002) in their pseudo-Newtonian solution: there is no sharp feature at the ISCO. This has implications for iron line profiles, as discussed by Reynolds and Begelman (1997). The lower left panel also shows the run of internal energy density in the simulation (it is zero by assumption in the inflow solution). Again, there is no sharp feature at the ISCO, just a gentle rise inward toward the event horizon. Because the density is nearly constant with radius this implies that entropy is increasing inward. Therefore there is some dissipation of kinetic or magnetic energy into internal energy in the inflow region. The lower left panel of Figure 4.10 shows the run of magnetic energy density b2/2 in the inflow solution (lower blue line) and simulation (green line). The normalization of the inflow magnetic energy is a parameter, but its radial slope is not.

Finally, the inﬂow solution predicts that ω = ΩISCO. Figure 4.8a shows the run of ω/ΩH on the horizon for the a = 0.5 model. The dashed line shows the ISCO value of ω/ΩH . At the equator the time-averaged numerical value lies within about 10% of the ISCO value: the numerical average

ω/ΩH = 0.685, while the ΩISCO/ΩH = 0.8136 at the ISCO. In the a = 0.938 run the numerical average ω/ΩH = 0.681, while ΩISCO/ΩH = 0.745 at the ISCO. To sum up, the inﬂow model does a surprisingly good job of matching some aspects of the

143 time-averaged simulation. It does not match the profile or boundary condition at the ISCO for the radial velocity or the total angular momentum and energy fluxes, because the simulation flow is hot, while the inflow solution has zero temperature by assumption. What is most surprising is that the energy per baryon accreted in the numerical model matches the thin disk prediction. The inflow model predicts that the energy per baryon accreted should be lower than the thin disk prediction, enhancing the nominal accretion efficiency (Gammie, 1999; Krolik, 1999a; Agol and Krolik, 2000). The difference is apparently due to the finite temperature of the numerical model and the consequent change in boundary conditions at the ISCO. These boundary conditions evidently adjust themselves to maintain the energy flux at the thin disk value. The angular momentum flux is affected by the field, however, with the specific angular momentum of the accreted material in the fiducial run about 25% lower than the thin disk.

4.5 Parameter Study

Our numerical model has a number of physical and numerical parameters. Here we check the sensitivity of the model to: (1) black hole spin parameter a; (2) initial magnetic field geometry and initial magnetic field strength; and (3) numerical parameters such as (a) location of the inner boundary (rin); (b) outer radial (rout) boundary; (c) radial and θ resolution, including the coordinate parameter h; and (d) parameters describing the density and internal energy floors.

4.5.1 Black Hole Spin

The fiducial run has a rather low outgoing electromagnetic energy flux compared to the ingoing matter energy flux. It is possible that this varies sharply with black hole spin and that more rapidly rotating holes exhibit much larger electromagnetic luminosity. We have performed a survey over a, keeping all parameters identical to those in the fiducial run, except that the resolution is lowered to 2562 and the location of the pressure maximum is adjusted to keep H/R ≈ const. The results are shown in Figure 4.11 and described in Table 4.2. The figure shows the measured ratio of electromagnetic to rest-mass energy flux; the dashed line shows a fit

˙ (EM) E 2 ≈ −0.068(2 − r+) . (4.61) E˙ (MA) This fit applies only to this particular sequence of models; models with different initial field geometries give different results, as we shall see below. For all a > 0 we find E˙ (EM) > 0 in the funnel. For a < 0.5 this outward funnel flux is balanced by an inward electromagnetic energy flux near the equator. For our most extreme run with a = 0.969 the outward electromagnetic flux is still dominated by the inward particle flux. The ratio of electromagnetic luminosity to nominal accretion luminosity is L˜ = 27%, so the nominal accretion luminosity dominates over the BZ luminosity. The accretion rate of angular momentum is also a strong function of spin. As discussed in Gammie et al. (2004), accretion flows around rapidly spinning holes have da/dt < 0. Our fiducial

144 Figure 4.11 The ratio of electromagnetic to matter energy ﬂux on the horizon. The solid line indicates numerical data while the dotted line indicates a best ﬁt of E˙ (EM)/E˙ (MA) = −0.068(2 − 2 r+) . See Section 4.5.1 for a discussion.

145 Table 4.2. Black Hole Spin Study

4 (EM) (MA) a 10 × E˙ /E˙ E/˙ M˙ 0 L/˙ M˙ 0 a/˙ M˙ 0 M˙ 0 L˜

-0.938 105 0.958 3.806 -5.583 -0.908 -0.24 0.000 34.4 0.950 3.068 -3.049 -0.870 -0.065 0.050 31.2 0.952 3.025 -2.921 -0.709 -0.062 0.100 35.8 0.948 2.896 -2.713 -0.767 -0.066 0.150 29.7 0.949 2.881 -2.597 -0.796 -0.055 0.200 26.9 0.948 2.817 -2.439 -0.776 -0.050 0.250 9.17 0.946 2.749 -2.302 -0.747 -0.016 0.300 3.30 0.937 2.759 -2.217 -0.571 -0.0049 0.350 1.32 0.933 2.605 -1.975 -0.620 -0.0018 0.400 1.15 0.937 2.763 -1.986 -0.241 -0.0017 0.500 -9.85 0.933 2.583 -1.665 -0.252 0.014 0.600 -28.5 0.929 2.489 -1.347 -0.318 0.037 0.750 -81.8 0.908 2.150 -0.808 -0.276 0.083 0.875 -291 0.852 1.440 -0.152 -0.170 0.17 0.895 -254 0.891 1.723 -0.204 -0.215 0.20 0.900 -315 0.882 1.674 -0.118 -0.193 0.24 0.938 -318 0.856 1.396 0.067 -0.203 0.23 0.969 -410 0.869 1.374 0.217 -0.172 0.27

2 Note. — All models same as ﬁducial except at a resolution of 256 and rmax is used to keep H/R ∼ constant. These values can be compared to Tables 3 and 4. The eﬃciency is 1−E/˙ M˙ 0. A positivea/ ˙ M˙ 0 corresponds to a spindown of the black hole since M˙ 0 < 0.

146 model, in fact, is spinning down. Previous estimates suggested that spin equilibrium is reached at a ∼ 0.998 (Thorne, 1974). Our models reach spin equilibrium at a ∼ 0.92. The variation of field strength and geometry with black hole spin is also of interest. To measure variation of field strength, we probe the flow near four locations: 1) in the funnel near the horizon (“funnel/horizon”); 2) in the plunging region near the horizon (“plunging/horizon”); 3) at the ISCO; and 4) at the pressure maximum. We then take a time and spatial average of the comoving electromagnetic energy density b2/2 over a small region near each of these locations. The ratio of b2(funnel/horizon) to b2(plunging/horizon) changes from 0.43 at a = 0 to 0.74 at a = 0.938. The ratio of b2(funnel/horizon) to b2(ISCO) varies from 2.53 at a = 0 to 2.14 at a = 0.938. The ratio b2(funnel/horizon) to pressure maximum varies from 4.8 at a = 0 to 15.7 at a = 0.938. In summary, the field strength increases from the ISCO to the horizon by a factor of ∼ 3 at a = 0 and by a factor of ∼ 6 at a = 0.938, and on the horizon is slightly larger at the equator than at the poles by a factor of ∼ 2. Only the ratio of b2(pressure maximum) to other locations in the plunging region or at the horizon depends strongly on black hole spin. Our observed increase in horizon field strength with black hole spin agrees with results reported by De Villiers et al. (2003a). (Livio et al., 1999) suggest that 1) there is no reason the field strength near the black hole horizon should be stronger than in the central regions of the disk, which they suggest implies the electromagnetic output from the disk (corona) dominates the BZ luminosity ; and 2) the spin of the hole is probably irrelevant to the electromagnetic output in the wind. We have found that the field strength from the pressure maximum of the disk to the horizon increases by a factor ∼ 5 − 16 depending on the black hole spin. Also, for a = 0.5 we find that the BZ luminosity contributes about 1/5 to the total E˙ (EM) in the wind, and for a = 0.938 about 1/2. The rest of E˙ (EM) in the wind is provided by the disk corona, which is mostly dissipated into kinetic energy of the wind. This suggests that black hole spin is important in determining the electromagnetic energy flux in the wind, and that the BZ luminosity in the funnel can be a significant contribution to Poynting flux at large distances. We see no sign of the expulsion of flux from the horizon reported by Bicak and Janis (1985), who find that the flux through one hemisphere of the horizon, due to external sources and calculated in axisymmetry using vacuum electrodynamics, vanishes when the spin of the hole is maximal. It is possible that we have not gone close enough to a = 1 to observe this effect. To investigate the variation of field geometry in the funnel region with a we trace field lines from θin on the horizon to θout on the outer boundary and define a collimation factor θin/θout. The collimation factor is similar for all field lines in the funnel region. It reaches a minimum of ≈ 5/2 for the fiducial run, and rises to nearly 2 for a = 0 and again to nearly 2 for a ∼ 1. The collimation factor depends on the location of the outer boundary; for models with rout = 400 the collimation factor is 10 and the field lines are nearly cylindrical at the outer boundary.

We have also studied the variation of the ﬁeld rotation frequency ω in the funnel. ω/ΩH varies weakly with a, from 0.53 at a = 0.25 to 0.45 at a = 0.938, consistent with the hypothesis advanced by Thorne et al. (1986) that ω/ΩH ≈ 1/2.

147 Table 4.3. Field Strength and Geometry Study

(EM) (MA) Field Geometry Aφ β E˙ /E˙ E/˙ M˙ 0 L/˙ M˙ 0 a/˙ M˙ 0 M˙ 0 L˜

0 Aφ 100 −0.0312 0.856 1.40 0.0674 −0.203 0.21 0 Aφ 500 −0.0115 0.879 1.94 −0.293 −0.0474 0.085 0 Aφ sin (log (r/h)) 100 −0.0355 0.892 1.24 0.278 −0.541 0.42 0 Aφ| sin(2θ)| 100 −0.0112 0.888 1.91 −0.299 −0.0746 0.083 r sin θ 100 −0.147 0.773 −0.997 1.807 −1.769 0.79 r sin θ 400 −0.157 0.813 0.0617 1.184 −0.715 0.67

0 Note. — Aφ is the fiducial model field geometry and β = 100 is the fiducial ratio of gas to magnetic pressure. The r sin θ field geometry is a uniform vertical field model with β set by disk values at the equator. All other model and numerical parameters are as in the 2 fiducial model except that the resolution is 256 . The efficiency is 1−E/˙ M˙ 0. A positivea/ ˙ M˙ 0 corresponds to a spindown of the black hole because M˙ 0 < 0.

4.5.2 Field Geometry and Strength

The outcome of the simulation may also depend on the field geometry and strength in the initial conditions. This seems more likely for axisymmetric models such as ours where the evolution may retain a stronger memory of the initial conditions than comparable three dimensional models. We begin by investigating the dependence of outcome on initial field strength, parameterized by β ≡ pgas,max/pmag,max (notice that the two maxima never occur at the same location in space, so this ratio varies over a wide range when evaluated at individual locations in the disk). We consider models with β = (100, 500) and find a weak dependence on β. For the β = 100 model 2 (EM) (MA) (the fiducial model at a resolution of 256 ) we find ω/ΩH ≈ 0.45, E˙ /E˙ ≈ −3.1%, and (EM) (MA) L˜ = 21%. β = 500 leads to ω/ΩH ≈ 0.42, E˙ /E˙ = −1.2%, and L˜ = 8.5%. Notice that a higher spatial resolution is required to fully resolve weak field models, although all runs in this comparison were done at 2562; the decrease in electromagnetic energy extracted at β = 500 may therefore be due to resolution. We also vary the field geometry from the single loop used in our fiducial model, which has vector potential Aφ ∝ MAX(P/Pmax − 0.2, 0). We do this by multiplying the vector potential by sin(log(r/h)) or | sin(2θ)|. The former decompresses the field lines at the inner radial edge giving a field strength that is more uniform around the loop (for an extended disk this would yield a sequence of field loops centered at the midplane with alternating sense of circulation). The latter yields two loops, one centered above the equator and the other below, with the same sense of circulation. (EM) (MA) The sin(log(r/h)) modulation gives ω/ΩH ≈ 0.44, E˙ /E˙ ≈ −3.6%, and L˜ = 42%. The (EM) (MA) | sin(2θ)| modulation gives ω/ΩH ≈ 0.40, E˙ /E˙ ≈ −1.1%, and L˜ = 8.3%. Increasing the

148 number of initial field loops therefore leads to a weak (factor of 2 − 3) decrease in E˙ (EM)/E˙ (MA), while making the field strength more uniform around the loop increases L˜ by a factor of 2 with a nearly constant E˙ (EM)/E˙ (MA). Higher resolution studies may better resolve these simulations and show weaker dependence on field geometry.

We have also considered a purely vertical field geometry: Aφ ∝ r sin θ. In a Newtonian context this would correspond to a uniform z field in cylindrical coordinates. The field is normalized so that β = pgas,max/pmag,max = 100 and 400 in the equator of the torus. The outcome is different from any of the other models. The funnel field in the vertical field run is strong compared to the disk field. The accretion rate is larger, by a factor of 5, than the fiducial run. In the early stages there is a brief net outflow of energy from the black hole (although the total energy released from the hole is negligible compared to the energy gained at later times). The β = 100 model has a high mean efficiency; E/˙ M˙ 0 = 0.77, compared to 0.82 expected for a thin disk. There is also a net outflow of angular momentum from the black hole, with L/˙ M˙ 0 = −1.00, compared to 1.95 expected for a thin disk. The wind has a peak asymptotic radial velocityv ˜r = 0.94c, attained near the outer boundary, compared tov ˜r = 0.75c (EM) (MA) for the fiducial run. Finally, the model has ω/ΩH ≈ 0.41, E˙ /E˙ ≈ −15%, and L˜ = 79%. The β = 400 vertical field model has very similar properties, which suggests that we are resolving the β = 100 model. Table 4.3 summarizes measurements from the varying field geometry models. The models with net vertical field exhibit markedly different behavior from the fiducial model. It seems likely that some of this difference is due to the axisymmetric nature of the model; in 3D matter can accrete between the vertical field lines without having to push them into the hole. That is, in 3D, it would be easier for the hole to rid itself of the dipole moment that it acquires in the net vertical field calculation. But we cannot say with any confidence what the outcome is until a full 3D experiment on a disk with nonnegligible magnetic dipole moment.

4.5.3 Numerical Parameters

We have run the fiducial model at resolutions of 642, 1282, 128 × 64, 2562, and 4562. There is a weak dependence on resolution in the sense that E˙ (EM)/E˙ (MA) is smaller at higher resolutions. Lower resolution models do not sustain turbulence for as long as high resolution models, so we average over 500 < t < 1000, when all models are turbulent. Table 4.4 gives a summary of results from the resolution study. In every case the nominal radiative efficiency is close to the thin disk value. Resolution of the near-horizon region, where the energy density is large, is also a concern, because our accretion rates are measured there. We have checked dependence on radial numerical resolution of the near-horizon region by modifying the coordinate definition in equation (7) to read x x r = R0 + e 1 rather than r = e 1 . Increasing R0 from 0 to the horizon radius increases the number of grid zones located near the horizon. We ran a model with R0 = 0.5 and found no significant difference from a comparable model with R0 = 0. This suggests that we are adequately resolving

149 Table 4.4. Resolution Study

(EM) (MA) Resolution E˙ /E˙ E/˙ M˙ 0 L/˙ M˙ 0 a/˙ M˙ 0 M˙ 0 L˜

642 -0.0528 0.914 1.630 0.036 -0.159 0.55 128 × 64 -0.0438 0.841 1.420 0.121 -0.165 0.23 1282 -0.0447 0.887 1.518 0.087 -0.167 0.38 2562 -0.0316 0.874 1.274 0.198 -0.186 0.27 4562 -0.0261 0.865 1.381 0.216 -0.299 0.18

Note. — Numerator and denominators are separately time averaged from 500 < t < 1000 at the horizon. This interval is chosen so that all models are turbulent (in the lowest resolution model turbulence decays shortly after t = 1000). The 4562 model is the ﬁducial model. The nominal radiative eﬃciency is 1 − E/˙ M˙ 0. A positivea/ ˙ M˙ 0 corresponds to a spindown of the black hole because M˙ 0 < 0.

the near-horizon region. (EM) (MA) r 2 We also varied rin and rout and found no measurable difference in E˙ , E˙ ,(B ) , and ω on the horizon. We have moved rout from 40 to 400 and rin from 0.7r+ to 0.98r+ and find negligible differences in these quantities on the horizon. The solution is not sensitive to the location of the inner or outer boundary. Moving the inner boundary of the computational domain outside the horizon (e.g. 1.05r+) leads to strong reflections from the boundary conditions and, ultimately, failure of the run. It is possible that better inner boundary conditions or higher resolutions could overcome this difficulty, but it seems cleaner to simply leave the boundary inside the event horizon at r = 0.98r+, out of causal contact with the rest of the simulation.

The model with larger rout = 400 does exhibit some new features. The magnetic ﬁeld lines in the funnel region have a collimation factor of 10 by the time they reach the outer boundary. At

R = 40, however, both the rout = 400 model and the ﬁducial model have a collimation factor of

5/2. By rout = 400 the field lines are nearly cylindrical. The peak of the radial component of the asymptotic 3-velocity in the wind is identical to the fiducial run withv ˜r = 0.75c, indicating little acceleration between R = 40 and R = 400. The main numerical uncertainty in our experiments arise from the floor on the density and −4 −3/2 −6 −5/2 internal energy. We varied the floor scaling from ρ0,min = 10 r and umin = 10 r to −4 −2.7 −6 −3.7 2 ρ0,min = 10 r and umin = 10 r (we chose these scalings so that b /ρ0 would be nearly 2 constant with radius in the funnel). While this significantly affects b /ρ0, it does not otherwise affect E˙ (EM) and E˙ (MA) or the mean values of Br and ω measured on the horizon. −4 −6 We varied the floor normalization at r = 1 from the fiducial values (ρ0,min, εmin) = (10 , 10 ) to (10−5, 10−7), and (10−6, 10−8). This causes almost no change in the flow near the horizon. In

150 2 the funnel, however, we are at the limit of our ability to integrate the MHD equations (b /ρ0 À 1). −5 Our integration fails when we attempt to use a mass density floor ρ0,min(r = 1) . 10 when r rout À the outer edge of the initial torus. Lower floors lead to faster outflowsv ˜ À 0.99c in the funnel region, which are more likely to be numerically unstable. These results hint that low density models will produce fast outflows, but a confirmation awaits a more stable GRMHD algorithm. 2 The funnel region is difficult to integrate reliably, because when b /ρ0 À 1 small fractional errors in field evolution lead to large fractional errors in the evolution of other flow variables. This is a consequence of our conservative scheme, in which all the dependent variables are coupled together by the interconversion of primitive and conserved variables. Evolution of the MHD equations in nonconservative form (e.g. using an internal, rather than total, energy equation), as in De Villiers and Hawley (2003a), may be slightly more robust, although De Villiers and Hawley eventually experience similar problems in the funnel. In any event, the close correspondence between the numerical experiment and the BZ model raises confidence in the results and suggests that the magnetic field, if not the mass density and internal energy density, is being evolved reliably.

4.6 Discussion

We have used a general relativistic MHD code, HARM, to evolve a weakly magnetized thick disk around a Kerr black hole. Our main result is that we find an outward electromagnetic energy flux on the event horizon, as anticipated by Blandford and Znajek (1977). The funnel region near the polar axis of the black hole is consistent with the Blandford-Znajek model. The outward electromagnetic energy flux is, however, overwhelmed by the inward flux of energy associated with the rest-mass and internal energy of the accreting plasma. This result essentially confirms work by Ghosh and Abramowicz (1997) that suggested the BZ luminosity should be small or comparable to the nominal accretion luminosity (L˜ . 1). One of our models discussed here, however, begins with a vertical field threading the torus, exhibits a brief episode of outward net energy flux. This appears to be a transient associated with the initial conditions. The same model exhibits a steady net outflow of angular momentum from the black hole. Of all our models, the vertical field model has the largest negative −E˙ (EM)/E˙ (MA) ≈ 15% (ratio of the electromagnetic energy flux to ingoing matter energy flux) and largest L˜ = (EM) E˙ /(−²M˙ 0) ≈ 80% (ratio of electromagnetic luminosity to nominal accretion luminosity). This suggests that the BZ effect could play a significant role if the disk has a net dipole moment and accumulates magnetic flux that crosses the horizon. This possibility will be considered in future work. Consistent with the results found earlier by De Villiers et al. (2003a), we find that our models 2 can be divided into four regions: (1) a “funnel” region with b /ρ0 & 1 and β ¿ 1; (2) a corona with 1 . β . 3; (3); an equatorial disk with β > 3; and (4) a plunging region between the disk and event horizon with β ∼ 1 and a nearly laminar inflow from the disk to the black hole. We also find no feature in the surface density at or near the ISCO (see Figure 4.10), which agrees with the

151 results by Krolik and Hawley (2002); De Villiers et al. (2003a) and consistent with Reynolds and Begelman (1997). This is contrary to the sharp transition predicted by thin disk models and used by XSPEC to fit X-ray spectra. We have shown that the funnel region is nearly force-free, and is well-described by the stationary force-free magnetosphere model of Blandford and Znajek (1977), for which we have presented a self- contained derivation in Kerr-Schild coordinates. We find agreement between the BZ model and our simulations in measurements of energy flux, magnetic field, efficiency of accretion, and spindown power output. In all cases we find that in the force-free region the field rotation frequency is about half the black hole spin frequency, ΩH ≡ a/(2r+). This spin frequency maximizes electromagnetic energy output from the hole. This result is consistent with expectations of MacDonald and Thorne (1982) and the force-free numerical results of Komissarov (2001). We have also compared the time-average of the plunging region in our fiducial model with the stationary MHD inflow model of Gammie (1999), which assumes that the flow matches a cold disk at the ISCO. The inflow model matches the simulated rest-mass flux and electromagnetic flux of energy and angular momentum surprisingly well, particularly considering the strongly variable nature of the simulated flow in the plunging region. The inflow model fails to match other aspects of the flow, such as the radial component of the four-velocity. This is mainly due to the finite temperature of the simulated flow; the inflow solution assumes zero temperature. It is slightly surprising that the total angular momentum flux is close to the value predicted by the zero temperature inflow solution, and 20% less than what is predicted by the thin disk, yet the total energy flux is almost exactly what is predicted by the thin disk. It is as yet unclear whether this is due to coincidence or conspiracy. For a set of models similar to the fiducial model, the ratio of electromagnetic to matter energy fluxes is sensitive to the black hole spin, reaching −7% for a ∼ 1. The evolution is sensitive to the initial field geometry. Models with a net vertical field are more efficient, and more electromagnetically active than models with comparable field strength but zero net vertical field. Our models have a weak dependence on resolution in the sense that as resolution increases the relative importance of electromagnetic energy fluxes on the horizon diminishes.

With an rout = 400 model we demonstrate that an outgoing electromagnetic energy flux can reach large radii. The field in the funnel region does not connect back into the disk. Rather the poloidal components lie parallel to the polar axis. The field lines are collimated by a factor of 5/2 at r = 40 and by a factor of 10 at r = 400. An outflow along the boundaries of the funnel reaches a maximumv ˜r ≈ 0.75c, but this is sensitive to the value of our artificial density “floor”: a model with lower density reaches even larger radial velocities at the outer boundary of the computational domain.

Koide et al. (2002) have evolved a cold, highly magnetized uniform density plasma (ρ0/p = 2 0.06, b /ρ0 = 10) as it falls into a rapidly spinning (a = 0.99995) black hole in Boyer-Lindquist coordinates for a time ≈ 14GM/c3 using the MHD approximation. This initial state does not correspond to an accretion disk system. They demonstrated, however, that a transient net energy

152 extraction is possible from a spinning black hole. Because of the short evolution time they are unable to say whether the energy extraction process is possible in steady state. Koide (2003) gives an expanded discussion of the above system. In contrast to the results of Koide et al. (2002) and Koide (2003), we model a disk with an initially hydrodynamic equilibrium fluid that is weakly magnetized. We also use a Kerr-Schild (horizon penetrating) coordinate system that avoids potential problems associated with the treatment of inner boundary condition in Boyer-Lindquist coordinates. In our simulation the Balbus-Hawley instability drives turbulence and accretion in a steady state where we evolve for a time 2000GM/c3. We measure a sustained outward electromagnetic energy flux that is smaller than the inward matter energy flux (i.e. net inward energy flux). Their model is evolved for too short a time to observe the unbound mass outflow in the funnel region as seen by us and De Villiers et al. (2003a, 2004). De Villiers et al. (2004)(hereafter DH) have also considered the numerical evolution of weakly magnetized tori around rotating black holes. Their models are quite similar to ours in many respects, although they differ in that: (1) their models are three dimensional while our models are axisymmetric; (2) they use a nonconservative numerical method (De Villiers and Hawley, 2003a); (3) DH use Boyer-Lindquist while we use Kerr-Schild coordinates; (4) DH choose γ = 5/3 while we use γ = 4/3; (5) DH’s initial pressure maximum is located at 25M, while ours are typically at 12M. Our results for the energy and angular momentum per baryon accreted from Table 4.2 can be compared to Table 1 of DH by computing E/˙ M˙ 0 = ∆Ei/∆Mi and L/˙ M˙ 0 = ∆Li/∆Mi. For models with a = (0, 0.5, 0.9) DH find E/˙ M˙ 0 = (0.91, 0.91, 0.84) while we find E/˙ M˙ 0 = (0.96, 0.93, 0.88).

For the same models DH find L/˙ M˙ 0 = (3.1, 2.6, 1.9), while we find L/˙ M˙ 0 = (3.1, 2.6, 1.7). Given the differences in the models and numerical methods, this quantitative agreement is remarkable. Our models and De Villier and Hawley’s models also agree qualitatively in the sense that both show a similar geometry of disk, corona, and funnel and both imply that spin equilibrium is achieved at a ∼ 0.9 (see Gammie et al. 2004). Komissarov (2001) finds the BZ solution to be stable in force-free electrodynamics, and Komis- sarov (2002a, 2004a) find the BZ solution to be causal, but inconsistent with the membrane paradigm. We find our numerical solutions to be consistent with the BZ solution in the low-density funnel region around the black hole. A numerical general relativistic MHD study of strongly magnetized (monopole magnetic field) accretion by Komissarov (2004b) is also consistent with the BZ solution. For the strong field chosen he finds a considerably faster outflow (Lorentz factors of ≈ 14) than found in our models (Lorentz factors of ≈ 1.5 − 3.0). Komissarov’s model does not contain a disk. The limitations of the numerical models presented here include the assumption of axisymmetry and a nonradiative gas. The effect of axisymmetry can be tested by comparing our models with the three dimensional models of De Villiers et al. (2004); the angular momentum and energy per accreted baryon in the two models differs by only a few percent. In addition the jet structure observed in De Villiers et al. (2004) is nearly axisymmetric. This is encouraging, although it is unlikely that an axisymmetric calculation can capture the full range of possible dynamical behavior

153 in the accretion flow. The radiation field, which we have completely neglected here, is likely to play a significant role in the flow dynamics, through radiation force on the outflowing plasma in the wind and through photon bubbles in the disk Gammie (1998); Socrates and Blaes (2002). It will also, of course, play a significant role in heating and cooling the plasma. This is clearly the most significant limitation of our calculation– particularly from the standpoint of comparison with observations– and clearly the most numerically difficult problem to overcome.

154 5 Future Studies

For future work, I plan to perform simulations that will 1) establish the power of the BZ eﬀect as a function of disk thickness ; 2) determine the connection between the disk and the jet; 3) determine the relationship between the black hole spin and jet; and 4) establish the requirement of GRMHD to jet speed, collimation, and stability. I also plan to speciﬁcally study the collapsar model of GRBs using a GRMHD model with additional GRB-type microphysics.

5.1 Thin Disks

Chapter 4 presented a study of the BZ luminosity of a rotating black hole surrounded by a thick disk, for which the scale height (H) is comparable to the radius (R). I plan to extend those results to the thin disk regime, which may be applicable to some black hole accretion disk systems. Prior analytic work suggests that thin disks may have an insignificant BZ luminosity (Ghosh and Abramowicz, 1997; Livio et al., 1999). If thin disks have a weak BZ-effect and if the energy from the BZ-effect dominates the energy content of a jet in black hole systems, then those systems with a black hole and jet most certainly have a thick accretion disk near a rapidly rotating black hole. A self-consistent treatment of radiation generally leads to a disk with a radially dependent H/R, especially for disks near the black hole event horizon. While radiative effects are important, a general relativistic treatment is non-trivial (see, e.g., Cardall and Mezzacappa 2003; Cardall 2004). I plan to model thin disks with an ad hoc cooling function that sustains a constant H/R for all radii. The proposed H/R study may be sufficient to obtain an estimate for the BZ power output as a function of H/R without resorting to radiative transport. Compared to an otherwise equivalent thick disk simulation, the planned thin disk simulation requires a larger radial numerical resolution in order to resolve the MRI. Resolving the MRI, with about 6 zones per fastest growing wavelength, is crucial in sustaining turbulence for a sufficient period of time to reach steady state. For example, a typical thick (H/R ∼ 0.26) disk simulation requires a resolution of 256 × 256. A thin disk model with a = 0 and H/R = 0.01 requires a resolution of 6500 × 1200 and a thin disk model with a = 0.938 and H/R = 0.02 requires a resolution of 9800 × 920. Also, in order to avoid artificially loading the (larger) funnel region with baryons, the thin disk model requires a lower density floor than the thick disk model. Also, a lower density floor is required for convergence of the magnetic field in the funnel region. Using a lower 2 density floor (and thus increasing b /ρ0) may require improving the algorithms used in HARM.

155 5.2 Relativistic Jet-Disk Connection

GRMHD computational studies are on the verge of understanding the mechanism of jet formation from an accretion disk around a rotating black hole (McKinney and Gammie, 2004; De Villiers et al., 2004). These numerical models are beginning to identify 1) the mechanism of jet acceleration ; 2) the relative contributions of electromagnetic and kinetic energy to the jet; and 3) the baryon loading in the jet – i.e. the amount of rest-mass in the jet. The resolution of these issues are one focus of my future study, and some work has begun as described below. The source of energy generating a jet could arise from the BZ mechanism (an electromagnetic black hole wind), from the Blandford-Payne mechanism (a matter disk wind), from an electromagnetic disk-corona wind, radiation driving, or some combination of all these processes. I performed GRMHD simulations of rapidly rotating (a & 0.5) black holes that are spun-down by the BZ-effect, which generates an outgoing electromagnetic energy flux in a mildly relativistic (v/c ∼ 0.3), collimated outflow that reaches large distances. The black hole spin energy is extracted directly from the rotating black hole and flows along collimated field lines frozen within unbound, baryon-poor matter. For the particular parameter space I explored, the Poynting flux always dominates the matter-energy flux in the jet. There is a small matter-energy jet at the edge between the corona and funnel in our models with high black hole spin. The electromagnetic component of this “disk wind” appears to only provide a source of acceleration for the outgoing matter in the funnel region, since the fluid is tied-up in the magnetic field within the corona. Hawley’s group at Virginia, and our group, have performed simulations that show that black hole spin plays a significant role in determining the existence and power of a jet, but the details are only beginning to be studied (De Villiers et al., 2004).

5.3 Gamma-Ray Bursts: GRMHD Collapsar Model

I plan to use a GRMHD model to study the accretion disk that likely forms during a GRB, in the framework of the collapsar model (Woosley, 1993; Paczyński,1998; MacFadyen and Woosley, 1999). Some interesting unresolved questions include 1) what fraction of the observed luminosity of GRBs is due to power by black hole spin energy extraction, accretion disk luminosity, or neutrino annihilation, 2) what is the mechanism for jet production, collimation, and GRB variability ; 3) what is the efficiency of neutrino annihilation and magnetic field reconnection ; and 4) is the jet composed of ion/electron or positron/electron pair plasma and what particle/radiation physics dominates GRB radiation. Numerical studies of the Type I collapsar model have been performed using Newtonian hydrodynamic (HD) (MacFadyen and Woosley, 1999) and special relativistic HD models (Aloy et al., 2000; Zhang et al., 2003). Type II collapsars have been studied using Newtonian HD numerical models (MacFadyen et al., 2001). All of these models include a realistic EOS, neutrino cooling, and some estimates of neutrino annihilation. These unmagnetized numerical models include an ad hoc

156 alpha-viscosity model of angular momentum transport (Shakura and Sunyaev, 1973). However, the magnetic field is likely explicitly required to launch and collimate jets (Cameron, 2001; Wheeler et al., 2002), and the angular momentum transport is likely driven by the MRI (Balbus and Hawley, 1991). Recent numerical studies of non-self-gravitating accretion disks use the nonrelativistic MHD approximation (Proga et al., 2003) and use a code based on ZEUS (Stone and Norman, 1992). They include a realistic EOS, photodisintegration of helium, and neutrino cooling. They observe a neutrino luminosity of ∼ 1052 erg/s, which is close to the expected GRB luminosity. However, their model does not include neutrino annihilation, so their jet contains no energy from annihilating neutrinos that may alter the jet structure. Also, they cannot estimate what fraction of the energy in the jet comes from neutrino annihilation compared to electromagnetic luminosity from the disk. Neutrino-generated jets are likely important (see, e.g., Fryer and Mészáros2003). They observe a jet that is dominated by Poynting flux, rather than matter-energy flux. However, since their model is Newtonian, they are unable to measure the Lorentz factor of the jet and cannot model a spinning black hole, which may be required to generate a Poynting flux via the BZ-effect. As discussed above, I used a similar ZEUS-based MHD code with a pseudo-Newtonian potential (Paczyńskiand Wiita, 1980) to study similar flows without microphysics. My results suggest that their studies of the jet could be dominated by numerical artifacts. For example, I found it impossible to remove the inner radial boundary condition from causal contact with the rest of the flow (i.e. one cannot keep the inner radial boundary inside the fast point in MHD). Also, while I found a large Poynting flux from the horizon, the Newtonian model should not be capable of producing such an effect. The results of Proga et al. (2003) are likely similar. Most of the Poynting flux energy could be artificially generated by contact between the inner radial boundary and the flow at larger radii. Indeed, GRMHD simulations show that accreting black holes with spin parameter a = 0 generate weak outflows and much smaller Poynting flux at large distances (McKinney and Gammie, 2004). GRMHD numerical models suggest Poynting flux jets are a natural result of accretion around a rapidly rotating black hole (De Villiers et al., 2003a; Koide, 2003; McKinney and Gammie, 2004). However, such simulations ignore possibly important sources of jet energy, such as neutrino annihilation, and lack accurate models of reconnection (see, e.g., Sikora et al. 2003). Recent GRMHD collapsar simulations have been performed that fail to generate the estimated observed GRB energy and Lorentz factors (Mizuno et al., 2004a,b). However, their model includes no microphysics; they use a shock model, rather than the core of a massive star, as their initial conditions; and the simulation lasts for only a small number of dynamical times (∼ 200GM/c3). The black hole accretion disk system itself may not generate the large Lorentz factors. For example, a radial pressure gradient in the star and external atmosphere can accelerate the jet to Γ ∼ 44, close to that required by observations (see, e.g., Aloy et al. 2000; Zhang et al. 2003). The current GRMHD numerical model has an ideal-gas equation of state and includes no radiation of neutrinos. To the current GRMHD numerical model, I plan to add a realistic equation of state (EOS) and neutrino cooling and annihilation in the optically thin parts of the flow. I plan to

157 first include an approximate EOS (Popham et al., 1999; Kohri and Mineshige, 2002), which includes electrons, positrons, protons, neutrons, alpha particles, and photons as a single-component fluid. All particles are assumed to be in thermodynamic and nuclear statistical equilibrium. This approximate EOS allows the electrons, positrons, protons, and neutrons to have (fairly) arbitrary relativity and degeneracy. I later plan to include an more accurate EOS solver that allows for any relativity and degeneracy (Lattimer and Swesty, 1991; Lattimer, 1996; Blinnikov et al., 1996; Lattimer and Prakash, 2000). The disk is optically thick to all particles except potentially neutrinos, so the single-component fluid approximation is not applicable to neutrinos. Since photons and positrons are in thermal equilibrium, they will be treated as an additional source of radiative pressure. The photodisintegration of alpha particles and recombination of free nuclei into alpha particles will also be included. Neutrinos are to be included as part of the fluid for parts of the disk that are optically thick to neutrinos and otherwise stream freely for the optically thin parts. The neutrinos can radiate in the neutrino “optically” thin parts of the disk (see, e.g., Itoh et al. 1996; MacFadyen and Woosley 1999), while neutrinos provide a pressure in the neutrino “optically” thick parts of the flow. This neutrino pressure can be estimated as if optically thick, but with an extinction factor in the radiative neutrino emissivity. As a result, the pressure is reduced by a corresponding factor (Lee et al., 2004). A diffusion scheme for the radiation of neutrinos in the optically thick regime will then be applied. If required, a full Boltzmann transport scheme will be used (see, e.g., Cardall and Mezzacappa 2003; Cardall 2004). Initially, I do not plan to include nuclear burning, since it is likely only relevant to the supernova component and not to the GRB jet.

158 A Model of Plasmas

This appendix discusses the assumptions of the fluid, MHD, and ideal MHD approximations of a plasma (for a classic review see, e.g. Landau and Lifshitz 1959, 1980). The purpose of this appendix is to present an overview of the parameters estimated in Section 1.5, which shows that the single-component ideal MHD approximation is valid for the accretion flow likely to be present in GRBs, X-ray binaries, and AGN. As described there, only thick accretion disks, with their diffuse plasmas, are not explicitly treatable by the fluid approximation. However, the solar wind too cannot be explicitly treated as a fluid, yet the MHD approximation is a reasonable approximation (see, e.g., Usmanov et al. 2000). Plasma instabilities likely force the solar wind to behave as a fluid (Feldman and Marsch, 1997), and a similar phenomena may occur in thick disks. A discussion of plasma instabilities is beyond the scope of this thesis, so the discussion below is restricted to the fluid, MHD, and ideal MHD approximations. First, a general overview of these approximations is given. If a gas predominantly collides elastically and contains many particles moving randomly within a characteristic length scale (L), then the gas can be approximated as conforming to a space-velocity distribution function. Liouville’s theorem and expressions for the body-forces are used to derive the distribution evolution equation. This results in the so-called Vlasov or collisionless Boltzmann equation. Collisions can be introduced as a source or sink of particles within the distribution, resulting in the collisional Boltzmann equation. The fluid approximation further assumes that these collisions drive the system through a series of statistical equilibrium states. The fluid approximation can be derived as a series of mass-weighted velocity moments of the Boltzmann equation. The MHD approximation can be derived by including a series of charge- weighted velocity moments of the Boltzmann equation. In general, the fluid has many species, but these can often be reduced into a single-component fluid or MHD approximation. The single- component approximation is obtained by summing each of the equations over species and by appropriately defining the average quantities. For perfect (ideal) fluids, collisions only drive the system to an equilibrium state, and other effects on the distribution function are ignored. For imperfect fluids, a closure scheme, such as the asymptotic closure scheme of Chapman-Enskog (Chapman and

Cowling, 1953), is used to relate higher order (say, in ² = λmfp/L) collisional terms as functions

of ﬂuid quantities obtained from the velocity moments, where λmfp is the mean free path between particle collisions. The discussion that follows does not present a text-book derivation of the ﬂuid, MHD, and ideal MHD approximations, but rather presents a summary of the salient features of the approximations and gives parameters for estimating the validity of the approximations. Note, however, a general

159 plasma is sufficiently complicated that no set of dimensionless parameters can absolutely determine whether each part of the approximations, described below, are valid. Specific configurations of matter and magnetic field can develop macroscopic instabilities, for example as with the magnetorotational instability (Balbus and Hawley, 1991). Also, microscopic particle effects can build up to become macroscopically relevant, for example during reconnection (see, e.g., Parker 1963, 1979; Syrovatskii 1981; Biskamp 1986; Taylor 1986; Parker 1988; Masuda et al. 1994; Low 1996) or due to plasma instabilities (see reviews by, e.g., Melrose 1986; Hasegawa 1975; Begelman and Chiueh 1988). Only a full nonlinear study can verify the dimensional arguments presented below.

A.1 Equation of State

An equation of state (EOS) is needed to close the fluid or MHD equations of motion. The EOS of matter near the black hole depends mostly on the density and temperature if the system is in nuclear and thermodynamic statistical equilibrium. Thermodynamic statistical equilibrium is maintained if all species are in thermal contact through collisions. Thus, one needs to know the elastic and inelastic scattering and absorption cross sections. The elastic scattering cross sections can be used to estimate not only the validity of the fluid approximation, but also to estimate the thermodynamic coupling between species. If some species are thermally coupled faster than the accretion time scale, then they will equilibrate to similar temperatures. However, if they do not transfer energy by collisions faster than they are accreted, then they could decouple, resulting in a two-temperature flow for a gas of protons and electrons. For example, it is plausible that in X-ray binaries and AGN a hot two-temperature disk forms (Shapiro et al., 1976; Narayan and Yi, 1995). In two-temperature or disk-corona models, the electrons can cool by Comptonization, while the protons and nuclei have no mechanism to cool efficiently before being accreted by the black hole. This effect is neglected for the EOS discussed here. The particle distributions need not be thermal, nor are observations constrained to only observe the most average electrons within the thermal distribution. Low-frequency radio observations of SgrA* could be explained if a small fraction of electrons reside in a power-law tail of energies (Yuan et al., 2003). Non-thermal electrons may be generated in reconnection events, such as has been observed in tokamak experiments of high density plasmas generated by MHD modes (Savrukhin, 2002) or in solar flares (Priest, 1981; Kahler, 1992; Zank and Gaisser, 1992; Miller et al., 1997; Priest and Forbes, 2002).

A.1.1 EOS for GRBs

Assuming that GRBs arise from a stellar core-collapse scenario (Woosley, 1993; Paczy´nski,1998; MacFadyen and Woosley, 1999), the accretion disk formed during a GRB likely has mostly free neutrons, protons, alpha particles, electrons, positrons, various heavy nuclei, and neutrino and photon radiation (see, e.g., MacFadyen and Woosley 1999). In thermodynamic and nuclear statistical equilibrium, the GRB disk state can be treated using an ideal Fermi EOS with arbitrary

160 2 degeneracy for all species and arbitrary relativity parameter (x) for all species, where x ≡ kbT/mc for a species with mass m (Blinnikov et al., 1996). Speciﬁcally, the approximations of Kohri and Mineshige (2002) are used in this thesis. A modiﬁed form of equation (26) in Kohri and Mineshige (2002) is used to improve the accuracy of the results near the transition between degenerate and non-degenerate electrons (Q = mn − me is reintroduced into that equation, as required for consistency). A Fermi EOS leads to important deviations from an ideal gas EOS due to neutronization in thin disks that may form during a GRB. The Fermi EOS determines the fraction of electrons per baryon (Ye). Fully ionized helium (alpha particles) can be dynamically important in the accretion shock that develops during GRB disk formation. The accretion shock, where nuclei are photodisintegrated into free nucleons, is located at r ∼ 120GM/c2 in the collapsar model (MacFadyen and Woosley, 1999). Also, within the neutrino-cooled disk or after the free nucleons are ejected out in a wind, they recombine to form heavier nuclei (e.g., alpha particles and 56Ni). Indeed, nucleosynthesis is important for understanding the generation of supernova and the GRB-supernova connection. These estimates ignore the supernova component and focus on the GRB event itself, which likely does not require following nucleosynthesis (MacFadyen and Woosley, 1999). This will at most require understanding the photodisintegration and recombination of alpha particles. If nuclear statistical equilibrium (NSE) holds (Khokhlov, 1989), then a Saha-like equation can be used to determine the fraction of free nucleons from µ ¶ Ã ! T 9/8 ³ ρ ´−3/4 −6.096 X = 30.97 exp (A.1) nuc 1010 1010 T ( 1010 )

(Woosley and Baron, 1992). The remainder of nucleons are bound in α particles.

The gas pressure (pgas) is deﬁned as the sum of each species’ (proton, electron, neutron, alpha particles, nuclei, radiation, and neutrino) partial pressure, with baryons or electrons being either completely nondegenerate or completely degenerate, and completely relativistic or completely nonrelativistic (Kohri and Mineshige, 2002). It has been found that such approximate equations of state agree to within 10% (Popham et al., 1999) of more general calculations by Blinnikov et al. (1996). In this approximation, if the disk is optically-thick to photons, then the photons are treated as a radiation pressure. If the disk is optically thin to photons, then photons stream out with a pressure extinction factor of about 1 − exp(−τγ), where τγ is the photon optical depth given in equation 1.3.

The local cooling rate would be multiplied by exp(−τγ). This does not model radiative transport processes that may lead to radiative instabilities, such as the photon-bubble instability (Gammie, 1998). The same radiation pressure extinction prescription is applied to the neutrinos (Lee et al., 2004) with a neutrino optical depth of

τν ∼ τs,ν + τa,eN + τa,νν¯, (A.2) ³ ´2 ¡ ¢ ³ ´2 ¡ ¢ −7 Tp ρ −7 Tp ρ where τs,ν = 2.7 × 10 1011 1010 H, τa,eN = 4.5 × 10 Xnuc 1011 1010 H,

161 ³ ´5 −7 Tp τa,νν¯ = 2.5 × 10 1011 H, Tp is the proton temperature, and H is a typical disk scale height (Di Matteo et al., 2002). These three terms respectively correspond to 1) neutrino-neutron/proton scattering ; 2) neutrino absorption onto free protons or neutrons ; and 3) inverse electron-positron pair annihilation (i.e. inverse URCA). These processes dominate the inverses of the bremsstrahlung and plasmon neutrino emission processes, which are thus neglected. For an overview, see also Shapiro and Teukolsky (1983) §18.5. In the electron degeneracy regime, the neutrino emission is slightly reduced due to a reduction of Ye (Kohri and Mineshige, 2002). The effect of electron degeneracy on the optical depth is not accounted for in this thesis, so the above estimated τν should be considered a reasonable estimate, but strictly only an upper bound. Some numerical studies of simplistic pseudo-analytic viscous (unmagnetized) Fermi gas models of GRB disks have been performed (Popham et al., 1999; Narayan et al., 2001; Kohri and Mineshige, 2002; Di Matteo et al., 2002). GRMHD (or even magnetized) numerical models of accretion disks that incorporate a self-consistent Fermi gas EOS have yet to be performed by anyone. In Section 1.5, where Ye, pgas, and other quantities are reported for the GRB disks, the relation between mass density, pressure, and magnetic field is actually from the GRMHD numerical model, rather than from a self-consistent model that includes a Fermi EOS. Section 1.5 reports the ratio of Fermi gas pressure to ideal gas pressure to check the consistency of this actually inconsistent approach. It turns out that this approach is fairly consistent, and certainly good enough for estimating the validity of the fluid, MHD, and ideal MHD approximations.

A.1.2 EOS for X-ray binaries and AGN

The species’ abundances in accretion disks in X-ray binary and AGN are most likely similar to cosmic abundances. A disk with cosmic abundances would be composed of mostly ionized hydrogen, some ionized helium, few ionized metals, electrons, and photon radiation (Fabian, 1998). For the temperatures and densities of these accretion disks, the appropriate approximate EOS is simply an ideal gamma-law gas + radiation pressure (which is simply a reduced form of the GRB EOS described above). This EOS is used to compute the actual structure of the accretion disk for these objects using SS73 thin disk (Shakura and Sunyaev, 1973) or ADAF thick disk (Narayan and Yi, 1995) models.

A.2 Validity of the Fluid Approximation

This section summarizes the parameters used to estimate the validity of the fluid approximation. The fluid approximation is valid only if the characteristic time (T ) is much longer than the effective collision time τ. Also, the effective mean free path for particles λmfp must be much less than the length scale (L) of the problem. An “effective” collision is one that has any nonzero cross section but generates a 90◦ change in the direction of motion. Essentially all particles interact at some density and energy, but this section only considers the dominant, typically elastic, scattering

162 mechanisms. Most elastic scattering processes dominate inelastic scattering when energies are less than the mass energy of a particle. In order to determine the mean free path, we should consider the total scattering cross section σ = σ(E), where E is the center of mass energy. The eﬀective mean free path for particle type A to collide with particle type B is p µAB/mA λmfp,A−B = , (A.3) σABnB where nB is the number density of particle type B, and µAB = mAmB/(mA + mB) is the reduced mass. Notice that the mean free path for particle A to hit particle B is not necessarily the same as the mean free path for particle B to hit particle A. To determine the collisional rate between each species, the relativistic relative velocity

vA + vB vrel,A−B = 2 (A.4) 1 + (vAvB/c ) is used to estimate the mean relative velocity within a thermal distribution. This expression for the relative velocity assumes that each species is an ideal gas moving with relativistic thermal velocity p x(2 + x) v = c , (A.5) therm 1 + x where x = k T [K]/(mc2) is the relativity parameter. This is simply the relativistic extension of b p the nonrelativistic thermal velocity vtherm,non−rel. ∼ KE/m. The relative velocity can be used to ﬁnd the binary collision rate vrel,A−B νc,A−B = . (A.6) λmfp,A−B

The proton-electron scattering cross section σpe is determined by the Coulomb interaction. The 1 2 2 nonrelativistic Coulomb energy U(r, v) = 2 mv − Ze /r of one charged particle in the electrostatic ﬁeld of another vanishes U(r, v) = 0 at a radius

Ze2 λc = , (A.7) kbTp

2 where Z is the atomic number. This is the characteristic radius for interaction and thus σZe ∼ λc . ³ ´2 2 1+x 4 2 Relativistic corrections, of order unity, give σZe ∼ λc x(2+x) csc (θ/2), where x = kbTp/(mec ) and θ is the scattering angle. There are additional corrections of order unity not considered (see, e.g., McKinley and Feshbach 1948). A more accurate cross section accounting for Debye screening is discussed below. The Thomson scattering cross section for photon-electron scattering is σT,e = −24 2 −31 2 0.665 × 10 cm . For photon-proton scattering σT,p = 1.967 × 10 cm .

163 Additional Scattering Cross Sections for GRB Disks

The proton-proton elastic scattering cross section is also dominated by the Coulomb interaction, except at high energies that occur in hot (thick) disks. For example, strong nuclear forces at 11 −24 2 50 MeV, corresponding to Tp ∼ 6 × 10 K, give σpp ∼ 0.110 cm (Berdoz et al., 1986). Notice −30 2 that at such energies the Coulomb cross section is only σc,pp = 8 × 10 cm , which is much smaller than the total scattering cross section. The Coulomb term dominates the nuclear term at low energies. At low energies the nuclear neutron-proton scattering cross section reaches up to −24 2 σpp,nuc.max = 36 × 10 cm (see Table 3-3 in Wong 1990). The transition between Coulomb and 8 strong interaction domination is at Tp ∼ 10 K. Proton-proton inelastic scattering becomes important above energy production for mesons such as pions. The lightest meson is π± with E ∼ 140 MeV, which corresponds to a temperature of 12 Tp ∼ 1.6 × 10 K. This corresponds to twice the upper limit of the typical temperatures in a GRB accretion disk, although such processes may be important in short moments when/where the plasma may be signiﬁcantly hotter. The cross section for pion production in pp-scattering above −24 2 E ∼ 140 MeV is σpp,π . 0.0110 cm (see ﬁg 3-4 in Wong 1990). Our own galactic nucleus may emit gamma-rays due to proton-proton collisions in a thick disk due to neutral pions decaying into ∼ 70 MeV gamma-rays (Mahadevan et al., 1997). Nucleons in GRB disks likely have kinetic energies of order 9 MeV. Neutron-proton cross sec- −24 2 tions for 9 MeV are σnp ∼ 0.4 × 10 cm (see, e.g., Peterson et al. 1960, and references therein). 13 −25 2 At Tn,p ∼ 10 K ∼ 860 MeV, σnp ∼ 0.4 × 10 cm . As energies reach zero, the nuclear neutron- −24 2 proton scattering cross section reaches up to σnp,nuc.max = 70 × 10 cm (see Table 3-3 in Wong 1990). Scattering cross sections between the dipole moment of a neutron and electron can be found in Yakovlev and Shalybkov (1990).

A.3 Validity of the Ideal Fluid Approximation

This section summarizes the parameters used to estimate the validity of the ideal fluid approximation. In the approximation of the Boltzmann equation, higher order (² = λmfp/L ¿ 1) terms are associated with, for example, shear viscosity, kinematic viscosity, and thermal conductivity. In the MHD approximation, higher order terms are associated with Ohmic losses and the diffusion of magnetic field. All these effects are generally non-adiabatic and so alter the entropy of a fluid element. See Shu (1992) for more detailed discussions. A standard measure of the importance of viscosity is the ratio (Re), known as the Reynolds number, of the inertial term to the viscous term in the momentum equation, where

v L Re = (A.8) νvisc and νvisc ≡ µ/ρ is the kinematic viscosity associated with diﬀusion of vorticity, v is the relative shear velocity, L is the characteristic length scale, and the shear viscosity coeﬃcient is µ ∼ mvthermal/σ.

164 Thus νvisc ∼ vthermalλmfp gives Re ∼ vdr/(vthermalλmfp). The Reynolds number can also be 2 considered as the ratio of the viscous drag time on the largest scales (L /νvisc) to the eddy turnover time of a fluid element (L/v). The Reynolds number should be measured relative to the comoving frame of the fluid to be consistent with the scattering cross section measurement, and so the only velocity that matters is a relative (or differential) velocity. If the flow is supersonic and λmfp ¿ L, then Re À 1 and so viscous forces are much less important than inertial effects. In order to motivate an inviscid (ideal) approximation for the fluid, the Reynolds number must satisfy Re À 1. As shown in Equation A.8, the limit to such an approximation is the smallest characteristic differential velocity that needs to be resolved over a characteristic length scale. In an accretion disk, one smallest characteristic differential velocity desired to be resolved is the radial differential shear velocity µ ¶¯ dv ¯ v = dv ≡ dr K ¯ , (A.9) dr ¯ (r=rfid) where dr = L is the smallest differential distance to be resolved at r = rfid, vK ≡ rΩK is the Keplerian velocity, and dv is evaluated at fiducial outer radius of r = 40GM/c2, within which fidp this thesis is focused. The nonrelativistic Keplerian velocity is v = GM/r, while the general √ K 3/2 relativistic Keplerian velocity is vK = r GM/(r + a) for black hole spin parameter a for orbits beyond the ISCO. Another useful characteristic velocity is the sound speed v = cs and characteristic length scale corresponding to the disk scale height L = H or L = (H/R)R.

A.4 Validity of the Plasma and MHD Approximations

This section summarizes the quantities used to estimate the validity of the plasma approximation. This discussion of the plasma, MHD, and ideal MHD approximations follows any plasma (Chap- man and Cowling, 1953; Spitzer, 1956; Stix, 1962; Boyd and Sanderson, 1969; Biskamp, 1993; Baumjohann and Treumann, 1996; Treumann and Baumjohann, 1997), astrophysics (Shu, 1992), or electrodynamics (Jackson, 1962) book. The standard plasma approximation assumes that 1) the fluid is ionized; 2) the time scale of interest (T ) is larger than the time scale for electric field oscillations, and the smallest length scale of interest (L) is larger than the electric field screening length (λD) ; and 3) the plasma is weakly coupled. The MHD approximation in addition assumes that the particles behave like a fluid. A plasma is assumed to be at least partially ionized, since otherwise the fluid approximation is sufficient. In thermal statistical equilibrium (and ignoring radiative ionization), the number of ionized atoms can be determined from the fractional ionization by the Saha equation. The thermal ionization fraction (X) for nonrelativistic (i.e. x ¿ 1) hydrogen (H) is found from the Saha equation

2 µ ¶ X 1 3/2 IH = 3 (2πmekbTH ) exp − , (A.10) 1 − X nh kbTH where IH ∼ 13.6 eV is the ionization potential energy. Metals, such as iron, have an ionization

165 energy of Ipe ∼ 7.9024 eV for the outermost electron. At the temperatures of typical accretion disks, the resulting ionization fractions are similar for hydrogen and metals. The disks in all of the systems studied in this thesis are estimated to have high ionization fractions. Macroscopic electric ﬁelds are present in otherwise neutral plasmas due to charge separation, which can lead to plasma oscillations. The square of the plasma oscillation frequency is given by √ 2 2 2 2 2 ωp = ωpe + ωpp = ωpe 1 + R ≈ ωpe, (A.11)

2 2 2 2 where R = me/mp, ωpe = 4πe ne/me, and ωpp = 4πe ne/mp. The oscillation frequency of electrons and protons is identical, but the oscillation (velocity) amplitude of the protons is R times that of the electrons. Thus, the protons are relatively immobile for plasma oscillations. Note that only a warm plasma with proton-acoustic modes oscillates at ωpp. Diﬀerent geometries for the plasma oscillations (such as a spherical charge compression oscillation) give a plasma frequency to within factors of 10 of ωp. Relativistic corrections are diﬃcult to derive analytically, but a numerical solution is easily found. For electronic oscillations reaching Lorentz factor Γ ∼ 18 (estimated for a

GRB-type accretion disk), the fully relativistic plasma frequency is ∼ ωp/4. If T ωp/(2π) À 1, then plasma oscillations can be ignored on time scales longer than T , the shortest time scale of interest by deﬁnition. This is estimated to be true for all the black hole accretion systems studied in this thesis.

While ωp is the plasma frequency for a cold plasma, a hot plasma has a range of allowed frequencies, with a lower limit of ωp. No electric waves can propagate with frequency ω < ωp, and waves with wavelength λ < λD are Landau damped (see below). A relativistic treatment of the plasma waves is necessary if the maximum nonrelativistic velocity of the electrons (v = √ max,e vmax,p/R = x02πωpe/ 1 + R ∼ x02πωpe ∼ x02πωp) gives ∼ x02πωp & c. One might expect that the typical displacement of x0 would be the Debye length that is described below, which then simply 2 suggests that a relativistic correction is needed if vtherm,e & c or kbTe & mec . In the ultrarelativistic 2 2 limit of a hot plasma of electrons and nonrelativistic protons, ωp ∼ 4πe nec /(3kbTe) (see, e.g. Medvedev 1999). Thus, relativistic corrections always lead to a lower plasma frequency. Since the plasma approximation is only valid for large ωp, relativistic corrections should be included to validate whether T ωp/(2π) À 1. Even for the most relativistic ﬂows studied in this thesis,

T ωp/(2π) À 1, so the plasma approximation is valid.

For a hot plasma with electrons moving at speed vtherm,e, a characteristic length scale is the Debye length ωp λD = . (A.12) vtherm,e If there are many electrons within the Debye length scale of a proton, then one can show that the electric field of a proton is screened by the electrons on a length scale of λD (see, e.g., Shu 1992). This sets the outer scale for the Coulomb interaction, while the inner scale is set by the radius, at r = λc, at which particle thermal energy is equal to the potential energy. By symmetry, if electrons screen the field of protons, then protons have screened the field of electrons. At scales

166 much larger than λD the plasma acts collectively, while at scales much smaller than λD the plasma acts as individual charges. For scales less than λD, plasma oscillations are Landau damped due to resonant wave-particle interactions (for a physical interpretation of Landau damping see, e.g.,

Shu 1992, p.401). Thus, if λD ¿ L, then the plasma acts collectively on scales larger than L, the smallest scale of interest by deﬁnition. All the black hole accretion systems studied in this thesis are estimated to have λD ¿ L, so the plasmas in these systems act collectively. The plasma parameter is deﬁned as the reciprocal of the Coulomb logarithm, which is the number of particles Λ with number density ne = np within the screening sphere of radius r = 2λD. 3 3 This gives Λ = 4/3πne(2λD) ∼ 10πneλD. A more careful derivation for a hydrogen plasma gives

3 Λ = 24πneλD. (A.13)

3/2 −1/3 One can show that 4πΛ ∼ (λn/λc) ∼ (λD/λc) ∼ ωp/νc, where λn = n is the interparticle spacing and νc is the Coulomb collision frequency described below. A weakly coupled plasma is defined as having Λ À 1, such that Debye screening is effective and charge neutrality (ne− − ne+ ∼ np) holds. In a weakly coupled plasma, the particle kinetic energies are large compared to potential energies. This is what is typically meant by a plasma. A strongly coupled plasma is defined as Λ ¿ 1, which implies the particle potential energies are large compared to the kinetic energies. A strongly coupled plasma is irrelevant for the plasmas studied in this thesis. Notice 1/2 3 3/2 from Λ = (3/(4πn)) (1/e )(kbTp) that strongly coupled plasmas tend to be cold and dense, whereas weakly coupled plasmas are diffuse and hot. All black hole systems studied in this thesis have Λ À 1, so they are weakly coupled plasmas. For weakly coupled plasmas in which the Coulomb interaction is screened on the Debye length, binary collisions are well-defined. Since Coulomb collisions involve many small angle scattering events, νc must account for the inner and outer Coulomb interaction scales. The typical collision 2 frequency due to binary Coulomb interactions is νc ∼ ν0 ∼ vthermalnλc . A more accurate accounting of Debye screening gives µ ¶ λD νc = ν0 log . (A.14) λc

In summary, a medium is a weakly coupled plasma if Xion ∼ 1, λD ¿ L, T ωp/(2π) À 1, and Λ À 1. As first discussed for the fluid approximation, the MHD approximation of a plasma assumes that collisions sustain a statistical equilibrium space-velocity distribution function of particles on time scales much shorter than T , the shortest time scale of interest by definition (λmfp ¿ L and

T νc À 1).

A.5 Validity of the Ideal MHD Approximation

This section discusses non-ideal MHD eﬀects and the parameters used to estimate whether they are negligible. Non-ideal MHD eﬀects are due to particle collisions. Particle collisions can be modeled

167 via plasma transport coefficients, which result from higher-order expansions of the Boltzmann equation and from a closure relationship between the higher-order terms and the lower-order terms. It is assumed that any higher-order Boltzmann transport properties can be treated as diffusive terms in a relaxation theory (see, e.g., §19.31 in Chapman and Cowling 1953). Below we discuss some of these higher-order effects and the validity of approximations that neglect them. The general multi-component MHD approximation makes no explicit assumptions about the nature of the electromagnetic field or current and charge sources on scales of L or T for L À λD and T À 2π/ωp. However, for a specific derivation of the MHD approximation, the current (J) and charge (ρe) densities must be chosen to be consistent with the electric and magnetic fields as defined by Maxwell’s equations. Several reasonable simplifications can be made, such as to 1) model the protons and electrons (or any other particles) as one component — thereby ignoring drift among species and the inertia of electrons ; 2) treat the plasma as ideal – thereby ignoring the effects of collisions (apart from the fluid approximation itself) ; and 3) neglect the displacement current (as done in the nonrelativistic approximation). These are referred to as the single-component, ideal, and nonrelativistic MHD approximations, respectively.

If np = ne, then the current distribution for protons and electrons is J = enpup − eneue ≡

−eneve, where ve is the drift velocity of electrons relative to the protons. Ampere’s law then gives that the drift velocity of electrons relative to protons to sustain the magnetic ﬁeld is

cB vdrift,B ∼ (A.15) 4πeneL

(Shu, 1992). For all the black hole accretion flows considered in this thesis, the drift velocity is much smaller than any shear velocity. This further supports the single-component ideal MHD approximation. In weakly coupled plasmas, the magnetic field can effectively play the role of collisions by constraining charged particles to magnetic field lines within a Larmor radius. Indeed, there is such a thing as a “collisionless shock” (Tidman and Krall, 1971). A charged particle gyrates around a field line until interrupted by collisions. The gyration frequency for a constant speed around a field line is eB ω = , (A.16) g Γmc which assumes that the field is created by an external current. The gyration (Larmor) radius is

ωg λg = . (A.17) vthermal

The ideal MHD approximation assumes that the gyration radii and gyration time scale of particles are negligible, such that L À λg and T À 2π/ωg. If the magnetic ﬁeld is nonuniform over the length . λg, then the gyration itself produces a net current and the electrons can drift signiﬁcantly relative to the protons. One can show that gyration only produces a net current if the electron inertia is non-negligible, which leads to a break-down in the single-component approximation. In

168 current sheets, the gyration of particles and electron inertia must be accounted for. Ignoring 2 current sheets, the gyration must be considered if T ωgp À (L/λgp) À 1, which is the so-called “finite Larmor radius” (FLR) ordering associated with the so-called FLR-MHD approximation. 2 The finite size of the gyration can be ignored if (L/λgp) À T ωgp À 1, which is the so-called MHD ordering associated with the common MHD approximation. For the black hole accretion flows studied in this thesis, the MHD ordering is estimated to hold. This further supports the single-component ideal MHD approximation. In the following it is assumed that charge neutrality holds, the flow can be modeled as a single- component, and the gyration of particles can be ignored. All that remains to close the equation is a charge current density J. In the most trivial form J = σ0E, as given by Ohm’s law. A so- called generalized nonrelativistic Ohm’s law (see Krall and Trivelpiece 1973, Eq. 3.5.9) includes the effects of a finite current-rise time (∂J/∂t term), Hall-effect (J × B term), and an anisotropic charge pressure – the so-called pressure effect (∇ · P term, where P is the pressure tensor). A general relativistic generalized Ohm’s law in addition accounts for relativistic thermal velocities, relativistic beaming of particles, and is covariant (Meier, 2004). The MHD approximation is often 2 2 used in a form that ignores these effects, where J×B can be neglected if Lωp/ωg,edv/c À 1, ∂J/∂t can be neglected if T ωp À 1, off-diagonal pressure terms can be neglected if Lωg,p/vtherm,p À 1, 2 and ∇pe can be neglected if Ldvωg,e/vthermal,e À 1, where pe is the electron pressure (Krall and Trivelpiece, 1973). By ignoring ∂J/∂t, currents are generated by fields instantly, which is acausal (if only negligibly so in practice). The off-diagonal pressure terms are related to the neglect of the finite Larmor radius, since the protons and electrons feel a different average force due to their different orbital radii. All these effects are estimated to be negligible in the black hole accretion systems studied in this thesis, so the rest of this section discusses a less generalized Ohm’s law. From a single-component negligible-drift mostly-ideal (ideal except for standard Ohmic dissipation) nonrelativistic MHD approximation, one can show that the conduction electron current in the ion rest frame (primed) quantities is

0 0 0 J = −eneve = σ0E , (A.18) or in terms of lab frame (unprimed) quantities is

J = σ0(E + v × B/c), (A.19)

2 where σ0 = nee /(meνc) is the electrical conductivity. One can show that the evolution of the magnetic field, given by Faraday’s law of induction, contains a purely inductive term called the 2 Lorentz effect and a purely diffusive term associated with an electrical resistivity η = c /(4πσ0) = 2 2 c νc/ωp. Associated with the resistivity is the conversion of fluid motion into heat energy called Joule or Ohmic dissipation, which is a similar phenomena as found for viscous fluids. The time 2 scale for diffusion is tD ∼ L /η, so typically in astrophysical (large L) systems, with an otherwise similar resistivity, tD À t. The resistive term ηJ in the generalized Ohm’s law can be neglected if

169 the so-called magnetic Reynolds number

Ldv R ≡ (A.20) M η gives RM À 1, where RM is derived as the ratio of the EMF (Lorentz) to diffusion terms in the induction equation. The magnetic Reynolds number can also be considered as the ratio of the magnetic field decay time (L2/η) to the eddy turnover time (L/v). The black hole accretion flows studied in this thesis have an estimated RM À 1. This further supports the single-component ideal MHD approximation. This simple picture of resistivity is inaccurate in the presence of current sheets, which are considered to be present in, for example, solar flares. In current sheets, the resistivity is suspected to be driven by more complicated processes, such as by plasma or MHD instabilities. These processes are sometimes modeled by a larger anomalous resistivity, generically referred to as Bohm 2 diffusion. Some Bohm models suggest an effective resistivity with ηeff ∼ vtherm/ωg. Some self- consistent mechanisms have been proposed (see, e.g. Drake et al. 1994), but the study of current sheets is an active area of research. A nonuniform magnetic field leads to E × B-like curvature, gradient, and pitch drifts. The pitch drift can lead to the reflection of charged particles due to magnetic mirrors, such as in a magnetic bottle or the Earth’s van Allen belts. Gravity acts as an effective electric field to produce a gravitational drift. The curvature and gradient drift speeds are . vtherm(λg/L), which can be compared to typical flow speed v or typical differential flow speed dv. One can derive so-called gyrokinetic-MHD hybrid models to allow for such effects. These magnetic drift speeds are small in all systems of interest in this thesis since λg/L ¿ 1 and vtherm ∼ v, dv. This further supports the single-component ideal MHD approximation. A partially ionized plasma has neutrals. The drift velocity between neutrals and charged particles due to ambipolar diffusion can be compared to the characteristic smallest velocity (Draine et al., 1983), or it can be estimated by considering the ratio of the ambipolar diffusion to the inductive term in the induction equation (Balbus and Terquem, 2001). By setting the drag force 1 fd = γdrag,i−nρnρivdrift,i−n equal to the nonrelativistic Lorentz force fl = 4π (∇ × B) × B, the drift velocity can be estimated as B2 vdrift,i−n ∼ , (A.21) 4πγdrag,i−nρnρiL where γdrag,i−n = hvrel,i−nσini/(mi + µ), vrel,i−n is the speed of the charged particle i as seen in the rest frame of the neutrals n, σin is the scattering cross section between i and n, µ = ρ/nn is the mean mass per particle, ρ is the mass density, and nn is the number density of neutral particles (see, e.g., Shu 1992). For slow velocities, neutral-ion or neutral-electron coupling oc- −1 curs due to an induced dipole moment in the neutral particle, and then σin ∝ vrel,i−n, so that 13 3 −1 −1 γdrag,i−n ≈ Const. ∼ 3 × 10 cm g s . At large enough drift velocities this effect is dimin- ished and eventually the scattering cross section is dominated by the particle geometry, giving

170 ³ 2 ´ σ ∼ 4π (rn+ri) (|v |). The drift of electrons relative to neutrals can also be estimated in mn+mi drift,i−n (see, e.g., Balbus and Terquem 2001). Despite the fact that some black hole accretion disks studied in this thesis have low ionization fractions, most have negligible drift velocities compared to the characteristic smallest velocity, dv. Only thick (hot) disks modeled for an X-ray binary and some AGN give nonnegligible drift velocities between protons and neutrals, but they are fully ionized so this is not relevant. This further supports the single-component ideal MHD approximation. 11 At Tn,p & 10 K, which may occur in a GRB disk, the nuclear interactions dominate the proton-neutron scattering cross section. At all relevant temperatures, the GRB disk has a large neutron-proton collision frequency and so neutrals do not likely drift. Estimating the GRB disk neutral drift is nontrivial, so this is not estimated precisely in this thesis. Rough estimates show that the protons and neutrons do not drift, although direct electron-neutron drift could occur were it not for the proton-electron electrostatic attraction. Some relevant estimates are provided in Goldreich and Reisenegger (1992), where the neutron star matter cross sections are taken from Yakovlev and Shalybkov (1990).

For (Re, RM ) → ∞, Ohm’s law gives

E + v × B/c = 0, (A.22) which if combined with the equations of motion is referred to as the ideal MHD approximation. The ideal MHD approximation assumes there is no electric field in the frame comoving with the protons. In this case, dissipation of magnetic energy does not occur, and the magnetic field lines are frozen into the fluid (so-called “flux-freezing”). The single-component ideal MHD approximation is estimated to be valid for the black hole accretion disk systems studied in this thesis.

171 B Beowulf cluster

In this appendix, we discuss clustering of computers to achieve an eﬃcient, low cost way of doing massive calculations through parallel processing. Such a computing cluster is referred to as a Beowulf cluster. A Beowulf cluster consists of otherwise independent computers, called nodes, connected by a network. In the simplest setup, all nodes connect to a single switch. For the purposes of access from the outside, another switch with Internet access is used. The parallel computing power of a Beowulf cluster falls somewhere between that of a massively parallel machine, such as the Cray T3D or SGI Origin 2000, and a simple network of workstations with no collective processing. We give a synoptic history of our cluster design in § B.1, discuss performance issues in § B.2, and Internet accessibility in § B.3. In § B.4, we discuss the hardware that comprises a cluster, such as the motherboard, CPU, memory, network, and hard drive (HD). In § B.5 we discuss the operating system (OS), message passing interface (MPI), and how to make a hyperbolic physics code a parallel program. In § B.6 we discuss important tests of a cluster, such as testing CPU performance, reliability, code compiler, memory, network interface, and how our code performs. Finally in § B.7 we summarize the results and conclusions of this appendix.

B.1 History of Design Decisions for our Clusters

A short history of Beowulf clusters can be found at the Beowulf website1. Several factors influenced the desire to build clusters of computers for parallel computation using commercial off-the-shelf (COTS) products, which includes: 1) in the early 1990s, the performance per unit price of individual computers became comparable to classical supercomputers, for which the CPUs obtain each other’s data via proprietary communication technology rather than what is classified as network technology; 2) in the mid-1990s, computer hardware and the Linux OS became highly reliable; 3) in the mid- 1990s, committees formed to promote and develop parallel communication technology, such as the MPI2 (Message Passing Interface) Forum; and 4) developers at Argonne National Laboratory3 (ANL) and Mississippi State University (MSU) developed MPICH4, a portable implementation of MPI. Parallel processing on a cluster of computers became both practical and more affordable than supercomputers. Computer centers such as NCSA began experimenting with Beowulf clusters in

1http://www.beowulf.org/beowulf/history.html 2http://www.mpi-forum.org/ 3http://www.anl.gov/ 4http://www-unix.mcs.anl.gov/mpi/mpich/

172 the year 2000 with a cluster called Posic consisting of 550Mhz Intel Pentium III systems. In 2001, NCSA created a production cluster called Platinum consisting of 1Ghz Intel Pentium III systems (now retired by Tungsten)5. The typical lifetime of a Beowulf cluster has been about 2 years. NCSA offers a large amount of resources that can be obtained by submitting a proposal through an allocations board6. Based upon one’s request for computer time, they decide how much computer time one should be given. For example, in 2002 we obtained 100,000 CPU-hours of computer time on NCSA’s Platinum cluster of 1GHz Intel P3s, and in 2003-2004 we obtained 300,000 CPU- hours on NCSA’s Tungsten cluster of 3Ghz P4 Xeons (each Tungsten time unit is worth about 5X that of a Platinum time unit). An NCSA allocation expires after about a year from the point of activation. Using an NCSA cluster removes the technical hassle associated with computer hardware and OS details. The major problem with supercomputer centers such as NCSA is they often explore bleeding edge computer science and make it difficult to perform bleeding edge science. For example, the NCSA Tungsten cluster was scheduled to be ready October 2003, but it was not in production mode until June 2004. The primary reason for the delay was their attempt to use a new file system called Lustre7, which turned out to have bugs. The purpose of Lustre is to replace NFS (network file system)-type technologies for distributed file systems. Since 1999 we have investigated Beowulf clusters as platforms for high-resolution astrophysical simulations. Our ultimate goal was to perform general relativistic accretion flow simulations using the to-be developed HARM (see chapter 3). In comparison to computer time on an NCSA cluster, a local cluster is more easily accessible, allows us complete control over what programs are installed, and is self-managed. Self-managing a cluster gives the user “root” privileges, allowing the user to control the basic Linux environment, such as the version of compiler, MPICH, and other utilities that cannot be installed by a user on an NCSA cluster. We also considered a local cluster useful for test simulations before using NCSA computer time to perform production simulations. NCSA computer time can be used rather quickly when using many CPUs, and mistakes can be costly. With a small budget one can create a cluster that rivals a large NCSA cluster in the quantity of CPU-hours over the period of a year available for our private use. The number of hours in a year of computing time as allocated by NCSA is 8760. For an equivalent of 100,000 CPU-hours on Platinum, we would require 6 dual-CPU 1Ghz P3s. This is quite inexpensive. For an equivalent of 300,000 CPU-hours on Tungsten, we would require about 17 3.06Ghz dual-CPU nodes. The largest expense of a Beowulf cluster is the initial cost of hardware and associated envi- ronmental expenses (such as cooling units and power circuits). Much time can be spent installing the OS. As we discuss below, some Linux distributions essentially eliminate the complications and time needed to install and upgrade the OS. The primary expense of money and time after the cluster is built is replacing failed node components. In particular, HD failure is the limiting factor in determining the number of nodes to purchase. In our experience, HDs fail the fastest of any

5http://www.ncsa.uiuc.edu/UserInfo/Resources/ 6http://www.ncsa.uiuc.edu/UserInfo/Allocations/ 7http://www.lustre.org/

173 hardware component, at a rate of about 1%8 per year for mechanical failure, and based on our experience, the failure rate is about 5% per year for partial data loss requiring reinstalling the OS. If we assume it would be reasonable to reinstall a node’s hard drive about once per month and each node has one HD, then the largest cluster one should purchase is 240 nodes. If one is only limited by one mechanical failure per month, then the largest cluster one should purchase is 1200 nodes. This gives an operational cost of about $200/month for the new HD, plus time and effort to install the new HD. Below we discuss the purchase of a 16 node 2.4Ghz P4 Xeon cluster, that together with our collaborators will soon (as of this writing) expand into a 28 node Xeon cluster. We should expect about one HD failure per year resulting in loss of data that requires reinstalling the OS. The current cost per node is about $1600 and network equipment about $5000. This will give us a total number of Tungsten-equivalent CPU-hours of about 192,000 for about $50,000 of NSF funds (equivalent to about 3 graduate students for one year). This equates to $0.26 per Tungsten CPU-hour. We would require about $78,000 to obtain about 46 nodes to have 300,000 Tungsten CPU-hours per year. Before discussing our cluster in more detail, we first discuss the history of our older clusters. We then discuss the history of our decision making for our Xeon-based cluster. Our first cluster was based on 2 dual-CPU Alpha-based nodes. At the time, Alpha CPUs were considerably faster than Intel CPUs (Pentium III at 550Mhz). Despite the high price of the Alpha systems, 2 nodes are much easier to manage than the equivalent 6-8 dual-CPU Intel nodes needed. Fewer nodes also allows for higher performance for network intense operations. At the time we chose a Myrinet9 network technology running at 1000Mbit/s, which was significantly faster than the relatively inexpensive Ethernet technology that ran at 100Mbit/s. The cluster operated as expected, so we added an additional node when funding became available. By choosing the most expensive cluster components, the cluster was shielded from the performance uncertainties associated with low-end technology. However, the Alphas were sufficiently high-end to be cumbersome in Linux. For example, they were difficult to upgrade to new Linux versions and incompatible with several programs. This 3-node Alpha cluster served us for about 2 years. In 2001, we received additional funding to purchase a new cluster consisting of 3 2.0Ghz dual- CPU Intel nodes connected by Myrinet and gigabit Ethernet, where the Myrinet boards from the Alphas were used in the Intel systems. The nodes were named Alphadog, Gravitas, and Horizon, so the cluster was called the AGH cluster. The Intel-based systems were easier to manage than the Alpha systems due to the lack of full Linux support for Alphas, although the 2-year old Alphas had comparable performance. This system served us for about a year as a 3-node cluster, and for the past 1.5 years has been the backbone of our Xeon cluster described below. In 2002, our group and collaborators received $50,000 in NSF funds to purchase a computer cluster to perform simulations of relativistic magnetized gravitating flows. Given our group’s prior cluster experience we decided to split the $50,000 between the 2 groups. Our group forged ahead

8Typical manufacturer reported HD annualized failure rate 9http://www.myrinet.com/

174 and developed a cluster while our collaborators researched how their code performs on such systems. This gave us $25,000 to purchase a computer cluster. We originally considered simply purchasing systems as in the AGH cluster, but we first decided to study what systems would give the highest performance per unit cost. We thought it plausible to test whether single-CPU Intel Pentium 4 (P4) systems with either gigabit or Myrinet would perform as well as the AGH nodes. This would be advantageous since the P4 nodes were about 2-3X lower in cost than each node in the AGH cluster. The P4 test cluster could also be tested against the AGH cluster, and any gigabit or Myrinet board can be swapped between the P4 and AGH cluster for comparison. We purchased a test-cluster of 4 P4 nodes with the hope that they would eventually form part of a larger cluster, where in the worst case scenario these nodes would become new workstations. We tested the P4 nodes with 32-bit 33Mhz 100Mbit Ethernet onboard (82801 BA/BAM/CA/CAM), the AGH cluster nodes with 64-bit 133Mhz gigabit Ethernet onboard (82544EI rev2), P4 and AGH nodes with Myrinet boards (LANai 7 version PCI64A and M2L-PCI64B-2, 2MB boards, 66Mhz), and 4-5 different gigabit boards provided by our supplier, SWT10. These gigabit boards included the 32-bit 33Mhz Intel Pro/1000 MT Desktop (Intel 82545EM rev01), 64-bit 66Mhz Intel Pro/1000 MT Server (Intel 82540EM rev02), and a 64-bit 66Mhz 3com 3C996B-T (Broadcom NetXtreme BCM5701 rev 21). We did not really consider buying expensive gigabit boards (>$200), but we wanted to understand how such boards performed. Chips on the motherboard are generally much less expensive than add-on boards. Onboard chips can often provide increased performance due to an efficient design and because they are integrated with the latest features included on the motherboard. Myrinet at the time cost $1200 per board, and a Myrinet switch for 16 hosts cost $5000. Today the Myrinet boards are 50% cheaper, but the switches cost about the same. The gigabit boards range from $50 for the Intel Desktop to $500 for the 3COM board. We purchased an 8-port non-blocking switch for $700 in order to test gigabit, a Linksys EG000811. We compared all permutations of boards, systems, and number of processors involved in a calculation using MPI bandwidth/latency benchmarks, a ZEUS-based code, and HARM in order to test system+network performance. We found that the AGH cluster using its onboard gigabit performed the best. The AGH cluster using onboard gigabit even outperformed Myrinet in dual- CPU HARM calculations using all 3 nodes and 6 CPUs. We found that the P4 cluster with any board gives about 45MB/s peak bandwidth, whereas the AGH cluster gives about 90MB/s with any 64-bit board. The P4 systems with any gigabit board obtained a latency of ∼ 70µ s for transferring small message sizes, Myrinet obtained ∼ 9µ s, the add-on 32/64-bit 66Mhz gigabit obtained ∼ 50µ s on AGH, and the AGH onboard gigabit obtained ∼ 50µ s. HARM performed best on the AGH onboard gigabit by passing relatively large blocks per unit of computation. In such a case the latency of ∼ 50µ s vs. ∼ 9µ s for Myrinet is indistinguishable. The ZEUS-based code with many more small communications per computation performed best on Myrinet. Note that MPI tests

10http://www.swt.com/ 11http://www.linksys.com/products/product.asp?grid=29&prid=407

175 on 64-bit 66Mhz systems show year 2004 Myrinet technology to achieve about 225MB/s and small message size latency of about 8µ s12, so its plausible that Myrinet could offer a significant advantage over gigabit today. However, today’s 10Gigabit technology may counter that peak bandwidth13. Despite the likely higher peak bandwidth of 10Gigabit, the latency will still be bounded by the TCP overhead of standard Linux drivers. The lower P4 performance is likely due to the 32-bit 33Mhz PCI bus14,15. As of this writing no single-CPU Intel workstation motherboard has a faster bus than 32-bit 33Mhz. The 32-bit 33Mhz Intel Desktop gigabit performed poorly in an AGH node compared to 64-bit 66Mhz gigabit boards, indicating that the 32-bit board, rather than the rest of the system, was limiting the communication. The onboard AGH gigabit likely performed better than the add-on 64-bit 66Mhz gigabit boards because the AGH cluster nodes have a 64-bit 133Mhz PCI-X bus connection to the gigabit, and the built-in design allows motherboard manufactures to design an optimal interface that allows the onboard gigabit to operate as efficiently as possible at the full speed of PCI-X. Based upon these results we considered purchasing systems similar to AGH as we had originally intended. However, around this time in early December 2002, we learned of a soon-to-be-released type of motherboard that would have all the technical benefits of a server type motherboard that is typically an ATX-E (extended) form factor16, but would be compact in ATX form factor like workstation motherboards. The so-called Value Server motherboard17 (at the time the Tyan Tiger i7501) costs about the same as a high-end workstation motherboard ($250), while costing about 1/2-1/3 as much as the typical server motherboard18(at the time the Tyan Thunder i7501). There is always an uncertainty about performance for untested systems, so we had doubts that this was a proper choice. The AGH motherboard was from Tyan and worked great, the Value Server motherboard was also from Tyan, the Value Server motherboard is actually a similar model to the full server motherboard, and both contain very similar components. In particular, the onboard Intel gigabit chip was only slightly different and supposedly improved from the AGH gigabit chip. Thus, we decided to take the risk. This allowed us to obtain two more nodes due to the lower cost per node. In the worst case scenario these new nodes would be turned into a computer farm19. We decided to obtain about 12 nodes to add to the AGH cluster for a total of 15 nodes20. The delivery time for providing the systems was delayed a few times due to the lack of product availability, but we finally had the systems by the beginning of February 2003. A gigabit network switch, as opposed to a network HUB, is required for fast communication between nodes. A switch communicates to a network board only that information meant for it

12http://www.myrinet.com/myrinet/performance/MPICH-GM/index.html 13http://www.10gea.org/index.htm 14http://www.compute-aid.com/64bitpci.html 15http://www.intel.com/design/bridge/ 16“Form factor” denotes the style and size of the way in which the motherboard attaches to the computer case. 17http://www.tyan.com/products/html/tigeri7501.html 18http://www.tyan.com/products/html/thunderi7501.html 19Most of the cluster’s computer cycles have been used for single CPU processes since its activation. 20At the time (Aug 2004) of this writing we have added 3 more new nodes, and we have removed 1 older AGH-type node to serve as a web server

176 based upon its IP address. A HUB performs no such routing and every network board receives data meant for all nodes. High-end switches are typically non-blocking, which means that the communication is never limited by the switch and all network boards can achieve maximum speed. We originally planned on purchasing an HP 4108gl 21, but we required a gigabit switch that followed the always changing University of Illinois regulations as determined by the Network Design Office 22 (NDO). The typical function of the NDO is to design a network that suites the needs of University staff that fits with the NDO’s idea of what constitutes appropriate networking equipment. They knew of problems with the HP 4108gl that could cause network failures for certain devices on the University network. After providing the NDO with the desired purpose and design of our network, the NDO agreed on the HP 5308xl 23 switch, which we ultimately purchased for the new cluster. This switch is composed of a chassis and up to 8 modules. The chassis holds the modules, contains the technology to allow all the modules to communicate, and powers and cools the entire switch. The modules contain the technology to handle a specific type of network connector, such as copper or optical fiber. The cost of the switch chassis (model J4819A) was $2036. The gigabit copper modules with 4 ports each (HP Procurve Switch XL 100/1000-T module (J4821A)) were $746 each. For 15 nodes and one Internet connection we required 4 modules, but purchased 5 modules since we would likely add more nodes later. The final cluster element was the rack to hold the system. The purpose of the rack is to save floor space by stacking vertically while still allowing easy access to each node. We decided to buy a metal rack from Bed Bath and Beyond from InterMETRO24. Each unit supports 800lbs and each of the 4 shelves (including top) supports about 200lbs. We purchased a unit that would contain 12 new nodes and the switch on one shelf, while the Alpha cluster and 3 older AGH nodes would be on another shelf. Each new node is about 30lbs, each Alpha about 70lbs, old AGH nodes about 50lbs, and the switch is about 30lbs. This gives 360lbs for each shelf, so the shelving is more than sufficient. With the equipment to be purchased settled, we set out to determine whether there was sufficient power and cooling for the new cluster in the Astronomy building’s computer room. We needed to support our old Alpha cluster as well. One can determine the peak power required for a given type of equipment using the APC calculators25 online or estimate it from individual specification sheets. A typical new or AGH-type node consumes 270Watts of power, the HP 5308xl gigabit switch consumes about 620Watts, and the 3 old Alpha systems consume about 370 Watts each. With an efficiency factor of 1.33 for Watts to VA, we obtain ∼77A for a 120V line. Thus we required a total of about 4 20A electrical circuits. During this investigation it was important to determine the electrical layout and total power available for our system. The electrical outlet and hardware components in the Astronomy Building

21http://www.hp.com/rnd/products/switches/switch4100glseries/overview.htm 22http://www.cites.uiuc.edu/commtech/ndo.html 23http://www.hp.com/rnd/products/switches/switch5300xlseries/overview.htm 24http://www.metro.com/consumer/index.cfm 25http://www.apcc.com/template/size/apc/index.cfm

177 Figure B.1 Astronomy Building computer room electrical layout as of Feb, 2004. BH is located north and center. This excludes the addition of another new node for BH and the collaborating group’s cluster of 12 nodes. computer room as of early year 2004 are shown in Figure B.1. We determined that the addition of our computers required rewiring 2 240V 30A circuits into 3 120V 20A circuits. Often older computers used 240V since higher power consumption is more efficient at higher voltages. Before making these rewiring changes we tried putting the cluster into operation by using any available outlets. In particular, 1 20A outlet was connected to a set of 8 nodes, and we experienced power failures when fully using the cluster. The peak requirement for 8 nodes is 24A, so we were simply underestimating the likelihood of the peak power use. This was temporarily corrected by further distributing the power over the outlets with remaining capacity until the 240V 30A outlets were converted to 120V 20A. The power in Watts consumed also gives the heat generated, where the typical cooling unit is quantified in how many tons of cooling it provides, where 1 ton is 12000 BTU/hr and 1 Watt is 3.413 BTU/hr. Thus we required about 1.6 tons of cooling. The items already in the machine room included 7 300Watt machines, 1 switch at 1600 Watts, and lighting at about 300 Watts for a total of 3700 Watts. Thus the total peak cooling required was about 2.7 tons, while the cooling unit at the time was only 2 tons provided by a Liebert Challenger 226. Thus in principle we did not have sufficient cooling. In practice these are peak cooling requirements not often necessary except for

26http://www.liebert.com/support/training/env/manuals_obsolete.asp

178 extended periods of maximum use of all systems. Essentially the heating of electrical components is much more extended in time than the electrical demand. Thus, while electrical requirements are strict for any moment in time, only prolonged periods of maximum use require maximum cooling. The cooling unit handled our needs, and we have never had an overheating incident since the cluster was installed. Later additions by us and our collaborators in mid-2004 increased the need for power and cooling. After accounting for the addition of 3 new machines by us and a future addition of 12 nodes (attached to our switch) by our collaborating group, the total required cooling is 4 tons for the entire machine room. This required the purchase of an additional 5 tons of cooling provided by a Liebert Challenger 300027. The new 5 ton unit will supply the entire computer room, while the older 2 ton unit will operate only when the 5 ton unit fails. This allows time to shut down computers properly. These additional computers also required 3 more 20A circuits. The total estimated cost for this cooling+electric upgrade is $28,500. Our new cluster is called BH, an acronym for Black Hole. A basic schematic of the BH cluster is shown in Figure B.2. A digital photograph taken soon after the OS, software, and MPI were installed is shown in Figure B.3. BH currently has 17 nodes where one is used primarily for compilation of code, initiating parallel programs, and services such as a web server.

B.2 Cluster Performance and Advanced Network Drivers

Soon after BH was set up, we measured the latency and peak bandwidth for the new nodes and were disappointed that the peak bandwidth was only 60MB/s and the latency was 60µ s. This realized our worst case scenario that the nodes would be turned into a computer farm. This outcome was a result of impatience and the desire to avoiding wasting money by purchasing more test units that we would not use in a cluster. We considered methods to optimize the network performance. We hypothesized that the default Intel drivers were not properly dealing with the new chip in the new BH nodes. We could use specialized network drivers or specialized Linux OSs with modified kernels to improve performance. Some of the projects or companies out there that offer related services include GAMMA28, EMP29, M-VIA30, Scyld31, Score32, MPI/Pro33, and Scali34. Some of these projects or companies offer specific Linux distributions with modified kernels, some offer highly specific Linux distributions with management software and no modified kernel or drivers, and others offer specialized Linux drivers to be used with any Linux distribution. As far as speed is concerned, the specialized Linux

27http://www.liebert.com/dynamic/displayproduct.asp?id=542&cycles=60Hz 28http://www.disi.unige.it/project/gamma/index.html 29http://www.osc.edu/~pw/emp/ 30http://www.nersc.gov/research/ftg/via/ 31http://www.scyld.com/ 32http://www.pccluster.org/ 33http://www.mpi-softtech.com/ 34http://www.scali.com

179 NODES BH00-BH16

17NETWORKCABLES 1NETWORKCABLE

CLUSTERSWITCH INTERNETSWITCH

Figure B.2 BH Beowulf cluster primary elements.

180 Figure B.3 Digital photograph of BH cluster.

181 drivers offer improvements by including an OS-bypass or user-level communication drivers. The typical Linux network driver requires support of TCP and other archaic OS-level protocols that are not optimal for intensive communications. These OS-bypass drivers separate their activity from the Linux kernel and only provide special services such as a protocol for MPI communication. Another feature that improves network efficiency is called zero-copy. This feature allows the network to run without interaction with the CPU in processing memory buffers. Today, many companies, such as Intel, are including the zero-copy feature in Linux network drivers. These features are included in the commercial product Myrinet when using their GM drivers. The most interesting driver modification is GAMMA, which obtains a latency of 11µ s and 113MB/s on a Server-based motherboard 1Ghz P3 cluster, exceeding the performance of Myrinet at the time. Donald Becker, master of the networking universe, father of the Beowulf cluster, and chief technical officer & founder of Scyld, considered GAMMA the most promising technology at the time. However, this seemed an unreliable option since many projects such as GAMMA have a very short life (e.g. Giganet-VIA, ServernetII-VIA, InfiniBand, U-Net, AM II, LPC, PM, FM, GigaE- PM, and BIP are all dead) and were no longer being worked on at the time of our investigations. Specifically, GAMMA supported different devices and obtained quite excellent performance on an optical-based Netgear GA621, but by the time of this investigation we already purchased a copper- based gigabit switch. The successor to the GA621 with a copper connection called Netgear GA622 actually includes an unrelated chip that performs poorly by comparison. The M-VIA project only supported one still-available expensive $500 board (SysKonnect35). The EMP project seemed to be quite interesting from a performance perspective, however few chips were supported and the network boards were unavailable or expensive. The typical problem for these programmers is that by the time their specialized driver is written for a chip, the manufacturer no longer produces the board with that chip. Often a single board will have many revisions of the same chip with entirely different, incompatible functions that have to be reprogrammed (or re-hacked). Support from manufacturers for Linux driver developers has always typically been difficult to obtain due to the secrecy associated with corporate technology. Typically, chips are hacked by programmers until the drivers work, but driver development has not kept up pace with chip revisions. Clearly this process leads to unreliable support, where the obsolescence of a chip can mean the death of an associated project. Only companies that develop both the hardware and drivers with OS-bypass (such as Myricom) can offer reasonable reliability and stability to the user. Another problem is that these OS-bypass related drivers and OSs are generally non-trivial to install and only come with obscure instructions. Drivers such as GAMMA come with many restrictions on how the network can be used due to limitations in the state of the development of the specialized driver or OS. Discussions with Donald Becker regarding OS-bypass provided a historical insight. He says it took about 20 years to “get TCP right”, so likely it will take a while before OS-bypass or a related technology is cheap and reliable.

35http://www.syskonnect.com/syskonnect/performance/gig-over-copper.htm

182 We did communicate with Marc Ehlert who designed GAMMA and he used our ZEUS-based Newtonian 2D MHD code to test GAMMA on his 1Ghz P3 cluster. GAMMA obtained no better performance than default Linux drivers with our code for a typical problem and resolution. It is possible that a faster CPU would make better use of the higher bandwidth and lower latency GAMMA offers. Ultimately, the problem with GAMMA was that it only supported single-CPU nodes. Donald Becker, who has been a network driver programmer for several network chips, reports to us that the SMP (Shared Memory Processor) issue is a critical reason why OS-bypass has not been successfully implemented (except at the time by Myricom with Myrinet). Apparently the driver must be much more sophisticated to be compatible with SMP. Since most Beowulf clusters take advantage of SMP nodes, this is a large conflict. At the time of this writing the GAMMA project does support dual-CPU nodes, but still only supports the NetGear GA621 for gigabit. Ultimately we got “lucky” and found a large increase in performance on BH by using the advice of Intel and Tyan regarding our performance problems. Tyan explained that the newer gigabit chips were actually not malfunctioning, but the Intel driver was flawed for the newer chips, which was a known issue to Intel. They suggested 2 trivial changes to the “/etc/modules.conf” file to pass parameters to the Intel driver. This includes the “InterruptThrottleRate=0” and “TxIntDelay=0” lines. We are currently using Intel driver “e1000-4.4.19”. After incorporating this change the AGH- type node latency improved from 50µ s to 30µ s, and the peak bandwidth changed from 90MB/s to 83MB/s. The new Value Server system latency improved from 60µ s to 27µ s, and the peak bandwidth improved from 60MB/s to 100MB/s.

B.3 Cluster access to Internet

The University of Illinois at Urbana-Champaign (UIUC) provided BH with a range of private IP addresses. A private IP address can access the UIUC network, but not the Internet outside. Private addresses were provided due to the scarcity of addresses available on the UIUC domain. Private addresses also enhance security by not providing “Internet intruders” with any means to connect. To use the private IP address we connect our gigabit switch to the Astronomy Department network switch, which connects to the UIUC network. In order for part of BH to be visible to the Internet we also obtained a single public IP address. One computer (BH00) is connected to the Internet by directly connecting one of its network interfaces to the Astronomy Department network switch. This single computer can thus provide web, ftp, ssh, and any other service to computers on the Internet. Actually, the Internet related services are not required to run on BH and certain programs can be moved to other nodes to obtain a higher level of security for BH. Security must be maintained by updating the OS when patches that remove vulnerabilities are available. The ﬁnal basic set of connections made in a cluster is between every node in the cluster and the cluster switch. For BH, we have chosen a simple star-type network topology that provides maximum performance per unit cost.

183 B.4 Hardware

The hardware in a cluster must be reliable, relatively stable against new products, high performance, and all the equipment must work together seamlessly. A typical system primarily consists of a motherboard, CPU, memory, network interface, and hard drive. Figure B.4 is a cluster block diagram for the BH cluster and includes all the important elements of a cluster that we discuss in this section, along with the size and bandwidth of the pipe between each chip and the network components. The motherboard forms the backbone of each node by providing a means for communication between various types of hardware. Chips on the motherboard communicate through many diﬀerent protocols at various speeds to other chips on the motherboard. A motherboard typically contains those hardware chips that are relatively inexpensive (compared to additional hardware), commonly in use, and have no need to be upgraded. The primary chips on the motherboard are called the chipset. This is comprised of some number of chips that coordinate activity between any secondary chips on the motherboard, the CPU, memory, and other additional hardware. The BH nodes use the E7501 chipset36. The chipset is primarily composed of 3 chips: 1) Memory Controller Hub (MCH) ; 2) I/O Controller Hub (ICH) ; and 3) PCI Controller Hub (PCH). The MCH is the spinal cord of the nervous system and coordinates activity between the CPU, ICH, and PCH. Secondary chips could include hard drive controllers, hardware monitoring, external I/O controller, PCI controller, and BIOS. Today’s motherboards often incorporate other chips that would otherwise be added as a PCI board. This includes a video controller or one or more SCSI (advanced hard drive) controllers. The motherboard used for the new BH nodes is the Tyan Tiger 750137. The brain of a computer is the CPU, which interprets and executes program instructions that can be operated on program information. The new BH nodes have 2.4Ghz Intel P4 Xeon CPUs38 with a 512KB L2 cache. In today’s typical system, the set of CPUs communicate on a single bus called the front side bus (FSB). Each CPU in a dual-CPU system communicates to other CPUs or the MCH using this bus that operates at 400 - 533Mhz for the BH cluster systems. The FSB protocol is 64-bit and thus the peak bandwidth is 3.2GB/s - 4.27 GB/s, respectively (each period of the cycle contains 64-bits of data). The CPU itself has memory units called L1, L2, L3 caches (L stands for level). The L1 cache is the smallest and fastest cache. Progressively higher numbered L caches are larger and slower. The purpose of a CPU cache is to temporarily store program instructions and information. In the BH cluster the L1 cache is a mere 8K, but operates at 16.4GB/s. The L2 cache is 512K and operates at 14GB/s. The size and speed of the CPU caches are determined by cost eﬀectiveness for a given amount of performance as predicted by market prices for memory. There are several advanced features included in various CPUs. The Intel and AMD lines of CPUs include SIMD (sin-

36http://www.intel.com/design/chipsets/e7501/index.htm 37http://www.tyan.com/products/html/tigeri7501_spec.html 38http://www.intel.com/products/server/processors/server/xeon/index.htm

184 Figure B.4 Cluster block diagram of BH cluster. Figure shows bandwidth between elements on motherboard and switch.

185 gle instruction, multiple data) type instructions that allow for a single instruction, which executes every fixed number of CPU clock cycles, to operate on multiple data. This reduces the overhead associated with instructions and allows integer and floating point operations to be performed 2X faster for instructions operating on 2X the standard data size. Another feature of relevance for clusters is Hyper-Threading. This feature is meant to increase CPU efficiency by having a single physical processor support up to 2 thread (or processes). This improves performance when multiple programs compete for CPU time in a multi-tasking environment. However, typically in a cluster environment one only wants to have 1 process per CPU, so this is not typically beneficial. Some benchmarks with 1 process per CPU show that Hyper-Threading actually slows down per- formance39. We find no difference in performance for our codes running 1 process per CPU with or without Hyper-Threading. The main memory of a computer serves as a relatively fast repository of program data. The main memory is essentially an extension of the CPU caches but operates over the FSB bus through the MCH. A typical Intel server based motherboard has dual-channel memory40, which means the bus connecting the memory to the MCH has 2 independent channels for 2 sets of memory chips. This allows the 2 CPUs (or any other device) to independently operate on those 2 sets of memory chips. Each channel of memory in each node of the BH cluster has memory that communicates to the MCH in 64-bit chunks at 266Mhz for an effective total of 128-bit memory. The bus between the MCH and memory is actually 144-bit, which includes 16 bits of error correction encoding to guarantee the data is accurate. The memory used in the BH nodes is called PC2100 DDR. The PC refers to personal computer and the 2100 refers to 2100MB/s, which is the approximate effective peak bandwidth in MB/s per channel. Memory will continue to increase in bandwidth, but historically the large increases in memory bandwidth often do little for overall computational performance. Most scientific codes do well to make optimal use of the CPU caches. The network provides the means to transfer data between all the nodes. Some classes of problems solvable by a computer can be solved on a cluster of networked nodes in order to model the function of a single computer. Any problem that can be broken into a collection of localized problems, which can each be efficiently solved on one node, can benefit from a cluster of nodes in a network. While there are many ways of connecting a network that can be specialized for some classes of problems, the most general topology for one connection per node is simply to connect the node to a network switch. In this case it is best if the network switch is non-blocking. A non-blocking switch allows every port to communicate to every other port simultaneously, bidirectionally at maximum speed. Current network technology is fastest per unit cost for gigabit networks. Gigabit network chips are often incorporated into server-type motherboards which greatly lowers the cost. A gigabit network chip has a peak bidirectional bandwidth of 0.25GB/s. One can potentially increase the bandwidth between any 2 nodes by giving a node another

39http://www.Linuxclustersinstitute.org/Linux-HPC-Revolution/Archive/PDF02/11-Leng_T.pdf 40http://www.cpuplanet.com/features/article.php/1587771

186 network connection. However, one has to determine if the CPU, memory, and PCI bus have sufficient bandwidth to make this worthwhile. Generically, the CPU and memory are much faster than the PCI bus, and thus the PCI bus is the bottleneck. However, the true measured bandwidth can significantly differ from theoretical estimates, so in practice one should do experiments on various hardware setups. For example, Intel P4 motherboards with a 32-bit 33Mhz PCI bus have a peak bandwidth of 133MB/s, while on that bus the gigabit network performance is typically 45MB/s independent of the hardware. Older server systems have a 33Mhz 64-bit PCI bus which gives a peak bandwidth of 264MB/s, while peak gigabit performance is about 90MB/s. Newer Myrinet and 10Gigabit technologies can reach near the theoretical limits of a 66Mhz 64-bit bus operating at a bidirectional total of 440MB/s. For motherboards that have PCI-X technology, which is a standard 64-bit bus at 133Mhz, the peak bandwidth is about 1GB/s. Thus raw estimates would conclude that the PCI-X bus has sufficient capability to handle up to 4 gigabit network chips. The actual measured bandwidth of a gigabit network chip on the PCI-X bus is not measurably different from the 64-bit PCI bus at 33Mhz at 90MB/s. Actual hardware never operates at peak bandwidth for a variety of reasons, such as network drivers and true hardware operating conditions. As described in Section B.1, we did consider advanced drivers that eliminate the overhead associated with the CPU interaction and OS. The GAMMA based driver can achieve a much lower latency and a bit more peak bandwidth than default Linux drivers. However, we found this option to be unreliable and impractical due to several limitations of the current technology and scarce support for existing hardware. A gigabit switch connects all the nodes in a cluster. A network switch should be non-blocking for maximum performance. A non-blocking switch never limits the traffic between each node by processing all routing at sufficiently high rate so data is never stalled. Other features include hardware support for jumbo-frames. The typical mean transfer unit (MTU), or smallest message size, is no more than 1500 bytes. Jumbo-frames support typically allows up to 9000 bytes. This value is the maximum at which the standard TCP 32-bit CRC error correction check loses effectiveness against data loss. However, this is only useful for very long message size data transfers not typical of all computing done on clusters. The BH cluster uses an HP 5308xl non-blocking switch and obtains up to 9.6GB/s bandwidth on the so-called “backplane”. This describes the connections behind all the ports. Each port input can go to 1 port output, and each pair of input/outputs uses 0.125GB/s of bandwidth. Thus, theoretically the switch could handle up to 76 ports before performance is inhibited. Currently the BH cluster uses 17 ports and our collaborating group will soon require 12 more ports, yet this will still be far from over-utilizing the switch. Unlike computer OSs, network switches are very efficient at network processing due to their proprietary, optimized OSs. Older forms of the gigabit network technology exist, such as the 10Mbps Ethernet that theoretically has a peak bidirectional bandwidth of 2.5MB/s, but actually only obtains (under any OS or driver known to the author) about 0.5MB/s. Likewise, a 100Mbps Ethernet network theoretically

187 could obtain a bandwidth of 25MB/s, but achieves about a peak of 10MB/s under a broad number of OSs, hardware, and drivers. We use hard drives for both immediate simulation diagnostics and storage of old simulations. Each node has the capability to hold 4 HDs on the EIDE channel, and by distributing the HDs among the many nodes the effective bandwidth is parallelized and the reliability is enhanced compared to a single large hard drive system. The hard drive used in computer clusters can be a weak point in terms of reliability, and can be critical since failure can mean loss of data – generally the most important item in any cluster. The typical HD manufacturer specification sheets report a rate of failure using the mean time between failure (MTBF), the number of bits before 1 bit is read or written with errors, the annualized failure rate (AFR), and the “minimum” design lifetime of a drive. At the time of this writing, the typical high performance EIDE HD has a MTBF of 500,000 to 1,000,000 hours, a bit error rate of 1 out of 1013 − 1015, an annualized failure rate of about 1%, and a “minimum” design lifetime of 5 years. The MTBF and bit error rate only indicate HD error in reading or writing and assumes the drive does not otherwise fail catastrophically, such as from bearing failure (often associated with clicking sounds). The AFR is most representative in that it reports the actual rate of return due to failure by customers. Over the 4 years of dealing with approximately 30 HDs (in an arbitrary number of computers), we have had about 6 HD failures, where about 2 have been terminal (i.e. not restorable after attempting a reformat). If this were representative of HD failure rates, this represents a 5% chance that in a year a HD will fail with loss of data. Our experience of terminal failures is a ∼2% annual failure rate. The reported failure rate by manufacturers for actual used drives and returned due to any problem is 1% per year. Perhaps our luck is 2-5X worse than average, but more likely our problems are associated with obtaining the latest technology since we are interested in purchasing the new, larger hard drives. Also, we likely accessed our HDs more frequently than the average user. In summary, we suggest replacing drives that are still alive every 5 years and expect about 2-5% to die each year. Making backups to other HDs or having HDs perform some form of redundancy of data is critical. We have often considered getting a tape backup system, but have always ended up buying a HD due to their simplicity, general robustness, and low price. Tape backups are generally more robust than hard drives, but are of course sensitive to a magnetic field. The author has had personal experience with consumer tape backups before CDRs became available, and often had reliability problems with only 2 year old tape backups that were thought to be in a safe location and environment. We do have a CD burner and recently added a DVD burner to store small data sets or home directories containing code and other source material. CDs and DVDs are not immune to data loss problems. Typical CDs or DVDs, especially those one burns using consumer devices, only have a guaranteed lifetime of about 5-10 years, with some perhaps having as little as 2 years of

188 life 41,42. CD-R manufacturers performing media longevity studies using industry deﬁned tests and mathematical modeling, claim an theoretical longevity from 70 years to over 200 years. We suggest that any data on CDs or CDRs, for which one needs to guarantee reliability, be refreshed every 5 years.

B.5 Software

In this section we discuss the software required to build, operate, and maintain a cluster using Linux. We discuss the MPI library and how a ﬂuid code uses MPI to perform parallel calculations.

B.5.1 Choices for OS

Linux43 offers a reliable, stable, and high performance OS for a cluster of computers. Compared to Windows and other OSs that have come and gone (such as IBM’s OS/2), Linux is a programmer’s dream. Linux is highly configurable, with open source44 access to the entire system45. Since the author started using Linux in 1994 several versions of Linux have come and gone. Today RedHat Linux dominates the commercial market, with the free version of RedHat no longer being supported except through the Fedora project. Other versions worth mentioning are Mandrake, Knoppix, Debian, SUSE, Gentoo, and Slackware. The author has used all of these versions and has found RedHat to be the easiest to deal with. RedHat keeps to the simplicity of the old Linux versions, thus avoiding confusing configuration issues introduced by versions such as SUSE. Essentially SUSE adds all the problems of a Windows-like configuration without the benefits of Windows’ seamlessness. For example, several parts of SUSE’s configuration program “lock-up” and can corrupt the configuration files. The SUSE configuration files are more complicated than other distributions, and are generally not human readable. Most problems are due to lack of broad support by hardware companies for Linux. Since the demise of a free RedHat version, Fedora and Mandrake are the biggest contenders that are vying for the non-commercial market, which is applicable to private university computer clusters. We are currently evaluating whether Fedora is a reasonable migration path from our current installation of RedHat. Also, there are several “cluster” Linux distributions46 that include the means to install a cluster from one master node by simply having the slave nodes boot off a CD or floppy. The master node distributes and installs Linux on the entire cluster. Probably the most promising is the Scyld47 distribution that provides a single system image cluster using specially developed tools and libraries. Scyld not only makes installation trivial, but the entire cluster can be managed and operated like

41http://news.independent.co.uk/world/science_technology/story.jsp?story=513486 42http://www.pctechguide.com/09cdr-rw.htm 43http://www.Linux.org/info/ 44http://www.gnu.org/ 45Several commercial products and drivers exist for Linux that are closed source, but these have never been used for this cluster. 46http://lcic.org/distros.html\#cluster 47http://www.scyld.com/

189 a single stand-alone computer. Scyld also modifies the kernel to achieve optimal performance for a broad range of network hardware. The price is typically $200 per node. Another commercial product, Scali48, competes with Scyld without the installation and management benefits, but with the performance benefits. The performance of Scyld and Scali can be better than of a standard Linux distribution, although generally they require advanced network hardware to achieve higher performance than standard Linux distributions. Scali and Scyld on Myrinet achieve a drop in latency from 7-8µ s to 4µ s and an increase in peak bandwidth from 220MB/s to 260MB/s. OS- CAR49,50 is a free cluster distribution that offers the same installation and management benefits as Scyld without the performance enhancements. Note that none of these products attempts to develop OS-bypass network drivers for specific hardware, such as the GAMMA project discussed in Section B.2.

B.5.2 OS Installation

For our relatively small 15-17 node BH cluster, we decided to install RedHat and manage the cluster ourselves. We wanted to avoid the complications associated with learning a new OS, specific configuration issues, and we had a fear of losing support due to the death of several cluster projects. The performance of our cluster using standard Intel drivers with modified parameters was excellent compared to any cluster with such cheap technology. Below we describe the steps we followed to install the cluster by “hand”, after the cluster hardware installation has been completed. During this discussion we mention places where we learned of problems with the system. First, download the distribution. If you plan to install from CD, the best advice we can give is download an ISO51 distribution from the fastest server you can find. The availability of online ISO images to burn a CD to install Linux is certainly becoming more common today. In the past, most distributions online were set up for direct installation. This is generally an unreliable means to install Linux due to network slowdowns or outages. If you plan to do a network installation, download the distribution in any form (e.g. extract the ISO images) and place the distribution on a local directory that has an ftp server. All computers can use ftp to do a network installation from your own ftp server. For RedHat the floppy is used to boot each node to install in network mode. Second, install Linux on each node if you are not using a second-generation Beowulf Linux distribution that installs the OS automatically. We simply installed Linux on each node manually, fairly simultaneously, by doing a network installation on each node. It takes about 5 minutes to start the installation off the floppy. This requires switching a keyboard and monitor to each node, but is fairly efficient. For RedHat we chose a “custom installation” with “install everything” selected so that all libraries and applications are installed. Each BH node took about 30 minutes

48http://www.scali.com/ 49http://sourceforge.net/projects/oscar/ 50http://www.openclustergroup.org/ 51http://www.Linuxiso.org/

190 to install Linux. The time it takes for each computer to install Linux is a good first measure of whether the systems are independently performing properly. We had 3 systems take 45-60 minutes to install. We found them to have slower than average hard drive performance by using “hdparm -tT /dev/hda”. In particular the fast computers showed a buffered read bandwidth of 45MB/s, buffered-cache read of ∼ 700MB/s, while the slower computers had a large variation in HD speeds from 15MB/s to 35MB/s and no different buffered-cache read performance. This result was based upon repeated tests52. Swapping hard drives, cables, CPU or chipset heat sinks, etc. lead to no differences. Only after replacing the motherboard did the slow systems reach the normal performance of all the other nodes. Third, recompile the kernel for a streamlined Linux. We compiled our own kernel to trim down unnecessary modules and features. The only clear general statement that can be made about compiling a kernel is to avoid unneeded hardware drivers. One should go through every option available and attempt to fully understand whether it is needed or not. Generally new kernels change only slightly so this is not a difficult process when learned once. We installed Linux 2.4.20, and the kernel config file53 is available online. Linux 2.654 will likely be more desirable than Linux 2.4, since it includes many network related optimizations that may reduce network latency and increase peak bandwidth.

B.5.3 Software Installation and Usage Notes

Here we discuss the installation of MPI, Kerberos, Nagios, the Intel compiler, and group software. We discuss how we installed this on one node and then distributed that single modified state of the file system to the other nodes. The installation procedure for MPI is well documented55. We found that compiling with the SMP option “-comm=shared” leads to significant loss in performance, despite the CPUs apparently being fully utilized. Using the standard options results in good performance, even if the CPUs are under-utilized due to communication time taking longer than CPU time for a given calculation. Kerberized56 versions of several programs are available. Kerberos offers the same benefits as ssh but with higher performance. The default settings for Kerberos connection are encrypted password exchange, but unencrypted connections. This results in faster communications. For buffered transfers, we find that a typical “scp” copy using OpenSSH, or any other ssh program, uses 80-130% of the CPU(s) on the client side and < 1% on the server side for transfers on gigabit, while only obtaining about 13MB/s. Unencrypted or Kerberized versions of “ftp” or “rcp” give about 100MB/s transfer bandwidth and 30% CPU utilization on the server side and 10% on the client side. 52http://rainman.astro.uiuc.edu/cluster/scripts/hdparm.sh 53http://rainman.astro.uiuc.edu/cluster/scripts/jon2.4.20smp 54http://www.kernel.org/ 55http://www-unix.mcs.anl.gov/mpi/mpich/ 56http://web.mit.edu/kerberos/www/

191 We have also found Kerberos is very useful for scripting between nodes. We find that ssh commands can timeout after generating greater than about 10 simultaneous ssh commands from a single node. This means that a scripted run through many computers may or may not reach all computers, an intolerable unreliability. We have never encountered such problems with “rsh”, so we use this for scripting when generating simultaneous commands to the cluster. The “rsh” program is also much faster than ssh for starting MPI jobs. On a related note, we found “ncftp” on RedHat 8.0 to be slow. We obtain about 4-6MB/s on gigabit and the server side uses 100% CPU (“wu-ftpd” or “in.ftpd”) and 13-20% CPU on the client side. Using normal ftp gives 90-100MB/s at 100% CPU. For unknown reasons “ncftp” is causing the CPU to be under-utilized. No such problems were encountered under SUSE. We found it useful to install a cluster monitoring program called Nagios57. Nagios can be used for many advanced purposes and is easily and highly configurable. We use it to monitor each node in the BH cluster and all our workstations for 1) ping, 2) network services availability (SMTP, ftp, ssh, HTTP), 3) number of users, 4) total processes, 5) too high CPU load, temperature problems using the lm sensors58 package (currently version “lm sensors-2.7.0” which requires “i2c-2.7.0”), and 6) too small hard drive space available. It “nags” a specified user or group about problems via email, which is why SMTP (sendmail) is run on the cluster. It reports the status of the computers and allows changes to be made through a well designed web interface, which is why HTTP (apache) is run on the cluster. Nagios can be run on any computer that can access the relevant services on the remote computers. Other Nagios-like programs are available, such as Big Brother59 and Ganglia60. The compiler of choice for Intel-compatible nodes is “icc” for C and C++ and “ifc” for Fortran. These offer about 2-3X faster performance than “gcc” for C or g77 (f77) for Fortran, for our codes. A free non-commercial license is available online for both the C/C++ compiler61 and the Fortran compiler62. The standard “gcc” remains necessary for several programs that are incompatible with “icc” due to Intel’s lack of full Linux support. For scientific codes this is often a moot point because they make limited use of complicated libraries. However, several programs and libraries cannot be compiled with “icc”. Attempts will either generate compiler errors (that can sometimes be “fixed”), or the code will simply not function properly (sometimes generating insidious failures of apparently unrelated origin). We suggest only compiling something with “icc” if the code does not use complicated routines or external libraries. Intel’s “icc” packages include a debugger and one can purchase a profiler, but the standard gnu debugger “gdb” and gnu profiler “gprof” can be used with “icc”. The most accurate profile report from gprof on a code compiled with “icc” is with no optimizations, which can alter the profile significantly compared to an optimized compilation.

57http://www.nagios.org/ 58http://secure.netroedge.com/~lm78/ 59http://www.bb4.org/ 60http://ganglia.sourceforge.net/ 61http://www.intel.com/software/products/compilers/clin/noncom.htm 62http://www.intel.com/software/products/compilers/flin/noncom.htm

192 When compiling with optimizations, gprof can fail to create a reasonable report, so we only use gprof as a guide. Compiling MPI, especially the Myrinet version of MPICH over GM, is often problematic with “icc”. GM is Myrinet’s network device driver for accessing Myrinet hardware using MPI functions. Also, compiling Fortran 90 support for MPI with simultaneous support for C/C++ can be tricky and has unresolved problems due to Intel’s incomplete C and Fortran support. A script we used to compile the standard MPI is available online63. Group software should be placed in “/usr/local” so the software can be easily copied to any computer. Any additional software that is installed from source (in “/usr/src”) should be installed to “/usr/local” and “encapped” with EPKG64 or a similar program. EPKG allows one to easily install and uninstall any installed package without worrying about corrupting the “/usr/local” path with old programs. One simply makes a link from your program directory to “/usr/local/encap” and run inside that directory “epkg -i ”. Many of the installation issues, such as what services to install, can be worked out on one computer and distributed to all other computers. See the script online for a guide to how this was done for BH65. All BH nodes have ssh, Kerberized (ftp,telnet,rsh,klogin,kshell,eklogin), while currently the master node (BH00) has also “httpd”, “smtp”, and “kerberos-adm” installed. We found it useful to create a cluster equivalent to Linux programs “top” and “df” that would post to a website. This required updating the procps version to “procps-3.1.6”, which includes, for example, improved “top” and “ps” programs. The new “top” program has many options, and we set these options by including a “.toprc”66 file in the home directory of root. These options determine the final output of running “top” as root. We then run a cron job every 3 minutes for the cluster “top” and cluster “df” by adding the line to the “/etc/crontab/” list67. We also run a script every hour that shows the user usage of disk space. The scripts for the cluster “top” and “df” are available online68. The list of available top and df websites is available online69. As an aside, we suggest using the C language for complicated scripts. C scripts can be powerful, fast, and incorporated with pure scripting languages like bash. C scripts have the advantage of being easy to understand, easy to make, and are easy to read. Bash70, perl, and other scripting languages have many exceptions to the basic rules of syntax, and this syntax complexity makes it difficult to write marginally complicated tasks. Prior to the BH cluster we had been using NFS (network file system) to allow every computer to see the disk of every computer with a large hard drive for general storage. While this may have security concerns, the primary practical problem with this is a lack of stability. We found

63http://rainman.astro.uiuc.edu/cluster/scripts/mpich.make.intel.p4 64http://www.encap.org/epkg/ 65http://rainman.astro.uiuc.edu/cluster/scripts/clusterinstall.sh 66http://rainman.astro.uiuc.edu/cluster/scripts/.toprc 67http://rainman.astro.uiuc.edu/cluster/cronstuff/crontab 68http://rainman.astro.uiuc.edu/cluster/cronstuff 69http://rainman.astro.uiuc.edu/cluster/computers.html 70http://www.tldp.org/LDP/abs/html/

193 that if any single computer with a storage drive went down or became unavailable, all computers mapping that drive locally would lockup. This destroys the usefulness of NFS by eliminating the distributed nature of a cluster of workstations. For this reason we do not use NFS, and we simply assume that each person can copy information using “scp” or “rcp”, and that MPI programs will manage their diagnostics to write to disk in parallel or collect the data to one drive manually. That is, the MPI program should not assume that the disk is one large disk for all nodes. This loses the transparency of disk storage provided by NFS, but is worth the increased stability and parallel performance. Lustre71 may provide an NFS-like distributed file system, but with improved stability, performance, and security. NFS is fairly insecure and has stability and performance problems that Lustre seeks to eliminate. Lustre removes the NFS stability problems by avoiding any single point that can bring total failure of the distributed file system. Lustre allows up to 10,000 nodes with nearly 90% efficiency of total hard drive bandwidth for parallel file I/O. We maintain security by periodically updating packages from RedHat. Since the demise of the free RedHat, we may attempt to move to Fedora, Mandrake, or some commercial cluster distribution like Scyld or Scali. We maintain security by using “tcp wrappers”. A “tcp wrapper” is a program that checks all incoming connections for certain services and only allows computers with pre-specified IP addresses to connect. This is controlled by the “hosts.allow” and “hosts.deny” files in “/etc/”. For the website we control access either through standard entries in “httpd.conf”, or by using “.htaccess” files and assigned passwords. The former is used to control access to somewhat sensitive web pages by computers outside our group, and the latter is used to control Nagios access. Mistakes can be made in setting up which services are running. One can check what services are running on a computer using “nmap ”.

B.5.4 MPI Implementation in Fluid Codes

MPI is a library specification for message-passing, proposed as a standard by a committee of vendors, implementers, and users. MPICH72 is a freely available, portable implementation of MPI. Historically, for those who develop their own physics codes, there have been 2 distinct approaches to parallelizing a code. One approach is to use a pre-existing library suite that permits a simplified, but prescribed method of parallelization. The other approach involves directly using an MPI library, such as MPICH, and developing an interface. Interfaces to the MPI libraries, such as DAGH73,74, Kelp75, and CHOMBO76, invite the programmer to ignore the details of MPI and write good physics code. They often offer very complicated services that only take a handful of functions to use and so are easy to implement, such as parallel adaptive mesh support in CHOMBO. Problems with such MPI interfaces include: (1) a lack of continued support (DAGH, Kelp), (2)

71http://www.Lustre.org/documentation.html 72http://www-unix.mcs.anl.gov/mpi/mpich/ 73http://www.caip.rutgers.edu/~parashar/DAGH/ 74http://www.cs.utexas.edu/users/dagh/ 75http://www.cs.ucsd.edu/groups/hpcl/scg/kelp/ 76http://seesar.lbl.gov/anag/chombo/

194 over-generalization and slowness (DAGH), and (3) integration into code is fixed and very dependent on that library. As with any Linux project, support is often only guaranteed when supported by a large set of corporations or universities. Projects like DAGH and Kelp were only supported by a few universities and later collapsed as funding was not available and key programmers moved on. We wished to avoid performance problems, being dependent on a high-level library, and stability uncertainties. We decided to write our own MPI interfaces for intra-nodal setup, all inter-nodal communications, and for parallel file I/O using the MPICH library. We only use about 10 MPI functions to do all our parallel processing related tasks. These functions include 1) initialization type functions: MPI Init, MPI Comm size, MPI Comm rank, MPI Get processor name, which are trivially used and involve about 5 lines of code; 2) management functions: MPI Barrier, used to fully synchronize all CPUs at 1 point in the code, which is rarely used; 3) Transfer functions: MPI Bcast, MPI Reduce, MPI Allreduce, MPI Wait, MPI Irecv, MPI Isend; and 4) finishing functions: MPI Abort, MPI Finalize, which are trivially used at the end of either an aborted or successful simulation, using about 5 lines of code. The transfer functions involve the most complicated part of MPI by 1) setting up the communications between nodes (about 40 lines of code); 2) the communication of boundary values shared to nodes, which is about 100 lines of code; and 3) the diagnostics of parallel file output and other collective operations to obtain integrated quantities, which involves about 2000 lines of code. Clearly the diagnostics are the most difficult and time consuming part of making a code parallel using MPI, despite our effort to modularize the procedures. Once the modularization is done it is trivial to output new diagnostics by following examples in the code. Several MPI-277 (MPI version 2) parallel I/O functions can be used, as we have done. However, support for MPI-2 features was non-existent in 1999 when we were developing our code to use MPI. Even in 2002, NCSA’s Platinum cluster had no support for parallel I/O. In order to use NCSA clusters as soon as possible we implemented our own file I/O procedures. Our file I/O routines use a fixed small memory storage buffer regardless of the number of nodes or number of columns of data per zone. These routines can write to disk on a single master node or write in parallel to each node’s hard drive. The second most complicated MPI coding involves the transfer functions. Ideal fluid codes operate on a grid of zones with typically hyperbolic equations of motion. Such a code can be decomposed (“domain decomposition”) into smaller per-CPU grids and the boundary values on each CPU can be exchanged to the neighboring CPUs when necessary. The routine that transfers the boundary values between nodes is quite trivial and was developed and debugged in a week by the author for a ZEUS-based code in 1999. There are at least 2 methods to improve transfer efficiency for a simple homogeneous domain decomposition: 1) use non-blocking routines; and 2) use multi-layered boundary value zones. A non-blocking routine allows the CPU to continue processing while the network hardware is transferring data to other CPUs. This compared to “standard” blocking routines that halt all involved CPUs until the communication is complete. Blocking routines are a bit more trivial to implement, which is why they are more common. Non-

77http://www-unix.mcs.anl.gov/mpi/mpich2/

195 blocking routines alone might allow a small gain in efficiency by parallelizing transfers between all nodes when transferring boundary values between all computers each time step. Optimal use of non-blocking routines is obtained by using 2 layers of boundary zones per CPU. One boundary layer is the standard boundary conditions layer, while the next layer is a 2-zone wide ring of non-boundary zones just inside the boundary conditions layer. This assumes the stencil size is 2-zone each direction which is typical of ZEUS and HARM. The procedure to follow is simple: 1) start computations on all zones; 2) exchange boundary values using non-blocking routine; 3) continue calculation on the remaining subset of non-boundary layer zones not including the 2- zone wide ring; 4) once the subset calculation is complete, complete the non-blocking call with an MPI Wait() call; and 5) calculate the 2-zone ring that required the updated boundary values. This procedure continues indefinitely. This procedure allows the CPU to compute while boundary values were being transferred, allowing maximum CPU utilization. The new time step is computed by first computing the minimum time step required for each CPU, then using an MPI Allreduce() call. This communicates to every CPU the minimum of all the CPUs values for the time step. When a algorithmic failure occurs in MPI, its often difficult to track down the error without careful MPI considerations. Failure checks can be organized by having every function that could return a 1 on failure and 0 on success, and checking every function call’s return value. Upon success the code would continue. Upon failure the code would return a 1 for failure. As long as the failure checks are placed before each MPI-type call, then all the CPUs will fail synchronously. Ignoring this issue will leave hung jobs and no clear record of the cause of the failure.

B.5.5 Running MPI Jobs

For a large cluster, or for a large number of users for a relatively small cluster, preexisting schedulers and batch processing applications are necessary. The most widely used program to help organize all the requests of users to perform computations is the OpenPBS78,79 batch system (PBS stands for portable batch system). This system accepts user requests to run MPI programs on a cluster, runs the program, and can even schedule when the program should run. However, the most widely used scheduler for determining when someone’s program should run is the Maui80 scheduler. These are popular programs, and NCSA uses both PBS and Maui. These programs are not trivial to set up. This author set up these systems on the BH cluster, but we decided that they were too restrictive for our small group. For relatively small clusters or for clusters with few people, its easy to rely on the users to check what processors are used and how much hard drive space is available to ultimately decide what nodes the parallel job will run on. In order to facilitate this process scripts are useful to generate easy to access lists of this information. As described previously, we have a script that generates a

78http://www.openpbs.org/ 79http://www-unix.mcs.anl.gov/openpbs/ 80http://www.supercluster.org/maui/

196 webpage showing the “cluster top” and “cluster df”. These show available CPUs, what processes are on each CPU, CPU usage, and hard drive space available on each node. This is useful for both parallel and single CPU runs. We leave it up to the user to run MPI programs or single-CPU programs. The standard “mpirun” script that is included with MPICH has several seemingly complicated restrictions on starting an MPI program. For example, the job must be started on the first node that will perform the calculations. There are some ways around this particular problem, but other problems remain. The author has written a simple script that makes it easier to start a batch job from one node and then run the job on other nodes. This “mympirun.sh” script will also create a working directory on every node and copy the binary file to that directory on every node. This is useful in a non-NFS file system environment where each hard drive space on each node is invisible to the other nodes. The script uses either “ssh” or “rsh” and “scp” or “rcp” to perform the startup tasks. The programs “rsh” and “rcp” are much faster for starting jobs on many nodes. The script is available online81.

B.6 Testing Cluster Reliability and Performance

Benchmarks can provide insight into how diﬀerent software, hardware, and conﬁgurations of each of these modify performance. Ultimately one should test the code to be run on a system, but synthetic benchmarks can be useful for more general performance characteristics. A synthetic benchmark is simply a test that measures an isolated hardware component or feature. Synthetic benchmarks also allow one to compare all nodes to each other to verify that each node’s hardware components operate nominally.

B.6.1 Reliability and Performance Issues

Chips on the motherboard are unlikely to fail if cooled properly. However, the heat sinks on the CPUs can be unreliable. The fans can stop working or the thermal contact between the CPU and the heat sink can be poor. One can use an IR (infrared) thermometer (available at, e.g., Radio Shack) to obtain a precise temperature measurement for all the chips. We obtained an IR thermometer to measure the CPU temperature to be sure the “lm sensors” package was measuring the temperature accurately since some CPU temperatures were reported as being quite high. We found those temperatures to be accurate, and so we decided to remove the sides from the cases. This dropped the CPU temperatures from 50C to 25-40C depending upon the vertical location of the computer in the computer room (higher systems are hotter). During this process we found that the gigabit chip runs at about 70C, the hottest of all the chips on the motherboard. Tyan discussed this with Intel and it was determined that this chip is known to run this hot. An eﬃcient way to ﬁnd problems in hardware and software is to stress test, or burn-in, the cluster. Burning-in a computer cluster involves stress-testing the cluster to maximum levels of

81http://rainman.astro.uiuc.edu/cluster/scripts/mympirun.sh

197 CPU, memory use, hard drive use, and network communication for long periods of time. This tests whether there is sufficient power and cooling available and tests whether any hardware components might easily malfunction. CPUs from AMD and Intel are extremely reliable and generally operate as specified. We test CPU performance using lapack, specfp, stream82, our own codes, and a few other interesting benchmarks. The Benchmark HQ website83 and sourceforge84 are good resources for Linux benchmarking software. Despite the reliability of AMD and Intel CPUs, the author’s experience is that motherboards made for AMD CPUs (AMD does not make motherboards), and chipsets not made by AMD or Intel for AMD or Intel CPUs, are much less reliable than those chipsets and motherboards designed directly by AMD or Intel. There are several computer companies85,86,87 that do not offer anything except Intel products for chipsets, motherboards, and CPUs due to reliability problems with AMD motherboards and chipsets or non-Intel chipsets for Intel CPUs. Assuring compatibility with the CPU and other motherboard components is a complex task. The CPU has generally been the most complex piece of equipment on a motherboard88, so a motherboard and chipset designed by a CPU manufacturer is more likely to be reliable and compatible with existing technology. We chose an Intel 2.4Ghz Xeon CPU with an Intel chipset and a Tyan motherboard. Tyan is a well-respected motherboard manufacturer for Intel CPUs. We chose a CPU speed that was not overpriced by Intel, so was comparably priced to AMD processors of the same speed. It is possible that over the past couple years that AMD related chipsets and motherboards have become more robust. It is yet to be seen how reliable the AMD Opteron motherboards and chipsets are. Memory performance can often be a bottleneck in physics computations by having (1) limited bandwidth and high latency to/from the CPU, (2) limited bandwidth and high latency to/from the system bus, (3) limited bandwidth to/from the interface boards. Typically ECC (error correction coding) is used in memory for cluster computers to prevent errors in memory affecting the rest of the system. Most server-based motherboards require ECC memory. Generally memory in today’s market is reliable assuming one chooses a memory module that is a brand-name rather than generic. A special memory feature of server motherboards is dual-channel memory. Dual-channel memory allows interleaved access to the memory by the 2 CPUs, which allows each CPU in dual-CPU nodes to access the memory at nearly the peak memory bandwidth. Dual-channel memory reduces so-called “memory contention” that can occur when both CPUs access the memory simultaneously. We have found that specific memory speeds have little impact on the speed of our codes. That is, within the range of memory speeds available, the fastest memory seems to be little different from the standard memory. We purchased DDR PC2100 memory, where DDR stands for double data

82http://www.cs.virginia.edu/stream/ 83http://www.benchmarkhq.ru/english.html 84http://lbs.sourceforge.net/ 85http://www.dell.com/ 86http://www.gateway.com 87http://www.simplifiedcomputers.com/ 88Actually, for the past 2 years 3D graphics chips have been more complicated than even CPUs

198 rate, and the memory operates at 2100MB/s peak bandwidth. We use “memtest86”89 to check single-CPU memory speed and check for memory errors during a reboot of a system. Memtest86 acts as an entire OS that loads like any Linux kernel. One simply adds another entry to the “/etc/grub.conf” or “/etc/lilo.conf” ﬁles. “Grub” and “lilo” are programs that run at boot time that load the primary OS component and pass a few key arguments to the OS. One can also test the memory using “stream”.

B.6.2 Bandwidth and Latency of Network

Two basic measures of network performance are (1) peak bandwidth for large messages and (2) latency for small messages. These 2 measures characterize a full analysis of the amount of time it takes to transfer a message of arbitrary size. In comparing various onboard gigabit chips and PCI boards in 64-bit 66Mhz and 133Mhz PCI slots, we realized that the onboard chips operate faster or equal to add-on boards. In sections B.1 and B.2 we gave the performance results for all our tests on gigabit and Myrinet. These benchmarks used the “mpptest” program included with MPICH in the “examples/perftest” directory, and an example is available online90. Figure B.5 shows the 1-way bandwidth and latency for MPI communications between 2 2.4Ghz BH nodes for a range of message sizes. Notice that gigabit only achieves peak bandwidth for message sizes greater than about 20kB reaching a peak bandwidth of about 100MB/s. Also, notice that the latency for small message sizes is about 27µ s. These performance figures can be compared to similar tests with modern Myrinet boards on MPICH over GM91 that achieve a peak bandwidth of about 250MB/s and a latency of about 9µ s. NetPIPE92 can also be used to measure network performance, but it does not account for overhead due to MPI. One can measure the data overhead associated with MPI on the network by comparing the “tcpdump” results for the number of bytes transferred on a device with the known number of transfers in, say, a fluid code. From a fluid code one can count the total number of bytes transferred as boundary values per time step as N = NvNBC NxNb/vNt, where Nv is the number of named single unit variables transferred, NBC is the number of boundary zones per named variable, Nx is the typical dimension of the edge between CPUs, Nb/v is the number of bytes per variable, and Nt is the number of unidirectional transfers per time step per CPU to CPU communication.

For our ZEUS-based MHD code, we have Nv = 36, NBC = 2, Nb/v = 8, and Nt = 2. This gives N = 1152Nx, or about 1kB per zone edge. The typical message passed has Nv = 1 and

Nt = 1, so gives N = 16Nx. For a typical value of Nx = 128 the typical message size is 2kB per time step. This is a small message size, and from Figure B.5 we see that the bandwidth is only

20MB/s. Often when running on hundreds of CPUs (e.g. 200 = 25 × 8) Nx = 64 leads to a more practical total resolution. In this case the bandwidth on gigabit is only about 10MB/s, 1/10 the

89http://www.memtest86.com/ 90http://rainman.astro.uiuc.edu/cluster/scripts/mpptest.sh 91http://www.myri.com/myrinet/performance/MPICH-GM/index.html 92http://www.scl.ameslab.gov/netpipe/

199 Figure B.5 Bandwidth (left) and latency (right) for gigabit Ethernet connection on BH cluster.

peak bandwidth. For a 3D simulation no such limits occur since then Nx = N1N2. A practical resolution is N1 = N2 = 32. In this case the smallest message size is N = 16kB giving 85MB/s on gigabit, near the peak bandwidth. HARM operates by always transferring Nv = 8, so N = 128Nx, and for Nx = 64 we have the smallest message size is N = 8kB for a bandwidth of 60MB/s, which is reasonably near the peak bandwidth. HARM in 3D will easily reach the peak bandwidth of gigabit, and consequently require a faster network. One can also check for whether the communication is completing faster or slower than the CPU processing by checking the percent of CPU usage in “top” (assuming MPICH is compiled without “-comm=shared”). For example, our HARM code with 1282 zones per node achieves about 80% CPU efficiency when using 16 CPUs. We advise not running single processes on the nodes running MPI jobs with lower than 100% efficiency, since we find that the actual speed of the MPI processes slows down nonlinearly in proportion to their CPU usage when the CPU is oversubscribed. This could be due to memory contention. Simply using “nice” to “renice” the other processes to a nice level of 19 does not completely remove the cache inefficiency produced by oversubscribed CPUs, but leads to performance more proportional to the CPU usage shown by “top”.

B.6.3 Code Performance

Older NCSA systems, such as the Origin 2000 (now replaced by Copper), included utilities such as “perfex” that gave detailed information about cache use (percent miss), MFLOPs, and other vital statistics useful for proﬁling a code. There are some dead open source projects with “perfex”-like

200 utilities93. The current prevailing and actively supported “perfex”-like utility is PAPI94. For 32-bit Intel-compatible CPUs, PAPI requires the kernel be recompiled after a patch is installed. We assume all our codes are well designed for the cache and follow high performance coding etiquette95,96. We use the well-known performance metric of “zone cycles per second” (ZCPS) for measuring fluid code performance. This refers to the number of grid elements that are calculated per real time second on a given system. The code efficiency for a given number of CPUs is defined as % Eff= ZCPS/(NCPU ∗ZCPS1CPU )∗100, where NCPU is the number of CPUs and ZCPS1CPU is the ZCPS for 1 CPU. In general, testing of a fluid code should be performed both with and without diagnostics that are written to file. This allows one to test both raw processing speed and normal processing speed, which allows us to evaluate how the HD is limiting raw performance. Below we only report the raw processing speed in ZCPS. We consider the performance of our viscous hydrodynamics (VHD), 2D magnetohydrodynamics (MHD), 3D MHD, and HARM codes on some of the systems available to our group over the years. All these codes can be run in parallel using MPI. The systems tested include the NCSA SGI Origin 2000 composed of 256 R10000 250Mhz CPUs networked internally by SGI’s proprietary hardware connecting each CPU’s memory, the NCSA Platinum cluster composed of 1024 1Ghz Intel Pentium 3 (P3) CPUs on 512 dual-CPU nodes networked with Myrinet, the NCSA Itanium cluster composed of 268 Intel 800Mhz Itanium I CPUs on 134 dual-CPU nodes networked with Myrinet, and our BH cluster composed of 28 2.4Ghz P4 Xeon CPUs on 14 dual-CPU nodes networked with gigabit (we exclude the 2.0Ghz P4 Xeons from these tests). We expect NCSA’s Tungsten 3.06Ghz P4 Xeon cluster to show similar performance to BH, with Tungsten using Myrinet showing improved efficiency for a large number of processes. We provide tables of the number of ZCPS and efficiency of some of our codes on each system. We run all codes in double precision. Single precision offers some speed advantages, but often is insufficient to preserve important mathematical constraints. For example, after the number of time steps needed to study accretion flows, a single-precision based solenoidal constraint leads to errors larger than truncation error. All the codes are run with similar initial conditions that model the accretion flow near a black hole. As a baseline for parallel performance comparison, the number of ZCPS for each of our codes for a single CPU run is shown in Table B.1. The HARM code tests shown are from our original HARM code that had no failure checks, so is faster than our latest version of HARM with many failure checks. These checks slow our new HARM down by about a factor of 2X. We plan to profile the new HARM to optimize these failure checks. Notice that for our codes the Intel Itanium I processor is about a factor of 2X slower than the older 1Ghz Intel P3 processor, which was unexpected based upon synthetic benchmarks. Our collaborating group performing numerical relativity simulations found Itanium I CPUs to be about 2X faster than the 1Ghz Intel P3 processor. This demonstrates

93http://www.osc.edu/~troy/lperfex/ 94http://icl.cs.utk.edu/papi/ 95http://www.intel.com/cd/ids/developer/asmo-na/eng/microprocessors/ia32/pentium4/optimization/ 43896.htm 96http://rainman.astro.uiuc.edu/cluster/sgiperf.pdf

201 Table B.1. Single CPU performance in ZCPS

CPU VHD 2D MHD 3D MHD HARM

R10000 110 100 Alpha 21264 210 204 110 30 1GHz P3 155 141 70 17 800Mhz Itanium I 82 75 30 8 2.4Ghz P4 Xeon 300 282 144 40

Note. — ZCPS from actual measurements. The resolution per CPU is 642 for 2D and 403 for 3D. Tungsten performance can be estimated by simply multiplying the Xeon performance by ∼ 1.3.

the necessity to profile one’s own code rather than relying on anecdotes or synthetic benchmarks. Notice also that the 4-5 year old Alpha CPUs are nearly as fast as the 2.4Ghz Intel P4 Xeon CPUs. We briefly mention our Alpha cluster with Myrinet that has only 6 CPUs. For a 4-CPU MPI test of the 2D MHD we find an efficiency of 80%, while for a 6-CPU test we find an efficiency of 70%. When the NCSA Origin 2000 was available, we performed MPI tests using our 2D MHD code with artificial resistivity and the Alfvén-limiter enabled. File writing of diagnostics was disabled. The results of these tests are shown in Table B.2. We found the Origin 2000 to have disk performance problems for more than about 64 CPUs, which showed up when turning on file writing of our diagnostics. File writing with all diagnostics using 256 CPUs drops the efficiency to 37%, while with half the diagnostics drops to 55-65%. Otherwise the performance levels off at about 70% efficiency even up to 256 CPUs, which is excellent for the 2D MHD code. The tables of 2D & 3D MHD and HARM performance on Platinum and BH show columns for 1) the number of cpus; 2) the number of CPUs per node used; 3) the number of nodes used; 4) the CPU tile geometry; 5) the number of ZCPS per unit 1000; and 6) the parallel efficiency. Note that the 2D and 3D MHD code’s inner loop stride is the “i” or “x”-direction. Thus, the

CPU tile geometry is denoted as Tx × Ty, where Tx is the number of tiles in the x-direction and

Ty is the number of tiles in the y-direction. Alternatively, HARM’s inner loop stride is the “j” or “y”-direction. For comparison purposes the CPU tile geometry in the table is ﬂipped to show “y × x”. This allows direct table-to-table comparisons. We performed nonrelativistic MHD simulations using NCSA’s Platinum cluster using the 2D and 3D MHD codes. The results of the 2D and 3D MHD tests are shown in Table B.3. Notice that Platinum operates about 20% faster on 1cpu/node than 2cpus/node. Our collaborating group performing numerical relativity simulations found a decrease in performance between 30-60% when

202 Table B.2. Origin 2000 MPI performance in kZCPS for 2D MHD code

# CPUs Speedup % Eﬀ kZCPS

1 1.0 100 110 4 3.7 93 410 9 8.7 90 955 16 11.2 70 1235 36 24.2 67 2662 49 34.1 70 3751 64 36.5 57 4015 121 92.7 77 10201 256 173.9 72 19129

Note. — The resolution per CPU is 642 in 2D. using 2cpus/node compared to 1cpu/node, so they avoid using 2cpus/node. NCSA jobs are charged per node rather than per CPU, so a 30% performance decrease is reasonable to avoid the factor of 2X charge in CPU-hours. We have underlined those tests that show notably low efficiencies compared to similar runs. No simple reason could be found for these low efficiencies. See Table B.4 for MPI tests of the 2D MHD code on our BH cluster. All these tests are run on the 2.4Ghz Xeon nodes only, thus we test up to only 14 nodes (rather than 17 – the total number of nodes). These simulations are in 2D spherical polar coordinates. We test both 642 and 2562 tile sizes per CPU. Artificial resistivity and the Alfvén-limiter are enabled. File writing is disabled. Timing is based upon the wall clock time with nodes otherwise free of processes. The code runs for a fixed number of time steps (about 1500 for 642 and about 90 for 2562). We have underlined those tests that show notably low efficiencies compared to similar runs. No simple reason could be found for these low efficiencies. See Table B.5 for MPI tests of the 3D MHD code. These simulations use a tile size per CPU of 403. The grid is based upon a 3D Cartesian coordinates with inner and outer Cartesian approximations to spherical shells. Artificial resistivity and the Alfvén-limiter are enabled. File writing is disabled. Timing is based upon the wall clock time with nodes otherwise free of processes. The code is run for a fixed number of time steps (about 120). See Table B.6 for MPI tests of HARM. These simulations are in 2D spherical polar coordinates in Kerr-Schild coordinates. We test both 642 and 2562 tile sizes per CPU. The timing is based upon the wall clock time with nodes otherwise free of processes. The code runs for a fixed number of time steps (about 100 for 642 tile and about 10 for 2562). We use the latest HARM code running a fiducial simulation as in McKinney and Gammie (2004) with a fixed 3 iterations per zone for

203 Table B.3. NCSA Platinum MPI performance in kZCPS for 2D & 3D MHD code

# CPUs cpus/node # nodes geom kZCPS % Eﬀ

2D MHD 1 1 1 1 × 1 141 100 2 2 1 1 × 2 218 77 2 1 2 1 × 2 246 87 3 1 3 1 × 3 351 83 4 2 2 2 × 2 190 34 4 1 4 2 × 2 437 77 6 2 3 2 × 3 267 32 6 1 6 2 × 3 589 70 32 2 16 4 × 8 2146 48 32 1 32 4 × 8 2800 62 3D MHD 1 1 1 1 × 1 × 1 70 100 2 2 1 1 × 1 × 2 117 84 2 1 2 1 × 1 × 2 137 98 4 2 2 1 × 2 × 2 220 79 4 1 4 1 × 2 × 2 265 95 6 2 3 1 × 2 × 3 308 73 6 1 6 1 × 2 × 3 377 90 8 2 4 2 × 2 × 2 418 75 8 1 8 2 × 2 × 2 514 92 16 2 8 2 × 2 × 4 775 69 16 1 16 2 × 2 × 4 964 86 24 2 12 2 × 3 × 4 985 59 24 1 24 2 × 3 × 4 1212 72 32 2 16 2 × 4 × 4 1321 59 32 1 32 2 × 4 × 4 1582 70

Note. — The resolution per CPU is 642 in 2D and 403 in 3D. Using 2562 gives a bit lower per CPU performance and a bit better MPI eﬃciency.

204 Table B.4. BH Xeon Cluster MPI performance in kZCPS for ZEUS-based 2D MHD code

# CPUs cpus/node # nodes geom 1kZCPS 1%Eﬀ 2kZCPS 2%Eﬀ

1 1 1 1 × 1 282 100 305 100 2 2 1 1 × 2 511 91 513 84 2 1 2 1 × 2 513 91 572 94 3 1 3 1 × 3 696 82 753 82 4 2 2 2 × 2 720 64 843 70 4 1 4 2 × 2 787 70 950 78 4 1 4 4 × 1 638 57 848 70 4 1 4 1 × 4 650 58 1007 83 6 2 3 2 × 3 872 52 1110 61 6 1 6 1 × 6 1297 77 1428 78 6 1 6 6 × 1 954 56 1195 65 6 1 6 2 × 3 976 57 1218 66 12 2 6 2 × 6 1587 47 2176 60 12 1 12 2 × 6 1626 48 2108 58 12 1 12 1 × 12 2069 61 2375 65 12 1 12 12 × 1 1990 59 2000 55 14 2 7 2 × 7 1767 45 2493 58 14 1 14 2 × 7 1762 45 2297 54 14 1 14 7 × 2 1951 50 2335 55 14 1 14 14 × 1 2352 60 2359 55 14 1 14 1 × 14 2354 60 2680 63 28 2 14 4 × 7 2245 28 4431 52 28 2 14 1 × 28 3849 49 5094 60 28 2 14 28 × 1 4191 53 4187 49

Note. — (1)642 per CPU tile size. (2)2562 per CPU tile size.

205 Table B.5. BH Xeon Cluster MPI performance in kZCPS for 3D MHD code

# CPUs cpus/node # nodes geom kZCPS % Eﬀ

1 1 1 1 × 1 × 1 161 100 2 2 1 1 × 1 × 2 282 88 2 1 2 1 × 1 × 2 307 95 4 2 2 1 × 2 × 2 528 82 4 1 4 1 × 2 × 2 600 93 6 2 3 1 × 2 × 3 750 78 6 1 6 1 × 2 × 3 854 88 8 2 4 2 × 2 × 2 931 72 8 1 8 2 × 2 × 2 1117 87 12 2 6 2 × 2 × 3 1317 68 12 1 12 2 × 2 × 3 1323 68 16 1 16 2 × 2 × 4 1673 65 16 1 16 1 × 4 × 4 1603 62 16 1 16 1 × 1 × 16 1784 69 16 1 16 16 × 1 × 1 1609 66 24 2 12 2 × 3 × 4 2247 58 28 2 14 2 × 2 × 7 2740 61

Note. — Per CPU tile size is 403.

206 Table B.6. BH Xeon BH Cluster MPI performance in kZCPS for 2D HARM code

# CPUs cpus/node # nodes geom 1kZCPS 1%Eﬀ 2kZCPS 2%Eﬀ

1 1 1 1 × 1 17 100 18 100 2 2 1 1 × 2 33 97 32 89 2 2 1 2 × 1 33 97 33 92 2 1 2 2 × 1 35 100 34 94 2 1 2 1 × 2 35 100 34 94 3 1 3 3 × 1 46 90 49 91 4 2 2 2 × 2 61 90 67 93 4 1 4 2 × 2 61 90 62 86 4 1 4 1 × 4 57 84 62 86 4 1 4 4 × 1 61 90 62 86 6 2 3 2 × 3 91 89 96 89 6 2 3 3 × 2 86 84 97 90 6 1 6 6 × 1 88 86 93 86 6 1 6 1 × 6 86 84 101 94 6 1 6 3 × 2 92 90 100 93 12 2 6 2 × 6 175 86 152 70 12 2 6 6 × 2 174 85 192 89 12 1 12 6 × 2 164 80 184 85 12 1 12 12 × 1 169 83 185 86 12 1 12 1 × 12 172 84 184 85 14 2 7 2 × 7 209 88 169 67 14 2 7 7 × 2 207 87 219 87 14 1 14 7 × 2 198 83 191 76 14 1 14 2 × 7 200 84 168 67 14 1 14 1 × 14 203 85 205 81 14 1 14 14 × 1 209 88 213 85 28 2 14 7 × 4 348 73 430 85 28 2 14 4 × 7 369 78 377 75 28 2 14 1 × 28 335 70 427 85 28 2 14 28 × 1 396 83 408 81

Note. — (1) 642 per CPU tile size. (2) 2562 per CPU tile size. Tile geometry is ﬂipped for direct comparison to the 2D & 3D MHD code tables. Actual tile geometry shown is Ty × Tx, where Tx is the number of CPU tiles in the “x-direction” and Ty is the number of CPU tiles in the “y-direction”.

207 variable inversion, which is typical. Some simulations with many failure corrections use up to 15 iterations per zone, reducing per CPU performance by about 50%. The results of this table may differ from other results obtained using older versions of HARM. These differences are due to how the code was configured when tested and a lack of optimizations in the latest version of HARM. We disable file writing for these tests. All these tests are run on the 2.4Ghz Xeon nodes (i.e. not the 2.0Ghz Xeon nodes). We have underlined those tests that show notably low efficiencies compared to similar runs with different tile geometries. In these poorly performing cases, the CPU geometry is extended in the inner loop stride direction. Similar tests performed on the BH cluster using 100Mbps Ethernet show similar performance to gigabit for a small number (≤ 8) of CPUs for HARM and 3D MHD tests, but 2D MHD test results for any CPU number & 4 results in poor performance due to the higher latency of Ethernet. HARM is not functional on the Tungsten cluster due to issues that are not yet understood. Tungsten started operating in production mode with Myrinet on June 23, 2004, about the time this appendix was written. We expect Tungsten to operate better than BH due to the Myrinet interface, and we expect Tungsten using gigabit will operate similarly to BH after taking the CPU speed difference into account. We expect that for a large number of CPUs (say 200 operating on 25 × 8 CPU tile geometry), Tungsten with Myrinet will achieve an efficiency of about 70%. We find that for HARM on the BH cluster, using only 1cpu/node gives faster performance than 2cpus/node by 10-17%. The older Alpha cluster shows no change in performance between 1cpu/node and 2cpus/node. Some Platinum results show 2cpu/node performance problems, especially for 2D MHD simulations that show up to 50% performance degradation on 2cpus/node compared to 1cpu/node. HARM performs quite similarly on both 1 and 2 cpus/node and on any tile geometries. Our collaborating group performing numerical relativity calculations using their latest codes finds an even larger performance drop of 50-70% using 2cpus/node from 1cpu/node. Despite this effect they still plan to purchase dual-CPU nodes due to the cost effectiveness of dual- CPU nodes. They might plausibly modify their code or develop new code to improve dual-CPU efficiency. Our collaborating group finds overall higher MPI efficiencies due to a large number of computations per zone, although they may improve MPI performance by changing from DAGH (an antiquated MPI library) to directly using MPI functions. Notice also that the CPU tile geometry can significantly impact performance. For a fixed tile size, the number of bytes transferred is smallest for an equal number of tiles per dimension. However, for codes that require many small messages such as the 2D MHD code, extended CPU geometries can be more efficient by increasing the bandwidth of message passing and making the CPU to do more work per unit of communication. Otherwise, the CPU is under-utilized due to the system waiting for the messages to complete passing. For example, the 2D MHD code using 28 CPUs and 2cpus/node with 4 × 7 CPU tile geometry achieves only 40% CPU utilization, while with 1 × 28 CPU tile geometry achieves 80% CPU utilization. Assuming the code uses the CPU cache efficiently on the inner loop stride, elongating along the inner loop stride has no benefits. In fact, we find a slightly larger performance for tile geometries elongated along the outer loop stride

208 for 28 CPUs with 2cpus/node for the 2D MHD code. However, for 4 CPUs and 1cpus/node a 2 × 2 geometry is optimal. Increasing the tile size can also aﬀect performance for codes that operate on small message sizes. By comparing the 642 and 2562 cases in Table B.4, notice that the larger 2562 per CPU tile size typically runs faster per CPU and is typically more eﬃcient in MPI due to larger message sizes.

B.7 Beowulf Cluster Summary

We have described the method used to design, build, test, and use a Beowulf cluster from start to finish. The design process involves testing plausible configurations of hardware and software by setting up small test clusters. This process is critical in verifying a specific configuration will be successful. Building a Beowulf cluster involves mounting the nodes on a rack or shelf, considering cooling and electrical issues, and setting up the physical network connections and physical components of the network switch. We found that a simple, cheap rack suffices for the number of nodes (15-28) we purchased. Modifying the electric outlets and circuits in the computer room took time to authorize and complete, and in the meantime we learned that stress-testing a system is key to verifying that the power supplied to the cluster is sufficient. The network has been transparent and working at 100MB/s peak bandwidth and 27µ s latency using standard Intel drivers with some modified parameters. We found that synthetic benchmarks tell only a small part of the performance story, and that testing codes expected to run on the cluster (or similar test cluster) is the only clean way to estimate cluster performance. We generally perform hyperbolic, fluid dynamic simulations of accretion flows around black holes. Our collaborators perform numerical relativistic dynamical space-time simulations. They find very different results on nearly every system we tested, including the final BH cluster. For example, they find their code performs 2X faster on Itanium I CPUs than on Platinum 1Ghz Intel P3 processors, while we find our code performs 2X slower. We have provided tables of single CPU and parallel performance of all our accretion flow codes, including our VHD, 2D MHD, 3D MHD, and GRMHD (HARM) codes. We have tested these codes on several systems, including the SGI Origin 2000, the NCSA Platinum cluster, the NCSA Itanium, and our BH cluster. We expect NCSA’s latest cluster, Tungsten, to show similar performance to BH. Tungsten using Myrinet should show improved efficiency for a large number of processes. We found that tile size, tile geometry, and whether one uses 1cpu/node or 2cpus/node can significantly affect parallel performance. Our HARM code is least affected by such considerations, with 10X the operations per CPU per zone compared to the 2D MHD code. This results in a much longer time between communications, leading naturally to a higher parallel code efficiency. HARM and the 2D MHD codes perform marginally better at 2562 tile size than at 642 tile size. HARM typically performs best with a larger number of tiles along the outer loop stride, especially for the 2562 tile size per CPU. However, the 2D and 3D MHD codes show no consistent preference for tile geometry for their worst case efficiencies. HARM and the 3D MHD codes essentially run

209 the same on 1cpu/node and 2cpus/node, while the 2D MHD code clearly operates more eﬃciently when using 1cpu/node.

210 References

D. C. Abbott. The theory of radiatively driven stellar winds. II - The line acceleration. ApJ, 259: 282–301, August 1982.

T. Abel, G. L. Bryan, and M. L. Norman. The Formation and Fragmentation of Primordial Molecular Clouds. ApJ, 540:39–44, September 2000.

T. Abel, G. L. Bryan, and M. L. Norman. The Formation of the First Star in the Universe. Science, 295:93–98, January 2002.

M. Abramowicz, M. Jaroszynski, and M. Sikora. Relativistic, accreting disks. A&A, 63:221–224, February 1978.

M. A. Abramowicz. Innermost parts of accretion disks are thermally and secularly stable. Nature, 294:235, November 1981.

M. A. Abramowicz, X. Chen, S. Kato, J. Lasota, and O. Regev. Thermal equilibria of accretion disks. ApJ, 438:L37–L39, January 1995.

M. A. Abramowicz, W. Klu´zniak,and J.-P. Lasota. No observational proof of the black-hole event-horizon. A&A, 396:L31–L34, December 2002.

M. A. Abramowicz, W. Kluzniak, Z. Stuchlik, and G. Torok. The orbital resonance model for twin peak kHz QPOs. astro-ph/0401464, January 2004.

E. Agol and M. Kamionkowski. X-rays from isolated black holes in the Milky Way. MNRAS, 334: 553–562, August 2002.

E. Agol, M. Kamionkowski, L. V. E. Koopmans, and R. D. Blandford. Finding Black Holes with Microlensing. ApJ, 576:L131–L135, September 2002.

E. Agol, J. Krolik, N. J. Turner, and J. M. Stone. Two-dimensional Hydrodynamic Simulations of Convection in Radiation-dominated Accretion Disks. ApJ, 558:543–552, September 2001.

E. Agol and J. H. Krolik. Magnetic Stress at the Marginally Stable Orbit: Altered Disk Structure, Radiation, and Black Hole Spin Evolution. ApJ, 528:161–170, January 2000.

F. Aharonian and A. Neronov. High energy gamma rays from the massive black hole in the Galactic Center. astro-ph/0408303, August 2004.

211 C. Akerlof, R. Balsano, S. Barthelemy, J. Bloch, P. Butterworth, D. Casperson, T. Cline, S. Fletcher, F. Frontera, G. Gisler, J. Heise, J. Hills, R. Kehoe, B. Lee, S. Marshall, T. McKay, R. Miller, L. Piro, W. Priedhorsky, J. Szymanski, and J. Wren. Observation of contemporaneous optical radiation from a gamma-ray burst. Nature, 398:400–402, 1999.

S. Akiyama, J. C. Wheeler, D. L. Meier, and I. Lichtenstadt. The Magnetorotational Instability in Core-Collapse Supernova Explosions. ApJ, 584:954–970, February 2003.

A. Akmal, V. R. Pandharipande, and D. G. Ravenhall. Equation of state of nucleon matter and neutron star structure. Phys. Rev. C, 58:1804–1828, September 1998.

M. A. Aloy, E. Müller,J. M. Ibáñez,J. M. Mart´ı, and A. MacFadyen. Relativistic Jets from Collapsars. ApJ, 531:L119–L122, March 2000.

M. I. Andersen, J. Hjorth, H. Pedersen, B. L. Jensen, L. K. Hunt, J. Gorosabel, P. Møller, J. Fynbo, R. M. Kippen, B. Thomsen, L. F. Olsen, L. Christensen, M. Vestergaard, N. Masetti, E. Palazzi, K. Hurley, T. Cline, L. Kaper, and A. O. Jaunsen. VLT identiﬁcation of the optical afterglow of the gamma-ray burst GRB 000131 at z=4.50. A&A, 364:L54–L61, December 2000.

A.M. Anile. Relativistic Fluids and Magnetoﬂuids. Cambridge University Press, Cambridge, U.K., 1989.

P. J. Armitage. Turbulence and Angular Momentum Transport in Global Accretion Disk Simula- tion. ApJ, 501:L189, July 1998.

P. J. Armitage. Theory of disk accretion onto supermassive black holes. astro-ph/0405170, May 2004.

P. J. Armitage and P. Natarajan. The Blandford-Znajek Mechanism and the Emission from Isolated Accreting Black Holes. ApJ, 523:L7–L10, September 1999.

P. J. Armitage, C. S. Reynolds, and J. Chiang. Simulations of Accretion Flows Crossing the Last Stable Orbit. ApJ, 548:868–875, February 2001.

J. W. Armstrong, B. J. Rickett, and S. R. Spangler. Electron density power spectrum in the local interstellar medium. ApJ, 443:209–221, April 1995.

W. Baade. Polarization in the Jet of Messier 87. ApJ, 123:550–551, May 1956.

S. V. Babak and L. P. Grishchuk. Int. J. Mod. Phys. D, 12:1905–1960, 2003.

F. K. Baganoﬀ, Y. Maeda, M. Morris, M. W. Bautz, W. N. Brandt, W. Cui, J. P. Doty, E. D. Feigelson, G. P. Garmire, S. H. Pravdo, G. R. Ricker, and L. K. Townsley. Chandra X-Ray Spectroscopic Imaging of Sagittarius A* and the Central Parsec of the Galaxy. ApJ, 591:891–915, July 2003.

212 S. Bahcall, B. W. Lynn, and S. B. Selipsky. New models for neutron stars. ApJ, 362:251–255, October 1990.

S. Balberg and S. L. Shapiro. Gravothermal Collapse of Self-Interacting Dark Matter Halos and the Origin of Massive Black Holes. Phys. Rev. Lett., 88:101301, March 2002.

S. A. Balbus and J. F. Hawley. A powerful local shear instability in weakly magnetized disks. I - Linear analysis. II - Nonlinear evolution. ApJ, 376:214–233, July 1991.

S. A. Balbus and J. F. Hawley. Instability, turbulence, and enhanced transport in accretion disks. Reviews of Modern Physics, 70:1–53, January 1998.

S. A. Balbus and J. F. Hawley. On the Nature of Angular Momentum Transport in Nonradiative Accretion Flows. ApJ, 573:749–753, July 2002.

S. A. Balbus and C. Terquem. Linear Analysis of the Hall Eﬀect in Protostellar Disks. ApJ, 552: 235–247, May 2001.

D. R. Ballantyne, R. R. Ross, and A. C. Fabian. X-ray reﬂection by photoionized accretion discs. MNRAS, 327:10–22, October 2001.

D. Balsara. Total Variation Diminishing Scheme for Relativistic Magnetohydrodynamics. ApJS, 132:83–101, January 2001.

J. M. Bardeen. Kerr Metric Black Holes. Nature, 226:64, 1970.

T. W. Baumgarte and S. L. Shapiro. Evolution of Rotating Supermassive Stars to the Onset of Collapse. ApJ, 526:941–952, December 1999.

W. Baumjohann and R. A. Treumann. Basic space plasma physics. London: Imperial College Press, 1996.

M. C. Begelman. Super-Eddington Atmospheres That Do Not Blow Away. ApJ, 551:897–906, April 2001.

M. C. Begelman, R. D. Blandford, and M. J. Rees. Theory of extragalactic radio sources. Reviews of Modern Physics, 56:255–351, April 1984.

M. C. Begelman and T. Chiueh. Thermal coupling of ions and electrons by collective eﬀects in two-temperature accretion ﬂows. ApJ, 332:872–890, September 1988.

J. W. Belcher and L. Jr. Davis. Large Amplitude Alfven Waves in the Interplanetary Medium. J. Geophys. Res., 2:3534, 1971.

K. R. Bell and D. N. C. Lin. Using FU Orionis outbursts to constrain self-regulated protostellar disk models. ApJ, 427:987–1004, June 1994.

213 D. P. Bennett, A. C. Becker, J. L. Quinn, A. B. Tomaney, C. Alcock, R. A. Allsman, D. R. Alves, T. S. Axelrod, J. J. Calitz, K. H. Cook, A. J. Drake, P. C. Fragile, K. C. Freeman, M. Geha, K. Griest, B. R. Johnson, S. C. Keller, C. Laws, M. J. Lehner, S. L. Marshall, D. Minniti, C. A. Nelson, B. A. Peterson, P. Popowski, M. R. Pratt, P. J. Quinn, S. H. Rhie, C. W. Stubbs, W. Sutherland, T. Vandehei, and D. Welch. Gravitational Microlensing Events Due to Stellar- Mass Black Holes. ApJ, 579:639–659, November 2002.

A Berdoz, F Foroughi, and C Nussbaum. The diﬀerential cross section of elastic proton-proton scattering at 50 mev. Journal of Physics G: Nuclear Physics, 12(6):L133–L137, 1986.

P. N. Bhat, G. J. Fishman, C. A. Meegan, R. B. Wilson, M. N. Brock, and W. S. Paciesas. Evidence of sub-millisecond structure in a gamma-ray burst. Nature, 359:217, September 1992.

J. Bicak and V. Janis. Magnetic ﬂuxes across black holes. MNRAS, 212:899–915, February 1985.

J. A. Biretta, W. B. Sparks, and F. Macchetto. Hubble Space Telescope Observations of Super- luminal Motion in the M87 Jet. ApJ, 520:621–626, August 1999.

J. A. Biretta, C. P. Stern, and D. E. Harris. The radio to X-ray spectrum of the M87 jet and nucleus. AJ, 101:1632–1646, May 1991.

D. Biskamp. Magnetic reconnection via current sheets. Physics of Fluids, 29:1520–1531, May 1986.

D. Biskamp. Nonlinear magnetohydrodynamics. Cambridge Monographs on Plasma Physics, Cambridge [England]; New York, NY: Cambridge University Press, 1993.

G. S. Bisnovatyi-Kogan. Accretion disks around black holes with account of magnetic ﬁelds. astro-ph/0406466, June 2004.

G. S. Bisnovatyi-Kogan, I. P. Popov, and A. A. Samokhin. The magnetohydrodynamic rotational model of supernova explosion. Ap&SS, 41:287–320, June 1976.

G. S. Bisnovatyi-Kogan and A. A. Ruzmaikin. The Accretion of Matter by a Collapsing Star in the Presence of a Magnetic Field. Ap&SS, 28:45, 1974.

R. D. Blandford. Models of active galactic nuclei. New York Academy Sciences Annals, 422: 303–318, 1984.

R. D. Blandford and M. C. Begelman. On the fate of gas accreting at a low rate on to a black hole. MNRAS, 303:L1–L5, February 1999.

R. D. Blandford and C. F. McKee. Fluid dynamics of relativistic blast waves. Physics of Fluids, 19:1130–1138, August 1976.

214 R. D. Blandford and C. F. McKee. Reverberation mapping of the emission line regions of Seyfert galaxies and quasars. ApJ, 255:419–439, April 1982.

R. D. Blandford and D. G. Payne. Hydromagnetic ﬂows from accretion discs and the production of radio jets. MNRAS, 199:883–903, June 1982.

R. D. Blandford and R. L. Znajek. Electromagnetic extraction of energy from Kerr black holes. MNRAS, 179:433–456, May 1977.

S. I. Blinnikov, N. V. Dunina-Barkovskaya, and D. K. Nadyozhin. Equation of State of a Fermi Gas: Approximations for Various Degrees of Relativism and Degeneracy. ApJS, 106:171, September 1996.

L. Blitz, J. Binney, K. Y. Lo, J. Bally, and P. T. P. Ho. The centre of the Milky Way. Nature, 361:417–424, February 1993.

J. M. Blondin, K. J. Borkowski, and S. P. Reynolds. Dynamics of Fe Bubbles in Young Supernova Remnants. ApJ, 557:782–791, August 2001.

J. S. Bloom, D. A. Frail, and S. R. Kulkarni. Gamma-Ray Burst Energetics and the Gamma-Ray Burst Hubble Diagram: Promises and Limitations. ApJ, 594:674–683, September 2003.

P. Bodenheimer and S. E. Woosley. A two-dimensional supernova model with rotation and nuclear burning. ApJ, 269:281–291, June 1983.

R. C. Bohlin, B. D. Savage, and J. F. Drake. A survey of interstellar H I from L-alpha absorption measurements. II. ApJ, 224:132–142, August 1978.

C. Bolton. Nature, 235:271, 1972.

H. Bondi. On spherically symmetrical accretion. MNRAS, 112:195, 1952.

T. J. M. Boyd and J. J. Sanderson. Plasma dynamics. Plasma dynamics, by T.J.M. Boyd and J.J. Sanderson. London, Nelson, Applications of mathematics series ISBN : 177616113, 1969.

A. H. Bridle, D. H. Hough, C. J. Lonsdale, J. O. Burns, and R. A. Laing. Deep VLA imaging of twelve extended 3CR quasars. AJ, 108:766–820, September 1994.

A. H. Bridle and R. A. Perley. Extragalactic Radio Jets. ARA&A, 22:319–358, 1984.

M. Brio and C. C. Wu. An upwind diﬀerencing scheme for the equations of ideal magnetohydrodynamics. J. Comp. Phys., 85:400–422, 1988.

C. Brocksopp, P. J. Groot, and J. Wilms. The optical long ‘period’ of LMC X-3. MNRAS, 328: 139–146, November 2001.

215 B. C. Bromley, K. Chen, and W. A. Miller. Line Emission from an Accretion Disk around a Rotating Black Hole: Toward a Measurement of Frame Dragging. ApJ, 475:57, January 1997.

V. Bromm, P. S. Coppi, and R. B. Larson. The Formation of the First Stars. I. The Primordial Star-forming Cloud. ApJ, 564:23–51, January 2002.

C. H. Bruneau and E. Creus. Towards a transparent boundary condition for compressible navier- stokes equations. International Journal for Numerical Methods in Fluids, 36(7):807–840, 2001.

A. Burrows, T. Young, P. Pinto, R. Eastman, and T. A. Thompson. A New Algorithm for Supernova Neutrino Transport and Some Applications. ApJ, 539:865–887, August 2000.

N. R. Butler, H. L. Marshall, G. R. Ricker, R. K. Vanderspek, P. G. Ford, G. B. Crew, D. Q. Lamb, and J. G. Jernigan. The X-Ray Afterglows of GRB 020813 and GRB 021004 with Chandra HETGS: Possible Evidence for a Supernova Prior to GRB 020813. ApJ, 597:1010–1016, November 2003.

M. Camenzind. Hydromagnetic ﬂows from rapidly rotating compact objects. I - Cold relativistic ﬂows from rapid rotators. A&A, 162:32–44, July 1986.

A. G. W. Cameron. Some Properties of r-Process Accretion Disks and Jets. ApJ, 562:456–469, November 2001.

C. Cardall and A. Mezzacappa. Conservative formulations of general relativistic kinetic theory. Phys. Rev. D, 68(2):023006, July 2003.

C. Y. Cardall. Conservative Formulations of General Relativistic Radiative Transfer. astro- ph/0404254, April 2004.

J. I. Castor, D. C. Abbott, and R. I. Klein. Radiation-driven winds in Of stars. ApJ, 195:157–174, January 1975.

A. Cavaliere and P. Padovani. Eddington ratios as a constraint on the activity patterns of the active galactic nuclei. ApJ, 333:L33–L36, October 1988.

S. Chandrasekhar. The highly collapsed conﬁgurations of a stellar mass. MNRAS, 91:456, March 1931a.

S. Chandrasekhar. The Maximum Mass of Ideal White Dwarfs. ApJ, 74:81, July 1931b.

S. Chandrasekhar. Hydrodynamic and hydromagnetic stability. International Series of Monographs on Physics, Oxford: Clarendon, 1961.

S. Chapman and T. G. Cowling. The Mathematical Theory of Non-uniform Gases; An Account of the Kinetic Theory of Viscosity, Thermal Conduction and Diﬀusion in Gases. (Cambridge University Press, 1939, 1952, 1970), 1953.

216 K. Chen and J. P. Halpern. Structure of line-emitting accretion disks in active galactic nuclei - ARP 102B. ApJ, 344:115–124, September 1989.

W. Coburn and S. E. Boggs. Polarization of the prompt γ-ray emission from the γ-ray burst of 6 December 2002. Nature, 423:415–417, May 2003.

R. Cornelisse, J. J. M. in’t Zand, F. Verbunt, E. Kuulkers, J. Heise, P. R. den Hartog, M. Coc- chi, L. Natalucci, A. Bazzano, and P. Ubertini. Six years of BeppoSAX Wide Field Cameras observations of nine galactic type I X-ray bursters. A&A, 405:1033–1042, July 2003.

A. P. Cowley. Evidence for black holes in stellar binary systems. ARA&A, 30:287–310, 1992.

A. P. Cowley, D. Crampton, J. B. Hutchings, R. Remillard, and J. E. Penfold. Discovery of a massive unseen star in LMC X-3. ApJ, 272:118–122, September 1983.

A. P. Cowley, P. C. Schmidtke, K. Ebisawa, F. Makino, R. A. Remillard, D. Crampton, J. B. Hutchings, S. Kitamoto, and A. Treves. Discovery of a long-term periodic variation in LMC X-3. ApJ, 381:526–533, November 1991.

T. G. Cowling. The stability of gaseous stars. MNRAS, 94:768–782, June 1934.

C. T. Cunningham. The eﬀects of redshifts and focusing on the spectrum of an accretion disk around a Kerr black hole. ApJ, 202:788–802, December 1975.

Y. Dabrowski, A. C. Fabian, K. Iwasawa, A. N. Lasenby, and C. S. Reynolds. The proﬁle and equivalent width of the X-ray iron emission line from a disc around a Kerr black hole. MNRAS, 288:L11–L15, June 1997.

S. Dado, A. Dar, and A. De R´ujula.On the optical and X-ray afterglows of gamma ray bursts. A&A, 388:1079–1105, June 2002.

J. De Villiers and J. F. Hawley. Three-dimensional Hydrodynamic Simulations of Accretion Tori in Kerr Spacetimes. ApJ, 577:866–879, October 2002.

J. De Villiers and J. F. Hawley. A Numerical Method for General Relativistic Magnetohydrody- namics. ApJ, 589:458–480, May 2003a.

J. De Villiers and J. F. Hawley. Global General Relativistic Magnetohydrodynamic Simulations of Accretion Tori. astro-ph/0303241, March 2003b.

J. De Villiers, J. F. Hawley, and J. H. Krolik. Magnetically Driven Accretion Flows in the Kerr Metric. I. Models and Overall Structure. ApJ, 599:1238–1253, December 2003a.

J. De Villiers, J. F. Hawley, and J. H. Krolik. Magnetically Driven Accretion Flows in the Kerr Metric I: Models and Overall Structure. astro-ph/0307260, July 2003b.

217 J. De Villiers, J. F. Hawley, J. H. Krolik, and S. Hirose. Magnetically Driven Accretion in the Kerr Metric III: Unbound Outﬂows. astro-ph/0407092, July 2004.

Simon DeDeo and Dimitrios Psaltis. Towards new tests of strong-ﬁeld gravity with measurements of surface atomic line redshifts from neutron stars. Phys. Rev. Lett., 90(14):141101, 2003.

A. Dedner, D. Kroner, I. L. Sofronov, and M. Wesenberg. Transparent boundary conditions for mhd simulations in stratiﬁed atmospheres. J. Comp. Phys., 171(2):448–478, Aug 2001.

L. Del Zanna and N. Bucciantini. An eﬃcient shock-capturing central-type scheme for multidimensional relativistic ﬂows. I. Hydrodynamics. A&A, 390:1177–1186, August 2002.

L. Del Zanna, N. Bucciantini, and P. Londrillo. An eﬃcient shock-capturing central-type scheme for multidimensional relativistic ﬂows. II. Magnetohydrodynamics. A&A, 400:397–413, March 2003.

T. Di Matteo, R. Perna, and R. Narayan. Neutrino Trapping and Accretion Models for Gamma- Ray Bursts. ApJ, 579:706–715, November 2002.

T. di Matteo and D. Psaltis. Quasi-Periodic Variability and the Inner Radii of Thin Accretion Disks in Galactic Black Hole Systems. ApJ, 526:L101–L104, December 1999.

S. S. Doeleman, Z.-Q. Shen, A. E. E. Rogers, G. C. Bower, M. C. H. Wright, J. H. Zhao, D. C. Backer, J. W. Crowley, R. W. Freund, P. T. P. Ho, K. Y. Lo, and D. P. Woody. Structure of Sagittarius A* at 86 GHZ using VLBI Closure Quantities. AJ, 121:2610–2617, May 2001.

C. Done, G. Wardzi´nski,and M. Gierli´nski. GRS 1915+105: the brightest Galactic black hole. MNRAS, 349:393–403, April 2004.

B. T. Draine, W. G. Roberge, and A. Dalgarno. Magnetohydrodynamic shock waves in molecular clouds. ApJ, 264:485–507, January 1983.

J. F. Drake, R. G. Kleva, and M. E. Mandt. Structure of thin current layers: Implications for magnetic reconnection. Phys. Rev. Lett., 73:1251–1254, August 1994.

G. Drenkhahn. Acceleration of GRB outﬂows by Poynting ﬂux dissipation. A&A, 387:714–724, May 2002.

G. Drenkhahn and H. C. Spruit. Efficient acceleration and radiation in Poynting flux powered GRB outflows. A&A, 391:1141–1153, September 2002.

R. C. Duncan and C. Thompson. Formation of very strongly magnetized neutron stars - Implica- tions for gamma-ray bursts. ApJ, 392:L9–L13, June 1992.

D. M. Eardley and A. P. Lightman. Magnetic viscosity in relativistic accretion disks. ApJ, 200: 187–203, August 1975.

218 K. Ebisawa, F. Makino, K. Mitsuda, T. Belloni, A. P. Cowley, P. C. Schmidtke, and A. Treves. Spectral variations of LMC X-3 observed with GINGA. ApJ, 403:684–689, February 1993.

A. Eckart and R. Genzel. Stellar proper motions in the central 0.1 PC of the Galaxy. MNRAS, 284:576–598, January 1997.

A. Eckart, R. Genzel, T. Ott, and R. Sch¨odel.Stellar orbits near Sagittarius A*. MNRAS, 331: 917–934, April 2002.

J. P. Eckman. Roads to Turbulence in Dissipative Dynamical Systems. Reviews of Modern Physics, 53:643–654, 1981.

A. S. Eddington. On “relativistic degeneracy”. MNRAS, 95:194–206, January 1935.

R. Edelson and S. Vaughan. The Rapidly Variable X-ray Spectrum of MCG-6-30-15: Evidence for Variability in Both the Iron K-alpha Line Flux and Continuum Slope of MCG–6-30-15. Bulletin of the American Astronomical Society, 32:1189, October 2000.

D. Eichler, M. Livio, T. Piran, and D. N. Schramm. Nucleosynthesis, neutrino bursts and gamma- rays from coalescing neutron stars. Nature, 340:126–128, July 1989.

A. Einstein. Die Grundlage der allgemeinen Relativittstheorie. Annalen der Physik, 49:769, 1916.

M. Elvis. A Structure for Quasars. ApJ, 545:63–76, December 2000.

M. Elvis, G. Risaliti, and G. Zamorani. Most Supermassive Black Holes Must Be Rapidly Rotating. ApJ, 565:L75–L77, February 2002.

T. Erber. High-Energy Electromagnetic Conversion Processes in Intense Magnetic Fields. Reviews of Modern Physics, 38:626–659, October 1966.

C. R. Evans and J. F. Hawley. Simulation of magnetohydrodynamic ﬂows - A constrained transport method. ApJ, 332:659–677, September 1988.

N. J. Evans. Physical Conditions in Regions of Star Formation. ARA&A, 37:311–362, 1999.

G. Fabbiano, A. Zezas, A. R. King, T. J. Ponman, A. Rots, and F. Schweizer. The Time-variable Ultraluminous X-Ray Sources of “The Antennae”. ApJ, 584:L5–L8, February 2003.

A. C. Fabian. X-Ray Iron Line Observations of Accretion in AGN. In American Institute of Physics Conference Series, page 247, 1998.

A. C. Fabian, M. J. Rees, L. Stella, and N. E. White. X-ray ﬂuorescence from the inner disc in Cygnus X-1. MNRAS, 238:729–736, May 1989.

A. C. Fabian and S. Vaughan. The iron line in MCG-6-30-15 from XMM-Newton: evidence for gravitational light bending? MNRAS, 340:L28–L32, April 2003.

219 A. C. Fabian, S. Vaughan, K. Nandra, K. Iwasawa, D. R. Ballantyne, J. C. Lee, A. De Rosa, A. Turner, and A. J. Young. A long hard look at MCG-6-30-15 with XMM-Newton. MNRAS, 335:L1–L5, September 2002.

W. C. Feldman and E. Marsch. Kinetic Phenomena in the Solar Wind. in Cosmic Winds and the Heliosphere, edited by J. R. Jokipii et al., University of Arizona Press, Tucson, pp. 617676, 1997.

R. Fender and T. Belloni. GRS 1915+105 and the Disc-Jet Coupling in Accreting Black Hole Systems. astro-ph/0406483, June 2004.

L. Ferrarese, H. C. Ford, and W. Jaﬀe. Evidence for a Massive Black Hole in the Active Galaxy NGC 4261 from Hubble Space Telescope Images and Spectra. ApJ, 470:444, October 1996.

L. G. Fishbone and V. Moncrief. Relativistic ﬂuid disks in orbit around Kerr black holes. ApJ, 207:962–976, August 1976.

J. A. Font. Numerical Hydrodynamics in General Relativity. Living Reviews in Relativity, 6:4, August 2003.

H. C. Ford, R. J. Harms, Z. I. Tsvetanov, G. F. Hartig, L. L. Dressel, G. A. Kriss, R. C. Bohlin, A. F. Davidsen, B. Margon, and A. K. Kochhar. Narrowband HST images of M87: Evidence for a disk of ionized gas around a massive black hole. ApJ, 435:L27–L30, November 1994.

D A Frail, S R Kulkarni, L Nicastro, M Feroci, and G B Taylor. The radio afterglow from the γ-ray burst of 8 may 1997. Nature, 389(6648):261–263, 1997.

D. A. Frail, S. R. Kulkarni, R. Sari, S. G. Djorgovski, J. S. Bloom, T. J. Galama, D. E. Reichart, E. Berger, F. A. Harrison, P. A. Price, S. A. Yost, A. Diercks, R. W. Goodrich, and F. Chaﬀee. Beaming in Gamma-Ray Bursts: Evidence for a Standard Energy Reservoir. ApJ, 562:L55–L58, November 2001.

J. Frank, A. King, and D. Raine. Accretion Power in Astrophysics. Accretion Power in Astro- physics, ISBN 0521408636, Cambridge University Press, 1992.

J. Frank, A. King, and D. J. Raine. Accretion Power in Astrophysics: Third Edition. Accretion Power in Astrophysics: Third Edition, by Juhan Frank, Andrew King, and Derek J. Raine. Cam- bridge University Press, 398 pp., 2002.

U. Frisch. Turbulence. The legacy of A.N. Kolmogorov. Cambridge: Cambridge University Press, 1995.

A. S. Fruchter, S. E. Thorsett, M. R. Metzger, K. C. Sahu, L. Petro, M. Livio, H. Ferguson, E. Pian, D. W. Hogg, T. Galama, T. R. Gull, C. Kouveliotou, D. Macchetto, J. van Paradijs, H. Pedersen, and A. Smette. Hubble Space Telescope and Palomar Imaging of GRB 990123:

220 Implications for the Nature of Gamma-Ray Bursts and Their Hosts. ApJ, 519:L13–L16, July 1999.

C. L. Fryer. Mass Limits For Black Hole Formation. ApJ, 522:413–418, September 1999.

C. L. Fryer and P. M´esz´aros.Neutrino-driven Explosions in Gamma-Ray Bursts and Hypernovae. ApJ, 588:L25–L28, May 2003.

T. J. Galama, P. M. Vreeswijk, J. van Paradijs, C. Kouveliotou, T. Augusteijn, H. Bohnhardt, J. P. Brewer, V. Doublier, J.-F. Gonzalez, B. Leibundgut, C. Lidman, O. R. Hainaut, F. Patat, J. Heise, J. in ’t Zand, K. Hurley, P. J. Groot, R. G. Strom, P. A. Mazzali, K. Iwamoto, K. Nomoto, H. Umeda, T. Nakamura, T. R. Young, T. Suzuki, T. Shigeyama, T. Koshut, M. Kippen, C. Robin- son, P. de Wildt, R. A. M. J. Wijers, N. Tanvir, J. Greiner, E. Pian, E. Palazzi, F. Frontera, N. Masetti, L. Nicastro, M. Feroci, E. Costa, L. Piro, B. A. Peterson, C. Tinney, B. Boyle, R. Cannon, R. Stathakis, E. Sadler, M. C. Begam, and P. Ianna. An unusual supernova in the error box of the gamma-ray burst of 25 April 1998. Nature, 395:670–672, 1998.

A. A. Galeev, R. Rosner, and G. S. Vaiana. Structured coronae of accretion disks. ApJ, 229: 318–326, April 1979.

C. F. Gammie. Photon bubbles in accretion discs. MNRAS, 297:929–935, July 1998.

C. F. Gammie. Eﬃciency of Magnetized Thin Accretion Disks in the Kerr Metric. ApJ, 522: L57–L60, September 1999.

C. F. Gammie. The Magnetorotational Instability in the Kerr Metric. astro-ph/0406532, June 2004.

C. F. Gammie, J. C. McKinney, and G. T´oth. HARM: A Numerical Scheme for General Rela- tivistic Magnetohydrodynamics. ApJ, 589:444–457, May 2003.

C. F. Gammie, R. Narayan, and R. Blandford. What Is the Accretion Rate in NGC 4258? ApJ, 516:177–186, May 1999.

C. F. Gammie and R. Popham. Advection-dominated Accretion Flows in the Kerr Metric. I. Basic Equations. ApJ, 498:313, May 1998.

C. F. Gammie, S. L. Shapiro, and J. C. McKinney. Black Hole Spin Evolution. ApJ, 602:312–319, February 2004.

M. R. Garcia, J. E. McClintock, R. Narayan, P. Callanan, D. Barret, and S. S. Murray. New Evidence for Black Hole Event Horizons from Chandra. ApJ, 553:L47–L50, May 2001.

R. Genzel, A. Eckart, T. Ott, and F. Eisenhauer. On the nature of the dark mass in the centre of the Milky Way. MNRAS, 291:219–234, October 1997.

221 R. Genzel and C. H. Townes. Physical conditions, dynamics and mass distribution in the center of the galaxy. NASA STI/Recon Technical Report N, 87:29391, April 1987.

I. M. George and A. C. Fabian. X-ray reﬂection from cold matter in active galactic nuclei and X-ray binaries. MNRAS, 249:352–367, March 1991.

A. M. Ghez, B. L. Klein, M. Morris, and E. E. Becklin. High Proper-Motion Stars in the Vicinity of Sagittarius A*: Evidence for a Supermassive Black Hole at the Center of Our Galaxy. ApJ, 509:678–686, December 1998.

A. M. Ghez, M. Morris, E. E. Becklin, A. Tanner, and T. Kremenek. The accelerations of stars orbiting the Milky Way’s central black hole . Nature, 407:349–351, September 2000.

P. Ghosh and M. A. Abramowicz. Electromagnetic Extraction of Rotational Energy from Disk-fed Black Holes: the Strength of the Blandford-Znajek Process. Bulletin of the American Astronomical Society, 29:1313, December 1997.

D. R. Gies and C. T. Bolton. The optical spectrum of HDE 226868 = Cygnus X-1. II - Spec- trophotometry and mass estimates. ApJ, 304:371–393, May 1986.

P. Goldreich and W. H. Julian. Pulsar Electrodynamics. ApJ, 157:869, August 1969.

P. Goldreich and A. Reisenegger. Magnetic ﬁeld decay in isolated neutron stars. ApJ, 395:250–258, August 1992.

M. L. Goldstein, D. A. Roberts, and W. H. Matthaeus. Magnetohydrodynamic Turbulence In The Solar Wind. ARA&A, 33:283–326, 1995.

L. J. Greenhill, C. Henkel, R. Becker, T. L. Wilson, and J. G. A. Wouterloot. Centripetal acceleration within the subparsec nuclear maser disk of NGC4258. A&A, 304:21, December 1995a.

L. J. Greenhill, D. R. Jiang, J. M. Moran, M. J. Reid, K. Y. Lo, and M. J. Claussen. Detection of a Subparsec Diameter Disk in the Nucleus of NGC 4258. ApJ, 440:619, February 1995b.

J. Greiner, J. G. Cuby, and M. J. McCaughrean. An unusually massive stellar black hole in the Galaxy. Nature, 414:522–525, November 2001a.

J. Greiner, J. G. Cuby, M. J. McCaughrean, A. J. Castro-Tirado, and R. E. Mennickent. Identi- ﬁcation of the donor in the X-ray binary GRS 1915+105. A&A, 373:L37–L40, July 2001b.

J. Greiner, S. Klose, K. Reinsch, H. M. Schmid, R. Sari, D. H. Hartmann, C. Kouveliotou, A. Rau, E. Palazzi, C. Straubmeier, B. Stecklum, S. Zharikov, G. Tovmassian, O. Barnbantner, C. Ries, E. Jehin, A. Henden, A. A. Kaas, T. Grav, J. Hjorth, H. Pedersen, R. A. M. J. Wijers, A. Kaufer, H. S. Park, G. Williams, and O. Reimer. The polarization evolution of the optical afterglow of GRB 030329. astro-ph/0312089, December 2003a.

222 J. Greiner, M. Peimbert, C. Estaban, A. Kaufer, A. Jaunsen, J. Smoke, S. Klose, and O. Reimer. Redshift of GRB 030329. GRB Circular Network, 2020:1, 2003b.

H.-J. Grimm, M. Gilfanov, and R. Sunyaev. The Milky Way in X-rays for an outside observer. Log(N)-Log(S) and luminosity function of X-ray binaries from RXTE/ASM data. A&A, 391: 923–944, September 2002.

Z. Haiman and A. Loeb. What Is the Highest Plausible Redshift of Luminous Quasars? ApJ, 552:459–463, May 2001.

M. J. Hardcastle, D. M. Worrall, R. P. Kraft, W. R. Forman, C. Jones, and S. S. Murray. Radio and X-Ray Observations of the Jet in Centaurus A. ApJ, 593:169–183, August 2003.

E. T. Harlaftis and J. Greiner. The rotational broadening and the mass of the donor star of GRS 1915+105. A&A, 414:L13–L16, January 2004.

R. J. Harms, H. C. Ford, Z. I. Tsvetanov, G. F. Hartig, L. L. Dressel, G. A. Kriss, R. Bohlin, A. F. Davidsen, B. Margon, and A. K. Kochhar. HST FOS spectroscopy of M87: Evidence for a disk of ionized gas around a massive black hole. ApJ, 435:L35–L38, November 1994.

A. Harten, P.D. Lax, and B. van Leer. On upstream diﬀerencing and godunov-type schemes for hyperbolic conservation laws. SIAM Rev., 25:35–61, 1983.

J. B. Hartle. Bounds on the mass and moment of inertia of non-rotating neutron stars. Physics Reports, 46(6):201–247, 1978.

A. Hasegawa. Plasma instabilities and nonlinear eﬀects. Physics and Chemistry in Space, Berlin: Springer, 1975.

S. W. Hawking and G. F. R. Ellis. The large scale structure of space-time. Cambridge Monographs on Mathematical Physics, London: Cambridge University Press, 1973.

S. W. Hawking and J. B. Hartle. Energy and angular momentum ﬂow into black holes. Commu- nications in Mathematical Physics, 27:283, 1972.

J. F. Hawley. Three-dimensional simulations of black hole tori. ApJ, 381:496–507, November 1991.

J. F. Hawley. Global Magnetohydrodynamical Simulations of Accretion Tori. ApJ, 528:462–479, January 2000.

J. F. Hawley. Global Magnetohydrodynamic Simulations of Cylindrical Keplerian Disks. ApJ, 554:534–547, June 2001.

J. F. Hawley and S. A. Balbus. A Powerful Local Shear Instability in Weakly Magnetized Disks. II. Nonlinear Evolution. ApJ, 376:223, July 1991.

223 J. F. Hawley, S. A. Balbus, and J. M. Stone. A Magnetohydrodynamic Nonradiative Accretion Flow in Three Dimensions. ApJ, 554:L49–L52, June 2001.

J. F. Hawley, C. F. Gammie, and S. A. Balbus. Local Three-dimensional Magnetohydrodynamic Simulations of Accretion Disks. ApJ, 440:742, February 1995.

J. F. Hawley and J. H. Krolik. Global MHD Simulation of the Inner Accretion Disk in a Pseudo- Newtonian Potential. ApJ, 548:348–367, February 2001.

J. F. Hawley and J. H. Krolik. High-Resolution Simulations of the Plunging Region in a Pseudo- Newtonian Potential: Dependence on Numerical Resolution and Field Topology. ApJ, 566:164– 180, February 2002.

J. F. Hawley, J. R. Wilson, and L. L. Smarr. A numerical study of nonspherical black hole accretion. II - Finite diﬀerencing and code calibration. ApJS, 55:211–246, June 1984.

J. Heise, J. in’t Zand, R. M. Kippen, and P. M. Woods. X-Ray Flashes and X-Ray Rich Gamma Ray Bursts. In Gamma-ray Bursts in the Afterglow Era, page 16, 2001.

A. Herrero, R. P. Kudritzki, R. Gabler, J. M. Vilchez, and A. Gabler. Fundamental parameters of galactic luminous OB stars. II. A spectroscopic analysis of HDE 226868 and the mass of Cygnus X-1. A&A, 297:556, May 1995.

J. R. Herrnstein, L. J. Greenhill, J. M. Moran, P. J. Diamond, M. Inoue, N. Nakai, and M. Miyoshi. VLBA Continuum Observations of NGC 4258: Constraints on an Advection-dominated Accretion Flow. ApJ, 497:L69, April 1998.

J. R. Herrnstein, J. M. Moran, L. J. Greenhill, P. J. Diamond, M. Inoue, N. Nakai, M. Miyoshi, C. Henkel, and A. Riess. A geometric distance to the galaxy NGC 4258 from orbital motions in a nuclear gas disk. Nature, 400:539–541, 1999.

J. S. Heyl and L. Hernquist. Nonlinear QED eﬀects in strong-ﬁeld magnetohydrodynamics. Phys. Rev. D, 59(4):045005, February 1999.

S. Hirose, J. H. Krolik, J. De Villiers, and J. F. Hawley. Magnetically Driven Accretion Flows in the Kerr Metric II: Structure of the Magnetic Field. astro-ph/0311500, November 2003.

J. Hjorth, J. Sollerman, P. Møller, J. P. U. Fynbo, S. E. Woosley, C. Kouveliotou, N. R. Tanvir, J. Greiner, M. I. Andersen, A. J. Castro-Tirado, J. M. Castro Cer´on,A. S. Fruchter, J. Gorosabel, P. Jakobsson, L. Kaper, S. Klose, N. Masetti, H. Pedersen, K. Pedersen, E. Pian, E. Palazzi, J. E. Rhoads, E. Rol, E. P. J. van den Heuvel, P. M. Vreeswijk, D. Watson, and R. A. M. J. Wijers. A very energetic supernova associated with the γ-ray burst of 29 March 2003. Nature, 423:847–850, June 2003.

224 L. C. Ho. The Spectral Energy Distributions of Low-Luminosity Active Galactic Nuclei. ApJ, 516:672–682, May 1999.

Y. F. Huang, Z. G. Dai, T. Lu, K. S. Cheng, and X. F. Wu. Gamma-Ray Bursts from Neutron Star Kicks. ApJ, 594:919–923, September 2003.

S. A. Hughes and R. D. Blandford. Black Hole Mass and Spin Coevolution by Mergers. ApJ, 585: L101–L104, March 2003.

S. Ichimaru. Bimodal behavior of accretion disks - Theory and application to Cygnus X-1 transitions. ApJ, 214:840–855, June 1977.

I. V. Igumenshchev and M. A. Abramowicz. Rotating accretion ﬂows around black holes: convection and variability. MNRAS, 303:309–320, February 1999.

I. V. Igumenshchev and M. A. Abramowicz. Two-dimensional Models of Hydrodynamical Accre- tion Flows into Black Holes. ApJS, 130:463–484, October 2000.

I. V. Igumenshchev, M. A. Abramowicz, and R. Narayan. Numerical Simulations of Convective Accretion Flows in Three Dimensions. ApJ, 537:L27–L30, July 2000.

R. R. Islam, J. E. Taylor, and J. Silk. Massive black hole remnants of the ﬁrst stars in galactic haloes. MNRAS, 340:647–656, April 2003.

N. Itoh, T. Adachi, M. Nakagawa, Y. Kohyama, and H. Munakata. Neutrino energy loss in stellar interiors. III - Pair, photo-, plasma, and bremsstrahlung processes. ApJ, 339:354–364, April 1989.

N. Itoh, H. Hayashi, A. Nishikawa, and Y. Kohyama. Neutrino Energy Loss in Stellar Interiors. VII. Pair, Photo-, Plasma, Bremsstrahlung, and Recombination Neutrino Processes. ApJS, 102: 411, February 1996.

K. Iwamoto, P. A. Mazzali, K. Nomoto, H. Umeda, T. Nakamura, F. Patat, I. J. Danziger, T. R. Young, T. Suzuki, T. Shigeyama, T. Augusteijn, V. Doublier, J.-F. Gonzalez, H. Boehnhardt, J. Brewer, O. R. Hainaut, C. Lidman, B. Leibundgut, E. Cappellaro, M. Turatto, T. J. Galama, P. M. Vreeswijk, C. Kouveliotou, J. van Paradijs, E. Pian, E. Palazzi, and F. Frontera. A hypernova model for the supernova associated with the gamma-ray burst of 25 April 1998. Nature, 395:672– 674, 1998.

J. D. Jackson. Classical Electrodynamics. New York: Wiley, 1962.

W. Jaﬀe, H. C. Ford, L. Ferrarese, F. van den Bosch, and R. W. O’Connell. A large nuclear accretion disk in the active galaxy NGC4261. Nature, 364:213–215, July 1993.

M. Jaroszynski, M. A. Abramowicz, and B. Paczy´nski.Supercritical accretion disks around black holes. Acta Astronomica, 30:1–34, 1980.

225 W. Junor, J. A. Biretta, and M. Livio. Formation of the radio jet in M87 at 100 Schwarzschild radii from the central black hole. Nature, 401:891–892, October 1999.

P. Kaaret, A. H. Prestwich, A. Zezas, S. S. Murray, D.-W. Kim, R. E. Kilgard, E. M. Schlegel, and M. J. Ward. Chandra High-Resolution Camera observations of the luminous X-ray source in the starburst galaxy M82. MNRAS, 321:L29–L32, February 2001.

B. B. Kadomtsev. Plasma turbulence. New York: Academic Press, 1965.

S. W. Kahler. Solar ﬂares and coronal mass ejections. ARA&A, 30:113–141, 1992.

V. Karas, D. Vokrouhlicky, and A. G. Polnarev. In the vicinity of a rotating black hole - A fast numerical code for computing observational eﬀects. MNRAS, 259:569–575, December 1992.

S. Karni. To the boundary and back - a numerical study. International Journal on Numerical Methods for Fluids, 13:201–216, 1991.

S. Kato. Mass and Spin of GRS 1915+105 Based on a Resonance Model of QPOs. astro- ph/0407251, July 2004.

K. S. Kawabata, J. Deng, L. Wang, P. Mazzali, K. Nomoto, K. Maeda, N. Tominaga, H. Umeda, M. Iye, G. Kosugi, Y. Ohyama, T. Sasaki, P. Höflich, J. C. Wheeler, D. J. Jeffery, K. Aoki, N. Kashikawa, T. Takata, N. Kawai, T. Sakamoto, Y. Urata, A. Yoshida, T. Tamagawa, K. Torii, W. Aoki, N. Kobayashi, Y. Komiyama, Y. Mizumoto, J. Noumaru, R. Ogasawara, K. Sekiguchi, Y. Shirasaki, T. Totani, J. Watanabe, and T. Yamada. On the Spectrum and Spectropolarimetry of Type Ic Hypernova SN 2003dh/GRB 030329. ApJ, 593:L19–L22, August 2003.

J. W. Keohane, W. T. Reach, J. Rho, and T. H. Jarrett. Near Infrared Observations of the Super- nova Remnant W49B: A Possible Collapsar Remnant. American Astronomical Society Meeting, 204, May 2004.

A. M. Khokhlov. The structure of detonation waves in supernovae. MNRAS, 239:785–808, August 1989.

A. M. Khokhlov, P. A. H¨oﬂich, E. S. Oran, J. C. Wheeler, L. Wang, and A. Y. Chtchelkanova. Jet-induced Explosions of Core Collapse Supernovae. ApJ, 524:L107–L110, October 1999.

H. Kim, H. M. Lee, C. H. Lee, and H. K. Lee. Pulsar Magnetosphere : A General Relativistic Treatment. astro-ph/0407500, July 2004.

A. R. King, M. B. Davies, M. J. Ward, G. Fabbiano, and M. Elvis. Ultraluminous X-Ray Sources in External Galaxies. ApJ, 552:L109–L112, May 2001.

R. M. Kippen. GRB 990510: BATSE observations. GRB Circular Network, 322:1, 1999.

226 R. W. Klebesadel, I. B. Strong, and R. A. Olson. Observations of Gamma-Ray Bursts of Cosmic Origin. ApJ, 182:L85, June 1973.

K. Kohri and S. Mineshige. Can Neutrino-cooled Accretion Disks Be an Origin of Gamma-Ray Bursts? ApJ, 577:311–321, September 2002.

S. Koide. Magnetic extraction of black hole rotational energy: Method and results of general relativistic magnetohydrodynamic simulations in Kerr space-time. Phys. Rev. D, 67:104010, May 2003.

S. Koide, D. L. Meier, K. Shibata, and T. Kudoh. General Relativistic Simulations of Early Jet Formation in a Rapidly Rotating Black Hole Magnetosphere. ApJ, 536:668–674, June 2000.

S. Koide, K. Shibata, and T. Kudoh. Relativistic Jet Formation from Black Hole Magnetized Accretion Disks: Method, Tests, and Applications of a General RelativisticMagnetohydrodynamic Numerical Code. ApJ, 522:727–752, September 1999.

S. Koide, K. Shibata, T. Kudoh, and D. L. Meier. Extraction of Black Hole Rotational Energy by a Magnetic Field and the Formation of Relativistic Jets. Science, 295:1688–1691, March 2002.

A. V. Koldoba, O. A. Kuznetsov, and G. V. Ustyugova. An approximate Riemann solver for relativistic magnetohydrodynamics. MNRAS, 333:932–942, July 2002.

A. N. Kolmogorov. Dissipation of energy in the locally isotropic turbulence. Comptes rendus (Doklady) de l’Acadmie des Sciences de l’U.R.S.S., 32:16–18, 1941a.

A. N. Kolmogorov. The local structure of turbulence in incompressible viscous ﬂuid for very large Reynold’s numbers. Comptes rendus (Doklady) de l’Acadmie des Sciences de l’U.R.S.S., 30: 301–305, 1941b.

S. S. Komissarov. A Godunov-type scheme for relativistic magnetohydrodynamics. MNRAS, 303: 343–366, February 1999.

S. S. Komissarov. Direct numerical simulations of the Blandford-Znajek eﬀect. MNRAS, 326: L41–L44, September 2001.

S. S. Komissarov. Limit shocks of relativistic magnetohydrodynamics, Punsly’s waveguide and the Blandford-Znajek solution. astro-ph/0206076, June 2002a.

S. S. Komissarov. Test problems for relativistic magnetohydrodynamics. astro-ph/0209213, September 2002b.

S. S. Komissarov. Time-dependent, force-free, degenerate electrodynamics. MNRAS, 336:759–766, November 2002c.

227 S. S. Komissarov. Electrodynamics of black hole magnetospheres. astro-ph/0402403, February 2004a.

S. S. Komissarov. General relativistic MHD simulations of monopole magnetospheres of black holes. astro-ph/0402430, February 2004b.

J. Kormendy and K. Gebhardt. Supermassive Black Holes in Galactic Nuclei (Plenary Talk). In AIP Conf. Proc. 586: 20th Texas Symposium on relativistic astrophysics, page 363, 2001.

J. Kormendy and D. Richstone. Inward Bound—The Search For Supermassive Black Holes In Galactic Nuclei. ARA&A, 33:581, 1995.

C. Kouveliotou, C. A. Meegan, G. J. Fishman, N. P. Bhat, M. S. Briggs, T. M. Koshut, W. S. Paciesas, and G. N. Pendleton. Identiﬁcation of two classes of gamma-ray bursts. ApJ, 413: L101–L104, August 1993.

R. P. Kraft, S. E. V´azquez,W. R. Forman, C. Jones, S. S. Murray, M. J. Hardcastle, D. M. Worrall, and E. Churazov. X-Ray Emission from the Hot Interstellar Medium and Southwest Radio Lobe of the Nearby Radio Galaxy Centaurus A. ApJ, 592:129–146, July 2003.

N. A. Krall and A. W. Trivelpiece. Principles of plasma physics. International Student Edition - International Series in Pure and Applied Physics, Tokyo: McGraw-Hill Kogakusha, 1973.

R. Krasnopolsky, Z. Li, and R. Blandford. Magnetocentrifugal Launching of Jets from Accretion Disks. I. Cold Axisymmetric Flows. ApJ, 526:631–642, December 1999.

R. Krasnopolsky, Z. Li, and R. D. Blandford. Magnetocentrifugal Launching of Jets from Accretion Disks. II. Inner Disk-driven Winds. ApJ, 595:631–642, October 2003.

J. H. Krolik. Active galactic nuclei : from the central black hole to the galactic environment. Active galactic nuclei : from the central black hole to the galactic environment /Julian H. Kro- lik. Princeton, N. J. : Princeton University Press, 1999a.

J. H. Krolik. Magnetized Accretion inside the Marginally Stable Orbit around a Black Hole. ApJ, 515:L73–L76, April 1999b.

J. H. Krolik. Are Ultra-Luminous X-ray Sources Intermediate Mass Black Holes Accreting from Molecular Clouds? astro-ph/0407285, July 2004.

J. H. Krolik and J. F. Hawley. Where Is the Inner Edge of an Accretion Disk around a Black Hole? ApJ, 573:754–763, July 2002.

O. A. Kuznetsov, G. S. Bisnovatyi-Kogan, and D. Molteni. On the development of hydrodynamical turbulence in accretion discs: numerical simulations. MNRAS, submitted, 2004.

228 L. D. Landau and E. M. Lifshitz. Fluid mechanics. Course of theoretical physics, Oxford: Perga- mon Press, 1959.

L. D. Landau and E. M. Lifshitz. Statistical physics. Pt.1, Pt.2. Course of theoretical physics, Pergamon International Library of Science, Technology, Engineering and Social Studies, Oxford: Pergamon Press, 3rd rev.and enlarg. ed., 1980.

A. Laor. Line proﬁles from a disk around a rotating black hole. ApJ, 376:90–94, July 1991.

R. B. Larson. The Stellar Initial Mass Function and Beyond. astro-ph/0205466, May 2002.

J. M. Lattimer. The Nuclear Equation of State and Supernovae. Nuclear Equation of State, ed. A. Ansari & L. Satpathy (World Scientiﬁc, Singapore), 1996.

J. M. Lattimer and M. Prakash. Nuclear matter and its role in supernovae, neutron stars and compact object binary mergers. Phys. Rep., 333:121–146, 2000.

J. M. Lattimer and D. Swesty. Nucl. Phys. A, 535:3310376, 1991.

P.D. Lax and B. Wendroﬀ. Systems of conservation laws. Commun. Pure Appl. Math., 13:217–237, 1960.

A. Lazarian, D. Pogosyan, and A. Esquivel. Quest for HI Turbulence Statistics: New Techniques. astro-ph/0112368, December 2001.

R. J. Leamon, C. W. Smith, N. F. Ness, W. H. Matthaeus, and H. K. Wong. Observational constraints on the dynamics of the interplanetary magnetic ﬁeld dissipation range. J. Geophys. Res., 103:4775, March 1998.

J. M. Leblanc and J. R. Wilson. A Numerical Example of the Collapse of a Rotating Magnetized Star. ApJ, 161:541, August 1970.

J. C. Lee, A. C. Fabian, C. S. Reynolds, W. N. Brandt, and K. Iwasawa. The X-ray variability of the Seyfert 1 galaxy MCG-6-30-15 from long ASCA and RXTE observations. MNRAS, 318: 857–874, November 2000.

W. H. Lee, E. Ramirez-Ruiz, and D. Page. Opaque or Transparent? A Link between Neutrino Optical Depths and the Characteristic Duration of Short Gamma-Ray Bursts. ApJ, 608:L5–L8, June 2004.

R.J. LeVeque. Nonlinear Conservation Laws and Finite Volume Methods, pages 1–159. Springer, Berlin, Germany, 1998.

W. H. G. Lewin, J. van Paradijs, and E. P. J. van den Heuvel. X-ray binaries. Cambridge Astrophysics Series, Cambridge, MA: Cambridge University Press, edited by Lewin, Walter H.G.; Van Paradijs, Jan; Van den Heuvel, Edward P.J., 1995.

229 L. Li. Energetics of Black Hole-Accretion Disk System with Magnetic Connection: Limit of Low Accretion Rate. astro-ph/0406353, June 2004.

E. W. Liang, X. F. Wu, and Z. G. Dai. Testing the Predictions of the Universal-Structured Jet Model of Gamma-Ray Bursts by Simulations. astro-ph/0407443, July 2004.

M. Liebend¨orfer,O. E. B. Messer, A. Mezzacappa, S. W. Bruenn, C. Y. Cardall, and F.-K. Thiele- mann. A Finite Diﬀerence Representation of Neutrino Radiation Hydrodynamics in Spherically Symmetric General Relativistic Spacetime. ApJS, 150:263–316, January 2004.

M. Liebend¨orfer,A. Mezzacappa, F. Thielemann, O. E. Messer, W. R. Hix, and S. W. Bruenn. Probing the gravitational well: No supernova explosion in spherical symmetry with general relativistic Boltzmann neutrino transport. Phys. Rev. D, 63(10):103004, May 2001.

M. Liebendoerfer. Fifty-Nine Reasons for a Supernova to not Explode. astro-ph/0405029, May 2004.

A. P. Lightman and D. M. Eardley. Black Holes in Binary Systems: Instability of Disk Accretion. ApJ, 187:L1, January 1974.

Q. Z. Liu, J. van Paradijs, and E. P. J. van den Heuvel. A catalogue of low-mass X-ray binaries. A&A, 368:1021–1054, March 2001.

M. Livio, G. I. Ogilvie, and J. E. Pringle. Extracting Energy from Black Holes: The Relative Importance of the Blandford-Znajek Mechanism. ApJ, 512:100–104, February 1999.

K. Y. Lo, Z. Shen, J. Zhao, and P. T. P. Ho. Intrinsic Size of Sagittarius A*: 72 Schwarzschild Radii. ApJ, 508:L61–L64, November 1998.

B. C. Low. Solar Activity and the Corona. Sol. Phys., 167:217–265, August 1996.

L. B. Lucy and P. M. Solomon. Mass Loss by Hot Stars. ApJ, 159:879, March 1970.

D. Lynden-Bell. Galactic Nuclei as Collapsed Old Quasars. Nature, 223:690, 1969.

D. Lynden-Bell and J. E. Pringle. The evolution of viscous discs and the origin of the nebular variables. MNRAS, 168:603–637, September 1974.

M. Lyutikov, V. I. Pariev, and R. D. Blandford. Polarization of Prompt Gamma-Ray Burst Emission: Evidence for Electromagnetically Dominated Outﬂow. ApJ, 597:998–1009, November 2003.

P. M´esz´aros. Gamma-Ray Bursts: Accumulating Afterglow Implications, Progenitor Clues, and Prospects. Science, 291:79–84, January 2001.

P. M´esz´aros.Theories of Gamma-Ray Bursts. ARA&A, 40:137–169, 2002.

230 P. M´esz´aros. γ-ray bursts: The supernova connection. Nature, 423:809–810, June 2003.

F. Macchetto, A. Marconi, D. J. Axon, A. Capetti, W. Sparks, and P. Crane. The Supermassive Black Hole of M87 and the Kinematics of Its Associated Gaseous Disk. ApJ, 489:579, November 1997.

D. MacDonald and K. S. Thorne. Black-hole electrodynamics - an absolute-space/universal-time formulation. MNRAS, 198:345–382, January 1982.

A. I. MacFadyen and S. E. Woosley. Collapsars: Gamma-Ray Bursts and Explosions in “Failed Supernovae”. ApJ, 524:262–289, October 1999.

A. I. MacFadyen, S. E. Woosley, and A. Heger. Supernovae, Jets, and Collapsars. ApJ, 550: 410–425, March 2001.

J. Magorrian, S. Tremaine, D. Richstone, R. Bender, G. Bower, A. Dressler, S. M. Faber, K. Geb- hardt, R. Green, C. Grillmair, J. Kormendy, and T. Lauer. The Demography of Massive Dark Objects in Galaxy Centers. AJ, 115:2285–2305, June 1998.

R. Mahadevan, R. Narayan, and J. Krolik. Gamma-Ray Emission from Advection-dominated Accretion Flows around Black Holes: Application to the Galactic Center. ApJ, 486:268, September 1997.

K. Makishima. Iron Lines from Galactic and Extragalactic X-ray Sources. Lecture Notes in Physics, Berlin Springer Verlag, 266:249, 1986.

K. Makishima, R. Fujimoto, Y. Ishisaki, T. Kii, M. Loewenstein, R. Mushotzky, P. Serlemitsos, T. Sonobe, M. Tashiro, and T. Yaqoob. Discovery of an obscured low luminosity active nucleus in the spiral galaxy NGC 4258. PASJ, 46:L77–L80, June 1994.

E. Maoz. Dynamical Constraints on Alternatives to Supermassive Black Holes in Galactic Nuclei. ApJ, 494:L181, February 1998.

L. Maraschi and F. Tavecchio. The Jet-Disk Connection and Blazar Uniﬁcation. ApJ, 593:667–675, August 2003.

B. Marder. Ca method for incorporating gauss’ law into electromagnetic pic codes. J. Comp. Phys., 68:48, 1987.

B. Margon and J. P. Ostriker. The Luminosity Function of Galactic X-Ray Sources - a Cutoﬀ and a “standard Candle”? ApJ, 186:91–96, November 1973.

J. M. Mart´ıand E. M¨uller.Numerical Hydrodynamics in Special Relativity. Living Reviews in Relativity, 6:7, December 2003.

231 J. Mason. Astrophysics update : topical and timely reviews on astrophysics. Astrophysics update : topical and timely reviews on astrophysics, by John Mason. Springer-Praxis books in astrophysics and astronomy Berlin: Springer, 2004.

S. Masuda, T. Kosugi, H. Hara, S. Tsuneta, and Y. Ogawara. A Loop-Top Hard X-Ray Source in a Compact Solar Flare as Evidence for Magnetic Reconnection. Nature, 371:495, October 1994.

C. Matsumoto, H. Inoue, A. C. Fabian, and K. Iwasawa. Iron Line Properties of the Seyfert 1 Galaxy MCG-6-30-15 Observed with ASCA. PASJ, 55:615–623, June 2003.

H. Matsumoto, T. G. Tsuru, K. Koyama, H. Awaki, C. R. Canizares, N. Kawai, S. Matsushita, and R. Kawabe. Discovery of a Luminous, Variable, Oﬀ-Center Source in the Nucleus of M82 with the Chandra High-Resolution Camera. ApJ, 547:L25–L28, January 2001.

G. Matt, A. C. Fabian, and C. S. Reynolds. Geometrical and chemical dependence of K-shell X-ray features. MNRAS, 289:175–184, July 1997.

J. E. McClintock, R. Narayan, and G. B. Rybicki. On the Lack of Thermal Emission from the Quiescent Black Hole XTE J1118+480: Evidence for the Event Horizon. astro-ph/0403251, March 2004.

J. E. McClintock and R. A. Remillard. Black Hole Binaries. astro-ph/0306213, June 2003.

N. McCrady, A. M. Gilbert, and J. R. Graham. Kinematic Masses of Super-Star Clusters in M82 from High-Resolution Near-Infrared Spectroscopy. ApJ, 596:240–252, October 2003.

W. A. McKinley and H. Feshbach. The Coulomb Scattering of Relativistic Electrons by Nuclei. Physical Review, 74:1759–1763, December 1948.

J. C. McKinney and C. F. Gammie. Numerical Models of Viscous Accretion Flows near Black Holes. ApJ, 573:728–737, July 2002.

J. C. McKinney and C. F. Gammie. A Measurement of the Electromagnetic Luminosity of a Kerr Black Hole. ApJ, 611:977–995, August 2004.

M. V. Medvedev. Thermodynamics of photons in relativistic e+e−γ plasmas. Phys. Rev. E, 59: 4766–, May 1999.

C. A. Meegan, G. J. Fishman, R. B. Wilson, J. M. Horack, M. N. Brock, W. S. Paciesas, G. N. Pendleton, and C. Kouveliotou. Spatial distribution of gamma-ray bursts observed by BATSE. Nature, 355:143–145, January 1992.

D. L. Meier. Ohm’s Law in the Fast Lane: General Relativistic Charge Dynamics. ApJ, 605: 340–349, April 2004.

232 D. L. Meier, S. Koide, and Y. Uchida. Magnetohydrodynamic Production of Relativistic Jets. Science, 291:84–92, January 2001.

F. Melia and H. Falcke. The Supermassive Black Hole at the Galactic Center. ARA&A, 39: 309–352, 2001.

D. B. Melrose. Instabilities in Space and Laboratory Plasmas. Instabilities in Space and Laboratory Plasmas, ISBN 0521305411, Cambridge University Press, 1986.

K. Menou, A. A. Esin, R. Narayan, M. R. Garcia, J. Lasota, and J. E. McClintock. Black Hole and Neutron Star Transients in Quiescence. ApJ, 520:276–291, July 1999.

A. Merloni. Beyond the standard accretion disc model: coupled magnetic disc-corona solutions with a physically motivated viscosity law. MNRAS, 341:1051–1056, May 2003.

A. Merloni. The anti-hierarchical growth of supermassive black holes. astro-ph/0402495, February 2004.

O. E. B. Messer, A. Mezzacappa, S. W. Bruenn, and M. W. Guidry. A Comparison of Boltz- mann and Multigroup Flux-limited Diﬀusion Neutrino Transport during the Postbounce Shock Reheating Phase in Core-Collapse Supernovae. ApJ, 507:353–360, November 1998.

P. M´esz´arosand M. J. Rees. Poynting Jets from Black Holes and Cosmological Gamma-Ray Bursts. ApJ, 482:L29, June 1997.

M. R. Metzger, S. G. Djorgovski, S. R. Kulkarni, C. C. Steidel, K. L. Adelberger, D. A. Frail, E. Costa, and F. Frontera. Spectral constraints on the redshift of the optical counterpart to the gamma-ray burst of 8 May 1997. Nature, 387:878–880, 1997.

A. Mezzacappa, M. Liebend¨orfer,O. E. Messer, W. R. Hix, F. Thielemann, and S. W. Bruenn. Simulation of the Spherically Symmetric Stellar Core Collapse, Bounce, and Postbounce Evolution of a Star of 13 Solar Masses with Boltzmann Neutrino Transport, and Its Implications for the Supernova Mechanism. Phys. Rev. Lett., 86:1935–1938, March 2001.

F. C. Michel. Rotating Magnetospheres: an Exact 3-D Solution. ApJ, 180:L133, March 1973.

D. Mihalas and J. Binney. Galactic astronomy: Structure and kinematics /2nd edition/. San Francisco, CA, W. H. Freeman and Co., 608 p., 1981.

G. E. Miller and J. M. Scalo. The initial mass function and stellar birthrate in the solar neighborhood. ApJS, 41:513–547, November 1979.

J. A. Miller, P. J. Cargill, A. G. Emslie, G. D. Holman, B. R. Dennis, T. N. LaRosa, R. M. Winglee, S. G. Benka, and S. Tsuneta. Critical issues for understanding particle acceleration in impulsive solar ﬂares. J. Geophys. Res., 102:14631–14660, July 1997.

233 J. M. Miller, A. C. Fabian, C. S. Reynolds, M. A. Nowak, J. Homan, M. J. Freyberg, M. Ehle, T. Belloni, R. Wijnands, M. van der Klis, P. A. Charles, and W. H. G. Lewin. Evidence of Black Hole Spin in GX 339-4: XMM-Newton/EPIC-pn and RXTE Spectroscopy of the Very High State. ApJ, 606:L131–L134, May 2004.

J. M. Miller, A. C. Fabian, R. Wijnands, C. S. Reynolds, M. Ehle, M. J. Freyberg, M. van der Klis, W. H. G. Lewin, C. Sanchez-Fernandez, and A. J. Castro-Tirado. Evidence of Spin and Energy Extraction in a Galactic Black Hole Candidate: The XMM-Newton/EPIC-pn Spectrum of XTE J1650-500. ApJ, 570:L69–L73, May 2002.

K. A. Miller and J. M. Stone. The Formation and Structure of a Strongly Magnetized Corona above a Weakly Magnetized Accretion Disk. ApJ, 534:398–419, May 2000.

I. F. Mirabel and L. F. Rodr´ıguez. A Superluminal Source in the Galaxy. Nature, 371:46, September 1994.

I. F. Mirabel and L. F. Rodr´ıguez. Sources of Relativistic Jets in the Galaxy. ARA&A, 37: 409–443, 1999.

I. F. Mirabel, L. F. Rodr´ıguez,B. Cordier, J. Paul, and F. Lebrun. A double-sided radio jet from the compact Galactic Centre annihilator 1E140.7 - 2942. Nature, 358:215–217, July 1992.

C. W. Misner, K. S. Thorne, and J. A. Wheeler. Gravitation. San Francisco: W.H. Freeman and Co., 1973.

M. Miyoshi, J. Moran, J. Herrnstein, L. Greenhill, N. Nakai, P. Diamond, and M. Inoue. Evidence for a Black-Hole from High Rotation Velocities in a Sub-Parsec Region of NGC4258. Nature, 373: 127, January 1995.

Y. Mizuno, S. Yamada, S. Koide, and K. Shibata. General Relativistic Magnetohydrodynamic Simulations of Collapsars. ApJ, 606:395–412, May 2004a.

Y. Mizuno, S. Yamada, S. Koide, and K. Shibata. General Relativistic Magnetohydrodynamic Simulations of Collapsars: Rotating Black Hole Cases. astro-ph/0404152, April 2004b.

M. Morris and E. Serabyn. The Galactic Center Environment. ARA&A, 34:645–702, 1996.

N. Murray, J. Chiang, S. A. Grossman, and G. M. Voit. Accretion Disk Winds from Active Galactic Nuclei. ApJ, 451:498, October 1995.

K. Nandra, K. A. Pounds, G. C. Stewart, I. M. George, K. Hayashida, F. Makino, and T. Ohashi. Compton reﬂection and the variable X-ray spectrum of NGC 5548. MNRAS, 248:760–772, Febru- ary 1991.

R. Narayan. Black holes: Sparks of interest. Nature, 425:908–909, October 2003a.

234 R. Narayan. George Darwin Lecture: Evidence for the black hole event horizon. Astronomy and Geophysics, 44:22–6, December 2003b.

R. Narayan, D. Barret, and J. E. McClintock. Advection-dominated Accretion Model of the Black Hole V404 Cygni in Quiescence. ApJ, 482:448, June 1997.

R. Narayan and J. S. Heyl. On the Lack of Type I X-Ray Bursts in Black Hole X-Ray Binaries: Evidence for the Event Horizon? ApJ, 574:L139–L142, August 2002.

R. Narayan and J. S. Heyl. Thermonuclear Stability of Material Accreting onto a Neutron Star. ApJ, 599:419–449, December 2003.

R. Narayan, R. Mahadevan, J. E. Grindlay, R. G. Popham, and C. Gammie. Advection-dominated accretion model of Sagittarius A*: evidence for a black hole at the Galactic center. ApJ, 492: 554–568, January 1998.

R. Narayan, B. Paczy´nski,and T. Piran. Gamma-ray bursts as the death throes of massive binary stars. ApJ, 395:L83–L86, August 1992.

R. Narayan, T. Piran, and P. Kumar. Accretion Models of Gamma-Ray Bursts. ApJ, 557:949–957, August 2001.

R. Narayan, T. Piran, and A. Shemi. Neutron star and black hole binaries in the Galaxy. ApJ, 379:L17–L20, September 1991.

R. Narayan and R. Popham. Hard X-rays from accretion disk boundary layers. Nature, 362: 820–822, April 1993.

R. Narayan and I. Yi. Advection-dominated accretion: A self-similar solution. ApJ, 428:L13–L16, June 1994.

R. Narayan and I. Yi. Advection-dominated Accretion: Underfed Black Holes and Neutron Stars. ApJ, 452:710, October 1995.

R. Narayan, I. Yi, and R. Mahadevan. Explaining the Spectrum of SAGITTARIUS-A* with a Model of an Accreting Black-Hole. Nature, 374:623, April 1995.

M.L. Norman and K.-H.A. Winkler. Why ultrarelativistic hydrodynamics is diﬃcult. In M.L. Norman and K.-H.A. Winkler, editors, Astrophysical Radiation Hydrodynamics: Proceedings of the NATO Advanced Workshop held in Garching, August 2-13, 1982, volume 188 of NATO ASI Series C, pages 449–476, Dordrecht, Netherlands, 1986. Reidel.

I. D. Novikov and K. S. Thorne. in Black Holes. (Eds C De Witt, B S De Witt) (London: Gordon and Breach, 1973), 1973.

235 M. Oguri, N. Inada, C. R. Keeton, B. Pindor, J. F. Hennawi, M. D. Gregg, R. H. Becker, K. Chiu, W. Zheng, S. Ichikawa, Y. Suto, E. L. Turner, J. Annis, N. A. Bahcall, J. Brinkmann, F. J. Castander, D. J. Eisenstein, J. A. Frieman, T. Goto, J. E. Gunn, D. E. Johnston, S. M. Kent, R. C. Nichol, G. T. Richards, H. Rix, D. P. Schneider, E. S. Sheldon, and A. S. Szalay. Observations and Theoretical Implications of the Large-Separation Lensed Quasar SDSS J1004+4112. ApJ, 605: 78–97, April 2004.

A. Orszag and C. M. Tang. Small-scale structure of two-dimensional magnetohydrodynamic turbulence. Journal of Fluid Mechanics, 90:129, 1979.

W. S. Paciesas, C. A. Meegan, G. N. Pendleton, M. S. Briggs, C. Kouveliotou, T. M. Koshut, J. P. Lestrade, M. L. McCollough, J. J. Brainerd, J. Hakkila, W. Henze, R. D. Preece, V. Connaughton, R. M. Kippen, R. S. Mallozzi, G. J. Fishman, G. A. Richardson, and M. Sahi. The Fourth BATSE Gamma-Ray Burst Catalog (Revised). ApJS, 122:465–495, June 1999.

B. Paczy´nski.Gamma-ray bursters at cosmological distances. ApJ, 308:L43–L46, September 1986.

B. Paczy´nski.Are Gamma-Ray Bursts in Star-Forming Regions? ApJ, 494:L45, February 1998.

B. Paczy´nskiand P. J. Wiita. Thick accretion disks and supercritical luminosities. A&A, 88: 23–31, August 1980.

D. N. Page and K. S. Thorne. Disk-Accretion onto a Black Hole. Time-Averaged Structure of Accretion Disk. ApJ, 191:499–506, July 1974.

A. Panaitescu and P. Kumar. Jet Energy and Other Parameters for the Afterglows of GRB 980703, GRB 990123, GRB 990510, and GRB 991216 Determined from Modeling of Multifrequency Data. ApJ, 554:667–677, June 2001.

J. C. B. Papaloizou and J. E. Pringle. The time-dependence of non-planar accretion discs. MNRAS, 202:1181–1194, March 1983.

J. C. B. Papaloizou and J. E. Pringle. The dynamical stability of diﬀerentially rotating discs with constant speciﬁc angular momentum. MNRAS, 208:721–750, June 1984.

V. I. Pariev and B. C. Bromley. Line Emission from an Accretion Disk Around a Black Hole: Eﬀects of Disk Structure. ApJ, 508:590–600, December 1998.

E. N. Parker. The Solar-Flare Phenomenon and the Theory of Reconnection and Annihiliation of Magnetic Fields. ApJS, 8:177, July 1963.

E. N. Parker. Cosmical magnetic ﬁelds: Their origin and their activity. Oxford, Clarendon Press; New York, Oxford University Press, 858 pp., 1979.

E. N. Parker. Nanoﬂares and the solar X-ray corona. ApJ, 330:474–479, July 1988.

236 R. Penrose. Gravitational collapse: The role of general relativity. Riv. del Nuovo Cim., 1:252–276, 1969.

B. M. Peterson. Reverberation mapping of active galactic nuclei. PASP, 105:247–268, March 1993.

B. M. Peterson, L. Ferrarese, K. M. Gilbert, S. Kaspi, M. A. Malkan, D. Maoz, D. Merritt, H. Netzer, C. A. Onken, R. W. Pogge, M. Vestergaard, and A. Wandel. Central Masses and Broad- Line Region Sizes of Active Galactic Nuclei. II. A Homogeneous Analysis of a Large Reverberation- Mapping Database. astro-ph/0407299, July 2004.

J. M. Peterson, A. Bratenahl, and J. P. Stoering. Neutron Total Cross Sections in the 17- to 29-Mev Region. Phys. Rev., 120:521–526, 1960.

E. S. Phinney. A Theory of radio sources. Ph.D. Thesis (Institute of Astronomy and Churchill College: University of Cambridge), 1983.

T. Piran. The role of viscosity and cooling mechanisms in the stability of accretion disks. ApJ, 221:652–660, April 1978.

T. Piran. Gamma-ray bursts and the ﬁreball model. Phys. Rep., 314:575, 1999.

T. Piran. The Physics of Gamma-Ray Bursts. astro-ph/0405503, May 2004.

R. Popham and C. F. Gammie. Advection-dominated Accretion Flows in the Kerr Metric. II. Steady State Global Solutions. ApJ, 504:419, September 1998.

R. Popham, S. E. Woosley, and C. Fryer. Hyperaccreting Black Holes and Gamma-Ray Bursts. ApJ, 518:356–374, June 1999.

S. F. Portegies Zwart, J. Makino, S. L. W. McMillan, and P. Hut. Star cluster ecology. III. Runaway collisions in young compact star clusters. A&A, 348:117–126, August 1999.

S. F. Portegies Zwart and S. L. W. McMillan. The Runaway Growth of Intermediate-Mass Black Holes in Dense Star Clusters. ApJ, 576:899–907, September 2002.

K. A. Postnov and A. M. Cherepashchuk. Masses of Stellar Black Holes and Testing Theories of Gravitation. astro-ph/0401466, January 2004.

L. A. Pozdniakov, I. M. Sobol, and R. A. Siuniaev. Comptonization and the shaping of X-ray source spectra - Monte Carlo calculations. Astrophysics and Space Physics Reviews, 2:189–331, 1983.

W. H. Press and S. A. Teukolsky. Floating Orbits, Superradiant Scattering and the Black-Hole Bomb. Nature, 238:211, 1972.

237 E. R. Priest. Solar ﬂare magnetohydrodynamics. Solar Flare Magnetohydrodynamics, 1981.

E. R. Priest and T. G. Forbes. The magnetic nature of solar ﬂares. A&A Rev., 10:313–377, 2002.

J. E. Pringle. Thermal instabilities in accretion discs. MNRAS, 177:65–71, October 1976.

J. E. Pringle. Accretion discs in astrophysics. ARA&A, 19:137–162, 1981.

J. E. Pringle and M. J. Rees. Accretion Disc Models for Compact X-Ray Sources. A&A, 21:1, October 1972.

D. Proga, A. I. MacFadyen, P. J. Armitage, and M. C. Begelman. Axisymmetric Magnetohydro- dynamic Simulations of the Collapsar Model for Gamma-Ray Bursts. ApJ, 599:L5–L8, December 2003.

D. Proga, J. M. Stone, and J. E. Drew. Radiation-driven winds from luminous accretion discs. MNRAS, 295:595, April 1998.

M. Punch, C. W. Akerlof, M. F. Cawley, M. Chantell, D. J. Fegan, S. Fennell, J. A. Gaidos, J. Hagan, A. M. Hillas, Y. Jiang, A. D. Kerrick, R. C. Lamb, M. A. Lawrence, D. A. Lewis, D. I. Meyer, G. Mohanty, K. S. O’Flaherty, P. T. Reynolds, A. C. Rovero, M. S. Schubnell, G. Sembroski, T. C. Weekes, and C. Wilson. Detection of TeV photons from the active galaxy Markarian 421. Nature, 358:477, August 1992.

B. Punsly. Black hole gravitohydromagnetics. Black hole gravitohydromagnetics, Berlin: Springer, xii, 400 p., Astronomy and astrophysics library, ISBN 3540414665, 2001.

B. Punsly. Force-free Waves and Black Hole Magnetospheric Causality. ApJ, 583:842–852, Febru- ary 2003.

E. Quataert and A. Gruzinov. Convection-dominated Accretion Flows. ApJ, 539:809–814, August 2000.

E. Quataert, R. Narayan, and M. J. Reid. What Is the Accretion Rate in Sagittarius A*? ApJ, 517:L101–L104, June 1999.

M. Rampp and H.-T. Janka. Spherically Symmetric Simulation with Boltzmann Neutrino Trans- port of Core Collapse and Postbounce Evolution of a 15 M¯ Star. ApJ, 539:L33–L36, August 2000.

S. Rappaport, P. Podsiadlowski, and E. Pfahl. Stellar Mass Black Hole Binaries as ULXs. astro- ph/0408032, August 2004.

M. J. Rees. Black Hole Models for Active Galactic Nuclei. ARA&A, 22:471–506, 1984.

M. J. Rees. Astrophysical Evidence for Black Holes. In Black Holes and Relativistic Stars, page 79, 1998.

238 M. J. Rees. Why AGN studies need higher resolution. In astro-ph/0101266, page 2, January 2001.

M. J. Rees and P. M´esz´aros.Fe Kα Emission from a Decaying Magnetar Model of Gamma-Ray Bursts. ApJ, 545:L73–L75, December 2000.

M. J. Rees, E. S. Phinney, M. C. Begelman, and R. D. Blandford. Ion-supported tori and the origin of radio jets. Nature, 295:17–21, January 1982.

M. J. Reid. The distance to the center of the Galaxy. ARA&A, 31:345–372, 1993.

C. S. Reynolds. On the Lack of X-Ray Iron Line Reverberation in MCG -6-30-15: Implications for the Black Hole Mass and Accretion Disk Structure. ApJ, 533:811–820, April 2000.

C. S. Reynolds and P. J. Armitage. A Variable Eﬃciency for Thin-Disk Black Hole Accretion. ApJ, 561:L81–L84, November 2001.

C. S. Reynolds and M. C. Begelman. Iron Fluorescence from within the Innermost Stable Orbit of Black Hole Accretion Disks. ApJ, 488:109, October 1997.

C. S. Reynolds, T. di Matteo, A. C. Fabian, U. Hwang, and C. R. Canizares. The ‘quiescent’ black hole in M87. MNRAS, 283:L111–L116, December 1996.

C. S. Reynolds, A. C. Fabian, K. Nandra, H. Inoue, H. Kunieda, and K. Iwasawa. ASCA PV observations of the Seyfert 1 galaxy MCG-6-30-15: rapid variability of the warm absorber. MNRAS, 277:901–912, December 1995.

C. S. Reynolds, M. J. Ward, A. C. Fabian, and A. Celotti. A multiwavelength study of the Seyfert 1 galaxy MCG-6-30-15. MNRAS, 291:403, November 1997.

C. E. Rhoades and R. Ruﬃni. Maximum Mass of a Neutron Star. Phys. Rev. Lett., 32:324–327, February 1974.

J. E. Rhoads. The Dynamics and Light Curves of Beamed Gamma-Ray Burst Afterglows. ApJ, 525:737–749, November 1999.

D. Richstone, E. A. Ajhar, R. Bender, G. Bower, A. Dressler, S. M. Faber, A. V. Filippenko, K. Gebhardt, R. Green, L. C. Ho, J. Kormendy, T. R. Lauer, J. Magorrian, and S. Tremaine. Supermassive black holes and the evolution of galaxies. Nature, 395:A14, 1998.

P. L. Roe. Remote boundary conditions for unsteady multidimensional aerodynamic computations. Computers and Fluids, 17(1):221–231, 1989.

G. E. Romero. Microquasars and gamma-ray sources. astro-ph/0407461, July 2004.

L. Rudnick and B. K. Edgar. Alternating-side ejection in extragalactic radio sources. ApJ, 279: 74–85, April 1984.

239 G. B. Rybicki and A. P. Lightman. Radiative processes in astrophysics. New York, Wiley- Interscience, 393 pp., 1979.

D. Ryu and T. W. Jones. Numerical magetohydrodynamics in astrophysics: Algorithm and tests for one-dimensional ﬂow. ApJ, 442:228–258, March 1995.

K. C. Sahu, M. Livio, L. Petro, F. D. Macchetto, J. van Paradijs, C. Kouveliotou, G. J. Fishman, C. A. Meegan, P. J. Groot, and T. Galama. The optical counterpart to gamma-ray burst GRB 970228 observed using the Hubble Space Telescope. Nature, 387:476–478, 1997.

E. E. Salpeter. The Luminosity Function and Stellar Evolution. ApJ, 121:161, January 1955.

E. E. Salpeter. Accretion of Interstellar Matter by Massive Objects. ApJ, 140:796–800, August 1964.

W. L. W. Sargent, P. J. Young, C. R. Lynds, A. Boksenberg, K. Shortridge, and F. D. A. Hartwick. Dynamical evidence for a central mass concentration in the galaxy M87. ApJ, 221:731–744, May 1978.

R. Sari. Linear Polarization and Proper Motion in the Afterglow of Beamed Gamma-Ray Bursts. ApJ, 524:L43–L46, October 1999.

R. Sari, T. Piran, and J. P. Halpern. Jets in Gamma-Ray Bursts. ApJ, 519:L17–L20, July 1999.

P. V. Savrukhin. Eﬀect of the magnetohydrodynamic modes on generation of the nonthermal electrons in high density tokamak plasma. Physics of Plasmas, 9(8):3421–3428, 2002.

S. Sazonov, A. Lutovinov, and R. Sunyaev. An apparently normal gamma-ray burst with an unusually low luminosity. astro-ph/0408095, August 2004.

J. M. Scalo. The stellar initial mass function. Fundamentals of Cosmic Physics, 11:1–278, May 1986.

R. Sch¨odel,T. Ott, R. Genzel, R. Hofmann, M. Lehnert, A. Eckart, N. Mouawad, T. Alexander, M. J. Reid, R. Lenzen, M. Hartung, F. Lacombe, D. Rouan, E. Gendron, G. Rousset, A.-M. Lagrange, W. Brandner, N. Ageorges, C. Lidman, A. F. M. Moorwood, J. Spyromilio, N. Hubin, and K. M. Menten. A star in a 15.2-year orbit around the supermassive black hole at the centre of the Milky Way. Nature, 419:694–696, October 2002.

R. Schneider, A. Ferrara, P. Natarajan, and K. Omukai. First Stars, Very Massive Black Holes, and Metals. ApJ, 571:30–39, May 2002.

K. Schwarzschild. ber das Gravitationsfeld eines Massenpunktes nach der Einsteinschen Theorie. Sitzungsberichte der Kniglich Preussischen Akademie der Wissenschaften, 1:189–196, 1916.

240 N. I. Shakura. Disk Model of Gas Accretion on a Relativistic Star in a Close Binary System. AZh, 49:921, October 1972.

N. I. Shakura and R. A. Sunyaev. Black holes in binary systems. Observational appearance. A&A, 24:337–355, 1973.

S. L. Shapiro, A. P. Lightman, and D. M. Eardley. A two-temperature accretion disk model for Cygnus X-1 - Structure and spectrum. ApJ, 204:187–199, February 1976.

S. L. Shapiro and M. Shibata. Collapse of a Rotating Supermassive Star to a Supermassive Black Hole: Analytic Determination of the Black Hole Mass and Spin. ApJ, 577:904–908, October 2002.

S. L. Shapiro and S. A. Teukolsky. Black holes, white dwarfs, and neutron stars: The physics of compact objects. Research supported by the National Science Foundation. New York, Wiley- Interscience, 663 p., 1983.

Z. She and E. Leveque. Universal scaling laws in fully developed turbulence. Phys. Rev. Lett., 72: 336–339, January 1994.

M. Shibata and S. L. Shapiro. Collapse of a Rotating Supermassive Star to a Supermassive Black Hole: Fully Relativistic Simulations. ApJ, 572:L39–L43, June 2002.

T. Shimura and F. Takahara. On the spectral hardening factor of the X-ray emission from accretion disks in black hole candidates. ApJ, 445:780–788, June 1995.

L. S. Shklovski. A consequence of the asymmetry of jets in quasars and active nuclei of galaxies. In IAU Symp. 97: Extragalactic Radio Sources, pages 475–481, 1982.

F. H. Shu. Physics of Astrophysics, Vol. II. Physics of Astrophysics, Vol. II, by Frank H. Shu. Pub- lished by University Science Books, ISBN 0-935702-65-2, 476pp, 1992.

H. Shukla, M. S. Yun, and N. Z. Scoville. Dense Ionized and Neutral Gas Surrounding Sgr A*. astro-ph/0408136, August 2004.

M. Sikora, M. C. Begelman, P. Coppi, and D. Proga. Gamma-ray bursts from alternating-current jets. astro-ph/0309504, September 2003.

A. Socrates and O. Blaes. Dynamical Instabilities in Radiation Pressure Dominated Accretion Disks. APS Meeting Abstracts, page 17091, April 2002.

A. M. Soderberg, S. R. Kulkarni, E. Berger, D. W. Fox, M. Sako, D. A. Frail, A. Gal-Yam, D. S. Moon, S. B. Cenko, S. A. Yost, M. M. Phillips, S. E. Persson, W. L. Freedman, P. Wyatt, R. Jayawardhana, and D. Paulson. The sub-energetic GRB 031203 as a cosmic analogue to GRB 980425. astro-ph/0408096, August 2004.

A. Soltan. Masses of quasars. MNRAS, 200:115–122, July 1982.

241 R. Soria, K. Wu, M. J. Page, and I. Sakelliou. XMM-Newton optical monitor observations of LMC X-3. A&A, 365:L273–L276, January 2001.

R. Speith, H. Riﬀert, and H. Ruder. The photon transfer function for accretion disks around a Kerr black hole. Computer Physics Communications, 88:109–120, August 1995.

D. N. Spergel, L. Verde, H. V. Peiris, E. Komatsu, M. R. Nolta, C. L. Bennett, M. Halpern, G. Hinshaw, N. Jarosik, A. Kogut, M. Limon, S. S. Meyer, L. Page, G. S. Tucker, J. L. Weiland, E. Wollack, and E. L. Wright. First-Year Wilkinson Microwave Anisotropy Probe (WMAP) Observations: Determination of Cosmological Parameters. ApJS, 148:175–194, September 2003.

L. Spitzer. Physics of Fully Ionized Gases. Physics of Fully Ionized Gases, New York: Interscience Publishers, 1956.

H. C. Spruit, F. Daigne, and G. Drenkhahn. Large scale magnetic ﬁelds and their dissipation in GRB ﬁreballs. A&A, 369:694–705, April 2001.

K. Z. Stanek, T. Matheson, P. M. Garnavich, P. Martini, P. Berlind, N. Caldwell, P. Challis, W. R. Brown, R. Schild, K. Krisciunas, M. L. Calkins, J. C. Lee, N. Hathi, R. A. Jansen, R. Windhorst, L. Echevarria, D. J. Eisenstein, B. Pindor, E. W. Olszewski, P. Harding, S. T. Holland, and D. Bersier. Spectroscopic Discovery of the Supernova 2003dh Associated with GRB 030329. ApJ, 591:L17–L20, July 2003.

N. Stergioulas. Rotating Stars in Relativity. Living Reviews in Relativity, 6:3, June 2003.

B. E. Stern, J. Poutanen, R. Svensson, M. Sikora, and M. C. Begelman. On the Geometry of the X-Ray–Emitting Region in Seyfert Galaxies. ApJ, 449:L13, August 1995.

T. H. Stix. The Theory of Plasma Waves. The Theory of Plasma Waves, New York: McGraw-Hill, 1962.

J. M. Stone, J. F. Hawley, C. F. Gammie, and S. A. Balbus. Three-dimensional Magnetohydro- dynamical Simulations of Vertically Stratiﬁed Accretion Disks. ApJ, 463:656, June 1996.

J. M. Stone and M. L. Norman. ZEUS-2D: A radiation magnetohydrodynamics code for astrophysical ﬂows in two space dimensions. I - The hydrodynamic algorithms and tests. ApJS, 80: 753–790, June 1992.

J. M. Stone and J. E. Pringle. Magnetohydrodynamical non-radiative accretion ﬂows in two dimensions. MNRAS, 322:461–472, April 2001.

J. M. Stone, J. E. Pringle, and M. C. Begelman. Hydrodynamical non-radiative accretion ﬂows in two dimensions. MNRAS, 310:1002–1016, December 1999.

T. E. Strohmayer. Discovery of a 450 HZ Quasi-periodic Oscillation from the Microquasar GRO J1655-40 with the Rossi X-Ray Timing Explorer. ApJ, 552:L49–L53, May 2001.

242 H. Sudou, S. Iguchi, Y. Murata, and Y. Taniguchi. Orbital Motion in the Radio Galaxy 3C 66B: Evidence for a Supermassive Black Hole Binary. Science, 300:1263–1265, May 2003.

R. A. Sunyaev and J. Truemper. Hard X-ray spectrum of CYG X-1. Nature, 279:506–508, June 1979.

E. M. D. Symbalisty. Magnetorotational iron core collapse. ApJ, 285:729–746, October 1984.

S. I. Syrovatskii. Pinch sheets and reconnection in astrophysics. ARA&A, 19:163–229, 1981.

M. Takahashi, S. Nitta, Y. Tatematsu, and A. Tomimatsu. Magnetohydrodynamic ﬂows in Kerr geometry - Energy extraction from black holes. ApJ, 363:206–217, November 1990.

Y. Tanaka, K. Nandra, A. C. Fabian, H. Inoue, C. Otani, T. Dotani, K. Hayashida, K. Iwasawa, T. Kii, H. Kunieda, F. Makino, and M. Matsuoka. Gravitationally Redshifted Emission Implying an Accretion Disk and Massive Black-Hole in the Active Galaxy MCG:-6-30-15. Nature, 375:659, June 1995.

Y. Tanaka and N. Shibazaki. X-ray Novae. ARA&A, 34:607–644, 1996.

V. I. Tatarski. Wave propagation in a turbulent medium. Wave propagation in a turbulent medium, McGraw-Hill, 1961.

J. B. Taylor. Relaxation and magnetic reconnection in plasmas. Reviews of Modern Physics, 58: 741–763, July 1986.

H. Tennekes and J. L. Lumley. First Course in Turbulence. First Course in Turbulence, Cambridge: MIT Press, 1972.

S. A. Teukolsky and W. H. Press. Perturbations of a rotating black hole. III - Interaction of the hole with gravitational and electromagnetic radiation. ApJ, 193:443–461, October 1974.

C. Thompson. A Model of Gamma-Ray Bursts. MNRAS, 270:480, October 1994.

T. A. Thompson, P. Chang, and E. Quataert. Magnetar Spin-Down, Hyperenergetic Supernovae, and Gamma-Ray Bursts. ApJ, 611:380–393, August 2004a.

T. A. Thompson, P. Chang, and E. Quataert. Magnetar Spindown, Hyper-Energetic Supernovae, and Gamma Ray Bursts. astro-ph/0401555, January 2004b.

K. S. Thorne. Disk-Accretion onto a Black Hole. II. Evolution of the Hole. ApJ, 191:507–520, July 1974.

K. S. Thorne. Black holes and time warps: Einstein’s outrageous legacy. Commonwealth Fund Book Program, New York, NY: W.W. Norton and London: Picador, 1994.

243 K. S. Thorne and D. MacDonald. Electrodynamics in Curved Spacetime - 3+1 Formulation. MNRAS, 198:339, January 1982.

K. S. Thorne and R. H. Price. Cygnus X-1 - an interpretation of the spectrum and its variability. ApJ, 195:L101–L105, February 1975.

K. S. Thorne, R. H. Price, and D. A. MacDonald, editors. Black holes: The membrane paradigm, 1986.

D. A. Tidman and N. A. Krall. Shock waves in collisionless plasmas. Wiley Series in Plasma Physics, New York: Wiley-Interscience, 1971.

J. E. Tohline. The Origin of Binary Stars. ARA&A, 40:349–385, 2002.

Gabor Toth. The ∇ · b = 0 constraint in shock-capturing magnetohydrodynamics codes. J. Comp. Phys., 161(2):605–652, Jul 2000.

Gabor Toth and Dusan Odstrcil. Comparison of some ﬂux corrected transport and total variation diminishing numerical schemes for hydrodynamic and magnetohydrodynamic problems. J. Comp. Phys., 128(1):82–100, 1996.

D. Tournear, E. Raﬀauf, E. D. Bloom, W. Focke, B. Giebels, G. Godfrey, P. M. S. Parkinson, K. T. Reilly, K. S. Wood, P. S. Ray, M. T. Wolﬀ, R. M. Bandyopadhyay, M. N. Lovellette, and J. D. Scargle. X-Ray Bursts in Neutron Star and Black Hole Binaries from Unconventional Stellar Aspect Experiment and Rossi X-Ray Timing Explorer Data: Detections and Upper Limits. ApJ, 595:1058–1065, October 2003.

S. Tremaine, K. Gebhardt, R. Bender, G. Bower, A. Dressler, S. M. Faber, A. V. Filippenko, R. Green, C. Grillmair, L. C. Ho, J. Kormendy, T. R. Lauer, J. Magorrian, J. Pinkney, and D. Richstone. The Slope of the Black Hole Mass versus Velocity Dispersion Correlation. ApJ, 574:740–753, August 2002.

R. A. Treumann and W. Baumjohann. Advanced space plasma physics. London: Imperial College Press, 1997.

A. Treves, T. Belloni, L. Chiappetti, L. Maraschi, L. Stella, E. G. Tanzi, and M. van der Klis. X-ray spectrum and variability of the black hole candidate LMC X-3. ApJ, 325:119–127, February 1988.

N. J. Turner. On the Vertical Structure of Radiation-Dominated Accretion Disks. astro- ph/0402539, February 2004.

N. J. Turner and J. M. Stone. A Module for Radiation Hydrodynamic Calculations with ZEUS-2D Using Flux-limited Diﬀusion. ApJS, 135:95–107, July 2001.

244 N. J. Turner, J. M. Stone, and T. Sano. Local Axisymmetric Simulations of Magnetorotational Instability in Radiation-dominated Accretion Disks. ApJ, 566:148–163, February 2002.

T. Uchida. Linear perturbations in force-free black hole magnetospheres -II. Wave propagation. MNRAS, 291:125–144, October 1997.

M. Uemura, T. Kato, R. Ishioka, H. Yamaoka, B. Monard, D. Nogami, H. Maehara, A. Sugie, and S. Takahashi. Structure in the early afterglow light curve of the γ-ray burst of 29 March 2003. Nature, 423:843–844, June 2003.

A. V. Usmanov, M. L. Goldstein, B. P. Besser, and J. M. Fritzer. A global MHD solar wind model with WKB Alfv´enwaves: Comparison with Ulysses data. J. Geophys. Res., pages 12675–12696, June 2000.

V. V. Usov. Millisecond pulsars with extremely strong magnetic ﬁelds as a cosmological source of gamma-ray bursts. Nature, 357:472–474, June 1992.

R. P. van der Marel. The Black Hole Mass Distribution in Early-Type Galaxies: Cusps in Hubble Space Telescope Photometry Interpreted through Adiabatic Black Hole Growth. AJ, 117:744–763, February 1999.

R. P. Van Der Marel. Intermediate-Mass Black Holes in the Universe: A Review of Formation Theories and Observational Constraints. astro-ph/0302101, February 2004.

M. H. van Putten, A. Levinson, H. K. Lee, T. Regimbau, M. Punturo, and G. M. Harry. Gravita- tional radiation from gamma-ray burst-supernovae as observational opportunities for LIGO and VIRGO. Phys. Rev. D, 69:044007, February 2004.

M. H. P. M. van Putten and E. C. Ostriker. Hyper- and Suspended-Accretion States of Rotating Black Holes and the Durations of Gamma-Ray Bursts. ApJ, 552:L31–L34, May 2001.

R. Vanderspek, G. Crew, J. Doty, J. Villasenor, G. Monnelly, N. Butler, T. Cline, J. G. Jerni- gan, A. Levine, F. Martel, E. Morgan, G. Prigozhin, G. Azzibrouck, J. Braga, R. Manchanda, G. Pizzichini, G. Ricker, J.-L. Atteia, N. Kawai, D. Lamb, S. Woosley, T. Donaghy, M. Suzuki, Y. Shirasaki, C. Graziani, M. Matsuoka, T. Tamagawa, K. Torii, T. Sakamoto, A. Yoshida, E. Fen- imore, M. Galassi, T. Tavenner, Y. Nakagawa, D. Takahashi, R. Satoh, Y. Urata, M. Boer, J.-F. Olive, J.-P. Dezalay, C. Barraud, and K. Hurley. GRB030329 (=H2652): a long, extremely bright GRB localized by the HETE WXM and SXC. GRB Circular Network, 1997:1, 2003.

Maurice H. P. M. vanPutten. A numerical implementation of mhd in divergence form. J. Comp. Phys., 105(2):339–353, 1993.

S. Vaughan and A. C. Fabian. A long hard look at MCG-6-30-15 with XMM-Newton- II. Detailed EPIC analysis and modelling. MNRAS, 348:1415–1438, March 2004.

245 S. Vaughan, A. C. Fabian, and K. Nandra. X-ray continuum variability of MCG-6-30-15. MNRAS, 339:1237–1255, March 2003.

E. P. Velikhov. Stability of an ideally conducting liquid ﬂowing between rotating cylinders in a magnetic ﬁeld. J. Exper. Theor. Phys., 36:1398–1404, 1959.

P. A. J. Vitello and I. Shlosman. Line-driven winds from accretion disks. I - Eﬀects of the ionization structure. ApJ, 327:680–692, April 1988.

P. M. Vreeswijk, T. J. Galama, E. Rol, B. Stappers, E. Palazzi, E. Pian, N. Masetti, F. Frontera, J. van Paradijs, C. Kouveliotou, and H. Boehnhardt. VLT spectrum of GRB990510. GRB Circular Network, 324:1, 1999.

L. Wang, D. Baade, P. H¨oﬂich, and J. C. Wheeler. Spectropolarimetry of the Type Ic Supernova SN 2002ap in M74: More Evidence for Asymmetric Core Collapse. ApJ, 592:457–466, July 2003.

L. Wang, D. A. Howell, P. H¨oﬂich, and J. C. Wheeler. Bipolar Supernova Explosions. ApJ, 550: 1030–1035, April 2001.

L. Wang and J. C. Wheeler. Polarization of SN 1987A Revisited. ApJ, 462:L27, May 1996.

L. Wang, J. C. Wheeler, P. Höflich, A. Khokhlov, D. Baade, D. Branch, P. Challis, A. V. Filip- penko, C. Fransson, P. Garnavich, R. P. Kirshner, P. Lundqvist, R. McCray, N. Panagia, C. S. J. Pun, M. M. Phillips, G. Sonneborn, and N. B. Suntzeff. The Axisymmetric Ejecta of Supernova 1987A. ApJ, 579:671–677, November 2002.

B. Warner. Cataclysmic variable stars. Cambridge Astrophysics Series, Cambridge, New York: Cambridge University Press, 1995.

W. D. Watson and B. K. Wallin. Evidence from masers for a rapidly rotating disk at the nucleus of NGC 4258. ApJ, 432:L35–L38, September 1994.

E. J. Weber and L. J. Davis. The Angular Momentum of the Solar Wind. ApJ, 148:217, April 1967.

B. Webster and P. Murdin. Nature, 235:37, 1972.

A. E. Wehrle, B. G. Piner, S. C. Unwin, A. C. Zook, W. Xu, A. P. Marscher, H. Ter¨asranta, and E. Valtaoja. Kinematics of the Parsec-Scale Relativistic Jet in Quasar 3C 279: 1991-1997. ApJS, 133:297–320, April 2001.

A. A. West, K. I. Kellermann, R. C. Vermeulen, J. A. Zensus, and M. H. Cohen. Kinematics of Quasars and AGN. Bulletin of the American Astronomical Society, 30:1410, December 1998.

J. C. Wheeler, D. L. Meier, and J. R. Wilson. Asymmetric Supernovae from Magnetocentrifugal Jets. ApJ, 568:807–819, April 2002.

246 J. C. Wheeler, I. Yi, P. H¨oﬂich, and L. Wang. Asymmetric Supernovae, Pulsars, Magnetars, and Gamma-Ray Bursts. ApJ, 537:810–823, July 2000.

N. E. White and F. E. Marshall. The unusually soft X-ray spectrum of LMC X-3. ApJ, 281: 354–359, June 1984.

B. C. Whitmore, W. B. Sparks, R. A. Lucas, F. D. Macchetto, and J. A. Biretta. Hubble Space Telescope Observations of Globular Clusters in M87 and an Estimate of H 0. ApJ, 454:L73, December 1995.

B. J. Wilkes, G. D. Schmidt, P. S. Smith, S. Mathur, and K. K. McLeod. Optical Detection of the Hidden Nuclear Engine in NGC 4258. ApJ, 455:L13, December 1995.

J. Wilms, M. A. Nowak, K. Pottschmidt, W. A. Heindl, J. B. Dove, and M. C. Begelman. Discovery of recurring soft-to-hard state transitions in LMC X-3. MNRAS, 320:327–340, January 2001a.

J. Wilms, C. S. Reynolds, M. C. Begelman, J. Reeves, S. Molendi, R. Staubert, and E. Kendziorra. XMM-EPIC observation of MCG-6-30-15: direct evidence for the extraction of energy from a spinning black hole? MNRAS, 328:L27–L31, December 2001b.

J. R. Wilson. In in Proc. of the First Marcel Grossman Meeting on General Relativity, ed. R. Ruﬃni (Amsterdam: North-Holland), page 393, 1977.

S. S. M. Wong. Introductory nuclear physics. Prentice-Hall, 486 pp., 1990.

J. Woo and C. M. Urry. Active Galactic Nucleus Black Hole Masses and Bolometric Luminosities. ApJ, 579:530–544, November 2002.

S. E. Woosley. Gamma-ray bursts from stellar mass accretion disks around black holes. ApJ, 405: 273–277, March 1993.

S. E. Woosley and E. Baron. The collapse of white dwarfs to neutron stars. ApJ, 391:228–235, May 1992.

S. E. Woosley, A. Heger, A. Cumming, R. D. Hoﬀman, J. Pruet, T. Rauscher, J. L. Fisker, H. Schatz, B. A. Brown, and M. Wiescher. Models for Type I X-Ray Bursts with Improved Nuclear Physics. ApJS, 151:75–102, March 2004.

S. E. Woosley, N. Langer, and T. A. Weaver. The evolution of massive stars including mass loss - Presupernova models and explosion. ApJ, 411:823–839, July 1993.

S. E. Woosley and T. A. Weaver. The physics of supernova explosions. ARA&A, 24:205–253, 1986.

D. G. Yakovlev and D. A. Shalybkov. Electrical Conductivity and Resistivity in Magnetized Cores of Neutron Stars. Soviet Astronomy Letters, 16:86, April 1990.

247 S. Yamada, H. Janka, and H. Suzuki. Neutrino transport in type II supernovae: Boltzmann solver vs. Monte Carlo method. A&A, 344:533–550, April 1999.

R. Yamazaki, K. Ioka, and T. Nakamura. X-Ray Flashes from Oﬀ-Axis Gamma-Ray Bursts. ApJ, 571:L31–L35, May 2002.

T. Yaqoob, B. McKernan, A. Ptak, K. Nandra, and P. J. Serlemitsos. A Simple New Method for Analyzing Gapped Time Series: Search for a High-frequency Cutoﬀ in the X-Ray Power Spectrum of the Seyfert Galaxy MCG -6-30-15. ApJ, 490:L25, November 1997.

M. Yokosawa. Energy and angular momentum transport in magnetohydrodynamical accretion onto a rotating black hole. PASJ, 45:207–218, April 1993.

P. J. Young, J. A. Westphal, J. Kristian, C. P. Wilson, and F. P. Landauer. Evidence for a supermassive object in the nucleus of the galaxy M87 from SIT and CCD area photometry. ApJ, 221:721–730, May 1978.

Q. Yu and S. Tremaine. Observational constraints on growth of massive black holes. MNRAS, 335:965–976, October 2002.

F. Yuan, S. Markoﬀ, H. Falcke, and P. L. Biermann. NGC 4258: A jet-dominated low-luminosity AGN? A&A, 391:139–148, August 2002.

F. Yuan, E. Quataert, and R. Narayan. Nonthermal Electrons in Radiatively Ineﬃcient Accretion Flow Models of Sagittarius A*. ApJ, 598:301–312, November 2003.

Y. Yuan, R. Narayan, and M. J. Rees. Constraining Alternate Models of Black Holes: Type I X- ray Bursts on Accreting Fermion-Fermion and Boson-Fermion Stars. astro-ph/0401549, January 2004.

J. i. Zand, F. Verbunt, J. Heise, A. Bazzano, M. Cocchi, R. Cornelisse, E. Kuulkers, L. Natalucci, and P. Ubertini. BeppoSAX-WFC monitoring of the Galactic Center region. astro-ph/0403120, March 2004.

J.J.M.i Zand, J. Heise, R. M. Kippen, P.M. Woods, C. Guidorzi, E. Montanari, and F. Frontera. Long and faint GRBs in archival BeppoSAX-WFC data. astro-ph/0305361, May 2003.

G. P. Zank and T. K. Gaisser, editors. Particle acceleration in cosmic plasmas; Proceedings of the Workshop, Bartol Research Inst., Newark, DE, Dec. 4-6, 1991, 1992.

Ya. B. Zeldovich. The Fate of a Star and the Evolution of Gravitional Energy upon Accretion. Soviet Physics Doklady, 9:195, 1964.

W. Zhang, S. E. Woosley, and A. I. MacFadyen. Relativistic Jets in Collapsars. ApJ, 586:356–371, March 2003.

248 R. L. Znajek. Black hole electrodynamics and the Carter tetrad. MNRAS, 179:457–472, May 1977.

249 Curriculum Vitae

Jonathan C. McKinney

Address

Physics Department University of Illinois at Urbana-Champaign 1110 West Green Street Urbana, IL 61801

Phone: (217) 766-6555 fax: (217) 244-7638 email: [email protected] web: rainman.astro.uiuc.edu/ jon/

Personal

Birth November 22, 1973 in Methuen, MA, USA

Education

2004 Ph.D. in Physics, Theoretical Astrophysics, UIUC

Dissertation: Black Hole Accretion Disk, Jet, and Corona (Prof. Gammie, advisor)

1999 M.S. in Physics, UIUC

1996 B.S. in Physics (Magna Cum Laude), Texas A&M Univ., College Station

Sr. Thesis: 2-D Wavefunction Time Evolution using Wavelets/Fourier transforms investigating Periodic Potentials (Prof. Siu Ah Chin, advisor)

Jr. Thesis: Atmospheric Polarization due to Incoherent Light from Sun: A Layer Model of the Atmosphere (Prof. George W. Kattawar, advisor)

250 Fellowships, Awards, & Contributions

2004 Institute for Theory and Computation (ITC) fellowship, CfA, Harvard College Observatory

2003 Coauthored NCSA NRAC proposal awarded 300,000 Tungsten SUs, with a signiﬁcant portion for my research. PI: Charles F. Gammie

2002 Contributed to NSF ITR Program Award 0205155, MHD Simulations in Full General Rela- tivity . PI: Charles F. Gammie

2001 Contributed to NSF Award 0093091, Theory of Black Hole Accretion Flows . PI: Charles F. Gammie.

2001-2004 NASA GSRP Fellow (S01-GSRP-044), annual proposals, sole author, source of current funding

1997, 2000 General Electric Fellow (for scholastic excellence as graduate)

1996-1997 Faculty Achievement Award (for leadership in the College of Science at Texas A&M)

1996 Summer Research Fellowship, University of Illinios at U-C

1994-2003 Golden Key National Honor Award: National Honor Society

Employment and Training

2004- ITC Postdoctoral Fellow, CfA, Harvard College Observatory, Harvard University

2001-2004 NASA Fellow, UIUC

1999-2000 Research Assistant, UIUC Computational and theoretical study of black hole accretion disks. Advisor: Prof. Charles Gammie Molecular Clouds, Galactic Dynamics, Accretion Disks

2000 Teaching Assistant, UIUC, Graduate level, The Physics of Compact Objects , Prof. Stuart L. Shapiro

1998 Research Assistant, UIUC General relativistic hydrodynamic processes involving shocks as applied to cosmological sheets. Advisor: Prof. Mike Norman, Sr. res. scientist, NCSA (now UCSD) Numerical methods to model astrophysical ﬂuid dynamical systems

1997-1999 Teaching Assistant, UIUC, ENGR. level: Q.M., E&M, Mech., and Stat. Mech.

251 1996-1997 Teaching Assistant, Texas A&M, Graduate level: Stat. Mech.

1997Sum Electrical Engineer Assistant for Turnkey in Plano, TX for Mr. Steve Williams

1996Sum Research Assistant (REU), UIUC Symmetry properties and chaos in electron transport in semiconductor superlattices. Advisor: Prof. David K. Campbell (now Dean Boston University) Nonlinear Dynamics of Electrons in Mesoscopic Nanostructures

1996 Research Assistant, Texas A&M University 1-D and 2-D wavefunction time evolution using wavelet/Fourier transform. Periodic reﬂectionless quantum waveguides. Advisor: Prof. Siu Ah Chin Theoretical nuclear physics; high-density matter; lattice calculations; Monte Carlo methods

1995Sum Repair and Maintenance Technician for Dallas Semiconductor in Dallas, TX for Mr. David Massey

1995 Research Assistant, Texas A&M University Atmospheric polarization due to incoherent light from Sun interacting with the layers of the sky. Light from Moon and Sun that create the green and blue shock on sunsets and sunrises. Advisor: Prof. George W. Kattawar Atmospheric/oceanic optics; radiative transfer w/ elastic and inelastic scattering in the atmosphere- ocean system

1994 Computer Technician and Autocad design for B.L. & P. Engineers in Dallas, TX for Scott Brady

Publications

McKinney, J.C. and Gammie, C.F., A measurement of the hydromagnetic luminosity of a Kerr black hole , 2004, ApJ, 611 , 977M

Watson, W. D., Wiebe, D. S., McKinney, J. C., and Gammie, C. F., Anisotropy of magnetohydrodynamic turbulence and the polarized spectra of OH masers , 2004, ApJ, 604 , 707W

Gammie, C.F., Shapiro, S.L., and McKinney, J.C., Black hole spin evolution , 2004, ApJ, 602 , 312G

Gammie, Charles F., McKinney, Jonathan C., and T´oth,G´abor, HARM: A numerical scheme for general relativistic magnetohydrodynamics , 2003, ApJ, 589 , 444G

McKinney, J. C. and Gammie, C. F., Numerical models of viscous accretion ﬂows near black holes , 2002, ApJ, 573 , 728M

252 Anninos, P. and McKinney, J. Relativistic hydrodynamics of cosmological sheets , 1999, Phys. Rev. D 60 , 064011

K. N. Alekseev, E. H. Cannon, J. C. McKinney, F. V. Kusmartsev, and D. K. Campbell. Symmetry- breaking and chaos in electron transport in semiconductor superlattices , 1998, Physica D. 113 , 129-133

K. N. Alekseev, E. H. Cannon, J. C. McKinney, F. V. Kuzmartsev, and D. K. Campbell. Sponta- neous dc current generation in a resistively shunted semiconductor superlattice driven by a terahertz ﬁeld , 1998, Phys. Rev. Lett. 80 , 2669-2672

McKinney, J. C., Alekseev, K. N., Cannon, E. H., and Campbell, D. K., Dissipative chaos and symmetry-breaking in semiconductor superlattices, 1996, REU Thesis, U. of Illinois, Urbana

Publications in Preparation

McKinney, J.C. and Gammie, C.F., General relativistic MHD simulations of thin disks , 2004, in preparation

Refereed Journals 2002 - Astrophysical Journal, Astrophysical Journal Letters

Invited Talks

CTA Seminar on Theoretical Astrophysics & General Relativity http://www.physics.uiuc.edu/Research/CTA/seminars/

2003Spr Intermediate-Mass Black Holes: Formation Theories & Observational Constraints

2002Spr Eﬃcient Acceleration and Radiation in Poynting Flux Powered GRB Outﬂows

2002Fal High-Energy Gamma Rays from AGN, GRBs, and Plerions

2001Fal Black Hole accretion in Active Galactic Nuclei (also prelim. exam)

2001Spr Bar-Driven Dark Matter Halo Evolution: A Resolution of the Cusp-Core Controversy

2000Fal Gamma-Ray bursts: Magnetized Collapsars and duration of GRBs

253 2000Spr Planet Formation: The Eﬀects of Thermal Energetics on 3-D Hydrodynamic Instabilities in Massive Protostellar Disks

1999Fal Discussion of Numerical Methods on the study of AGN: GRMHD

1999Spr Global Magnetohydrodynamical Simulations of Accretion Tori

REU (Research Experience for Undergraduates) http://www.physics.uiuc.edu/education/undergrad/reu/

1996Sum Semiconductor Superlattices

Professional Memberships

1999- American Astronomical Society (AAS)

1999- American Physical Society (APS)

Undergraduate Memberships

1994-1997 Society of Physics students (SPS)

1992-1996 Texas A&M Physics Club Meets every Tuesday with guest speakers Ongoing Projects such as the high school laser show Participated in all these and weekend shows to the public

1993-1997 Texas A&M Astronomy Club Meets every Friday Night Observe and photograph diﬀerent phenomena, such as the Shoemaker-Levy Comet impact on Jupiter http://www.physics.sfasu.edu/astro/sl9.html

Computational Experience

Beowulf Cluster Principle designer, builder, and manager of a 32 CPU Linux gigabit & Myrinet cluster for testing, development, and up to medium scale simulations. Design and associated publications: http://rainman.astro.uiuc.edu/cluster/

254 Digital Demo Room Helped create a web portal for astrophysical simulations: http://ddr.astro. uiuc.edu

Software Written Developed ZEUS-like parallel 3D MHD code from scratch Contributed signiﬁcantly to writing HARM Parallelized all our groups codes using MPI

Workshops NCSA Microprocessor Performance Tuning, Jan 2002 NCSA Linux Clusters Institute Workshop, Oct 2001 NCSA MPI Workshop, Mar 2001

Science Applications Expert: Mathematica, Supermongo, Matlab Basic: Maple

Operating Systems Expert: All forms of Linux, DOS, and Windows Basic: VMS, Solaris, SUN

Programming Expert: C, FORTRAN77 & 90, C++, Visual C++, Bash

References

Prof. Charles F. Gammie Dept. of Physics, MC-704 University of Illinois at Urbana-Champaign 1110 West Green Street Urbana, IL 61801-3080 (217) 333-8646 (oﬃce) (217) 244-7638 (fax) [email protected]

Prof. Stuart L. Shapiro Dept of Physics, MC-704 University of Illinois at Urbana-Champaign 1110 West Green Street Urbana, IL 61801-3080 (217) 333-5427 (oﬃce) (217) 333-9783 (lab) [email protected]

Prof. William D. Watson Dept. of Physics, MC-704 University of Illinois

255 at Urbana-Champaign 1110 West Green Street Urbana, IL 61801-3080 (217) 333-7240 (oﬃce) w-watson@staﬀ.uiuc.edu

Prof. & Dean David K. Campbell Boston University College of Engineering 44 Cummington Street Boston, MA 02215 (617) 353-2800 (oﬃce) (617) 353-5929 (fax) [email protected]

Dr. Peter Anninos University of California Lawrence Livermore National Laboratory Livermore, CA 94550 [email protected]

256