Object-Oriented Design of Database Stored Procedures
Michael Blaha, [email protected] Bill Huth, [email protected] Peter Cheung, [email protected]
Abstract base I/O. One nice feature of SPs is their support for op- tional input parameters. An SP may have optional inputs Object-oriented (OO) software engineering techniques are if: (1) the definition lists all outputs before inputs and (2) often used with programming languages, but they also ap- optional inputs have default values. ply to relational databases. OO techniques are not only helpful for determining database structure, but also for de- 3. Financial Application Example signing stored procedure code. In fact, we were surprised by the magnitude of the stored procedure benefits. OO The examples in this article are based on an application for techniques boost development productivity as well as the managing syndicated loans. The loans can be huge, involv- quality, clarity, and maintainability of the resulting code. ing billions of dollars, and can arise from lending to gov- ernments and multinational corporations. Participation in a 1. Introduction loan must be spread across multiple lenders because the risk is too large for a single lender. Object-oriented (OO) techniques are versatile. They are not Figure 1 shows an excerpt of a UML class model for only helpful for developing programs and structuring data- the application. An Asset is something of value and can be bases, but they are also effective for designing stored proce- a Currency (such as US Dollars, Euros, and Yen), a Loan- dures. A stored procedure is programming code that runs in Instrument, or an OtherAsset (such as bonds and stock). An the address space of a database management system Asset can involve various CounterPartyRoles. Methods (DBMS). All the major DBMS products (such as SQL Serv- (SPs) link an Asset to a CounterPartyRole (bindCPR), un- er, Oracle, MySQL, DB2) now support stored procedures. link an Asset from a CounterPartyRole (unbindCPR), and When we build database applications, we start by pre- retrieve the collection of CounterPartyRoles for an Asset paring a UML class model and then create a corresponding (getCPRs). database structure [2]. One of the subsequent steps is to im- A CounterParty is some organization that is relevant plement methods, that is the behavior for classes. We have for the purpose of tracking loans. (The term CounterParty found stored procedures to be helpful for implementing is banking jargon.) CounterPartyRoleType specifies the methods. ways that CounterParties can be involved in loans; exam- 2. The SQL Server SP Language ples include borrower, lender, guarantor, and agent. A CounterPartyRole combines a CounterParty with a Coun- We have been using SQL Server in our recent applications terPartyRoleType. The software not only tracks who is in- and emphasize it in this article. However, we believe that volved with Assets, but also tracks how they are involved. many of the techniques we discuss here would apply to oth- Methods can create a new object (new), delete an object (de- er DBMSs. lete), modify data (update), and retrieve data (getWithID). SQL Server has three kinds of stored procedures: ex- A LoanInstrument can be a FacilityAgreement, tended stored procedures (XPs), common-language-run- Tranche, or TrancheItem. A FacilityAgreement is an over- time stored procedures (CLRPs, new to SQL Server 2005 all financial amount that is available to a borrower. A [1]), and interpreted stored procedures (SPs). Developers Tranche is a loan that is made under the auspices of a Fa- write XPs and CLRPs in a programming language (such as cilityAgreement. Tranches are important because they pro- C or C++), compile them, and then link them into the data- vide the means for managing Drawdowns (borrowed base. In contrast, SPs are written with Microsoft’s Trans- funds). A TrancheItem is the portion of a Tranche covered act-SQL language and are interpreted with linking an im- by a particular lender. Thus a Tranche splits into plicit part of the language. TrancheItems, one for each lender. For convenience, we have favored SPs in our applica- Via inheritance from Asset, the objects in the different tions. Of the three approaches, SPs are the easiest to write subclasses can have CounterPartyRoles. Thus TrancheIt- and the interpreter overhead is small compared to data- em inherits CounterPartyRoles. The model does not en-
Page 1 Object-Oriented Design of Database Stored Procedures
Asset * * CounterPartyRoleType CounterPartyRole bindCPR 1 * * getCPRs name delete unbindCPR delete getWithID new new update update * 1 Currency LoanInstrument OtherAsset name name CounterParty deleteCommitment name getWithID deleteCommitments baseCurrency 1 getCommitments Commitment delete sumCommitments beginDate getWithID new 1 endDate * * commitmentAmount update 1 lender
FacilityAgreement Tranche TrancheItem * name name DrawdownItem beginCommitmentDate beginDate addCommitment drawdownItemAmount endCommitmentDate endDate delete manualOrComputed getAll addCommitment addCommitment getDrawdownItems * delete delete getLender getAll getAll 1 1 * 1 * getWithID getTranches getDrawdowns new Drawdown getWithID getWithID update getWithName getWithName drawdownAmount updateCommitment new new drawDate update update maturityDate updateCommitment updateCommitment apportion 1 bindCPR delete * getCPRs * getDrawdownItems getWithID new unbindCPR update
Figure 1 Excerpt of UML class model with public methods force some business constraints: (1) each TrancheItem has them. In contrast, methods to add and update Commitments exactly one lender and (2) of the many TrancheItems for a have constraints that vary, so the subclasses (Facil- Tranche, exactly one TrancheItem pertains to each lender. ityAgreement, Tranche, and TrancheItem) individually Since an Asset can be bound to any number of CounterPar- specify them. Similarly, the new, delete, update, and tyRoles, the model cannot enforce these constraints. How- getWithID methods also vary by subclass. By definition, ever the database structure can store the data and SPs can each TrancheItem has a single lender that is assigned when then enforce the constraints. a new TrancheItem is created. The getLender method re- A LoanInstrument may have any number of Commit- trieves a TrancheItem’s lender. ments that are summed to limit the amount available for The FacilityAgreement Commitments limit Draw- Drawdowns. Each Commitment has begin and end dates; downs for a borrower. The Tranche Commitment limits thus the total amount available varies over time. Drawdowns for a loan and the TrancheItem Commitments The methods to delete, get, and sum Commitments do limit exposure for a lender to a loan. Ultimately, the sum of not vary across subclasses, so LoanInstrument specifies the Commitments for a Tranche cannot exceed the sum of
Page 2 Michael Blaha, Bill Huth, Peter Cheung Object-Oriented Design of Database Stored Procedures the Commitments for its TrancheItems. (The total funds are important in that they improve understandability, doc- from lenders must be able to cover the withdrawals of a umentation, and productivity. borrower.) This is another constraint that SPs must check. A Drawdown is a removal of funds within the scope of 5. Public vs. Private a Tranche. A Tranche can involve multiple Drawdowns A public SP is available to other applications and the user that occur on various dates—the drawDate must lie within interface. A private SP is only for the use of other SPs in the begin and end dates for the Tranche. There are addition- the same application. In the syndicated loan application al constraints. The total Drawdowns cannot exceed the about 90% of the SPs were public and 10% were private. Commitment for its Tranche. Furthermore, the total Draw- Figure 1 shows only public methods. downs cannot exceed the total Commitment for its Facil- SQL Server does not distinguish between public and ityAgreement (traverse from Drawdown to Tranche and private. So we established a convention to indicate the dif- from the Tranche to FacilityAgreement). ference. A prefix of usp (user stored procedure) denotes a A DrawdownItem is the portion of a Drawdown for a public SP; a prefix of isp (internal stored procedure) de- particular lender. Consequently, a Drawdown may be asso- notes a private SP. Developers must be disciplined in ad- ciated with any relevant CounterPartyRoles, except for hering to this convention—isp SPs are solely for internal lenders. The apportioning of a Drawdown to Drawdown- use. Items is normally consistent with the splitting of its Similarly, we distinguished between private views Tranche into TrancheItems (manualOrComputed = “com- (prefix of iv) and public views (prefix of uv). puted”), but sometimes a DrawdownItem is manually ad- Private SPs are helpful in a number of situations. For justed (manualOrComputed = “manual”). example, superclass SPs that are overridden by subclasses The methods can link a Drawdown to a CounterParty- should be declared as private. Similarly, computational ar- Role (bindCPR), unlink a Drawdown from a CounterPar- tifacts are not part of the published interface and should tyRole (unbindCPR), and retrieve the collection of Coun- also be declared as private. terPartyRoles for a Drawdown (getCPRs). There is a meth- od to apportion a Drawdown to DrawdownItems consistent We found that it was more difficult to determine the with the splitting of its Tranche into TrancheItems. The ap- class owner for private methods than for public meth- portion method sets manualOrComputed to “computed”. ods—this is not surprising. Public SPs have intrinsic Reference [3] presents another application for which meaning; private SPs are really just for implementation we used OO techniques to design stored procedures—soft- convenience. coded values that can handle miscellaneous data for objects. 6. Coping with the Lack of Inheritance 4. SP Naming Inheritance is a mechanism that organizes classes by their similarities and differences. An operation is invoked on an A large application can have many SPs and naming can object and the language infrastructure automatically re- quickly become an issue. Developers must be able to find solves the operation to a method in the inheritance hierar- SPs with the desired functionality, avoid duplication (SPs chy. Microsoft’s SQL Server language is not an OO lan- with the same logic and different names), and avoid name guage and consequently lacks inheritance and support for clashes. We found it helpful to organize SPs about class method resolution. As a result, client SPs must know where names, using the convention
Page 3 Michael Blaha, Bill Huth, Peter Cheung Object-Oriented Design of Database Stored Procedures hierarchy. We infrequently move methods within the hierar- stored procedure invocation. Instead, you must first eval- chy so direct calling has not been a problem. We have been uate the function and store it in a variable. Then you pass using a simple tool that searches all SPs for occurrences of the variable as an argument. a string, so we can quickly find the callers of an SP. As an example, several SPs perform currency conver- We handle destructors differently. (A destructor is a sion. The SPs include a function that converts an amount method that deletes an instance of a class.) We use referen- from a source currency to a target currency. The conver- tial integrity and specify on delete cascade for each sub- sion function is called within a SQL statement. class relative to its superclass. So in essence, destructors • Views. Views can be handy for combining simple and are built into the fabric of the database structure. Whenever aggregate data. For example a view could return Tranche we need to delete an object that is described by an inheri- data along with the total drawdownAmount as of a date. tance hierarchy, we just delete the root-level superclass re- We also used views to dynamically compute derived cord and let deletion cascade down the inheritance tree. data. We found the performance to be reasonable, as long We have avoided multiple inheritance with SQL Server. as we carefully wrote the views. We tested our views to 7. Error Handling verify their performance. Given acceptable performance, it is better to compute derived data as needed rather than Our SPs have extensive error checking. In general, the store redundant data that must be kept consistent across presence of an error causes an SP to do nothing and return updates to underlying base data. an error message. For convenience, we use strings instead • Layers of views. Layers of views can be helpful for ef- of numeric error codes. The syntax for error strings is class- fecting a sequence of calculations. For example, we Name + ‘_’ + operationName + ‘_’ + errorMessage. (We wrote a series of views that computes Commitment time have tried to make error messages consistent across classes intervals for LoanInstrument and sums the applicable and operations.) commitmentAmounts for each time interval. It is often convenient to define a transaction, tentative- These techniques reduce portability because they exploit ly write the data, check the data, and then rollback if neces- SQL Server features. However, we believe the techniques sary. The pseudocode in Figure 2 uses the UML’s Object could be ported to another DBMS. Constraint Language [5] to illustrate this approach. First As a matter of policy we try to minimize the use of cur- update a Drawdown’s drawdownAmount and/or drawDate. sors. Cursors increase the size of stored procedures, take Then get the Tranche (T) for the Drawdown. Then check a longer to write, and are more error prone than SQL state- business rule for T—the total drawdownAmount cannot ex- ments. Where possible, we place computation into a SQL ceed the total commitmentAmount as of the drawDate. statement rather than loop through a collection with pro- gramming code. Given the combination of SQL queries, BEGIN TRANSACTION user-defined functions, and views, we seldom need cursors. update data for aDrawdown Some authors have noted the drawbacks of nulls in da- T := aDrawdown.Tranche tabases [4]. We agree that nulls are troublesome, but it is check that sum(T.Drawdown.drawdownAmount) <= not practical to entirely forego nulls. For example, nulls sum(T.Commitment.commitmentAmount) naturally arise from outer joins. We permitted nulls in our as of aDrawdown.drawDate implementation. COMMIT or ROLLBACK TRANSACTION 9. Idiom for Record Sets Figure 2 Error checking. Transactions can SQL code naturally deals with record sets—the inputs to an ease error checking. SQL query are tables and the result is a table. In contrast, programming code deals with variables and individual re- 8. Substituting SQL Code for Programming cords. Consequently there is a mismatch between SQL and programming languages. SQL is a powerful language and we were able to reduce de- Stored procedures have ordinary input and output pa- velopment effort by substituting SQL code for program- rameters that are intended for simple variables (integer, ming code. SQL Server can flexibly mix stored procedures date, text, and so forth). So how can a stored procedure re- with views and functions. turn a set of records? The SQL Server documentation is un- • User-defined functions. User-defined functions behave clear about how to resolve this problem. We considered the same as SQL Server’s built-in functions. Both can be several possibilities. freely combined with SQL code. About the only restric- • Hardcoding. Hard code the columns for a fixed number tion is that a function call cannot be an argument in a of records. For example, we could write an SP that takes
Page 4 Michael Blaha, Bill Huth, Peter Cheung Object-Oriented Design of Database Stored Procedures
a facilityAgreementID as input and returns the name, be- • Record handshaking. We could handshake on each out- ginDate, and endDate for each of its Tranches. With put record, using a parameter to hold state. The SP re- hardcoding, the output parameters would be name1, turns one output record at a time as Figure 3 illustrates. beginDate1, endDate1, name2, and so forth. This is a For brevity we have omitted some error checking. This clumsy approach that we would only pursue as a last re- approach is a reasonable option and is similar to the cur- sort. It is awkward to limit results to a fixed number of sor approach, but with a cursor SQL Server maintains records and the large number of output parameters would state. With record handshaking, application code main- make the code verbose and difficult to read. tains state. Database caching would mitigate the perfor- mance impact of handling one record at a time. • String of IDs. Return many IDs folded into a long string. • Select query. Define the SP as a select query. This works A separate function gets a record at a time using the ID. but has an odd syntax and is poorly documented. Figure This is also a clumsy approach. The string length would 4 shows an example. limit the number of IDs and we would need to write ad- ditional SPs to manipulate ID strings. • Callee temp table. In our experiments, we could not cre- ate a local SQL Server temp table within an SP and have • XML string. Store multiple records within an XML it persist across multiple invocations. (A global temp ta- string. Once again the string length would limit the ble works, but would be visible to other applications.) amount of data. Construction and parsing of the string • Caller temp table. The calling SP creates a local SQL would be straightforward. Server temp table. The called SP knows the temp table • Cursor. SQL Server cursors are easy to use within an name and stores the record set in the temp table for use SP, but it is not obvious how to maintain a cursor across by the calling SP. Figure 5 illustrates the idea. repeated SP calls. This does not appear to be a viable so- • Table-valued function. SQL Server lets developers de- lution. fine a function that returns a table. Such a function is es-
/* Get the next Tranche for a FacilityAgreement (in ascending name order). */ CREATE PROCEDURE usp_FacilityAgreement_getNextTranche ( @p_trancheID INTEGER OUTPUT, @p_name VARCHAR(50) OUTPUT, @p_beginDate DATETIME OUTPUT, @p_endDate DATETIME OUTPUT, @p_return VARCHAR(50) OUTPUT, @p_facilityAgreementID INTEGER, @p_priorTrancheID INTEGER = NULL ) AS
IF @p_priorTrancheID IS NULL BEGIN /* get the first Tranche */ SELECT @p_trancheID = T.trancheID, @p_name = T.name, @p_beginDate = T.beginDate, @p_endDate = T.endDate FROM Tranche AS T WHERE T.name IN (SELECT MIN(name) FROM Tranche WHERE facilityAgreementID = @p_facilityAgreementID) END ELSE BEGIN /* get the next Tranche */ SELECT @p_trancheID = T1.trancheID, @p_name = T1.name, @p_beginDate = T1.beginDate, @p_endDate = T1.endDate FROM Tranche AS T1 WHERE T1.name IN ( SELECT MIN(T2.name) FROM Tranche AS T2, Tranche AS T3 WHERE T2.name > T3.name AND T3.trancheID = @p_priorTrancheID AND T2.facilityAgreementID = @p_facilityAgreementID ) END
IF @p_trancheID IS NULL BEGIN /* trying to retrieve, after the last Tranche */ SET @p_return = 'FacilityAgreement_getNextTranche_priorTrancheIDisLast'; RETURN END
Figure 3 Record handshaking. A set can be retrieved one record at a time.
Page 5 Michael Blaha, Bill Huth, Peter Cheung Object-Oriented Design of Database Stored Procedures
CREATE PROCEDURE usp_FacilityAgreement_getTranches ( @p_facilityAgreementID INTEGER ) AS
SELECT trancheID, name, beginDate, endDate FROM Tranche WHERE facilityAgreementID = @p_facilityAgreementID
Figure 4 Select query. SQL Server can return a record collection.
/* If #temp_Tranches exists, get the Tranches for a FacilityAgreement.*/ CREATE PROCEDURE usp_FacilityAgreement_getTranches ( @p_return VARCHAR(50) OUTPUT, @p_facilityAgreementID INTEGER ) AS
IF EXISTS (SELECT * FROM tempdb.dbo.sysobjects WHERE id = object_id('[tempdb].[dbo].[#temp_Tranches]')) BEGIN DELETE FROM #temp_Tranches WHERE facilityAgreementID = @p_facilityAgreementID
INSERT INTO #temp_Tranches (facilityAgreementID, trancheID, name, beginDate, endDate) SELECT facilityAgreementID, trancheID, name, beginDate, endDate FROM Tranche WHERE facilityAgreementID = @p_facilityAgreementID END
Figure 5 Caller temp table. The SP stores records in a temp table with a predefined name.
sentially a parameterized view. Table-valued functions the field—otherwise, we do nothing. The performance of have restrictions. For example, they cannot update the these updates is acceptable, since there is a locality of ref- database and cannot execute SPs. Also there is no provi- erence for I/O (the database schema physically clusters an sion to return an error code for an unexpected error. object’s values together on the disc). Of the possible options, six are viable (string of IDs, XML For semantic reasons, we disallow update of some string, record handshaking, select query, caller temp table, fields. For example, usp_Tranche_update cannot update and table-valued function). We have been using the select facilityAgreementID. query in most of our applications. For mass updates, an alternative is to process an XML When possible, we imposed a meaningful order when file of bulk updates. Figure 6 demonstrates the approach. retrieving a collection of records. For example, when re- trieving the Tranches for a FacilityAgreement, we order the 11. Conclusion Tranches by name. We have found OO techniques to be helpful for designing 10. Idiom for Updates database stored procedures. The benefits were greater than we had expected. OO techniques have improved the clarity We have also devised an idiom for updates. We want to be of our thought, tightened the coupling between the data- able to update an entire record or any combination of fields base structure and the stored procedures, and improved our within a record, all with the same SP. We do this by de- ability to understand, organize, and manage the resulting faulting the input to an ‘impossible’ value for each updat- code mass. We recommend the following practices. able field. Then we can detect genuine updates. Some ex- • SP names. Each SP implements a method and conse- amples of the ‘impossible’ values are: quently belongs to a class. Therefore you should orga- • String = '@#%$&' nize SP names about class names. The convention
Page 6 Michael Blaha, Bill Huth, Peter Cheung Object-Oriented Design of Database Stored Procedures
CREATE PROCEDURE usp_Drawdown_updateMany ( @drawdownInfo VARCHAR(1000) = '
/* Put the info into a temporary table variable */ EXEC master.dbo.sp_xml_preparedocument @hdoc OUTPUT, @drawdownInfo
INSERT INTO @tblDrawdownInfo (did,trancheID,drawdownAmount,drawDate,maturityDate) SELECT did, trancheID, drawdownAmount, drawDate, maturityDate FROM openxml (@hdoc, '/Drawdowns/Drawdown') WITH (did INT, trancheID INT, drawdownAmount MONEY, drawDate DATETIME, maturityDate DATETIME) temp
/* Remove the internal xml representation. */ EXEC master.dbo.sp_xml_removedocument @hdoc
UPDATE D SET drawdownAmount = TDDI.drawdownAmount, drawDate = TDDI.drawDate, maturityDate = TDDI.maturityDate FROM Drawdown D INNER JOIN @tblDrawdownInfo TDDI ON TDDI.did = D.drawdownId
RETURN
Figure 6 Bulk update using XML. XML can handle bulk inserts and updates.
around of a prefix (such as ‘usp’ for public and ‘isp’ for string, record handshaking, select query, caller temp ta- private) to indicate intent. Software developers must ble, and table-valued function are viable options. have the discipline for adhering to the intent. • Idiom for updates. If you follow our suggestions, an up- • Inheritance. SPs lack support for inheritance. Nonethe- date SP can update any combination of fields for a class. less, it is still helpful to think about inheritance during 12. References conceptual design. We found the best workaround to be direct calling of SPs and foregoing the use of polymor- [1] Bob Beauchemin, Niels Berglund, and Dan Sullivan. A phism. First Look at SQL Server 2005 for Developers. Boston: Ad- dison-Wesley, 2004. • Substituting SQL code for programming. SQL is a [2] Michael Blaha and William Premerlani. Object-Oriented powerful language and developers should take advantage Modeling and Design for Database Applications. Upper of it. Many DBMSs can flexibly mix SPs with views and Saddle River, NJ: Prentice Hall, 1998. functions, making it convenient to substitute concise [3] Michael Blaha. Designing and Implementing Softcoded SQL code for verbose programming code. Values. IEEE-CS ReadyNote, 2006. [4] C.J. Date. Relational Database: Writings 1991-1994. Bos- • Idiom for record sets. SPs have no natural mechanism ton: Addison-Wesley, 1995. for returning a record collection. We considered various [5] Jos Warmer and Anneke Kleppe. The Object Constraint alternatives and determined that string of IDs, XML Language. Boston: Addison-Wesley, 1999.
Page 7 Michael Blaha, Bill Huth, Peter Cheung