A Realistic Approach to Spatial and Temporal Complexities of Computational Algorithms

Mathematics and Computers in Business, Manufacturing and Tourism

AHMED TAREK AHMED FARHAN Department of Science Owen J. Roberts High School Cecil College, University System of Maryland (USM) 981 Ridge Road One Seahawk Drive, North East, MD 21901-1900 Pottstown, PA 19465 UNITED STATES OF AMERICA UNITED STATES OF AMERICA [email protected] [email protected]

Abstract: Computational complexity is a widely researched topic in Theoretical Computer Science. A vast majority of the related results are purely of theoretical interest, and only a few have treated the model of complexity as applicable for real-life computation. Even if, some individuals performed research in complexity, the results would be hard to realize in practice. A number of these papers lack in simultaneous consideration of temporal and spatial complexities, which together depict the overall computational scenario. Treating one without the other makes the analysis insufﬁcient and the model presented may not depict the true computational scenario, since both are of paramount importance in computer implementation of computational algorithms. This paper exceeds some of these limitations prevailing in the computer science literature through a clearly depicted model of computation and a meticulous analysis of spatial and temporal complexities. The paper also deliberates computational complexities due to a wide variety of coding constructs arising frequently in the design and analysis of practical algorithms.

Key–Words: Design and analysis of algorithms, Model of computation, Real-life computation, Spatial complexity, Temporal complexity

1 Introduction putations require pre and/or post processing with the remaining operations carried out fast enough. With Computation time and memory space require- these computational scenarios, an Amortized Analysis ments are two major constraints in computer imple- is used. With Amortization, the total time required for mentation of real-life algorithms. Though there are a n operations is bounded asymptotically from above by ∈ G(n) wide variety of formal notations for expressing these a function g(n), and g(n) O( n ). Hence, Amor- computational constraints, big-oh notation is the most tized Time is an upper bound for the average time of customarily used. Big-oh provides the upper-bound an operation in the worst case. So essentially, the in- on temporal(time) and spatial(space) requirements of vestment in the pre-work and/or post-work during the a computer algorithm, which is the most difficult ob- computation amortize or average out. struction to cross in digital computation. This paper Computer memory is a re-usable resource from primarily focuses on the upper-bounds as expressed the operating system standpoint and may be released through the standard big-oh notation for both tempo- for further reallocation. Temporal resources are con- ral and spatial complexities. sumable, and once spent, there is no return to that Temporal complexity is the quantity of CPU time point in time. Before today, computer algorithms were necessitated by an algorithm for its computer imple- not capable of time travel, which is turning back the mentation. Spatial complexity is the number of mem- clock - whether it’s a parallel or a pure sequential al- ory cells that an algorithm truly requires for compu- gorithm. Though there is a significant difference be- tation. A good algorithm tends to keep both of these tween temporal and spatial complexities from the re- requirements as small as possible. However, in real- allocation standpoint, spatial complexity shares many ity, it is strenuous to design an algorithm that is both of the same features due to temporal complexity. time and space efficient. In that event, there remains a This paper considers the temporal complexity to- trade-off between these two requirements. In certain wards the beginning. Commencing with the simplistic situations, not each of the n computer operations re- coding constructs arising frequently in real-life com- quire the same amount of time. Specially, some computation, the paper gradually transitions to the rela-

ISBN: 978-960-474-332-2 23 Mathematics and Computers in Business, Manufacturing and Tourism

tively complex models of computation in practice, and problems that can be solved by a Non-deterministic for each one of them, provides the upper-bound for Turing Machine using space in the order of O(f(n)), temporal resource requirements. Following the tem- and unlimited time. Therefore, the computation is poral complexity, the paper deals with the spatial com- done in O(f(n)) space by relaxing time to unlimited. plexity in an elaborate and easy to comprehend fash- NSP ACE stands for Non-deterministic Space Com- ion. In particular, spatial complexity is important to putation, which is the Non-deterministic counter-part sorting algorithms. Starting with the simple spatial in- of DSPACE. stances, the paper gradually approaches the relatively Please refer to [2] and [4] for the definition of complex computational models. big-oh complexity. Following is the basis of Com- In Section 2, specific terms and notations used in plexity Order using big-oh, and provides the complex- this paper are discussed briefly. Section 3 deals with ity class hierarchy. 2 k n a variety of coding constructs that frequently arise in log2n < n < nlog2n < n < . . . < n < 2 < realistic computations and provides the big-oh time Cn < n! complexity for each one of them. Section 4 considers spatial complexity in an elaborate fashion. Section 3 Temporal Complexity of Computational 5 explores the realistic issues in temporal and spatial Algorithms complexity models discussed in this paper. Section 6 deduces conclusions based on models and analysis in The aggregate time required to execute a pro- the paper, and explores future research avenues. gram depends both on the compile time and the run time. Compile time does not depend on the dynamic 2 Terminology and Notations parameters involved during computation. Therefore, the same compiled program may be executed several In this paper, following notations are used. times, each time with an individual input of different n: Denotes input or instance size. size. Execution time of an algorithm is the principal g(n): Highest order term in the expression for com- focus in temporal complexity analysis. plexity without coefficients determining the complex- In determining the temporal complexity, there ity class of algorithms. are: operations count and step count. Operations f(n): Complexity function with an input size, n. count is the number of additions, multiplications, f´(n): Complexity function without the constant coef- comparisons and other operations used during the ficients in f(n). computation. Success in operations count depends on O(g(n)): Big-oh complexity with problem size, n, the ability to identify the crucial operations that con- which represents a set corresponding to the complex- tribute most to the temporal complexity. Step count ity class, g(n). accounts for all time spent in all parts of the pro- T (n): Temporal complexity of an algorithm or a func- gram/function. Since 100 additions may be 1 step or tion with an input size, n. 200 multiplications may also be 1 step, a program step S(n): Spatial complexity of an algorithm or a func- is the syntactically or semantically meaningful seg- tion with an input of size n. ment of a program for which the execution time is B(n): Space-time Bandwidth Product for an input of independent of the instance characteristic. size n. Time complexity calculates the total time con- C: Any constant value. sumed for the given instance characteristic. How- Space-time Tradeoff: A situation where memory us- ever, the instance characteristic varies from input to age can be reduced at the cost of slower program exe- input even with the same problem size. For exam- cution, or vice versa: the computation time can be re- ple, the number of swaps performed by the Bubble duced at the cost of increased memory consumption. sort depends on both array size and array elements. DSPACE(f, n): DSP ACE stands for the De- With the same array size n, the array elements may be terministic Space Computation. In this paper, vastly different, which will result in different number DSP ACE(f, n) denotes the total number of mem- of swaps performed by the algorithm. As a result, the ory cells used during the deterministic computation operation count may not be uniquely determined by of f(n). Here, f indicates the algorithm or func- the selected instance characteristic. Hence, it is neces- tion under consideration, and n is the input of size, sary to have the best, the worst and the average num- |n|. It is often abbreviated as DSP ACE (f). Hence, ber of operation counts yielding the best, the worst DSP ACE(f, n) ∈ O(g(n)). However, DSP ACE and the average time complexities. For vast majority (f) is not defined whenever the computation of f(n) of the applications, average complexities suffice. does not halt. Big-oh complexity is usually expressed by the NSPACE(g, n): It denotes the set of all decision fastest growing term in the complexity function.

ISBN: 978-960-474-332-2 24 Mathematics and Computers in Business, Manufacturing and Tourism

There are 4 algorithmic steps in determining the big- the CPU time consumed to execute statement, Si, oh complexity that may be employed as a computer for∑i = 1, 2, . . . , k, then the total time consumed k program. The complete algorithm is described below: is, i=1 ci = C, which is a constant. Applying Algorithm big ohComplexity(n) the algorithm to determine the big-oh complex- Purpose: This algorithm determines big oh com- ity, the complexity order is, O(1). plexity with instance size n (single instance variable). 2. Simple Loops: The following is a pro- Input: Complexity function, f(n) on n with k totype for loop found in many pro- terms. grams. Output: Big oh complexity, O(g(n)). for(i = 0; i < n; i + +) {St;} int[] arr1 = new int[k] {array, arr1 holds the power of n for each term in the k-term complex- Total time to execute with the loop = (the num- × ity function, f(n).} ber of times the loop executes) (time required int highest power = arr1[0] {Because the terms for each execution of the loop). The statement se- are not ordered, determine the true highest order quence, St consumes C amount of constant time. term in f(n).} With the loop, St executes n different times. for j=1 to k − 1 do Therefore, aggregate time spent in the loop is, if arr1[j] > highest power then Cn. Utilizing algorithm to determine the big-oh highest power = arr1[j] complexity, this simple loop is, O(n). end if end for 3. Nested Loops: g n = power(n, highest power) for(i = 0; i < n; i + +) Print O(g n) as the big oh complexity. for(j = 0; j < n; j + +) {St;} Above algorithm is general, and may be extended Here, the statement sequence St consumes C for complexity functions involving multiple instance amount of constant time. The nested loop exe- characteristics or for fractional expressions. cutes n times for each execution of the outer for loop. Therefore, it executes n × n = n2 times in Example 1 Consider the complexity function, total. The total CPU time consumed is, C × n2. 2 − f(n, m) = (h(n, m) + 4n 3m + 2) on n The big-oh complexity is, O(n2). If there are k 2 and m. Also, function, h(n, m) = 2n + m. nested loops, each executing n times, the com- Here, h(n, m) is nested within function, f(n, m). plexity order will be O(nk). Hence, f(n, m) = (2n2 + m)2 + 4n − 3m + 2 4 2 2 − = 4n + m + 4n m + 4n 3m + 2. Re- 4. Inner Loop Index Depends on Outer Loop In- ´ moving the constant coefﬁcients, f(n, m) = dex: (n4 + m2 + n2m + n + m + 1). As the order for(i = 0; i < n; i + +) of n2m is, 3, the highest order term without coefﬁ- for(j = 0; j < i; j + +) {S ;} cients is, g(n, m) = n4, and f(n, m) ∈ O(n4). This t Upper-bound is independent of m. An example of Here, for n execution of the outer loop, the such complexity functions are the Graph algorithms, nested loop executes, (0+1+2+...+(n−1)) or ∑ − − × − where n corresponds to the set of vertices, and m n 1 k or (n 1) (n 1+1) times and has a com- represents the set of edges. k=0 2 plexity order of, O(n2).

3.1 Guide to Determining the Temporal 5. If-then-else statements: With If-then-else state- Complexity ments, the worst-case time complexity is impor- Common coding constructs with brief guidelines tant. The worst-case time is the time required by to their temporal complexity analysis are presented as the test, plus either the then part or the else part follows. Real-life coding constructs contain one or time, whichever is larger. Consider the following more of these fundamental coding structures. code: if (x is equal to y) then 1. Simple Statement Sequence: A set of indepen- return false; dent statements following one after another. A else general structure is, S1; S2; ... ; Sk, where k is { for(i = 0; i < n; i + +) { St; } a constant, and each Si, 1 ≤ i ≤ k represents an return true; } independent program statement. If ci designates end if

ISBN: 978-960-474-332-2 25 Mathematics and Computers in Business, Manufacturing and Tourism

In this example, either the if or the else part will while (j ≥ 1) { St; j = j/k; } be executed. Let the time for the if test be c0. Suppose the while loop executes i times. If each return statement takes c1 amount of time, n ≤ ≤ i Therefore, i 1. This provides, n k , then the else part (larger) will yield with the time k or logk(n) ≤ i. Hence, the minimum i is, complexity function, f(n) = (c + c + Cn). 0 1 imin = logk(n). Each execution of the This if-then-else structure has a linear time com- while loop consumes C amount of time. plexity, O(n). Therefore, the total time required is, C × imin. Also, log (n) = log2(n) × log (2), 6. Loop Index Varies Nonlinearly: These are also k k with logk(2) being a constant. Using the al- known as Logarithmic loops, since their com- gorithm, big-oh complexity for the division plexity order is always logarithmic. There are loop is O(log2(n)). three types of logarithmic loops: Multiplication loops, Division loops, and a Combination of (c) Multiplication and Division Loop Com- Multiplication and Division loops that occur dur- bination: A few complex coding con- ing computation in practice. structs might include a combination of multiplication and division loops, one be- (a) Multiplication Loops: The structure being nested within the other. The complexity order of such nested loops is gins with the loop control variable initial- r ized to its minimum value, which is usu- expressed as, O((log2n) )). Here, r ally 1. The control variable then increases is the total number of loops inside the by a constant real or integer multiplica- nested coding construct. An example fol- tion factor greater than 1.0 (k > 1.0) at lows. each loop iteration up to the upper-bound j = n; (n). Once the value of the loop control // Here, j is a loop control variable. variable exceeds the upper-bound, the loop- while (j ≥ 1) { l = 1; ing terminates. The following is an exam- // Here, l is another loop control variable. ple: while (l ≤ n) { S1; ∗ } j = 1; l = l k1; // Here, k1 > 1.0. //In above, j is the loop control variable. S2; j = j/k ; } while (j ≤ n) { St; j = j ∗ k; } 2 // Here, k2 is an integer or a real factor. Suppose the while loop executes i times. // Always, k2 > 1.0. Therefore, 1×ki ≤ n. Using the properties Suppose that the inner loop executes d1 of logarithm, i ≤ logk(n). Hence, the max- times and the outer loop executes d2 times. imum possible value for i, imax = logk(n). × If each execution of the while loop takes C The inner loop continues as long as, 1 d1 ≤ amount of constant time, the total time re- k1 n. This eventually provides, log (n) ≥ d . The maximum possible quired is, C × i = C × log (n). Us- k1 1 max k iteration of the inner loop is, d = ing the properties of logarithm, log (n) = 1max k log (n). Similarly, the minimum possi- log (n) × log (2). But log (2) is a con- k1 2 k k ble iteration of the outer loop is, d = stant. Hence, the big-oh complexity for the 2min log (n). For each execution of the outer multiply loop is, O(log (n)). k2 2 loop, the inner loop executes a maximum (b) Division Loops: A division loop com- of logk1 (n) times. Hence, for logk2 (n) iter- mences with the loop control variable ini- ation of the outer loop, the inner loop ex- tialized to its upper bound (the maximum × ecutes for a total of logk2 (n) logk1 (n) value, n), which then gradually decreases times. Suppose each execution of the in- by a constant division factor (k) greater ner loop takes C1 amount of constant time. than 1.0 (k > 1.0) at each loop iteration Hence, total time consumed in executing up to the given lower-bound (usually 1). × × the inner loop is, C1 logk2 (n) logk1 (n), Once the value of the control variable be- which is the highest order term in complex- comes less than the pre-set minimum, the ity function. Using the properties of log- loop terminates. Following is an exam- × × arithm, logk2 (n) logk1 (n) = log2(n) ple: × × logk2 (2) log2(n) logk1 (2). Here, both

j = n; logk2 (2) and logk1 (2) are constants. As- × // Here, j is the loop control variable. suming logk2 (2) logk1 (2) = C, the highest

ISBN: 978-960-474-332-2 26 Mathematics and Computers in Business, Manufacturing and Tourism

order term becomes C1 × C × log2(n) × program terminates, complexity of the recursive func- log2(n). Therefore, the complexity order tion, direct versus indirect recursion, and other re- 2 is, O((log2(n)) ). If there are r nested lated factors. Recursive functions are relatively easy loops, exercising a similar approach, the to code in programming languages, but they impose r overall time complexity is, O((log2(n)) ). large computational burdens to the computer, espe- Here, r is an integer, and r ≥ 2. cially memory space requirement. There are two dis- tinct phases that the computer passes through in stor- Following result is obvious. ing information as activation records on the Stack Frames [4] inside the Recursive Stack Space. Dur- Theorem 2 (Loop Control Factor Theorem) For ing Phase 1 (Build-up Phase), values are stored on the Multiplication and/or Division Loop(s) to converge, stack frames. During Phase 2 (Return Phase), stored the multiplication and/or the division factor, k used values inside the stack frames are popped out and used together with the Loop Control Variable(s) should be in computation. an integer or a real number strictly greater than 1.0. For spatial requirement, there are two parts. The Fixed Part, which is independent of the input and in- Proof: The factor k is used to gradually reduce the stance characteristic, includes the instruction space loop control variable to converge it to the set pre- and the data space for holding simple variables, con- determined maximum for the multiplication loop(s), stants, compiler generated temporary variables, etc. or to the pre-determined minimum for the division The Variable Memory Requirement depends on the loop(s). Looping continues until k reaches or exceeds size of the input parameters and the particular problem the preset value. If k < 1.0, the loop control vari- instance being solved. This part involves dynamically able value will decrease for multiplication loop(s) and allocated memory, recursive stack space, etc. The to- will increase for the division loop(s). As a result, the tal computational memory (RAM for P Cs) require- variable will never reach or exceed the preset maxi- ment may be expressed as, S = Sf + Sv = C + Sv. mum and/or minimum value, and will result in an in- Here, S is the total spatial requirement, Sf is the Fixed finite loop. If k = 1.0, the loop control variable value Space (which is a constant C for a specific computer will never change, and as a result, will never reach or program) and Sv is the Variable Space. Using the transcend the preset value, also resulting in an infinite DSP ACE notation introduced earlier in section 2, loop. Therefore, strictly, k > 1.0. ⋄ DSP ACE(f, n) = S = (C + Sv). Following is a comprehensive guideline to the spatial complexity 4 Spatial Complexity using Big-Oh analysis of algorithms and functions.

Spatial complexity may be compared to the num- 4.1 Spatial Complexity Analysis of Algo- ber of scratch papers required while solving a prob- rithms and Functions lem with pen and paper. Temporal complexity may be compared to the amount of time spent in working out In general, spatial complexity plays a major role the problem using pen and paper. Big-oh complex- in dynamic memory allocation and recursive compu- ity may provide upper-bound on the memory require- tations. When it comes to recursion, it is not rec- ments as well. Recursive and dynamic data structures ommended to look at the Average Space Complex- most frequently inﬂuence the big-oh notational space ity, since failure to comply with the recursive space complexity. requirement would eventually lead to program crash. Total space requirements for a coding construct Therefore, in most of the cases, the analysis is done may be attributed to 3 separate areas, which are the for the Worst-case Spatial Complexity. Instruction Space, the Data Space and the Environ- ment Stack Space. Instruction space is required to 1. Simple Dynamic Array: Java frequently sup- load the compiler, programming language libraries, ports dynamic array declaration inside its heap operating system modules for the program’s execu- space. tion, etc. Data space is required to load constants, // Following determines the minimum in an ar- local and global variables, etc. Environment Stack ray. Space is required to deﬁne stack areas inside compu- int k = 0, n; tational memory. With recursive function calls, com- String n string = keyboard.readLine(); puter builds up one or more recursive stack area(s) in- n = Integer.parseInt(n string); side the environment stack space. Size of the recur- int A[] = new int[n]; sive stack largely depends on the depth of recursion, Random generator = new Random(1000000); the number of recursive function calls made until the for (int i = 0; i < n; i + +) {

ISBN: 978-960-474-332-2 27 Mathematics and Computers in Business, Manufacturing and Tourism

A[i] = (generator.nextInt(3 ∗ n) + 1); } If the recursion is a tail recursion, it needs for (int i = 1; i < n; i + +) {if (A[i] < A[k]) to be converted to the corresponding iterative { k = i; }} construct. Tail recursion always incurs a space penalty due to the allocation of the recursive Assuming, each of an integer and a string type stack space. The iterative structure correspond- element occupies 1 unit of memory, C1 = 1+1+ ing to a tail recursion does not necessitate recur- 1+1 = 4 for holding the values of k, n string, n sion stack space or any other data structure that and i. Also, generator takes up a fixed k units of would consume additional memory space [3]. As memory space. Hence, C = (4 + k). The actual a result, this replacement usually improves spa- value of n will depend on the keyboard input, tial complexity. which in turn determines the dynamic array size. not a tail recursion Hence, Sv = n. Therefore, DSP ACE(f, n) = If the recursion is , its con- n + 4 + k, and DSP ACE(f, n) ∈ O(n). version to the iterative construct is much more intricate. While a recursive data structure can 2. Nested Dynamic Arrays: Java supports dy- always be converted into an equivalent itera- namically allocated nested arrays on its heap tive data structure, without the tail recursion, space. this inter-conversion process involves painstak- // Java code that performs matrix addition. ing consideration of information, which would int n, i, j; otherwise automatically be considered by the String n string = keyboard.readLine(); run-time support system [4] associated with the n = Integer.parseInt(n string); recursion stack. This automated consideration int P [ ][ ] = new int[n][n]; also involves a certain degree of hardware sup- int Q[ ][ ] = new int[n][n]; port, which involves saving the copies of all pa- int R[ ][ ] = new int[n][n]; rameters, local variables, and reentry point in a Random generator = new Random(1000000); separate activation record for each recursive call. for (i = 0; i < n; i + +) { For recursion to iteration inter-conversion, if ad- for (j = 0; j < n; j + +) { ditional data structures are required, recursion P [i][j] = (generator.nextInt(4∗n)+1); would be more efficient. Q[i][j] = (generator.nextInt(4∗n)+1); With recursion, the readability of the code im- R[i][j] = P [i][j] + Q[i][j]; }} proves, and the maintainability of the software becomes easier [3]. With some algorithms, the There are 3, n×n matrices required for the above use of recursive data structure even yields bet- code inside the dynamic memory area. The size ter performance in comparison to its iterative of each matrix is contingent upon the input size, counter part. As an example, consider the fol- n. Hence, total dynamic memory requirement, lowing table, in which, iMS stands for Itera- S = 3n2. The fixed space for storing i, j, v tive Merge Sort, rMS is Recursive Merge Sort, n string, n and generator is C = 4 + k units. iQS abbreviates Iterative Quick Sort, rQS is Re- Therefore, DSP ACE(f, n) = 3n2 + 4 + k. Ac- cursive Quick Sort, and pQS is the acronym for cordingly, DSP ACE(f, n) ∈ O(n2), with the Pseudo-Iterative Quick Sort. Java code segment having a quadratic space complexity. From Table 1, the Recursive Quick Sort has consumed the least amount of time throughout the 3. Recursive Data Structures: In any recursive computation. Though both iterative and recur- method, space is always required to hold the sive quick sorts have the same average time com- stack frames [4] created by the recursive calls in- plexity of O(nlog(n)), the recursive version runs side the dynamic memory area. Hence, recursive faster compared to it’s iterative counter part with structures require scrupulous analysis for spatial considerably large values of input, n. This is due complexity. The maximum dynamic memory re- to the constant coefficients with the highest or- quired to hold the stack frames is, SFn = (size der term in the complexity expression. Pseudo- of each stack frame)× (the depth of recursion). iterative quick sort combines the flavor of recur- Here, n is the input size. For a specific recursion with iteration by using the stack operation. sive data structure, the size of each stack frame Stack operations simulate the behavior of recur- is a constant, Csf . Hence, SFn = Csf × (depth sive stack space. Therefore, this hybrid algo- of recursion). Stated another way, SFn ∝ (depth rithm is known as the Pseudo-Iterative Quick of recursion). The depth of recursion, DRn is a Sort. The purely Iterative Quick Sort makes use function of input size, n, and SFn ∝ DRn. of an array and uses swap operations to perform

ISBN: 978-960-474-332-2 28 Mathematics and Computers in Business, Manufacturing and Tourism

The average Merge Sort exacts an amount of Table 1. Computation time in milliseconds space proportional to the average height of for different sorting algorithms. the tree structure describing the recursive calls, which is O(log2(n)) [3]. Hence, the algorithm Size, n iMS rMS iQS rQS pQS designers might think that O(log2(n)) space is sufficient in general. If applications are designed 105 31 31 31 16 16 with the average space in mind, and if for suf- 5 × 105 140 172 109 78 109 ficiently large values of n, the worst case space complexity occurs due to some problem instance, 106 312 360 218 172 203 × 6 and the size of the allocated memory is exceeded, 1.5 10 406 547 328 281 297 the program will simply crash! 2 × 106 547 735 438 375 406 Merge Sort works by coalescing the sorted sub- 2.5 × 106 703 938 531 468 484 lists. If implemented as a recursive algorithm, × 6 3 10 813 1, 172 641 562 594 the depth of recursion for Merge Sort with an 6 3.5 × 10 1, 172 1, 328 781 656 703 input of size n is, log(n), and hence, for the Recursive Merge Sort, the space complexity is, O(log(n)). For the Iterative Merge Sort using an array, the dynamic memory requirement is sorting. However, it does not make use of any contingent upon the input size, n. The iterative stack operation for its implementation. On av- merge sort is usually performed using a scratch erage, the pseudo-iterative implementation con- pad buffer in the dynamic memory area, which is sumes time, which is larger than the recursive proportional to the size of the array. Hence, the implementation but smaller than the purely it- iterative merge sort space complexity is, O(n). time (n) ≤ erative version. In general, recur This is also the lower-bound on memory require- time (n) ≤ time (n) pseudo iter with considerable ments due to the size of the scratch pad buffer. n input size, . Hence, S(n) ∈ Θ(n). The memory postulations Though both Iterative and Recursive Merge Sorts for merge sort may be decreased considerably have the same average and the worst case time with linked list implementation, which yields, complexities of O(nlog(n)) as the Quick Sort, S(n) ∈ O(1). Table 2 shows the spatial com- they usually run slower than their Quick Sort plexity of the commonly used sorting algorithms. counterparts with large enough values for n. This is due to the constant coefficients associated with the highest order terms in their time complex- Table 2. Spatial complexity of the frequently ity expressions. Though the worst case time used sorting algorithms. complexity of the Quick Sort implementations is O(n2), for most of the cases, the average com- Sorting Algorithm Space Complexity plexity holds with the program’s computation time behavior. As a result, quick sort usually per- Quick Sort O(n) forms much faster than merge sort. The experi- Merge Sort(recursive) O(log(n)) 1 mental results in Table were obtained using a Merge Sort(iterative) O(n) Sony VAIO desktop running on Intel’s 2.8 GHz Bubble Sort O(1) Pentium 4 processor with 512 MB of memory (RAM) space. Selection Sort O(1) Insertion Sort O(1) 4. Sorting Algorithms: Quick Sort works by par- Shell Sort O(1) titioning the list elements, and is an in-place Heap Sort O(1) sorting algorithm. The algorithm is fundamen- Radix Sort O(n) tally recursive [3], and fits nicely with the recursive data structure. All extra space required for sorting with quick sort comes from the stack of the recursive calls in the environment stack 5 Simultaneous Consideration of Spatial and space. Its space complexity is O(n), since in Temporal Constraints worst-case computation, the number of activation records [4] on the recursive stack space can The temporal and spatial complexities largely in- be in O(n). fluence algorithm design. Algorithms for Fibonacci

ISBN: 978-960-474-332-2 29 Mathematics and Computers in Business, Manufacturing and Tourism sequence are widely researched in computing. There reduced to O(1) through using a static instead of are four major algorithms for the Fibonacci sequence, a dynamic array. and each one of them is considered below 3. Space Efficient Fibonacci Sequence Al- 1. Recursive Fibonacci Sequence: The following gorithm: The DP algorithm for fibonacci program recursively computes the Fibonacci se- sequence may be modified to use a much quence. In the program code, F [n] computes the smaller amount of computational mem- nth fibonacci number. ory. int F (int n) { if (n ≤ 2) return 1; // Code to compute the nth fibonacci number. else return F [n − 1] + F [n − 2]; } int fibonacci(int n) { int a = 1, b = 1; for (int i = 3; i ≤ n; i + +) {int c = a + b; The temporal complexity of this recursive ver- a = b; b = c;} sion is, T (n) ∈ O(tn), which is exponential. return b;} Here, t = 1.61803, and is the golden ratio. Space complexity is, S(n) ∈ O(n), which is In the above program, c represents F [i], b repre- linear on the size of the recursive stack space. sents F [i − 1], and a represents F [i − 2]. The The algorithm needs to perpetuate two recur- 2 extra assignments after the sum shift those 2 sive stacks at the same time due to the nature values over in preparation for the next iteration. of recursion. The space-time bandwidth product, The time complexity for this iterative algorithm B(n) ∈ O(ntn), is pseudo-exponential. is, T (n) ∈ O(n). 2. Dynamic Programming (DP) Fibonacci Se- With this modified version, each step through the quence: The dynamic programming-based fi- loop uses only two previous values of F (n). In- bonacci sequence uses the following coding con- stead of storing these values in a static or in a dy- struct. namic array, they are stored as two independent variables. Though this requires some swapping int[ ] fibonacci(int n) { around of values, the computational memory re- int[ ] F = new int[n + 1]; quirement substantially reduces. int F [1] = F [2] = 1; for (int i = 3; i ≤ n; i + +) 4. Fibonacci Sequence Through Binet’s For- F [i] = F [i − 1] + F [i − 2]; mula: Binet’s formula is efficacious considering return F ;} both time & space, since it does not use recursion The recursive fibonacci algorithm is exponen- or iteration. tially slow, since for many of the intermediate // Binet’s Formula:√ The nth fibonacci number. ( ( 5+1) )n values, the algorithm recomputes the same sub- F (n) = Round( √2 ); problems for the same fibonacci numbers repeat- 5 edly. The DP version surmounts this problem A relatively complex structure is considered next. by storing the intermediate fibonacci numbers in The straightforward recursive technique to compute a table inside the computational memory(RAM the determinant of a square matrix uses the determi- for P Cs) when they are initially computed once. nant of a square matrix 1 less in each of its 2 different As a result, it is necessary to have a one dimen- dimensions than the current square matrix. The algo- sional array F with (n + 1) elements. Each fi- rithm computes n such determinants to get the orig- bonacci number F [i] within the array is com- inal determinant. The temporal complexity function − puted iteratively, by adding together F [i 1] and for this classic version is, f(n) = n×f(n−1) + lower − F [i 2], which can be retrieved from the dy- order terms = n × (n − 1) × f(n − 2) + lower order namically filled table within the computer’s main terms = ... = n × (n − 1) × (n − 2) × ... × 1 + memory area (RAM). lower order terms = n! + lower order terms. Hence, The time complexity function, f(n) for this DP g(n) = n!, and T (n) ∈ O(n!). This is a factorial version is, f(n) = (n−1)+(n−2)+3 = 2n. The complexity class algorithm and is considerably time constant additive term 3 in f(n) is due to the fact inefficient. The alternative Gauss-Jordan Algorithm that the first 3 statements are always executed. reduces a square matrix to its diagonal form, and has Also, T (n) ∈ O(n). The Java implementation a time complexity order of, O(n3). of the above algorithm requires a dynamic array Consider matrix multiplication using dynamic ar- of size (n + 1) to hold the fibonacci numbers. rays. If P is a n×p, Q is a p×m,∑ and R is a n×m ma- × p As a result, the spatial complexity for this imple- trix, and R = P Q, then Ri,j = k=1 Pi,kQk,j com- mentation is also, S(n) ∈ O(n), which may be putes R. Here, i = 1, 2, . . . , n and j = 1, 2, . . . , m.

ISBN: 978-960-474-332-2 30 Mathematics and Computers in Business, Manufacturing and Tourism

The algorithm involves n × m × p multiplications and inside the computational memory (RAM) space. additions. The highest order term in the time com- Hence, it is not an acceptable programming practice to plexity function is 2nmp, and T (n, m, p) ∈ O(nmp). allow occasional crashes in the computational struc- If m = n = p, T (n) ∈ O(n3). Total dynamic tures. Modern compilers incorporate VMM (Vir- memory cells required = (np + mp + nm). Hence, tual Memory Management) techniques [3]. With the S(n, m, p) ∈ (np + mp + nm). If m = n = p, VMM, if the memory capacity exceeds, spooling S(n) ∈ O(n2). The space-time band-width product occurs where computation time augments drastically is, B(n, m, p) ∈ (n2mp2 + nm2p2 + n2m2p). With due to frequent page transfers to and from the hard m = n = p, B(n) ∈ O(n5). For the matrix-vector disk. It is extremely undesirable to design a program multiplication, multiplying a matrix, P of n × n di- that will eventually occupy the entire computational mension by a vector, Q of n dimension−→ produces an-−→ memory for an input leading to a program crash. It other vector, C of n dimension.∑ Hence, C = P × Q is also undesirable to design and implement a coding n × may be computed using, Ci = j=1 Pi,j Qj, where, construct that will consume an unrealistic amount of i = 1, 2, . . . , n. The temporal complexity is, T (n) ∈ computation time with a practical size of the input. O(n2). The algorithm requires 2 one dimensional ar- These two requirements are expressed jointly through rays, and one n × n, 2 dimensional array in the dy- a unique parameter known as the space-time band- namic memory area. Hence, the number of dynamic width product, denoted as B(n). For a temporal memory cells required is 2n+n2, and S(n) ∈ O(n2). complexity, T (n) ∈ O(g1(n)), and the correspond- The bandwidth product is B(n) ∈ O(n4). This re- ing spatial complexity, S(n) ∈ O(g2(n)), the space- source constraint order is 1 less than the pure matrix time bandwidth product is, B(n) ∈ O(g1(n)×g2(n)). multiplication order. Bandwidth product shows the upper-bound on computational resources requirements for a given coding 6 Conclusion construct. Following table shows the bandwidth prod- ucts for the frequently used sorting algorithms. In Ta- Both the temporal and the spatial requirements ble 3, n is the input size and k is the number of digits are of paramount importance in computer implemen- in each input. Basically, the output of a sorting algo- tation of algorithms. For frequently used coding con- rithm is just a Permutation or a rearrangement of the structs, the spatial complexity may be expressed as input elements, which is computationally significant constant, logarithmic or linear. For iterative data from computer science standpoint. structures, most have linear or O(1) space complexities. With recursion, space complexity is largely contingent upon the depth of recursion and varies from Table 3. Space-Time Bandwidth Product for structure to structure. Research has revealed that tail frequently used sorting algorithms. recursions are usually associated with a considerable space penalty due to the allocation of recursive stack Sorting Algorithm Bandwidth P roduct space. On the other hand, the techniques of computation that allow replacing tail recursions by the iter- 2 ative counterparts mandate no recursive stack or any Quick Sort O(n log2(n)) 2 other data structure that would necessitate additional Merge Sort(recursive) O(n(log2(n)) ) 2 computational memory. As a consequence, replacing Merge Sort(iterative) O(n log2(n)) tail recursions by the iterative structures significantly Bubble Sort O(n2) improves the space complexity of the resulting pro- Selection Sort O(n2) grams. In general, the implementation of recursive Insertion Sort O(n2) structures in a programming environment is a fairly Shell Sort O(n1.25) intricate process, and the time complexity of the iter- Heap Sort O(nlog2(n)) ative counterpart is usually better. 2 Though the average case behavior suffices for Radix Sort O(kn ) the vast majority of the applications, it is not recom- mended to allocate memory based on average performance. If the program passes through the worst case Using the complexity class hierarchy in section computational scenario, it will run out of memory, and 2, heap sort has the best known bandwidth prod- may crash eventually. Program crashes may have sig- uct, which is O(nlog2(n)). The algorithm also nificant ramifications that go beyond the abnormal ter- has the best temporal complexity in the order of mination of the software, and may affect the proper O(nlog2(n)). However, the best known sorting algo- functioning of other concurrent programs residing rithm with O(nlog2(n)) temporal complexity is the

ISBN: 978-960-474-332-2 31 Mathematics and Computers in Business, Manufacturing and Tourism

quick sort [4]. Hence, the algorithm with the best In future, the model presented in this paper will be bandwidth product is not necessarily the algorithm considered for real-life computation. This avenue is with the best performance. The bandwidth product not well-explored yet, and may unveil a new paradigm is just an indicator of the upper-bound on the over- in computational research. all computational resource requirement. Quite often, the space-time tradeoff is a better option for select- Acknowledgements: This research was incalcula- ing practical algorithms. If there is an abundant of bly supported by the Academic Programs at Cecil computational memory available, there is no need to College. Located in the beautiful Cecil County, Cecil compensate on time. If the fastness of execution is not College is a part of the University System of Mary- a major concern, and the memory available is less, it land (USM) in the State of Maryland. The college is is not possible to compensate on spatial complexity. a rapidly growing institution in the state. Cecil Col- For systems that handle more data than usually fits lege always advocates Faculty Scholarship and Pro- into RAM, such as databases, and the related applica- fessional Development (FSPD) to maintain a sus- tions, spatial complexity is given the prime consider- tainable faculty growth. ation. Consider a database with 10 gigabytes (GB) of records. We are required to verify whether a partic- References: ular record is within the database or not. This verifi- cation may be accomplished in two different ways. It [1] Ahmed Tarek, ”A Generalized Set Theo- is possible to load the entire 10 GB of data into main retic Approach for Time and Space Complex- memory and check the records one-by-one. This is a ity Analysis of Algorithms and Functions”, faster process with better time complexity but requires WSEAS TRANSACTIONS ON MATHEMATICS, a huge amount of RAM space (10 GB). Alternatively, Publisher: World Scientific and Engineering Academy and Society, Issue 1, Volume 6, Jan- it is possible to bring one record at a time from the − − hard disk and then check. This is a slower process uary 2007 (ISSN 1109 2769), pp. 60 68. with improved space complexity. [2] Donald E. Knuth, The Art of Computer Pro- Although Table 3 represents the space-time band- gramming, Volume 3 Sorting and Searching, width product for useful sorting algorithms, the World Second Edition, (New Jersey, Addison-Wesley, of Sorting is not done yet. Many computer researchers 1998). consider sorting to be an already solved problem. [3] Ernst L. Leiss, A Programmer’s Companion to However, useful and computationally more power- Algorithm Analysis, (Florida, CRC Press, 2006). ful new sorting algorithms are still being invented. [4] Richard F. Gilberg and Behrouz A. Forouzan, For example, Library Sort was first published only in Data Structures - A Pseudocode Approach with 2004. C + +, (California, Brooks/Cole, Thomson Spatial complexity plays a significant role in the Learning, 2001). Recursive Models of Computation. One significant aspect of recursion is the variability of the size of the recursion stack. Typically, the exact stack requirement of a recursive computation is unpredictable, since it varies from computation to computation. In most cases, it is possible to derive an upper-bound providing with a general understanding of computational resource requirements. Also, the analysis of spatial requirement deviates from that of temporal resource requisite, since memory is a re-usable resource whereas time is a consumable resource. The selection of an algorithm largely depends on the application. Some applications justify select- ing faster algorithms at the expense of more memory space consumption. This is specially true of time- sensitive applications. Also, there are applications that demand use of space efficient algorithms at the outlay of slower program execution. On the contrary, there are certain computing applications that obligate a balance between computation time and computational memory for efficient program execution.

ISBN: 978-960-474-332-2 32