算法设计与分析 Lecture 2: Asymptotic Notation

卢杨 厦门大学信息学院计算机科学系 [email protected] Asymptotic Notation

¡ Intuitively, just look at the dominant term. � � = 0.1� + 10� + 5� + 25 ¡ Drop lower-order terms 10� + 5n + 25. ¡ Ignore constant 0.1. ¡ But we can’t say that � � equals to �. ¡ It grows like �. But it doesn’t equal to �. ¡ We define asymptotic notations (渐进符号) like � � = Θ(�) to describe the asymptotic running time of an . ¡ “Asymptotic” here means “as something tends to infinity”, as we want to compare for very large �.

1 Review

Definition ¡ Useful identities for all real � > 0, � > 0, � > 0, and �, and where log � is the unique number � s.t. � = �. logarithm bases are not 1： ¡ log (��) = log � + log � ¡ Notations: , , - / ¡ log. � = �log. � ¡ lg � = log! � (binary logarithm) ) ¡ ln� = log � () ¡ log = − log � " . + . # # ¡ lg � = lg � () 0) ¡ log. � = log+ � ¡ lg lg � = lg(lg �) (composition) ¡ �%&'" , = �%&'" + ¡ Derivative: %&'# + ¡ log. � = $ %&' ( ) %&'# . ¡ ! = $( ( %* + ¡ � = �%&'" +

2

Definition 2.2 For a given complexity function �(�), � � � is the set of complexity functions � � for which there exists some positive real constant � and some nonnegative �! such that for all � ≥ �!, 0 ≤ � � ≤ ��(�). ¡ � � � is a set of functions in terms of � � that satisfy the definition. ¡ If � � = � � � , it represents that �(�) is an element in � � � . We say that � � is “big O (大O)” of � � . ¡ Strictly, we should use “∈” instead of “=”. However, it is conventional to use “=” for asymptotic notations.

3 Big O Notation

��(�) ¡ No matter how large � � = �(�(�)) � � is, it will eventually be smaller than �� � for some �(�) � and some �. ¡ Big O notation describes an upper bound. We use it to bound the worst- case running time of an algorithm on arbitrary inputs. �! � 4

Image source: 图2.2, 张德富, 算法设计与分析, 国防工业出版社, 2009. Display of Growth of Functions

5

Image source: http://bigocheatsheet.com/img/big-o-complexity-chart.png Big O Notation

Example 1 We show that � + 10� = �(�). Because, for � ≥ 1, � + 10� ≤ � + 10� = 11�, we can take � = 11 and � = 1 to obtain our result. ¡ To show a function is in big O of another function, the key is to find a specific value of � and � that make the inequality hold. ¡ More examples of functions in �(�): ¡ �, � + �, � + 1000�, 1000� + 1000�, �, �/1000, �., �/ lg lg lg �.

6 Classroom Exercise

Use the definition of Big O notation to show: Is 2 = �(2)?

7 Classroom Exercise

Proof: We prove it by contradiction. Assume there exist constants � > 0 and �! ≥ 0, such that 2"# ≤ �2#, for all � ≥ �!. Then 2"# = 2#2# ≤ �2#, 2# ≤ �. # But we can’t find any constant � is greater than 2 for all � ≥ �!. So the assumption leads to a contradiction. Then we can certify that 2"# ≠ �(2#). How about 2#$% = �(2#)?

8 Big Ω Notation

Definition 2.3 For a given complexity function �(�), Ω � � is the set of complexity functions � � for which there exists some positive real constant � and some nonnegative integer � such that for all � ≥ �, 0 ≤ ��(�) ≤ � � . ¡ Ω � � is the opposite of � � � . ¡ If � � = Ω � � , it represents that �(�) is an element in Ω � � . We say that � � is “big Ω (大Ω)” of � � .

9 Big Ω Notation

¡ No matter how small � � = Ω(�(�)) � � is, it will eventually be larger �(�) than �� � for some � and some �. ¡ Big Ω notation ��(�) describes an lower bound. We use it to bound the best-case running time of an algorithm on arbitrary inputs. �! � 10

Image source: 图2.3, 张德富, 算法设计与分析, 国防工业出版社, 2009. Big Θ Notation

Definition 2.1 For a given complexity function �(�), Θ � � is the set of complexity functions � � for which there exists some positive real constants � and � and some nonnegative integer � such that, for all � ≥ �,

0 ≤ �� � ≤ � � ≤ ��(�). ¡ If � � = Θ � � , we say that � � is “big Θ (大Θ)” or has the same order (数量级) of � � . ¡ Θ � � = � � � ∩ Ω � � .

11 Relation between Big O, Big Ω and Big Θ

¡ Now we have O, Θ, and Ω. Intuitively, they just like “≤”, “=”, and “≥” for complexity functions.

12

Image source: Figure 1.6, Richard E. Neapolitan, Foundations of Algorithms (5th Edition), Jones & Bartlett Learning, 2014 Big Θ Notation

� � = Θ(�(�)) ¡ � � = Θ � � implies � �(�) � � = � � � and # � � = Ω � � . ¡ Big Θ can also be used to �(�) bound the worst-case .

¡ For insertion sort, the worst- �"�(�) case is both Θ �! and � �! . ¡ However, we usually use Big O notation because we don’t care the best-case. �! �

13

Image source: 图2.1, 张德富, 算法设计与分析, 国防工业出版社, 2009. Properties of Asymptotic Notations

Theorem 2.1 For any two functions �(�) and �(�), �(�) = Θ � � if and only if �(�) = � � � and �(�) = Ω � � . ¡ Θ = O and Ω. Theorem 2.2

For any two functions �(�) and �(�), if � � = � � � and � � = � � � , we have � � + � � = �(max {� � , � � }). ¡ Pick the larger one.

14 Properties of Asymptotic Notations

¡ Transitivity (传递性) ¡ Reflexivity (自反性) ¡ If � � = Θ � � and � � = ¡ If � � = Θ � � . Θ ℎ � then � � = Θ ℎ � . ¡ Same for O and Ω. ¡ Ω Same for O and . ¡ Symmetry (对称性) ¡ Additivity (可加性) ¡ � � = Θ � � if and only if ¡ If � � = Θ ℎ � and � � = � � = Θ � � . Θ ℎ � then � � + �(�) = ¡ Not hold for O and Ω. Θ ℎ � . ¡ Same for O and Ω.

15 Properties of Asymptotic Notations

¡ Consider the following ordering of complexity categories: Θ lg � Θ � Θ �lg � Θ �" Θ �& Θ �' Θ �# Θ �# Θ(�!) where � ≥ � ≥ 2 and � ≥ � ≥ 1. ¡ If � � is to the left of � � in the above sequence, then � � = � � � ¡ Notice: Big Θ is a set of functions. We can’t say Θ lg � < Θ � .

16 Properties of Asymptotic Notations

Example 2 % Given � � = �(� − 1), prove that � � = Θ(�") " Proof: By the property, we first show that � � = �(�"): % % % % % � � − 1 = �" − � ≤ �" (for � = and � = 0). " " " " " ! Then we show that � � = Ω �" : % % % % % % % % � � − 1 = �" − � ≥ �" − � � = �" (for � = and � = 2). " " " " " " ( ( ! Thus � � = Θ(�").

17 Using Limit to Determine Order

¡ In addition to proving by definition, we can also use limit to get asymptotic notations. = � implies � � = Θ � � if 0 < � < ∞ �(�) lim ≠ ∞ implies � � = � � � → �(�) ≠ 0 implies � � = Ω � �

18 Using a Limit to Determine Order

Example 3 Compare the orders of growth of �(� − 1) and �. 1 �(� − 1) 1 � − � 1 1 1 lim 2 = lim = lim (1 − ) = , → � 2 → � 2 → � 2 Thus, � � − 1 = Θ(�).

19 Classroom Exercise

Compare the orders of growth of � and �, when � > � > 0

20 Classroom Exercise

Solution: � � lim = lim = 0. → � → � The limit is 0 because 0 < < 1. Thus, � = � � .

21 Using a Limit to Determine Order

() ¡ When calculating lim , how to deal with the following → () cases? lim �(�) = lim �(�) = 0 or ± ∞ → →

22 Using a Limit to Determine Order

23

Image source: https://tieba.baidu.com/p/5933589166 Using a Limit to Determine Order

L‘Hôpital’s Rule (洛必达法则) If �(�) and �(�) are both differentiable with derivatives �′(�) and �′(�), respectively, and if lim � � = lim � � = 0 or ± ∞, → → then �(�) �′(�) lim = lim , → �(�) → �′(�) whenever the limit on the right exists.

24 Using a Limit to Determine Order

Example 4 � log � 1 $ = lg � = �(�) �� � ln � because lg � �(lg �)/�� 1/(� ln2) lim = lim = lim = 0. → � → ��/�� → 1

25 Exercises

Show the correctness of the following statements. ¡ lg � = � � ¡ � = �(� lg �) ¡ � lg � = �(�) ¡ 2 = Ω(5 ) ¡ lg � = �(�.)

26 Conclusion

After this lecture, you should know: ¡ Why do we need asymptotic notation? ¡ What are the meaning of these asymptotic notations big O, big Θ, or big Ω? ¡ How to prove a complexity function is big O, big Θ, or big Ω? ¡ How to compare the order of two complexity function?

27 Homework

¡ Page 19 2.1 2.2 2.3 2.9

28 谢谢

有问题欢迎随时跟我讨论

29