Introduction, Structure, and Advanced Programming Techniques

APQE12-all

Advanced Programming in Quantitative Economics Introduction, structure, and advanced programming techniques

Charles S. Bos

VU University Amsterdam [email protected]

20 – 24 August 2012, Aarhus, Denmark

1/260 APQE12-all Outline

OutlineI Introduction Concepts: Data, variables, functions, actions Elements Install Example: Gauss elimination Getting started Why programming? Programming in theory Questions Blocks & names Input/output Intermezzo: Stack-loss Elements Droste 2/260 APQE12-all Outline

OutlineII KISS Steps Flow Recap of main concepts Floating point numbers and rounding errors Eﬃciency System Algorithm Operators Loops Loops and conditionals Conditionals Memory Optimization 3/260 APQE12-all Outline

OutlineIII Optimization pitfalls Maximize Standard deviations Standard deviations Restrictions MaxSQP Transforming parameters Fixing parameters Include packages Magic numbers Declaration ﬁles Alternative: Command line arguments OxDraw Speed 4/260 APQE12-all Outline

OutlineIV Include packages

SsfPack

Input and output

High frequency data

Data selection

OxDraw

Speed

C-Code

Fortran Code

5/260 APQE12-all Outline

Day 1 - Morning

9.30 Introduction

I Target of course I Science, data, hypothesis, model, estimation I Bit of background I Concepts of

I Data, Variables, Functions, Addresses I Programming by example

I Gauss elimination

I (Installation/getting started) 11.00 Tutorial: Do it yourself 12.30 Lunch

6/260 APQE12-all Introduction

Target of course

I Learn

I structured

I programming

I and organisation

I (in Ox or other language) Not: Just learn more syntax...

7/260 APQE12-all Introduction

What? Why? Wrong answer: For the fun of it A correct answer To get to the results we need, in a fashion that is controllable, where we are free to implement the newest and greatest, and where we can be ‘reasonably’ sure of the answers Science 1 Data

1 0 0 0 1 1 Estimation 0 1 0 0000 0 E²= m² (c²)2 Model E= m c2 Programming Hypothesis E= f(m)

8/260 APQE12-all Introduction

Aims and objectives

I Use computer power to enhance productivity

I Productive Econometric Research: combination of interactive modules and programming tools

I Data Analysis, Modelling, Reporting

I Accessible Scientiﬁc Documentation (no black box)

I Adaptable, Extendable and Maintainable (object oriented)

I Econometrics, statistics and numerical mathematics procedures

I Fast and reliable computation and simulation

9/260 APQE12-all Introduction

Options for programming

GUI CLI Program Speed QuanEcon Comment EViews + - - +/- + Black box, TS TSMod + - +/- +/- + Alternative Stata +/- + - - - Less programming Matlab + + + + +/- Expensive, other audience Gauss +/- +/- + +/- + ‘Ugly’ code, unstable S+/R +/- + + - +/- Graph +, speed - Ox + +/- + + + Links to C, ectrics Python + + + +/- Neat syntax, common C(++)/Fortran - - + ++ - Very quick, diﬃcult

Here: Use Ox as environment, apply theory elsewhere 10/260 APQE12-all Introduction

History There was once... C-Programmer Memory leaks Shell around C Matrices ...and Ox was born. More possibilities, also computationally:

Timings for OLS (30 observations, 4 regressors): 2009 Neh 2.67Ghz 64b 670.000†/sec 2008 Xeon 2.8Ghz OSX 392.000†/sec 2006 Opt 2.4Ghz 64b 340.000†/sec 2006 AMD3500+ 64b 320.000†/sec 2006 AMD3500+ 4.04 273.000†/sec Increase: 2004 AMD3500+ 3.40 218.000†/sec ≈ × 1000 in 15 years Note: 2004 PM-1200 147.000†/sec 2001 PIII-1000 104.000†/sec ≈ × 10000 in 25 years. 2000 PIII-500 60.000/sec 1996 PPro200 30.000/sec 1993 P5-90 6.000/sec 1989 386/387 300/sec 1981 86/87 (est.) 30/sec For further speed increase, use multi-cpu. See: Speed.

11/260 APQE12-all Introduction

Speed increase — but keep thinking

Z x −1 x ∼ NIG(α, β, δ, µ) P(X < x) = f (z)dz = F (x) xq = F (q) 0 x1−q + xq − 2x 1 1 S(q) = 2 0 < q < x1−q − xq 2

0.15 1.2 0.7 S x q KL x l KR x r σ σ 0.6 σ S 1 KL KR x r 0.1 E(S) E(KL) x l E(KR) x r 0.8 0.5 0.05 0.4 0.6 0.3 0 0.4 0.2 0.2 -0.05 0.1 0 0 -0.1 -0.2 -0.1 -0.15 -0.4 -0.2 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1

Direct calculation of graph: > 40 min — Pre-calc quantiles: 5 sec 12/260 APQE12-all Introduction

OxMetrics

PcGive STAMP G@RCH TSP Ox Packages A P + x12arima + SsfPack DPD, MSVAR P + PcNaive Arﬁma, etc. S Ox programs OxMetrics Ox C O interactive graphics numerical programming R data manipulation computational engine E results storage interface wrapper code editor

13/260 APQE12-all Concepts: Data, variables, functions, actions

What is programming about?

Managing DATA, in the form of VARIABLES, usually through a set of predeﬁned FUNCTIONS or ACTIONS Of central importance: Understand variables, functions at all times... So let’s exagerate

14/260 APQE12-all Concepts: Data, variables, functions, actions Variables Variable

I A variable is an item which can have a certain value.

I Each variable has one value at each point in time.

I The value is of a speciﬁc type.

I A program works by managing variables, changing the values until reaching a ﬁnal outcome [ Example: Paper integer 5 ]

15/260 APQE12-all Concepts: Data, variables, functions, actions Variables Integer

iX= 5;

I An integer is a number without fractional part, in between −231 and 231 − 1 (limits are language dependent)

I Distinguish between the name and value of a variable.

I A variable can usually change value, but never change its name

16/260 APQE12-all Concepts: Data, variables, functions, actions Variables Double

dX= 5.5;

5.5

I A double is a number with possibly a fractional part. I Note that 5.0 is a double, while 5 is an integer. I A computer is not ‘exact’, careful when comparing integers and doubles I If you add a double to an integer, the result is double (in Ox at least, language dependent) [ Example: dAdd= 1/3; dD= 0; dD= dD + dAdd; etc. ] 17/260 APQE12-all Concepts: Data, variables, functions, actions Variables String sY= "Hello world"; sX= "A";

A Hello world

I A character is a string of length one. I A string is a collection of characters. I The " are not part of the string, they are the string delimiters. I One single element of a string, sY[3] for instance, is an integer, with the ASCII value of the character. I Multiple elements of a string are a string as well, sY[0:4], also sX[0:0] is a string. [ Example: sX= "Hello world"; ] 18/260 APQE12-all Concepts: Data, variables, functions, actions Variables ‘Simple’ types

I Integer

I Double

I Character/String ‘Derived’ type

I boolean, integer 0 is FALSE, integer 1 is TRUE [ Example: print (TRUE); ]

19/260 APQE12-all Concepts: Data, variables, functions, actions Variables ‘Diﬃcult’ types

I Function

I Address

I Matrix

I Array

I File

I Object

20/260 APQE12-all Concepts: Data, variables, functions, actions Variables Function

print ("Hello world");

print()

I A function performs a certain task, usually on a (number of) variables

I Hopefully the name of the function helps you to understand its task

I You can assign a function to a variable, fnMyPrintFunction= print; [ Example: fnMyPrintFunction("Hello world"); ] 21/260 APQE12-all Concepts: Data, variables, functions, actions Variables Address, real world School of Economics and Management University of Aarhus Building 1322 DK-8000 Aarhus C

A building at the university The address

22/260 APQE12-all Concepts: Data, variables, functions, actions Variables Address

adX= &dX; adX

5.5

I Now the address is the value (of variable adX)

I Any variable has an address (&iX, &dX, &sX etc)

I Each object exists only once: Whether I use dX or what’s at the address adX, it is the same thing.

23/260 APQE12-all Concepts: Data, variables, functions, actions Variables Matrix

mX= <1, 2; 3, 4>;

1.0 2.0

3.0 4.0

I A matrix is a collection of doubles. I A matrix has two dimensions. I A matrix of size k × 1 or 1 × k we tend to call a vector, vX. I Later on we’ll see how matrix operations can simplify/speed up calculations.

24/260 APQE12-all Concepts: Data, variables, functions, actions Variables Array

aX= {"Beta", 5, <5.5>};

Beta 5 5.5

I An array is a collection of other objects. I An array itself has one dimension. I An element of an array can be of any type (integer, double, function, address, matrix, array) I An array of an array of an array has three dimensions etc. [ Example: aX= {}; ] 25/260 APQE12-all Concepts: Data, variables, functions, actions Variables File

fh= fopen("data/aex-trades-0711.csv", "r");

I A file variable ‘points to’ an opened file I This can be of use to read or write a file e.g. line-by-line I Useful for successively writing results, or handling enormous data-files [ Example: fh= fopen("data/mydata.csv", "r"); ] 26/260 APQE12-all Concepts: Data, variables, functions, actions Variables Object

hh= new house(); db= new Database();

I An object variable is an ‘object’ I It can have certain characteristics or function members, which can be changed in turn. E.g. hh.OpenWindow(); or db.GetVar("Returns"); I Useful for building higher level programs, with functionality hidden away in member functions. I Communication of research (Arfima example)

27/260 APQE12-all Concepts: Data, variables, functions, actions Variables Ox and other languages Concepts are similar

I Ox (and Gauss/Matlab/Python) have automatic typing. Use it, but carefully... I C/C++/Fortran need to have types and sizes specified at the start. More difficult, but still same concept of variables. I Precise manner for specifying a matrix differs from language to language. Ox rather similar to C in many respects I Remember: An element has a value and a name I A program moves the elements around, hopefully in a smart manner Keep track of your variables, know what is their scope

28/260 APQE12-all Concepts: Data, variables, functions, actions Variables All languages

Programming is exact science

I Keep track of your variables

I Know what is their scope

I Program in small bits

I Program extremely structured

I Think about algorithms, data storage, outcomes etc.

29/260 APQE12-all Elements

Elements to consider

I Comments: /* (block) */ or // (until end of line)

I Declarations: Up front in each routine

I Spacing

I Variables, types and naming in Ox: scalar integer iN= 20; scalar double dC= 4.5; string sName="Beta1"; matrix mX= <1, 2.5; 3, 4>; array of X aX= {1, <1>, "Gamma"}; address of variable: amX= &mX; function fnFunc = olsr; class object db= new Database();

30/260 APQE12-all Elements

Imagine elements

iX= 5 dX= 5.5 sX= "Beta"

5 5.5 Beta

mX= <1, 2; 3, 4> aX= {"Beta", 5, <5.5>} amX= &mX

amX

1.0 2.0 Beta 5 5.5

3.0 4.0

Every element has its representation in memory — no magic 31/260 APQE12-all Elements

Try out elements Listing 1: oxelements.ox #include

main() { decla, mX, sX;

a= 5; println(" Integer : ",a);

a= 5.5; println(" Double : ",a);

a= sX=" Beta "; println(" String : ",a);

a= mX= <1, 2; 3, 4>; println(" Matrix : ",a);

a= &mX; println(" Address of matrix : ",a);

a= &sX; println(" Address of string : ",a);

a= olsr; println(" Function : ",a); } 32/260 APQE12-all Elements Hungarian notation Hungarian notation preﬁxes

preﬁx type example i integer iX b boolean (f is also used) bX d double dX m matrix mX v vector vX s string sX fn Function fnX a array oraddress aX as array ofstrings asX am array ofmatrices amX c class object variable cX m classmember variable m mX g external variable withglobal scope g mX s static external variable (ﬁle scope) s mX

Use them everywhere, always. Possible exception: Counters i, j, k etc.

33/260 APQE12-all Elements Hungarian notation Hungarian 2 Ox does not force Hungarian notation: Correct but very ugly is

Listing 2: oxnohun.ox #include main() { decl sX, iX;

iX=" Hello "; sX= 5; } Instead, always use

Listing 3: oxhun.ox #include main() { decl sX, iX;

sX=" Hello "; iX= 5; }

34/260 APQE12-all Install

Installation

1. Install the appropriate version (academic/professional), http://www.doornik.com, for Ox and possibly OxMetrics 2. Make the Ox documentation the homepage in your browser (c:\program files\oxmetrics6\ox\doc\index.html) 3. Install the necessary tools for OxEdit, if needed Optional steps: I Continue with downloading and installing extra packages ssfpack, arfima, gnudraw, dpd etc. into the Ox directory c:\program files\oxmetrics6\ox\packages\ssfpack etc, each in its own subdirectory below ox\packages.

35/260 APQE12-all Install

Installation (advanced)

What if:

I No graphics, no OxMetrics license Then:

I Install GnuDraw package with Ox, and

I Install GnuPlot (google it for a download) in c:\program files\gnuplot

36/260 APQE12-all Example: Gauss elimination

Programming by example

I Enough theory

I Example: How to solve a system of linear equations

I Goal: Simple situation, program to solve it

I Broad concepts, details follow

37/260 APQE12-all Example: Gauss elimination

Setup: Linear system Solve for x: A x = b, with       a11 a12 ··· a1n x1 b1  0 a22 ··· a2n x2 b2 A =   , x =   b =    . .. .   .   .   . . .   .   .  0 ··· 0 ann xn bn Solution: xn = bn/ann   X xi = bi − aij xj  /aii , i = n − 1, .., 1 j>i

I.e.: Start at the end, solve backwards.

But ... only works for upper triangular A... 38/260 APQE12-all Example: Gauss elimination

Elimination Hence: Create triangular matrix...

2 1 x 1 2 1 x 1 1 = ⇔ 1 = 4 6 x2 4 0 4 x2 2

Subtract multiple ajk /akk times equation k from rows (k) j = k + 1, ..., n, such that ajk ≡ 0. Note: The x’s don’t change, only elements of A and b. Extended matrix:   a11 ······ a1n b1  .. . .  a21 . . .  (A, b) =    . .. . .   . . . .  an1 ······ ann bn

39/260 APQE12-all Example: Gauss elimination

Example elimination

 6 −2 2 4 | 16   12 −8 6 10 | 26  [A|b] =    3 −13 9 3 | −19 −6 4 1 −18 | −34 6 −2 2 4 | 16  iteration 1 (1) 0 −4 2 2 | −6  ⇔ [A|b] =   0 −12 8 1 | −27 0 2 3 −14 | −18

Let’s concentrate on one row at a time: How to eliminate the row starting with 12?

(See ge0.ox)

40/260 APQE12-all Example: Gauss elimination

Program by Example 0

I Use commenting

I One main function: main() {}

I Declarations on top (...)

I Get the matrices, mA= <1, 2; 3, 4>;

I Concatenate, mAB= mA ~ vB;

I Debug → println() Recognize Magic Numbers, initial settings

PbE 1: Eliminate a row

I What row/column are we working with? Start counting at 0...

I Calculate multiplicity

I Subtract a row at a time

41/260 APQE12-all Example: Gauss elimination

PbE 2: Eliminate a row in a function

As we might want to eliminate more rows, it could be programmed as a separate function...

I Function header: Deﬁne what goes in/out

I Use commenting

I First use of address amAB= &mAB; PbE 3: Eliminate multiple rows

I Use a loop around the function, for (start condition; check; increment)

42/260 APQE12-all Example: Gauss elimination

PbE 4: Eliminate multiple columns PbE 4: Eliminate multiple columns

I Use a loop around the loop. What columns should be eliminated? PbE 5: Use another function

I Use a function to eliminate a column

I Call the function multiple times from the loop

Resulting program:

I Clean

I Readable chunks

I Debugging was done step by step, function/action at a time

I In future, functions are easily re-utilizable.

43/260 APQE12-all Getting started

Chapter 1: Getting started Exercise: 1. Copy the file /samples/myfirst.ox to your personal directory. 2. Open the file in OxEdit (e.g. Windows Explorer, walk there, right mouse button, Send To - OxEdit) 3. Run the program (through Modules - Run - Ox) (If there is no Ox option under the Run menu, load the .tool file from the students directory, using Tools -

Add/remove modules - Load from) Output

Ox version 5.10 (Linux_64/MT)(C)J.A. Doornik, 1994-2008 two matrices 2.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 1.0000

0.0000 0.0000 0.0000 1.0000 1.0000 1.0000

44/260 APQE12-all Getting started

Using OxEdit One tab has program Running the program puts output in separate file/sheet Errors in code can appear in output file Workspace indicates opened files

45/260 APQE12-all Getting started

Type of errors 1. Compilation errors: Like the above, error in the syntax of Ox Listing 4: myfirst err.ox print" two matrices ", m1, m2); // gives compile-time error ------Ox version 5.10 (Linux_64/MT)(C)J.A. Doornik, 1994-2008 myfirst_err.ox (12):’ ;’ expected but found’ ’ myfirst_err.ox (12):’ ;’ expected but found’ )’ myfirst_err.ox (12):’ )’ out of place 2. Runtime errors: Impossible computations or commands Listing 5: myfirst err.ox print(" product of two matrices ", m1* m2); // gives run-time error ------Ox version 5.10 (Linux_64/MT)(C)J.A. Doornik, 1994-2008 ... Runtime error:’ matrix [3][3] * matrix [2][3] ’ bad operand Runtime error occurred in main(14), call trace: myfirst_err.ox (14): main One error can lead to multiple messages: Start solving first in list. 46/260 APQE12-all Getting started

Chapter 2: Syntax - Comments

/* This is standard comment, which/* may be nested*/ . */ declx; // declare the variablex Use them well, use them extensively, use them consistently

47/260 APQE12-all Getting started

/* ** olsc(const mY, const mX, const amB) ** ** Purpose: ** PerformsOLS, expecting the data in columns. ** ** Inputs: ** mY iTx iN matrix of regressorsY ** mX iTx iK matrix of explanatory variablesX ** ** Outputs: ** amB address of iKx iN matrix with iN sets ofOLS coefficients ** ** Return value: ** integer, 1: success, 2: rescaling advised, ** -1:X’X is singular, -2: combines2 and -1. ** ** Example: ** ir= olsc(mY, mX,&mB); ** ** Last changed ** 21-04-96(Marius Ooms): made documentation ** 06-08-09(Charles Bos): adapted documentation */ Use explanation, consistently, before every function, detailing name, purpose, inputs, outputs, return value (and possibly date, author, once per ﬁle)

48/260 APQE12-all Getting started

Program layout A minimal complete program is: Listing 6: oxtut2b.ox #include

main() { println(" Hello world "); } Contains: 1. Include statement, to deﬁne all standard functions in Ox; between < and > to indicate oxstd.h is an intrinsic part of Ox 2. One function header, called main, taking no arguments () 3. Function body for main(), enclosed in {}, with a println statement Note: Syntax terribly similar to C or Java. 49/260 APQE12-all Getting started

Statements Listing 7: oxtut2c-hun.ox #include

main() { decl iN, dSigma, mX, vBeta, vEps;

iN = 4; dSigma = 0.25; mX=1 ∼ ranu(iN, 2); vBeta = <1; 2; 3>;

vEps= dSigma* rann(iN, 1);

print(" x", mX," beta ", vBeta," epsilon ", vEps); } (note: Stick to Hungarian, don’t follow the Introduction to Ox literally here) I Declaration: Automatic typing I Assignment: Integer, double, matrix-function, matrix-constant, function result. I Print statement 50/260 APQE12-all Getting started

Day 1 - Afternoon

13.30 Structuring your thoughts

I What is programming? I Preparation of a program 14.30 Tutorial: Do it yourself

I Exercise to hand in I Work through ‘Introduction to Ox Ch 1-5’

51/260 APQE12-all Why programming?

Repeat: What? Why? Wrong answer: For the fun of it A correct answer: To get to the results we need, in a fashion that is controllable, where we are free to implement the newest and greatest, and where we can be ‘reasonably’ sure of the answers

Science 1 Data

1 0 0 0 1 1 Estimation 0 1 0 0000 0 E†= m† (c†)2 Model E= m c2 Programming Hypothesis E= f(m) 52/260 APQE12-all Programming in theory Questions Programming in Theory

Plan ahead

I Research question: What do I want to know?

I Data: What inputs do I have?

I Output: What kind of output do I expect/need? I Modelling:

I What is the structure of the problem? I Can I write it down in equations?

I Estimation: What procedure for estimation is needed (OLS, ML, simulated ML, GMM, nonlinear optimisation, Bayesian simulation, etc)?

53/260 APQE12-all Programming in theory Blocks & names Closer to practice Blocks:

I Is the project separable into blocks, independent, or possibly dependent?

I What separate routines could I write?

I Are there any routines available, in my own old code, or from other sources?

I Can I check intermediate answers?

I How does the program ﬂow from routine to routine? ... names:

I How can I give functions and variables names that I am sure to recognise later (i.e., also after 3 months)? Use (always) Hungarian notation

54/260 APQE12-all Programming in theory Input/output Even closer to practice

Deﬁne, on paper, for each routine/step/function:

I What inputs it has (shape, size, type, meaning), exactly

I What the outputs are (shape, size, type, meaning), also exactly...

I What the purpose is... Also for your main program:

I Inputs can be magic numbers, (name of) data ﬁle, but also speciﬁcation of model

I Outputs could be screen output, ﬁle with cleansed data, estimation results etc. etc.

55/260 APQE12-all Intermezzo: Stack-loss

Intermezzo: Stack-loss data

21 Observations on 4 variables (source: Brownlee (1965)). It concerns the oxidation of ammonia to nitric acid, as a function of air ﬂow, water temperature, and acid concentration. See also: Justel & Pe˜na(1996) Regression model - outliers? - OLS - standard deviation - R2 Data:

80 80 75 62 62 62 ... 27 27 25 24 22 23 ... 89 88 90 87 87 87 ... 42 37 37 28 18 18 ...

56/260 APQE12-all Intermezzo: Stack-loss

Intermezzo II: What to do?

Stack loss x Air Flow Stack loss x Water Temp Stack loss x Acid Conc 45 45 45 40 0 40 0 40 0 35 2 1 35 2 1 35 1 2 30 3 30 3 30 3 25 25 25 20 4567 20 4 5 67 20 45 67 15 199108 20 15 910 1920 8 15 9198 10 20 121311 111213 12 11 13 10 14161718 10 14161718 10 16 1718 14 5 15 5 15 5 15 50 55 60 65 70 75 80 16 18 20 22 24 26 28 70 75 80 85 90 95 Data: Stack loss = f (Air ﬂow, Water temperature, Acid concentration) As a starter: Look at the data. Check which series is which, ranges, means, outliers, transformations etc.

Conclusion: Regression, indeed? 57/260 APQE12-all Intermezzo: Stack-loss

Intermezzo III: What to do? 2 Model yi = Xi β + ui , ui ∼ N (0, σ ) Estimation βˆ = (X 0X )−1X 0y Data yi = Line 0 of data in data/stackloss.mat, running from 7-42 Xi = Lines 1-3 of data; and constant!

Ugly programming: Listing 8: stack/stackols ugly #include main() { decly= <42;37;37;28;18;18;19;20;15;14;14;13;11;12; 8; 7; 8; 8; 9;15;15>; declx= <1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1;80,80,75,62,62,62,62, 62,58,58,58,58,58,58,50,50,50,50,50,56,70;27,27,25,24,22,23,24,24,23,18, 18,17,18,19,18,18,19,19,20,20,20;89,88,90,87,87,87,93,93,87,80,89,88,82, 93,89,86,72,79,80,82,91>’ ; print ((x’x)^(-1)*x’ y); }

Errors... 58/260 APQE12-all Intermezzo: Stack-loss

Intermezzo III: Nicer programming ** StackOls ** ** Purpose: ** Estimatea regression model on the stackloss data set ** ** Inputs: ** The program expects the file data/stackloss.mat to contain ** the data, with size information ** ** Author: ** Charles Bos ** ** Date: ** 16/2/2005 */ #include

main() { decl mStackloss, vY, mX, ir, vBeta;

mStackloss= loadmat(" data / stackloss . mat "); // Load the data vY= mStackloss[3][]; // Read out row3 mX= mStackloss[0:2][]; // Read out row 0-2 mX= 1|mX; // Appenda constant ir= olsr(vY, mX,&vBeta); // RunOLS on rows

print(" Ols estimates of Beta : ", vBeta); } 59/260 APQE12-all Elements

Elements to consider

I Explanation: Be generous (enough)

Listing 9: stack/stackols.ox /* **NAME ** ** Purpose: ** Short description of main idea ** ** Inputs: ** Clearly indicate what should have been prepared ** ** Author: ** Who amI ** ** Date: ** When didI write this version... */ #include

main() { ... }

60/260 APQE12-all Elements

Elements to consider II

I Explanation: Be generous (enough)

I Initialise from main

Listing 10: stack/stackols.ox /* ... */ #include

main() { // Initialisation }

61/260 APQE12-all Elements

Elements to consider III

I Explanation: Be generous (enough)

I Initialise from main

I Then do the estimation

Listing 11: stack/stackols.ox /* ... */ #include

main() { // Initialisation

// Estimation }

62/260 APQE12-all Elements

Elements to consider IV

I Explanation: Be generous (enough) I Initialise from main I Then do the estimation I ... and give results

Listing 12: stack/stackols.ox /* ... */ #include

main() { // Initialisation

// Estimation

// Results } NB: These steps are usually split into separate functions 63/260 APQE12-all Droste

The ‘Droste eﬀect’

I The program performs a certain function I The main function is split in three (here) I Each subtask is again a certain function that has to be performed

Apply the Droste eﬀect:

I Think in terms of functions

I Analyse each function to split it

I Write in smallest building blocks

64/260 APQE12-all Droste

Preparation of program What do you do for preparation of a program? 1. Turn oﬀ computer 2. On paper, analyse your inputs 3. Transformations/cleaning needed? Do it in a separate program... 4. With input clear, think about output: What do you want the program to do? 5. Getting there: What steps do you recognise? 6. Algorithms 7. Available software/routines 8. Debugging options/checks Work it all out, before starting to type... KISS 65/260 APQE12-all KISS

KISS

Keep it simple, stupid Implications:

I Simple functions, doing one thing only

I Short functions (one-two screenfuls)

I With commenting on top

I Clear variable names (but not too long either)

I Consistency everywhere

I Catch bugs before they catch you Reference: http://kerneltrap.org/files/Jeremy/CodingStyle.txt

66/260 APQE12-all KISS

KISS: Example

Remember Gauss elimination:

I Eliminate a matrix ≡

I (K − 1)× [eliminate a column ≡

I (K − k)× [eliminate a single row ≡

I subtracting f times row k]] Separate actions, separately programmed, each debugged separately

67/260 APQE12-all KISS

Day 2 - Morning

9.00L Structuring

I Recursive programming I Building blocks I Declarations/data/actions/output I Revise:

I Passing data back and forth 10.30P Tutorial

I Addresses I Minimal blocks 12.00 Lunch

68/260 APQE12-all KISS

Reprise: What? Why? Wrong answer: For the fun of it A correct answer To get to the results we need, in a fashion that is controllable, where we are free to implement the newest and greatest, and where we can be ‘reasonably’ sure of the answers Science 1 Data

1 0 0 0 1 1 Estimation 0 1 0 0000 0 E²= m² (c²)2 Model E= m c2 Programming Hypothesis E= f(m)

69/260 APQE12-all Steps

Step 1: Analyse the data

I Read the original data file I Make a first set of plots, look at it I Transform as necessary (aggregate, logs, first differences, combine with other data sets) I Calculate statistics I Save a file in a convenient format for later analysis

P1 Data 1 1 0 0 1 1 0 0 0 0 0 Estimation 111 E²= m² (c²)2 0 Model E= m c2 Programming Hypothesis E= f(m)

savemat(" data / fx9709 . fmt ", mX); savemat(" data / fx9709 . in7 ", vDay∼mX,{" Date "," UKUS "," EUUS "," JPUS " });

70/260 APQE12-all Steps

Step 2: Analyse the model

I Can you simulate data from the model?

I Does it look ‘similar’ to empirical data?

I Is it ‘the same’ type of input?

Data 1 0 1 1 1 0 10 1 Estimation 1 01 1 E²= m² (c²)2 1 0 Model E= m c2 Programming Hypothesis P2 E= f(m)

mU= rann(4, iT); // Log-returnsUS,UK,EU,JP mF= cumulate(mU’ )’; // Log-currencies mFX= exp(mF[1:][] - mF[0][]); //FXUKEUJP

71/260 APQE12-all Steps

Step 3: Estimate the model

I Take input (either empirical or simulated data)

I Implement model estimation

I Prepare useful outcome

1 Data 1 0 0 1 0 1 1 0 Estimation 0 0 1 ² ² ² 2 1 E = m (c ) 1 0 Model P3 E= m c2 Programming Hypothesis E= f(m)

72/260 APQE12-all Steps

Step 4: Extract results

I Use estimated model parameters

I Create tables/graphs

I Calculate policy outcome etc.

Data 0 01 1 1 Estimation 0 1 00 1 ² ² ² 2 0 E = m (c ) 0 Model 1 Results E= m c2 Programming P4 Hypothesis E= f(m)

73/260 APQE12-all Steps

Result of steps main() { decl sData, asIn, vYears, vDay, mRet, vP, vS, dLnPdf, mFilt, ir;

// Prepare’magic numbers’ sData=" data / fx9709 . in7 "; // Or use"data/sim9709.in7"; asIn= {" UKUS "," EUUS "," JPUS "}; vYears= <1997, 2009>;

// Perform analysis, in steps} Initialise(&vDay,&mRet, sData, asIn, vYears); ir= Estimate(&vP,&vS,&dLnPdf, mRet, asIn); ExtractResults(&mFilt, vP, vS, mRet); Output(vP, vS, mRet, mFilt, ir); }

I Short main I Starts oﬀ with setting items that might be changed: Only up front in main (magic numbers) I Debug one part at a time! I Easy for later re-use, if you write clean small blocks of code I Input to estimation program is prepared data ﬁle, not raw

data. 74/260 APQE12-all Flow

Ch 5: Program ﬂow Last main chapter on low-level Ox language

I Read your program. There is only one route the program will take. You can follow it as well.

I Statements are executed in order, starting at main()

I A statement can call a function: The statements within the function are executed in order, until encountering a return statement or the end of the function

I A statement can be a looping or conditional statement, repeating or skipping some statements. See below.

I (The order can also be broken by break, continue or goto statements. Don’t use, ugly.) And that is all, any program follows these lines. (Sidenote: Objects etc)

75/260 APQE12-all Flow

Flow 2: Reading easily

As a general hint: I Main ﬁle:

I #include routines (see later) I Contains only main() I Preferably only contains calls to routines (Initialise, Estimate, Output)

I Each routine: Maximum 30 lines / one page. If longer, split!

76/260 APQE12-all Recap of main concepts Functions All work in functions All work is done in functions

Listing 13: recap1.ox #include

main() { decl dX, dX2;

dX= 5.5; dX2= dX^2; println(" The square of ", dX," is ", dX2); } According to the function header main() the function main takes no arguments. This function uses only println as a function, rest of the work is done locally.

77/260 APQE12-all Recap of main concepts Functions Squaring and printing Use other functions to do your work for you

printsquare(const dIn) { decl dIn2; dIn2= sqr(dIn); println(" The square of ", dIn," is ", dIn2); }

main() { decl dX;

dX= 5.5; printsquare(dX);

printsquare(6.3); } Here, printsquare does not give a return value, only screen output. printsquare takes in one argument, with a value locally called dIn. Can either be a true variable (dX), a constant (6.3), or even the outcome of a calculation (dX-5). 78/260 APQE12-all Recap of main concepts Return statement return

Alternatively, use return to give a value back to the calling function (as e.g. the ones() function also gives a value back).

Listing 14: return.ox #include

onesL(const iR, const iC) { decl mX; mX= zeros(iR, iC) + 1; return mX; }

main() { decl mX;

mX= onesL(2, 4); print(" Ones matrix , using local function onesL : ", mX); }

79/260 APQE12-all Recap of main concepts Return statement Indexing A matrix consists of multiple doubles, a string of multiple characters, an array of multiple elements. Get to those elements by using indices (starting at 0):

index(const mA, const sB, const aC) { println(" Element [0][1] of ", mA," is ", mA[0][1]); println(" Elements [0:4] of ’", sB," ’ are ’", sB[0:4]," ’"); println(" Element [4] of ’", sB," ’ is ASCII number ", sB[4]); println(" Element [1] of ", aC," is ’", aC[1]," ’"); }

main() { decl mX, sY, aZ;

mX= rann(2, 3); sY=" Hello world "; aZ= {mX, sY, 6.3};

index(mX, sY, aZ); } Check out how sB[i:i] is a string, and sB[i] the ASCII-number

representing the letter (65=A, 66=B, ...) 80/260 APQE12-all Recap of main concepts Scope Scope Each variable has a scope, a part of the program where it is known. printsquare(const dIn) { decl dIn2; dIn2= sqr(dIn); println(" The square of ", dIn," is ", dIn2); } main() { decl dX; printsquare(dX); printsquare(6.3); } Possibilities: 1. Local declarations decl dX, or decl dIn2: Only known in the present block, until closing parenthesis of the function. 2. Function arguments: Local name for argument to function, in order. Compare local name (dIn) to call (dX, 6.3). 3. [Later] Global variables static decl s_vY, s_mX: Only used in special situations, with great care; these have full scope for the remainder of the ﬁle/program. 81/260 APQE12-all Recap of main concepts Arrays and such Arrays and multiple assignment

Not speciﬁc to functions are arrays and multiple assignments:

Listing 15: multassign.ox #include

main() { decl aiRC, iR, iC;

aiRC= {2, 4}; // Create an array with two integers [iR, iC]= aiRC; // Assign the two elements of the array

// Or usea function, assigning the array of returns [iR, iC]= SomeFunctionReturningArrayOfSizeTwo(); }

82/260 APQE12-all Recap of main concepts Constant arguments Arguments cannot be changed Arguments to a function cannot be changed in a lasting way. After returning from the function, the old value is back. Listing 16: changeme.ox #include

changemeerror(const dA) { dA= 5; }

changemenoerror(dA) { dA= 5; }

main() { decl dX;

dX= 3; changemeerror(dX); changemenoerror(dX); println(" Result : ", dX); }

83/260 APQE12-all Recap of main concepts Addresses Before the addresses

If you prefer, stop here for the moment...

Use constant arguments, return values using return statement. Everything could be written this way.

84/260 APQE12-all Recap of main concepts Addresses Those addresses again...

As I cannot change the argument itself, pass along the (ﬁxed) address of a variable:

Listing 17: changemedef.ox changemedef(const adX) { adX[0]= 7; // Do not change the address, but the value at the address }

main() { decl dX;

dX= 3; println(" Value before ChangeMeDef : ", dX); changemedef(&dX); println(" Value after ChangeMeDef : ", dX); }

85/260 APQE12-all Recap of main concepts Indexing Addresses and indexing

Indexing works with one index at a time. If you have the address of an array with a matrix in 3rd place, of which you want to change element [6][2], just check the indexing carefully.

Listing 18: index.ox main() { decl mX, aMany, aaMany;

mX= rann(7, 4); // Matrix aMany= {45, olsc, mX, 4.9}; // Array with mX and others aaMany= &aMany; // Address of array

aaMany[0][2][6][2]= 10000; print(" Address : ", aaMany); // Print address, with underlying array print(" Array : ", aaMany[0]); // Print array at address }

86/260 APQE12-all Recap of main concepts Indexing Day 2 - Afternoon

13.00L Background of computations

I Floating point numbers and rounding errors I Compilers and CPUs I Computing environment at Aarhus University 14.30P Tutorial

I Simulate data duration model I Apply concepts of the day I Think of rounding errors 16.00 End

87/260 APQE12-all Floating point numbers and rounding errors

Precision

Not all numbers are made equal... Example: What is 1/3 + 1/3 + 1/3 + ...?

Listing 19: precision/onethird.ox main() { decli,j, dD, dSum;

dD= 1/3; dSum= 0.0; for(i= 0;i < 10; ++i) for(j= 0;j < 3; ++j) { print(dSum ∼i∼(dSum-i)); dSum+= dD; // Successively adda third } } See outcome: It starts going wrong after 16 digits...

88/260 APQE12-all Floating point numbers and rounding errors

Decimal or Binary

1-to-10 (Source: XKCD, http://xkcd.com/953/)

89/260 APQE12-all Floating point numbers and rounding errors

Representation

I Integers are represented exactly using 4 bytes/32 bits, in range [INT_MIN, INT_MAX]= [-2147483648, 2147483647] I Doubles are represented in 64 bits. This gives a total of 264 ≈ 1.84467 × 1019 diﬀerent numbers that can be represented. How?

Double ﬂoating point format (Graph source: Wikipedia) Split double in I Sign (one bit) I Exponent (11 bits)

I Fraction or mantissa (52 bits) 90/260 APQE12-all Floating point numbers and rounding errors

Double representation

 sign 1−1023  (−1) × 2 × 0.mantissa if exponent=0x.000 x = (−1)sign × ∞ if exponent=0x.7ﬀ  (−1)sign × 2exponent−1023 × 1.mantissa else Note: Base-2 arithmetic

Sign Expon Mantissa Result 0 0x.3ﬀ 0000 0000 0000 −10 × 2(1023−1023) × 1.0 = 0 0 0x.3ﬀ 0000 0000 0001 −10 × 2(1023−1023) × 1.000000000000000222 = 1.000000000000000222 0 0x.400 0000 0000 0000 −10 × 2(1024−1023) × 1.0 = 2 0 0x.400 0000 0000 0001 −10 × 2(1024−1023) × 1.000000000000000222 = 2.000000000000000444 Bit weird... 91/260 APQE12-all Floating point numbers and rounding errors

Consequence: Addition

Let’s work in Base-10 arithmetic, assuming 4 signiﬁcant digits:

Sign Exponent Mantissa Result x + 4 0.1234 0.1234 × 104 1234 + 3 0.5670 0.5670 × 103 567

What is the sum? Sign Exponent Mantissa Result x + 4 0.1234 0.1234 × 104 1234 + 4 0.0567 0.0567 × 104 567 + 4 0.1801 0.1801 × 104 1801

Shift to same exponent, add mantissas, perfect

92/260 APQE12-all Floating point numbers and rounding errors

Consequence: Addition II Let’s use dissimilar numbers: Sign Exponent Mantissa Result x + 4 0.1234 0.1234 × 104 1234 + 1 0.5670 0.5670 × 101 5.67 What is the sum? Sign Exponent Mantissa Result x + 4 0.1234 0.1234 × 104 1234 + 4 0.000567 0.05 × 104 5 + 4 0.1239 0.1239 × 104 1239 Shift to same exponent, add mantissas, loose precision... Further consequence: Add numbers of similar size together, preferably! In Ox/C/Java/Matlab/Octave/Gauss: 16 digits (≈ 52 bits) available instead of 4 here 93/260 APQE12-all Floating point numbers and rounding errors

Consequence: Addition III Check what happens in practice:

Listing 20: precision/accuracy.ox main() { decl dA, dB, dC;

dA= 0.123456 * 10^0; dB= 0.471132 * 10^15; dC= -dB;

println(" a: ", dA," , b: ", dB," , c: ", dC); println(" a + b + c: ", dA+dB+dC); println(" a + (b + c): ", dA+(dB+dC)); println(" (a + b) + c: ",(dA+dB)+dC); } results in

Ox Professional version 6.00 (Linux_64/MT)(C)J.A. Doornik, 1994-2009 a: 0.123456,b: 4.71132e+14,c: -4.71132e+14 a+b+c: 0.125 a+(b+c): 0.123456 (a+b) +c: 0.125

94/260 APQE12-all Floating point numbers and rounding errors

Other hints

I Adding/subtracting tends to be better than multiplying P Q I Hence, log-likelihood log Li is better than likelihood Li I Use true integers when possible

I Simplify your equations, minimize number of operations

I Don’t do x = exp(log(z)) if you can escape it (Now forget this list... use your brains, just remember that a computer is not exact...)

95/260 APQE12-all Eﬃciency

On architecture, algorithms, languages, and machines

Why do we program (repeat)? To get to the results we need, in a fashion that is controllable, where we are free to implement the newest and greatest, and where we can be ‘reasonably’ sure of the answers What is important here? 1. To get correct code (≈ maintainable, clear, adjustable) 2. To get eﬃcient code (≈ quick?) Correct code: See rest of course

96/260 APQE12-all Eﬃciency

Eﬃciency II Eﬃcient code: Depends on 1. system (processor, memory size/structure, drives, network, operating system) 2. language 3. algorithm 4. more...? Use an example: What is the sum of all returns on the SP500 stock index, over 20 years?

N X S = rt t=1

97/260 APQE12-all Eﬃciency System Eﬃciency: System

What do you prefer: 1. Old Apple II (1Mhz, 64KB) 2. Older Power Mac (2.5GHz, PowerPC processor) 3. This laptop (C2Duo 1.4Ghz, 1GB) 4. Home machine (I7 quadcore 2.66Ghz, 6GB) 5. Work machine (Nehalem dual quadcore 2.66Ghz, 12GB) 6. SARA Lisa supercomputer (536 dual quadcores, loads of memory) Diﬀerence?

98/260 APQE12-all Eﬃciency System Eﬃciency: System II

Diﬀerence: 1. Apple does not even have the memory to store the data. Read from tape? 2. Mac: PowerPC had low-level possibilities to sum vectors of numbers quickly. 3. C2D laptop: Not bad, fraction of a second 4. Core I7: Better memory management in CPU, quicker 5. 2xI7: Can one use 8 cores for the summation? 6. SARA: Can one use 536 x 2 x 4 cores for summing 1000 numbers? In general: Recent ≡ better... More memory ≡ better... More cores/CPUs 6≡ better necessarily

99/260 APQE12-all Eﬃciency System CPU

Figure : Itanium CPU architecture (Source: Wikipedia)

I Hard drive → memory → CPU I Efficiency differs per CPU architecture I Does your data fit in memory/cache? Or should you go back to hard drive to reload it I Communication between parts of system: Speed matters I ... 100/260 APQE12-all Efficiency System Multi-core?

N N/4 N/2 3N/4 N X X X X X S = rt = rt + rt + rt + rt t=1 t=1 t=N/4+1 t=N/2+1 t=3N/4

= S1 + S2 + S3 + S4 Use 4 cores for summation? 1. Decide on how to split the problem 2. Send each core the right amount of data 3. Wait for the answers to come back

4. Sum S = S1 + S2 + S3 + S4 Drawback: Overhead in communication between cores/nodes, not always faster... (GPU, supercomputer, MPI, Ox7, ...) 101/260 APQE12-all Eﬃciency Algorithm Algorithm

inti, iN; //C code CFORTRANCODE double dS; INTEGERi, iN REAL dS iN= ...; dS= 0.0; iN= ... for(i= 0;i< iN; ++i) dS= 0.0 dS+= vR[0][i]; DO 10i= 1, iN dS= dS+ vR(0,i) 10CONTINUE

decli, dS, iN; // Ox code decl dS; // Ox code, using function iN= sizerc(vR); dS= 0; dS= sumc(vR); for(i= 0;i< iN; ++i) dS+= vR[i];

I Fortran or C would be quickest, as compiled code can be executed directly by CPU

I Ox code is similar to C, but far slower: Each command has to be ‘translated’ to executable code, so even a loop takes time...

I Ox version using sumc is virtually as fast as Fortran or C

102/260 APQE12-all Eﬃciency Algorithm Algorithm II Think... Where do returns come from? First diﬀerence of log prices

pt = log Pt

rt = pt − pt−1 What does this imply for the sum of the returns? X X S = rt = (pt − pt−1) t

= (p1 − p0) + (p2 − p1) + ··· + (pN − pN−1) = pN − p0

I Quicker algorithm I This does more than quicker computer, quicker language... I Think/measure where your program is slow: Work on bottlenecks ﬁrst 103/260 APQE12-all Eﬃciency Algorithm Algorithm III

Good algorithms: Where to ﬁnd?

I BLAS library (C/Fortran)

I LAPack library (C/Fortran)

I Within higher-level languages (Matlab/Octave/Ox/Gauss) Examples: Regression, matrix inversion, random number generator, optimization... Easy to code little robustly, hard to get right...

104/260 APQE12-all Eﬃciency Operators Low-level ‘algorithms’: Operators

Not all operators are equal: Speed Operation Example Fast Integer addition and subtraction I+J; I-J Floating point addition and subtraction A+B; A-B Int multiply, fl. point multiply/divide I*J, A*B, A/B . . Integer divide I/J Exponentiation to a positive int. constant A^2 Exponentiation to a positive int. variable AÎ Slow Exponentiation to a floating point variable A^B What is the difference?

for(dI= 0.0; dI< iN; ++dI) for(i= 0;i< iN; ++i) dosomething(); dosomething();

105/260 APQE12-all Eﬃciency Operators Operators, do’s and don’ts

Looking only at memory use and speed of operators: Don’t Do A/2.0 0.5*A A^2 A * A B (E + F) - C*(E+F) (B - C) * (E + F) A^0.5 sqrt(A) B/C/D/E B/(C*D*E) T1 = B + C, T2 = D*E, A = T1 + T2 A= B+C+D*E T1 = X + Y, A= T1 + log(T1) A= (X + Y) + log(X + Y) Note: Think also of clarity of your program...

106/260 APQE12-all Eﬃciency Loops Loops and execution: Gather

Loops take most time (even empty loops take time!) Preferably use one loop, gather statements together

// Don’t // Do dS= dQ= 0; dS= dQ= 0; for(i= 0;i< iN; ++i) for(i= 0;i< iN; ++i) dS+= vR1[i]; { for(i= 0;i< iN; ++i) dS+= vR1[i]; dQ+= vR2[i]; dQ+= vR2[i]; }

107/260 APQE12-all Eﬃciency Loops and conditionals Loops and execution: If-then-else

Conditional constructs are hard for CPU, take if-then-else outside of loop if possible

// Don’t // Do dS= 0; dS= 0; for(i= 0;i< iN; ++i) if(iDay == 4) if(iDay == 4) for(i= 0;i< iN; ++i) dS+= vR1[i]; dS+= vR1[i]; else else dS+= vR2[i]; for(i= 0;i< iN; ++i) dS+= vR2[i];

108/260 APQE12-all Eﬃciency Conditionals If-then-else

Test for most common condition ﬁrst

// Don’t // Do if(iDay == 4) if(iDay != 4) dS= sumr(vR1); dS= sumr(vR2); else else dS= sumr(vR2); dS= sumr(vR1);

109/260 APQE12-all Eﬃciency Memory Memory and speed Should you care about memory? Yes and no... Make some speed diﬀerence. 0 1 2 Matrices A = is stored: 3 4 5

I Column-wise: Fortran, Matlab

I Row-wise: C

I Row-wise (with row pointers): Ox

r[0] r[1]

0 3 1 4 2 5 0 1 2 3 4 5 0 1 2 3 4 5

Fortran/Matlab mA C mA Ox mA

110/260 APQE12-all Eﬃciency Memory Function calls When you call a function:

I Local variables declare their memory for every function call. Can be expensive, for many/large variables. For speed: Use globals...

I It takes time: Putting small procedures in-line is quicker

I the passing of parameters also takes time. Passing pointers to variables, or using globals, is again faster.

I In general, using memory takes time (in C: new). Think about the constructs you need.

I Pointers are faster than copying data but: Forget this for now...

111/260 APQE12-all Eﬃciency Memory Structure!

Your time is more valuable than computer time...

Concentrate on structured, readable code. Afterwards, with bug-free code

I proﬁle your code to see which routine takes most time

I think of putting mostly used data constructs global

I aforementioned tricks can shave 5-20% oﬀ execution time. Better algorithm can improve inﬁnitely more...

I move from laptop to recent desktop, think of using MPI, split program in smaller tasks, etc.

112/260 APQE12-all Eﬃciency Memory Day 3 - Morning

9.00L Optimization

I Newton-Raphson and quadratic optimisation I Hessian: Importance and problems I Loglikelihood and covariance I MaxBFGS 10.30P Estimating a duration model

I Likelihood I Optimization I Covariance 12.00 Lunch

113/260 APQE12-all Optimization Target function Maximization: Theory Rosenbrock function: 2 2 2 g(x) = 100 ∗ (x2 − x1 ) + (1 − x1) f (x) = −g(x)

3.5

2.5 1000 100 2 2 x 10 1.5 1 1 0.1 -2 0.5 0.01 -1.5 -1 0.001-0.5 -0.5 0 0 0.5 0 x 1 1.5 0.5 1 2 2.5 1 x 3 3.5 1.5 -0.5 2 4 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2

Minimum of g(x) at (1, 1) — Steep function — minimum in ‘narrow crooked alley’ 114/260 APQE12-all Optimization Random Wrong approach: Purely random Suppose: 2 I Search in area [−10, 10] I Throw darts randomly I Continue until you get precision = 0.25 from minimum, in both coeﬃcients How many darts would you need? P(|x − x∗| < ) = 2 = .025 i i 20 2 P(|x − x∗| < , i = 1, 2) = P(|x − x∗| < )2 = 4 = 0.000625 i i i i 202 E(n) = 1/p = 1600 Way too many...

See rb_darts.ox 115/260 APQE12-all Optimization Use function characteristics Better approach: Use function characteristics

(k) I Start in some point x

I Choose a direction s (k+1) (k) I Move distance α in that direction, x = x + αs

I Increase k, and possibly continue from 116 Direction s: Linked to gradient? Optimum: Gradient 0, second derivative positive deﬁnite?

116/260 APQE12-all Optimization Use function characteristics Ingredients

f (x):

∂2f (x) ∂2f (x) = H = HT ∂xi ∂xj ∂xj ∂xi Hessian symmetric

117/260 APQE12-all Optimization Newton-Raphson & co. Newton-Raphson

I Approach f (x) locally with quadratic function 1 f (x + h) ≈ q(h) = f (x) + hT f 0(x) + hT f 00(x)h 2

I Minimise q(h) (instead of f (x + h))

q0(h) = f 0(x)+f 00(x)h = 0 ⇔ f 00(x)h = −f 0(x) or Hh = −g

by solving last expression, h = −H−1g

I Choose x = x + h, and repeat as necessary Problems:

I Is H positive deﬁnite/invertible, at each step?

I Is step h, of length ||h||, too big or small?

118/260 APQE12-all Optimization Newton-Raphson & co. Problematic Hessian?

Algorithms based on NR need H(k) = f 00(x(k)). Problematic:

I Taking derivatives is not stable (...)

I Needs many function-evaluations

I H not guaranteed to be positive deﬁnite Problem is in step

(k)−1 sk = −H gk

(k)−1 Replace H by some Mk , positive deﬁnite by deﬁnition?

119/260 APQE12-all Optimization Newton-Raphson & co. BFGS/DFP Broyden, Fletcher, Goldfarb and Shanno (BFGS, choose θ = 1), but also Davison, Fletcher en Powell (DFP, θ = 0) thought of following trick: 1. Start with k = 0 and positive deﬁnite Mk , e.g. M = I 0 (k) 2. Calculate sk = −Mk gk , with gk = f (x ) (k+1) (k) 3. Find new x = x + hk , hk = αsk 4. Calculate, with qk = gk+1 − gk 0 0 qk Mk qk hk hk 1 − θ 0 Mk+1 = Mk + 1 + θ 0 0 − 0 Mk qk qk Mk hk qk hk qk qk Mk qk Result: θ 0 0 − 0 hk qk Mk + Mk qk hk I No Hessian needed hk qk I Still good convergence I No problems with negative deﬁnite Hk ⇒ MaxBFGS in Ox, similar routines in Matlab/Gauss/other. 120/260 APQE12-all Optimization pitfalls

Pitfalls I

5 15 f(x) ∆ f(x) 4 10 3 5 2 0 1 -5 0 -1 -10 -2 -15 -1 -0.5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 x x

I Local optima: Diﬀerent starting points? Simulation-based optimisation (simulated annealing, anyone?)

121/260 APQE12-all Optimization pitfalls

Pitfalls II

3.5

2.5 1000 100 2 2 x 10 1.5 1 1 0.1 -2 0.5 0.01 -1.5 -1 0.001-0.5 -0.5 0 0 0.5 0 x 1 1.5 0.5 1 2 2.5 1 x 3 3.5 1.5 -0.5 2 4 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2

I Flat surface: Diﬀerent starting points; transformation of parameter (θ = log σ tends to optimise a lot better...)

122/260 APQE12-all Optimization pitfalls

Pitfalls III

10 1000 LL x σ d(LL) x σ 0 LL+Noise x σ 800 d(LL+Noise) x σ

-10 600

-20 400

-30 200

-40 0

-50 -200

-60 -400 0.8 0.85 0.9 0.95 1 1.05 1.1 1.15 1.2 0.8 0.85 0.9 0.95 1 1.05 1.1 1.15 1.2

I Noise on derivatives/function: Only smooth functions can be optimised using NR-approach. Use analytical derivatives. Robust programming of target function (optimise AVERAGE LOG likelihood, never likelihood itself).

123/260 APQE12-all Optimization pitfalls

Pitfalls IV

-1

-2

-3

-4

-5

-6

-7 LL x σ -8 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2

I Non-uniqueness of optimum/indeterminacy in model speciﬁcation: Optimise σ, both σ = −1, σ = 1 give same likelihood... 124/260 APQE12-all Optimization pitfalls

Pitfalls V

1.5

0.5

0 0 0.5 1 1.5 2 2.5 3 3.5 4

I Bad Taylor-approximation: Sometimes transformation of parameters helps a bit, or it is just bad luck

125/260 APQE12-all Optimization pitfalls

Pitfalls VI

2.5

1.5

0.5

-0.5

-1 0 1 2 3 4 5 6

I Other numerical trouble: AVERAGE LOG likelihood, check range of optimisor, use grid search for some parameters, estimation by parts, move to simulation-based optimisation, improve starting values...

126/260 APQE12-all Maximize

A package: Maximize Doing Econometrics ≡ estimating models, e.g.: 1. Optimise likelihood 2. Minimise sum of squared residuals 3. Mimimise diﬀerence in moments 4. Do Bayesian simulation, MCMC Options 1-3 evolve around

θˆ = argmax ± f (y; θ) θ Inputs: I f , use average log likelihood, or average (negative) sum-of-squares.

I Starting value θ0 0 I Possibly f , analytical ﬁrst derivatives of f . 127/260 APQE12-all Maximize Likelihood Maximize I: Regression

2 yi = Xi β + i i ∼ N (0, σ )

ML maximises likelihood (other options: Minimise sum-of-squares, optimise utility etc):

2 Y 1 (yi − Xi β) L(y; θ) = √ exp − 2 2σ2 i 2πσ 2 − N 1 0 = (2πσ ) 2 exp − (y − X β) (y − X β) 2σ2

In this case, θ = (β, σ2)

128/260 APQE12-all Maximize Likelihood Maximize II

The maximize package provides features to maximize a function. So maximize AvgLnLRegr(θ) = LL(θ)/N

First write (and test!) this function:

Listing 21: eststack ml.ox #include #import // Two globals for likelihood function static decl s_vY, s_mX;

AvgLnLiklRegr(const vP, const adLnPdf, const avScore, const amHess) { adLnPdf[0]= ...;

return!ismissing(adLnPdf[0]); // Check if not missing} }

129/260 APQE12-all Maximize Likelihood Maximize III

Function to optimize

/* ** AvgLnLiklRegr(const vTheta, const adLnPdf, const avScore, const amHess) ** ** Purpose: ** Compute average loglikelihood of the regression model ** ** Inputs: ** vThetak+1x1 vector of parameters ** s_vY global,nx1 vector of regressor ** s_mX global,nxk matrix of explanatory variables ** ** Outputs: ** adLnPdf address, on output the average loglikelihood ** avScore(not used for now) ** amHess(not used) ** ** Return value: ** ir boolean,TRUE if computation succeeded,FALSE otherwise */

130/260 APQE12-all Maximize Likelihood syntax Syntax Function to maximize should have format

fnFunc(const vP, const adLnPdf, const avScore, const amHess)

I Choose your own logical function name

I vP is a p × 1 COLUMN vector with parameters

I adLnPdf is a pointer to a variable, which on return should contain the function value, or a missing if function could not be evaluated

I avScore can be either 0 or a pointer. In the latter case the function should ﬁll it in with the p × 1 score vector

I amHess can be either 0 or a pointer, but isn’t used in MaxBFGS

I The function should return either TRUE (if the calculation succeeded) or FALSE, if not.

131/260 APQE12-all Maximize Likelihood syntax Writing the likelihood Let’s do it... To remember: 2 − N 1 0 L(y; θ) = (2πσ ) 2 exp − (y − X β) (y − X β) 2σ2 1 e0e log L(y; θ) = − N log 2π + N log σ2 + 2 σ2 In this case, θ = (β, σ) or θ = (β, σ2). I Start oﬀ your adLnPdf[0] at a missing (just so you won’t forget giving it a value I Extract your parameters from the vector, use sensible names 2 I Check if your parameters are valid; if not (σ < 0?) return FALSE I At the end, test for a !ismissing(adLnPdf[0]) I Careful: This routine is used often; don’t do unnecessary work I And test... 132/260 APQE12-all Maximize Calling syntax Syntax II Call MaxBFGS according to #import ir= MaxBFGS(fnFunc, avP, adLnPdf, amInvHess, bNumDer);

I fnFunc is the name of the function I avP is a pointer to the initial vector of parameters, on return it will contain the optimal parameters I adLnPdf is a pointer which on return will contain the optimal value I amInvHess can be 0 or pointer to k × k matrix with initial inverse Hessian estimate; on output it gives a (bad) estimate of this matrix I bNumDer is a boolean, indicating if numerical derivatives have to be used The return value ir indicates the type of convergence; MaxConvergenceMsg(ir) returns an intelligible message-string. 133/260 APQE12-all Maximize Optimisation & ﬂow Optimisation Approach for general criterion function Q(β): Write

1 Q∗(β) = Q(β ) + g 0 (β − β ) + (β − β )0H (β − β ) (0) (0) (0) 2 (0) (0) (0) ∂ g(β) = Q(β) ∂β ∂2 H(β) = Q(β) ∂β∂β0

Optimise approximate Q∗(β):

g(0) + H(0)(β − β(0)) = 0 First order conditions −1 ⇔ β(1) = β(0) − H(0) g(0) and iterate into oblivion.

134/260 APQE12-all Maximize Optimisation & flow MaxBFGS: Program flow MaxBFGS follows quasi-Newton method according to Broyden, Fletcher, Goldfarb and Shanno (BFGS). Can use either numerical or analytical first derivatives. Analytical derivatives are more robust, exact, quicker.

No conv

MaxBFGS Gradient Move End Conv

fn fn fn fn

135/260 APQE12-all Maximize Average loglikelihood Maximize: Average

Why use average loglikelihood? 1. Likelihood function L(y; θ) tends to have tiny values → possible problem with precision 2. Loglikelihood function log L(y; θ) depends on number of observations: Large sample may lead to |large LL|, not stable 3. Average loglikelihood tends to be moderate in numbers, well-scaled... Better from a numerical precision point-of-view. Warning: Take care with score and standard errors (see later)

136/260 APQE12-all Maximize Precision/convergence Maximize: Precision

Strong convergence is said to occur if (roughly): i i i i 1. |qj θj | ≤ 1, ∀j, with qj the jth element of the score at θ , at iteration i: Scores are relatively small. i i−1 |θj −θj | 2. i ≤ 101: Change in parameter is relatively small |θj | Note: This also depends on the scale of your parameters... Preferably θ ≈ 1, not θ ≈ 1e − 15!

Adapt the precision with MaxControlEps(const dEps1, const dEps2);, default is dEps1= 1e-4, dEps2= 5e-3.

137/260 APQE12-all Maximize Score function Maximize: Scores

10 LL x σ 0

-10 -20 Optimising ≡ ‘going up’ -30 ≡ ﬁnding gradient. -40

-50

-60 0.8 0.85 0.9 0.95 1 1.05 1.1 1.15 1.2 Numerical gradient, for small h:

∂f (θ) f (θ + h) − f (θ) f (θ + h) − f (θ − h) f 0(θ) = ≈ ≈ ∂θ h 2h

Function evaluations: 2 × dim(θ)

Preferred: Analytical score f 0(θ)

138/260 APQE12-all Maximize Score function Maximize: Scores II

AvgLnLiklRegr(const vTheta, const adLnPdf, const avScore, const amHess) { ... if(isarray(avScore)) // Check if score is requested { avScore[0]= ???; // Compute the score, in whatever way } }

I Only compute the score when requested

I Test if avScore is an array (as this looks similar to an address)

I Or test if avScore is non-zero: if (avScore) ... , should do the same thing

I Work out vector of scores, of same size as θ.

I DEBUG! Check your score against Num1Derivative()

139/260 APQE12-all Maximize Score function Maximize: Scores IIb

I ...

I DEBUG! Check your score against Num1Derivative()

#import ... AvgLnLiklRegr(vTheta,&dLnPdf,&vS1, 0); // Compute analytical score Num1Derivative(AvgLnLiklRegr, vTheta,&vS2); // Compute numerical score

print(" Theta and scores : ", vTheta∼vS1∼vS2∼(vS1-vS2)); // Compare scores

Don’t ever forget debugging this (goes wrong 100% of the time...)

140/260 APQE12-all Maximize Score function Maximize: Scores III Let’s do it... To remember: 1 e0e f (y; θ) = − log 2π + 2 log σ + 2 Nσ2 e = y − X β ∂f (y; θ) = ... ∂β ∂f (y; θ) = ... ∂σ

2 I In this case, it matters whether θ = (β, σ) or θ = (β, σ )! I Find score of AVERAGE loglikelihood 141/260 APQE12-all Standard deviations

Standard deviations

Given a model with

L(Y ; θ) Likelihood function l(Y ; θ) = log L(Y ; θ) Log likelihood function ˆ θ = argmaxθl(Y ; θ) ML estimator

what is the vector of standard deviations, σ(θˆ)? Assuming correct model speciﬁcation,

Σ(θˆ) = −H(θˆ)−1 2 ˆ δ l(Y ; θ) H(θ) = 0 δθδθ θ=θˆ

142/260 APQE12-all Standard deviations

SD2: Average likelihood For numerical stability, optimise average loglikelihood. For regression model, e.g. the stackloss model, (y − X β)0(y − X β) l(Y ; θ) = − − N log 2πσ2 + c 2σ2 (y − X β)0(y − X β) l(Y ; θ) = − − log 2πσ2 + c0 2Nσ2 δ2l(Y ; θ) 1 δ2l(Y ; θ) 1 H ≡ = Σ(ˆ θˆ) = (−H )−1 l δθδθ0 N δθδθ0 N l

mS2= 0; if(Num2Derivative(AvgLnLiklRegr, avP[0], &mH)) mS2= invertgen(-mH, 30)/iN,

avS[0]= sqrt(diagonal(mS2)’ );

print (" MaxBFGS returns ", MaxConvergenceMsg (ir), " with LL= ", adLL [0]* iN , "at parameters ", "%c", {" Par ", " Std "}, avP [0]∼avS [0]);

143/260 APQE12-all Standard deviations

Day 3 - Afternoon

13.00L Optimization II

I Restrictions and transformations I Delta method: Covariance estimation I (ﬁxing parameters) 14.30P Implementing covariance estimation

I Duration model restricting α and/or β I Covariance of parameters 16.00 End 18.00 Course dinner at ‘Sct. Oluf’

144/260 APQE12-all Restrictions

Optimization and restrictions Take model

y = X β + , ∼ N (0, σ2)

Parameter vector θ = (β0, σ)0 is clearly restricted, as σ ∈ [0, ∞) or σ2 ∈ [0, ∞)

I Newton-based method (BFGS) doesn’t know about ranges

I Alternative optimization (SQP) may be(?) slower/worse convergence, but simpler Hence: First tricks for MaxSQP/MaxSQPF.

Warning: Don’t use MaxSQP unless you know what you’re doing (the function looks attractive, but isn’t always...)

145/260 APQE12-all Restrictions MaxSQP Restrictions: MaxSQP MaxSQP is an alternative to MaxBFGS I Without restrictions, delivers exactly MaxBFGS I Allows for sequential quadratic programming solution, for linear and non-linear restrictions.

#import MaxSQP(const func, const avP, const adFunc, const amHessian, const fNumDer, const cfunc_gt0, const cfunc_eq0, const vLo, const vHi); MaxSQPF(const func, const avP, const adFunc, const amHessian, const fNumDer, const cfunc_gt0, const cfunc_eq0, const vLo, const vHi); Restrictions: 1. cfunc_gt0(const avF, const vP); Fill avF[0]. Restriction: f (p) > 0 2. cfunc_eq0(const avF, const vP); Fill avF[0]. Restriction: f (p) = 0. 3. vLo: Restriction p >vLo 4. vHi: Restriction p

Advantages:

I Simple

I Implements restrictions on parameter space (e.g. σ > 0, 0 < α + δ < 1) Disadvantages:

I BFGS is meant for global optimisation; MaxSQP might work worse

I Often better to incorporate restrictions in parameter transformation: Estimate θ = log σ, −∞ < θ < ∞ So check out transformations...

147/260 APQE12-all Transforming parameters

Transforming parameters

Variance parameter positive? Solutions: 1. Use σ2 as parameter, have AvgLnLiklRegr return 0 when negative σ2 is found

2. Use σ ≡ |θk+1| as parameter, ie forget the sign altogether (doesn’t matter for optimisation, interpret negative σ in outcome as positive value) ∗ 3. Transform, optimise θk+1 = log σ ∈ (−∞, ∞), no trouble for optimisation Last option most common, most robust, neatest.

148/260 APQE12-all Transforming parameters

Transform: Common transformations

Constraint θ∗ θ [0, ∞) log(θ) exp(θ∗) θ exp(θ∗) [0, 1] log 1−θ 1+exp(θ∗)

Of course, to get a range of [L, U], use a rescaled [0, 1] transformation.

Note: See also exercise transpar

149/260 APQE12-all Transforming parameters

Transform: General solution Distinguish θ = (β0, σ)0 and θ∗ = (β0, log σ)0. Steps:

I Get starting values θ ∗ I Transform to θ ∗ I Optimize θ , transforming back within LL routine ∗ I Transform optimal θ back to θ

AvgLnLiklRegrTr(const vPTr, const adLnPdf, const avScore, const amHess) { ... vBeta= vPTr[:iK-1]; // Extract parameters dS= exp(vPTr[iK]); // in terms of sigma ... }

main() { ... vP= zeros(iK, 1)|1; vPTr= vP[:iK-1]|log(vP[iK]); ir= MaxBFGS(AvgLnLiklRegrTr,&vPTr,&dLnPdf, 0,TRUE); vP= vPTr[:iK-1]|exp(vPTr[iK]); }

150/260 APQE12-all Transforming parameters

Transform: Use functions

Notice code before: Transformations are performed 1. Before MaxBFGS 2. After MaxBFGS 3. Within AvgLnLiklRegrTr 4. And probably more often for computing standard errors Premium source for bugs... Solution: Deﬁne ∗ I TransPar(const avPTr, const vP): θ → θ ∗ I ir= TransBackPar(const avP, const vPTr) θ → θ And test (in a separate program) whether transformation works right. Necessary when using multiple transformed parameters.

151/260 APQE12-all Transforming parameters

Standard deviations

Remember:

Σ(θˆ) = −H(θˆ)−1 2 2 ˆ δ l(Y ; θ) δ l(Y ; θ) H(θ) = 0 = N 0 δθδθ θ=θˆ δθδθ θ=θˆ Therefore, we need (average) loglikelihood in terms of θ, not θ∗ for sd’s...

152/260 APQE12-all Transforming parameters

Transforming parameters II: SD Question: How to construct standard deviations? Answers: 1. Use transformation in estimation, not in calculation of standard deviation. Advantage: Simpler. Disadvantage: Troublesome when parameter close to border. 2. Use transformation throughout, use Delta-method to compute standard errors. Advantage: Fits with theory. Disadvantage: Is standard deviation of σ informative, is its likelihood suﬃciently peaked/symmetric? 3. After estimation, compute bootstrap standard errors 4. Who needs standard errors? Compute 95% bounds on θ, translate those to 95% bounds on parameter of interest. Advantage: Theoretically nicer. Disadvantage: Not everybody understands advantage. See next slides. 153/260 APQE12-all Transforming parameters

Transforming: Temporary Use global indicator s_bTrans indicating whether the parameters are transformed or not:

s_bTrans=TRUE; TransPar(&vPTr, avP[0]); println(" Transforming initial parameters ", avP[0]’ , " to ", vPTr ’); ir= MaxBFGS(AvgLnLiklRegr,&vPTr, adLL, 0,TRUE);

// Get back standard parameters TransBackPar(avP, vPTr); s_bTrans=FALSE; println(" Transforming back estimated parameters ", vPTr’ , " to ", avP [0] ’);

TransBackPar(const avP, const vPTr) { ... avP[0]= vPTr; if(s_bTrans) avP[0][iQ-1]= exp(vPTr[iQ-1]); returnTRUE; // Correct transform? Needed for NumJacobian } AvgLnLiklRegr(vPTr, ...) takes parameters, and transforms them to normal parameters at the start.

154/260 APQE12-all Transforming parameters

Transforming: Delta

1/2 a ∞ n (θˆ − θ0) ∼ N 0, V (θˆ) γˆ = g(θˆ) 0 γˆ ≈ g(θ0) + g (θ0)(θˆ − θ0) 1/2 a 0 1/2 ˆ a 0 2 ∞ ˆ n (ˆγ − γ0) = g0n (θ − θ0) ∼ N (0, (g0) V (θ)) scalar 1/2 a ∞ ˆ 0 n (ˆγ − γ0) ∼ N (0, G0V (θ)G0) vector

In practice: Use

var(ˆγ) = Gˆ var(θˆ)Gˆ 0 δg(θ) Gˆ = = dg(θ) dg(θ) ··· dg(θ) = Jacobian δθ0 dθ1 dθ2 dθk

155/260 APQE12-all Transforming parameters

Transforming: Delta in Ox

Listing 22: stack/eststack ml tr2.ox // Estimate transformed parameters TransPar(&vPTr, avP[0]); ir= MaxBFGS(AvgLnLiklRegr,&vPTr, adLL, 0,TRUE);

if(Num2Derivative(AvgLnLiklRegr, vPTr,&mH)) mS2Th= invertgen(-mH, 30)/iN,

// Get matrix of first derivatives of transformation NumJacobian(TransBackPar, vPTr,&mG); mS2= mG* mS2Th* mG’ ;

// Transform back parameters TransBackPar (avP , vPTr ); Use NumJacobian(TransBackPar, vPTr, &mG) to compute Jacobian. Note: TransBackPar needs to return TRUE if transformation succeeded.

156/260 APQE12-all Transforming parameters

Transforming: Bootstrap

I Estimate model, resulting inγ ˆ = g(θˆ) s I From the model, generate B bootstrap samples yj (ˆγ) s ˆs ˆs −1 s I For each sample, estimateγ ˆj = g(θj ), θj = g (ˆγj ) ˆ ˆs ˆs I Report var(θ) = var(θ1,..., θB ) I.e, report variance/standard deviation among those B estimates of the parameters, assuming your parameter estimates are used in the DGP. Simple, somewhat computer-intensive?

157/260 APQE12-all Transforming parameters

Transforming: Bootstrap in Ox

Listing 23: stack/eststack ml tr2 boot.ox BootstrapS(const avS, const mX, const vP, const iB) { ... for(i= 0;i< iB; ++i) { // Simulate data fromDGP GenerateData(&vY, mX, vP);

s_vY= vY; // Prepare globals for AvgLnLiklRegr

TransPar(&vPTr, vP); ir= MaxBFGS(AvgLnLiklRegr,&vPTr,&dLL, 0,TRUE); TransBackPar(&vPB, vPTr);

mG[][i]= vPB; // Record estimated parameters } mS2= variance(mG’ ); avS [0]= sqrt ( diagonal ( mS2 )’); } For the tutorial: Try it out for the normal model?

158/260 APQE12-all Fixing parameters

Fixing parameters What if some of the parameters are fixed? 1. Rewrite likelihood function for each special case 2. Keep track of fixed parameters in a flexible way Obviously, option (2) is preferred. Ideas: I Put full vector of parameters, including restricted, in global s_vPar I Keep track of free parameters by their indices, s_vIdxFreePar I In likelihood function, place free pars within full vector, s_vPar[s_vIdxFreePar]= vP;, and continue with s_vPar as if nothing happens. I Use functions FixPar(const dP, const iIdx) and FreePar(iIdx) to fix or free parameters Advantage: General, flexible, clean... 159/260 APQE12-all Fixing parameters

Fixing: Ox code

See live coding...

160/260 APQE12-all Fixing parameters

Other options

Study stack/eststack.ox, using

I choice of estimation methods

I log-ﬁle

I choice of output ﬁlename

I saving estimation results for later initialisation What is meaning of

oxl eststack x Constant AirFlow met 1 sa load base output/stb

161/260 APQE12-all Fixing parameters

Day 4 - Morning 9.00L Topics

I Finish standard errors, see Day 3b I Including packages I Including magic numbers I Including graphs 10.30P Estimating a duration model

I Transform 0.5 < β2 < 1 I Graph the durations I Advanced: 2 I Draw N = 1000, yi ∼ N (0, σ ) for a σ of choice. Make a QQ plot using DrawQQ I Make the QQ plot ‘by hand’ using DrawXMatrix, drawing the empirical quantiles of the y’s against the theoretical quantiles of the normal density α I Make a residual plot Ei = (Λi yi ) for your y’s of the duration model, and a QQ-plot against the Exp(1) density

162/260 APQE12-all Fixing parameters

Day 4 - Afternoon

12.00 Lunch 14.15 Empty slot? 16.00L Added capabilities 17.00 The end for today

163/260 APQE12-all Include packages

Include

Enlarging the capabilities of ox beyond oxstd.h capabilities: Either

#include

(to include the mentioned ﬁle literally within the program at that point, and will be compiled in), or

#import

(to import the code when needed; pre-compiled code is used when available)

164/260 APQE12-all Include packages Ox-provided packages Ox-provided packages

Package Purpose oxprob.h Extra probability densities oxfloat.h Definition of constants oxdraw.h Graphics capabilities (∗) arma.h ARMA filters and generators quadpack.h Univeriate numerical integration maximize Optimization using Gauss-Newton or Simplex methods (∗) maxsqp Maximize non-linear function with sequential quadratic programming solvenle Solve a system of non-linear equations solveqp Solve a quadratic program with restrictions database General class for creating a database modelbase General class for building a model simulation General class for simulation exercise

165/260 APQE12-all Include packages User-provided packages User-provided packages Package Author Purpose arﬁma Doornik, Ooms Long memory modelling dcm Eklof, Weeks Discrete choice models dpd Doornik, Arellano, Bond Dynamic Panel Data models ﬁnancialnr Ødegaard Financial numerical recipes gnudraw.h Bos Alternative graphing capabilities maxsa.h Bos Simulated Annealing mc2pack.h Bos Markov chain Monte Carlo estimation msvar Krolzig Markov switching oxutils.h Bos Some convenient utilities (∗) oxdbi Bruche A database independent abstraction layer for Ox ssfpack.h Koopman, Shephard, Doornik State space models ...... and many others m@ximize Laurent, Urbain Use CML optimisation in OxGauss oxgauss Doornik Run Gauss code through Ox

I Packages reside either in ox-home/packages, or in a local packages folder. I After including the package, the package is supposed to work seamlessly with Ox I Easy and clean way of communicating research

166/260 APQE12-all Include packages OxUtils A package: oxutils

What does ‘seamless’ mean? Standard situation: What is the size of a matrix I’m using?

main() { ... print(rows(mX)|columns(mX)); } How often would you use this code while debugging?

167/260 APQE12-all Include packages OxUtils A package: oxutils II Alternative: Use a package with some extra functions, not previously available

#include

main() { ... print(size(mX)); } Check manual /packages/oxutils/doc/oxutils.html Other routines I use plenty: info Measure time an iteration takes, time until end of program TrackRoutine Routine to proﬁle your program printtex Replacement for print, outputting in LATEX format ReadArg Read arguments from command line setseed Reset the random seed, psuedo-randomly

168/260 APQE12-all Include packages OxUtils A package: oxutils III (From tomorrow’s slides on speed) Use TrackTime("concat") to proﬁle a piece of code, get a report using TrackReport()

#include

main() { Output: decl iN, iK, mX,j; Ox Professional version 6.00 (Linux_64/MT) iN= 1000; // Size of matrix iK= 100; Time spent in routines concat 2.42 0.99 TrackTime(" concat "); predefined 0.02 0.01 mX= <>; for(j= 0;j< iN; ++j) Total: 2.44 mX|= rann(1, iK);

TrackTime(" predefined "); mX= zeros(iN, iK); for(j= 0;j< iN; ++j) mX[j][]= rann(1, iK); TrackTime(-1);

TrackReport(); }

169/260 APQE12-all Magic numbers

Magic numbers and declarations

Magic numbers:

I Those numbers/strings/settings deﬁning what your program will do

I Might be changed regularly (testing diﬀerent sample sizes, regressors etc.) Ugly solution: Change program Better solutions: 1. Specify them ‘outside’ program? 2. Specify them on command line

170/260 APQE12-all Declaration ﬁles

Include Enlarging the capabilities of ox beyond oxstd.h capabilities: Either

#include

(to include the mentioned ﬁle literally within the program at that point, and will be compiled in), or

#import

(to import the code when needed; pre-compiled code is used when available) You can also include a declaration ﬁle:

#include "simox.dec"

with special settings for your program.

171/260 APQE12-all Declaration ﬁles

Declaration ﬁle I

Remember previous exercise: Run with n = 100, run with n = 1000, run with n = 10000 etc. Options: 1. Build program very general, including a loop over diﬀerent values of n 2. Build a general program for one value of n; indicate the value to use in a declaration ﬁle 3. Possibly allow settings to be changed on the command line oxl lrdecl n 50 base thisversion e.g. using the ReadArg statement from OxUtils

172/260 APQE12-all Declaration ﬁles

Using a declaration file Wouldn’t it be useful to have Listing 24: stack/eststack.dec /* ** EstStack.dec ** ** Purpose: ** Contain definitions for estimating stack-loss data ** ** Date: ** 18/9/06 ** ** Author: ** Charles Bos */ static decl g_sData=" data / stackloss . in7 "; static decl g_sYVar=" StackLoss "; static decl g_asXVar= {" AirFlow "," WaterTemperature "," AcidConcentration "}; Use fewer X -vars: Change array here, leave program untouched Use different data set: Change g_sData and variable names, leave program untouched ⇒ Clean, touch program to change computation, put settings aside in separate file. 173/260 APQE12-all Declaration files

Prepare data set Listing 25: stack/loadstack.ox main() { decl mX, asNames;

mX= loadmat(" data / stackloss . mat "); // Read the data into rows mX= mX|1; // Add ina constant asNames= {" AirFlow "," WaterTemperature "," AcidConcentration ", " StackLoss "," Constant "}; savemat(" data / stackloss . in7 ", mX’ , asNames ); // Save data in columns print ("% r", asNames , meanr (mX)∼varr (mX )); // Check data } to reload it in estimation program with Listing 26: stack/eststack ols.ox InitData(const avY, const amX, const sYVar, const asXVar, const sData) { decl db;

db= new Database(); db.LoadIn7(sData); // Read database avY[0]= db.GetVar(sYVar); // ExtractY from database amX[0]= db.GetVar(asXVar); // Extract Xs from database

delete db; 174/260 return(sizerc(avY[0]) > 0) && (sizerc(amX[0]) > 0); } APQE12-all Declaration ﬁles

Using the declarations

Initialise your settings in a separate routine, reading out the declaration ﬁle, e.g.

Listing 27: stack/eststack ols.ox Initialise(const asYVar, const aasXVar, const asData) { asData[0]= g_sData; asYVar[0]= g_sYVar; aasXVar[0]= g_asXVar; } Preferably: Touch globals as little as possible, in few places.

Is this the only way of specifying the settings? No...

175/260 APQE12-all Declaration ﬁles

From the manual of OxUtils (dd 14/9/07)

176/260 APQE12-all Alternative: Command line arguments

Using ReadArg/ReadArgUsed

Listing 28: stack/eststack ols.ox Initialise(const asYVar, const aasXVar, const asData) { ... ReadArg(asData," data ", -1); // Read string with data file ReadArg(asYVar," y", -1); // Read string withy-variable ReadArg(aasXVar," x", -5); // Read array of strings withx-variable

ReadArgUsed(); // Show the arguments } Call Ox from command line using oxl eststack_ols data data/gnp.in7 y GNP x Constant IP (if Ox installed within the path) or use OxRun to indicate the parameters.

177/260 APQE12-all OxDraw

A package: OxDraw (or GnuDraw...) Ox graphics are displayed within OxMetrics. Needs the professional version for Windows. Alternatively, use GnuDraw: Displays graphics in GnuPlot on Windows, Unix. Compatible in usage, easy to switch.

Listing 29: stack/drawstack.ox #include //#include// Alternatively

// Draw stackloss regressors onY, stackloss itself onX axis DrawXMatrix(0, mX’ , asXVar , vY ’, sYVar, 1, 2); SaveDrawWindow(" graphs / stackloss . eps "); ShowDrawWindow(); From the manual:

DrawXMatrix(const iArea, const mYt, const asY, const vX, const sX,...); DrawXMatrix(const iArea, const mYt, const asY, const vX, const sX, const iSymbol, const iIndex);

178/260 APQE12-all OxDraw

OxDraw (or GnuDraw...) II

I Graphing appears in graphing area, ﬁrst argument I Draws rows at a time I Puts in a label. For multiple Y-values, give an array of labels {"yHat", "y", "cons"} I Can draw XY data, time series data, densities, QQ-plots etc. I Takes extra arguments specifying line types, colours etc. I After drawing the graph, and before showing it, the last graphing command can be adjusted using DrawAdjust(...); for daily data, draw XY with X=julian date, and use DrawAdjust(ADJ_AXISSCALE, AXIS_DATE); I Save the graphics in eps, pdf or gwg format (oxdraw), or also plb, png, tex and others (gnudraw) I Can show the graph on the screen (professional version of Ox) I Close the graph if necessary before continuing

179/260 APQE12-all Speed

Speed

I Use matrices, avoid loops

I Use the const argument qualiﬁer

I Use built-in functions

I Optimise inner loop

I Avoid using ‘hat’ matrices/outer products over large dimensions

I Matrices are stored by row

I Link in C or Fortran code

180/260 APQE12-all Speed Loops Speed: Loops vs matrices Avoid loops like the plague. Most of the time there is a matrix alternative, like for constructing dummies: Listing 30: speed loop2.ox #include #include

main() { decl iN, iR, vY, vDY,i,r;

iN= 10000; iR= 1000; vY= rann(iN, 1); vDY= zeros(vY);

TrackTime(" Loop "); for(r= 0;r< iR; ++r) for(i= 0;i< iN; ++i) if(vY[i] > 0) vDY[i]= 1; else vDY[i]= -1;

TrackTime(" Matrix "); for(r= 0;r< iR; ++r) vDY= vY .> 0 .? 1 .: -1; TrackTime(-1); TrackReport(); } 181/260 APQE12-all Speed const

Speed: const

Listing 31: speed const.ox SomeFuncConst(const mX) { // Do nothing }

SomeFuncNotConst(mX) { // Do nothing }

main() { decl iN, iK, iR, iRr, mX,r;

... mX= rann(iN, iK); for(r= 0;r< iR; ++r) SomeFuncNotConst(mX);

for(r= 0;r< iR; ++r) SomeFuncConst(mX); }

182/260 APQE12-all Speed Functions Speed: Built-in functions

Listing 32: speed builtin.ox #include #include

MyOlsc(const vY, const mX, const avBeta) { avBeta[0]= invert(mX’ mX )* mX ’vY;

return!ismissing(avBeta[0]); }

main() { decl iN, iK, iR, vY, mX, vBeta,r;

// Generate regression data ...

for(r= 0;r< iR; ++r) MyOlsc(vY, mX,&vBeta);

for(r= 0;r< iR*iRr; ++r) ols2c(vY, mX,&vBeta); }

183/260 APQE12-all Speed Optimise inner Speed: Inner/optimise

Target: Compute SSR= y − X β for a series of q diﬀerent vectors of β’s:

Listing 33: speed outer.ox ... mBeta= vBeta+ ranu(iK, iQ); mE= vY- mX*mBeta;

TrackTime(" Outer "); for(r= 0;r< iR; ++r) vE2= diagonal(mE’ mE );

TrackTime (" Inner - matrix "); for (r= 0; r < iR; ++r) vE2 = sumc (mE .* mE );

TrackTime (" Inner "); for (r= 0; r < iR; ++r) vE2 = sumsqrc (mE ); TrackTime ( -1);

184/260 APQE12-all Speed Rows/columns Speed: Rows/columns

Target: Successive ﬁll either a large row or column of a matrix

Listing 34: speed rows.ox ... iN= 10; // Size of matrix iK= 10000; iR= 100; // Number of repetitions

TrackTime(" columns "); mX= zeros(iK, iN); for(i= 0;i< iR; ++i) for(j= 0;j< iN; ++j) mX[][j]= rann(iK, 1);

TrackTime(" rows "); mX= zeros(iN, iK); for(i= 0;i< iR; ++i) for(j= 0;j< iN; ++j) mX[j][]= rann(1, iK); TrackTime(-1);

185/260 APQE12-all Speed Concatenation Speed: Concatenation or predeﬁne

In a simulation with a matrix of outcomes, predeﬁne the matrix to be of the correct size, then ﬁll in the rows. The other option, concatenating rows to previous results, takes a lot longer.

Listing 35: speed concat.ox ... iN= 1000; // Size of matrix iK= 100;

TrackTime(" concat "); mX= <>; for(j= 0;j< iN; ++j) mX|= rann(1, iK);

TrackTime(" predefined "); mX= zeros(iN, iK); for(j= 0;j< iN; ++j) mX[j][]= rann(1, iK);

186/260 APQE12-all Speed Ox vs C Speed: Ox vs C vs more optimised C

Replace heavy loops for C-code

Listing 36: speed c.ox ... TrackTime(" SsfLik Ox"); for(r= 0;r< iR; ++r) ir= SsfLikOx(&dLnLik,&dVar, vY, mPhi, mOmega, mSigma);

TrackTime(" SsfLik C"); for(r= 0;r< iR; ++r) ir= SsfLik(&dLnLik,&dVar, vY, mPhi, mOmega, mSigma);

TrackTime(" SsfLikEx optimised C"); for(r= 0;r< iR; ++r) ir= SsfLikEx(&dLnLik,&dVar, vY, mPhi, mOmega, mSigma);

187/260 APQE12-all Speed Overview Speed: Overview Method I II III Loop vs matrix 95 5 Const argument 100 0 Built-in function 69 31 Inner loop/hat matrices 60 32 8 Columns vs rows 75 25 Concatenation vs predeﬁned matrix 98 2 Ox vs C vs more optimised C 92 5 3 Conclusions: I If your program takes more than a few seconds, optimise I Track the time spent in parts of the program, optimise what takes longest I Declare your arguments as CONST I C is a lot faster in loops than Ox, for matrices it doesn’t matter much. 188/260 APQE12-all Speed Overview Day 4 - Afternoon

14.15 [Seminar] 16.00L Added capabilities

I Graphics packages I SsfPack/Arﬁma and others 17.00 End

189/260 APQE12-all Include packages

Include

Enlarging the capabilities of ox beyond oxstd.h capabilities: Either

#include

(to include the mentioned ﬁle literally within the program at that point, and will be compiled in), or

#import

(to import the code when needed; pre-compiled code is used when available)

190/260 APQE12-all Include packages Ox-provided packages Ox-provided packages

191/260 APQE12-all Include packages User-provided packages User-provided packages Package Author Purpose arﬁma Doornik, Ooms Long memory modelling dcm Eklof, Weeks Discrete choice models dpd Doornik, Arellano, Bond Dynamic Panel Data models ﬁnancialnr Ødegaard Financial numerical recipes gnudraw.h Bos Alternative graphing capabilities maxsa.h Bos Simulated Annealing mc2pack.h Bos Markov chain Monte Carlo estimation msvar Krolzig Markov switching oxutils.h Bos Some convenient utilities (∗) oxdbi Bruche A database independent abstraction layer for Ox ssfpack.h Koopman, Shephard, Doornik State space models ...... and many others m@ximize Laurent, Urbain Use CML optimisation in OxGauss oxgauss Doornik Run Gauss code through Ox

192/260 APQE12-all Include packages OxUtils A package: oxutils

What does ‘seamless’ mean? Standard situation: What is the size of a matrix I’m using?

main() { ... print(rows(mX)|columns(mX)); } How often would you use this code while debugging?

193/260 APQE12-all Include packages OxUtils A package: oxutils II Alternative: Use a package with some extra functions, not previously available

#include

194/260 APQE12-all Include packages OxUtils A package: oxutils III (From tomorrow’s slides on speed) Use TrackTime("concat") to proﬁle a piece of code, get a report using TrackReport()

#include

TrackTime(" predefined "); mX= zeros(iN, iK); for(j= 0;j< iN; ++j) mX[j][]= rann(1, iK); TrackTime(-1);

TrackReport(); }

195/260 APQE12-all SsfPack

A package: SsfPack Model in State Space:

αt+1 = d + T αt + ηt

yt = c + Zαt + t HH0 HG 0 (η0 , 0 )0 ∼ N t t GH0 GG 0

I Flexible structure for all linear Gaussian time series models

I Incorporates ARMA models, Exponentially Weighted Moving Average, regression models

I Allows for distinction of level, trend, seasonal components

I Can easily handle time series with missing observations

I Likelihood expressed analytically (through recursion formula)

196/260 APQE12-all SsfPack

SsfPack: Recursions

Recursion formula:

I Kalman ﬁlter (and related routines)

I Cumbersome to program in robust and general way ⇒ SsfPack (S.J. Koopman, N. Shephard, J. Doornik)

197/260 APQE12-all SsfPack

SsfPack: LLN

Some notation: αt+1 = δ + Φαt + ut ut ∼ N (0, Ω) yt

Simple Local Level with noise or Random Walk with Noise model:

0 1 .01 0 δ ≡ Φ = Ω = 0 1 0 1

αt+1 = αt + ηt ηt ∼ N (0,.01) Random walk

yt = αt + t t ∼ N (0, 1) with noise

198/260 APQE12-all SsfPack

SsfPack: Likelihood Deﬁne at+1 = E(αt+1|Yt ), with corresponding Pt+1 = cov(αt+1|Yt ). Then, observing yt+1 provides extra info on αt+1 ⇒ Filter... 0 0 vt = yt − Zat Ft = ZPt Z + GG 0 0 −1 Kt = (TPt Z + HG )Ft 0 0 0 at+1 = Tat + Kt vt Pt+1 = TPt T + HH − Kt Ft Kt

Prediction error vt |Yt ∼ N (0, Ft ) n X log L(Yn; θ) = log p(yt |y1, .., yt−1; θ) t=1 n 1 X = − log(2π) − (log |F | + v 0F −1v ) 2 2 t t t t t Hassle to compute likelihood, recursion, and slow... 199/260 APQE12-all SsfPack

SsfPack: Package Alternative: Use package...

main() { decl iN, vP, vAlpha, vY, mPhi, mOmega, mSigma, ir, dLnPdf, dVar;

iN= 10000; vP= <.1; 1>; vAlpha= cumulate(vP[0]*rann(iN, 1))’ ; // Generate state vY= vAlpha + vP [1] * rann (1, iN ); // Generate data

mPhi = <1; 1>; mOmega = diag (vP ); // Place variances mSigma = <-1; 0>; ir= SsfLik (& dLnPdf , &dVar , vY , mPhi , mOmega , mSigma ); } What is this SsfLik? 1. Internal Ox function, with heading in ssfpack.h (compare oxdraw.h)? 2. Ox function, deﬁned in ssfpack.h 3. Other type of function, reference in ssfpack.h? 200/260 APQE12-all SsfPack

SsfPack: Load those ﬁles Let’s take a peek...

Listing 37: .../ssfpack.h extern" packages / ssfpack / ssfpack , FnSsfLik " SsfLik(const adLik, const adVar, const mY, const mTZ, const mHG,...); (check extern statement)

Apparently

I Routine is internally called FnSsfLik

I Heading is deﬁned here in Ox terms

I Code is declared externally

I ... in some .dll/.so ﬁle

I ⇒ C-code included for Ox functionality...

201/260 APQE12-all SsfPack

SsfPack: Summary

SsfPack

I provides easy access to wealth of routines concerning State Space models

I gives seamless integration into Ox

I codes routines in optimized C-code

I stems from some of the leading researchers in the ﬁeld

202/260 APQE12-all SsfPack

Day 5 - Morning

9.00L Data handling

I Diﬀerence in formats I Reading large datasets I Selecting and transforming 10.30P Selected exercises. Choice of

I Reading HF data I Implement HF Autoregressive Duration Model I Using graphing package I AR(p) estimation with/without ARFIMA package 12.00 Lunch

203/260 APQE12-all Input and output

Input and Output

file type extension Names Remark ASCII matrix file .mat - Convenient input ASCII data file with load information .dat + PcGive/OxMetrics data file .in7 (with .bn7) + Retains variable names Excel spreadsheet file .xls + Common format, take care Lotus spreadsheet file .wks/.wk1 + Gauss data file .dht (with .dat) + Gauss matrix file .fmt - Small, save results Stata data file .dta - Input only text file using fscan/fprint functions ? Lots of control binary file using fread/fwrite functions ?

main() { decl mX, asVar;

mX= loadmat(" data / data . xls ",&asVar); println(" Saving data in 5 types of files : ... "); savemat(" excl / lm_data . mat ", mX); savemat(" excl / lm_data1 . dat ", mX, asVar); savemat(" excl / lm_data . dht ", mX, asVar); savemat(" excl / lm_data . in7 ", mX, asVar); savemat(" excl / lm_data . fmt ", mX); println(" done ."); }

204/260 APQE12-all Input and output Format: mat Input: mat format

I Text-based format, very convenient for inputting your data I Starts with two numbers, rows and columns of matrix I Followed by numbers; output matrix is filled row-by-row I Numbers are separated by space, comma or new-line I ‘.’, ‘m’, ‘M’ and ’.NaN’ are considered missing I ‘.Inf’ is infinity I Other text leads to skipping the remainder of the line Paths I You may specify a full path (but relative paths are easier). Use either ‘/’ (preferably) or ‘\\’ in your path. I The file is searched first starting from the present directory, then along the value of the OX6PATH environment variable (i.e., usually in the main Ox directory and its include subdirectory). 205/260 APQE12-all Input and output Format: mat stackloss.mat example

Listing 38: stack/data/stackloss.mat 4 21 // Stackloss.mat // // Hoeting,J.A., Madigan,D., and Raftery,A.E. (1996)‘‘A Method for // Simultaneous Variable Selection and Outlier Identification in Linear // Regression,’’{\em Journal of Computational Statistics and Data // Analysis},{\bf 22}, 251-270. // // Data source: // Brownlee,K.A. (1965),"Statistical Theory and Methodology in // Science and Engineering",2nd edition, New York:Wiley. // // Rows contain: // Air Flow, Water Temperature, Acid Concentration, Stack Loss 80 80 75 62 62 62 62 62 58 58 58 58 58 58 50 50 50 50 50 56 70 27 27 25 24 22 23 24 24 23 18 18 17 18 19 18 18 19 19 20 20 20 89 88 90 87 87 87 93 93 87 80 89 88 82 93 89 86 72 79 80 82 91 42 37 37 28 18 18 19 20 15 14 14 13 11 12 8 7 8 8 9 15 15 Great format to save your data, easy reference etc.

206/260 APQE12-all Input and output Format: XLS Spreadsheet ﬁles

ABCDE 1 CONS INC INFLAT OUTPUT 2 1953-1 890.45 908.21 3.66 1203.77 3 1953-2 886.54 900.68 2.76 1200.36 4 1953-3 886.33 899.8 2.52 1193.63 5 1953-4 884.88 898.48 1.72 1193.04

Rules:

I Variables in columns I Row of variable names I Column of dates, unlabelled, with year and period: YYYYxP I First sheet only I Periods are only useful when loading the ﬁle into the Ox Database class (see later)

I Names can be loaded along, with mX= loadmat("data/data.xls", &asVar). 207/260 APQE12-all Input and output Format: in7 OxMetrics files OxMetrics files are combination of.in7 file (Names of variables) and .bn7 file (Binary data). Small format, containing names, flexible for use within Ox. Can be used in 1. OxMetrics: Analyse data in spreadsheet environment 2. Ox: mX= loadmat("data/stackloss.in7", &asNames);, allows for loading including names 3. Ox-Database: Convenient format to put in database, using Object Oriented programming (see elsewhere) Hint: I Prepare/analyse your data in small Ox program I Save treated series (demeaned, log-transform, including dummies, without unneccessary variables) in .in7 format I Use .in7 for input in estimation program ⇒ Less probability of mistakes, makes sure you use correct data. 208/260 APQE12-all Input and output Format: Example Intermezzo: Stack loss

Prepare (and check!) data:

Listing 39: stack/loadstack.ox main() { decl mX, asNames,i;

mX= loadmat(" data / stackloss . mat "); asNames= {" AirFlow "," WaterTemperature ", " AcidConcentration "," StackLoss "}; savemat(" data / stackloss . in7 ", mX’ , asNames ); // Save data in columns print ("% r", asNames , meanr (mX)∼varr (mX )); // Check data

for (i= 0; i < 3; ++i) // Plot data , correctly read ? DrawXMatrix (i, mX[i][], asNames [i], mX [3][] , asNames [3] , 1); ShowDrawWindow (); }

209/260 APQE12-all Input and output Format: Example Intermezzo: Stack loss II Estimate using OxMetrics ﬁle: mStackloss= loadmat(" data / stackloss . in7 ",&asNames); // Load the data // Find the index of the stack loss variable i= strifind(asNames," StackLoss "); vY= mStackloss[][i]; // Read out column Stack loss i= strifind(asNames,{" AirFlow "," WaterTemperature "," AcidConcentration " }); mX= mStackloss[][i]; // Get other columns ir= olsc(vY, mX,&vBeta); // RunOLS on columns Or using a database: #import

main() { decl db, mStackloss, asNames, vY, mX, ir, vBeta,i;

db= new Database(); db.LoadIn7(" data / stackloss . in7 "); // Load the data

vY= db.GetVar(" StackLoss "); // Read out column Stack loss mX= db.GetVar({" Constant ", // Get other columns " AirFlow "," WaterTemperature "," AcidConcentration " }); ir= olsc(vY, mX,&vBeta); // RunOLS on columns

print(" Ols estimates of Beta : ", vBeta’ ); } 210/260 APQE12-all High frequency data

High frequency data

Biggest trouble: Data has not been saved for your speciﬁc purposes...

I Enormous ﬁles (Euronext Amsterdam: Tick-by-tick data, only stocks in Amsterdam, >6GB per month)

I Dirty: Not always saved exactly in order, sorting/cleaning/linking necessary

I Order book: What is best order available? Depends on orders as they arrive and orders as they are fulﬁlled; but those aren’t indicated clearly in dataﬁle... Lots of processing needed

I Timing: What eﬀect does daylight saving time have? What happens at market opening? Intra-day periodicity Not really something to solve here...

211/260 APQE12-all High frequency data

Step one: Reading ﬁrst ﬁle

Technicality: 32-bit Operating systems (Windows XP and before; many versions of Vista; older versions of Linux/OSX) can at most handle a file of 2GB length Even though a file is smaller, you might not have sufficient memory to read it... Have to work in smaller chunks.

Linux/OSX tools, available for Windows as well head -n file.csv Show ﬁrst n lines tail -n file.csv Show last n lines grep ABN file.csv Show lines containing ‘ABN’

212/260 APQE12-all High frequency data

Step one.B: Reading ﬁrst ﬁle

First get a smaller ﬁle, to test your procedures: Write 10.000 lines as eu_short.csv head-n 10000 EuronextAmsterdam-TemporaryOrders -200711-1.csv> eu_short.csv or choose a speciﬁc stock (Fortis, in this case):

head-n1 EuronextAmsterdam-TemporaryOrders -200711-1.csv> ft0711.csv grep BE0003801181 EuronextAmsterdam-TemporaryOrders -200711-1.csv >> ft0711.csv Pipe/append the outcome of grep to a ﬁle, using > ft0711.csv or >> ft0711.csv Resulting ﬁle looks like

66694;BE0003801181;FORTIS ;Amsterdam;2007-09-25;09:38:05;11276;0;L; ;.;0;V;78;0;29.170000;2007-09-25;.;: : 0; ;. 66694;BE0003801181;FORTIS ;Amsterdam;2007-09-25;10:07:15;15381;0;L;D;2007-12-31;0;A;1000;0;19.100000;2007-09-25;2007-11-08;10:07:15;X;. 66694;BE0003801181;FORTIS ;Amsterdam;2007-09-25;11:13:00;27943;0;L;D;2008-09-24;0;A;1000;0;20.000000;2007-09-25;2007-11-02;11:13:00;X;. 66694;BE0003801181;FORTIS ;Amsterdam;2007-09-26;12:36:11;12968;0;L; ;.;0;V;50;0;25.500000;2007-09-26;.;: : 0; ;.

213/260 APQE12-all High frequency data

Step two: Reading smaller ﬁle

Several options: 1. Use editor to cut out unnecessary columns, get it into easily readable format 2. Use spreadsheet for this purpose 3. Write a script in Perl or Python 4. Write a program in Ox or other

214/260 APQE12-all High frequency data

Step two: Editor + Ox

Could you get it into .mat or .csv format ? Try

mX= loadmat("myfile.csv");

I Advantage: Little ox-coding needed

I Disadvantage: You have to hand-edit the downloaded ﬁle...

215/260 APQE12-all High frequency data

Step two: Reading in Ox

Bring out Swiss army knife: Read line by line, analyse block-by-block, store in prefered format

fh= fopen(" ft0711 . csv "," r"); while(!feof(fh)) { // Reada line ir= fscan(fh," %z",&sLine);

// Analyse line

// Store line } Analyse: Build a routine reading in the items you need. Check out examples on sscan() and tokens.

216/260 APQE12-all High frequency data

Step two: Read a line With ir= fscan(fh, "%z", &sLine); you have one line as a string. Droste-eﬀect: Take small steps... I One option: Split at ‘;’ I Get all separate strings I Easier to handle afterwards Listing 40: hf/readft.ox SplitLine(const aasElem, const sLine, const sSep) { decli,j;

aasElem[0]= {}; i=j= 0; while(i< sizeof(sLine)) { j= strifind(sLine[i:], sSep); // Find sep afteri if(j < 0) // If not found j= sizeof(sLine)-i; // indicate end of string aasElem[0] ∼= sLine[i:i+j-1]; // Read out next element i+=j+1; } } 217/260 But: See also simpler alternative below... APQE12-all High frequency data

Step two: Read the elements

Now we have an array with strings: How to convert a string to a number? Check sscan: ir= sscan(" 2007 -11 -04 "," %d",&iYY," -%d",&iMM," -%d",&iDD); This

I Reads the string

I Assigns ﬁrst integer (format=‘%d’) to iYY

I Skips the minus, and assigns second integer to iMM

I etc

I and returns the number of values read correctly (here: 3)

218/260 APQE12-all High frequency data

Step two: Read the elements II

Read date (element 4), time (element 5), size (element 13) and price (element 15)

ReadElements(const avX, const asX) { decl ir, iYY, iMM, iDD, iH, iM, iS, dSize, dPrice, dDate;

ir= sscan(asX[4]," %d",&iYY," -%d",&iMM," -%d",&iDD) + sscan(asX[5]," %d",&iH," :%d",&iM," :%d",&iS) + sscan(asX[13]," %d",&dSize) + sscan(asX[15]," %g",&dPrice); // Get date number, including time-of-day} dDate= dayofcalendar(iYY, iMM, iDD)+timeofday(iH, iM, iS);

avX[0]= dDate ∼dSize∼dPrice;

return ir; }

219/260 APQE12-all High frequency data

Step two: Simpler alternative

If the input file has fixed-width fields, you could just count what columns you are interested in. Say you want the date, time, and the last price.

06/25/1979,1759,1.3580,1.3760,1.3580,1.3725,0,0 06/26/1979,1759,1.3725,1.3725,1.3554,1.3564,0,0 06/27/1979,1759,1.3564,1.3645,1.3564,1.3635,0,0 Then you want to read an integer (”%2i”), a slash+integer (”/%2i”), slash+integer (”/%4i”), comma, two integers of width two, (”,%2i”, ”%2i”), bunch of nothing, (”%*22c”) which is not assigned to anything, and a double (”%g”)...

ir= sscan(sLine," %2i",&iMM," /%2 i",&iDD," /%2 i",&iYY, " ,%2i",&ihh," %2i",&imm," %*22 c"," %g",&dPrice); print(" Number of elements : ", ir, " Date : ", iYY∼iMM∼iDD, " Time and price : ", ihh∼imm∼dPrice);

220/260 APQE12-all High frequency data

Step two: Store the results

Remember speed-example: Concatenation is bad... Don’t do

mX= <>; while(!feof(fh)) { ... ReadElements(&vX, asX); mX|= vX; } How can you initialise mX in that case?

221/260 APQE12-all High frequency data

Step two: Store the results II Do it in batches...

I Choose batch-size iB I Initialise mX at the batch size: mX= zeros(iB, 3); I Fill in the lines mX[iL++][]= vX; I If you run out of space, extend mX: mX|= zeros(iB, 3); I At the end, throw out empty lines: mX= mX[:iL-1][]; ⇒ Relatively low number of concatenations Quick code, especially for HF data! Warning:

I HF data tends to be dirty I Cleaning is a necessity I Hard work, have to know the source, and construction of dataset...

222/260 APQE12-all High frequency data

HF Results After reading Fortis data over the ﬁrst days of November 2007 (during discussion of sale to ABNAMRO), 115461 ‘good’ trades out of 307924 result.

800000 700000 Size 600000 500000 400000 300000 200000 100000 0 01/11 02/11 03/11 04/11 05/11 06/11 07/11 08/11 09/11 10/11

25 Price 20 15 10 5 0 01/11 02/11 03/11 04/11 05/11 06/11 07/11 08/11 09/11 10/11 Fortis trade size and price, November 2007 223/260 APQE12-all High frequency data

HF Alternative

Using ‘Private’ package packages/loadcsv/loadcsv.ox:

I Similar to loadmat(), simple loading

I Loads all data, including strings

I No hassle with SplitLine etc.

I But far slower...

Program readft2.ox readft3.ox Method SplitLine loadcsv Time 0m24s 2m01s

224/260 APQE12-all Data selection

Data selection

Standard situation

I Large dataset of observations, many individuals I Only those

I older than 18, I female, I with an income between 25-50k should go into the sample... Three words: selectifr, deleteifr, vecrindex Some others: thinc, range

225/260 APQE12-all Data selection

Data: selectifr First two either select or delete speciﬁc rows from a matrix.

Listing 41: seldata/seldata.ox main() { ... // Load data including names mX= loadmat(" ../ data / duration . in7 ",&asNames);

// Find out what are indices in data set vSel= strfind(asNames,{" Age "," Sex "," Income " });

mX= selectifr(mX, mX[][vSel[0]] .> 18); // Older than 18 mX= selectifr(mX,!mX[][vSel[1]]); // Only female mX= selectifr(mX,(mX[][vSel[2]] .>= 25000) .&& (mX[][vSel[2]] .<= 50000)); // 25k-50k income ... } selectifr(mA, mB): Select only those rows of mX which have (at least one) non-zero element in the corresponding rows of the second argument.

226/260 APQE12-all Data selection

Data: deleteifr

You could also do it the other way around:

Listing 42: seldata/deldata.ox main() { ... // Load data including names mX= loadmat(" ../ data / duration . in7 ",&asNames);

// Find out what are indices in data set vSel= strfind(asNames,{" Age "," Sex "," Income " });

mX= deleteifr(mX, mX[][vSel[0]] .<= 18); // Not older than 18 mX= deleteifr(mX, mX[][vSel[1]]); // Not female mX= deleteifr(mX,(mX[][vSel[2]] .< 25000) .|| (mX[][vSel[2]] .> 50000)); // <25k or >50k income ... }

227/260 APQE12-all Data selection

Data: vecrindex

For more extensive situations: Use vecrindex?

Listing 43: seldata/inddata.ox // Create vector of zero/ones, indicating observations to use vI= (mX[][vSel[0]] .> 18) .&& !mX[][vSel[1]] .&& (mX[][vSel[2]] .>= 25000) .&& (mX[][vSel[2]] .<= 50000); // Re-using vSel, to indicate observations to use} vSel= vecrindex(vI); print(" Limits of indexed observations : ", limits(mX[vSel][]));

// Or, old tricks: mX= selectifr(mX, vI);

I These functions are used extensively in Initialise()

I Warning: Don’t do this within likelihood function...

I Changing big data matrices, sorting, whatever: Relatively slow

228/260 APQE12-all Data selection

Data: thinr/thinc

If a matrix is too large to handle, thinr can select a number of rows from a large matrix:

mXSmall= thinr(mX, 1000); // Get 1000 rows from mX, if possible A similar eﬀect, with more control, can be obtained using range:

iN= rows(mX); vI= range(0, iN-1, 100); mXSmall= mX[vI][]; // Get row 0, 100, 200,... from mX

229/260 APQE12-all OxDraw

Listing 44: stack/drawstack.ox #include //#include// Alternatively

// Draw stackloss regressors onY, stackloss itself onX axis DrawXMatrix(0, mX’ , asXVar , vY ’, sYVar, 1, 2); SaveDrawWindow(" graphs / stackloss . eps "); ShowDrawWindow(); From the manual:

DrawXMatrix(const iArea, const mYt, const asY, const vX, const sX,...); DrawXMatrix(const iArea, const mYt, const asY, const vX, const sX, const iSymbol, const iIndex);

230/260 APQE12-all OxDraw

OxDraw (or GnuDraw...) II

231/260 APQE12-all OxDraw

Day 5 - Afternoon

13.00L Eﬃciency and remaining topics

I Alternative algorithms I Matrices I Own code vs C-code I Others 14.30P Finding the diﬀerence

I Concatenation I Loop vs AR(p) vs ARFIMA package 15.30- Handing out exam, ﬁnal remarks

232/260 APQE12-all Speed

Speed

I Use matrices, avoid loops

I Use the const argument qualiﬁer

I Use built-in functions

I Optimise inner loop

I Avoid using ‘hat’ matrices/outer products over large dimensions

I Matrices are stored by row

I Link in C or Fortran code

233/260 APQE12-all Speed Loops Speed: Loops vs matrices Avoid loops like the plague. Most of the time there is a matrix alternative, like for constructing dummies: Listing 45: speed loop2.ox #include #include

main() { decl iN, iR, vY, vDY,i,r;

iN= 10000; iR= 1000; vY= rann(iN, 1); vDY= zeros(vY);

TrackTime(" Loop "); for(r= 0;r< iR; ++r) for(i= 0;i< iN; ++i) if(vY[i] > 0) vDY[i]= 1; else vDY[i]= -1;

TrackTime(" Matrix "); for(r= 0;r< iR; ++r) vDY= vY .> 0 .? 1 .: -1; TrackTime(-1); TrackReport(); } 234/260 APQE12-all Speed const

Speed: const

Listing 46: speed const.ox SomeFuncConst(const mX) { // Do nothing }

SomeFuncNotConst(mX) { // Do nothing }

main() { decl iN, iK, iR, iRr, mX,r;

... mX= rann(iN, iK); for(r= 0;r< iR; ++r) SomeFuncNotConst(mX);

for(r= 0;r< iR; ++r) SomeFuncConst(mX); }

235/260 APQE12-all Speed Functions Speed: Built-in functions

Listing 47: speed builtin.ox #include #include

MyOlsc(const vY, const mX, const avBeta) { avBeta[0]= invert(mX’ mX )* mX ’vY;

return!ismissing(avBeta[0]); }

main() { decl iN, iK, iR, vY, mX, vBeta,r;

// Generate regression data ...

for(r= 0;r< iR; ++r) MyOlsc(vY, mX,&vBeta);

for(r= 0;r< iR*iRr; ++r) ols2c(vY, mX,&vBeta); }

236/260 APQE12-all Speed Optimise inner Speed: Inner/optimise

Target: Compute SSR= y − X β for a series of q diﬀerent vectors of β’s:

Listing 48: speed outer.ox ... mBeta= vBeta+ ranu(iK, iQ); mE= vY- mX*mBeta;

TrackTime(" Outer "); for(r= 0;r< iR; ++r) vE2= diagonal(mE’ mE );

TrackTime (" Inner - matrix "); for (r= 0; r < iR; ++r) vE2 = sumc (mE .* mE );

TrackTime (" Inner "); for (r= 0; r < iR; ++r) vE2 = sumsqrc (mE ); TrackTime ( -1);

237/260 APQE12-all Speed Rows/columns Speed: Rows/columns

Target: Successive ﬁll either a large row or column of a matrix

Listing 49: speed rows.ox ... iN= 10; // Size of matrix iK= 10000; iR= 100; // Number of repetitions

TrackTime(" columns "); mX= zeros(iK, iN); for(i= 0;i< iR; ++i) for(j= 0;j< iN; ++j) mX[][j]= rann(iK, 1);

TrackTime(" rows "); mX= zeros(iN, iK); for(i= 0;i< iR; ++i) for(j= 0;j< iN; ++j) mX[j][]= rann(1, iK); TrackTime(-1);

238/260 APQE12-all Speed Concatenation Speed: Concatenation or predeﬁne

In a simulation with a matrix of outcomes, predeﬁne the matrix to be of the correct size, then ﬁll in the rows. The other option, concatenating rows to previous results, takes a lot longer.

Listing 50: speed concat.ox ... iN= 1000; // Size of matrix iK= 100;

TrackTime(" concat "); mX= <>; for(j= 0;j< iN; ++j) mX|= rann(1, iK);

TrackTime(" predefined "); mX= zeros(iN, iK); for(j= 0;j< iN; ++j) mX[j][]= rann(1, iK);

239/260 APQE12-all Speed Ox vs C Speed: Ox vs C vs more optimised C

Replace heavy loops for C-code

Listing 51: speed c.ox ... TrackTime(" SsfLik Ox"); for(r= 0;r< iR; ++r) ir= SsfLikOx(&dLnLik,&dVar, vY, mPhi, mOmega, mSigma);

TrackTime(" SsfLik C"); for(r= 0;r< iR; ++r) ir= SsfLik(&dLnLik,&dVar, vY, mPhi, mOmega, mSigma);

TrackTime(" SsfLikEx optimised C"); for(r= 0;r< iR; ++r) ir= SsfLikEx(&dLnLik,&dVar, vY, mPhi, mOmega, mSigma);

240/260 APQE12-all Speed Overview Speed: Overview Method I II III Loop vs matrix 95 5 Const argument 100 0 Built-in function 69 31 Inner loop/hat matrices 60 32 8 Columns vs rows 75 25 Concatenation vs predeﬁned matrix 98 2 Ox vs C vs more optimised C 92 5 3 Conclusions: I If your program takes more than a few seconds, optimise I Track the time spent in parts of the program, optimise what takes longest I Declare your arguments as CONST I C is a lot faster in loops than Ox, for matrices it doesn’t matter much. 241/260 APQE12-all C-Code

Linking Ox and C-Code

Origin of Ox:

I ‘Shell’ around C

I Getting rid of memory allocation/deallocation

I Quicker implementation Disadvantage:

I Some extra overhead

I Slower loops =⇒ Go back to C where it pays oﬀ

242/260 APQE12-all C-Code

Example: Kalman ﬁlter

Random walk plus noise model:

2 µt+1 = µt + ηt ηt ∼ N (0, ση) 2 yt = µt + t t ∼ N (0, σ )

Listing 52: kf/kf0.ox GenrData(const avY, const avMu, const vP, const iT) { decl dSEps, dSEta;

[dSEta, dSEps]= {vP[0], vP[1]}; // Generate Mu, random walk avMu[0]= cumulate(dSEta* rann(iT, 1))’ ; // Generate observations , Mu + noise avY [0]= avMu [0] + dSEps * rann (1, iT ); }

243/260 APQE12-all C-Code

KF: Filter equations Deﬁne: 0 0 αt+1 Tt Ht Ht Ht Gt = αt + ut ut ∼ N 0, 0 0 yt Zt Gt Ht Gt Gt

at+1 ≡ E(αt+1|Yt ) Pt+1 ≡ cov(αt+1|Yt )

Then ﬁlter (Harvey 1989):

vt = yt − Zt at [Prediction error] 0 0 Ft = Zt Pt Zt + Gt Gt [Prediction error variance] 0 0 −1 Kt = (Tt Pt Zt + Ht Gt )Ft [Kalman gain] at+1 = Tt at + Kt vt [Prediction update] 0 0 0 Pt+1 = Tt Pt Tt + Ht Ht − Kt Ft Kt [Variance update]

Likelihood: L(yt |Yt−1, θ) ≡ L(vt |Yt−1, θ) ∼ N (0, Ft ) 244/260 APQE12-all C-Code

KF in Ox

Listing 53: kf/kf0.ox KalmanFilOx(const mYt, const vP) { ... for(i= 0;i< iT; ++i) { ... dv= mYt[][i]- da; dF= dP+ dS2Eps; dK= (dP + 0)/dF;

da= da+ dK* dv; dP= dP+ dS2Eta- dK* dF* dK; ... } return mKF; }

245/260 APQE12-all C-Code

KF: Testing

6 Y 5 Mu org 4 Mu pred 3 2 1 0 -1 -2 -3 0 10 20 30 40 50 60 70 80 90 100

Random walk plus noise, and predicted mean E(αt+1|Yt ) To use the ﬁlter, e.g. build it into a loglikelihood, and optimise: MaxBFGS returns Strong convergence at parameters 0.49104 1.0103 Time elapsed using Ox: 0.90 for1 iterations Time per iteration: 0.90 Using T = 10000, optimising only once. Bootstrap/simulation? 246/260 APQE12-all C-Code

KF in C

Move routine back from Ox to C? Worthwhile if

I One routine takes most of the time (check e.g. TrackTime("KalmanFil") in OxUtils package)

I Routine is relatively simple (not too many inputs/outputs)

I Routine takes time in looping, not in matrix manipulations Ergo: Kalman ﬁlter fulﬁlls these prerequisites.

Documentation: Ox Appendices, example on creating a matrix with 3’s...

247/260 APQE12-all C-Code

KF in C II

Items to consider:

I Compiler: GCC on Linux, e.g. MinGW on Windows

I Ox development kit (http://www.doornik.com)

I ox/dev/samples/threes/win_gcc/readme.txt: Further hints on installing the compiler and

I makeﬁle: How to compile?

I project.def: What functions are deﬁned? [Only on Windows/BCC?]

I the c-code itself... First: Let’s check the Ox code in detail (see/write kf0.ox/kf.ox)

248/260 APQE12-all C-Code

KF in C: inckalman Listing 54: kf/ccode/inckalman.c #include" oxexport .h"

voidOXCALL FnKalmanFilC(OxVALUE*rtn, OxVALUE*pv, int cArg) { int iT;

// Check arguments if(cArg != 2) OxRunError(ER_ARGS,NULL); OxLibCheckType(OX_MATRIX, pv, 0, 1); iT= OxMatc(pv, 0); OxLibCheckMatrixSize(pv, 0, 0, 1, iT); OxLibCheckMatrixSize(pv, 1, 1, 2, 1); ... } I Include Ox exported functions I Fixed function heading void OXCALL FnName(OxVALUE *rtn, OxVALUE *pv, int cArg) containing pointer to return value, pointer to function arguments, and number of arguments I Always ﬁrst check arguments, in detail (number, type, size) I See Ox appendices for available functions

249/260 APQE12-all C-Code

KF in C: inckalman II

double dS2Eta, dS2Eps; MATRIX mYt, vP, mKF; ... // Prepare output mKF OxLibValMatMalloc(rtn, 5, iT);

mYt= OxMat(pv, 0); vP= OxMat(pv, 1); mKF= OxMat(rtn, 0);

dS2Eta=SQR(vP[0][0]); dS2Eps=SQR(vP[1][0]);

I No automatic typing in C I Inherited from Ox: Matrices I Allocate memory if you need it (and deallocate if necessary...) mYt is address of Ox matrix, ﬁrst element in array of arguments pv, vP is second element etc. Space is allocated for return pointer; for simplicity, mKF is made to point to this same space. 250/260 APQE12-all C-Code

KF in C: inckalman III

Listing 55: kf/ccode/inckalman.c ... da= 0.0; dP= 5.0; for(i= 0;i< iT; ++i) { mKF[3][i]= da; mKF[4][i]= dP;

dv= mYt[0][i]- da; dF= dP+ dS2Eps; dK= (dP + 0)/dF;

da= da+ dK* dv; dP= dP+ dS2Eta- dK* dF* dK; mKF[0][i]= dv; mKF[1][i]= dK; mKF[2][i]= 1.0 / dF; } // End of routine, done } As mKF ≡ return value, all is done at end of routine. Similar use of matrices, but only scalar operators directly available. Ox functions can be used from C; convenient for more complex calculations. 251/260 APQE12-all C-Code

KF in C: Calling from Ox

Calling C-code from Ox:

I Make function available, as an external function loaded from a dynamic link library

I Use it as any other function, completely transparant

extern" ccode / inckalman , FnKalmanFilC " KalmanFilC(const mYt, const vP);

AvgSsfLikC(const vP, const adLnPdf, const avScore, const amHess) { decl mKF;

mKF= KalmanFilC(m_mYt, exp(vP)); adLnPdf[0]= -0.5*meanr(log(M_2PI) + sqr(mKF[0][]) .* mKF[2][] - log(mKF[2][]));

return!ismissing(adLnPdf[0]); } See diﬀerence...

252/260 APQE12-all C-Code

KF in C: Conclusions

What did we ﬁnd?

I Large speed-up possible (here: up to 30× faster)

I Only attainable in pure, heavy loops

I Mainly matrix algebra: C eﬀectively slower than Ox (as Ox is heavily optimised, more than your C-code will ever be)

I Only useful for internal loops

I Compare extra programming eﬀort to gain in speed But: Similarity between C and Ox syntax does pay oﬀ.

253/260 APQE12-all Fortran Code

Linking Ox and Fortran Code

Origin of Ox:

I ‘Shell’ around C

I Getting rid of memory allocation/deallocation

I Quicker implementation Disadvantage:

I Some extra overhead

I Slower loops =⇒ Go back to C or even Fortran where it pays oﬀ.

254/260 APQE12-all Fortran Code

Linking Ox and Fortran Code II Ox is based on C: ⇒ Easier communication with C. Hence easiest option: I Write function in C for communication with Ox (see before) I Call Fortran from C Elements to keep in mind: I Arguments to Fortran are always by reference: Changing Fortran argument will change the element in the calling function I Therefore, one should pass pointers to Fortran from C I C uses doubles, make sure Fortran does the same I Names might be ‘decorated’. GCC/GFortran compiler will rename ‘kalmanfil’ to ‘kalmanfil_f’ I Fortran77 has no dynamic memory allocation: Declare any matrices from C 255/260 APQE12-all Fortran Code

Building blocks: C side

In C, do communication with Ox like before, with additionally an external declaration to the Fortran function. Enumerate the argument for the Fortran function, as pointers to doubles or integers, usually:

// external declaration of Fortran function extern void kalmanfilf_(double*, double*, double*, int*);

voidOXCALL FnKalmanFilFC(OxVALUE*rtn, OxVALUE*pv, int cArg) { ... // Prepare output mKF OxLibValMatMalloc(rtn, 5, iT); mKF= OxMat(rtn, 0);

// Call Fortran function kalmanfilf_(mKF[0], mYt[0], vP[0], &iT); ... }

256/260 APQE12-all Fortran Code

Building blocks: Fortran side

On the Fortran side, note that the matrix mKF comes in transposed. For vectors, direction does not matter.

subroutineKALMANFILF(mKFt, mYt, vP, iT)

* Note size of matrices; mKFt comes in transposed, asC uses row-first * order, Fortran column-first integer iT double precision vP(2), mYt(iT), mKFt(iT,5)

* Local variables double precision dS2Eta, dS2Eps, da, dP, dv, dF, dK;

dS2Eta= vP(1) * vP(1) dS2Eps= vP(2) * vP(2) ...

257/260 APQE12-all Fortran Code

Building blocks: Fortran side II

Computations are standard Fortran (but use doubles, explicitly!)

da= 0.0d0 dP= 5.0d0

doi= 1, iT mKFt(i, 4)= da mKFt(i, 5)= dP

dv= mYt(i)- da dF= dP+ dS2Eps dK= dP/dF

da= da+ dK* dv dP= dP+ dS2Eta- dK* dF* dK mKFt(i, 1)= dv mKFt(i, 2)= dK mKFt(i, 3)= 1.0 / dF end do

258/260 APQE12-all Fortran Code

Compilation Compilation can be done with one of the prepared makeﬁles. These perform eﬀectively

gfortran-Wall-fPIC-O3-c-o inckalmanf.o inckalmanf.f gcc-Wall-fPIC-O3-c-o inckalmanfc.o inckalmanfc.c gcc-Wall-fPIC-O3-shared-o inckalmanfc_64.so inckalmanfc.o inckalmanf.o Note that compilation often goes wrong. If Ox complains of not being able to load the dll, check 1. did compilation end with an error? 2. are all routines available? Check with ‘nm inckalmanfc_64.so’ 3. possibly include extra libraries, e.g. add option ‘-lgfortran’ at the linking stage If nothing works, go back to a ‘last known good’ conﬁguration, and add in small steps.

259/260 APQE12-all Fortran Code

Results

Include the C-wrapper function into Ox. Results:

Time elapsed using FortranKF: 9.89 for 1000 iterations withT=10000 Time elapsed usingC: 10.85 for 1000 iterations withT=10000 ⇒ Fortran seems marginally quicker in this case

260/260