1 QCOBJ a Python package to handle

2 quantity-aware configuration files

1 2 3 Roberto Vidmar and Nicola Creati

1 4 OGS - Istituto Nazionale di Oceanografia e di Geofisica Sperimentale 2 5 OGS - Istituto Nazionale di Oceanografia e di Geofisica Sperimentale

6 Corresponding author: 1 7 Roberto Vidmar

8 Email address: [email protected]

9 ABSTRACT

10 Configuration files are widely used by scientists and researchers to configure the parameters and initial 11 settings for their computer programs. 12 We present here a Python package that adds physical quantities to these parameters and validates them 13 against user defined specifications to ensure that they are in the correct range and eventually converted 14 to the requested unit of measurement. The package contains also a graphical user interface class to 15 display, edit configuration file content, and to compare them side by side highlighting their differences.

16 INTRODUCTION

17 Scientists often use configuration files (cfg files) to set the parameters and initial conditions for their 18 computer programs or simulations. When these parameters are not limited to numbers or strings but 19 represent physical quantities their unit of measure must be taken into account. Researchers are used to 20 convert derived physical quantities by hand or with the help of some computer program but this operation 21 slows down the process and is inherently error prone.

22 We developed a package to give an answer to this problem integrating unit of measure and hence 23 dimensionality into parameters. This approach ensures that programs using this package will always get 24 numbers in the requested range and in the correct unit of measure independently of the units used in the 25 configuration file.

26 CODE DESIGN

27 Scientific work implies the writing of many lines of code and researchers try to capitalize their production 28 creating reusable code that is driven by cfg files. These are essentially text files in which keys are 29 associated to values and these can be numeric or symbolic. Usually comments can be added to explain 30 the meaning of the keywords and other useful information for the end users.

31 The standard packages that allow Python programmers to take benefit of cfg files are ConfigParser 32 (Langa, 2017) and ConfigObj (Foord and Larosa, 2017).

33 The main difference between them is that ConfigObj has the capability to validate a cfg file against 34 a specification file called configspec. The configspec defines the allowed data types and ranges for the 35 keywords and can set default values. Thus the cfg file just needs to specify values that differ from defaults.

36 However when programs use various physical quantities the user must take care of converting them to 37 the numbers assigned in the cfg file that will be validated against the configspec specifications.

38 We found this process time consuming and error prone because it needs frequent review of correctly 39 developed and tested functions to search for not existing bugs. Our solution was the integration of physical 40 quantities into configuration files.

41 Many Python libraries exists that manage physical units manipulation but we found that Pint (Grecco, 42 2017) is the most easy to read and has a clean syntax to specify units. Pint integrates unit parsing: prefixed 43 and pluralized forms of units are recognized without explicitly defining them. In other words: since the

PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.3418v1 | CC BY 4.0 Open Access | rec: 17 Nov 2017, publ: 17 Nov 2017 44 prefix kilo and the unit meter are defined, Pint understands kilometers. Pint can also handle units provided 45 as strings and this capability was the main reason of our choice.

Figure 1. A simple application using CfgGui class.

46 Eventually we created a Python library, or more properly a pacakge to integrate ConfigObj and Pint 47 and we called it called QCOBJ. QCOBJ is composed by three main classes.

48 Q

49 This is the physical quantity container class. It provides all the methods to manipulate the strings that 50 define physical quantities and supports conversion to and from different units. It implements also methods 51 for their representation in clear human readable form. Q instances can be added or compared only if they 52 have the same dimensionality.

53 QValidator

54 Validation is a transparent layer to access data stored as strings. The validation checks if the data is correct 55 and converts it to the expected type.

56 The QValidator class is an extension of the original Validator class that understands the new syntax 57 created for the physical quantities. Moreover it ensures that the values used in the cfg file are dimensionally

2/7 PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.3418v1 | CC BY 4.0 Open Access | rec: 17 Nov 2017, publ: 17 Nov 2017 58 correct and converts them to the units specified by configspec. If configspec defines also a minimum 59 and/or a maximum value the QValidator raises errors when the user supplied values are out of range.

60 QConfigObj

61 It extends the ConfigObj class adding methods to integrate the QValidator. Default cfg files can be created 62 when the validator instance is supplied to an empty instance of this class. QConfigObj has also methods 63 for converting user defined quantities to the units used in configspec. For example a keyword defining a 64 velocity in m/s in configspec can always be converted to m/s even if its value is set to knots in cfg or is set 65 during program execution to any other velocity unit of measure.

66 QConfigObj has a reserved keyword configFiles that allows the inclusion of a list of files. Long cfg, 67 hundreds or thousands lines, can be split into smaller units thus improving readability and reuse.

Figure 2. The expanded tree view of the comparison of two cfg files with the differences highlighted in reverse text/background colors.

3/7 PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.3418v1 | CC BY 4.0 Open Access | rec: 17 Nov 2017, publ: 17 Nov 2017 68 Graphical User Interface (GUI) 69 While comparing long cfg files we noticed that spotting differences between them was quite difficult. 70 Standard tools are available (Wikipedia, 2017a) to compare files but these highlight also difference in 71 indentation and in comment lines hence we found them rather unusable. The CfgGui class included in the 72 package defines a simple GUI to explore, edit and compare already defined configuration files. 73 We chose a tree diagram representation for the GUI as configuration files are organized in sections 74 and these can be nested to any level. In Figure 1 there is an expanded tree view of a cfg file. Values are 75 coloured according their data types: quantities in red, boolean in green, strings in blue. Other data types 76 in black. CfgGui can display more than one cfg file side by side as in Figure 2 and allows the expansion 77 and compression of every single section as well as synchronized scrolling. Comment lines are ignored, 78 values are colored according to their data types and differences between them are highlighted reverting 79 background and foreground colors. Hovering with the mouse over a value pops up a help tooltip window 80 with the valid range for that parameter. Comparison of more than two files (3-way comparison) is also 81 supported. 82 Our implementation uses the Model-View-Controller (MVC) (Wikipedia, 2017b) pattern and among 83 the many GUi toolkits available from the Python literature (Alves, 2017) (Polo, 2017) we stick to 84 PyQt/PySide (Riverbank Computing Limited, 2017) (The Qt Company, 2017) since they provide great 85 flexibility and user control. 86 PyQt and PySide are almost identical from the user point of view, PyQt being a much more mature, 87 efficient and stable project. On the other side PySide provides LGPL-licensed Python bindings for the 88 Qt framework and this feature can be important when deciding how to distribute the software. QCOBJ 89 includes a compatibility module to leave the user free to choose the preferred library at runtime.

90 USAGE

91 Since configuration files use physical quantities it is mandatory to create first a configspec file that defines 92 the keywords and the valid data types allowed for each of them. The keywords can be organized into 93 sections and subsections in a hierarchical form. 94 Cfg files can be written with any text editor but we provided a utility function makeSpec that helps 95 building configspec sections with the correct syntax and indentation through a short Python script.

96 l e v e l = 1 # The hierarchical level of this section 1 97 secname = ’Ingredients’ 2 98 subsection = collections.OrderedDict(( 3 99 ( ’ s u g a r ’ , ( 4 100 ’ Enable s u g a r ’ , 5 101 ’ b oole an ’ , 6 102 F a l s e ) ) , 7 103 (’fruits’, ( 8 104 ’A l i s t of e x a c t l y two f r u i t s a t your c h o i c e ’ , 9 105 ’ s t r i n g l i s t 2 2 ’ , 10 106 ” apple , orange ” ) ) , 11 107 (’roomTemp’, ( 12 108 ’Room temperature’, 13 109 ’ degC , 18 , 26 ’ , 14 110 2 0 . 0 ) ) , 15 111 (’fraction’, ( 16 112 ’Some dec imal v a l u e ( f l o a t s a r e welcome , as always ) ’ , 17 113 ’ f l o a t , 0 , 1 ’ , 18 114 0 . 2 5 ) ) , 19 115 )) 20

116 117 subspec = makeSpec(secname, subsection , level) 21 Listing 1. makeSpec.py - Python script to create a section using the makeSpec function.

118 Every section/subsection can be built filling an ordered Python dictionary in which each keyword is 119 associated with a tuple. The last two elements of it are the range of valid values and the default while the 120 remaining values will appear as comments in the configspec. The second last element defines also the 121 type of the keyword that can be a physical quantity according to the Pint syntax. The keyword roomTemp

4/7 PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.3418v1 | CC BY 4.0 Open Access | rec: 17 Nov 2017, publ: 17 Nov 2017 122 of listing 1 al line 12 for example defines a temperature that can assume any value between 18 and 26 ◦ 123 degrees Celsius with a default value of 20 C. The subsection instance of the same listing at line 3 is then 124 processed by makeSpec at line 21 of listing 1 leaving in the subspec string what appears in listing 2 with 125 the correct syntax and proper indentation.

126 Listing 3 is an example of configspec file.

127 [Ingredients] 1 128 # Enable sugar 2 129 sugar = boolean(default=False) 3 130 #A list of two fruits at your choice 4 131 fruits = string list(default=list(apple, orange)) 5 132 # Room temperature 6 133 roomTemp = quantity(units=degC, min=18, max=26, default=’20.0 degC’) 7 134 # Some decimal value(floats are welcome, as always) 8

135 136 fraction = float(min=0, max=1, default=0.25) 9 Listing 2. Section created with the script of Listing 1.

137 # 1 138 # Header of this configspec file, date, authors and version 2 139 # 3 140 description = string(default=’A meaningful descripion of this configuration file’) 4 141 ... 5 142 voltage = quantity(units=V, min=0, max=100, default=’13.8 V’) 6 143 UG = quantity(units=N ∗ m∗∗2 / kg∗∗2, default=’6.672e−11 N ∗ m∗∗2 / kg∗∗2’) 7 144 # List of more configuration files blank separated 8 145 configFiles = string(default=’’) 9 146 [Ingredients] 10 147 ... 11 148 roomTemp = quantity(units=degC, min=18, max=26, default=’20.0 degC’) 12 149 [[Regions]] 13 150 ... 14 151 # More Sections like this can be added with different names 15 152 [[[ many ]]] 16 153 ... 17 154 # Constant Temperature 18

155 156 T = quantity(units=degC, default=’18.0 degC’) 19 Listing 3. my configspec.cfg - Example of configspec file.

157 The quantity in line 7 defines the unit of mesure for the keyword UG and hence its physical dimensions. 158 The accepted quantities as well as their alias are defined in the Pint unit definitions file but the user can 159 easily edit this text file to add any other unit needed.

160 Lower and upper limits for each quantity can be specified like in line 6 leaving to the QValidator class 161 the duty to check that the value defined in the actual configuration file is in the defined range.

162 The special section many defines sub-sections to be validated using the same keywords and 163 specification.

164 The creation of this configspec can be speeded using a template file instead of writing it from scratch. 165 Such a file can be generated form its configspec assigning to all keywords their default values with the 166 following command:

167 python −c ’from qcobj.qconfigobj import QConfigObj; 168 template = QConfigObj(”my configuration file.cfg”, configspec=” 169 configspec.cfg”);

170 171 template.write()’

172 The just created my configuration file.cfg can later be tailored by the user with less effort.

173 The physical quantities designated in configspec can now be assigned in any unit of measurement 174 provided its value is dimensionally correct and, if converted to the units already specified in configspec, it 175 is in the allowed range (if defined). For example voltage can be assigned as

5/7 PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.3418v1 | CC BY 4.0 Open Access | rec: 17 Nov 2017, publ: 17 Nov 2017 176 voltage = 12.6 V # or 177 voltage = 12.6 volt # or 178 ∗ 179 voltage = 12.6 ohm ampere

180 Pressure can be expressed in pascal or Pa or newton / m**2 or force kilogram / cm**2.

181 Configuration file content is accessible to Python scripts in this way:

182 from q co b j import q c o n f i g o b j 183 qcobj = qconfigobj.QConfigObj(”my configuration f i l e . c f g ” ,

184 185 configspec=”my configspec.cfg”)

186 Once the QCOBJ object has been instantiated, physical quantities in it can be accessed with the 187 standard ConfigObj syntax:

188 section = qcobj[’Ingredients’] 189 roomTemperature = section[’roomTemp’] 190 # Being a validated quantity , roomTemperature can be converted by 191 # the script to any other units and eventually its value is 192 # available for computation 193 absoluteRoomTemperature = roomTemperature. to(’K’).magnitude 194 # Increase temperature 195 absoluteRoomTemperature += 5.2 196 # Convert now this ∗∗ number ∗∗ back to its physical quantity 197 # as in configspec: in this case degC ∗∗ r e g a r d l e s s ∗∗ of the units used in 198 # m y configuration f i l e . c f g 199 newTempQuantity = qconfigobj.qLike(

200 201 absoluteRoomTemperature , section , ’roomTemp’)

202 More utility functions are available in the qconfigobj package to simplify the use of quantities from 203 Python scripts. Full documentation of the package and of all the classes and functions in it is available 204 both in HTML and pdf format. A few examples are also available to learn the basic usage.

205 FINAL REMARKS

206 QCOBJ has been evaluated and tuned during the developemnt of a geodynamic parallel numerical 207 simulation suite (Nicola Creati et al., 2015). We found it of great help in managing the hundreds of 208 physical quantities (parameters) needed for the computation and its use solved the troublesome process of 209 units conversions leaving more time available for the research. CfgGui was necessitous for comparing cfg 210 files that drove models with hundreds of parameters.

211 ACKNOWLEDGMENTS

212 We thank Hernan´ Grecco for the time and effort he spent in helping us to achieve this result.

213 REFERENCES

214 Alves, M. (2017). GuiProgramming - Python Wiki. https://wiki.python.org/moin/ 215 GuiProgramming [Accessed: 20 September”]. 216 Foord, M. and Larosa, N. (2017). Reading and Writing Config Files. http://www.voidspace. 217 org.uk/python/configobj.html [Accessed: 20 September”]. 218 Grecco, H. E. (2017). Pint: makes units easy. https://pint.readthedocs.io/ [Accessed: 20 219 September”]. 220 Langa, Ł. (2017). 14.2. configparser - Configuratin file parser. https://docs.python.org/3/ 221 library/configparser.html# [Accessed: 20 September”]. 222 Nicola Creati, Roberto Vidmar, and Paolo Sterzai (2015). Geodynamic simulations in HPC with Python. 223 In Kathryn Huff and James Bergstra, editors, Proceedings of the 14th Python in Science Conference, 224 pages 158 – 163. 225 Polo, G. (2017). PyGTK, PyQT, Tkinter and wxPython comparison. http://ojs.pythonpapers. 226 org/index.php/tpp/article/download/61/57 [Accessed: 20 September”].

6/7 PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.3418v1 | CC BY 4.0 Open Access | rec: 17 Nov 2017, publ: 17 Nov 2017 227 Riverbank Computing Limited (2017). Riverbank — Software — PyQt — What is PyQt? https: 228 //riverbankcomputing.com/software/pyqt/ [Accessed: 20 September”]. 229 The Qt Company (2017). PySide - Qt Wiki. https://wiki.qt.io/PySide [Accessed: 26 Octo- 230 ber”]. 231 Wikipedia (2017a). Comparison of file comparison tools - Wikipedia. https://en.wikipedia. 232 org/wiki/Comparison_of_file_comparison_tools [Accessed: 20 September”]. 233 Wikipedia (2017b). Model-view-controller - Wikipedia. https://en.wikipedia.org/wiki/ 234 Model-view-controller [Accessed: 20 September”].

7/7 PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.3418v1 | CC BY 4.0 Open Access | rec: 17 Nov 2017, publ: 17 Nov 2017