arXiv:2107.11347v1 [cs.PL] 23 Jul 2021 ecnie w prahs an approaches: two consider we questio the but data, incomplete for practice in used widely h etdRltoa acls(R)[]i ihlvlque high-level a is [2] (NRC) Calculus Relational Nested The h etdRltoa acls(R)hsbe ninfluential an been has (NRC) Calculus Relational Nested The INTRODUCTION 1 res conservativity integratio and normalization language of in extensions e sketch and nulls implicit handling the discuss relating approaches, translations give We default. by t primitive nullable an (possibly) represent to used are p types this In thoroughly. la studied programming been typed have a to into not queries appears guage, such integrate or informat NRC, incomplete in and nulls SQL-style accommodate to how Null logic. two-valued on n based is NRC logic, ‘standard’ inc with three-valued of patible and treatment nulls SQL’s using However, information, Links. plete and Scala, langu F#, programming as in such query language-integrated for been basis also has a It queries. SQL standard to while translation flexibility lowing and power providing language, query level ABSTRACT atfrlnug-nertdqeymcaim ospott support to mechanisms query Neverthele language-integrated i for [11]. so tant nulls, bugs involve applications subtle and databases to real all counte lead most in can result that and behavior th misunderstood itive although easily SQL, also in used are widely features and standard is approach critic this Despite “unknown”. value, truth third three-va a with have a mantics to val or predicates extending these “null”, and be on nulls, propagate to operations values primitive field extending allowing sent/missing, proposal early Co an recognized. clearly made been has sup information of incomplete importance ing the languages, query associated and model agae u orfi o nefcn ihata incomple ty actual ambient with interfacing an for into fit poor queries a database but language, integrating make This for data. spe fit of no good piece is a absent an there indicating which value in null und languages, cial as programming etc. typed strings, most booleans, in integers, include types base htmpNCqeisoe a aat Q queries. SQL to data flat over queries NRC map that ure hntasomn a nust a upt 2] Th [22]. outputs flat to inputs called property, flat rela transforming flat when than expressive queries more no t are Despite queries 20]. NRC 15, flexibility, integra 8, [3, query languages and functional programming high-level con- m database in types, can for collection use and to and composed natural record it freely of nesting be with can values queries struct NRC SQL, as such ug rvdn prtosfrcletos(es as li pecially bags, (sets, collections for operations providing guage otpeettoso R n eae agae shwnull eschew languages related and NRC of presentations Most dat relational the of development the of years early the From implicit comprehensions prahi hc ye r rae spossibly-null as treated are types which in approach conservativity nvriyo Edinburgh of University [email protected] ncnrs osadr ur languages query standard to contrast In . ae Cheney James stebssfrrwiigalgorithms rewriting for basis the is , explicit prahi hc option which in approach opeedn nulls Comprehending t,ec) es- etc.), sts, simpor- is t i added his ps and ypes, s [10], ism ults. erstood tcom- ot udse- lued sdas used edata. te ,and n, tl al- still xplicit e to ues NRC s ylan- ry d[6] dd tional hem. aking s al- ss, rintu- high- are s port- ages aper of n tion ped om- ion ese n- b- is s: a - atdt elwt ul fohrbs ye,w ol aeto have would we types, base other of nulls with deal to wanted hnw d ulvle oNC ihih ehia suswh issues technical highlight NRC, to values null add we when au ouei lc fNL o nee fields. integer for NULL of place in use to value ulyed necpin rmtv prtr uha addit as such ( operators equality Primitive exception. an yields null a eetatdfo the from extracted be can ycekn h ola field Boolean the checking by utrtivn aa(osbyicuigNLs rmteda the from NULLs) including (possibly data retrieving just aei airt rt ure novn uhotoa data optional such involving queries write use to be easier can it types make option on nul operations is convenient result and the loading then null is input any if does: SQL like nulls ulvle hte a Whether value. null a eur rgamr ob osiu fwihfilsaenulla are fields which of conscious exceptions, be to and programmers coercions require implicit w on reasonably rely work but to practice, appear th in approaches On Quill sensible. is and type F# each singl the of hand, a values using null that of clear place isn’t in default it and value, default a provide eipiil ore to coerced implicitly be EET*FO iessWEEnm 'covid-19' = name WHERE diseases FROM as * such query SELECT a execute w would diseases we to name, related given information a showing type). page known a web have a option yet produce is not might type diseases the new (some but nullable non-nullable) ident (i.e. The required (integer). are type name and (string), 20 name [3, (integer), F# fier in LINQ in and sett [15], three library in Quill currently Scala’s handled [8], is Links it how gi discuss first We and nulls. example lacks that language SQL) (e.g. host language functional query typed, database typical a in nulls of ment ut r ie ieettype: different a given are sults globa single a set process to that allows be code Links fail, the Currently, by results. simply expected query would not was field value type NULL the the cause from NULLs read to tempted usin.I atclrw osdrtefloigissues: following the consider we outlin particular and In known, questions. already or straightforward are solutions nertdqeysse hudsilb beda ihthem. with deal able be lang still should a system imperfect, query is integrated nulls to approach SQL’s while Moreover, biul,teLnsslto sltl etrta ak i hack: a than better little is solution Links the Obviously, nQil ulbefilsaegvnoto ye,adSaaov Scala and types, option given are fields nullable Quill, In fw okbyn h ipeseai bv nwihw are we which in above scenario simple the beyond look we If ups ehv al otiigdsae,ec ihident with each diseases, containing table a have we Suppose nF,i otat ulbefilsi aaaetbe rquer or tables database in fields nullable contrast, in F#, In nLns ni eety tepigt xct ure tha queries execute to attempting recently, until Links, In einconsiderations. Design nti hr ae,w netgt h einise htar that issues design the investigate we paper, short this In 1 hudnlsb rae mlcty(iei Q)o expli or SQL) in (like implicitly treated be nulls Should (1) 3 ocasclrslsnee o rnltn R queries NRC translating type for base at needed just results or classical type, any Do at (3) available be nulls Should (2) lk pinvle nfntoa languages)? functional in values option (like Q otnet odi h rsneo nulls? of presence the in hold to continue SQL + , = r itdt ulbevrin ( versions nullable to lifted are ) [email protected] nvriyo Edinburgh of University imrRicciotti Wilmer Nullable
∗ The syntax of NRCnull is NRC extended with a null constant and Here 5 is the primitive operation 5 lifted to apply to options, i.e. ∗ with a nullness test, as follows. We assume the presence of primi- 5 (some(E1),..., some(E=)) = some(5 (E1,...,E=)) and otherwise ∗ tive operations including at least the logical connectives ∧, ∨, ¬. 5 (. . . none . . .) = none. These operations are definable in NRCopt, as are the other null-sensitive operations such as equality and log- Terms ",# ::= ···| null | isNull(") ical connectives. Conditionals must be translated so that the then- We do not allow nulls at record or collection types. For collection branch is executed only if the test is true, and the else-branch if the types in particular, the expected behavior of nulls is unclear. The test is false or null. To ensure this we use the auxiliary operation semantics of logical connectives is three-valued, as in SQL. The se- isTrue(G) = case G of (none ⇒ false | some(~)⇒ ~). mantics of other primitive operations is strict: if any argument is null then the result is too, otherwise the primitive operation is per- 6 HANDLING NULLS IN QUERY RESULTS formed on the non-null inputs. Finally, if the Boolean in a where The translations above establish that NRC and NRC are equally statement is null, then the statement evaluates to an empty collec- opt null expressive (and equally expressive to NRC provided all base types tion (similarly to false and contrary to true). This behavior can have default values). In principle one could allow programmers to be specified by adding the following rewriting rules to the standard write queries in NRC , generate and evaluate the correspond- NRC ones: null ing SQL queries, and translate the results at the end to host lan- isNull(null) { true isNull(2) { false guage values involving options. How can we make it easy to work 5 (. . . null . . .) { null " where null { ∅ with these results in a host language where field types do not have nulls? 5 TRANSLATIONS Nullable type tracking. This idea is a slightly strengthened form The implicit and explicit approaches have complementary advan- of F#’s approach. The type system could be extended to track nul- tages. NRCopt is essentially a special case of the nested relational lability information in queries, and using this information try to calculus with binary sum types. However, if many fields are nul- minimize the amount of optional tagging that must be added. In lable, writing queries in NRCopt is excruciating. On the other hand, particular, this approach could cope with the overloaded behavior NRCnull seems easier to relate to plain SQL queries, and writing of primitive operations on nulls, by giving them types that indicate queries that operate over possibly-null values is more straightfor- that the result may be null only if one of the inputs may be null; if ward (albeit with the same pitfalls as SQL), but normalization re- all inputs are nonnull then so is the result. This approach could be sults for NRC with implicit nulls do not follow immediately from encoded using a sufficiently rich type system, e.g. dependent types prior work. We consider translations in each direction. or Haskell’s type families. However, if schemas lack accurate infor- mation about nullability, any benefits may be limited. From NRCopt to NRCnull. The main issue arising in this transla- tion is the fact that option types can be nested inside other type Null handlers. This idea is loosely inspired by the common lan- constructors, including options: for example (int? × bool)? repre- guage feature of exception handling, and by Quill’s pragmatic ap- sents an optional pair the first element of which is also optional. To proach to dealing with optional values inside queries. Given aquery deal with this generality, we translate options and cases as follows: returning flat records in NRCnull, we could consider a small domain- g? = hisnull : bool, val : g i specific language of null handlers that specify how to map the re- sult to an NRC value. A null handler is a record of instructions noneJ K = hisnull = true, val =J 3K i opt JgK defining what to do with each possibly-null field: someJ (")K = hisnull = false, val = " i (1) optional: return an option value case " J K if ".isnull J K u } (2) required: skip this record if this field is null of (none ⇒ #1 = then #1 (3) default E: return default value E if null v | some(G) ⇒ #2) ~ else #2 [ " .val/G] J K Syntactic sugar for declaring multiple fields optional or required where 3g is a default value of type g. Note thatJ nulls,K J isNullK (−) and may also be useful. Of course, it is possible to provide any other other null-sensitive primitive operations are not needed to handle desired behavior by returning all nullable results as optional values. options, assuming that there are constants of each base type in If nulls are tracked by the type system, then fields that are certainly NRCopt: this translation actually maps NRCopt to plain NRC. nonnull do not need to be mentioned. For example, the disease table query from Section 1 could have From NRC to NRCopt. Types are translated as follows: null (among others) two handlers: −−→ −−−−→ = = = 1 1? hℓ : gi hℓ : g i {g} { g } hid : required, name : required, type : default − 1i The mostJ K interestingJ cases ofK the termJ K translationJ K are:J K hid : required, name : required, type : requiredi 2 = some(2) The first one will use −1, an invalid type value, if a type field is ∗ null, while the second will skip any records that contain null type 5 ("1,...,"=J)K = 5 ( "1 ,..., "= ) fields. Nulls in the id and name fields could also lead to records be- J nullK = noneJ K J K ing dropped, but should not occur according to the schema. These isNullJ (")K = " = none handlers can be desugared to case analyses using isNull() (on if " thenJ#1 else #2K = Jif isTrueK ( " ) then #1 else #2 the database side) or case (in the host language). By desugaring J K J K J K J K James Cheney and Wilmer Riccio i to database-side case analyses, the handling can be performed in are used to embed SQL queries safely into an ambient typed pro- the database, possibly saving effort. gramming language, as in Links, F#, or Quill. In this short paper, we have outlined the main issues and design considerations we think 7 RELATED AND FUTURE WORK are important for a satisfactory solution to this problem. We have also outlined some initial technical steps towards a solution. Though nulls and incomplete information have been studied exten- sively for traditional query languages over flat data (see Libkin [13] ACKNOWLEDGMENTS for a recent overview), these features appear to have attracted lim- ited interest in the setting of nested relational calculus or complex This work was supported by ERC Consolidator Grant Skye (grant object query languages. The only work in this direction we know number ERC 682315), and by an ISCF Metrology Fellowship grant of is from the early years of ‘non-first-normal-form’ databases [12, provided by the UK government’s Department for Business, En- 19]. Roth et al. [19] studied nested relations with several variants ergy and Industrial Strategy (BEIS). of nulls, including no-information, does-not-exist, and unknown, while Levene and Loizou [12] considered only a single ‘no-information’ REFERENCES [1] V. Benzaken and E. Contejean. A Coq mechanised formal semantics for realistic null, however neither of these approaches corresponds exactly to SQL queries: formally reconciling SQL and bag relational algebra. In CPP, 2019. the treatment of nulls in SQL, as formalized recently by Guagliardo [2] P. Buneman, S. Naqvi, V. Tannen, and L. Wong. Principles of programming with and Libkin [11]. complex objects and collection types. Theor. Comput. Sci., 149(1), 1995. [3] J. Cheney, S. Lindley, and P. Wadler. A practical theory of language-integrated Sum types (of which g? is a special case) were studied in an NRC query. In ICFP, 2013. setting by Wong [22]. Wong showed normalization and conserva- [4] J. Cheney, S. Lindley, and P. Wadler. Query shredding: efficient relational evalu- tive extension properties hold in the presence of sums and later ation of queries over nested multisets. In SIGMOD. ACM, 2014. [5] J. Cheney and W. Ricciotti. Comprehending nulls (extended version). Technical Giorgidze et al. [9] showed that nonrecursive algebraic data types report, arXiv, 2021. (i.e. n-ary labeled sums) can be implemented in NRC by mapping [6] E. F. Codd. Extending the database relational model to capture more meaning. ACM Trans. Database Syst., 4(4):397–434, 1979. such datatypes to nested collections. However, for the purposes of [7] E. Cooper. The script-writer’s dream: How to write great SQL in your own normalizing queries and generating SQL, the latter approach has language, and be sure it will succeed. In DBPL, 2009. the disadvantage that query results would use nested collections to [8] E. Cooper, S. Lindley, P. Wadler, and J. Yallop. Links: web programming without tiers. In FMCO, 2007. represent options, requiring a further flattening or shredding step [9] G. Giorgidze, T. Grust, A. Ulrich, and J. Weijers. Algebraic data types for possibly resulting in executing several SQL queries [4, 21], which language-integrated queries. In DDFP, pages 5–10, 2013. is not needed in our translation. General sum types can also be sim- [10] J. Grant. Null values in SQL. SIGMOD Rec., 37(3):23–25, Sept. 2008. [11] P. Guagliardo and L. Libkin. A formal semantics of SQL queries, its validation, ulated using options, e.g. by representing g + f as h! : g?, ' : f?i. and applications. PVLDB, 2017. Implementing sum types using nulls is possible future work. [12] M. Levene and G. Loizou. Semantics for null extended nested relations. ACM Trans. Database Syst., 18(3):414–459, 1993. In this paper we have focused on nulls in a conventional NRC [13] L. Libkin. Incomplete data: what went wrong, and how to fix it. In PODS, pages with a single collection type, e.g. homogeneous sets or multisets. In 1–13, 2014. SQL, which contains operators with both set and multiset seman- [14] R. Okura and Y. Kameyama. Language-integrated query with nested data struc- tures and grouping. In FLOPS, pages 139–158, 2020. tics, as well as grouping and aggregation, nulls interact with sev- [15] Quill: Compile-time language integrated queries for Scala. Open source project. eral other features, such as multiset difference and aggregation, of- https://github.com/getquill/quill. ten in counterintuitive ways [1, 11]. Our focus has been on seman- [16] W. Ricciotti and J. Cheney. Mixing set and bag semantics. In DBPL, pages 70–73, 2019. tics of NRC queries in the presence of nulls. We conjecture that nor- [17] W. Ricciotti and J. Cheney. Strongly normalizing higher-order relational queries. malization and conservativity results hold for NRCnull and NRCopt In FSCD, pages 28:1–28:22, 2020. [18] W. Ricciotti and J. Cheney. Query lifting - language-integrated query for hetero- facilitating their translation to flat SQL queries. We are also inter- geneous nested collections. In ESOP, pages 579–606, 2021. ested in generalizing our treatment of nulls to queries over hetero- [19] M. A. Roth, H. F. Korth, and A. Silberschatz. Null values in nested relational geneous (set/bag) collections [16], higher-order functions [7, 17], databases. Acta Informatica, 26(7):615–642, 1989. [20] D. Syme. Leveraging .NET meta-programming components from F#: integrated grouping and aggregation [14], and to shredding queries that pro- queries and interoperable heterogeneous execution. In ML Workshop, 2006. duce nested results into multiple SQL queries [4, 18] and in ex- [21] A. Ulrich. Query Flattening and the Nested Data Parallelism Paradigm. PhD thesis, tending NRC to allow nulls at record and collection types. Such University of Tübingen, Germany, 2019. null [22] L. Wong. Normal forms and conservative extension properties for query lan- extensions seem possible but not necessarily straightforward. For guages over collection types. J. Comput. Syst. Sci., 52(3), 1996. example, should a union of a null collection with another be null, or should the result retain partial knowledge about the known el- ements?
8 CONCLUSIONS Incomplete information is needed in most real database situations. While incomplete information has been studied extensively both in theory (e.g. certain answer semantics [13]) and practice (e.g SQL’s pragmatic, but complex treatment of nulls and three-valued logic [11]), almost all such work has focused on conventional, flat relational data and queries, not nested relations. This gap in the literature is particularly noticeable where clean query languages such as NRC Comprehending nulls
A TYPING RULES B REWRITE RULES A.1 Rules for NRC B.1 Common rules
−→ −→ h...,ℓ = ",...i.ℓ { " 5 (+ ) { 5 (+ ) Γ Σ = J K G : g ∈ (2) 1 { { Ø{∅|G ← "} ∅ Ø{"|G ← ∅} ∅ Γ ⊢ G : g Γ ⊢ 2 : 1 {"|G ←{# }} { " [# /G] −→ ′ Ø Σ(5 ) = 1= ⇒ 1 (Γ ⊢ "8 : 18 )8=1,...,= −−→ ′ { Γ ⊢ 5 ("=) : 1 Ø{" ∪ # |G ← '} Ø{"|G ← '}∪ Ø{# |G ← '}
−−−−→ { (Γ ⊢ "8 : g8)8=1,...,= Γ ⊢ " : hℓ= : g=i 8 ∈ {1,...,=} Ø{"|G ← # ∪ '} Ø{"|G ← # }∪ Ø{"|G ← '} −−−−−−→ −−−−→ Γ Γ ⊢ hℓ= = "=i : hℓ= : g=i ⊢ ".ℓ8 : g8 Ø{"|~ ← Ø{'|G ← # }} { Γ ⊢ " : g Γ ⊢ " : {g} Γ ⊢ # : {g} Ø{Ø{"|~ ← '}|G ← # } (if G ∉ FV(")) Γ ⊢ ∅ : {g} Γ ⊢{"} : {g} Γ ⊢ " ∪ # : {g} { Ø{"|G ← # where !} Ø{"|G ← # } where ! Γ, G : f ⊢ " : {g} Γ ⊢ # : {f} Γ ⊢ " : {g} " where true { " " where false { ∅ Γ Γ empty bool ⊢ Ø{"|G ← # } : {g} ⊢ (") : ∅ where ! { ∅ Γ ⊢ " : bool Γ ⊢ # : g Γ ⊢ # : g 1 2 (" ∪ # ) where ! { (" where !)∪(# where !) Γ ⊢ if " then #1 else #2 : g Ø{"|G ← # } where ! { Ø{" where !|G ← # } (if G ∉ FV(!)) A.2 Additional rules for NRCopt (" where !1) where !2 { " where (!1 ∧ !2) For NRCopt the following typing rules are added to those of NRC: { empty " empty (Ø{hi|G ← "}) (if " is not relation-typed) −−−−−−−−−−−−−−−−−−−−−−−−−−→ if ! then " else # { hℓ = if ! then ".ℓ else # .ℓi −−−→ Γ ⊢ " : g (if ",# have type hℓ : fi) Γ ⊢ none : g? Γ ⊢ some(") : g? B.2 Additional rule for NRC For NRC, the following rewrite rule is added to the common rules:
if ! then " else # { (" where !)∪(# where ¬!) (if ",# have type {f} and # ≠ ∅)
Γ ⊢ " : g? Γ ⊢ #1 : f Γ, G : g ⊢ #2 : f Γ ⊢ case " of (none ⇒ #1 | some(G) ⇒ #2) : f B.3 Additional rules for NRCopt For NRCopt, the following rewrite rules are added to those of NRC:
A.3 Additional rules for NRCnull case none of (none ⇒ #1 | some(G) ⇒ #2) { #1 For NRCnull the following typing rules are added to those of NRC: case some(") of (none ⇒ #1 | some(G) ⇒ #2) { #2 ["/G]
We believe that the case analysis rules, being a special case of Γ ⊢ " : 1 sum types, are well behaved and preserve the strong normalization Γ ⊢ null : 1 Γ ⊢ isNull(") : bool property. James Cheney and Wilmer Riccio i
B.4 Additional rules for NRCnull Notice that the if-splitting rule is refined to account for the case where the condition is null; this additional check preserves For NRCnull, the following rewrite rules are added to the common rules: Girard-Tait reducibility and we thus believe the rewrite system to " where null { ∅ be strongly normalizing.
if ! then " else # { (" where !)∪(# where (isNull(!)∨¬!)) (if ",# have type {f} and # ≠ ∅)
isNull(null) { true isNull(2) { false
5 (. . . null . . .) { null (" where null) { ∅