Qt and Sqlite … Squeeze Every CPU Clock to Get Faster

Qt And SQLite … squeeze every CPU clock to get faster

Florence, April 2nd 2019 DBMS SQLite is a database management system

Qt and SQLite - Gianbattista Gualeni

2 Database management system

A database is an organised collection of data, generally stored and accessed electronically from a computer system. Where databases are more complex they are often developed using formal design and modeling techniques.

The database management system (DBMS) is the software that interacts with end users, applications, and the database itself to capture and analyse the data. The DBMS software additionally encompasses the core facilities provided to administer the database. The sum total of the database, the DBMS and the associated applications can be referred to as a "database system". Often the term "database" is also used to loosely refer to any of the DBMS, the database system or an application associated with the database.

Wikipedia

Qt and SQLite - Gianbattista Gualeni 2019 3 Not really new!

This was actually a DBMS used 2500 years b.C. I miss the programmer name, sorry. I don’t ﬁnd the project on GitHub!

Qt and SQLite - Gianbattista Gualeni 2019 4 Why SQLite

There are 2 Big Family •SQL •NOSQL

Among SQL we have: •Database server (MySQL, Oracle …) •Database Library (in-process database)

SQLite is a library, hence lower latency Speed? It depends. Implementation? Simpler, no excuse…

Qt and SQLite - Gianbattista Gualeni 2019 5 SQLite and Qt How we can use a database from Qt application

Qt and SQLite - Gianbattista Gualeni 2019 6 Fast lane to SQLite

The fastest way to play with SQLite database? DB Browser for SQLite

Qt and SQLite - Gianbattista Gualeni 2019 7 Integration

Add sql module to .pro

Import include

Qt and SQLite - Gianbattista Gualeni 2019 8 addDatabase / removeDatabase

A lot of Static! (Danger, High Voltage)

Qt and SQLite - Gianbattista Gualeni 2019 9 EXAMPLE A simple image database application

Qt and SQLite - Gianbattista Gualeni 2019 10 My Sample Project

1Milion Images… Again I create images using a Python script (6 hrs!) Images are organised in sub-folder 1000 folders x 1000 images = 1M 4kB for image = 4GB

Qt and SQLite - Gianbattista Gualeni 2019 11 Who I am /1

I am the Author of the book: Qt 5 Quanto Basta

And I am a Qt enthusiastic user since early 2000.

Currently I work for Datalogic as Machine Vision Product Specialist and I use Qt to play with images, logs and … private projects.

I’m working on another book? Yes! Fast and Furious

Qt and SQLite - Gianbattista Gualeni 2019 12 Who I am /2

Qt and SQLite - Gianbattista Gualeni 2019 13 Update the database Writing images info into the database

Qt and SQLite - Gianbattista Gualeni 2019 14 Create the database

DB Browser images table 4 cols Just text

Qt and SQLite - Gianbattista Gualeni 2019 15 INSERT INTO

Static again

For each folder, for each ﬁle in the folder…

Qt and SQLite - Gianbattista Gualeni 2019 16 SPEED?

Really, not the right word! 5.4 ms for each image 1h 30 min estimated for 1M images or 18.5 images for second

Using Core i7, 8th gen, 4 GHz turbo boost

Qt and SQLite - Gianbattista Gualeni 2019 17 Optimisation Can we speed-up this shame?

Qt and SQLite - Gianbattista Gualeni 2019 18 Speed Limit ?

If I leave just the ﬁle scan, query string creation but I disable the query?

So this means that: 1. We can run up to 50k images/sec 2. The limiting factor is the access to the database

Qt and SQLite - Gianbattista Gualeni 2019 19 Optimisation

During optimisation I use a sub-set of just 100 images

I test every possible options, but I’m going to save you from my boring experience.

Qt and SQLite - Gianbattista Gualeni 2019 20 Journal_mode = WAL

Write Ahead Log

1.955 ms means 511 Images / sec faster than 18.5 images /sec but far from the speed limit (50.000)

Qt and SQLite - Gianbattista Gualeni 2019 21 Journal_mode = MEMORY

We can save the backup in memory and not on a separate ﬁle

1709 ms means 585 Images/sec faster than 18.5 images/sec but far from the limit (50.000)

Qt and SQLite - Gianbattista Gualeni 2019 22 PRAGMA SYNCHRONOUS OFF

Don’t save to disk in synchro

0.887 ms means 1127 Images / sec faster than 18.5 images /sec but far from the limit (50.000)

Qt and SQLite - Gianbattista Gualeni 2019 23 BEGIN / END TRANSACTION We lock the database for our transaction till the end!

0.114 ms means 8772 Images / sec faster than 18.5 images /sec but far from the limit (50.000)

Qt and SQLite - Gianbattista Gualeni 2019 24 Playing with the MILION What happens when we play with the real beast?

Qt and SQLite - Gianbattista Gualeni 2019 25 Can we use multiple optimisations?

This is the table with optimisation results BEGIN / END TRANSACTION is way more better!

Qt and SQLite - Gianbattista Gualeni 2019 26 Sure!

Just apply BEGIN TRANSACTION as last. This is the optimal conﬁguration I test:

What happens with 1M Images?

Qt and SQLite - Gianbattista Gualeni 2019 27 1AMAZING MILION

Did you see the number? 0.016 ms This means 62k images per second Faster than the speed limit. Why so fast? Maybe the turboboost is more efﬁcient on a long run.

Qt and SQLite - Gianbattista Gualeni 2019 28 Last but not least: Bind

We need 2 steps: prepare and bind

Qt and SQLite - Gianbattista Gualeni 2019 29 1AMAZING MILION /2

This is INSANE! 0.011 ms per image

This means 90k images per second. Considering we start from 18.5…

Qt and SQLite - Gianbattista Gualeni 2019 30 1AMAZING MILION /2

Maybe I was not clear

90.000 Images per second!

Qt and SQLite - Gianbattista Gualeni 2019 31 Let the icon play What happen if we insert images?

Qt and SQLite - Gianbattista Gualeni 2019 32 Insert small and big icons

We add 2 Binary Large Object to store one small icon and one thumbnail. In my test I use

Qt and SQLite - Gianbattista Gualeni 2019 33 Creating Icons /1

Read the ﬁle, resize and write as JPG into a Byte Array

Just for ﬁrst image, because we focus on database not on image conversion

Qt and SQLite - Gianbattista Gualeni 2019 34 Creating Icons /2

This process is painfully slow, compared to the incredible level we reach

But this optimisation is out of scope

Can I use it in the next QtDay?

Qt and SQLite - Gianbattista Gualeni 2019 35 INSERT INTO + BLOB

How can I insert a blob into a query? Using bind

Qt and SQLite - Gianbattista Gualeni 2019 36 Performances

Considering that now the resulting database is 4GB, instead of 160 MB, I expect the a slower image rate …

I think it is still incredible We have 0.024 ms for each image (4 seconds for 1M Images) Or 42k Images per second.

Considering the disk speed is 400 MB/s It takes 10 secs to write the ﬁle (and 14 to do the rest…)

Qt and SQLite - Gianbattista Gualeni 2019 37 And the answer is… 42 (thousand images per second)

Qt and SQLite - Gianbattista Gualeni 2019 38 VISUALIZATION Can we see the database content?

Qt and SQLite - Gianbattista Gualeni 2019 39 Can we see the database content?

Qt and SQLite - Gianbattista Gualeni 2019 40 Table View and Model

We need a tableView We need a QSqlQueryModel We need a delegate to paint icons

Then we need the update event

Qt and SQLite - Gianbattista Gualeni 2019 41 The Delegate

Qt and SQLite - Gianbattista Gualeni 2019 42 Search How fast is searching an image?

Qt and SQLite - Gianbattista Gualeni 2019 43 SELECT * FROM WHERE

"SELECT * FROM images WHERE FileName LIKE '%1234.png%' ORDER BY FileName DESC;"

Qt and SQLite - Gianbattista Gualeni 2019 44 Search 12 second to search all images that contains 1234 in the name!

Qt and SQLite - Gianbattista Gualeni 2019 45 Index to speed-up search

Qt and SQLite - Gianbattista Gualeni 2019 46 Index effect

Amazing!

Few milli seconds instead of 12 seconds.

Qt and SQLite - Gianbattista Gualeni 2019 47 Questions?

Qt and SQLite - Gianbattista Gualeni 2019 48