Automated Approach for the Importing the New Photo Set to Private Photo Album to Make It More Searchable

УДК 004.855 DOI: 10.18523/2617-3808.2020.3.141-148 D. Nikulin, O. Buchko AUTOMATED APPROACH FOR THE IMPORTING THE NEW PHOTO SET TO PRIVATE PHOTO ALBUM TO MAKE IT MORE SEARCHABLE

This paper focuses on describing several Informational Retrieval (IR) Multimedia Systems that work primarily with private photo albums and it provides comparison of IR Multimedia Systems in terms of functionality. The Informational Retrieval Multimedia systems, used for comparison are selected as the most known and widely used as well. The Informational Retrieval Multimedia systems compared are Mylio, Google Photo, Apple Photo, Synology Moments and digiKam. Moreover, based on these systems comparison, this paper provides the definition of the end-point user needs. Then it provides the identification and the prioritization (based on the author’s experience and opinion) the issues in private information Retrieval multimedia systems to be addressed by any technical solution. The following issues have been addressed as the most important from the end-point user’s perspective: First of all, it is “Not existing EXIF file metadata”.(It is divided into two sub issues: not existing taken date and GEO data for place it was taken at); as second, the linking a photo to an author person entity (Based on camera brand and model given as metadata); And as the last one, linking the set of photos to the predefined type of event, like birthdays, anniversaries, holidays, trips etc.(Based on tagging the photos in semi-automated way). For each issue, this document has provided the technical approaches and program algorithms (in a pseudo code) as well. Also, the using of relevant tools is provided if applicable. Each tool mentioned is described in a few words and given with a reference to read about. This document has also described the human involvement into the process, if a totally automated process is not possible. As a conclusion this paper describes the key points that were identified during the identification and addressing the issues. The conclusion also gives a proposal for a technical solution (that is required to be used during importing the new photo set into existing IR Multimedia systems) with context diagram that represents the user, its devices with multimedia data, the system to import the data from user multimedia devices and Informational Retrieval multimedia system to keep the whole private multimedia and to search the target photo set. Finally, it gives the definition of workflow steps to describe the main activity flow (with human involvement and software as well) to be implemented in a technical solution in future. Keywords: IR Multimedia Systems, Metadata, Photo Album.

Definition of the user needs Ordinary people also might be interested in such in Multimedia IR systems kind of engines. For example, let us calculate the size of a photo album for an average family. Here This chapter provides a review of the areas of po we assume that an average family consists of 3 per- tential intended usage and implementation of Multi sons. Also, let us assume a family owns 2 devices media IR systems (that search through the non-textu- with a camera. On average, one person with a photo al information) in different fields. In corporate seg- device can take 5 photos per day, and thus, the entire ment, the target audience might be: family takes 10 photos per day (sometimes it might – people of different professions who need access be 0, but sometimes it might be hundreds). It means to images, like medical professionals searching that one family can take more than 3 thousand pho- for medical images or architects requiring the tos annually, and in 10 years – more than 30 thou- image examples to create the buildings; sand of photos. And, if such a family does not have – video content engineers, when looking for spe- a good tool to search through the 30 thousand of cial video segments and movies by their titles; photos of their photo album, the process of finding – journalists who work with camera to create mul- of target photos might be very difficult. timedia news content; From an average user’s point of view, there are – photographers that provide their services to the following requirements to be met by the Multi companies; media IR system: – car engineers who need photographs and sound – it should provide a container to keep different of car engines. types of multimedia;

– it should be possible to represent any kind of – Objects in it (forests, pets, cars); multimedia information (photo, video, and audio); – Events description (my father’s birthday, a trip – it should provide a simple way to edit the multi- to London); media data; – Commentary (any text comments); – it should be possible to search through the con- – Rating or marks (from 0 to 10); tent quickly and easily; – A combination of color, texture (a lot of blue – it should allow filtering of information by sev- (like the sky)); eral sets of criteria; – Emotions attached to an event (smiles, etc.) – relevance feedback shall be implemented; – Similar photos. – it should include indexing and cataloguing func- Let us provide the examples of requests based on tions; photo properties/attributes that people might make – it should be possible to find information by ex- when searching for certain photos: ample of image; – it should include a function of browsing through – My photos from April 2020; the search results. – All photos when I am at home; It is clear enough how to keep and represent the – My father’s anniversaries; multimedia information. However, the question is – Photos when I was in Italy; what kind of search requests are possible in this case. – Photos where I am in the forest; There are many different technical photo pro – All my cat/dog photos; perties/attributes, such as Aperture, Camera make, – Photos with my grandmother from 2000 till 2010 Camera model, etc. (see the figure below). All of at home; them could be used in the search. – Photos made by my brother.

Overview and comparison of the private Multimedia IR systems

This chapter provides an overview and comparison of several private Multimedia IR systems. There are photo organizers that provide the functions similar to private Multimedia IR systems. Therefore, they are chosen as a subject for the overview and comparison. They currently perform the search in photo set by organizing the photos based on their date, location, categories, etc. They suggest to create a folder structure and put the files on it. The folder name can be some- thing like Date_Shoot-Type_Event Name. It can also include a location or any other information the user wants. In such a way the user can create subfolders based on the number of photographers or cameras. The idea is to avoid photos having the same file name in the folder [5]. This [5] provides the list of Photo Organizer for professional pho- tographer and those organizers can be used by a Figure 1. Photo file properties/attributes usual user with a big photo album. There are even relevant online courses available In this paper we are focusing only on those prop- to guide the user how to organize the photos. erties/attributes that might be used in search by or- The multimedia IR systems are selected for the dinary people. They might be: overview according to the following set of criteria. – Data and time when the photo is taken (e.g.: One or several of them: 2020/04/20 21:22:21); – should be online; – The people in it (e.g.: my father, friend); – should be offline; – The place where it is taken (for instance, longi- – shall be free; tude + latitude or “my parents’ home”); – shall be paid. D. Nikulin, O. Buchko. Automated Approach for the Importing the New Photo Set to Private Photo Album to Make it More Searchable 143

Table 1 Comparison of IR multimedia systems by features coverage

Feature Google Apple Synology digiKam Feature description Mylio Priority Photo photo Moments [4] Keeping and presenting the Multimedia information 10 + + + + + Editing the Multimedia information 8 + - + + + Indexing and cataloguing 10 + + + + + Browsing the filtered information 10 + + + + + Search and filtering by date/time 9 + + + + + Search and filtering by location 10 + + + + + Search and filtering by people 8 + + + + + (using facial recognition) Search and filtering by objects 2 - - - + - Search and filtering by events 7 Tagging Tagging Tagging Tagging Tagging Search and filtering by votes/marks/user feedback 5 + + + + + Search and filtering by comments 3 + + + + + Search and filtering by emotions 2 - - - - - Search and filtering by similar photos 1 - + + - + Search and filtering by a combination of color, texture 2 - - - - + Search and filtering by a photo author 7 Tagging Tagging Tagging Tagging Tagging

The comparison table shows the following as- – only some systems perform the search by a com- pects: bination of color or texture; – almost all systems are able to support storage, – the search by photo author is successful only by editing, indexing, browsing and cataloguing the applying tagging. multimedia data; The more detailed comparison of photo organiz- – all systems are able to search using location data ers is provided by Wikipedia and available here [1]. and date/time metadata; – all systems are able to define a person using face Identification and prioritization recognition technology; of the issues in IR multimedia systems – only several systems can recognize the objects on the photos; This chapter dwells upon the identification and – searching by event is possible only through tagging; prioritization of the issues found during the over- – search by feedback and by comments ability is view and compassing the current IR multimedia provided by all systems; systems. – all the systems are unable to search by emotions; The identification and prioritization of the issues – only some systems can make the search by a si was performed based on private needs of the paper’s milar photo; author. Table 2 Identified and prioritized issues in IR multimedia systems

# Description Priority

1 It is undefined how to link the identified faces (thousands) to the person entities that can be used 5 in the search 2 It is undefined how to link the person entity to heritage tree node 2 3 It is undefined how to link a set of photos to the predefined type of event 7 4 It is undefined what to do ifmetadata does not exist 9 5 It is undefined how to integratetagging approach with other systems 4 6 Filtering by objects (dogs/cars/trees) is not widely implemented 3 7 It is undefined how to link a photo to an author person entity 8 144 ISSN 2617-3808. Наукові записки НаУКМА. Комп’ютерні науки. 2020. Том 3

Addressing the issues That metadata is usually kept inside of each file in private IR multimedia systems as a header with Exif format. There are several other formats of metadata like IPTC or XMP, but in this The following three issues with the highest pri- paper we are focusing mostly on Exif as the most ority are chosen to be addressed in this paper: common metadata header. – It is undefined what to do if metadata does not The first step is to identify if location or date/ exist; time data is not present. That might be done using – It is undefined how to link a photo to an author the tools like Apache Tika or digiKam[4]. person entity; As soon as the photos without the GPS and time/ – It is undefined how to link a set of photos to the date taken are identified, it is necessary to assign the predefined type of event. correct metadata to the photos in a manual or auto- The issues shall be addressed in the way men- mated way. Below is the description of the automated tioned above with a toolset available for a usual user way to update GPS and time/date of the photos. and not for the enterprises. To update the time/date of the photos, the fol- Snippets of pseudo code or tool screenshots ex- lowing approach might be used: amples will be provided to explain the approach in 1. To define the approximate date (not time) addressing the issues. when the set of photos was taken. For instance, The most automated approach possible should on 2020.05.11. be provided. In case there is a need to perform any 2. To put all the photos with assumed date taken manual steps, it should be justified and described as to folder with name “2020.05.11”. well. 3. Use the following algorithm to update the Exif time/date taken: Addressing the issues 1. For all photos inside of that folder (in our with not existing metadata case it is “2020.05.11”) 2. Do the following: Basing on the metadata inside of photo, the user a. put current photo file information into vari- might perform the search process by date/time and able info. location criteria. So, if location data or date/time b. compare info.lastModified() with assumed does not exist, the user is not able to search in the set date (encoded into the folder name). of photos by location and date/time requests. Those c. if assumed_date + 1 day > info.lastModi- requests are commonly and widely used, so it is fied() > assumed_date very important to identify if that data does not exist Call command line with “jhead -dsft “ + info. inside the photo. fileName()

Figure 2. Screenshot of digiKam[4] advanced search screen configured to find the photos withoutGPS data D. Nikulin, O. Buchko. Automated Approach for the Importing the New Photo Set to Private Photo Album to Make it More Searchable 145

Figure 3. The example of search results by the photo of a famous actor

If assumption about taken date/time was not cor- So, to assign GPS data the following algorithm rect – no photos will be updated and more attempts might be applied for each for photos without GPS need to be performed. coordinates: As a result of this algorithm, all photos with mo 1. Decrease a size of photo according to the web dification date/time that is inside of the assumed requirements. date will get Exif time/date taken that corresponds 2. Upload the photo to any webserver and save the to the “last modification time”. To updateExif meta- link to the photo. data we use “jhead” tool that is free to use. That al- 3. Call the Internet Search Engine API to get re- gorithm assumes that the user performing such ac- sults. “curl” tool might help in this and so the tion is aware of the assumed date when the photo request to Search Engine API will look like – was taken. As soon as date/time taken is attached to “curl.exe-G-k https://images.google.com/sear the photos, it will be possible to search those photos chbyimage?image_url=https%3A%2F%2Fi. by date/time criteria, because almost all multimedia imgur.com%2FHBrB8p1.png”. As a result, it search engines take into account the Exif date/time will return HTML or JSON with results that taken attribute. you need to parse in the next step. The next issue is assigning Exif geo data in au- 4. Inside of parsers the algorithm has to parse tomatic way. First, it should be mentioned that there HTML DOM to find special HTML-tag, class are some Internet tools that allow a user to reverse or id. To parse JSON it needs to find proper tag the images to get, for example, the person identifi by its path. cation by the photo or to get the geo position based 5. As soon as GPS coordinates from HTML or on photo details. For example, they include https:// JSON are obtained, the GPS data should be serpapi.com/google-reverse-image, https://images. assigned to the photo file with the help of the google.com/, https://mrisa.mage.me.uk/, etc. So, they already mentioned tool – “exiftool”. The com- look for the similarity in the photos and they return mand to assign the Exif geo data will look like actually. Most likely the researcher will need to pay “exiftool photoToSetGPS.jpg-gpslatitude=40. a fee to use such Internet engines. 25090-gpslongitude=75.132658-gpslatituderef= The local search engines can also provide the N-gpslongituderef=W”. same functionality. However, due to much smaller For not well-recognized places like private ho base, they return less accurate results. Still, they will mes, sea side or forest places (they look very simi- perform the task for the photos in famous and public lar), there is another approach to assign the GPS places. data that requires a bit of manual work: 146 ISSN 2617-3808. Наукові записки НаУКМА. Комп’ютерні науки. 2020. Том 3

1. Divide the photos into separate folders by The string representation of concatenation of grouping them by the same GPS position. “Camera Maker” and “Camera Model” can be a key 2. Define the exact geo position using the map to link the photo to the author. The following dic- service like Google Maps or OpenStreet maps tionary shall be created to support the linkage: for each photo set inside one folder. “Camera Maker” + “Camera Model“ to “author\ 3. For each photo file inside the folder with the author_name”. same geo position, call the “exiftool” com- For example: mand to set GPS data defined earlier like in “CanonCanonS3” - “author\dmytro_nikulin”; step 5 of the previous algorithm. “NokiaNokia6260i” – “author\vasia_nikulin”; As a result of these algorithm jobs, all photos “samsungSM-G965F” – “author\petia_nikulin”. will get Exif GPS data. Both the algorithms assume Having that dictionary, the following algorithm that the user performing such action is aware of the can be used to assign the photo to the author: assumed place where the photo was taken. As soon 1. Get the photo file; as GPS data is attached to the photos, it will be pos- 2. Extract the “Camera Maker” and “Camera sible to search those photos by position criteria, be- Model“ attributes; cause almost all multimedia search engines take 3. Put the extracted attributes to one string vari- into account the Exif GPS attributes. able CameraMaker&Model; 4. Try to search in the dictionary “Camera Ma Linking the photo to the author ker” + “Camera Model” to “author\author_ name” by CameraMaker&Model as a dic- Usually, during the search in the photo album tionary key; it is important to find the set of the photos taken by 5. If the search is successful – assign current a certain person. photo file tag to“author\author_name”. It is assumed that there is a predefined set of au- 6. If the search is not successful – thors – persons’ list in a form of tag set, and each 6.1. log the error/warning message with photo item of the list should be assigned to each photo. For file name, or example, “author\dmytro_nikulin; author\vasia_ni- 6.2. show the dialog to user to make it pos kulin; author\petia_nikulin”. sible to create a new item in the diction- It is also assumed that each photo file has Exif ary “Camera Maker” + “Camera Model” “Camera Make” and “Camera Model” attributes as- to “author\author_name”. signed. It might happen that “Camera Maker” + “Came- ra Model” key is not unique because of different people having the same device model. For example, the dictionary will be as follows: “samsungSM-G965F” – “author\vasia_nikulin”; “samsungSM-G965F” – “author\petia_nikulin”. In such cases user might be prompted to select the correct person from the list or a log message might be created to be analyzed after processing. As a result of that algorithm, all photos will get author tags assigned. The algorithm assumes a user to be involved in the preparation of the “Camera Maker” + “Camera Model“ to “author\author_name” dictionary and selection of proper photo author if the algorithm is unable to do it in automatic way. As soon as the author tags are attached to the photos, it will be able to search those photos by author.

Linking the set of photos to an event The current private IR multimedia systems can perform the search in photo album by the event if Figure 4. The example of the photo file with assigned appropriate tags are assigned to the photos. For ex- “Camera Maker” and “Camera Model” attributes ample, a photo file can be assigned the tag such as D. Nikulin, O. Buchko. Automated Approach for the Importing the New Photo Set to Private Photo Album to Make it More Searchable 147

“event/somebody’s anniversary”. In this case IR – The search attributes are the subjective ones multimedia system can find this photo by user search and might vary from other user needs to others. request: all photos with “event/somebody’s anni- – The current IR Multimedia systems don’t pro- versary”. vide a flexible and automated functionality to First of all, there should be a well-defined tags import the multimedia data into IR Multimedia structure. There are several Internet sources [2], [3] systems basing on user search needs. that describe the rules of creating a smart and flex- – Despite there are a lot of steps that can be au- ible tags structure. tomated, still a human involvement is re- The following tag structure can be used as a ref- quired. erence one: So, automation for importing the new photo set Event/ into photo album might be highly required feature With relatives/ for IR Multimedia systems as it can identify and Birthday/ address some issues that shall be fixed before user Wedding/ starts searching the data using IR Multimedia sys- NewYear&Christmas/ tem. With colleagues/ Organization1/ The next steps Birthday/ Basing on the statements above the following Wedding/ user needs are suggested to be covered: NewYear&Christmas/ – Asking a user for the media attributes he/she Organization2/ is going to use during the photo album search; With friends/ – Basing on the media attributes that user de- Birthday/ fined as important one, to keep search attrib- Wedding/ utes configuration; NewYear&Christmas/ – Basing on the search attributes configuration As soon as the tag event structure is defined, the to identify the gaps inside of multimedia data following algorithm to assign event to photos might that is going to imported into the current pho- be executed: to album; 1. Divide the photos in separate folders by group – As soon as any gap is identified to automati- ing by the same event. cally cover the missing data; 2. For each photo file inside the folder with same – To involve user interaction as soon as any gap event, assign the appropriate tag for file – for can not be addressed in automatic way. example, using the digiKam [4] tool. The software for importing the multimedia data to IR Multimedia systems can be developed to cover As a result of this algorithm, all photos will get the user needs described the event tag assigned. This algorithm assumes that The following picture shows the context diagram the user is involved in the preparation of the folder for solution proposed: structure with splitting the folders by common event and assigning a predefined event tag to the set of photos. As soon as the event tags are assigned to the photos, it will be possible to search those photos by event.

Conclusion As a result of issues identification and addressing, the following statements can be formulated: – The almost all IR Multimedia systems provide a user a possibility to search by a lot of parameter/attributes and provide a wide functionality for keeping and editing the photo album content. – The metadata is a main information basement Figure 5. The context diagram of software to search through the photo album for IR Multi that imports the multimedia data from user devices media systems. to storage of IR multimedia systems. 148 ISSN 2617-3808. Наукові записки НаУКМА. Комп’ютерні науки. 2020. Том 3

The work flow, according the figure, is fol 4. As soon as any missing information is de- lowing: tected it shall try to fill it in in automatic way. 1. The multimedia data from the user devices It shall prompt user if filling the missing data shall be placed inside of the input folder cannot be done in automatic way. where the software for import is running on. 5. Once any missing information is detected and 2. Once it is done the photo album owner shall filled in, it shall put the multimedia data into start the software for the import. the storage of IR multimedia systems. Storages 3. The software for import shall take the multi- of IR Multimedia systems are folders where media content from input folder and to iden- all user photo albums are kept and searchable. tify any missing data according to search pa- They might be Mylio, Google Photo, Apple rameter configuration. photo, Synology Moments or digiKam[4].

References

1. Image organizer. (2019). In Wikipedia. Retrieved from https:// 4. The digiKam Handbook | The digiKam developers team. (n.d.), en.wikipedia.org/wiki/Image_organizer. Retrieved from https://docs.kde.org/trunk5/en/extragear- 2. Tag your media. (n.d.). Retrieved from https://helpx.adobe.com/ graphics/digikam/index.html. elements-organizer/using/tag-media.html. 5. Boaz Eapen. (2019). Top 25 Photo Organizing Software and 3. Tagging Photos | Digital Photo Secrets. (n.d.). Retrieved from Apps of 2020. Retrieved from https://www.pixpa.com/blog/ https://www.digital-photo-secrets.com/tip/63/tagging-photos. photo-organiser.

Нікулін Д. М., Бучко О. А.

АВТОМАТИЗАЦІЯ ІМПОРТУВАННЯ НОВИХ ФОТОГРАФІЙ ДО ПРИВАТНОГО ФОТО-АЛЬБОМУ ДЛЯ ПОКРАЩЕННЯ ПОШУКУ

Цю статтю присвячено мультимедійним системам інформаційного пошуку, зокрема проведено їх порівняння з погляду функціональності. Розглянуто потреби користувача, визначено проблеми у приватних мультимедійних системах інформаційного пошуку. Проаналізовано найважливіші для кінцевого користувача питання: неіснуючі метадані, зв’язок фотографії з авторською особою, зв’язок набору фотографій із заданим типом події. Для кожної проблеми запропоновано підхід щодо вирішення, алгоритми та відповідні інструменти для використання. Також описано участь людини в цьому процесі, якщо повністю автоматизований процес неможливий.

Ключові слова: ІR-мультимедійні системи, метадані, фотоальбом.

Матеріал надійшов 13.05.2020

Creative Commons Attribution 4.0 International License (CC BY 4.0)