Partnership for Good Governance (2019-2021)

PROJECT Support to further strengthening the efficiency and quality of the judicial system in the Republic of Moldova

CEPEJ expert mission on improving the efficiency of courts and the quality of judicial services

Mission report

Based on the results of the consultants’ mission on 25-26 February 2020

Rado Brezovar (Slovenia) Anna Skrjabina (Latvia) Luigi Cipollini (Italy)

This document has been produced as part of a project co-funded by the European Union and the Council of Europe. The views expressed herein can in no way be taken to reflect the official views of either party.

Contents Contents ...... 1 INTRODUCTION ...... 3 USE OF KPIs TO EVALUATE THE IMPACT OF JUDICIAL REFORMS ...... 5 Objectives and background ...... 5 USE OF KPIs TO EVALUATE THE EFFICIENCY AND QUALITY OF COURTS ...... 9 Gathering data on KPIs and other statistics through the case management system ...... 11 Reporting on courts’ efficiency and quality ...... 12 IMPLEMENTATION OF A SUCCESSFUL OPEN REPORTING SYSTEM OF JUDICIAL DATA ...... 14 Open Reporting System - main characteristics ...... 14 Roadmap to the implementation of the open reporting system ...... 15 1. Working group ...... 15 2. Identify the data to be displayed ...... 16 3. Data gathering and data processing ...... 18 4. Tools for the implementation of the open reporting system ...... 18 5. Building the dashboard – data visualisation ...... 19 6. Storytelling ...... 21 7. Publication of the open reporting system and publicity ...... 21 Other opportunities ...... 22 Appendix I – Glossary and abbreviations ...... 23 Appendix II - Resources available on the web regarding visuals and dashboards ...... 25

2

INTRODUCTION

On 25-26 February 2020 a CEPEJ team including consultants from Slovenia, Italy and Latvia conducted a fact-finding and experience-sharing mission on judicial efficiency and quality in the Republic of Moldova. On this occasion discussions were conducted with national authorities on the collection, use and publication of judicial statistics by the example of the CEPEJ-STAT dynamic database1 and on a set of key performance indicators (KPIs) and other records to be applied with a view to evaluating the performance of national courts and the impact of judicial reforms in Moldova.

The mission was part of the project “Support to further strengthening the efficiency and quality of the judicial system in the Republic of Moldova” (the Project) aimed at the dissemination of CEPEJ tools in all courts of Moldova, based on the results of a court coaching programme implemented between 2015 and 2017. The participating CEPEJ consultants presented to their Moldovan peers the experience of the countries they represent in continuously improving the evaluation of courts’ performance and in promoting higher standards of efficiency, accountability and quality of judicial services.

Representatives from the key Moldovan partner institutions, such as the Ministry of Justice (MoJ), the Superior Council of Magistracy (SCM) and the Agency for Courts Administration (ACA), as well as development partners and non-governmental organisations actively involved in the reforms of the justice sector took part in the consultations. The major aim of the mission was to allow the CEPEJ consultants to assess the latest developments in the Moldovan judiciary and to develop a set of specific recommendations and proposals for activities with a view to supporting the implementation of the project’s objectives. While visiting the Chisinau Court of Appeal, the consultants got acquainted with the latest version of the ICMS applied in Moldovan courts2 and its advanced features.

The project “Support to further strengthening the efficiency and quality of the judicial system in the Republic of Moldova”, launched in December 2019 for a period of 24 months, is part of the Council of Europe / European Union Partnership for Good Governance 2019-2021 for Eastern Partnership countries (PGGII). It is funded by the European Commission and the Council of Europe and implemented by the Council of Europe.

In 2015-2017 six pilot courts (Supreme Court of Justice, Chisinau Court of Appeal, Cahul Court of Appeal, Riscani District Court of Chisinau Municipality, Ialoveni District Court, Soroca District Court) were engaged in testing the CEPEJ tools in their daily work. A report on the implementation of selected CEPEJ tools in pilot courts of the Republic of Moldova (CEPEJ-COOP(2017)3) was prepared, including a number of conclusions and recommendations on: - Judicial statistics; - Time management;

1 Please see the CEPEJ-STAT - Dynamic database of European judicial systems at https://www.coe.int/en/web/cepej/dynamic-database-of-european-judicial-systems 2 Information on the ICMS implemented in Moldovan courts can be consulted on http://aaij.justice.md/en- tech/content/ce-este-pigd and https://www.justitietransparenta.md/en/new-icms-implemented-national-courts/ 3

- Strengthening of the human resources; - Alignment of the functionalities and indicators introduced in the ICMS; - Approaching court users (surveys) and communication; - Dissemination of CEPEJ methodology at the national level etc.

According to the Project’s work plan, under the outcome 1 (the efficiency and quality of courts is enhanced through the application of CEPEJ tools at national level, while the mechanisms for judicial transparency and accountability are further consolidated) the following outcomes are foreseen: 1. Improved public access to the database of judicial statistics, based on the concept of CEPEJ- STAT database. 2. Increased skills and know-how for the collection and analysis of judicial statistics and ensuring qualitative data for policy-making and managerial purposes. 3. Improved court management and capacity of courts to deliver better quality of services.

The Project’s first Steering Committee meeting carried out in December 2019 stressed that, in the light of the expected result, the emphasis shall be put on: - continuing the discussions on the use of KPIs and statistics to evaluate court performance. In this regard it could be proposed to review the best experiences in usage of IT (introduction of the indicators into the ICMS), business intelligence (BI) tools for time management and efficiency; - the discussion on the distribution of resources based on objective data and indicators is an important element on the way to performance-based budgeting. Once discussing the use of indicators, attention should be paid to assessing the impact of implemented reforms; - in discussing the judicial timeframes, the recent developments by the CEPEJ SATURN (Centre for judicial time management) is to be taken into account, including the advancements on the analysis on case weighting methods.

The highlights of the activities conducted with the participation of the CEPEJ team on 25-26 February 2020 are the following: - A review of the results of the court coaching programme in 2015-2017 from the perspective of the main national stakeholders has been provided. The CEPEJ consultants were informed on the latest developments as regards the use of CEPEJ-recommended indicators and reporting on courts performance, on the current priorities of main stakeholders provided. - New CEPEJ instruments on the efficiency and quality of courts were presented, including CEPEJ-STAT tool and its potential for the use by national authorities. - The most recent version of ICMS – the case management system of the Moldovan judiciary presented. It has been discussed how ICMS supports implementation of CEPEJ-indicators and facilitates court management (case management, including time management; distribution of resources among and within courts; statistical reporting and analysis of court performance etc.). - National experiences in the evaluation of the results of judicial reforms in the Republic of Moldova and the use of KPIs for this purpose was presented. As well as plans under the Draft Strategy for Ensuring the Independence and Integrity of Justice Sector for 2020 - 2023. - Were provided examples of applying clear and comprehensive KPIs with a view to planning judicial reforms and evaluating their results in European states (Italy, Slovenia, and Latvia).

4

Once a system of gathering and monitoring of judicial KPIs is operational and regular reporting by key actors is put in place, the respective KPIs may be used as an instrument to: - Evaluate objectively the workload of courts and judges; - Assess the baseline and set targets for judicial efficiency and quality; - Evaluate the impact of the implemented judicial reforms; - Foresee further developments and plan related measures.

USE OF KPIs TO EVALUATE THE IMPACT OF JUDICIAL REFORMS

Objectives and background

It is important to promote the concept of a national comprehensive and standardised statistical reporting. This will provide an opportunity for the authorities such as the MoJ and the SCM of the Republic of Moldova to take decisions related to court administration on the basis of objective data (statistics, KPIs) related to human resources, budget, workload, backlogs and quality. This reporting also strengthens the position of MoJ and SCM in taking policy decisions and in evaluating the impact of justice reforms.

There seems to exist a clear and in-depth understating on the part of the MoJ, including the Agency for Court Administration (ACA) and the SCM on production and monitoring of KPIs. By a decision of the SCM of 19 December 2017 (no. 854/37)3, 17 KPIs were approved for the purpose of their implementation through the ICMS: 1. Ratio of cases cleared in the reported period4; 2. Clearance rate (CR); 3. Calculated time of disposition (DT); 4. Age of pending cases; 5. Ratio of timely examination of cases (or of cases cleared within the established deadlines); 6. Appeal ratio (AR); 7. Ratio of quashed decisions; 8. Ratio of modified decisions; 9. Case per judge (CPJ); 10. Efficiency rate (ER); 11. Staff per judge ratio (SPJ); 12. Cost per case (CPC); 13. Ratio of adjourned court hearings; 14. Case per staff (CPS); 15. Staff commitment; 16. Ratio of cases resolved after one court hearing;

3 See the Decision of the SCM in Romanian language at https://www.csm.md/files/Hotaririle/2017/37/854-37.pdf 4 This ratio is different from CR as it compares the number of cleared cases with the number of cases on the court’s register during the reporting period - both pending at the beginning of the period and registered during that period. This ratio cannot reach 100% because, normally, there will be pending cases at the end of a reporting period. In the Republic of Moldova this ratio is being used historically and is therefore easier understood by judges. 5

17. Satisfaction of court users.

The use of KPIs allows establishing a system in which, in the framework of annual reporting, available data is used to predict the workload and take the necessary decisions. Normally, all the information is available in the ICMS for further assessment. There is a possibility, with the potential for further development, to produce reports automatically using BI. According to the Agency for Court Administration, the work in this direction is on-going.

The aspects of disproportionate workload and of the necessity to have an in-depth evaluation in the context of the implemented judicial map reform were reported by Moldovan counterparts in the framework of the mission.

During the 2015-2017 project it was concluded that the court system of Moldova has reached a good record regarding the timeliness of court procedures and should make efforts to maintain and to possibly improve it. As the workload of courts is constantly rising, it is necessary to take preventive measures and anticipate any deficit of capacities within individual courts.

The appeal ratios reflect the limited potential of judicial statistics in view of assessing the quality of judicial decisions and it has been suggested to disseminate their analysis throughout the court system.

The objective of using KPIs to evaluate the impact of judicial reforms shall be put in line with the foreseen and implemented actions. The draft Strategy for Ensuring the Independence and Integrity of Justice Sector (SEIIJS) for 2020-20235 foresee a direction for the reforms in the coming years. The document provides continuity for objectives pursued by the Strategy for Justice Sector Reform implemented between 2011 and 2018. A number of reforms were taken during 2011-2018 under the umbrella of the mentioned document, inter alia: - Strengthening the self-governance capacities of the judiciary (SCM and its subordinated bodies). - Revision of the judicial map.

There is a number of activities foreseen for the new period, being mentioned that major reforms and legislative changes have been introduced during the previous period, and that the activities of the new SEIIJS shall be targeted at consolidating the positive results. It is also stressed that an essential element of the justice reform is the mechanism to assess the efficiency of its implementation, which is not possible without establishing clear and measurable performance indicators6. Several “progress indicators” to be used in order to measure the implementation of the SEIIJS are listed non-exhaustively. Along with some performance indicators, such as user satisfaction and level of trust among legal professionals, are listed several sources of verification. From this point of view, the draft SEIIJS may be further improved through references to KPIs recommended by the CEPEJ and by clearly separating performance indicators and sources of verification.

5 Please see under http://www.justice.gov.md/slidepageview.php?l=ro&idc=733 6 Strategy for Ensuring the Independence and Integrity of Justice Sector for 2020-2023, Chapter V “Estimation of the progress, impact and costs of implementation” 6

The reporting and monitoring procedure is established within the institutional framework with the objective to ensure the smooth implementation of the SEIIJS for 2020-2023. This structure is to be used as the main forum to discuss, measure and identify the impact of reforms. It is suggested to amend the progress indicators by introducing the KPIs which may be produced by the SCM or the ACA with the focus on following indicators: - Effectiveness (clearance rate, forecasted disposition time, age of pending cases) - Efficiency (case per judge, case per staff, cost per case) - Quality (appeal ratio, ratio of quashed and modified decisions)

As example it is proposed to discuss the possibility to evaluate on a regular basis the following KPIs linked with the specific objectives under SEIIJS:

Specific objective KPIs Objective 1.1. (SEIIJS) Strengthening the independence and Case per judge administration of the judiciary and the prosecutor’s office Case per staff Ensuring the court with the needed number of judges and Staff per judge supporting staff (with a link to specialisation) Positive impact on Clearance Rate and Disposition Time. Objective 1.2. (SEIIJS) Strengthening the self-administration, Cost per case7 independence and accountability of legal professionals Establishing clear rules for rate formation

Objective 2.2. (SEIIJS) Continuing the Process of Optimisation the Cost per case Judiciary and Prosecution Positive impact on Clearance Continuing the process of optimisation of courts Rate, Disposition Time and Age of pending cases. Impact on quality indicators to be monitored.

Objective 3.1. (SEIIJS) Improving the Efficiency of Justice Sector Clearance Rate, Disposition Procedure (simplification and streamlining of the procedures) Time, Age of pending cases. Amending the procedural and enforcement legislation to simplify Number of incoming cases v. and improve the efficient procedures data available on the usage of Identifying extrajudicial/administrative mechanisms for ADRs adjudicating certain categories of cases

Objective 2.3. (SEIIJS) Improving the quality of judicial documents Appeal ratio, ratio of quashed and uniformization of judicial practice and modified decision. Establishing criteria for quality and clarity of judicial documents Positive impact: level of the satisfaction among court users

7 Providing a clear link between the resources needed and the resources gathered through the court fees 7

Taking the decisions on reforms based on an in-depth analysis of data and KPIs on a long-term basis should result in: - Fair distribution of the workload - Optimised allocation of resources - Enhanced quality of court decisions - More uniform judicial practice and case law - Random and balanced distribution of cases - Enhanced specialisation of judges - Reduced length of the proceedings - Raising trust in the judiciary.

The involvement of the judiciary in planning of the judicial reforms is another important element to be mentioned and to be ensured at the largest extent possible (within the institutional setup for the evaluation of SEIIJS and on ad hoc basis).

The regular monitoring and evaluation of the KPIs (with the focus on costs per case) allows to develop performance-based budget and to optimise the resources along with monitoring the quality of the procedure.

Recommendation 1: KPIs shall be used as the instrument to evaluate the impact of implemented judicial reforms and foreseen developments and advancement in the framework of annual reporting system. The annual reporting system shall be established in the way to allow the MoJ (ACA) and the SCM to assess and to monitor the impact of the reforms. The reporting system could be introduced in the framework of the evaluation mechanism foreseen to evaluate the implementation of SEIIJS for 2020-2023. A broader involvement of the judiciary in the discussion shall be ensured.

Recommendation 2: The progress indicators mentioned in SEIIJS 2020-2023 shall be amended by adopting CEPEJ-recommended KPIs with the focus on: - Effectiveness (clearance rate, forecasted disposition time, age of pending cases) - Efficiency (case per judge, case per staff, cost per case) - Quality (appeal ratio, ratio of quashed and modified decisions)

As the imbalance in the workload among courts/judges and the necessity of an ex-post evaluation of the judicial map reform have been reported among the main problematical aspects, both elements shall be evaluated as the priority, linking with objectives and actions foreseen in the framework of SEIIJS for 2020-2023.

8

USE OF KPIs TO EVALUATE THE EFFICIENCY AND QUALITY OF COURTS

The purpose of the report is to cover the following topics: − The KPIs for motoring the efficiency and quality of the judiciary and of courts (based on the results of the court coaching and reports of the previous project; − Gathering data on KPIs and other statistics through the case management system; − Reporting on court efficiency and quality by all national courts and at national level.

The KPIs for motoring the efficiency and quality of the judiciary and of courts (based on the results of the court coaching and reports of the previous project

Moldovan judiciary implemented a modern concept of KPIs as a tool for motoring the efficiency and quality of the judiciary and of courts. CEPEJ standards, based on the CEPEJ Guidelines on Judicial Statistics (GOJUST) CEPEJ(2008)11 and the Saturn Guidelines for Judicial Time Management (3rd Revision) CEPEJ(2018)20R, were applied as a methodological basis. Some extra KPIs were set, based on the court coaching reports and the Final Report on the Implementation of selected CEPEJ Tools in Pilot Courts of the Republic of Moldova (please see Table 1 below).

CEPEJ set de facto standards with its Dynamic database of European judicial systems (CEPEJ-STAT).8 Every country implementing the Data Warehouse (DWH) and Business Intelligence (BI) solutions should consider the CEPEJ-STAT system as a guide. Some other examples of national implementation should be considered in this regard too.9 The EU publication of the “Quality of Public Administration - A Toolbox for Practitioners - Theme 7: Quality of justice systems” is directly linked to the topic of the quality of judicial administration.10

Table 1: KPIs (according to the Court Coaching Reports11)

Based on the CEPEJ Guidelines on Judicial Additional KPIs Statistics (GOJUST)12 1. CR – Clearance Rate 7. Judicial assistants / Judge ratio 2. DT – Disposition Time 8. Non-judicial staff / Judge ratio 3. CPJ – Case per judge (pending per judge) 9. Total number of staff / Judge ratio 4. ER – Efficiency rate (Solved cases / Judge 10. Cost per case (MDL) ratio) 5. ER – Efficiency rate (Solved cases / Staff 11. New cases / Judge ratios ratio)

8 Please see https://www.coe.int/en/web/cepej/dynamic-database-of-european-judicial-systems 9 Please see https://ec.europa.eu/digital-single-market/en/news/quality-public-administration-toolbox-practitioners 10 Please see https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8055&type=2&furtherPubs=no 11 Please see the Guide for the implementation of selected CEPEJ tools in the courts of the Republic of Moldova, p. 37 12 Please see https://www.coe.int/en/web/cepej/documentation/cepej-documents/guidelines 9

6. Duration of cases or Age of resolved cases 12. Appeal Ratio – Percentage of appealed decisions 13. Ratio of quashed or modified decisions 14.1. Ratios per categories of incoming cases in% 14.2. Ratios per categories of solved cases in% 14.3. Ratios per categories of pending cases in% 15. APC – Age of pending cases

In addition, a “user friendly” presentation of KPIs, which must be clearly understandable to the management and other possible target groups, is of utmost importance. Modern graphical presentation of statistical information, analyses and KPIs (but avoiding pies and 3D graphs) are recommended. Judicial dashboards should cover all the basic information concerning the functioning of the judiciary, which were covered through Tables 1, 2 and 3 of the spreadsheet used by the Moldovan pilot courts.13 Information (statistics, KPIs) concerning human resources, budget, workload, backlogs and the quality of judicial work shall be reflected.

A request for assistance in the field of determination of a dataset and of possible additional KPIs for the future DWH and “business” reports was expressed by Moldovan counterparts. It is of utmost importance to inspire the influential representatives and the top management in the judiciary to accept the concept of standardised reports and KPIs, in order to promote the monitoring of the efficiency and quality of the judiciary and courts. Currently more than 160 different statistical reports are generated without clear information on their impact on the performance of the judicial administration.

NOTE: A clear opinion was expressed by Moldovan interlocutors to implement the DWH/BI concept of reporting. This concept is used by the advanced national judicial administrations to facilitate the motoring of the efficiency and quality of the judiciary and of courts.

The implementation of the DWH/BI system is in the pilot phase. 17 different KPIs were set and some additional measures were elaborated (e.g. filed/resolved/pending per staff/judge). Setting additional KPIs shows clear understanding of the value of KPIs and the determination to improve judicial administration and court management. This will help Moldovan judicial administration to transform its work towards the predictive administration, which can react properly to evolving circumstances.

Recommendation 3: In an early phase of the DWH/BI implementation, it is recommended to develop dashboards which will contain all basic statistical information and KPIs (based on the CEPEJ GOJUST and Saturn Guidelines for Judicial Time Management).

13 ibidem 10

Gathering data on KPIs and other statistics through the case management system

The Integrated Case Management System (ICMS) was implemented in Moldovan courts at the end of 2008. An automated random distribution of cases was implemented in 2014. In addition, degrees of complexity for cases (case weighting) were implemented based on the categories of cases, number of parties, number and length of documents, number of witnesses and other elements, which give the score of complexity. The recommendation regarding the application of the concept of active judge/staff, based on the full-time equivalent (FTE), was implemented.

Since 2017, the ICMS version 5.0 has been used. The system is owned by the MoJ and held and maintained by the ACA.

The Moldovan judicial electronic system as a centralised solution consists of various components: 1. ICMS 2. eCourt (piloted) – for electronic filing the applications to courts 3. National portal – facilitating the access to information 4. AV solutions – AV conferences and recording 5. Payments module

The ICMS is used on daily basis and includes: 1. electronic record on the case 2. electronic record on parties (physical, legal entity) with interoperability functions to appropriate registers (civil, commercial) 3. electronic record on hearings 4. case summary 5. time management information 1. duration of the case 2. waiting time 3. action time 6. electronic schedule (linked to judge’s calendar) with warning functionality for the booked hearing room 7. search functionality 8. archive 9. anonymisation 10. publication 11. statistical reporting (with 160 different statistical reports) 12. performance of the judge

In order to provide interoperability with other governmental agencies, the mConnect is used. E.g. based on the ID numbers, the addresses of parties are updated automatically.

Currently the implementation of the DWH/BI concept is being piloted in three pilot courts. It is expected that the piloting period will finish at the end of March 2020. The pilot courts check daily all the data. Uploading of CEPEJ reports on the basis of the Excel spreadsheet used in 2015-2017 by

11

the pilot courts is enabled but needs to be further tuned and streamlined across the national judiciary. Special BI training is planned. ACA will draft curricula for different categories of users.

NOTE: The ICMS includes all the necessary information needed to deliver statistical reports in line with above mentioned CEPEJ Guidelines. DWH/BI approach is being developed and a beta version was presented at the Court of Appeal Chisinau. The system is used by all courts and by all staff members (some 15.000 users). A unique case number is used, which facilitates the automatic correlation of data and calculation of some very important KPIs concerning timeframes: e.g. duration of the procedure according to the reasonable time concept and the age of pending cases. A code list with more than 1.000 sub-categories of cases is used. This categorisation is not aligned to the CEPEJ approach, which defines only a limited number of subcategories under each category. This may cause problems in the identification of the necessary interventions in the business process and discrepancies between national reports and reports to the CEPEJ.

Recommendation 4: It is recommended to reconsider the solution of deploying the CEPEJ Excel spreadsheet, used to coach the pilot courts, into the ICMS.

Reporting on courts’ efficiency and quality

The Final Report on the Implementation of selected CEPEJ Tools in Pilot Courts of the Republic of Moldova, March 2017, CEPEJ-COOP(2017)3 (hereinafter “the Final report 2017”), which was developed as a part of the Project “Strengthening the efficiency of justice and support to lawyers’ profession in the Republic of Moldova” concluded in 2017 (hereinafter the 2017 Project) delivered several recommendations regarding statistics: 1. Continuation of implementing the CEPEJ methodology in statistical reporting; 2. Publishing of the statistical reports made in line with the CEPEJ methodology on courts’ websites; 3. Dissemination of the project’s results and the CEPEJ methodology within the Moldovan court system; 4. Improving the capacities and human resources for collection, analysis and reporting of statistical data; 5. Discontinuing the manual registration and collection of statistical data.

NOTE: The Guide for the implementation of selected CEPEJ tools in the courts of the Republic of Moldova (2nd edition), which is an appendix to the Final report 2017, shall be used as a tool for the dissemination of CEPEJ-recommended statistics and indicators. The main purpose of the Guide was to define the necessary steps and activities in order to facilitate the implementation of the conclusions and recommendations drawn up through the 2017 Project’s Court Coaching Reports concerning the implementation of the CEPEJ tools reporting.

Recommendation 5: It is recommended to draft a concrete Roadmap to facilitate the nationwide implementation of judicial statistical reporting. The aim of the Roadmap would 12

be the specification of needed activities in order to perform efficient implementation of statistics and indicators, which were defined and used by all pilot courts during the 2017 Project. The Roadmap should specify a set of activities, actions and deliverables, which will facilitate the nationwide implementation of collection, reporting, analysis of statistics, and also support the definition of policies, priorities, measures and monitoring of their implementation at different levels of the judiciary.

13

IMPLEMENTATION OF A SUCCESSFUL OPEN REPORTING SYSTEM OF JUDICIAL DATA

Data about courts and judges is data of public interest since it can reveal the efficiency and quality of the judiciary. Transparency of judicial data plays an important role in increasing public trust in the judiciary and in the fight against corruption. For this reason, Moldovan judicial authorities have the opportunity of making a drastic change in the way they communicate with the citizens, implementing an open reporting system (ORS) which displays figures about the judicial system.

Differently from a BI system, which is specifically designed for the management to evaluate the performance and efficiency of the judicial system, run regulatory impact analysis and in general to investigate what-if scenarios, an open reporting system is intended for the general public and it is specifically designed to portray an as much as possible accurate picture of the current state of the judicial system. It also improves the participation and engagement of the Moldovan citizens towards judicial matters.

An open reporting system differs form a BI solution in various ways: purposes, users, IT infrastructure, costs and investments, implementation, data reported, etc. The BI serves as a managerial tool whilst the ORS is a descriptive tool for illustrative purposes. The following table summarizes the main differences between the twos.

Business Intelligence Open Reporting System Top management General public Users Court management Researchers Single judges Media Infrastructure Integrated with the ICMS Integrated or stand-alone Depends on the IT infrastructure Limited budget if built in-house, Cost and already in use and the current without the intervention of external investments external providers providers Specific KPIs to address specific KPIs that illustrate the general KPI needs of the management functioning of the justice system It can provide highly detailed It only displays summarized Level of depth information information

Open Reporting System - main characteristics

Traditionally, reporting judicial data is performed through statistical tables only. However, in the last few decades, the reporting systems have become more refined and sophisticated. We moved from static statistical tables with limited or even non-existent interaction with the viewer to dashboards with a high degree of interactivity. The current reporting systems have become more sophisticated and now they make large use of interactive functions so that the users can navigate the underlying database as they please. The dashboards displaying judicial information are now

14

quite well spread in Europe and even though they have become more sophisticated, they are relatively easy to be implemented.

It must also be considered that moving from a set of static statistical tables to a more innovative reporting system is the only choice if we consider the tremendous increment of data produced by the judicial system as a whole. Simply reading through hundreds of tables in a traditional fashion becomes unfeasible and also counterproductive. Instead, the goal of an efficient organisation is to constantly search for those solutions that not only make the data look appealing but also make it easier to understand and interpret. In this perspective the ORS represent a major improvement.

Moreover, it is worth underling the fact that the ORS is not only a collection of data, but it represents an opportunity to communicate more effectively to the citizenship through a different narrative in a country where citizens have - on average - a negative attitude towards the judiciary. In this context the OCR will not only contribute to the openness and transparency of the judicial system but will also contribute to a better communication of the results achieved by the judiciary.

Roadmap to the implementation of the open reporting system

Implementing a successful ORS requires the involvement of different professional figures across the different stages of the project. In this document we will focus our attention to the following elements: 1. Working group; 2. Identify the data to be displayed; 3. Data gathering and data processing; 4. Tools for the implementation of the ORS; 5. Building the dashboard – data visualisation; 6. Storytelling; 7. Publication of the ORS and publicity; 8. Advertising both the solution itself and its benefits; 9. Other opportunities. Even though the present document is not meant to go into the technicalities of how to build dashboards, the reader may find useful a brief glossary in Appendix I for a better understanding of some technical terms mentioned in this text. Moreover, Appendix II contains a list of some useful online resources regarding visuals and dashboards.

1. Working group

The implementation of the ORS is a responsible initiative which requires the presence of a working group supervising the process during all its stages. As already mentioned, the implementation of a successful ORS requires the involvement of different professional figures in order to implement the ORS effectively and efficiently. A well-formed and balanced working group is suggested to be composed of judges, prosecutors, court staff, statisticians, data analysts, web developers, IT

15

specialists, judicial researches/experts. Each one of them will provide their expertise during the different stages of the implementation of the project.

Especially during the first stage of the project (i.e. identification of the data to be displayed and the discussion about the quality of the data) it is highly recommended to listen carefully and consider all the different needs, comments and views of the participants of the working group. This is a very sensitive part and it has deep repercussions on the entire project. On the other hand, the implementation phase will inevitably be up to technical professional figures (such as statisticians, data analysts and IT specialists).

2. Identify the data to be displayed

Identifying the data to make available to the public through the ORS is one of the first key element to be discussed by the working group. The choice is not obvious and must be pondered thoughtfully. Several fundamental aspects must be taken into consideration: availability of the data, ease of data extraction from the existing data sources, accuracy and reliability of the data, the relevance of the data to the final users, privacy and confidentiality issues, the possibility to repeat the extraction from the database in the future under the same conditions.

All the above-mentioned aspects address two main categories of issues: technical issues and opportunity-related issues. The participants to the working group will contribute to the discussion according to their areas of expertise.

To start with, the working group might consider the following list of areas of investigation: • Budgetary elements and justice expenses; • Legal aid; • Number of courts; • Population served by each court; • Case flow (pending cases, incoming and resolved case); • Length of the proceedings; • Number of judges; • Number of court staff; • ADR procedures. Each of the above areas are likely to be explored further for a more detailed and exhaustive examination. For instance, the length of the proceedings can be seen from different angles: actual length, disposition time, age of pending cases, age of resolved cases, timeframes and backlogs, etc. Another example is given by the number of judges or the number of court staff where the working group is supposed to describe how these figures should be collected and reported: by instance (first, second, and cassation), by gender, in full-time equivalent, etc.

Also, it is important to asses which variables are needed to investigate topics such as workload (of judges, prosecutors, court staff, etc.) and efficiency in all its various forms. Moreover, it is essential

16

to identify the main indicators and how to calculate them: litigation rate, appeal ratio, quashed and modified decisions ratio, clearance rate, etc.

In this regard, the biannual CEPEJ questionnaire on the evaluation of judicial systems is an excellent starting point for the definition of the data under scrutiny. It is suggested to look at the CEPEJ-STAT14 as a reference. The CEPEJ-STAT is the dynamic database of European judicial systems which allows general public to find various data on judicial systems of Council of Europe Member states, entities and observers collected by CEPEJ.

The working group is supposed to discuss not only about the availability of the data in the abstract but also about other practical consequences. For instance, when it comes to the case flow, one must also decide the period for which these measures/variables are available. This decision is not arbitrary as it must take into consideration not only the availability of the data but also its accuracy and its reliability. In this respect, it is also extremely important to have court staff involved in the working group as they have a better understanding of the case management systems, how it is populated in practice and therefore they are aware of its flaws and pitfalls. It is important to ponder their opinion on the quality of the data and the level of reliance of the data produced by the ICMS that ultimately feeds the ORS.

As a matter of fact, the ORS follow the so-called “GIGO” concept (garbage in, garbage out) according to which nonsense input data produces nonsense results as output. For this reason, it is fundamental to stress the importance of the quality of the data. It is important to make sure that everyone in the working group fully understand the collection method (how the data is retrieved from the system), the definitions and limitations of each variable under scrutiny.

Other than defining the period for which the data is available one should also define the level of detail in terms of courts. Should the data be referred only to national-level figures or should it also include information on different levels of jurisdiction (first instance, appellate courts, Supreme Court of Justice)? In other words, one of the major matters under discussion is the depth of details of the data, i.e. the level of details that the ORS is capable of managing. Please note that highly detailed data might raise privacy and confidentiality issues. In general, highly detailed data carries some risks, such as the publication of sensitive personal data, which need to be addressed. Since the access to elementary data might cause problems related to privacy and confidentiality, it is highly recommended to evaluate precisely what kind of information is made available through the ORS. Having said that, in general, the ORS is intended to provide general information about the functioning of the system so it should not be a major problem.

Other than the above-mentioned quantitative measures, the ORS is also supposed to provide information on qualitative aspects of the judiciary. It is worth mentioning that CEPEJ has implemented a specific explorer15 (see the glossary in appendix I) to address qualitative data.

14 Please see https://www.coe.int/en/web/cepej/dynamic-database-of-european-judicial-systems 15 Please see https://public.tableau.com/profile/cepej#!/vizhome/CEPEJ-Qualitativedata5EN/QualitativeData 17

3. Data gathering and data processing

Once the measures/variables have been identified, the next step is to select the data sources from which the data is retrieved. These aspects have a more technical nature compared to the mere identification of the variables under scrutiny. In general, the relevant data is most likely to be spread across different data sources and platforms. Case flow information usually comes from the case management system while budgetary data comes from budgetary and financial management systems. There might also be data (e.g. number of judges, population) which come from systems outside the Judiciary.

Data extraction is the process of retrieving data out of data sources and it is a very sensitive part of the workflow. After the extraction from the relevant database/s, the data must undergo a validation and a processing phase. Validation is required for purposes related to quality controls. After that, the data is usually not ready to be submitted to the ORS as it must undergo a processing phase to make sure that the resulting data meets the ORS requirements. This processing phase may be automated through ETL procedures (see the glossary in appendix I) or performed manually. Although highly advisable, automatization of the process is not always feasible. Moreover, the technical implementation of the data gathering and processing depends on both the actual technical implementation of the data sources on one hand and the system adopted for the reporting on the other.

During the processing phase it is important to detect missing data, duplicates and also promptly identify suspicious values and outliers which may affect the results. It is also important that the working group drafts a protocol that illustrates how to deal with missing data and suspicious values. Therefore, it is necessary to have a set of rules that establish under which circumstances suspicious values are removed from the database and when the outlier is addressed with a warning message.

When dealing with data gathering, especially from data sources outside the judiciary, it is advisable to pay special attention to the extraction criteria (i.e. the query), making sure that the queries remain the same over time, guaranteeing a certain level of consistency.

4. Tools for the implementation of the open reporting system

From a technical point of view the ORS can either be integrated with the existing ICMS and/or BI system or being a stand-alone solution. Both these approaches have their pros and cons. Having an ORS directly fed by the ICMS/BI solution is definitely a more elegant solution, but it might be a little more difficult to set-up and definitely takes more time to be implemented. In the market there are big players such as IBM, Microsoft, Oracle and SAP (just to name a few) and a plethora of small and medium-sized enterprises offering technical solutions. It comes without saying that the choice is dictated by pre-existing contracts with providers of the Moldovan judicial infrastructure.

On the other hand, a stand-alone solution is quite reasonably easier to be implemented in the short period. Moreover, the implementation of the ORS through an ad-hoc and stand-alone solution might be the result of an internal process without the need of outsourcing the implementation of

18

such system. Therefore, a convenient solution that does not necessary require the intervention of external expertise and consequently it is feasible with a limited budget.

There are many different solutions and visualisation tools available on the web and many of them are for free. CEPEJ is currently using Tableau Public16 for displaying its dynamic database of European judicial systems (CEPEJ-STAT). Tableau comes with different plans and different prices but they also offer a free plan (Tableau Public) which represents a valid option. As already mentioned, other than Tableau there is a multitude of other providers in the market. It is outside the scope of this document to suggest one specific solution. However, in appendix II - online resources the reader can find the link to a collection of different solutions for data visualisation available on the web.

5. Building the dashboard – data visualisation

Once the data is gathered, validated and processed, the next step is to build the actual ORS/dashboards, a system that allows the final users to navigate the judicial data through the most appropriate visualisation tools. In a nutshell, a dashboard is a collection of figures grouped into charts and tables. As already mentioned, today’s solutions come in all shapes and sizes. Still, the following considerations are independent from the solution/brand of choice.

There was a time when the only options we had to display figures visually were merely a bar chart or a pie chart. We have come a long way since then; today there are hundreds of different fancy graphs available which enable us to convey our data and messages more effectively. Being able to choose the right visualisation is paramount.

Professional dashboards are often designed by data visualisation designers, as they not only develop eye-catching solutions within the dashboard, but also use the right visuals in order to maximise the message we want to convey. They master the art of applying design standards to improve readability, clarity, and accessibility of the data through graphics. Data visualisation designer is a quite recent profession. For this reason, as an alternative, the dashboard can be developed by a mixed team of statisticians, data analysts and IT experts.

In general, when building the actual dashboards, one needs to have a full picture of all the data available and how they are linked one to another. For this reason, even though it may seem a mere technical task, it should be supervised by judicial experts and other components of the working group to share ideas and comments.

As already mentioned, it is important to make the dashboards aesthetically appealing, but it is even more important to use the most appropriate visual (e.g. bar chart, scatter plot, density plot) according to the variable/s under investigation and having in mind the message we want to convey. E.g. a “timeline” is useful when one wants to underline the improvement across time, a “choropleth map” come very handy to display differences between courts or regions while a “word cloud” to summarise the keywords / findings of satisfaction surveys.

16 Please see https://public.tableau.com/en-us/s/ 19

In Appendix II the reader can find a few useful online resources which may help choosing the right visuals depending on the variables under investigation.

The next question is how many dashboards should be the ORS composed of? Theoretically the ORS might consist of one single dashboard to navigate the whole judiciary dataset. This option has the advantage of having all the relevant data in one place but the downside is that it may be too slow to navigate because of the enormous amount of data. For this reason, it is advisable to create different dashboards that address specific topics such as budgetary elements, durations, professionals, etc. The number of these dashboard depend on various aspects (technical and opportunity related) that the working group might want to discuss. The CEPEJ itself has adopted this multi-report approach: the CEPEJ-STAT is composed of different dashboards and explorers to navigate the database thematically.

The idea of a dashboard is that the user is given the possibility to move freely between charts and tables using a user-friendly interface (UI - see the glossary in appendix I). Each individual table or chart should allow a series of operations such as “search”, “sort” and “filter by value”. More sophisticated solution might also allow additional functions such as filtering using a combination of AND / OR conditions or transformation (e.g. adding or removing columns).

Most of the time the initial view of a dashboard is intended to provide a summarised view of the topic we are representing. Starting from there, the user can autonomously get a more granular level using the user interface. This goes back to the discussion about the level of detail of the data. Today, most of the visualisation tools available have drill-down features, which allow users to navigate the data at different depths. For example, figures about civil proceedings might be explored to see further sequences or sub-categories of cases: Civil proceedings > Non-litigious cases > Family law > Mutual consent divorce cases as illustrated in the figure below. Figure 1 - This scheme is just for illustrative purposes

Civil Layer1 proceedings

Non litigious Litigious cases ... Layer 2 cases

Contract Family law ...... Layer 3 Disputes

Mutual consent ... Layer 4 divorce cases

20

Figure 1 illustrates an example of a 4-layer scheme. The only intent of this figure is to illustrate the drill-down concept. It comes without saying that the working group may come up with a different schema according to the peculiarities of the Moldovan judicial system. Moreover, some branches of the tree may go down to layer 4 while others terminate at an earlier layer (for example ‘contract disputes’ in Figure 1 cannot be expanded any further). The opposite operation to the drill-down is called roll-up (moving from something more specific to something more general). Moreover, the same concept applies to all categorical variables with a hierarchical structure, for example time (Year>Semester>Quarter) or location (National>Regional>District). It is the task of the working group to identify all the categorical variables and provide the hierarchy where appropriate.

When developing a dashboard, it is important to accompany each visual with relevant information (e.g. data source, updated date) and the description of the variables involved if not evident from the context.

Once the dashboards are in place it is a good idea to check them against the Data Visualisation Checklist (see the link is provided in Appendix II). It is a compilation of 24 guidelines on how graphs should be formatted to best show the story in your data. The guidelines are broken down into 5 sections: text, arrangement, colour, lines, and overall.

6. Storytelling

Sometimes numbers and figures without a context may lead to wrong conclusions. Since the ORS is open not only to the Moldovan citizens but also to the press, it is paramount to provide numbers and figures within the relevant contextual environment, to exclude to the extent possible the erroneous conclusions. Therefore, charts and tables should always be accompanied by notes that help the reader to interpret the figures correctly. For instance, time series are often accompanied by notes that describe legislative changes occurred over time. In this respect, the working group should identify those figures and charts which are more likely to be misinterpreted and draft additional notes to be added into the dashboard.

Moreover, the ORS is not only a collection of figures grouped into charts and tables, as it represents a modern and innovative way to disseminate and communicate judicial data to the public. In this respect, the judicial authorities have the opportunity to make a smarter use of this kind of instrument: telling a story through the numbers. For instance, illustrating the efficiency of the judicial system over time, or maybe illustrating the workload of judges across different courts.

7. Publication of the open reporting system and publicity

Once the setup of the ORS is accomplished, the next step is to make it available to the general public. In practice, the ORS may reside on a website of its own or being embedded into some other relevant website/s (MoJ/ACA/SCM). Despite the virtual place where the ORS is hosted / published, it is paramount to officially announce its publication through a press conference or any similar activity. It is important to communicate to the Moldovan citizens the fact that the Judiciary has introduced an open and innovative tool through which it is possible for anyone to access the judicial data and track the efficiency of the judiciary over time. 21

The promotion of the ORS may come through different channels: press, TV, court websites, posters and leaflets within the courts, etc. Other than the implementation of the ORS itself, it is important to convey the message of a new approach to transparency, thus increasing public confidence in the courts. Ideally, the publication of the ORS should be accompanied or followed by the publication of judicial decisions.

Other opportunities

9.1 Satisfaction surveys As already mentioned, the ORS represent not only a technical device to display judicial data more effectively and efficiently but also represents a better way for the judicial authorities to convey their message to the general public. In this respect, the ORS might also come handy to illustrate the results of satisfaction survey conducted across the country. All kind of surveys (those aimed at court users, lawyers, prosecutors) are eligible to be portrayed though the ORS. The Ministry of Justice and the courts have an opportunity to show the results of the satisfaction surveys conducted in recent years. This will have a double positive effect: 1) it shows the interest of the court management to listen to the needs of the court users, 2) it stresses the positive experience of court users regardless the overall negative perception/attitude of the citizens towards the judiciary. It is worth underlying that the latter point is supported by numbers and figures and it is not just an arguable propaganda statement.

9.2 Open data As already mentioned, the ORS is specifically designed to portray an accurate picture of the current state of the judicial system and it is intended to the general public. It also may well serve the needs of specific users like the media, researchers, students, NGOs, etc. All these actors are often eager to collect more detailed numbers and figures regarding the judiciary.

Noticeably, students and researchers are often looking for elementary/detailed data rather than consolidated data (statistics or summaries) for their research. To meet their reasonable expectations, it is possible to make available not only summary tables and graphs through the ORS but also the underlying elementary data that were used to produce the dashboards. In other words, it is possible to offer access to open judiciary data that can be eventually downloaded and analysed. Here the terms “open” refers to the fact that the users can freely access unprocessed atomic data and analyse it for their personal and/or academic purposes.

It is worth mentioning that having free access to judicial data does not mean having access to sensitive information of single procedures. It means having access to judicial data up to a certain level (decided beforehand). Open data is a very wide and complex topic which involves technical aspects, (e.g. meta-data), organisational aspects and confidentiality issues. Generally speaking, open government data is a worldwide growing phenomenon. Having said that, open data is outside the scope of the ORS and this document. Still, it represents a major opportunity to enhance the accountability, transparency and efficiency of the Moldovan judicial system and for this reason it is worth consideration.

22

Appendix I – Glossary and abbreviations17

• ACA – Agency for Court Administration. • API (Application programming interface) – a computing interface to a software component or a system, that defines how other components or systems can use it. • BI – Business Intelligence. • CEPEJ – European Commission for the Efficiency of Justice. • CEPEJ GOJUST – CEPEJ Guidelines on Judicial Statistics • CEPEJ SATURN – Working group on judicial time management. • CEPEJ-STAT – a dynamic database which allows you to find various data on judicial systems of Council of Europe Member states / entities and observers collected by the CEPEJ. • Chart-junk – a term that refers to all visual elements in charts and graphs that are not necessary to comprehend the information represented on the graph, or that distract the viewer from this information. • Choropleth map – a type of thematic map in which areas are shaded or patterned in proportion to a certain variable. • Consolidated data (or summary data) – data obtained from a collection of elementary data by applying some aggregation function. • Clearance rate (CR) – Relationship between the new cases and completed cases within a period, in percentage. • Dashboard – an easy-to-read user interface showing a graphical presentation of data; it is often composed of different tables, graphs and charts which the user can navigate through dynamically. • Data extraction – the process of retrieving data out of data sources for further data processing. • Dimensions – variables containing qualitative values (such as names, dates, or geographical data). Dimensions can be used to categorise, segment, and reveal the details in the data. • Disposition time (DT) – compares the number of resolved cases during the observed period and the number of unresolved cases at the end of the observed period. 365 is divided by the number of resolved cases divided by the number of unresolved cases at the end, so as to be able to express it in number of days. The ratio measures how quickly the judicial system (of the court) turns over received cases – that is, how long it takes for a type of cases to be resolved. This indicator provides further insight into how a judicial system manages its flow of cases. • Drill-down – the operation of exploring the data from general to specific. • DWH – Data Warehouse. • Elementary data – the most basic unit of data (i.e. data at the most granular level).

17 For further information about applied terminology please see the CEPEJ Glossary (CEPEJ(2020)Rev1) at https://rm.coe.int/cepej-2019-5final-glossaire-en-version-10-decembre-as/1680993c4c 23

• ETL (extract, transform, load) – a general procedure of copying data from one or more sources into a destination system which represents the data differently from the source(s) or in a different context than the source(s). • Explorer – an explorer is a subset of a dashboard which addresses a specific topic. • FTE – Full-Time Equivalent – number of persons working the standard number of hours. The number of persons working part time should be converted to full-time equivalent (e.g. when two people work half the standard number of hours, they count for one "full-time equivalent". One half-time worker should count for 0.5 of a full-time equivalent). • ICMS – Integrated Case Management System. • Infographic – a graphic visual representation of information or data intended to present information quickly and clearly. • Information retrieval – is the activity of obtaining information system resources that are relevant to an information need from a collection of those resources. • KPI – Key Performance Indicator. • Measures – variables containing numeric and quantitative values that can be measured. Measures can be aggregated. • MoJ – Ministry of Justice. • ORS – Open Reporting System • Pending cases - Cases that remain open / unresolved at the end of a reference period. • Quashed and modified decisions – decisions that are quashed (annulled) or modified by the hierarchically higher court. • Query – a formal request (in a specific programming language) for information from a database. • Roll-up – the operation of exploring the data from specific to general. • Scatterplot – a type of data display that shows the relationship between two numerical variables. • SCM – Superior Council of Magistracy. • SEIIJS – Strategy for Ensuring the Independence and Integrity of Justice Sector. • Smart table – a statistical table with a set of built-in features such as filtering and sorting. • Thematic map – a type of map specifically designed to show a particular theme connected with a specific geographic area. • Time series – a series of data points indexed (or listed or graphed) in time order. • UI (User interface) – the space where interactions between humans and machines occur. • Viz tool (visualisation tool) – a graphical representation of information and data. • Word cloud (or tag cloud) – a visual representation of text data, typically used to visualise free form text.

24

Appendix II - Resources available on the web regarding visuals and dashboards

Visual Vocabulary https://ft-interactive.github.io/visual-vocabulary/ A particularly useful vocabulary regarding visuals divided into several categories. It helps deciding which data relationship is most important and suggest the more relevant types of chart within each category.

Data visualisation resources https://www.visualisingdata.com/resources/ A collection of different solutions for data visualisation

The Chartmaker Directory https://chartmaker.visualisingdata.com/ The Chartmaker Directory gathers and organises a useful catalogue of references that will offer an answer to one of the most common questions in data visualisation: 'which tool do you need to make that chart?’.

Data Visualisation Checklist https://datavizchecklist.stephanieevergreen.com/assets/DataVizChecklist_Feb2018.pdf An extremely useful “Data Visualisation Checklist” developed by data analyst Dr. Stephanie Evergreen and her team which allows assessing the goodness of a dashboard.

25