EACL 2021 Submission ***. Confidential Review Copy. DO NOT DISTRIBUTE.

000 050 001 OPUS-CAT: Desktop NMT with CAT integration and local fine-tuning 051 002 052 003 053 004 Tommi Nieminen 054 005 University of Helsinki, Yliopistonkatu 3, 00014 University of Helsinki, Finland 055 006 [email protected] 056 007 057 008 058 009 059 010 060 011 061 Abstract translation professionals use MT only occasionally. 012 One of the factors slowing down the adoption of 062 013 OPUS-CAT is a collection of software which MT are risks related to confidentiality and security. 063 014 enables translators to use neural machine trans- There are well-known risks involved with using 064 lation in computer-assisted translation tools 015 web services, which also concern the web-based 065 without exposing themselves to security and 016 066 confidentiality risks inherent in online ma- NMT services available to translators and organi- 017 chine translation. OPUS-CAT uses the public zations: data sent to the service may be intercepted 067 018 OPUS-MT models, which en route, or it may be misused or handled care- 068 019 are available for over a thousand language lessly by the service provider. These security and 069 020 pairs. The generic OPUS-MT models can be confidentiality risks (even if they are unlikely to ac- 070 021 fine-tuned with OPUS-CAT on the desktop us- tualize) hinder MT use by independent translation 071 ing data for a specific client or domain. 022 professionals, since their clients often specifically 072 023 1 Introduction forbid or restrict the use of web-based MT (Euro- 073 024 pean Commission, 2019). Even if using web-based 074 025 Neural machine translation (NMT) has brought MT is not expressly forbidden, translators may con- 075 026 about a dramatic increase in the quality of ma- sider it unethical or they may fear it might expose 076 027 chine translation in the past five years. The re- them to unexpected legal liabilities (Kamocki et al., 077 028 sults of the latest European Language Industry Sur- 2016). 078 029 vey (FIT Europe et al., 2020) confirm that NMT is Producing MT directly on the translator’s com- 079 030 now routinely used in professional translation work. puter without any communication with external 080 031 NMT systems used in translation work are devel- services eliminates the confidentiality and security 081 oped by specialized machine translation vendors, 032 risks associated with web-based MT. This requires 082 translation agencies, and organizations that have an optimized NMT framework which is capable of 033 083 their own translation departments. Translators use running on Windows computers (as most CAT tools 034 084 NMT either at the request of a client, in which case are only available for Windows), and pre-trained 035 085 the client provides the NMT, or independently, in NMT models for all required language pairs. The 036 086 which case they usually rely on web-based services Marian NMT framework (Junczys-Dowmunt et al., 037 087 offered by large tech companies (such as Google or 2018) fulfills the first requirement, as it is highly op- 038 Microsoft) or specialized machine translation ven- timized and supports Windows builds. Pre-trained 088 039 dors. These web-based services are mainly used NMT models are available from the OPUS-MT 089 040 through machine translation plugins or integrations project (Tiedemann and Thottingal, 2020), which 090 041 that are available for all major computer-assisted trains and publishes Marian-compatible NMT mod- 091 042 translation (CAT) tools, such as SDL Trados and els with the data collected in the OPUS corpus 092 043 memoQ. (Tiedemann, 2012). OPUS-CAT is a software col- 093 044 Even though MT has been extensively used in lection which contains a local MT engine for Win- 094 045 the translation industry for over a decade (Doherty dows computers built around the Marian frame- 095 046 et al., 2013), there is still considerable scope for work and OPUS-MT models, and a selection of 096 047 growth: according to FIT Europe et al.(2020), plugins for CAT tools. OPUS-CAT is aimed at pro- 097 048 78 percent of language service companies plan to fessional translators, which is why it also supports 098 049 increase or start MT use, and most independent the fine-tuning of the base OPUS-MT models with 099

288

Proceedings of the 16th Conference of the European Chapter of the Association1 for Computational Linguistics: System Demonstrations, pages 288–294 April 19 - 23, 2021. ©2021 Association for Computational Linguistics EACL 2021 Submission ***. Confidential Review Copy. DO NOT DISTRIBUTE.

100 OPUS TMX file or Trados 150 101 corpora parallel text files plugin 151 102 152 103 153 train models extract fine-tuning 104 install material 154 105 models 155 OPUS-MT 106 locally OPUS-CAT API CAT tool memoQ 156 model 107 MT Engine integration plugin 157 repository 108 158 translate 109 finetune 159 with 110 models 160 111 models 161 112 Locally installed (as OmegaT 162 113 Marian NMT custom MT) plugin 163 114 164 115 Figure 1: Diagram of the software and models used in OPUS-CAT. 165 116 166 117 project-specific data. ing the text while newer models use SentencePiece 167 118 (Kudo and Richardson, 2018). Both Subword NMT 168 119 2 OPUS-CAT MT Engine and SentencePiece are coded in Python, but they 169 120 are distributed as standalone Windows executables 170 121 The main component of OPUS-CAT is the OPUS- with OPUS-CAT MT Engine, as requiring the users 171 122 CAT MT Engine, a locally installed Windows appli- to install Python in Windows would complicate the 172 123 cation with a graphical user interface. OPUS-CAT setup process. 173 MT Engine can be used to download NMT mod- 124 OPUS-CAT MT Engine user interface provides 174 els from the OPUS-MT model repository, which 125 a simple functionality for translating text, but the 175 contains models for over a thousand language pairs. 126 translations are mainly intended to be generated via 176 127 an API that the OPUS-CAT MT Engine exposes. 177 128 This API can be used via two protocols: net.tcp 178 129 and HTTP. net.tcp is used with plugins for the SDL 179 130 Trados and memoQ CAT tools, while HTTP is used 180 131 for other plugins and integration. The motivation 181 132 for using net.tcp is that exposing a net.tcp service 182 133 on the local Windows computer does not require 183 administrator privileges, which makes setting up 134 184 the OPUS-CAT MT Engine much easier for non- 135 185 technical users. However, Trados and memoQ are 136 186 the only CAT tools with sufficiently sophisticated 137 187 Figure 2: Install OPUS-MT models locally (1,000+ lan- plugin development kits to allow for net.scp con- 138 188 guage pairs available) nections, so the API can also be used via HTTP 139 with some extra configuration steps, so that it can 189 140 Once a model has been downloaded, OPUS-CAT be used from other tools. The API has three main 190 141 MT Engine can use it to generate translations by functionalities: 191 142 invoking a Marian executable included in the in- 192 • Translate: Generates a translation for a 143 stallation. Before the text is sent to the Marian 193 source sentence (or retrieves it from a cache) 144 executable, OPUS-CAT MT Engine automatically 194 and returns it as a reply to the request. 145 pre-processes the text using the same method that 195 146 was originally used for pre-processing the train- • PreorderBatch: Adds a batch of source sen- 196 147 ing corpus of the model. Pre-processing is model- tences to the translation queue and immedi- 197 148 specific, as the older OPUS-MT models use Sub- ately returns a confirmation without waiting 198 149 word NMT (Sennrich et al., 2016) for segment- for the translations to be generated. 199

289

2 EACL 2021 Submission ***. Confidential Review Copy. DO NOT DISTRIBUTE.

200 • Customize: Initiates model customization us- 250 201 ing the fine-tuning material included in the 251 202 request. 252 203 253 The OPUS-CAT MT Engine stores the local 204 254 NMT models in the user’s application data folder 205 255 in order to avoid file permission issues. Local ap- 206 256 plication data folder is used, as saving the models 207 257 in the roaming application data folder could lead 208 to unwanted copying of the models, if same user 258 209 profile is used on multiple computers. The user 259 210 260 interface of the OPUS-CAT MT Engine contains Figure 3: Trados plugin settings. Note the preordering 211 261 functionalities for managing models installed on function and model tag. 212 the computer, such as deletion of models, packag- 262 213 ing of models for migration to other systems, and 263 214 tagging the models with descriptive tags. The tags in the plugin settings) to the OPUS-CAT for trans- 264 215 can be used to select specific models in CAT tool lation. This means that when the translator moves 265 216 plugins, e.g. a model fine-tuned for a specific cus- to the next segment, it has already been translated 266 217 tomer can be tagged with the name of the customer. and can simply be retrieved from the OPUS-CAT 267 218 translation cache. 268 219 3 CAT tool plugins and integration 269 4 Local fine-tuning of models 220 OPUS-CAT contains plugins for three CAT tools: 270 221 SDL Trados, memoQ and OmegaT. OPUS-CAT OPUS-CAT is intended for professional transla- 271 222 can also be used with the Wordfast CAT tool, which tors, and the utility of generic NMT models in pro- 272 223 supports fetching translations from services via fessional translation is uncertain (Sanchez-Gij´ on´ 273 224 APIs. As SDL Trados is the established market et al., 2019), while performance improvements re- 274 225 leader among CAT tools, and it has the most exten- sulting from the use domain-adapted NMT models 275 226 sive plugin development support, the OPUS-CAT have been observed multiple times (Laubli¨ et al., 276 227 plugin for SDL Trados is more feature-rich than 2019; Macken et al., 2020). Because of this, OPUS- 277 228 the other plugins. The other plugins simply sup- CAT MT Engine includes a functionality for fine- 278 229 port fetching translations through the Translate API tuning models with small amounts of bilingual data. 279 230 method. SDL Trados plugin also contains an option The method of fine-tuning is simple: the generic 280 231 to initiate the fine-tuning of a model based on the OPUS-MT model for a language pair is duplicated, 281 and Marian training of the model is resumed with 232 bilingual material included in a translation project. 282 One difficult aspect of integrating MT services the model using the domain-specific data as the 233 283 with CAT tools is latency. Delays in presenting training set. This particular method of fine-tuning 234 284 the translation to the user affect the user experi- was first described in Luong and Manning(2015), 235 285 ence adversely and may even lower productivity but adaptation of statistical and neural MT models 236 286 significantly. For most MT services the delay is with domain-specific data has been common for 237 287 due to web latency, but for OPUS-CAT the gen- over a decade (Koehn and Schroeder, 2007). The 238 eration of translations itself may be so slow that fine-tuning is performed using the same Marian 288 239 it causes a visible delay, since OPUS-CAT uses a executable included with OPUS-CAT MT Engine 289 240 CPU for translation instead of a much faster GPU. installation, which is also used for generating trans- 290 241 In any CAT tool, this delay can be eliminated by lations. 291 242 pre-translating the translation project with the MT OPUS-CAT MT Engine is a Windows program 292 243 service prior to starting the translation. intended to run on mid-tier desktop and laptop com- 293 244 In the OPUS-CAT plugin for SDL Trados there puters that translators commonly use, so the fine- 294 245 is also a feature which can be used to initiate the tuning process cannot be computationally intensive. 295 246 translation of segments ahead of time. Whenever The fine-tuning must rely on CPUs, since GPUs 296 247 the translator moves to a new segment, the plugin suitable for neural network training are not avail- 297 248 will send the segments following the selected seg- able on the translators’ computers. This places 298 249 ment (the number of segments can be configured severe restrictions on the size of the fine-tuning set 299

290

3 EACL 2021 Submission ***. Confidential Review Copy. DO NOT DISTRIBUTE.

300 and the duration of the training. Furthermore, the vanced users can change these default settings in 350 301 fine-tuning functionality is intended to be used by the settings tab of the OPUS-CAT MT Engine. 351 302 translators without specialist knowledge about ma- 352 303 chine translation, so the users cannot be expected 353 304 to be able to adjust the fine-tuning settings. That is 354 305 why the fine-tuning functionality has to have a con- 355 306 servative set of default settings that work in almost 356 307 all environments and circumstances. 357 308 In a typical translation job, deadlines usually 358 309 allow for considerably more time for delivery of the 359 310 translation than the actual translation work requires. 360 311 This is due to the fact that translators normally 361 312 have multiple jobs underway or lined up at any Figure 4: Initiating fine-tuning from OPUS-CAT MT 362 313 given time, and extended deadlines are required Engine. 363 314 so that translators can organize and prioritize their 364 315 work. This means that there is generally at least Currently fine-tuning can be initiated directly 365 316 a couple of hours of time available for running from the OPUS-CAT MT Engine or from the SDL 366 the fine-tuning process before the actual translation 317 Trados plugin. When initiated from the OPUS- 367 work has to begin. On the basis of this estimate, the 318 CAT MT Engine, a .tmx file or a pair of source 368 fine-tuning process should generally take at most 319 and target files can be used as fine-tuning mate- 369 two hours. 320 rial. Fine-tuning from the SDL Trados plugin 370 321 Another consideration in fine-tuning is that the allows for much more sophisticated selection of 371 fine-tuning material. The fine-tuning functional- 322 process takes place on a computer that may be used 372 for other tasks during the fine-tuning. This means ity in the SDL Trados plugin is implemented as a 323 373 that the fine-tuning process cannot take advantage batch task, which is performed for a given transla- 324 374 of all the resources available on the computer, as do- tion project. Translation projects usually contain 325 375 ing so would cause performance issues for the user. segments which have already been translated (full 326 376 In order to keep the fine-tuning as non-intrusive matches). By default, the fine-tuning task extracts 327 as possible, OPUS-CAT MT Engine uses only a these segments as fine-tuning material. These are 377 328 single thread and a workspace of 2048 MB for assumed to be the most relevant material, since 378 329 fine-tuning. they pertain directly to the translation project. 379 330 380 Because of the limited amount of processing 331 381 power available and the target duration of at most 332 two hours, the fine-tuning is stopped after a single 382 333 epoch by default. The actual duration will vary 383 334 according to the sentence count and the sentence 384 335 length distribution of the fine-tuning set. The dura- 385 336 tion will be approximately two hours when the fine- 386 337 tuning set contains the default maximum amount 387 338 of sentences, which is 10,000. 388 339 It would be possible to allow the fine-tuning to 389 340 last for multiple epochs and to adjust the amount 390 341 of epochs based on the sentence count, but infor- 391 342 mal testing during development indicated that a 392 343 single epoch of fine-tuning tends to have a notice- 393 344 able effect on the MT output even with small fine- 394 345 tuning sets. Also, some output corruption indicat- Figure 5: Initiating fine-tuning from OPUS-CAT plu- 395 346 ing over-fitting was detected when fine-tuning was gin for SDL Trados.) 396 347 continued over many epochs. Because of these in- 397 348 dications of over-fitting, the learning rate was also If the translation project does not contain enough 398 349 lowered to 0.00002 from the default 0.0001. Ad- full matches, it is possible to extract translation 399

291

4 EACL 2021 Submission ***. Confidential Review Copy. DO NOT DISTRIBUTE.

400 units from the translation memories attached to SDL Trados plugin supports tag conversion). Place- 450 401 the translation project. The fine-tuning task uses holder tags and tag pairs are converted separately. 451 402 the fuzzy matching functionality of SDL Trados This approach to tag handling is similar to the one 452 403 to extract partially matching translation units ac- found in (Hanneman and Dinu, 2020), but simpler. 453 404 cording to the specified minimum fuzzy percentage. The main difference is that the same textual tag 454 405 The task can also extract fine-tuning material by marker is used for every tag, so the tag handling as- 455 406 performing a concordance search for words or se- sumes that the tag order is identical in both source 456 407 quences of words in the source segment. Finally, and target. 457 408 if the other methods have not managed to extract a 458 5 Related work 409 sufficient amount of fine-tuning material, the task 459 410 can simply bulk up the fine-tuning set by extracting Desktop MT systems have been available at least 460 411 segments from the translation memories, starting since the 1990s (Richards, 1994), when desktop 461 with the newest (assumed to be the most relevant). 412 computers became powerful enough to run rule- 462 413 based MT systems. In the SMT era, the higher 463 414 computational requirements made desktop MT dif- 464 415 ficult, but there were still some examples of desktop 465 416 SMT, such as (Slate Rocks!, 2021). Unlike OPUS- 466 417 CAT, these earlier desktop MT programs were com- 467 418 mercial products. As for NMT, the currently active 468 Bergamot project (Bergamot, 2021) aims to make 419 469 client-side MT available in web browsers, and also 420 470 uses Marian as its NMT framework. However, 421 471 Bergamot is aimed at the common public, while 422 472 OPUS-CAT is intended for professional translators. 423 473 To our knowledge, there is no other software that is 424 Figure 6: Monitoring fine-tuning progress.) free to use and offers a local NMT fine-tuning func- 474 425 tionality (commercial MT providers do provide 475 Progress of the fine-tuning can be monitored in 426 local MT engine installations, which may support 476 the OPUS-CAT MT Engine. When a fine-tuning 427 local fine-tuning). 477 428 job is initiated, part of the fine-tuning set is sepa- 478 429 rated for use as an in-domain validation set. For 6 Current status and future work 479 430 most language pairs, OPUS-CAT MT Engine also 480 contains out-of-domain validation sets, which have OPUS-CAT is based on software developed orig- 431 inally for the Fiskmo¨ project (Tiedemann et al., 481 432 been extracted from the Tatoeba corpus (Tiede- 482 mann, 2020). The in-domain and out-of-domain 2020), and it is currently being developed as part 433 483 validation sets are combined and evaluated period- of the European Language Grid programme. The 434 484 ically during fine-tuning with SacreBLEU (Post, previous version of the software has been used by 435 485 2018), which is also included in OPUS-CAT as a several organizations in Finland for professional 436 486 standalone Windows executable. The evaluation translation. Based on the feedback from users, the 437 487 results for each set are plotted as a graph in the most important features that translators would like 438 OPUS-CAT MT Engine. OPUS-CAT MT Engine to see are real-time adaptation of NMT models 488 439 also displays an estimate of the remaining dura- with new translations, and the enforcement of cor- 489 440 tion of the fine-tuning. These visual indications rect terminology and document-level consistency. 490 441 of progress are important, as the users of the fine- These will be the main priorities in the develop- 491 442 tuning functionality are translators without special- ment of OPUS-CAT. We will also be collecting 492 443 ist technical skills. user experiences on the local fine-tuning capability, 493 444 The fine-tuning functionality also has an op- and will develop the feature and its documentation 494 445 tion to include tags found in the fine-tuning set according to that feedback. 495 446 496 as generic tag markers. The fine-tuned model will 7 Conclusion 447 then learn to generate tag markers in the transla- 497 448 tions, and these can be used to transfer tags from OPUS-CAT is collection of software that makes it 498 449 source to target in CAT tools (currently only the possible to use NMT locally on desktop computers 499

292

5 EACL 2021 Submission ***. Confidential Review Copy. DO NOT DISTRIBUTE.

500 without risks posed by web-based services. It uses on Statistical Machine Translation, pages 224–227, 550 501 models from the OPUS-MT project, which offers Prague, Czech Republic. Association for Computa- 551 tional Linguistics. 502 NMT models for over a thousand language pairs. 552 503 OPUS-CAT is based on the efficient and optimized Taku Kudo and John Richardson. 2018. Sentencepiece: 553 504 Marian NMT framework, which is fast enough to A simple and language independent subword tok- 554 505 work usefully even with mid-tier computers. The enizer and detokenizer for neural text processing. 555 pages 66–71. 506 local fine-tuning functionality makes it possible to 556 507 adapt models to specific domains and clients, which Samuel Laubli,¨ Chantal Amrhein, Patrick Duggelin,¨ 557 508 is vital when using MT for professional translation. Beatriz Gonzalez, Alena Zwahlen, and M. Volk. 558 OPUS-CAT also contains plugins for several major 2019. Post-editing productivity with neural machine 509 translation: An empirical assessment of speed and 559 510 CAT tools, and exposes an API which can be used quality in the banking and finance domain. In MT- 560 511 in integrations with other tools. The OPUS-CAT Summit. 561 512 plugin SDL Trados is especially well suited for 562 integration into translation workflows due to its Minh-Thang Luong and Christopher D. Manning. 2015. 513 Stanford neural machine translation systems for spo- 563 sophisticated fine-tuning functionality, which is 514 ken language domains. 564 implemented as a workflow task. OPUS-CAT is 515 565 licensed under the MIT License, and the source Lieve Macken, Daniel Prou, and Arda Tezcan. 2020. 516 566 code and software releases are available at https: Quantifying the effect of machine translation in a 517 high-quality human translation production process. 567 //github.com/Helsinki-NLP/OPUS-CAT. 518 Informatics, 7. 568 519 Matt Post. 2018. A call for clarity in reporting BLEU 569 520 References scores. In Proceedings of the Third Conference on 570 521 Machine Translation: Research Papers, pages 186– 571 Bergamot. 2021. Bergamot. 191, Belgium, Brussels. Association for Computa- 522 tional Linguistics. 572 523 Stephen Doherty, Federico Gaspari, Declan Groves, 573 Josef Genabith, Lucia Specia, Arle Lommel, 524 John Richards. 1994. LogoVista E to J. In Proceedings 574 Aljoscha Burchardt, and Hans Uszkoreit. 2013. of the First Conference of the Association for Ma- 525 Mapping the industry i: Findings on translation tech- chine Translation in the Americas, Columbia, Mary- 575 526 nologies and quality assessment. Globalization and land, USA. 576 Localization Association. 527 577 Rico Sennrich, Barry Haddow, and Alexandra Birch. 528 European Commission. 2019. Tender specifications: 578 2016. Neural machine translation of rare words Translation of european union documents. 529 with subword units. In Proceedings of the 54th An- 579 530 FIT Europe, EUATC, ELIA, GALA, and LINDWeb. nual Meeting of the Association for Computational 580 Linguistics (Volume 1: Long Papers), pages 1715– 531 2020. European language industry survey 2020. 581 1725, Berlin, Germany. Association for Computa- 532 Greg Hanneman and Georgiana Dinu. 2020. How tional Linguistics. 582 533 should markup tags be translated? In Proceedings of 583 534 the Fifth Conference on Machine Translation, pages Slate Rocks! 2021. Slate desktop. 584 1160–1173, Online. Association for Computational 535 585 Linguistics. Pilar Sanchez-Gij´ on,´ Joss Moorkens, and Andy Way. 536 2019. Post-editing neural machine translation ver- 586 537 Marcin Junczys-Dowmunt, Roman Grundkiewicz, sus segments. Machine Transla- 587 Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, tion, 33:1–29. 538 588 Tom Neckermann, Frank Seide, Ulrich Germann, Al- 539 ham Fikri Aji, Nikolay Bogoychev, Andre´ F. T. Mar- Jorg¨ Tiedemann. 2020. The Tatoeba Translation Chal- 589 540 tins, and Alexandra Birch. 2018. Marian: Fast neu- lenge – Realistic data sets for low resource and mul- 590 tilingual MT. In Proceedings of the Fifth Confer- 541 ral machine translation in C++. In Proceedings of 591 ACL 2018, System Demonstrations, Melbourne, Aus- ence on Machine Translation (Volume 1: Research 542 tralia. Papers). Association for Computational Linguistics. 592 543 593 Jorg¨ Tiedemann, Tommi Nieminen, Mikko Aulamo, 544 Pawel Kamocki, Jim O’Regan, and Marc Stauch. 2016. 594 All Your Data Are Belong to us . European Perspec- Jenna Kanerva, Akseli Leino, Filip Ginter, and Niko 545 tives on Privacy Issues in ‘Free’ Online Machine Papula. 2020. The FISKMO¨ project: Resources 595 546 Translation Services. and tools for Finnish-Swedish machine translation 596 and cross-linguistic research. In Proceedings of 547 597 Philipp Koehn and Josh Schroeder. 2007. Experiments the 12th Language Resources and Evaluation Con- 548 in domain adaptation for statistical machine trans- ference, pages 3808–3815, Marseille, France. Euro- 598 549 lation. In Proceedings of the Second Workshop pean Language Resources Association. 599

293

6 EACL 2021 Submission ***. Confidential Review Copy. DO NOT DISTRIBUTE.

600 Jorg¨ Tiedemann and Santhosh Thottingal. 2020. 650 601 OPUS-MT — Building open translation services for 651 Proceedings of the 22nd Annual Con- 602 the World. In 652 ferenec of the European Association for Machine 603 Translation (EAMT), Lisbon, Portugal. 653 604 654 Jorg¨ Tiedemann. 2012. Parallel data, tools and inter- 605 655 faces in opus. In Proceedings of the Eight Interna- 606 tional Conference on Language Resources and Eval- 656 607 uation (LREC’12), Istanbul, Turkey. European Lan- 657 608 guage Resources Association (ELRA). 658 609 659 610 660 611 661 612 662 613 663 614 664 615 665 616 666 617 667 618 668 619 669 620 670 621 671 622 672 623 673 624 674 625 675 626 676 627 677 628 678 629 679 630 680 631 681 632 682 633 683 634 684 635 685 636 686 637 687 638 688 639 689 640 690 641 691 642 692 643 693 644 694 645 695 646 696 647 697 648 698 649 699

294

7