Python Focus Group Phase 2 or "Accelerating Python"

Felix Ehm, Ivan Sinkarenko, Vito Baggiolini, BE-CO-APS

1 Introduction

Vito Baggiolini

2 History of Python in the Acc Sector and CO

• 2015: request from MD and equipment groups to CO • 2016 1st BE Seminar on “Python tools for machine data analysis and equipment control” with 60 participants • 2016: Evian presentation “Breaking the Wall between OP and Expert Tools”  Collaborate to make Expert GUIs usable also in operations • 2016: Evian: CO announces Python Focus Group (Chaired by David Cobas) • 2017: CO provides a standard Python distribution based on EP/SFT distrib. • 2017: Evian presentation “A users view on exploiting the control system in MDs”  Promote Python to make useful tools both for MD and operations • 2018: Two new LD allocated to CO: one for Python GUIs/RAD (here) and one for Python as a Platform (coming)

3 CO is now fully committed to supporting Python

• Main working areas • Python as a Development Platform => Felix + 2nd LD • Python for GUIs/RAD => Ivan (1st LD) • Python APIs to the control system => Vito + 2nd LD

• CO uses Python also elsewhere: • NXCALS (Spark Python API) • Unified Controls Acquisition Processing (UCAP) virtual device framework • LN4 source control • Encore/Edge, Cheby • Etc. How and what we want to contribute • Join, participate and accelerate ;-) • Identify users and contributors; understand your needs and requirements • Consolidate and improve (the partly existing) infrastructure and tools • Contribute to Python Controls APIs (pyJAPC, pjLSA, pyRDA, ...), and gradually take over maintenance and support • Organize training courses and promote other learning resources • Recommend SW engineering best practices, especially for operational software • We hope that the community will remain as active as they are now • Learn from your existing expertise • Develop together with you, support new ideas • Rely on you for local first-line support • This is just the beginning of a new Phase - please shape this work with us!

5 Python as a development Platform

Felix Ehm - Development Tools Team

6 Content

• Context & Current State • Plans for 2019 • Further Contributions • Summary

7 Context

CO got officially appointed to • Provide a standardized development & execution environment for Python code • Take over existing Python libraries for accelerator controls

Development Tools Team

Felix Zsolt New Staff Mikkel ~July 2019 Feb 2019

8 Context

April 2017: Initiation of the Python focus group Dec 2017: A Python(3) distribution available

Since end of 2018 • We reviewed of current base Python distribution • We collected use cases and requirements from developers (ABP, BI, RF, ICS, CO, OP, MPE, EPC)

So far, our (little) work was in the background - now we like to join the community actively.

9 Current State

• BE-CO Python distro based on LCG-93 community version from EP-SFT • Current version from 2/2018, NFS based, available from TN • Packaged by David with a set of commonly used libraries (requests, numpy, scipy, ..) • Activated by a virtual environment • Third-party packages from pypi.org are installed through private SSH tunnels • Newer version of module is requested • TN is a restriction to reach packages in global index • Code is shared/re-used in several ways: distro, GIT, NFS or pypi.org • Python APIs for Controls are widely used • So far developed & maintained by individual users • CO has taken over responsibility for pyJAPC • Various ways of deploying/running Python code: NFS, GIT

10 Outstanding requests we got…

• Packaged Python distribution for local installation • Access to 3rd Party libraries on pypi.org • Access to newer package than the one in the distribution Most • Consensus on coding standards wanted • Standards on how to deploy and run Python code • Develop / run on TN / GPN

11 The Process – An Iterative Evolution

Initial Situation

Build upon the existing decisions & profit from existing expertise in the community Rollout Proposal

Review Implementation

12 Plans for 1st half 2019

• Python Distribution • Update 3rd Party modules according to LCG95 • Provide installable package & organize deployment on Controls machines • Decide on where in-house packages will be shared • Repository (Python Package Index) Service • Creation of an in-house package index for uploading packages (also from TN) • Enable access to pypi.org for 3rd party packages (also from TN) • Recommendations for tools, procedures, configuration • Maintenance for existing Control APIs = > see Vito’s slide

13 Plans for 2nd half 2019 - Further Contributions

• Development Environment • Provision of IDE and bootstrap mechanism => automate common tasks • Support usage of quality assurance tools/procedures • Advocate of Python best practices E.g. code formatting, testing, dependency management

• Deployment & Execution in the operational environment • Recommendations for deploying and running Python software • Recommendation for sharing packages

• Lifecycle management of products • Releasing new version, deprecation, cleanup

14 Current Progress and next steps

• A Nexus PyPI is available for testing: http://acc-python-repo:8081/ • Allows access to pypi.org & upload packages (GPN & TN)

• New distribution expected for mid April 2019 for testing • Based on own view from SFT team: /cvmfs/sft.cern.ch/lcg/views/devBE independent from LCG built - we can add/remove packages as we like • Entry point is no longer a virtual environment

• First version of bootstrap mechanism April 2019 • Comes with pip config, linting tools, supports upload of packages

15 Organization

• Entry point: Python Wikis Getting started guide, recommendations, communication links,..

• Stay connected: AccPy Mattermost - post questions & answer them :-) • “I have a problem with pyTIMBER: … Anyone experiencing the same?” • “Can we upgrade package XXX in the distro because of …” • “#release #pyJAPC New pyJAPC 3.2 available to testing.”

• More regular Accelerating-Python meetings

16 Summary

• We join the Python party :-) • We want to evolve and prosper the Python ecosystem • We have concrete plans to address high priority demands • We seek collaboration to create your development environment

Getting started https://wikis.cern.ch/display/ACCPY/ Support & Requests: [email protected] Mattermost: https://mattermost.web.cern.ch/py-acc

17 Python for GUI & Rapid Application Development Ivan Sinkarenko

18 Content

• Current situation & vision • GUI Platform • Rapid Application Development • Knowledge base • Plans for 2019

19 Current situation

Operational applications in CCC are implemented using: • /JavaFX • Inspector • PyQt • ...

So far, there was no officially recognized way to develop Python GUIs. • It lead to many ad-hoc applications and fragmented development.

20 Vision for Python GUI development

We are aiming to offer: • A PyQt-based ecosystem • Custom set of widgets • RAD framework • Guidance & best practices

21 What we’ve done so far…

• Update of the strategy for GUIs • Evaluated QML • Analysis of charting libraries • Proof of concept for PS fixed display • Proof of concept for RAD

22 Qt

Qt is a cross-platform ++ framework that has been most known for GUI development. It is used in a wide variety of industries, from coffee- machine screens to automotive dashboards and smartphones.

• 2 ways of developing GUIs: Qt Widgets & QtQuick/QML Widgets • Shipped as a set of libraries. • Full distribution comes with development tools • → Main IDE for C++/Qt • Qt Designer → WYSIWYG editor for Qt Widgets • QtQuick Designer → WYSIWYG editor for QtQuick widgets • etc...

23 PyQt

PyQt is a set of Python bindings to Qt • Powered by SIP binding generator • Both PyQt and SIP are made by Riverbank company

Just like Qt, PyQt provides flexible developer experience: • WYSIWYG (Using Qt Designer) for simple use-cases • Code-centric approach • Hybrid

24 Planned PyQt platform

PyQt distribution will be decoupled from Python distribution • Qt 5.12 (LTS) • PyQt 5.12 (+ Qt Designer plugin) • SIP 4.19

Keep in mind: • GUI development will be based on Qt Widgets • No plans to support QML, but we still can include it in the distribution

25 Planned PyQt platform

Qt Designer available as a standalone application. • Qt Creator is not part of the distribution • Code editing is done in Python IDE of your choice (we like PyCharm)

Possibilities to tailor Qt Designer to be more developer-friendly: • Custom widgets • Property sheet • Context menu • Action sheet

26 Qt Designer

Example plugin

27 Custom PyQt widgets

Besides the standard Qt Widgets, we want to provide a set of custom widgets common in CERN applications, e.g. • Spinner control • Charts • ...

Users can develop and share their own widgets • Sharing approach can be later agreed with the devtools team

28 Rapid Application Development in the Acc Sector

Rapid Application Development (RAD) • Enable people who don’t have much knowledge about software to develop simple GUIs • Ideal for prototypes & throw-away applications, however... • When something simple is created at first, it could be later evolved into an operational application for the control room.

We want to enable evolution of the RAD GUIs into full-fledged PyQt GUIs! • Might require some additional expertise

29 Rapid Application Development in the Acc Sector

Example use-cases: • Applications needed during development of a machine • Applications for equipment testing • Fixed displays (Vistars)

30 Planned RAD framework

• Based on PyQt • Launcher + frame of the application (PyDM @ SLAC) • Device/property model integration provided by the framework • Widgets connected to the control system • Qt Designer enhancements • Migration helper for Inspector users

31 RAD workflows

Basic users • Develop an application entirely in Qt Designer (with no or little code)

Intermediate users • Develop an application in Qt Designer mixing with Python code-behind • Migrate away from RAD to a standalone PyQt application

Expert users • Develop their own custom widgets • Develop their own Qt Designer extensions

32 Community help

We invite “adventurous souls” who want to become early adopters and give feedback: • To test RAD environment • To test charting components (BI?)

Another presentation concerning RAD framework with more detailed look will be following up soon!

33 Guidance & best practices for Python GUIs

Knowledge base could include: • Naming conventions (ComboBox or Combobox or DropdownList?) • Ways to structure the GUI code (decoupling presentation and logic) • Testing GUIs (pytest, QTest) Resources • Wikis • Mattermost • Public websites: Real Python, Python School, The Hitchhiker’s Guide to Python • Training for PyQt GUI development (with CERN training centre)

Let’s build common knowledge together!

34 Plans for 1st half 2019

• PyQt distribution, aligned with Python distribution • Getting started guide • PyQt GUI training with the CERN training centre • RAD v0.1-beta Requests for RAD alpha/beta testing are welcome!

35 Plans for 2nd half 2019

• RAD v0.1, stable release • Getting started guide • (Some) custom widgets • Qt Designer customization improvements • Migration helper improvements

36 Summary

• PyQt is our choice for Python GUIs • We’ll be providing an independent PyQt distribution • Help us shape Rapid Application Development with Python • Join the next RAD presentation • Get involved in the feedback loop • Look out for PyQt GUI training announcement

37 Python Controls APIs

Vito Baggiolini

38 Python APIs to the control system • Controls APIs • Give Python access to the controls system, e.g. pyJAPC, pjLSA, pyTIMBER, ... • Typically wrappers around existing Java APIs, using JPype • Developed by active developers in different groups • Our plans/proposals • Progressively accept responsibility for support/maintenance • Contribute other APIs (e.g. pyRDA) • Progressively replace Java/JPype wrappers by native bindings to C/C++ or REST APIs • Smooth API upgrades, no “brutal” changes • Development needs to be coordinated and aligned • Formatting, naming conventions, project layout (c.f. Felix’ presentation) • Agree on development workflow, e.g. using gitlab, issues, PR; testing and CI

39 Concrete plans for 2019

First half of 2019 • Provide support/maintenance for pyJAPC and cmmnbuild-dep-mgr • Update to latest version of JAPC (code and doc) • Write tests, Possibly add small features More in an upcoming • Better understand how JPype works "Accelerating Python" • Participate in discussions about other APIs meeting • E.g. to encourage common style, best practices, testing

Second half of 2019 (when the new Python Staff is here) • Take over support + maintenance for more APIs • Actively contribute development to existing and new Controls APIs

40 Summary

• CO now strongly supports and promotes Python in the accelerator sector • We have two new staff - one here, one expected to come in July • We plan to contribute with • A Python development platform • A framework for GUI and RAD • Controls APIs • Recommendations, mentoring, training • [ whatever there is a need for ]

• Take part: as normal user, beta tester, code contributor, visionary, ...

41