Python Focus Group Phase 2 or "Accelerating Python" Felix Ehm, Ivan Sinkarenko, Vito Baggiolini, BE-CO-APS 1 Introduction Vito Baggiolini 2 History of Python in the Acc Sector and CO • 2015: request from MD and equipment groups to CO • 2016 1st BE Seminar on “Python tools for machine data analysis and equipment control” with 60 participants • 2016: Evian presentation “Breaking the Wall between OP and Expert Tools” Collaborate to make Expert GUIs usable also in operations • 2016: Evian: CO announces Python Focus Group (Chaired by David Cobas) • 2017: CO provides a standard Python distribution based on EP/SFT distrib. • 2017: Evian presentation “A users view on exploiting the control system in MDs” Promote Python to make useful tools both for MD and operations • 2018: Two new LD allocated to CO: one for Python GUIs/RAD (here) and one for Python as a Platform (coming) 3 CO is now fully committed to supporting Python • Main working areas • Python as a Development Platform => Felix + 2nd LD • Python for GUIs/RAD => Ivan (1st LD) • Python APIs to the control system => Vito + 2nd LD • CO uses Python also elsewhere: • NXCALS (Spark Python API) • Unified Controls Acquisition Processing (UCAP) virtual device framework • LN4 source control • Encore/Edge, Cheby • Etc. How and what we want to contribute • Join, participate and accelerate ;-) • Identify users and contributors; understand your needs and requirements • Consolidate and improve (the partly existing) infrastructure and tools • Contribute to Python Controls APIs (pyJAPC, pjLSA, pyRDA, ...), and gradually take over maintenance and support • Organize training courses and promote other learning resources • Recommend SW engineering best practices, especially for operational software • We hope that the community will remain as active as they are now • Learn from your existing expertise • Develop together with you, support new ideas • Rely on you for local first-line support • This is just the beginning of a new Phase - please shape this work with us! 5 Python as a development Platform Felix Ehm - Development Tools Team 6 Content • Context & Current State • Plans for 2019 • Further Contributions • Summary 7 Context CO got officially appointed to • Provide a standardized development & execution environment for Python code • Take over existing Python libraries for accelerator controls Development Tools Team Felix Zsolt New Staff Mikkel ~July 2019 Feb 2019 8 Context April 2017: Initiation of the Python focus group Dec 2017: A Python(3) distribution available Since end of 2018 • We reviewed of current base Python distribution • We collected use cases and requirements from developers (ABP, BI, RF, ICS, CO, OP, MPE, EPC) So far, our (little) work was in the background - now we like to join the community actively. 9 Current State • BE-CO Python distro based on LCG-93 community version from EP-SFT • Current version from 2/2018, NFS based, available from TN • Packaged by David with a set of commonly used libraries (requests, numpy, scipy, ..) • Activated by a virtual environment • Third-party packages from pypi.org are installed through private SSH tunnels • Newer version of module is requested • TN is a restriction to reach packages in global index • Code is shared/re-used in several ways: distro, GIT, NFS or pypi.org • Python APIs for Controls are widely used • So far developed & maintained by individual users • CO has taken over responsibility for pyJAPC • Various ways of deploying/running Python code: NFS, GIT 10 Outstanding requests we got… • Packaged Python distribution for local installation • Access to 3rd Party libraries on pypi.org • Access to newer package than the one in the distribution Most • Consensus on coding standards wanted • Standards on how to deploy and run Python code • Develop / run on TN / GPN 11 The Process – An Iterative Evolution Initial Situation Build upon the existing decisions & profit from existing expertise in the community Rollout Proposal Review Implementation 12 Plans for 1st half 2019 • Python Distribution • Update 3rd Party modules according to LCG95 • Provide installable package & organize deployment on Linux Controls machines • Decide on where in-house packages will be shared • Repository (Python Package Index) Service • Creation of an in-house package index for uploading packages (also from TN) • Enable access to pypi.org for 3rd party packages (also from TN) • Recommendations for tools, procedures, configuration • Maintenance for existing Control APIs = > see Vito’s slide 13 Plans for 2nd half 2019 - Further Contributions • Development Environment • Provision of IDE and bootstrap mechanism => automate common tasks • Support usage of quality assurance tools/procedures • Advocate of Python best practices E.g. code formatting, testing, dependency management • Deployment & Execution in the operational environment • Recommendations for deploying and running Python software • Recommendation for sharing packages • Lifecycle management of products • Releasing new version, deprecation, cleanup 14 Current Progress and next steps • A Nexus PyPI is available for testing: http://acc-python-repo:8081/ • Allows access to pypi.org & upload packages (GPN & TN) • New distribution expected for mid April 2019 for testing • Based on own view from SFT team: /cvmfs/sft.cern.ch/lcg/views/devBE independent from LCG built - we can add/remove packages as we like • Entry point is no longer a virtual environment • First version of bootstrap mechanism April 2019 • Comes with pip config, linting tools, supports upload of packages 15 Organization • Entry point: Python Wikis Getting started guide, recommendations, communication links,.. • Stay connected: AccPy Mattermost - post questions & answer them :-) • “I have a problem with pyTIMBER: … Anyone experiencing the same?” • “Can we upgrade package XXX in the distro because of …” • “#release #pyJAPC New pyJAPC 3.2 available to testing.” • More regular Accelerating-Python meetings 16 Summary • We join the Python party :-) • We want to evolve and prosper the Python ecosystem • We have concrete plans to address high priority demands • We seek collaboration to create your development environment Getting started https://wikis.cern.ch/display/ACCPY/ Support & Requests: [email protected] Mattermost: https://mattermost.web.cern.ch/py-acc 17 Python for GUI & Rapid Application Development Ivan Sinkarenko 18 Content • Current situation & vision • GUI Platform • Rapid Application Development • Knowledge base • Plans for 2019 19 Current situation Operational applications in CCC are implemented using: • Swing/JavaFX • Inspector • PyQt • ... So far, there was no officially recognized way to develop Python GUIs. • It lead to many ad-hoc applications and fragmented development. 20 Vision for Python GUI development We are aiming to offer: • A PyQt-based ecosystem • Custom set of widgets • RAD framework • Guidance & best practices 21 What we’ve done so far… • Update of the strategy for GUIs • Evaluated QML • Analysis of Qt charting libraries • Proof of concept for PS fixed display • Proof of concept for RAD 22 Qt Qt is a cross-platform C++ framework that has been most known for GUI development. It is used in a wide variety of industries, from coffee- machine screens to automotive dashboards and smartphones. • 2 ways of developing GUIs: Qt Widgets & QtQuick/QML Widgets • Shipped as a set of libraries. • Full distribution comes with development tools • Qt Creator → Main IDE for C++/Qt • Qt Designer → WYSIWYG editor for Qt Widgets • QtQuick Designer → WYSIWYG editor for QtQuick widgets • etc... 23 PyQt PyQt is a set of Python bindings to Qt • Powered by SIP binding generator • Both PyQt and SIP are made by Riverbank company Just like Qt, PyQt provides flexible developer experience: • WYSIWYG (Using Qt Designer) for simple use-cases • Code-centric approach • Hybrid 24 Planned PyQt platform PyQt distribution will be decoupled from Python distribution • Qt 5.12 (LTS) • PyQt 5.12 (+ Qt Designer plugin) • SIP 4.19 Keep in mind: • GUI development will be based on Qt Widgets • No plans to support QML, but we still can include it in the distribution 25 Planned PyQt platform Qt Designer available as a standalone application. • Qt Creator is not part of the distribution • Code editing is done in Python IDE of your choice (we like PyCharm) Possibilities to tailor Qt Designer to be more developer-friendly: • Custom widgets • Property sheet • Context menu • Action sheet 26 Qt Designer Example plugin 27 Custom PyQt widgets Besides the standard Qt Widgets, we want to provide a set of custom widgets common in CERN applications, e.g. • Spinner control • Charts • ... Users can develop and share their own widgets • Sharing approach can be later agreed with the devtools team 28 Rapid Application Development in the Acc Sector Rapid Application Development (RAD) • Enable people who don’t have much knowledge about software to develop simple GUIs • Ideal for prototypes & throw-away applications, however... • When something simple is created at first, it could be later evolved into an operational application for the control room. We want to enable evolution of the RAD GUIs into full-fledged PyQt GUIs! • Might require some additional expertise 29 Rapid Application Development in the Acc Sector Example use-cases: • Applications needed during development of a machine • Applications for equipment testing • Fixed displays (Vistars) 30 Planned RAD framework • Based on PyQt • Launcher + frame of the application (PyDM @ SLAC) • Device/property model integration provided