Introduction Jupyter Prof Hans Fangohr, 5 October 2019 1 09:00 - 09:30 Jupyter Notebook and Ecosystem 30', Hans Fangohr (European XFEL GmbH) 09:30 - 10:00 Status updates from facilities ► Jupyter at Synchrotron Soleil (Alain Buteau & Gwenaelle Abeille) 5' ► Jupyter at CERN (Manuel Gonzalez & Jakub Wozniak) 5' ► Jupyter at European Southern Observatory (Gianluca Chiozzi) 5' ► Jupyter at J-PARC MLF (Kentaro Moriyama) 5’ ► Jupyter at MAX IV Synchrotron (Vincent Hardion) 5’ 10:00 - 10:30 Coffee break 10:30 - 12:30 JupyterLab Tutorial 2h0', Saul Shanabrook (Quansight) 12:30 - 14:00 Lunch and networking 14:00 - 14:30 Jupyter for processing neutron event data at ESS 30', Dr. Jonathan Taylor (European Spallation Source) 14:30 - 15:00 Jupyter at Brookhaven National Laboratory 30', Dr. Daniel B. Allan (Brookhaven National Lab) 15:00 - 15:20 Jupyter for Accelerator Physics 20', Dr. Jonathan Edelen (RadiaSoft LLC) 15:30 - 16:00 Coffee break 16:00 - 17:00 Questions and answers, discussion, Hans Fangohr (European XFEL GmbH) Introduction Jupyter notebook and ecosystem Hans Fangohr Data Analysis European X-ray Free Electron Laser (EuXFEL), Germany Professor of Computational Modelling University of Southampton, United Kingdom [email protected] @ProfCompMod Jupyter Workshop, ICALEPCS 2019, New York, US, Saturday 5 October 2019 https://indico.desy.de/indico/event/23354 Introduction Jupyter Prof Hans Fangohr, 5 October 2019 3 Outline Quick introduction Jupyter Notebook Introduce a number of use cases (most from European XFEL) Introduce a number of tools from Project Jupyter Format (for the day) Informal Ask questions to make it informal To guide the presenter what is useful Slides will be made available on Workshop site (speakers please send to Hans Fangohr) In [1]: # Import Python and libraries we need later %matplotlib inline Introduction Jupyter Prof Hans Fangohr,from numpy 5 October import 2019 exp, cos, linspace import pylab 4 from ipywidgets import interact Mathematical model: We would like to understand !(", #, $) = exp(−#") cos($") Jupyter Notebook Code: Here is an implementation: In [2]: def f(t, alpha, omega): """Computes and returns exp(-alpha*t) * cos(omega*t)""" Document hosted in web browser (demo) return exp(-alpha * t) * cos(omega * t) Interactive exploration: We can execute the function for values of ", # and $: Combines In [3]: f(t=0.1, alpha=1, omega=10) text (markdown with LaTeX support) Out[3]: 0.48888574340060287 Computer code (Python) Or produce a plot (in a function plot_f so it can be re-used for different parameters): Output from code In [4]: def plot_f(alpha, omega): ts = linspace(0, 5, 500) # 500 points in interval [0, 5] ys = f(ts, alpha, omega) Saved in one document pylab.plot(ts, ys, '-') *.ipynb [IPYthon NoteBook] In [5]: plot_f(alpha=0.1, omega=10) # call function and create plot Combines input and output cells JSON format Using interaction widgets, we can re-trigger execution of the plot_f function via GUI elements such as sliders. In [6]: interact(plot_f, alpha=(0, 2, 0.1), omega=(0, 50, 0.5)); alpha 1.50 omega 45.50 Conclusion: We observe that parameter # is responsible for damping, and $ for the frequency. Introduction Jupyter Prof Hans Fangohr, 5 October 2019 5 Toolbar Notebook name Kernel name Markdown cell Currently selected cell Code cell Code output In [1]: # Import Python and libraries we need later %matplotlib inline Introduction Jupyter Prof Hans Fangohr, 5 October 2019from numpy import exp, cos, linspace import pylab 6 from ipywidgets import interact Mathematical model: We would like to understand !(", #, $) = exp(−#") cos($") History of Jupyter Code: Here is an implementation: In [2]: def f(t, alpha, omega): """Computes and returns exp(-alpha*t) * cos(omega*t)""" return exp(-alpha * t) * cos(omega * t) IPython à IPYthon NoteBook à *.ipynb Interactive exploration: We can execute the function for values of ", # and $: In [3]: f(t=0.1, alpha=1, omega=10) IPython notebooks were language agnostic, as they run Out[3]: 0.48888574340060287 over open network protocols Or produce a plot (in a function plot_f so it can be re-used for different parameters): In [4]: def plot_f(alpha, omega): The community began adding other languages to the ts = linspace(0, 5, 500) # 500 points in interval [0, 5] ys = f(ts, alpha, omega) notebooks, starting with Julia and R pylab.plot(ts, ys, '-') In [5]: plot_f(alpha=0.1, omega=10) # call function and create plot As the proJect expanded away from Just Python, the notebooks had to be renamed JUlia, PYThon, and R à Jupyter “Jupyter” is also a homage to Galileo’s notebooks which recorded the discovery of Jupiter’s moons Using interaction widgets, we can re-trigger execution of the plot_f function via GUI elements such as sliders. In [6]: interact(plot_f, alpha=(0, 2, 0.1), omega=(0, 50, 0.5)); alpha 1.50 omega 45.50 Conclusion: We observe that parameter # is responsible for damping, and $ for the frequency. Introduction Jupyter Prof Hans Fangohr, 5 October 2019 7 How does it work? Starting jupyter-notebook starts the notebook server Opening a notebook starts up the kernel The kernel communicates to the notebook server via ZMQ The notebook server shows content via the browser JavaScript used in Browser Introduction Jupyter Prof Hans Fangohr, 5 October 2019 8 Supported Kernels 50+ languages supported. More complete list at https://github.com/jupyter/jupyter/wiki/Jupyter-kernels Introduction Jupyter Prof Hans Fangohr, 5 October 2019 9 Use case 1: data analysis in notebook Explorative data analysis Convenient combination of processing, results and interpretation Complete capture of all computational steps good record for reproducibility and re-use ► FAIR data Through export to HTML, easy to share with collaborators & supervisors Scientists are confident drivers of this Example on the right from SCS instrument Introduction Jupyter Prof Hans Fangohr, 5 October 2019 10 Sharing and exporting notebooks Can share *.ipynb files Can be displayed using jupyter-notebook Code can be re-executed on collaborator‘s machine ► Only if software is available, and data is available, and collaborators know how to start notebook “Static sharing“ through html and email (for example) Often sufficient – does not require installation or additional skills ► Effective to communicate with supervisors, line managers etc Convert using menu “File -> Download as -> html“ Or using nbconvert: $ jupyter-nbconvert --to html 2-widgets.ipynb [NbConvertApp] Converting notebook 2-widgets.ipynb to html [NbConvertApp] Writing 382609 bytes to 2-widgets.html Introduction Jupyter Prof Hans Fangohr, 5 October 2019 11 Use case 2: notebooks as recipes Pre-populate notebook with cells to carry out a particular type of data analysis Provide a directory full of such recipes to users Users execute cells during beamtime and later Convenient compromise between Static recipe (=script) Interactive exploration Experience Keep code in notebook cells short and move functionality into library (here “ToolBox“) Archive directory of modified recipes with data Introduction Jupyter Prof Hans Fangohr, 5 October 2019 12 Use case 4: notebooks as a script Use Jupyter Notebook as a script Can execute using nbconvert to take commands in notebook, execute them, save resulting notebook. Can create data files and plots in process. Use case: detector calibration pipeline Use of nbparametrize (or papermill) to insert run parameters into notebook before execution Automatic execution and creation of pdf Error messages embedded in output Introduction Jupyter Prof Hans Fangohr, 5 October 2019 13 Executing a notebook from the command line (nbconvert) Convert from ipynb to html: $ jupyter-nbconvert --to html 2-widgets.ipynb Can optionally set output name $ jupyter-nbconvert --to html --output myout.html 2-widgets.ipynb Can execute notebook before conversion: $ jupyter-nbconvert --execute --to html --output myout.html 2-widgets.ipynb Execute notebook and save results as notebook: $ jupyter-nbconvert --execute --to ipynb --output myout.ipynb 2-widgets.ipynb [NbConvertApp] Converting notebook 2-widgets.ipynb to ipynb [NbConvertApp] Executing notebook with kernel: python3 [NbConvertApp] Writing 103398 bytes to myout.ipynb Introduction Jupyter Prof Hans Fangohr, 5 October 2019 14 nbconvert nbconvert is a tool which can convert notebooks to various other formats such as: HTML, LaTeX, PDF, Markdown, ReStructuredText, Python, … Static HTML pages can be created as documentation or tutorial Nbsphinx -> Jupyter notebook provides section in sphinx documentation Homepage: https://github.com/jupyter/nbconvert Notebook as a script approach Can change parameters inside notebook before execution: ► nbparametrize or papermill Introduction Jupyter Prof Hans Fangohr, 5 October 2019 15 Use case 5: (remote) data analysis environment (JupyterHub) JupyterHub allows to users connect through browser and https use existing authentication systems serve notebooks on facility hardware connect to user’s file storage Example: Jupyter Hub at EuXFEL & DESY Uses Maxwell HPC cluster Popular with users: no software installation & browser of choice works locally and remotely the same Introduction Jupyter Prof Hans Fangohr, 5 October 2019 16 Using remote resources: JupyterHub In principle user can ssh to HPC resource, use port forwarding to connect local machine with HPC computer running notebook server JupyterHub helps orchestrate and manage individual Jupyter instances for multiple users It provides an interface allowing users to easily spawn and connect their own Jupyter Server Different options