Github.Com/Microsoft/Azure-Databricks-Operator

Total Page:16

File Type:pdf, Size:1020Kb

Github.Com/Microsoft/Azure-Databricks-Operator Azure Databricks Operator github.com/microsoft/azure-databricks-operator Azadeh Khojandi Senior Software Engineer @AzadehKhojandi Jordan Knight Principal Software Engineer @jakkaj Agenda Why, What, How https://aka.ms/azure-databricks-operator @AzadehKhojandi @jakkaj Why? Highly-cohesive, loosely-coupled, highly- directional, complex-configuration, multi- component, multi-technology, multi-platform, high-scale, high-availability, low-latency, big- data system. https://aka.ms/azure-databricks-operator A.K.A -> a stream processing pipeline https://aka.ms/azure-databricks-operator Pipelines are complex • Streams • Transforms • Storage • Security • Dev, test, prod Environments • External dependencies (DataBricks, PaaS services) https://aka.ms/azure-databricks-operator Pipelines are complex • Highly cohesive system, loosely coupled components • Relationship is king • Configuration can get messy, is based on relationships • Especially when it doesn’t exist at design time • … like external PaaS services etc. https://aka.ms/azure-databricks-operator Pipelines are complex • Build a re-usable catalog of components • Create many pipelines to suit particular use cases • … easily (maybe even using a UI?!) https://aka.ms/azure-databricks-operator Problem: Building it We had to build a pipeline system that: • Has unknown component config and relationships at build time • Is highly flexible and re-usable • Has 1-n custom transforms • Forks in data https://aka.ms/azure-databricks-operator Idea: Central Configuration • Single point of configuration for a pipeline • All components, relationships, parameters defined in the same place • Sounds like a DAG! https://aka.ms/azure-databricks-operator DAG as config • Start with a DAG view of the system • Can visualize the DAG, get it right • DevOps takes the DAG and “makes it real” • DAG is essentially converted to Helm Charts (using a special tool we’ve built) https://aka.ms/azure-databricks-operator Kubernetes to the rescue • Create desired state configuration platform • Flexible extension options (CRDs and operators!) https://aka.ms/azure-databricks-operator Custom components • … like DataBricks jobs, EventHubs etc • Operators to the rescue https://aka.ms/azure-databricks-operator Operators • Allow us to create a re-usable module simply • Represents as a first class object in the system • Easy to represent in the DAG • Easy to represent in Helm • Easy to deploy, update and remove • … are becoming well known, skills building in industry https://aka.ms/azure-databricks-operator What? https://docs.microsoft.com/en-us/azure/azure-databricks • Large-scale data processing • Speed • Scalable to petabytes of data and clusters of thousands of nodes • Connectors to S3, Azure Blob Storage, Redshift, Kafka • Security • Unified platform and Collaboration https://aka.ms/azure-databricks-operator Master Node 1 API Server Databricks Operator Databricks Job Node … Cluster: … Notebook Path:… Params:… Node n YAML File https://github.com/Azure-Samples/twitter-databricks-analyzer-cicd How? & Lessons learned Kubebuilder https://github.com/kubernetes-sigs/kubebuilder Adding custom resources A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind Webhook admission controllers • Mutating (set values, e.g. defaults) • Validating (synchronous input validation) Schema CRDs • OpenAPI v3 schema definition • Declarative validation (simple) https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ Declarative API A declarative API allows you to declare or specify the desired state of your resource and tries to keep the current state of Kubernetes objects in sync with the desired state Custom resource Custom controller https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ CRD Being used for third-party extensions • Add-ons • Stateful orchestration (e.g. MySQL) • Domain-specific APIs (e.g. Istio) • Higher levels of abstraction (e.g. kNative) More recently in Kubernetes • Storage Snapshots (integrates w/ PersistentVolumes) • RuntimeClass (integrates w/ Kubelet and CRI) • CSI https://speakerdeck.com/thockin/crds-arent-just-for-add-ons Vision Now: most/all new APIs are CRDs Eventually: Everything becomes a CRD (except things to run CRDs) There should be nothing that we can do that you can’t • Built-in: Namespaces, CRDs, Admission, etc. • CRDs: Pods, Services, Nodes, Deployments, ... • Kubernetes is a set of operators Tim Hockin @thockin Principal Software Engineer https://speakerdeck.com/thockin/crds-arent-just-for-add-ons KubeCon NA, Seattle, 12/2018 Kind https://kind.sigs.k8s.io/ go (1.11+) and docker GO111MODULE="on" go get sigs.k8s.io/[email protected] && kind create cluster Dev Containers Dev https://aka.ms/vscode-remote Thank You Star our Github repo, raise your requests or send PR https://github.com/microsoft/azure-databricks-operator @AzadehKhojandi @jakkaj.
Recommended publications
  • AAS Worldwide Telescope: Seamless, Cross-Platform Data Visualization Engine for Astronomy Research, Education, and Democratizing Data
    AAS WorldWide Telescope: Seamless, Cross-Platform Data Visualization Engine for Astronomy Research, Education, and Democratizing Data The Harvard community has made this article openly available. Please share how this access benefits you. Your story matters Citation Rosenfield, Philip, Jonathan Fay, Ronald K Gilchrist, Chenzhou Cui, A. David Weigel, Thomas Robitaille, Oderah Justin Otor, and Alyssa Goodman. 2018. AAS WorldWide Telescope: Seamless, Cross-Platform Data Visualization Engine for Astronomy Research, Education, and Democratizing Data. The Astrophysical Journal: Supplement Series 236, no. 1. Published Version https://iopscience-iop-org.ezp-prod1.hul.harvard.edu/ article/10.3847/1538-4365/aab776 Citable link http://nrs.harvard.edu/urn-3:HUL.InstRepos:41504669 Terms of Use This article was downloaded from Harvard University’s DASH repository, and is made available under the terms and conditions applicable to Open Access Policy Articles, as set forth at http:// nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of- use#OAP Draft version January 30, 2018 Typeset using LATEX twocolumn style in AASTeX62 AAS WorldWide Telescope: Seamless, Cross-Platform Data Visualization Engine for Astronomy Research, Education, and Democratizing Data Philip Rosenfield,1 Jonathan Fay,1 Ronald K Gilchrist,1 Chenzhou Cui,2 A. David Weigel,3 Thomas Robitaille,4 Oderah Justin Otor,1 and Alyssa Goodman5 1American Astronomical Society 1667 K St NW Suite 800 Washington, DC 20006, USA 2National Astronomical Observatories, Chinese Academy of Sciences 20A Datun Road, Chaoyang District Beijing, 100012, China 3Christenberry Planetarium, Samford University 800 Lakeshore Drive Birmingham, AL 35229, USA 4Aperio Software Ltd. Headingley Enterprise and Arts Centre, Bennett Road Leeds, LS6 3HN, United Kingdom 5Harvard Smithsonian Center for Astrophysics 60 Garden St.
    [Show full text]
  • XAMARIN.FORMS for BEGINNERS ABOUT ME Tom Soderling Sr
    XAMARIN.FORMS FOR BEGINNERS ABOUT ME Tom Soderling Sr. Mobile Apps Developer @ Polaris Industries; Ride Command Xamarin.Forms enthusiast DevOps hobbyist & machine learning beginner 4 year XCMD Blog: https://tomsoderling.github.io GitHub: https://github.com/TomSoderling Twitter: @tomsoderling How Deep Pickster Spaniel Is It? THE PLAN • Introduction: Why, What, and When • Overview of Xamarin.Forms Building Blocks • Building a Xamarin.Forms UI in XAML • Data Binding • View Customization • Next Steps & Resources • Please ask any questions that come up! THE PLAN • Introduction: Why, What, and When • Overview of Xamarin.Forms Building Blocks • Building a Xamarin.Forms UI in XAML • Data Binding • View Customization • Next Steps & Resources INTRODUCTION : WHY • WET: the soggy state of mobile app development • Write Everything Twice INTRODUCTION : WHY • WET: the soggy state of mobile app development • Write Everything Twice INTRODUCTION : WHAT • What is Xamarin.Forms? • Cross-platform UI framework • Platforms: • Mobile: iOS 8 and up, Android 4.0.3 (API 15) • Desktop: Windows 10 UWP, MacOS, WFP • Samsung Smart Devices: Tizen INTRODUCTION : WHAT • Brief History: • May 2011, Xamarin founded • MonoTouch and Mono for Android using MonoDevelop IDE • February 2013, release of Xamarin 2.0 • Xamarin Studio IDE & integration with Visual Studio • Renamed to Xamarin.Android and Xamarin.iOS • May 2014, Xamarin.Forms released as part of Xamarin 3 • February 24 2016, Xamarin acquired by Microsoft • Owned, actively developed on, and supported by Microsoft • Free
    [Show full text]
  • Azure Web Apps for Developers Microsoft Azure Essentials
    Azure Web Apps for Developers Microsoft Azure Essentials Rick Rainey Visit us today at microsoftpressstore.com • Hundreds of titles available – Books, eBooks, and online resources from industry experts • Free U.S. shipping • eBooks in multiple formats – Read on your computer, tablet, mobile device, or e-reader • Print & eBook Best Value Packs • eBook Deal of the Week – Save up to 60% on featured titles • Newsletter and special offers – Be the first to hear about new releases, specials, and more • Register your book – Get additional benefits Hear about it first. Get the latest news from Microsoft Press sent to your inbox. • New and upcoming books • Special offers • Free eBooks • How-to articles Sign up today at MicrosoftPressStore.com/Newsletters Wait, there’s more... Find more great content and resources in the Microsoft Press Guided Tours app. The Microsoft Press Guided Tours app provides insightful tours by Microsoft Press authors of new and evolving Microsoft technologies. • Share text, code, illustrations, videos, and links with peers and friends • Create and manage highlights and notes • View resources and download code samples • Tag resources as favorites or to read later • Watch explanatory videos • Copy complete code listings and scripts Download from Windows Store PUBLISHED BY Microsoft Press A division of Microsoft Corporation One Microsoft Way Redmond, Washington 98052-6399 Copyright © 2015 Microsoft Corporation. All rights reserved. No part of the contents of this book may be reproduced or transmitted in any form or by any means without the written permission of the publisher. ISBN: 978-1-5093-0059-4 Microsoft Press books are available through booksellers and distributors worldwide.
    [Show full text]
  • Microsoft Cloud Agreement
    Microsoft Cloud Agreement This Microsoft Cloud Agreement is entered into between the entity you represent, or, if you do not designate an entity in connection with a Subscription purchase or renewal, you individually (“Customer”), and Microsoft Ireland Operations Limited (“Microsoft”). It consists of the terms and conditions below, Use Rights, SLA, and all documents referenced within those documents (together, the “agreement”). It is effective on the date that your Reseller provisions your Subscription. Key terms are defined in Section 10. 1. Grants, rights and terms. All rights granted under this agreement are non-exclusive and non-transferable and apply as long as neither Customer nor any of its Affiliates is in material breach of this agreement. a. Software. Upon acceptance of each order, Microsoft grants Customer a limited right to use the Software in the quantities ordered. (i) Use Rights. The Use Rights in effect when Customer orders Software will apply to Customer’s use of the version of the Software that is current at the time. For future versions and new Software, the Use Rights in effect when those versions and Software are first released will apply. Changes Microsoft makes to the Use Rights for a particular version will not apply unless Customer chooses to have those changes apply. (ii) Temporary and perpetual licenses. Licenses available on a subscription basis are temporary. For all other licenses, the right to use Software becomes perpetual upon payment in full. b. Online Services. Customer may use the Online Services as provided in this agreement. (i) Online Services Terms. The Online Services Terms in effect when Customer orders or renews a subscription to an Online Service will apply for the applicable subscription term.
    [Show full text]
  • Licensing Information User Manual Release 9.1 F13415-01
    Oracle® Hospitality Cruise Fleet Management Licensing Information User Manual Release 9.1 F13415-01 August 2019 LICENSING INFORMATION USER MANUAL Oracle® Hospitality Fleet Management Licensing Information User Manual Version 9.1 Copyright © 2004, 2019, Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error- free. If you find any errors, please report them to us in writing. If this software or related documentation is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable: U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs.
    [Show full text]
  • Seeing the Sky Visualization & Astronomers
    Seeing the Sky Visualization & Astronomers Alyssa A. Goodman Harvard Smithsonian Center for Astrophysics & Radcliffe Institute for Advanced Study @aagie WorldWide Telescope Gm1m2 F= gluemultidimensional data exploration R2 Cognition “Paper of the Future” Language* Data Pictures Communication *“Language” includes words & math Why Galileo is my Hero Explore-Explain-Explore Notes for & re-productions of Siderius Nuncius 1610 WorldWide Telescope Galileo’s New Order, A WorldWide Telescope Tour by Goodman, Wong & Udomprasert 2010 WWT Software Wong (inventor, MS Research), Fay (architect, MS Reseearch), et al., now open source, hosted by AAS, Phil Rosenfield, Director see wwtambassadors.org for more on WWT Outreach WorldWide Telescope Galileo’s New Order, A WorldWide Telescope Tour by Goodman, Wong & Udomprasert 2010 WWT Software Wong (inventor, MS Research), Fay (architect, MS Reseearch), et al., now open source, hosted by AAS, Phil Rosenfield, Director see wwtambassadors.org for more on WWT Outreach Cognition “Paper of the Future” Language* Data Pictures Communication *“Language” includes words & math enabled by d3.js (javascript) outputs d3po Cognition Communication [demo] [video] Many thanks to Alberto Pepe, Josh Peek, Chris Beaumont, Tom Robitaille, Adrian Price-Whelan, Elizabeth Newton, Michelle Borkin & Matteo Cantiello for making this posible. 1610 4 Centuries from Galileo to Galileo 1665 1895 2009 2015 WorldWide Telescope Gm1m2 F= gluemultidimensional data exploration R2 WorldWide Telescope gluemultidimensional data exploration WorldWide Telescope gluemultidimensional data exploration Data, Dimensions, Display 1D: Columns = “Spectra”, “SEDs” or “Time Series” 2D: Faces or Slices = “Images” 3D: Volumes = “3D Renderings”, “2D Movies” 4D:4D Time Series of Volumes = “3D Movies” Data, Dimensions, Display Spectral Line Observations Loss of 1 dimension Mountain Range No loss of information Data, Dimensions, Display mm peak (Enoch et al.
    [Show full text]
  • Software License Agreement (EULA)
    Third-party Computer Software AutoVu™ ALPR cameras • angular-animate (https://docs.angularjs.org/api/ngAnimate) licensed under the terms of the MIT License (https://github.com/angular/angular.js/blob/master/LICENSE). © 2010-2016 Google, Inc. http://angularjs.org • angular-base64 (https://github.com/ninjatronic/angular-base64) licensed under the terms of the MIT License (https://github.com/ninjatronic/angular-base64/blob/master/LICENSE). © 2010 Nick Galbreath © 2013 Pete Martin • angular-translate (https://github.com/angular-translate/angular-translate) licensed under the terms of the MIT License (https://github.com/angular-translate/angular-translate/blob/master/LICENSE). © 2014 [email protected] • angular-translate-handler-log (https://github.com/angular-translate/bower-angular-translate-handler-log) licensed under the terms of the MIT License (https://github.com/angular-translate/angular-translate/blob/master/LICENSE). © 2014 [email protected] • angular-translate-loader-static-files (https://github.com/angular-translate/bower-angular-translate-loader-static-files) licensed under the terms of the MIT License (https://github.com/angular-translate/angular-translate/blob/master/LICENSE). © 2014 [email protected] • Angular Google Maps (http://angular-ui.github.io/angular-google-maps/#!/) licensed under the terms of the MIT License (https://opensource.org/licenses/MIT). © 2013-2016 angular-google-maps • AngularJS (http://angularjs.org/) licensed under the terms of the MIT License (https://github.com/angular/angular.js/blob/master/LICENSE). © 2010-2016 Google, Inc. http://angularjs.org • AngularUI Bootstrap (http://angular-ui.github.io/bootstrap/) licensed under the terms of the MIT License (https://github.com/angular- ui/bootstrap/blob/master/LICENSE).
    [Show full text]
  • How Github Secures Open Source Software
    How GitHub secures open source software Learn how GitHub works to protect you as you use, contribute to, and build on open source. HOW GITHUB SECURES OPEN SOURCE SOFTWARE PAGE — 1 That’s why we’ve built tools and processes that allow GitHub’s role in securing organizations and open source maintainers to code securely throughout the entire software development open source software lifecycle. Taking security and shifting it to the left allows organizations and projects to prevent errors and failures Open source software is everywhere, before a security incident happens. powering the languages, frameworks, and GitHub works hard to secure our community and applications your team uses every day. the open source software you use, build on, and contribute to. Through features, services, and security A study conducted by the Synopsys Center for Open initiatives, we provide the millions of open source Source Research and Innovation found that enterprise projects on GitHub—and the businesses that rely on software is now comprised of more than 90 percent them—with best practices to learn and leverage across open source code—and businesses are taking notice. their workflows. The State of Enterprise Open Source study by Red Hat confirmed that “95 percent of respondents say open source is strategically important” for organizations. Making code widely available has changed how Making open source software is built, with more reuse of code and complex more secure dependencies—but not without introducing security and compliance concerns. Open source projects, like all software, can have vulnerabilities. They can even be GitHub Advisory Database, vulnerable the target of malicious actors who may try to use open dependency alerts, and Dependabot source code to introduce vulnerabilities downstream, attacking the software supply chain.
    [Show full text]
  • Cyber AI Security for Microsoft Azure
    Data Sheet Cyber AI Security for Microsoft Azure Powered by self-learning Cyber AI, Darktrace brings real-time visibility and autonomous defense to your Azure cloud. Key Benefits Cyber AI Defense for Dynamic Protecting Azure Cloud From Novel Workforces and Workloads and Advanced Threats ✔ Learns ‘on the job’ to offer The Darktrace Immune System provides a unified, AI-native Data Exfiltration and Destruction continuous, context-based platform for autonomous threat detection, investigation, defense and response in Azure and across the enterprise, ensuring Detects anomalous device connections and user your dynamic workforce is always protected. access, as well as unusual resource deletion, ✔ Offers complete real-time visibility modification, and movement. of your organization’s Azure With advanced Cyber AI, Darktrace builds a deep environment understanding of normal behavior in your Azure cloud Critical Misconfigurations environment to identify even the most subtle deviations ✔ Autonomously neutralizes novel from usual activity that point to a threat – no matter how Identifies unusual permission changes and and advanced threats sophisticated or novel. anomalous activity around compliance-related data ✔ Cyber AI Analyst automates and devices. threat investigation, reducing time to triage by up to 92% Compromised credentials Spots brute-force attempts, unusual login source or time, and unusual user behavior including rule changes and password resets. “Darktrace complements Microsoft’s security products Insider Threat and Admin Abuse with
    [Show full text]
  • Use Chef Automate and Microsoft Azure for Speed, Scale and Consistency
    Use Chef Automate and Microsoft Azure for speed, scale and consistency Together, Chef Automate and Microsoft Azure give you everything you need to deliver infrastructure and applications quickly and safely. You can give your operations and development teams a common pipeline for building, testing, and deploying infrastructure and applications. Use Chef Automate on Azure and take advantage of the flexibility, scalability and reliability that Azure offers. Chef Automate on the Azure Marketplace makes it easy to deploy a fully-featured Chef Automate instance into your own Azure subscription. workflow. When compliance is code you can find problems early in the development process. Chef. With the Chef server and client, you describe your infrastructure as code, which means it’s versionable, human-readable, and testable. You can take advantage of cookbooks provided by the Chef community, which contain code for managing your infrastructure. Habitat. Habitat is automation that travels with the app. Habitat packages contain everything the Chef Automate is the leader in Continuous app needs to run with no outside dependencies. Automation. With Chef Automate, you have Habitat apps are isolated, immutable, and auditable. everything you need to build, deploy and manage They are atomically deployed, with self-organizing your applications and infrastructure at speed. peer relationships. With Habitat, your apps behave Collaborate. Chef Automate provides a pipeline for consistently in any runtime environment. It’s an the continuous deployment of infrastructure and ideal approach for deploying containers using the applications. Chef Automate also includes tools for Azure Container Service (ACS) or managing legacy local development and can integrate with a variety of application stacks using virtual machine instances in third-party products for developer workflows.
    [Show full text]
  • Presented by Alyssa Goodman, Center for Astrophysics | Harvard & Smithsonian, Radcliffe Institute for Advanced Study
    The Radcliffe Wave presented by Alyssa Goodman, Center for Astrophysics | Harvard & Smithsonian, Radcliffe Institute for Advanced Study Nature paper by: João Alves1,3, Catherine Zucker2, Alyssa Goodman2,3, Joshua Speagle2, Stefan Meingast1, Thomas Robitaille4, Douglas Finkbeiner3, Edward Schlafly5 & Gregory Green6 representing (1) University of Vienna; (2) Harvard University; (3) Radcliffe Insitute; (4) Aperio Software; (5) Lawrence Berkeley National Laboratory; (6) Kavli Insitute for Particle Physics and Cosmology The Radcliffe Wave CARTOON* DATA *drawn by Dr. Robert Hurt, in collaboration with Milky Way experts based on data; as shown in screenshot from AAS WorldWide Telescope The Radcliffe Wave Each red dot marks a star-forming blob of gas whose distance from us has been accurately measured. The Radcliffe Wave is 9000 light years long, and 400 light years wide, with crest and trough reaching 500 light years out of the Galactic Plane. Its gas mass is more than three million times the mass of the Sun. video created by the authors using AAS WorldWide Telescope (includes cartoon Milky Way by Robert Hurt) The Radcliffe Wave ACTUALLY 2 IMPORTANT DEVELOPMENTS DISTANCES!! RADWAVE We can now Surprising wave- measure distances like arrangement to gas clouds in our of star-forming gas own Milky Way is the “Local Arm” galaxy to ~5% of the Milky Way. accuracy. Zucker et al. 2019; 2020 Alves et al. 2020 “Why should I believe all this?” DISTANCES!! We can now requires special measure distances regions on the Sky to gas clouds in our (HII regions own Milky Way with masers) galaxy to ~5% accuracy. can be used anywhere there’s dust & measurable stellar properties Zucker et al.
    [Show full text]
  • Getting Started with Microsoft Azure Virtual Machines
    Getting Started with Microsoft Azure Virtual Machines Introduction You can use a Microsoft Azure Virtual Machine when you need a scalable, cloud-based server running a Windows or Linux operating system and any application of your choosing. By taking advantage of Microsoft Azure Infrastructure as a Service (IaaS), you can run a virtual machine on an ongoing basis, or you can stop and restart it later with no loss to your data or server settings. You can quickly provision a new virtual machine from one of the images available from Microsoft Azure. On the other hand, if you are already running applications in a VMware or Hyper-V virtualized environment, you can easily migrate your virtual machine to Microsoft Azure. Once you have created your virtual machine and added it to Microsoft Azure, you can work with it much like an on-premises server by attaching more disks for data storage or by installing and running applications on it. Virtual machines rely on Microsoft Azure Storage for high availability. When your virtual machine is provisioned, it is replicated to three separate locations within the data center to which you assign it. You have the option to enable geo-replication to have copies of your virtual machine available in a remote data center region. Considering Scenarios for a Virtual Machine Microsoft Azure gives you not only the flexibility to support many application platforms, but also the flexibility to scale up and scale down to suit your requirements. Furthermore, you can quickly provision a new virtual machine in a few minutes. A Microsoft Azure virtual machine is simply a fresh machine preloaded with an operating system of your choice—you can add any needed application easily.
    [Show full text]