DOCSLIB.ORG

Talmatic CV Vadim H. Data Scientist 1213

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Talmatic CV Vadim H. Data Scientist 1213 Vadym H. Team Lead/Scala Developer/Data Scientist Have over 4.5 years of experience as a Data Scientist and a Scala Developer. MAJOR SKILLS Field Level Experience Scala Expert 4.5 years Play Framework Expert 4 years Sbt Expert 4 years Slick Expert 4 years Akka Expert 4 years Kafka Expert 4 years OpenCV Expert 4 years Lucene Advanced 2 years Hadoop Expert 3 years Spark Expert 3 years Flink Advanced 2 years Docker Expert 3 years Kubernetes Expert 3 years Java Expert 4 years J2EE/JSP/Servlets Advanced 2 years Spring (Core, MVC) Advanced 2 years Hibernate Advanced 2 years Machine Learning Expert 4.5 years JavaScript/jQuery Expert 4 years HTML5/CSS3 Expert 4 years NoSQL (HBase, Cassandra) Expert 4 years Databases/SQL Expert 4 years Linux Expert 4 years Python Intermediate 2 years TensorFlow Expert 3 years Keras Expert 3 years Kubeflow Intermediate 1 year PyTorch Advanced 2 years PROFESSIONAL EXPERIENCE Company Soft Development Company, Ukraine Role Team Lead/Scala Developer/Data Scientist Date January 2017 – Present Responsibilities Development of reactive and scalable applications. Analysis of large amounts of data and designing reliable, fast, accurate AI for solving problems of prediction, estimation and classification. Leading a team of software engineers and data scientists MAJOR PROJECTS Project #1 Image Measuring App Platform Anaconda Databases MongoDB Technologies Python 3, Vue.js, OpenCV Third Parties Darknet Tools Flask, Git Role Data Scientist, Senior Software Developer Team size 3 Description An application for an insurance company that detects small damaged areas in houses, considers the area of damage and also conducts checks to avoid insurance fraud. Responsibilities ● Responsible for AI part and back-end server developing ● Collecting data for training ● Selected right tools and frameworks for the object detector ● Developed object detector using Darknet and Python3 ● Developed server to communicate with MongoDB Project #2 IoT aggregator Platform Quarkus, Kubernetes Databases MongoDB, Readis and Hazelcast as In-Memory cache Technologies Quarkus (GraalVM, Vert.x), Kafka, Kubernetes, Helm, Istio, Kubeflow, Angular, Maven, Java, Python3, Strimzi Third Parties None Tools Git, IntelliJ IDEA, Swagger codegen Role Senior Software Engineer, Data Scientist Team size 2 Description IoT aggregator aiming to provide a single access to all available smart devices. It uses complex and scalable architecture based on Vert.x as a web framework and Kubernetes as a platform for microservices. This project combines multiple technologies and approaches to achieve this goal: Machine Learning, Reactive Applications, etc. Responsibilities ● Developed microservices using Vert.X ● Maintained Kubernetes resources ● Participated in development of ML based solutions ● Developed new features and fixed bugs in front-end ● Created and configured Kafka cluster ● Provided data analysis based on collected data Project #3 Security System Platform Anaconda Databases None Technologies Python 3, TensorFlow, Keras, NumPy, Docker, Kubernetes, Darknet, OpenCV, ONNX Third Parties Luxand Tools PyCharm, Git Role Data Scientist Team size 1 Description Security System is an AI driven application that allows programmers to use their computers without fear that someone can watch their code or make some photos of it. The application consists of two parts: a face detector based on Luxand and a phone detector based on Darknet YOLO and OpenCV Dnn implementation. Responsibilities ● Responsible for AI part ● Selected right tools and frameworks for the face detector and phone detector, helped to integrate and configure Luxand ● Collected and labeled dataset for the phone detector ● Developed the phone detector using Darknet YOLO and OpenCV Dnn Project #4 Face Recognition Platform Anaconda Databases PostgreSQL Technologies Python 3, TensorFlow, Keras, NumPy, Docker, Kubernetes, OpenCV Third Parties None Tools PyCharm, Git Role Data Scientist Team size 2 Description Face Recognition is a neural-network based application that enables a higher level of security in an office. It allows checking a person’s ID without any key just by processing his or her face image/live video stream. This application consists of several sub networks that make the model scale invariant and robust to new images. Responsibilities ● Responsible for a ML pipeline ● Participated in a ML model implementation (selection, implementation, tuning), dataset collection, dataset preprocessing ● Created the robust and fast siamese neural network based on a triplet loss Project #5 Kubkaf Platform Microsoft Azure Databases PostgreSQL Technologies Scala, Slick, Akka, Kafka, Swagger, REST, Play Framework, SBT, Azure Java API, Skuber, WebSocket, Docker, Kubernetes Third Parties None Tools IntelliJ IDEA, Git Role Scala Developer Team size 4 Description Kubkaf is a Kafka-as-a-Service platform. It allows deploying Kafka clusters without any deep configuration in Microsoft Azure Cloud. It supports many Kafka versions and cluster configurations (number of nodes, etc.). Also, it utilizes all advantages of Docker and Kubernetes to provide a fault-tolerant, scalable, and resilient system. Responsibilities ● Сreating of a Kafka-as-a-Service module ● Microsoft Azure management ● Dockerfile creation ● Kubernetes management ● Kafka management Project #6 Recommendation System for a book shop Platform SBT, Docker Databases Cassandra Technologies Scala, Play Framework, Java, Phantom SBT, Spark, SparkML Third Parties None Tools IntelliJ IDEA, Git Role Scala Developer Team size 2 Description A system that’s used for predicting user’s rating and opinions about books. The system makes predictions based on the opinion of every user. Besides, it calculates user’s and book’s feature vectors alone, so developer/ administrator can control it. Responsibilities ● Responsible for a Recommendation System micro-service, writing tests and bug fixing ● Developed Recommendation Engine based on Spark Structured API (ALSModel), Streaming API (implemented the BSGD algorithm for it) and Akka ● Created a lot of Actors for Spark model synchronization and updating without any downtime ● Development of REST controllers based on a Play Framework later connected to Akka through a level of service classes Project #7 GDPR Compliance Platform SBT, Docker Databases CouchDB Technologies Java, Scala, Lucene, Hadoop, Spark Third Parties None Tools IntelliJ IDEA, Git Role Data Scientist Team size 2 Description This program provides the possibility to check how different Data Processors implement GDPR. It splits document into parts using key-words, saves them in the DB. Another part analyses parts and predicts the quality of implementation. Responsibilities ● Responsible for creation of ML module, development of unit tests and bug fixing ● Data engineering (visualization, preprocessing), feature engineering (extraction, generation) and model creation (choosing, validation, testing) ● Responsible for Spark Estimators and Transformers ● Participated in the development of a search engine used for data extraction, in particular, developed highlighter based on Lucene based highlighters ● All tests are done using the spec2 library Project #8 Travel Assistant Platform Play Framework, Sbt Databases PostgreSQL Technologies Scala, Flyway, Doobie, Akka, Cats, Swagger, REST Third Parties None Tools IntelliJ IDEA, Git Role Scala Developer Team size 5 Description A platform for planning and organizing international traveling. Includes an advanced system for booking plane tickets and a variety of ground and water transport, building travel routes, an advanced search system, customer care, email distribution and others. Responsibilities ● Responsible for the creation of test-database and manipulations with it. It was used Cats, Doobie and H2 to complete this task ● Worked with the services of the project, optimized a little bit of validation part of this service (constraints). Some tasks were done based on Akka for concurrent processing of user data and requests covered the project with a lot of tests Project #9 Company Team Chat Platform Maven Databases MySQL Technologies Java, Spring, Spring MVC, Hibernate, jQuery, JavaScript, Webpack Third Parties None Tools IntelliJ IDEA, Git Role Java Developer Team size 5 Description This portal provides an ability to exchange messages, files and store message history. A user can pick favorite messages, create rooms, set notifications, create public and private chat rooms, search messages in the archive by date or by some text pattern, etc. Responsibilities ● Responsible for the creation of some parts of a DAO system of this application, also participated in the development of Spring MVC controllers and service/model layer ● Managed configuration files for a Spring IoC. It was removed all repeatable code from the application using Spring AOP EDUCATION Date Degree HEI 2016 – 2020 Bachelor's degree V. N. Karazin Kharkiv National University LANGUAGES Language Level English Upper-Intermediate Russian Native Ukrainian Native.
Load more

Recommended publications
  • Open Source Used in Cisco DNA Center Platform Release 1.2.X
    Open Source Used In Cisco DNA Center Platform 1.2.x Cisco Systems, Inc. www.cisco.com Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the Cisco website at www.cisco.com/go/offices. Text Part Number: 78EE117C99-178119203 Open Source Used In Cisco DNA Center Platform 1.2.x 1 This document contains licenses and notices for open source software used in this product. With respect to the free/open source software listed in this document, if you have any questions or wish to receive a copy of any source code to which you may be entitled under the applicable free/open source license(s) (such as the GNU Lesser/General Public License), please contact us at [email protected]. In your requests please include the following reference number 78EE117C99-178119203 Contents 1.1 ajv 5.5.2 1.1.1 Available under license 1.2 ajv-keywords 3.1.0 1.2.1 Available under license 1.3 akkahttp 10.0.9 1.3.1 Available under license 1.4 akkahttpcore 10.0.9 1.5 akkahttpjackson 10.0.9 1.5.1 Available under license 1.6 akkahttptestkit 10.0.9 1.7 akkaslf4j 2.5.6 1.8 akkastream 2.5.6 1.9 api-spec-converter 2.6.0 1.9.1 Available under license 1.10 axios 0.16.2 1.10.1 Available under license 1.11 babel-cli 6.8.0 1.12 babel-cli 6.26.0 1.13 babel-core 6.26.0 1.14 babel-core 6.8.0 1.15 babel-eslint 8.2.2 1.15.1 Available under license 1.16 babel-jest 21.2.0 1.17 babel-jest 21.2.0 1.17.1 Available under license 1.18 babel-plugin-transform-async-to-generator 6.24.1 Open Source Used In Cisco DNA Center Platform
    [Show full text]
  • Getting Started with Sbt
    Getting Started with sbt Contents Preface ................................... 4 Installing sbt ................................ 4 Tips and Notes ............................ 5 Installing sbt on Mac ............................ 5 Installing from a third-party package ................ 5 Installing from a universal package ................. 5 Installing manually .......................... 5 Installing sbt on Windows ......................... 5 Windows installer ........................... 5 Installing from a universal package ................. 5 Installing manually .......................... 6 Installing sbt on Linux ........................... 6 Installing from a universal package ................. 6 RPM and DEB ............................ 6 Gentoo ................................. 6 Installing manually .......................... 6 Installing sbt manually ........................... 6 Unix .................................. 7 Windows ............................... 7 Hello, World ................................ 8 Create a project directory with source code ............ 8 Build definition ............................ 9 1 Setting the sbt version ........................ 10 Directory structure ............................. 10 Base directory ............................. 10 Source code .............................. 10 sbt build definition files ....................... 11 Build products ............................ 11 Configuring version control ..................... 11 Running ................................... 11 Interactive mode ..........................
    [Show full text]
  • Sbt-Native-Packager Release 1.0A1
    sbt-native-packager Release 1.0a1 Josh Suereth Sep 18, 2021 Contents 1 Introduction 1 1.1 Goals...................................................1 1.2 Scope...................................................1 1.3 Core Concepts..............................................2 2 Getting Started 5 2.1 Setup...................................................5 2.2 Your first package............................................5 3 Packaging Formats 7 3.1 Universal Plugin.............................................8 3.2 Linux Plugin............................................... 15 3.3 Debian Plugin.............................................. 21 3.4 Rpm Plugin................................................ 25 3.5 Docker Plugin.............................................. 31 3.6 Windows Plugin............................................. 38 3.7 JDKPackager Plugin........................................... 41 3.8 GraalVM Native Image Plugin...................................... 44 4 Project Archetypes 47 4.1 Java Application Archetype....................................... 47 4.2 Java Server Application Archetype................................... 54 4.3 Systemloaders.............................................. 59 4.4 Configuration Archetypes........................................ 62 4.5 Jlink Plugin................................................ 62 4.6 Archetype Cheatsheet.......................................... 64 5 Recipes 69 5.1 Custom Package Formats........................................ 69 5.2 Dealing with long classpaths......................................
    [Show full text]
  • Lift Documentation
    Lift Documentation The Lift Team Sep 11, 2019 Table of Contents: 1 Getting Started with Lift 3 1.1 Download.................................................3 1.2 Installation................................................4 1.3 Building Lift...............................................6 2 Overview of Lift 15 2.1 Patterns.................................................. 15 2.2 Rewrite Rules.............................................. 17 2.3 Compilation Flow............................................ 17 2.4 Inferring OpenCL Thread Counts.................................... 18 3 Developing Lift 19 3.1 Testing.................................................. 19 3.2 Guidelines for Contributing....................................... 20 3.3 How to . ................................................. 22 4 Generating and Running Kernels 27 4.1 Generating Kernels............................................ 27 4.2 Running Kernels............................................. 27 4.3 High-Level Rewrite........................................... 28 5 Indices and tables 31 i ii Lift Documentation This is the documentation of the Lift programming language and compiler. The lift project is a research project bya team at the University of Edinburgh together with further collaborators. Table of Contents: 1 Lift Documentation 2 Table of Contents: CHAPTER 1 Getting Started with Lift 1.1 Download To download Lift to your local machine perform the following steps: 1. Ensure that git and git-lfs are installed on your machine Lift is distributed via https://github.com
    [Show full text]
  • Peter Robinett [email protected] for DUSE VI, 2010-06-30
    "' L-* Fr#1'w3r/ f3r F72 #2& Pr3)6 Peter Robinett [email protected] for DUSE VI, 2010-06-30 W,3 #1 I? ● Background in web programming with interpreted languages (PHP, Python, Javascript, etc) ● Likes long walks on the beaches ● Lift + Scala programmer for one year ● Loves both cats AND dogs ● Lift committer for approx. 6 months BUT only has minor commit to lift-flot to my name ● Likes fine wine and smooth jazz ● BUT active on mailing list and wiki ● Isn't very good at making funny bullet points W,#6 . L-*? Lift is an expressive and elegant framework for writing web applications. Lift stresses the importance of security, maintainability, scalability and performance, while allowing for high levels of developer productivity. Lift is inspired by Seaside, Rails, Django, Wicket, and beyond. W,9 F72? class AskName extends CometActor { def render = ajaxForm(<div>What is your username?</div> ++ text("",name => answer(name.trim)) ++ <input type="submit" value="Enter"/>) } class Chat extends CometActor with CometListener { private var userName = "" private var chats: List[ChatLine] = Nil private lazy val infoId = uniqueId + "_info" C31'6 private lazy val infoIn = uniqueId + "_in" private lazy val inputArea = findKids(defaultXml, "chat", "input") private lazy val bodyArea = findKids(defaultXml, "chat", "body") private lazy val singleLine = deepFindKids(bodyArea, "chat", "list") // handle an update to the chat lists // by diffing the lists and then sending a partial update // to the browser override def lowPriority = { case ChatServerUpdate(value)
    [Show full text]
  • Full-Graph-Limited-Mvn-Deps.Pdf
    org.jboss.cl.jboss-cl-2.0.9.GA org.jboss.cl.jboss-cl-parent-2.2.1.GA org.jboss.cl.jboss-classloader-N/A org.jboss.cl.jboss-classloading-vfs-N/A org.jboss.cl.jboss-classloading-N/A org.primefaces.extensions.master-pom-1.0.0 org.sonatype.mercury.mercury-mp3-1.0-alpha-1 org.primefaces.themes.overcast-${primefaces.theme.version} org.primefaces.themes.dark-hive-${primefaces.theme.version}org.primefaces.themes.humanity-${primefaces.theme.version}org.primefaces.themes.le-frog-${primefaces.theme.version} org.primefaces.themes.south-street-${primefaces.theme.version}org.primefaces.themes.sunny-${primefaces.theme.version}org.primefaces.themes.hot-sneaks-${primefaces.theme.version}org.primefaces.themes.cupertino-${primefaces.theme.version} org.primefaces.themes.trontastic-${primefaces.theme.version}org.primefaces.themes.excite-bike-${primefaces.theme.version} org.apache.maven.mercury.mercury-external-N/A org.primefaces.themes.redmond-${primefaces.theme.version}org.primefaces.themes.afterwork-${primefaces.theme.version}org.primefaces.themes.glass-x-${primefaces.theme.version}org.primefaces.themes.home-${primefaces.theme.version} org.primefaces.themes.black-tie-${primefaces.theme.version}org.primefaces.themes.eggplant-${primefaces.theme.version} org.apache.maven.mercury.mercury-repo-remote-m2-N/Aorg.apache.maven.mercury.mercury-md-sat-N/A org.primefaces.themes.ui-lightness-${primefaces.theme.version}org.primefaces.themes.midnight-${primefaces.theme.version}org.primefaces.themes.mint-choc-${primefaces.theme.version}org.primefaces.themes.afternoon-${primefaces.theme.version}org.primefaces.themes.dot-luv-${primefaces.theme.version}org.primefaces.themes.smoothness-${primefaces.theme.version}org.primefaces.themes.swanky-purse-${primefaces.theme.version}
    [Show full text]
  • Introduction to Scala and Spark
    Introduction to Scala and Spark SATURN 2016 Bradley (Brad) S. Rubin, PhD Director, Center of Excellence for Big Data Graduate Programs in Software University of St. Thomas, St. Paul, MN [email protected] 1 Scala Spark Scala/Spark Examples Classroom Experience 2 What is Scala? • JVM-based language that can call, and be called, by Java New: Scala.js (Scala to JavaScript compiler) Dead: Scala.Net • A more concise, richer, Java + functional programming • Blends the object-oriented and functional paradigms • Strongly statically typed, yet feels dynamically typed • Stands for SCAlable LAnguage Little scripts to big projects, multiple programming paradigms, start small and grow knowledge as needed, multi-core, big data • Developed by Martin Odersky at EPFL (Switzerland) Worked on Java Generics and wrote javac • Released in 2004 3 Scala and Java javac Java Scala scalac JVM 4 Scala Adoption (TIOBE) Scala is 31st on the list 5 Freshman Computer Science 6 Job Demand Functional Languages 7 Scala Sampler Syntax and Features • Encourages the use of immutable state • No semicolons unless multiple statements per line • No need to specify types in all cases types follow variable and parameter names after a colon • Almost everything is an expression that returns a value of a type • Discourages using the keyword return • Traits, which are more powerful Interfaces • Case classes auto-generate a lot of boilerplate code • Leverages powerful pattern matching 8 Scala Sampler Syntax and Features • Discourages null by emphasizing the Option pattern • Unit, like Java void • Extremely powerful (and complicated) type system • Implicitly converts types, and lets you extend closed classes • No checked exceptions • Default, named, and variable parameters • Mandatory override declarations • A pure OO language all values are objects, all operations are methods 9 Language Opinions There are only two kinds of languages: the ones people complain about and the ones nobody uses.
    [Show full text]
  • POM Model Version 5.0.0
    POM Model Version 5.0.0 Status DRAFT Version Issue(s) Sources Developer(s) Stephen Connolly Status This RFC is currently in the DRAFT state. Nothing in this RFC has been agreed or confirmed. Contents Status Contents Introduction Background Classification of change requests New content to include in the POM Supports / provides style concepts Versioning related issues Lifecycle related changes Scope related changes Profile activation POM format Mix-ins Existing model Changes Dual usage XML vs custom DSL Elements vs Attributes Customizing build behavior / One-off projects Custom scopes Build vs Reporting Project Object Model <project> element <parent> element <mixin> element <extensions> element <extension> element TODO resolve the inheritance problem Appendix 1 - Issues flagged for consideration post-modelVersion 4.0.0 Introduction The next generation Project Object Model to be used by Maven 5.0+ Background Maven uses the Project Object Model as a descriptor for the declarative build requirements of a project. Maven 1.x used a model which contained a <modelVersion>3.0.0</modelVersion> element as an immediate child of the root. Maven 2.x / 3.x has used a <modelVersion>4.0.0</modelVersion> element. Due to the way Maven has been implemented, the current release versions will consider any modelVersion other than the one that they target as invalid and will fail to parse the model. For build time concerns, this is not that major a concern, and in fact may be desirable behaviour, e.g. I should not be able to build a Maven 2.x / 3.x project with Maven 1.x. Where the modelVersion becomes a constraint, however, is when it comes to transitive dependency resolution.
    [Show full text]
  • Why a Build Tool?
    Why a build tool? By Joshua Suereth and Matthew Farwell, SBT in Action Most everyone will agree that we need build automation. You can perfectly well automate your build with a shell script (or batch script). But why have a specific tool for this purpose? In this article, we talk about why by providing a couple of real-life examples. If you've spent any time working in a team of software developers, you'll have noticed one thing. People make mistakes. Have you ever heard of the old joke about asking 10 economists a question and getting 11 different answers? It's like that when you've got a list of things to do: if you have 10 things to do, you'll make 11 mistakes doing them. Another thing you'll notice is that people are different. Bob isn't the same person as Ted. When we look at Bob’s workstation, everything is organized and tidy. Everything is where it should be. No icons on his desktop. Ted, however, can't actually see his screen background for the icons. Don't know why he bothers having one in the first place. One thing is absolutely certain. If you ask Bob to do a build of the product, you'll get a different product than if you were to ask Ted. But you can’t tell which would have the fewer bugs though. A few years ago, one of our project teams had an “automated” build. It was 10 or 12 windows batch files, which built and deployed a web application; it compiled the java, built the web application and deployed it to server.
    [Show full text]
  • Customizing RISC-V Core Using Open Source Tools
    Customizing RISC-V core using open source tools Alex Badicioiu, NXP Agenda • Chisel HDL • Introduction • Development environment components and setup • Hello World project • Extending RISC-V core with new instruction • riscv-sodor designs • New instruction implementation, integration & simulation Chisel HDL • Introduction • Scala extension to generate hardware description • Chisel/Scala statements are mixed - when Scala program runs hardware description output is generated as Chisel statements are executed • This hardware description output is called FIRRTL – Flexible Intermediate Form RTL • FIRRTL is converted in low level Verilog by a FIRRTL compiler • This approach provides powerful design generators and comprehensive testing possibilities Chisel HDL • Local development environment setup example • Ubuntu 14.0.4 • Build essential • apt-get install build-essential autoconf flex bison • Java(OpenJDK1.7) • apt-get install openjdk-7-jdk • SBT/Scala - https://piccolo.link/sbt-0.13.16.tgz, Scala version 2.11.11 • Verilator (3.886) • git clone http://git.veripool.org/git/verilator • git checkout verilator_3.886 • autoconf && ./configure && make • FIRRTL – firrtl, firrtl-interpreter • https://github.com/freechipsproject/firrtl.git -3dd921f38f298c7c4aa338e14ac43bc77c652e8c • https://github.com/freechipsproject/firrtl-interpreter.git - cea56365cf1dce8dd13fa1379a4f5ed35347ee0b • Chisel3 • https://github.com/freechipsproject/chisel3.git - 8168a8eea6c3465966081c5acd0347e09791361c • chisel-testers • https://github.com/freechipsproject/chisel-testers.git
    [Show full text]
  • {DOWNLOAD} the Definitive Guide to Lift : a Scala-Based Web
    THE DEFINITIVE GUIDE TO LIFT : A SCALA-BASED WEB FRAMEWORK PDF, EPUB, EBOOK Marius Danciu | 200 pages | 30 Jul 2009 | Springer-Verlag Berlin and Heidelberg GmbH & Co. KG | 9781430224211 | English | Berlin, Germany The Definitive Guide to Lift : A Scala-based Web Framework PDF Book While many are concerned with building Lift or are build program archetypes, there are two you will probably encounter fairly frequently as a Lift user:. Studying others is a natural way to learn; however, it's as easy to learn bad habits through imitation as it is to acquire good ones. The better way to become HTML-fluent is through a comprehensive reference that covers the language syntax, semantics, and variations in detail and demonstrates the difference between good and bad usage. Learn how to install and use Express. JavaScript is currently disabled, this site works much better if you enable JavaScript in your browser. If you're a decent developer, the app should already be build in your mind. Lift makes it fun to develop because it lets you focus on the interesting parts of coding. The Definitive Guide to AdonisJs also covers how to create a front-end build chain, so that you can use a modern front-end framework, such as React. An archived version of blog. Everyone I know who uses spring complains about the difficulty in getting the configs and dependencies right, maven nightmares, etc. JavaScript is very fast, simple and integrate easily with other languages. He has been writing articles for Nordic APIs since Per our contributing guidelines , Issues on the Lift GitHub project are intended to describe actionable, already-discussed items.
    [Show full text]
  • Running Spark on EC2 Documentation
    Running Knn Spark on EC2 Documentation Pseudo code Preparing to use Amazon AWS First, open a Spark launcher instance. Open a m3.medium account with all default settings. Step 1: Login to the AWS console. Under the account name at the top right, select security credentials. Step 2: Click on the Access Keys tab and get or create the AWS access key id and AWS secret access key, then save them someplace for later. Step 3: Under the services tab in the top left, select EC2. Once on EC2 dashboard, go to left side and click on the Network and Security tab and select Key Pairs. Step 4: Create a new key pair, and save the key pair name as well as the .pem private key file. Installing Spark Step 1: Install Java Install java with the following set of commands sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt-get install oracle-jdk7-installer You can check whether java is installed correctly by using the command java –version If installed correctly, you should be seeing the following Step 2: Download and unzip Spark Using the following commands, download a Spark version and unzip it wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2.tgz tar –xvzf spark-1.0.2.tgz It will create a Spark directory. Step 3: Install Scala Use the following commands to download Scala, install and set the path wget http://www.scala-lang.org/files/archive/scala-2.10.4.tgz sudo mkdir /usr/local/src/scala sudo tar xvf scala-2.10.4.tgz -C /usr/local/src/scala/ Open bashrc file vi .bashrc Add the following lines at the end of the file export SCALA_HOME=/usr/local/src/scala/scala-2.10.4 export PATH=$SCALA_HOME/bin:$PATH Restart bashrc by the command .
    [Show full text]