Always On. 2018-10
Total Page:16
File Type:pdf, Size:1020Kb
From Overnight to Always On Enno Runne Reactive Summit 2018-10-24 Enno Runne • Alpakka Tech Lead @ Lightbend • Stockholm, Sweden File content as streams File-based integration Read Write from file to file Disk Disk Copying a file in traditional Java InputStream is = new BufferedInputStream( new FileInputStream(sourceFilename)); OutputStream os = new BufferedOutputStream( new FileOutputStream(targetFilename)); byte[] buffer = new byte[1024]; int length; while ((length = is.read(buffer)) > 0) { os.write(buffer, 0, length); } is.close(); os.close(); Let’s call it source and sink Source<Byte…> Sink<Byte…> Read Write from file to file Disk Disk File source and sink with Akka Streams Source<ByteString, …> Sink<ByteString, …> FileIO.fromPath(sourceFile) FileIO.toPath(targetFile) Read Write from file to file Disk Disk Connect source and sink Source<ByteString,…> CompletionStage<IOResult fileSource = FileIO.fromPath(sourceFile); Sink<ByteString,…>CompletionStage<IOResult>>fileSink = FileIO.toPath(targetFile); fileSource .to(fileSink) .run(materializer); Read Write from file to file Materialized values Source<ByteString, CompletionStage<IOResult>> fileSource = FileIO.fromPath(sourceFile); Sink<ByteString, CompletionStage<IOResult>> fileSink = FileIO.toPath(targetFile); CompletionStage<IOResult> handle = fileSource .to(fileSink) .run(materializer); Materialized values Source<ByteString, CompletionStage<IOResult>> fileSource = FileIO.fromPath(sourceFile); Sink<ByteString, CompletionStage<IOResult>> fileSink = FileIO.toPath(targetFile); CompletionStage<IOResult> handle = fileSource .runWith( fileSink, materializer ); Copying a file with Akka Streams CompletionStage<IOResult> handle = FileIO.fromPath(sourceFile) .runWith( FileIO.toPath(targetFile), materializer ); Read Write from file to file Copying a file with Akka Streams CompletionStage<IOResult> handle = FileIO.fromPath(sourceFile) .runWith( FileIO.toPath(targetFile), materializer ); Read Write from file to file Data flow Step 1 Step 2 Step 3 Step 4 Buffering data • May cure the immediate issue • All buffering is limited Streaming with back-pressure Source Flow Flow Sink Downstream Dynamic push/pull • Push when downstream is faster • Pull when upstream is faster Akka Streams Parts of a Stream T Source S Sink A B Flow Stream operators A B map A B via Flow A B mapAsync Overnight Batching Make our file-copy more useful Detect Read Write new file from file to file Disk Disk Detect a new file in the directory Source<Path, NotUsed> newFileDetector = DirectoryChangesSource.create( sourceDir, Detect pollingInterval, new file maxChangesKept ) // Pair<Path, DirectoryChange> .filter(pathChange -> DirectoryChange.Creation == pathChange.second() ) .map(Pair::first); We built our own source! Combine the source with a stream Detect Read Write new file from file to file Disk Disk Combine the source with a stream with a sink Detect Read Write new file from file to file Disk Disk Combine the source with a stream with a sink Nesting a stream execution within a stream newFileDetector .mapAsync(8, p -> { Path targetFile = targetDir.resolve(p.getFileName()); return createFileToFile(p, targetFile); }) .runWith(Sink.ignore(), materializer); Outer and inner flows Detect Read Write new file from file to file Disk Disk Reactive Streams reactive-streams.org Reactive Streams A standard for asynchronous stream processing with non- blocking back-pressure. Part of JDK 9 (java.util.concurrent.Flow) http://www.reactive-streams.org/ Compliant libraries allow full interoperability Reactive Integrations Reactive Integrations Cross-system back-pressure support is the key thing Reactive Integrations bring to the table. Overcoming file-based Overcoming file-based integration Read from file ? Disk From bytes to… Overcoming file-based integration Read bytes Parse messages from file bytes Disk Overcoming file-based integration Read bytes Parse messages from file CSV Parse JSON Disk Parse XML Parse as CSV with Alpakka ByteString Collection<ByteString> Map<String, String> Parse Convert CSV CSV lines lines to maps byteStringSource .via(CsvParsing.lineScanner()) .via(CsvToMap.toMapAsStrings(StandardCharsets.UTF_8)); Build your own flow Flow<ByteString, Map<String, String>, NotUsed> csvBytesToMap = Flow.of(ByteString.class) .via(CsvParsing.lineScanner()) .via(CsvToMap.toMapAsStrings( StandardCharsets.UTF_8 )); Use your own flow and apply data mapping Flow<ByteString, Map<String, String>, NotUsed> csvBytesToMap = Flow.of(ByteString.class) .via(CsvParsing.lineScanner()) .via(CsvToMap.toMapAsStrings(StandardCharsets.UTF_8)); JsonNodeFactory jsonNodeFactory = JsonNodeFactory.instance; FileIO.fromPath(p) .via(csvBytesToMap) .map(data -> { // Using raw Jackson to create JSON objects ObjectNode objectNode = jsonNodeFactory.objectNode(); data.forEach(objectNode::put); return objectNode; }) Overcoming file-based integration bytes Map JSON Read Parse Map to from file CSV JSON Disk What50,000 is a / s stream? 10 / s 1 / month Photo by blucolt - originally posted to Flickr as spearfish creek, CC BY-SA 2.0 Reactive File Integration Turn Detect Read Other into new file from file technology messages Disk Destination Alpakka Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka. The short version: “Endpoints for Akka Streams” My view on Apache Camel vs Alpakka Apache Camel Alpakka • Data is wrapped in Exchange instance, • Typed message interchange, type can only be inspected compiler tracks compatibility • No back-pressure awareness, can • Back-pressure as specified by connect to Reactive Streams compliant Reactive Streams systems • No OSGi support • OSGi support • Relies on Akka Streams; evolving • Full-featured framework to express rapidly, but many integration integrations requirements are not covered, yet • Comprehensive docs and books • Moving fast, docs split between Akka and Alpakka sites Alpakka connectors for cloud services Amazon DynamoDB Google Cloud Pub/ Azure Storage Queue Amazon Kinesis data Sub streams & firehose Google Firebase AWS Lambda Cloud Messaging Amazon S3 Amazon SNS Amazon SQS Alpakka connectors for data stores Elasticsearch Alpakka connectors for messaging JMS Java Messaging Service (Eclipse Paho) AMQP (RabbitMQ) IronMQ Apache Kafka … not as fancy logos, but very well suited for the streaming approach. Community connectors to Akka Streams Apache Camel Couchbase Eventuate FS2 Pulsar … if you know of more, please tell us. The road to Alpakka 1.0 Preparations for Alpakka 1.0 include • structure all modules the same • improve chances to stay binary compatible • ensure good test coverage Help with this is highly appreciated… Alpakka community Code, Issues, Pull Requests @ Github • https://github.com/akka/alpakka • https://github.com/akka/alpakka-kafka Questions, Discussions • https://discuss.lightbend.com/c/akka https://www.lightbend.com/alpakka @akkateam Thank you! Join the Alpakka community @ennru at github.com/akka/alpakka [email protected].