From Overnight to Always On
Enno Runne Reactive Summit 2018-10-24 Enno Runne
• Alpakka Tech Lead @ Lightbend • Stockholm, Sweden File content as streams File-based integration
Read Write from file to file
Disk Disk Copying a file in traditional Java
InputStream is = new BufferedInputStream( new FileInputStream(sourceFilename)); OutputStream os = new BufferedOutputStream( new FileOutputStream(targetFilename));
byte[] buffer = new byte[1024]; int length; while ((length = is.read(buffer)) > 0) { os.write(buffer, 0, length); }
is.close(); os.close(); Let’s call it source and sink
Source
Read Write from file to file
Disk Disk File source and sink with Akka Streams
Source
Read Write from file to file
Disk Disk Connect source and sink
Source
Read Write from file to file Materialized values
Source
CompletionStage
Source
CompletionStage
CompletionStage
Read Write from file to file Copying a file with Akka Streams
CompletionStage
Read Write from file to file Data flow
Step 1 Step 2 Step 3 Step 4
Buffering data • May cure the immediate issue • All buffering is limited Streaming with back-pressure
Source Flow Flow Sink
Downstream
Dynamic push/pull • Push when downstream is faster • Pull when upstream is faster Akka Streams Parts of a Stream T
Source
S
Sink
A B
Flow Stream operators
A B
map
A B via Flow A B mapAsync Overnight Batching Make our file-copy more useful
Detect Read Write new file from file to file
Disk Disk Detect a new file in the directory Source
Detect Read Write new file from file to file
Disk Disk Combine the source with a stream with a sink
Detect Read Write new file from file to file
Disk Disk Combine the source with a stream with a sink
Nesting a stream execution within a stream newFileDetector .mapAsync(8, p -> {
Path targetFile = targetDir.resolve(p.getFileName()); return createFileToFile(p, targetFile);
}) .runWith(Sink.ignore(), materializer); Outer and inner flows
Detect Read Write new file from file to file
Disk Disk Reactive Streams reactive-streams.org Reactive Streams
A standard for asynchronous stream processing with non- blocking back-pressure.
Part of JDK 9 (java.util.concurrent.Flow) http://www.reactive-streams.org/
Compliant libraries allow full interoperability Reactive Integrations Reactive Integrations
Cross-system back-pressure support is the key thing Reactive Integrations bring to the table. Overcoming file-based Overcoming file-based integration
Read from file ?
Disk From bytes to… Overcoming file-based integration
Read bytes Parse messages from file bytes
Disk Overcoming file-based integration
Read bytes Parse messages from file CSV
Parse JSON
Disk Parse XML Parse as CSV with Alpakka
ByteString Collection
Parse Convert CSV CSV lines lines to maps
byteStringSource .via(CsvParsing.lineScanner()) .via(CsvToMap.toMapAsStrings(StandardCharsets.UTF_8)); Build your own flow
Flow
Flow.of(ByteString.class)
.via(CsvParsing.lineScanner())
.via(CsvToMap.toMapAsStrings( StandardCharsets.UTF_8 )); Use your own flow and apply data mapping
Flow
JsonNodeFactory jsonNodeFactory = JsonNodeFactory.instance; FileIO.fromPath(p) .via(csvBytesToMap) .map(data -> { // Using raw Jackson to create JSON objects ObjectNode objectNode = jsonNodeFactory.objectNode(); data.forEach(objectNode::put); return objectNode; }) Overcoming file-based integration
bytes Map JSON Read Parse Map to from file CSV JSON
Disk What50,000 is a / s stream? 10 / s 1 / month Photo by blucolt - originally posted to Flickr as spearfish creek, CC BY-SA 2.0 Reactive File Integration
Turn Detect Read Other into new file from file technology messages
Disk Destination
Alpakka Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka.
The short version: “Endpoints for Akka Streams” My view on Apache Camel vs Alpakka
Apache Camel Alpakka • Data is wrapped in Exchange instance, • Typed message interchange, type can only be inspected compiler tracks compatibility • No back-pressure awareness, can • Back-pressure as specified by connect to Reactive Streams compliant Reactive Streams systems • No OSGi support • OSGi support • Relies on Akka Streams; evolving • Full-featured framework to express rapidly, but many integration integrations requirements are not covered, yet • Comprehensive docs and books • Moving fast, docs split between Akka and Alpakka sites Alpakka connectors for cloud services
Amazon DynamoDB Google Cloud Pub/ Azure Storage Queue Amazon Kinesis data Sub streams & firehose Google Firebase AWS Lambda Cloud Messaging Amazon S3 Amazon SNS Amazon SQS Alpakka connectors for data stores
Elasticsearch Alpakka connectors for messaging
JMS Java Messaging Service (Eclipse Paho) AMQP (RabbitMQ)
IronMQ Apache Kafka
… not as fancy logos, but very well suited for the streaming approach. Community connectors to Akka Streams
Apache Camel Couchbase Eventuate FS2 Pulsar
… if you know of more, please tell us.
The road to Alpakka 1.0 Preparations for Alpakka 1.0 include • structure all modules the same • improve chances to stay binary compatible • ensure good test coverage
Help with this is highly appreciated… Alpakka community
Code, Issues, Pull Requests @ Github • https://github.com/akka/alpakka • https://github.com/akka/alpakka-kafka
Questions, Discussions • https://discuss.lightbend.com/c/akka https://www.lightbend.com/alpakka @akkateam Thank you!
Join the Alpakka community @ennru at github.com/akka/alpakka [email protected]