Resource-Oriented Computing w/ NetKernel :
Software for the 21st Century
Brian Sletten Zepheira, LLC [email protected] Speaker Qualifications
Over 14 years of software development experience Has own software consulting company for design, mentoring, training and development Currently working in Semantic Web, AOP, Grid Computing, P2P and security consulting Pinky Committer Agenda
What the heck is NetKernel? Why We Care NetKernel Overview NetKernel Languages Advanced Concepts Pinky What the heck is NetKernel?
A micro-kernel based resource ecosystem built around the principles of REST, Unix Pipes and SOA Why We Care: The Simple Answer
Most enterprise systems use XML and are built in Java and this is still too much of a pain Complicated Different Languages for different technologies Predicting scalability Why We Care : The Complicated Answer
This is hard because of a general mismatch at the intersection of languages, data model, processing abstractions, architectural tiers, etc. Larger Trends
Data formatted
Code written
1970s 1980s 1990s 2000s NetKernel
A Ground-Up Resource-oriented computing environment Modern, microkernel architecture Takes the best of REST, Unix Pipes, SOA Open Source w/ Dual License for non-open source uses Resource-Oriented Computing
Everything is a Resource Everything is URI-Addressable (logical reference) “Computation” is the turning of logical resource references into physical representation Lossless conversion between representations REST
Representational State Transfer Architectural style described in Roy Fielding’s Ph.D. thesis URLs locate resources in Internet-sized address spaces Client passes all state in request History of Addressing
------
Documents Network
http://myserver.com/docs/reports/3rdQtr/ History of Addressing
Network
http://myserver.com/data/graph/1
Data History of Addressing
http://myserver.com/service/a
Network
Service A
Services URLs ! => Scalability
Addressability of the resources is important, but REST derives its properties from the architectural constraints REST Properties Unix Influences
Everything is a file (common interface/ resource abstraction) Small, focused, reusable tools Tools are scripted Processes have context from environment SOA Influences
Loosely-coupled Asynchronous requests Reusable services composed through orchestration languages Putting the Kernel in NetKernel
Internally, a microkernel manages threading resources Maps URI requests to managed resources and schedules them General Concept Calling a BeanShell Script XQuery XSLT Active URIs
Internally, NetKernel uses a special URI naming scheme A proposed IETF standard by HP and 1060 Research to encode complex URIs into standard forms General form active:
BeanShell active:[email protected][email protected]
XQuery active:[email protected][email protected]
XSLT active:xslt[email protected][email protected] NetKernel Benefits
“Deep” REST environment Appropriate abstractions Focus on information, not code! Layered approach to building applications Logical connections/requests get resolved at runtime (late-binding) NetKernel Properties
Fine-grained caching Self-optimizing based on use “Aspect-Oriented” architecture Security, logging, etc. behind URI interception Tiny Footprint Can do “real” work in a VM w/ 10-12MB Takes full advantage of multi-core, multi-CPU boxes Basic NetKernel Tour NetKernel Concepts (aka Really Cool Terms)
Accessors Transports Representations Aspects Transreptors Accessors
URI-addressable component that provides access to NetKernel resources and services Transport
A component that funnels requests from external sources into NetKernel URI requests (and ultimately Accessors) Examples HTTP SMTP JMS Representations
The ‘R’ in REST In NetKernel, an object representing the resource Exposed via immutable, cacheable “views” Aspects
Not AOP “Aspects” Immutable views of the underlying representation in a given form Examples JDOM, DOM, String, BinaryStream Transreptors
Fancy term for “transforming representations” Transreptors return a resource in a specified form if conversion is possible Useful for converting between Aspects NetKernel Languages DPML
A simple NetKernel scripting language for specifying URI requests Supports conditionals, loops, subroutines, etc. DPML Sample
XML Query Language W3C Candidate Recommendation Functional language FLWOR (For Let Where Order By Return) XQuery Sample
Supports embedded BSH interpreter Java source interpreter Interacts with other scripts, Java, etc. Makes for very fast scripting BeanShell Sample
void main() { // Extract all speeches with lines // containing the parameter argument req=context.createSubRequest(); req.setURI("active:xquery"); req.addArgument("operator","abbey.xq"); output=context.issueSubRequest(req);
//Finally return response response=context.createResponseFrom(output); context.setResponse(response); } “Real World” Examples XSLT Input Data XSLT Example
if( context.exists("this:param:param") ) { param=context.sourceAspect("this:param:param", IAspectNVP.class); mbid = param.getValue("mbid"); } req=context.createSubRequest(); req.setURI("active:httpGet"); url="http://musicbrainz.org/ws/1/artist/" +mbid+ "?type=xml&inc=sa-Album"; req.addArgument("url", url); output=context.issueSubRequest(req); sa= new StringAspect("
Results for
Simple Tree Manipulation Kind of a DSL for XML XML is tree-oriented, let’s use it that way stm:set
Sets element values to the specified value
Inserts specified value after the selected element
Applies the specified namespace to the selected elements
Asynchronous requests are easy in NetKernel
// Set up request 1 req1=context.createSubRequest(); req1.setURI("active:beanshell"); req1.addArgument("operator", “script1.bsh” ); req1.addArgument("param", param ); handle1 = context.issueAsyncSubRequest(req1);
// Set up request 2 req2=context.createSubRequest(); req2.setURI("active:beanshell"); req2.addArgument("operator", “script2.bsh” ); req2.addArgument("param", param ); handle2 = context.issueAsyncSubRequest(req2);
// Block until both calls finish retValue1 = handle1.join(); retValue2 = handle2.join(); Throttling Requests
Throttling requests is also easy // attempt to issue 100 concurrent requests to script1.bsh for (int i=0; i<100; i++) { req=context.createSubRequest("active:throttle"); req.addArgument("id","throttle:my_throttle"); req.addArgument("uri","active:beanshell"); req.addArgument(“operator”,”myscript.bsh”); context.issueAsyncSubRequest(req); } Expects to find a configuration file in ffcpl:/etc/ConfigThrottle.xml
Virtual Dependency Management Powerful technique for invalidating arbitrary collections of resources Easier to see than hear about! Scalability - Scenario 1
Long Computation wrapped in an Accessor called from a DPML script Scalability - Scenario 2
Shorter Computation called multiple times from DPML script Scalability - Scenario 3
Random document chosen from DPML script Modules
Real NetKernel Infrastructure is built around modules Modules (cont)
Ways of segmenting Address Space and functionality Modules export public URIs and class packages Separately Versioned units of deployment Modules (cont)
Can hold static resources, Java code, scripts, accessors, aspects, transreptors, transports, fragmentors and caches Import module definitions to access another namespace If you create a new module, you will need to at it to the deployedModules.xml Sample Modules
Module Functionality ext-xml-core-1.4.2.jar DOM, JDom, SAX, XDA, XMLBean ext-dpml-1.2.2.jar DPML Language runtime ext-kernel-1.0.6.jar System component accessors doc-netkernel-4.0.0.jar NetKernel Core Documentation mod-fulcrum-frontend HTTP Server FrontEnd Fulcrum mod-fulcrum-backend HTTP Server Backend Fulcrum tpt-http-2.1.8.jar HTTP Transport tpt-jms-1.0.2.jar JMS Transport app-address-book-1.1.4.jar Address Book application Module Example Pinky
Pipes in NK (Yahoo) Module for implementing Yahoo Pipes-like functionality Runs in your computational space No ads Full control
http://sourceforge.net/projects/pinky Pinky Accessors
Union - combine one or more feeds Truncate - keep things manageable Unique - remove duplicate stories Count - how many elements per feed Filter - restrict feed elements to filter criteria Pinky Example NetKernel Administration
Sophisticated management console System Healthcheck Manipulate Modules Monitor Memory, Throttle Settings, Workload Edit Kernel Parameters
http://localhost:1060/ep+name@app_ext_introspect_controlpanel Non-XML Examples NetKernel Links
Open Source Site http://www.1060.org
Commercial Site http://www.1060research.com
Article http://tinyurl.com/ghxg3
Front-End Fulcrum http://localhost:8080
Back-End Fulcrum http://localhost:1060
http://zepheira.com/presentations/2008/NetKernel/ Slides NetKernel.pdf http://zepheira.com/presentations/2008/NetKernel/ Tutorial tutorial.txt http://zepheira.com/presentations/2008/NetKernel/ Examples NetKernel-Examples.zip Please Write Reviews
Feedback/Questions: [email protected]