Lewis Shepherd Chief Technology Officer Microsoft Institute for Advanced Technology in Governments
KEYKEY FACTORSFACTORS ININ THETHE FUTUREFUTURE OFOF COMPUTINGCOMPUTING in the Government Enterprise AnAn ExerciseExercise inin Prediction,Prediction, withwith thethe IntelligenceIntelligence CommunityCommunity asas anan ExampleExample
““WhatWhat technologiestechnologies willwill bebe requiredrequired overover thethe nextnext 1010 yearsyears toto protectprotect U.S.U.S. interests?interests?””
WhatWhat ifif wewe hadhad askedasked thatthat question,question, 1010 yearsyears agoago?? SomeSome SurprisesSurprises PostPost--9/119/11
“Asymmetric adversary” = an information challenge (“hard target”) Seeming irrelevance of traditional methods for new targets - Order of battle (counting military elements) - State-to-state analysis - “Kremlinological” approaches
Challenges of IT during wartime - Stress on systems infrastructure of 2 wars - Stress on software (link-analysis, SNA, “search”) - Stress on collection capacity (sensor grids, Internet) - Stress on analysts’ – and technologists’ – imagination LimitsLimits ofof ““SearchSearch”” forfor PredictionPrediction
WeWeWe don’tdondon’’tt havehavehave aaa “Search”““SearchSearch”” capabilitycapabilitycapability tototo reachreachreach insideinsideinside enemyenemyenemy mindsmindsminds ……… yetyetyet ITIT Challenge:Challenge: LowLow--ObservableObservable AdversaryAdversary
HowHow doesdoes THISTHIS …… … … help help performperform analysisanalysis onon THISTHIS??
OurOur databasesdatabases hadhad nono fieldsfields forfor box-cutters,box-cutters, IMIM accountsaccounts PD1 CaseCase Study:Study: IntelligenceIntelligence CommunityCommunity
The IC’s post-9/11 challenge Some identified solutions: 1. Grid/Cloud computing 2. Secure SOA platform 3. Web 2.0 tools (Intellipedia, A-Space) Implementation challenges Slide 6
PD1 As with the rest of this template, this is a suggested agenda that may or may not fit your situation. Please feel free to make changes as you see fit, including adding and changing pages. In general, the idea is to educate folks on the technology challenge you faced, how you addressed it, and the business benefits the project ultimately delivered - or failed to deliver, as the case may be. Keep in mind that people can learn a lot by hearing what went wrong with your project, so don't be afraid to mix the bad with the good. Paul Desmond, 10/3/2007 WhatWhat DrivesDrives thethe FutureFuture ofof EnterpriseEnterprise ComputingComputing
The value for a new user of a service depends on the number of existing users of the service…
“Critical mass” can lead to “Bandwagon effect”… SideSide--EffectsEffects ofof NetworkNetwork EffectEffect
Exponential growth of networks, systems Requires Scale
Exposes networks to “edge audiences” Requires Security
Derives new wisdom from growing “crowd” Makes Smart Systems Scale Scale:Scale: aa ChallengeChallenge forfor LargeLarge CommercialCommercial EnterprisesEnterprises
“ GOVER NMEN T NO EXCEP TION” RemoteRemote OfficeOffice ITIT ScenariosScenarios NoNo InfrastructureInfrastructure MicrosoftMicrosoft Inc.Inc. asas anan EnterpriseEnterprise ExampleExample
141,000 end users 260,000 computers 550 Buildings in 98 countries 358,000 SharePoint sites 29 billion E-mails sent per day 2,500 internal applications 280 billion page views per 2,500,000 internal E-mails day per day 435 million unique users 136,000 E-mail Server accounts 6 billion instant 1,000,000 remote messages (IMs) per day connections per month DefenseDefense IntelligenceIntelligence AgencyAgency asas anan EnterpriseEnterprise ExampleExample
One of 16 agencies in the Intelligence Community
9,000+ personnel
DIA IT systems support the entire intelligence community 100,000+ users of DIA’s Top Secret network, apps, data
Global reach through IT support of all DoD Commands Pacific Command, European Command, etc.
The only true “all-source” agency in the IC Collection (signals intell, human intell,
measurements & signatures, etc) TheThe Challenge:Challenge: StovepipedStovepiped AnalyticAnalytic CapabilitiesCapabilities Security TheThe SecuritySecurity SideSide ofof ““EnterpriseEnterprise 2.02.0”” SecretSecret toto aa WalledWalled Garden:Garden: ControlControl
Definition: On the Internet, a walled garden is an environment that controls the user's access to Web content and services. In effect, the walled garden directs the user's navigation within particular areas, to allow access to a selection of material, or prevent access to other material. [SearchSecurity.com] WhyWhy WalledWalled--GardenGarden ContentContent && Systems?Systems?
Rationale on the Internet: Money Paid-Access Content Revenue Member-Fee Revenue Exclusive Ad Revenue (Segmented Eyeballs) Value of Intellectual Property
“Enterprise” Rationale: Security Trade Secrets in Operational Data Competitive Advantages “ GOVER NMEN T NO Regulatory Control over Data EXCEP TION” Smart Systems NeedNeed forfor AnalyticAnalytic ReformReform
Traditional IC output: ~50,000 stand-alone reports/year Many redundancies Produced in agency/organization silos Lack of collaborative capabilities across (and within) agencies “Intelink” (the IC-wide shared domain) seen as a backlot
Forcing Function: 9/11 Commission Report Key Recommendation: From Need-to-Know to Need-to-Share!
Realization: “Something that’s 80 percent accurate, on-time, and sharable, is better than something that is perfectly formatted, but too much, too late, and over-classified.” Chris Rasmussen, NGA PD3 BirthBirth PangsPangs ofof ICIC WebWeb 2.0:2.0: 20042004--20052005
Early Efforts were internal, agency-specific projects CIA’s internal blogs, 2004 DIA’s internal “IntelliPedia” wiki, 2004 NGA’s internal blogs, early 2005 DIA’s AJAX mashups in “Lab X,” 2004-05 CIA’s del.ici.ous lookalike, Tag/Connect, 2005
A “Wisdom of Crowds” Culture was forming by 2005 Joint trips to outside conferences Cross-agency collaboration on metadata tagging Formation of “IC Enterprise Services” group, or ICES
Tipping Points, sparked by ICES: August 2005 launch of “Intelink Blogs” April 2006 launch of IC-wide Intellipedia Slide 22
PD3 As with the rest of this template, this is a suggested agenda that may or may not fit your situation. Please feel free to make changes as you see fit, including adding and changing pages. In general, the idea is to educate folks on the technology challenge you faced, how you addressed it, and the business benefits the project ultimately delivered - or failed to deliver, as the case may be. Keep in mind that people can learn a lot by hearing what went wrong with your project, so don't be afraid to mix the bad with the good. Paul Desmond, 10/3/2007 OneOne thingthing wewe learnedlearned wikiwiki--wikiwiki…… KeyKey Distinctions,Distinctions, IntellipediaIntellipedia vsvs WikipediaWikipedia
Business Practices of intelligence analysis & reporting demanded certain technical features:
Not open to the public, only users with access to the IC’s Top Secret network (JWICS), accounts created by ICES.
No anonymity. All edits and additions are traceable.
Intellipedia does not enforce a “neutral point of view” Actually intended to represent various points of view; viewpoints are attributed to the agencies, offices, and individuals participating Consensus may or may not emerge! IntellipediaIntellipedia’’ss HockeyHockey--StickStick GrowthGrowth TheThe TopTop--SecretSecret WikiWiki GetsGets ClonedCloned
Summer 2007, ICES introduced 2 new Intellipedia versions: • One on the SECRET network “SIPRNET” • One on a “Sensitive But Unclassified” network “DNI(U)” (a protected trunk apart from the regular Internet)
Rationale: • Many military intelligence analysts (and most soldiers) only have access to SIPRNET • Many DHS personnel and Law Enforcement have no clearances whatsoever for classified information • Many IC personnel like to work at home on research and topical news items
WalledWalled GardensGardens WithinWithin WalledWalled Gardens:Gardens: RelativeRelative ValueValue ofof ClassifiedClassified InformationInformation
Relative Number of Users, Relative Growth in Intellipedia Pages Also Relative Volume of Data AnticipateAnticipate aa NetworkNetwork EffectEffect forfor DNI(U)?DNI(U)?
Expect increasing rates of growth for DNI(U) usage and information sharing Improved realtime Internet data-mining Awareness of value of collaboration outside traditional IC boundaries (DHS, LE, foreign partners) Improved Web 2.0 tools deployed on DNI(U) to mirror those on JWICS and the Internet IntellipediaIntellipedia TotalsTotals onon AllAll ThreeThree NetworksNetworks
64,782 users 2.3 million edits BottomBottom Line:Line: KnowledgeKnowledge WorkWork isis UniversalUniversal NewNew ICIC Focus:Focus: ““AnalyticAnalytic TransformationTransformation””
Launched by ODNI, April 2007 Both “analysis” side and “techie” side DDNI/A and DNI CIO are the two project owners
Several key programs: Community-wide “IC Data Layer” to aggregate access to “all” databases (no one knows the true number) A-Space, a16-agency “collaborative environment for analysis”
DNI assigned job of ICDL and A-Space to DIA on behalf of full IC - because of our SOA work DIADIA’’ss Alien:Alien: AlAlll--SourceSource IIntelligencentelligence EnEnvironmentvironment
SOASOA PlanningPlanning BegunBegun 20052005:: FullFull webweb--servicesservices frameworkframework Alien is a framework, not a single tool Reliant on globally networked set of data centers
New best-of-breed analytic software
AlienAlien datadata servicesservices –– tyingtying datadata togethertogether Message traffic and other text sources Traditional single-INT databases Integrated security architecture for single sign-on AlienAlien allowsallows toolstools toto exploitexploit semanticallysemantically--enhancedenhanced datadata
METS: Metadata Extraction & Tagging Service “Black-box Tagging Factory” combines 13 separate best-of-breed entity-identifiers, natural-language processors, disambiguators, tagging engines.
34 KeyKey DesiredDesired FeaturesFeatures ofof AA--SpaceSpace
Wikis, blogs, social networking, personalized RSS feeds, collaborative cloud-based word processing, mash-ups, and content tagging…
… all built atop an underlying SOA. AA--Space:Space: thinkthink ““iGoogle,iGoogle,”” ““LiveLive SpacesSpaces”” MetricsMetrics (a(a keykey postpost--9/119/11 recommendation)recommendation)
37 AA--SpaceSpace PilotPilot Schedule:Schedule: BridgeBridge TooToo Far?Far?
Pilot Awarded Sep 14, 2007 Pilot Development and Integration Sep 14-Nov 23, 2007 Pilot Development Freeze Nov 23, 2007 Integration Testing and IPAT Nov 26-30, 2007 Functional Testing (Approved Users) Dec 3-7, 2007 Final Clean Up Dec 10-12, 2007 C&A DIA* Dec 13-14, 2007 C&A DNI* Dec 17-19, 2007 Installation at DIA’s main Data Center Dec 20-28, 2007 Prototype available to IC users Dec 31, 2007 Lesson: Many Enterprise IT Projects Fall Short of Expectations
Average IT Project Success
Time overruns Budget overruns Incomplete features On time Incomplete functions On budget Desired features Desired functions
Cancelled prior to completion Abandoned “ GOVER NMEN T NO EXCEP TION” Source: CIO Executive Board research; Standish Group 2004 CHAOS Report OtherOther GovernmentGovernment Examples:Examples: epa.wik.isepa.wik.is
http://epa.wik.is/ epa.wik.isepa.wik.is goesgoes mashupmashup bigtimebigtime
41 Extensibility: Integration with Yahoo!, Windows Live, Google, Flickr, WidgetBox, YouTube, and much more.
“Data reuse in mashups will revolutionize EPA data architecture, data management, and data reuse applications!” EPA Architect Brand Niemann NearNear--FutureFuture ITIT EnablersEnablers forfor thethe ICIC
SemanticSemantic WebWeb -- GlobalGlobal all-sourceall-source systemsystem enablingenabling richrich ontologicalontological informationinformation managementmanagement ¾¾ autonomouslyautonomously andand presumptivelypresumptively alertingalerting analystsanalysts ¾¾ automaticallyautomatically populatingpopulating knowledgeknowledge basesbases ¾¾ cueingcueing otherother militarymilitary andand ITIT systemssystems
GIGINTGIGINT -- abilityability toto minemine andand controlcontrol thethe GlobalGlobal InformationInformation GridGrid withoutwithout humanhuman intervention,intervention, inclincludinguding thethe billionsbillions ofof sensor/sensor/ RFID/nano/autonomousRFID/nano/autonomous devicesdevices communicatingcommunicating withwith thethe Grid.Grid. ¾¾ Gartner:Gartner: ByBy 2013,2013, moremore thanthan 220000 billionbillion processorsprocessors willwill bebe inin dailydaily useuse aroundaround thethe worldworld VirtualVirtual WorldsWorlds
New methods of modeling, simulation, and collaboration are being created for analysts and collectors
“Knowledge Walls” and Crisis Centers can be built more cheaply in a Virtual World, still using real-time feeds
43 ResearchResearch UnderwayUnderway forfor FutureFuture EnterpriseEnterprise EffectivenessEffectiveness
1. SOA environments driven entirely by business processes 2. Cross-Domain capabilities as embedded, intuitive services 3. Rapid increases in speed/volume of sensor and analytic feeds 4. Stateless devices (the ultimate thin client “computer”) 5. Wideband agile human interfaces, and true video tele-presence 6. The far edges of technological support for analysis: Support to prediction; Crisis uncertainty management; Dynamic retasking of machines by machines... Lesson:Lesson: JointJoint LeadershipLeadership ResponsibilityResponsibility
100 Business Leaders
CIO Accountability 0 IT Portfolio Lifecycle
“ GOVER NMEN T NO EXCEP TION” Lewis Shepherd Microsoft Institute for Advanced Technology in Governments www.ShepherdsPi.com