open-emv.com The Team Roadmap • Introduction • Solution Overview • Energy Efficiency 101 • Site Tour • Under the Hood • Basic Machine Configuration (AMIs) • Software Configuration • Database Structure • The Web Application The Solution User's Browser jQuery OpenLayers Issue Tracking VCS (FogBugz) (beanstalk) User AWS EC2 m2.xlarge c1.medium (high-memory) (high-CPU) Database Applications Front-End Applications ZEO Zope Client Apache Zope Object Database Server Zope Client Zope Client PG Django PostgreSQL Database Bouncer (mod_wsgi) monitoring monitoring Development Machine zc.buildout mr.developer m1.small Monitoring Dashboard Developer Why CA Energy Efficiency? World Leader • 1.5 Million Jobs • $45 Billion in Payroll How does that compare to the rest of the U.S? California... • ... spends billions less on Electricity • ... has a ratio of GDP to Energy Consumed 68% higher • ... residents, per capita, pay lower utility bills !"#$%&'()*+(,()-+()./0'-,1)./023)4()&5)&-)*(,()$6)(7.&('-)$6)8$2&5/,'&$9:) ;))<9)=/(2)>(,,1?)@('-0,().$A&-$2&6-?)5/0'3(,)/5)=(B-)CD) Energy Efficiency and EMV How Does CA do Energy Efficiency? • CPUC established EE program cycle length and funding level • CPUC provides $ to utilities to implement EE Programs • Utilities report program accomplishments to CPUC How Does CA do Energy Efficiency? • CPUC hires independent EMV contractors to independently evaluate utility savings claims • CPUC awards $ to the utilities based on savings accomplishments • CPUC releases data publicly via website Rinse and Repeat Roadmap • Introduction • Solution Overview • Energy Efficiency 101 • Site Tour • Under the Hood • Basic Machine Configuration (AMIs) • Software Configuration • Database Structure • The Web Application Roadmap • Introduction • Solution Overview • Energy Efficiency 101 • Site Tour • Under the Hood • Basic Machine Configuration (AMIs) • Software Configuration • Database Structure • The Web Application AMI AMI (Amazon Machine Image) User's Browser jQuery OpenLayers Issue Tracking VCS (FogBugz) (beanstalk) User AWS EC2 m2.xlarge c1.medium (high-memory) (high-CPU) Database Applications Front-End Applications ZEO Zope Client Apache Zope Object Database Server Zope Client Zope Client PG Django PostgreSQL Database Bouncer (mod_wsgi) monitoring monitoring Development Machine zc.buildout mr.developer m1.small Monitoring Dashboard Developer m2.xlarge c1.medium (high-memory) (high-CPU) Database Applications Front-End Applications ZEO Zope Client Apache Zope Object Database Server Zope Client Zope Client PG Django PostgreSQL Database Bouncer (mod_wsgi) Custom DB AMI m2.xlarge (high-memory) Database Applications ZEO Zope Object Database Server PG PostgreSQL Database Bouncer monitoring Custom DB AMI m2.xlarge (high-memory) Database Applications PG PostgreSQL Database Bouncer monitoring Custom DB AMI m2.xlarge (high-memory) • Ubuntu 10.10 (Maverick) Database Applications PG PostgreSQL Database Bouncer monitoring Custom DB AMI m2.xlarge (high-memory) • Ubuntu 10.10 (Maverick) Database Applications • 64-bit PG PostgreSQL Database Bouncer monitoring Custom DB AMI m2.xlarge (high-memory) • Ubuntu 10.10 (Maverick) Database Applications • 64-bit • Kernel Settings PG PostgreSQL Database Bouncer monitoring Custom DB AMI m2.xlarge (high-memory) • Ubuntu 10.10 (Maverick) Database Applications • 64-bit • Kernel Settings kernel.shmmax = 8GB PG PostgreSQL Database Bouncer kernel.shmall = 8GB/4096 monitoring /etc/sysctl.conf Custom DB AMI m2.xlarge (high-memory) • Ubuntu 10.10 (Maverick) Database Applications • 64-bit • Kernel Settings PG PostgreSQL Database Bouncer • Spatial Database monitoring Custom DB AMI m2.xlarge (high-memory) • Ubuntu 10.10 (Maverick) Database Applications • 64-bit • Kernel Settings PG PostgreSQL Database Bouncer • Spatial Database apt-get install... PostgreSQL 9.0.3 monitoring Geos 3.2.2 GDAL 1.7.0 manual install postgis 1.5.2 Custom DB AMI m2.xlarge (high-memory) • Ubuntu 10.10 (Maverick) Database Applications • 64-bit • Kernel Settings PG PostgreSQL Database Bouncer • Spatial Database monitoring • Connection Pooling Custom DB AMI m2.xlarge (high-memory) • Ubuntu 10.10 (Maverick) Database Applications • 64-bit • Kernel Settings PG PostgreSQL Database Bouncer • Spatial Database monitoring • Connection Pooling apt-get install... pgbouncer 1.3.3 Custom DB AMI m2.xlarge (high-memory) • Ubuntu 10.10 (Maverick) Database Applications • 64-bit • Kernel Settings PG PostgreSQL Database Bouncer • Spatial Database monitoring • Connection Pooling • Monitoring Custom DB AMI m2.xlarge (high-memory) • Ubuntu 10.10 (Maverick) Database Applications • 64-bit • Kernel Settings PG PostgreSQL Database Bouncer • Spatial Database monitoring • Connection Pooling • Monitoring apt-get install... monit munin-node Custom Web App AMI c1.medium (high-CPU) Front-End Applications Zope Client Apache Zope Client Zope Client Django (mod_wsgi) monitoring Custom Web App AMI c1.medium (high-CPU) Front-End Applications Zope Client Apache Zope Client Zope Client Django (mod_wsgi) monitoring Custom Web App AMI c1.medium (high-CPU) Ubuntu 10.10 (Maverick) • Front-End Applications Zope Client Apache Zope Client Zope Client Django (mod_wsgi) monitoring Custom Web App AMI c1.medium (high-CPU) Ubuntu 10.10 (Maverick) • Front-End Applications • 32-bit Zope Client Apache Zope Client Zope Client Django (mod_wsgi) monitoring Custom Web App AMI c1.medium (high-CPU) Ubuntu 10.10 (Maverick) • Front-End Applications • 32-bit Zope Client Apache Zope Client Apache2 Zope Client • Django (mod_wsgi) monitoring Custom Web App AMI c1.medium (high-CPU) Ubuntu 10.10 (Maverick) • Front-End Applications • 32-bit Zope Client Apache Zope Client Apache2 Zope Client • Django (mod_wsgi) • mod_wsgi monitoring Custom Web App AMI c1.medium (high-CPU) Ubuntu 10.10 (Maverick) • Front-End Applications • 32-bit Zope Client Apache Zope Client Apache2 Zope Client • Django (mod_wsgi) • mod_wsgi monitoring • Monitoring We’ll make these public We’ll make these public – once I’ve polished them up a bit :) Additional Software Additional Software • Back-End Additional Software • Back-End • ZODB Additional Software • Back-End • ZODB • ZEO Server Additional Software • Back-End • Front-End • ZODB • ZEO Server Additional Software • Back-End • Front-End • ZODB • Django • ZEO Server Additional Software • Back-End • Front-End • ZODB • Django • ZEO Server • Plone Additional Software • Back-End • Front-End • ZODB • Django • ZEO Server • Plone • Add-Ons Additional Software • Back-End • Front-End • ZODB • Django • ZEO Server • Plone • Add-Ons • Zope Clients And Configuration • Back-End • Front-End • ZODB • Django • ZEO Server • Plone • Add-Ons • Zope Clients And Configuration • Back-End • Front-End • ZODB • Django • ZEO Server • Plone • Add-Ons • Zope Clients And Configuration • Back-End • Front-End • ZODB • Django • ZEO Server • Plone PostgreSQL • Add-Ons • Zope Clients And Configuration • Back-End • Front-End • ZODB • Django • ZEO Server • Plone PostgreSQL • Add-Ons • Zope Clients And Configuration • Back-End • Front-End • ZODB • Django • ZEO Server • Plone PostgreSQL • Add-Ons • Zope Clients Apache And Configuration • Back-End • Front-End • ZODB • Django • ZEO Server • Plone PostgreSQL • Add-Ons • Zope Clients Apache And Configuration • Back-End • Front-End • ZODB • Django • ZEO Server • Plone PostgreSQL • Add-Ons • Zope Clients Apache Monit And Configuration • Back-End • Front-End • ZODB • Django • ZEO Server • Plone PostgreSQL • Add-Ons • Zope Clients Replicate OnApache Development Machine(s) Monit Photo by EvelynGiggles via Flickr using a CC-BY license. This is a job for Superman! This is a job for Buildout! Wait – what? zc.buildout www.buildout.org developed for used extensively by also gaining traction with not limited to python dependencies dependencies $ apt-get install python-setuptools dependencies $ apt-get install python-setuptools $ easy-install distribute dependencies $ apt-get install python-setuptools $ easy-install distribute $ apt-get install python-dev The Basics The Basics $ ls basic-buildout bootstrap.py buildout.cfg The Basics The Basics $ more buildout.cfg [buildout] parts = foo [foo] recipe = collective.recipe.cmd cmd = echo foo Build It Run It Photo by Rich Moore via Flickr using a CC-BY license. let’s scale this up lots of cfg, and more base.cfg pg_parts.cfg A Closer Look Tie it all together... dev.cfg or, for deployment... backend.cfg Buildout Advantages Buildout Advantages • Easy to write Buildout Advantages • Easy to write • Lots of recipes, examples Buildout Advantages • Easy to write • Lots of recipes, examples • Manage complexity Buildout Advantages • Easy to write • Lots of recipes, examples • Manage complexity • Duplicate across platforms Buildout Advantages • Easy to write • Lots of recipes, examples • Manage complexity • Duplicate across platforms • Deploy rapidly Buildout Advantages • Easy to write • Lots of recipes, examples • Manage complexity • Duplicate across platforms • Deploy rapidly • Versioning of software setup and config not without issues not without issues • Can be network-dependent not without issues • Can be network-dependent • Interconnected parts mean complexity not without issues • Can be network-dependent • Interconnected parts mean complexity • ‘extends’ can be tricky to manage use it in joy Data ingest Data dimensions The Web Application 8 Axes of Data 8 Axes of Data • Time Frame (lifecycle or first-year) 8 Axes of Data • Time Frame (lifecycle or first-year) • Savings Type (but cost too) 8 Axes of Data • Time Frame (lifecycle or first-year) • Savings Type (but cost too)