sign in sign up
<<
Home , Information architecture

INFORMATION the power of data

2019/02/05 - [email protected] ANCIENT EGYPT PROBLEM solving large scale problems with data

Khufu “Khufu’s Horizon” Egypt Public works project: Date: ~2580 BC the largest man made building Population: ~2 million in human history to date

(and the tallest man-made structure in the world for the next 3,800 years) THE GREAT PYRAMID OF GIZA PROJECT executive sponsor and stakeholders

Executive Sponsor Pharaoh Stakeholders Vizier & Ruling Class THE GREAT PYRAMID OF GIZA PROJECT architecture

Vision

diagrams 146.7 metres (481 metres 146.7 (481 ft)

230.34 metres (756 ft) THE GREAT PYRAMID OF GIZA PROJECT engineering

Material: Limestone, 2.3 million blocks, Physical 2-10 metric ton each, Implementation nearby quarry; Technology, tools, materials Material: Granite, 25-80 metric ton each, quarry 500 miles away;

Height: 146.7 metres (481 ft) Base Length: 230.34 metres (756 ft)

Constraints: no wheel, or iron tools THE GREAT PYRAMID OF GIZA PROJECT quality control

Quality expectations for Precision: 0.5 millimetres (0.020 in) the final product Height: 146.7 metres (481 ft) (passing bar) Base Length: 230.34 metres (756 ft) THE GREAT PYRAMID OF GIZA PROJECT GPMO - great pyramid management office

Delivery date: 2560 BC Management Construction time: 20 years Schedule, tasks, 1 block / 5 minutes, 24/7 for 20 years workforce resource Required workforce: ~250,000 planning workers THE GREAT PYRAMID OF GIZA PROJECT the problem statement

how to motivate 250,000 citizens to dedicate 20 years of their lives to build the pyramid? THE GREAT PYRAMID OF GIZA PROJECT solving large scale problem with data

pharaoh, vizier The Main Event priests, nobles

scribes, soldiers, craftsmen, farmers, servants, slaves Ra - the sun god Osiris - the god of the dead and ruler of the underworld THE GREAT PYRAMID OF GIZA one of the seven wonders of the ancient world LESSON #1 power of data - ancient world, current times, future (?)

Around the 4th millennium BC, the complexity of trade and administration outgrew the power of memory, and writing became a more dependable method of recording and presenting transactions in a permanent form. LESSON #2 origins of engineering/architecture

Architecture Vision, Engineering Technology, tools, materials Project Management Schedule, tasks, resource planning Quality Control Quality expectation, compliance LESSON #3 data patterns: data - - intelligence

business knowledge + information strategic intelligence use

information context (viewpoints)

data facts + meaning Four domains of enterprise architecture by The Open Group Architecture Framework – TOGAF (TOGAF framework): ● Business architecture ● Data architecture ● Applications architecture ● Technology architecture Data Architecture approach In , data architecture (Wikipedia) is composed of models, policies, rules or standards that govern which data is collected, and how it is stored, arranged, integrated, and put to use in data systems and in organizations.

Data Data Data Data Collection Storage Integration Arrangement Data Usage Architecture Zachman Architecture Framework Data Landscape a few examples… • data creation, data collection, data exchange, data preservation • data integration, data presentation, data • data organization, data catalogs, • data quality, data profiling, data governance • data warehousing, business intelligence • , data discovery • master data, reference data • data federation, data virtualization • unstructured data, big data, of things , noSQL • cloud computing, data as a service • data on-premises, data in the cloud, data lakes • data mining, data science, artificial intelligence, machine learning • data encryption, distributed data, , bitcoin architecture • dark data… Data Modeling data models - perspectives Abbreviation Model name Focus on…

CDM Conceptual Data Broad understanding, ‘one- Model pager’ on how something works (key business concepts and rules) LDM Logical Data Detailed business solution on Model how something works (according to strict business rules and independent of technology) PDM Physical Data Detailed technical solution on Model how something works (DBMS technology specific) Data Modeling data models – detail level Model Elements LDM Entity Attribute Relationship Cardinality Domain Primary Key Foreign Key PDM Table Column Relationship Cardinality Data Type Primary Key Foreign Key Index Partitioning Data Modeling conceptual data model Conceptual Data Model shows the key concepts in a particular area and how these concepts interact with each other).

Benefits of a complete model: • Provides broad understanding (capture requirements) • Defines scope and direction (new business functionality) • Offers proactive analysis (important issues and concerns) • Builds rapport between IT and the business (reduce communication barriers) Data Modeling logical data model Data elements required for a given application or business process, organized into entities according to strict business rules and independent of technology.

Normalizing is a formal process of asking business questions. Normalization ensures that every data element is a fact about the key (1NF), the whole key (2NF), and nothing but key (3NF).

Model Scorecard (basic): • Captures requirements • Represents business • Follows naming standards • Good definitions • Arranged for readability • Structurally sound Data Patterns traditional RDBMS workflow Data Patterns ETL vs ELT Data Patterns big data workflow Data Patterns lambda architecture Data Patterns lambda architecture example (Azure) Data Patterns machine learning at scale Data Patterns natural language processing Data Patterns data preparation at scale Systems NoSQL use Cloud Patterns cloud development Patterns for building reliable, scalable, secure applications in the cloud. Challenges: ● Availability ● Data Management ● Design and Implementation ● Messaging ● Management and Monitoring ● Performance and Scalability ● Resiliency ● Security

Link: https://docs.microsoft.com/en-us/azure/architecture/patterns INFORMATION ARCHITECTURE the power of data

THANK YOU!

2019/02/05 - [email protected]