HPC, Cloud, BAO a BigData Milan Král IBM STG 8. července 2012 1
Zajímavé oblasti? HPC/Deep HPC/Deep Computing/ Computing/Technical Computing technické a vědecké výpočty modelování a simulace podpůrné sw (GPFS…) a hw
Cloud/ Cloud/Virtualizace/Automatizace Virtualizace/Automatizace
Standardizace možnosti cloudu IBM řešní pro budování cloudu
BAO & BIG DATA
Klasická architektura In Memory koncept BIG Data příklad
… nebo marketingové ‚buzzwordy‘? 2
HPC, Deep Computing, Technical Computing
3
Aplikační oblasti HPC
Derivative Analysis
Seismic Analysis
Reservoir Analysis
Actuarial Analysis Asset Liability Management
Process Simulation
Finite Element Analysis
Portfolio Risk Analysis Statistical Analysis
Energy
Mechanical/ Electric Design
Finance
Failure Analysis
Mfg
Drug Discovery Protein Folding
Collaborative Research Bandwidth Consumption Digital Rendering
Weather Analysis
Medical Imaging
Gaming
High Energy Physics
Life Sci.
Media
Gov’t
4
Klima a počasí
5
Příklad testů a očekávání Příklad testů a gridu modelů: - Meteo France models - ALADIN/ALARO with 5.0km horizontal resolution, 1080x960x60 grid - AROME with 2.5km horizontal resolution, 432x320x60 grid
-
Ale stále je prostor: - problémy (paralelizace, sdílení dat…) - Grid km/stovky m? - lokální předpovědi? přívalové srážky, tání sněhu, vliv na nestabilní energetické zdroje (fotovoltaika, větrně elektrárny…) … 6
Increasing demands for more accurate NWP (Numerical Weather Prediction) The Challenge facing Climate & Weather Scientific driving Scientificdemands demands are driving hugeincreases increases in in capacity capacity huge Applicationscaling scaling is forcing Application forcing moreensemble ensemble solutions solutions more
Global Regional
20km 10km
10 km 5km
Scalingisisstill still aa key key factor factor in Scaling in time sensitive production runs time sensitive production runs Data demands will be huge
Little sign of this slowing nextofdecade Littleinsign this slowing in next decade
Odhad rů růstu potř potřebné ebného výkonu:
Walter Zwielfhofer May 2008
2020-25x v nejbliž nejbližších ších 5ti letech 7
Crash simulace a výpočty proudění
Investigates the cast subframe prediction on the chassis of the new VW Passat. Courtesy of Volkswagen AG Internal combustion engine modeled Pressure contours in side-window buffeting simulation of passenger car, using computational aeroacoustics method. Courtesy Daimler Chrysler. Predition how car interior components, such as dashboard and trims, will behave under the occupants' impact.
Comparison of real and simulated frontal crash Courtesy of Volkswagen AG 8
Akustická optimalizace • model characteristics •
Structure DOFs:
11.5M
•
Acoustic DOFs:
1.2M
•
# Str. Modes:
2950
•
# Fluid Modes:
•
# Exc. Freq:
•
# Loads
•
# Design variables:
7.1 dB
200 4 190 (Panel thickness +/- 20%)
Pressure (dbA)
99
• objective •
minimize max acoustic pressure at driver’s ear
• optimization • • •
S1003_70001951_opt S1003_70003932_opt
11 optimization steps 93 hrs; 115TB of IO improvement of 7.1 dB in maximum pressure
0
50
100
150
200
Frequency (Hz)
• observations • •
increased complexity of CAE models is reducing the ability of engineers to intuitively steer design process computer based design optimization will increasing drive product design
Mladen Chargin CDH AG Advanced Engineering Am Marktplatz 6 79336 Herbolzheim Germany 9
Deep Computing/HPC nejsou jen servery… SW stack cluster management, workload management kompilátory, debuggery, trace atd. knihovny, MPI… paralelní filesystémy (GPFS…) …
Výkon, energie, chlazení výpočetní hustota energetická náročnost per rack (10x víc než před 10ti lety) chlazení a environment friendly přístup
Ukládání dat a konektivita Ethernet, Infininand, SAN Disková pole Zálohování
10
Rear Door Heat exchanger Perforated Door for clear airflow
IBM patented hex airflow design Lock handle to close/ open door
Swings to provide access to rear PDUs
No Leaks Sealed Internal coils
Industry Standard hose fittings 11
Trend chlazení teplou vodou 100% heat removal possible Passive heat extraction Low airside impedance for rack Low waterside impedance Uniform node airflow optmizes RDHX performance Exit air temperatures uniform across rack height (for 2U node configuration)
120 115 % Heat Removal
• • • • • •
% Heat Removal as function of Water Temperature and Flowrate for: 24C rack inlet air, 32KW load, Min Fan Speed
110 105
H2O Temp 12 C * 14 C * 16 C * 18 C 20 C
100 95 90 85 80 75
TREND: chlazení teplou vodou ažž 45° °C 8
10
12
Water Flowrate (gpm)
14
Energeticky usporné 12
Pohled zpět… •
CAE deployments • 1985 1 core • CRAY XMP • 1992 10 cores • CRAY YMP • 1998 100 cores • SGI PowerChallenge • 2004 1,000 cores • POWER4, Itanium, x86 • 2010 10,000+ cores • x86_64 • 2015 100,000 cores • ???
question: how can effective build, use, and manage such large systems
13
… a vpřed: Science and Technology Strategy / Roadmap 2000
2005
2010
2015
2020
2025
2030
Extending Si CMOS R
D
Subsystem Integration R
D
Post Si CMOS Options R
R&D
14
Konec HPC části…
15
© 2011 IBM Corporation
Cloud & IBM Technologie
Cloud Computing as “the Industrialization of IT” (Sam Palmisano, 2009) It’s about improved processes: “evolution” not “revolution”
• Cloud computing “is not really new, which is why so many of its ingredients feel familiar. Rather, it represents a confluence of forces that have been building” • “Enterprises spend a lot of time […] doing similar tasks in different ways. […] Cloud-based business services bring productivity and quality improvements”
Irving Wladawsky-Berger Chairman Emeritus, IBM Academy of Technology
http://blog.irvingwb.com/blog/2009/12/cloud-computing-and-the-evolution-toward-the-outsidein-virtual-enterprise.html#more
17
© 2011 IBM Corporation
Cloud becomes top CIO technology priority in 2011 Gartner 2011 CIO priorities
Source: Gartner – Reimagining IT: The 2011 CIO Agenda 18
© 2011 IBM Corporation
IDC: Hlavní výzvy Cloud Computingu
Q: Jaké dvě hlavní výzvy očekáváte, budete-li přecházet na privátní čí veřejný cloud?
IDC‘s Datacenter and Cloud Computing Survey Leden 2010
19
IDC, Cloud Computing Attitudes, Doc # 223077, Apr 2010
© 2011 IBM Corporation
Cloud will play a fundamental role in this new era.
Cloud
• Masks complexity of underlying components • Rapid deployment of processes and infrastructure • Consumer-driven services, process, deployment
Business Services Platform Infrastructure 20
Consume business processes, analytics applications running on Cloud infrastructure
Deploy consumer-created or -acquired applications on the Cloud infrastructure
Rapidly provision computing resources for deploying and running software
© 2011 IBM Corporation
Implementační modely Privátní
IT funkce je poskytována ”jako služba“ prostřednictvím intranetu, v rámci podniku a za firewallem
Hybridní
Služba IT je integrována pro interní a externí konzumenty
Enterprise data center
Enterprise data center
Enterprise
Private cloud
Managed private cloud
Hosted private cloud
• Privátní • U klienta • Klient provozuje/ spravuje 21
• Operováno třetí stranou • Vlastní klient • Vnitřní sít
Veřejný
• Vlastněno a operováno třetí stranou • Vnitřní síť • Bezpečnost
IT funkce poskytovány jako služba přes Internet
Enterprise A
B
Member cloud services • Sdílené infrastrukturní zdroje • Sdílené lidské zdroje • VPN přístup
Users A
B
Public cloud services • Sdílené/dedikované zdroje • Platba dle využití © 2011 IBM Corporation • Internet př.
Komponenty „Cloud Computingu“ Klient
Samoobsluž Samoobslužný portá portál
Katalog služ služeb
Implementač Implementační Modul - (provisioning) provisioning)
Další Další moduly Měření využití služby, účtování, monitorování
Klí Klíčové ové vlastnosti: vlastnosti: • Selfelf-service; service službu objednávám „samoobslužně“ • Služby vybírám ze standardizované standardizované nabídky (katalogu) • Služba je objednávána automatizovaně automatizovaně • Cloud zajistí provisioning, provisioning, tzn. připraví službu k použití • Služby jsem schopen měřit ěřit a účtovat tzv. „Pay-per-use“ • Flexibilní Flexibilní specifikace implementačních modelů 22
Virtualizovaná Virtualizovaná HW infrastruktura
© 2011 IBM Corporation
Proč se cloud computingem zabývat ? VIRTUALIZACE
+
STANDARDIZACE
+
Hmatatelné výsledky při užití IBM cloud computingu
Urychlení & Flexibilita
Příprava testovacího prostředí
Tradiční
Flexibilita
Cloud Minuty
Change management
Měsíc/Dny
Dny/Hodiny
Release management
Týdny
Minuty
Administrativní
Samoobslužžný
Komplexní
Sdílená
Fixní náklady
Variabilní náklady
10–20%
70–90%
Roky
Mě ěsíce
Standardizace Měř ěření využžití/úč čtování ěř
23
=
Náklady
Týdny
Přřístup ke služžbě ě
Redukce nákladů
AUTOMATIZACE
Využžitelnost HW Návratnost investic
© 2011 IBM Corporation
Metodologie kalkulace pro privátní cloud Obtížně vyčíslitelné benefity: • ‚hodnota‘ rychlosti poskytnutí služby/provisioning? • ‚hodnota‘ interního billingu, reportingu? 4
Známé (příp. odhadnuté) náklady: • hw, sw • hw a sw údržbu • facility, energie, prostory • lidské zdroje, ostatní OPEX
Enterprise deployment/Integrated 3 Limited Deployment/Evaluation
Očekávané hodnoty: • frekvence provisioning?
2 Discovery 1
Business Case Summary Total CPU Cores Used CPU Cores Total CPU Sockets #Logical Servers #Physical Servers Ave.Log.Srv RIP Total RIP Capacity Total RIP Workload Ave %Utilization
Current 8,530.00 8,530.00 4,394.0 4441.00 1943.00 1,267.9 5,630,837.1 1,185,482.0 22%
BAU + Cloud 3024 3000 756 4346 378.00 974.2 4,233,750.0 1,185,482.0 28%
Blades + Cloud 2224 1947 278 4346 139.00 547.4 2,378,990.6 1,185,482.0 50%
CloudBurst 1638 1561 273 4346 12.07 547.8 2,380,525.0 1,185,482.0 50%
Change -81% -82% -94% -2% -99% -57% -58% 0% 124%
3 Year Projection Millions
Sizing
45
Transition
40
R O
0 Hardware Maint
35
Space
30
Electric
25
Annual Operating Costs (AOC) Staff Cost Code System Software M&S Hardware Maint Space Electric Staff Depreciation Total AOC est.potential saving /yr
Today 5,623,864 0 210,832 995,135 1,617,461 0 8,447,292
Cloud 7,727,281 0 36,275 392,319 1,455,715 0 9,611,590 -1,164,298
Fewer servers 7,405,822 0 4,288 260,351 1,310,143 0 8,980,604 -533,312
CloudBurst 1,214,658 0 11,696 263,086 1,179,129 0 2,668,569 5,778,723
Software Purchase Hardware Purchase Transition Total OTC Write-Off Net Cash Investment
0 13,529,500 0 13,529,500 0
0 2,835,000 0 2,835,000 0 -10,694,500
0 3,118,635 0 3,118,635 0 -10,410,865
0 6,818,058 0 6,818,058 0 -6,711,442
OTC + 3x AOC 3 yr saving Payback Period
38,871,376
31,669,770 7,201,606 9yr2m
30,060,448 8,810,928 19yr6m
14,823,765 24,047,611 -2yr10m
One Time Costs (OTC)
Depreciation
20
Z V
-78% 0% -94% -74% -27% 0% -68%
No Action
15 10
Staff
Software Purchase
5
Hardware Purchase
0
C
3
2
1
System Software M&S
3 Year Projection
24
Project Time 0yr0m
© 2011 IBM Corporation
Základní „IBM produkty“ v oblasti budování Cloud Computingu
IBM Smart Cloud Entry (Starter Kit for Cloud) Foundation
Technologie pro vybudování 25 Cloudu
Services
Platforma IBM Datových Center nabízející služby Veřřejného (Public) Cloudu
Solutions
Software jako služžba © 2011 IBM Corporation
Smart Cloud Entry via Starter Kit for Cloud ‚Entry Entry‘ řešení privátního cloudu Samoobslužný portál Katalog služeb Workfow Provisioning a expirace Měření a reportování Intuitivní rozhraní Snadno nasaditelné Privátní cloud VMware KVM a PowerVM
26
© 2011 IBM Corporation
IBM Service Delivery Manager Integrované softwarové řešení pro ‚Advanced‘ cloud Samoobslužný portál Katalog služeb Pokroč čilé workfow Provisioning (včetně expirace) Měření, reportování, úč čtování Monitoring Kapacitní plánování Multitenancy
Uživatelé mohou požádat o službu kdykoliv potřebují, sledovat stav svých služeb a za zkonzumované služby dostávají vyúčtování
Privátní Cloud • Hybridní Cloud • Veřejný Cloud •
Multiplatformní (současně) VMWare, Xen, KVM • PowerVM, zVM • Hyper-V v roadmapě •
27
© 2011 IBM Corporation
IBM Power Systems
As a result, cloud is an increasingly attractive means of creating and delivering IT services.
Value delivered
From traditional
To cloud
Change management
Months
Days or hours
Test provisioning
Weeks
20 minutes
Install database
1 day
12 minutes
Install of operating system
1 day
30–60 minutes
Provisioning environment
▄
51% cost savings
Design and deploy business applications
Months
Days/Weeks
“Our commitment to informed decision making led us to consider private cloud delivery, which is the enabling foundation that makes possible +$20M savings over 5 years.” – IBM Office of the CIO
28
© 2011 IBM Corporation
Závěr Cloud části…
29
© 2011 IBM Corporation
BI, BA, BAO & Big Data
CIO #1 Concern Business Analytics
83% 76%
Virtualization 71%
Risk Management & Compliance Mobility Solutions
68%
Customer & Partner Collaboration
68%
Self-service Portals Application Harmonization
64%
Business Process Management
64%
SOA / Web Services Unified Communications
31 31
66%
Source: IBM Global CIO Study, n = 2345
61% 60%
Komponenty
Data Warehouse
Cubing Services Agregace Indexes
ETL
Operational Source Systems Structured/ Unstructured Data
32
Implementation Services
DB2 Utilities Suite
© 2011 IBM Corporation
33
34 34
Paměť RAM, SSD a pevné disky Processors
Memory
Very, very, very, very, very fast
Very, very, very fast
< 10’s ns
~100 ns
SSD
Fast
~200,000 ns
Disk
Very, very slow comparatively
1,000,000 8,000,000 ns
Access Speed
~1 second
~33 minutes
~ 12.5 hours
Human Time Context
35
© 2011 IBM Corporation
SAP In Memory řešení HANA
Hardware inovace
Multi-Core Architecture (8 x 8core CPU per blade)
Massive parallel scaling with many blades
36
SAP Software inovace
Row and Column Store
Compression
Partitioning
No Aggregate Tables
Insert Only on Delta
One blade ~$50.000 = 1 Enterprise Class Server 64bit address space – 2TB in current servers
100GB/s data throughput
Dramatic decline in price/performance
= IBM HANA Appliance
© 2011 IBM Corporation
IBM In Memory řešení: COGNOS TM1 Speed and flexibility real-time calculations, what-if and ad-hoc analysis, data refresh and write-back
Security complex management of user access based on roles and single-sign-on capability
Scalability it is possible to operate multiple servers with different applications on different places
In-memory the whole multidimensional database runs in the physical memory so everything is incredibly fast
Real-time both data refresh and calculations run real-time
Read/write not only data reading is supported, but also writing data back to database is no problem 37
© 2011 IBM Corporation
InfoSphere Streams: In-Motion Vs Traditional Analytics Stream Analytics
RTAP Non-Traditional / Non- Relational Data Sources
In-Motion Analytics
Millions of events or Terabytes of data per second Results with Microsecond latencies Ultra Low Latency Results
RHEL v5.3 or v5.4 x86 multicore hardware InfiniBand support Up to 125 servers
Audio, Video, emails…
OLAP / OLTP Traditional / Relational Data Sources
Traditional Analytics
(Alpha)Numeric, text…
Warehouse
Streams DOES NOT store data for analysis Traditional data is finite, saved and known. Streaming data is NONE of these
At-Rest Data Analytics
Results
Streams: Industry Focus
Government
Telco
•Security, foreign intelligence, foreign surveillance and cyber security •Complex analysis of data in motion such as audio, video, email etc. •Structured and unstructured data that require highly time sensitive •Security is the #1 spending initiative for the US Gov’t.
•Real time insights into network and subscriber behavior •Improve network performance, reduce customer churn, •Increase loyalty & appeal, prevent spam, •Focus on countries w/ emerging growth leapfrogging hw phone lines •Subscriber base doubling every 2 years.
FSS Algorithmic & high frequency trading, fraud detection in real time .Trading companies reducing costs as well as to add more value to their offerings. Brokerage firms looking to find ways to add significant value to their customers
Energy / Utilities Healthcare •Smart meters, sensors, smart grid •Analysis in real time for energy savings and prevention of outages. •Analytics and complex logic & the need for extreme speed to take action. •Significant ROI is emerging.
•Real time predictive modeling & fast analysis on “Big Data •Analytics on large volumes of physiological data from increasingly sophisticated sensors and medical devices. ie:Patient Monitoring in ICU
Real-Time BIG DATA: řešení Apache Hadoop Applying: SAP HANA – strukturovaný přístup Apache Hadoop – nestrukturovaná data IBM Servery a GPFS to retail point of sales and web log data
New Approach on business analytics and optimization
New Approach Traditional Approach
Predict and act Velocity
Lack of Insight
Inefficient Access
Sense and respond
Instinct and intuition
Real-time, fact-driven
Volume Skilled analytics experts
Variety
Inability to Predict
Back office
Everyone
Point of impact
Automated
Optimized
41
Konec BAO & Big Data části
These enterprises are addressing the challenges that emerged during the last era of computing… 32.6 million servers worldwide • 85% idle computer capacity • 15% of servers run 24/7 without being actively used on a daily basis
1.2 Zetabytes (1.2 trillion gigabytes) exist in the “digital universe” • 50% YTY growth • 25% of data is unique; 75% is a copy
Between 2000 and 2010 • servers grew 6x (‘00-’10) • storage grew 69x (‘00-’10) • virtual machines grew 51% CAGR (‘04-’10)
Data centers have doubled their energy use in the past five years • 18% increase in data center energy costs projected
Internet connected devices growing 42% per year
Since 2000 security vulnerabilities grew eightfold
…while IT budgets are growing less than 1% per year. 43
Otázky?
Děkuji za pozornost
[email protected] 44
© 2011 IBM Corporation
