Supporting scientific and Web 2.0 communities by desktop grids P. Kacsuk MTA SZTAKI (Univ. of Westminster) 1 The EDGeS/EDGI projects receive Community research funding
The strength of communities • Web 2.0 services provide frameworks for communities to put together contents • Volunteer computing (VC) is a community activity to provide desktop resources for typically scientific communities -> Desktop Grids (DGs) • Our goal is to bring together the two communities: – Web 2.0 communities can help scientific projects to advertise their projects and attract volunteer donors – Web 2.0 communities can extend the scope of their applications by using VC technology provided for science – Web 2.0 service providers can increase their services without further resource investment
2
Goals of the EDGI and Web2Grid projects • To support both scientific and Web 2.0 communities by desktop grids • EDGI: enables scientific communities to extend the existing service grids (SGs) with DG resources • Web2Grid enables Web 2.0 communities to create large computing infrastructure as community effort 3
Possible options to extend the infrastructure for communities Scientific U1 Community Users
U2
Un
U1
Um
Web 2.0 Community Users
Web 2.0 Community Service
Grid
Faster, Cloud more reliable but more expensive D1
U2
Cloud
DG
D2 Donors
Dj
D1
DG
Less expensive since uses the free cycles of existing PCs
D2 Donors Dk
4
EDGI: Extending the grid infrastructure for scientific communities
EDGeS scope only for compute intensive applications for EGEE (gLite)
EDGI scope for both compute and data intensive applications for EMI/EGI (gLite, ARC, Unicore) Extend Desktop Grids with Clouds for QoS 5
Extending the grid infrastructure for scientific communities U1
U2
Un
Scientific Community Users
Grid
Faster, Cloud more reliable but more expensive D1
Provides reliable donors from dedicated cloud resources
DG
D2 Donors
Dj
Cloud
6
Extending the grid infrastructure for scientific communities U1
U2
Un
Scientific Community Users
Grid
Provides reliable donors from dedicated cloud resources
DG
D1
D2 Donors
Dj
Cloud
7
Steps of the technical solution
The starting state DG Pro‐ ject DG client submit
gLite grid
Volunteer/ Institutional Resources 8
Security issues • SG -> DG direction – In DG projects • Applications are trusted • DG server is trusted • Clients are not trusted
– Therefore any SG application that is intended to run in a DG should be validated to make it trusted – Therefore EDGI provides: • Application validation service • Application repository (AR) where trusted applications are stored • EDGI bridge transfers only those applications to DGs that are stored in the AR
9
Step 1: Providing application repository • metadata templates describe applications and their implementations to support browse and search operations • access without registration for all users to browse and search the repository • API to enable repo access for modified CEs
10 10
Step 2: Submit job from gLite to DG project
3GBridge
down‐ load
DG Pro‐ ject DG client
submit
gLite grid
Volunteer/ Institutional Resources 11
Generic Grid-Grid (3G) Bridge to integrate SGs and DGs
Queue Manager
DC-API Plugin
SG
BOINC
Xtrem Web
GridHandler Interface
WS interface
Job Handler Interface
Job Database
XtremWe b Plugin
3G Bridge
Step 3: Submit job from ARC to DG project ARC grid
submit 3GBridge
down‐ load
DG Pro‐ ject DG client
submit
gLite grid
Volunteer/ Institutional Resources 13
Step 4: Support for Dataintensive applications upload
ARC grid UI
Attic FS
submit
3GBridge
down‐ load
attic
DG Pro‐ ject DG client attic
submit
gLite grid
Volunteer/ Institutional Resources 14
Usage of Attic P2P File System in EDGI
15
Step 5: QoS Support by Clouds upload
ARC grid
submit
UI
OpenNebula/ Amazon
Attic FS
DG client attic
3GBridge
down‐ load
attic
cloud
DG Pro‐ ject DG client attic
submit
gLite grid
Volunteer/ Institutional Resources 16
3G Bridge Extension for supporting Clouds
Host A 3G‐Bridge
Queue 1
Queue 2
… …
… …
1. Scheduler
2. 3.
BOINC Plugin
Cloud Plugin
4.
BOINC Master 5. Amazon/ Eucalyptus Cloud Interface
Cloud Resource N (BOINC Worker)
…
Cloud Resource 2 (BOINC Worker)
Cloud Resource 1 (BOINC Worker)
6.
Job is submitted from MCE to 3G Bridge. 3G Bridge submits the job to a BOINC DG using the BOINC Plugin The Scheduler keeps track of the number of jobs in the BOINC queue (Queue 1) and of the number of the running Cloud Resources (workers). If the BOINC DG is overloaded, the Scheduler starts new workers by submitting a job to the queue of the Cloud Plugin (Queue 2). Each job in Queue 2 launches a new BOINC Worker in the Cloud. If the cluster is underutilized, the Scheduler stops some workers (cloud resources) by sending cancel jobs into Queue 2.
Legend Job Command Information
17
Step 6: Monitoring upload
ARC grid
submit
UI
OpenNebula/ Amazon
Attic FS
DG client attic
3GBridge
down‐ load
attic
DG Pro‐ ject
cloud
monitor
DG client attic
submit
Monitor UI gLite grid
inspect
Volunteer/ Institutional Resources 18
Appl. name
Last day
Last week
Last month
EDGeS@home
19
EDGI business model U1
U2
Un
SC Users can ask QoS support (cloud resources) if they have collected credits
Grid Institutes of SC users can donate DG resources and can collect credits (by BOINC mechanism) Students and public can support universities by offering their credits
EDGI provides reliable donors from dedicated cloud resources
DG
D1
D2 Donors
Dj
Cloud
After EDGI, IDGF members or cloud providers can provide cloud resources 20
EDGI business model U1
U2
Un Universities can create local DGs (e.g. Univ. of Westminster, Univ. of Portsmouth, Univ. of Szeged, etc.)
Grid
University level DG could be connected to the EDGI DG and collect credit for the univ. researchers
DG
DG D1
D2 Donors
Dj
Cloud D1
D2 Donors Dk 21
WEB2GRID Project
Building comm.
Collect resources Existing community appl1 Existing community appl2 Applications Watermark appl. Media appl.
Services for comm.
New community appls
22 22
Facebook interface for watermarking photos
• To prevent the illegal use of unprotected photos • Use of steganographic methods, embedding hidden watermarks into the photos • Protected images can be found by a search-bot • Cryptography (timestamp and digital signature) is used to show the ownership rights 23
Architecture of Web2Grid system
Meg‐ rendelő általi fizetés
Web 2.0 User Web 2.0 felhasználói Interface (e.g. Facebook) interfész (Facebook/Orkut)
WS-PGRADE
Megrendelési modul Order Module
gUSE repository
Result Validator Module Eredmény validátor
Grid Task Handler Web2 alkalmazás Application GRID feladatkezelő
Results of Web2Grid Payment System Abaqoos
Új workflow
Új workflow Workflow állapot Workflow eredmények
Eredmények validálása
Workflow állapot
gUSE middleware
Results of EDGeS and EDGI BOINC BOINC jelentés report and generátor és validátor interface interfész validator
Koordinációs Module modul Coordinator
Accounting Module Elszámoló modul
BOINC szerver
Accounting System Elszámoló rendszer
BOINC alkalmazás
Appl.
Elszámolás munkavégzésről Munkacsomag
RegisztrációsModule modul Registrator
3GBridge
BOINC donor regiszter
Fizető rendszer
Donor munka‐ végzés kifizetése
gUSE workf low
gUSE WS interfész interface
Donor regisztráció
Eredmény
GRID
Boinc Boinckliens kliens Boinc kliens szoftverí0 Boinc kliens szoftver Boinc kliens szoftver BOINC client Boinc kliens szoftver szoftver szoftver 24 software
24
Architecture of Web2Grid system Web 2.0 application Web 2.0 User Web 2.0 felhasználói Interface (e.g. Facebook) interfész (Facebook/Orkut) interfaces
Web 2.0 Application Result Validator Module Eredmény validátor Support GRID feladatkezelő Grid Task Handler Services
WS-PGRADE
Megrendelési modul Order Module
Meg‐ rendelő általi fizetés
Web2 alkalmazás Application
Payment Abaqoos module Payment System Fizető rendszer
gUSE repository
Új workflow Workflow állapot Workflow eredmények
Eredmények validálása Új workflow
Workflow állapot
gUSE middleware
Grid
BOINC BOINC jelentés report and generátor és
validátor interface interfész validator
3GBridge
BOINC donor regiszter Koordinációs Module modul Coordinator
Donor munka‐ végzés kifizetése
gUSE workf low
gUSE WS interfész interface
Accounting Module Elszámoló modul Accounting System Regisztrációs modul Registration Module
Accounting System Elszámoló rendszer
BOINC szerver
BOINC alkalmazás
Appl.
Elszámolás munkavégzésről Munkacsomag Donor regisztráció
Eredmény
GRID
Boinc Boinckliens kliens Boinc kliens szoftverí0 Boinc kliens szoftver Boinc kliens szoftver BOINC client Boinc kliens szoftver szoftver szoftver 25 software
Grid resources
25
Grid applications
Job 1 Job 2
Watermark Workflow Media Workflow
Job 3
…
Micropayment (Abaqoos)
Example Web2Grid application: watermarking photos
3G Bridge
Desktop Grid Watermark App
Client Client
“Credit”
Digital signature
Client
Media App …
Client 26 26
Possible business models • Service provider provides the whole framework • Supplier model: – Service provider invites trusted individuals to connect clients – Pays for the collected credits
• Volunteer model: – Web 2.0 community members provide the clients – They collect credits – Credits are transferred to real money and offered to charity organizations 27
International Desktop Grid Federation (IDGF) Support those NGIs that want to extend their VO with volunteer or local DGs Support those universities, institutes, companies who want to set up local DG Run dissemination campaign to attract more volunteers to donate their computer resources Federation is open for organisations and individuals in research and industry. Current members: >100 individuals > 30 organizations (including companies) 28 Introduction Author: Robert Lovas, Ad Emmen
RI-261561 WP4 version: 3.0
Summary • Volunteer DG resources can be provided for scientific and Web 2.0 communities as an inexpensive alternative to cloud resources • EDGI and Web2Grid projects have developed the required technical solutions • They also show possible business models • Future task: to implement the business models 29
Thank you for your attention … Any questions?
For more information please visit the EDGeS and EDGI Websites: http://www.edges-grid.eu/ http://edgi-project.eu and/or send e-mail to me:
[email protected]
30
