ELI: Petawatt nebo Petabyte – kdo z koho? Pavel Bastl, Jaroslav Charfreitag CESNET Community Fórum 2014 - Zpracování a ukládání velkých dat
Local DAQ
Based on rack mount computer with PCIe interface
Amount of PCIe x8 interfaces should be available Optical fiber based NIC for DAQ network At the moment 4x SFP+ 10Gb/s NIC card (40 Gb/s overall throughput) Some SFP+ cages may be connected locally during local control More than one NIC may be inserted when needed and PCIe slots are available Digitizers Other “standard” measurement cards are available on the market Big amount of cheap DDR RAM is available Front access to devices in PCIe slots must be provided
ELI
ELI–BL–4442–PRE–00000000-A
Projekt ELI Extreme Light Infrastructure (ELI)
3 + 1 laser = ELI ELI rovná se tři laserová centra sdružená pod jednou hlavičkou První centrum (ELI Beamlines) bude umístěna v České republice a bude vytvářet novou generaci sekundárních zdrojů pro mezioborové aplikace (fyzika, medicína, biologii, materiálové inženýrství…) Druhé centrum (ELI Attosecond) se staví v Maďarsku a má se věnovat fyzice ultrakrátkých optických pulsů v řádu attosekund. Třetí centrum (ELI Nuclear Physics) orientované na fotonukleární fyziku, by mělo sídlit v Rumunsku. Čtvrté infrastruktury se v současné době jedná.
Projekt ELI – Beamlines Centrum Dolní Břežany První centrum (ELI Beamlines)
Projekt ELI – Plán umístění systémů
1. patro Podpůrné technologie
Přízemí Laserové haly
Podzemí Kompresorová hala Distribuce svazků ve vakuu Specializované experimentální haly (E1 – E6)
Projekt ELI – Technologie a zdroje dat
Obory a oblasti Stavební Strojní Laserové Experimentální Řídící (SCADA, kontrolní systémy) Zpracování dat (DAQ systémy) – největší zdroj Ostatní – HR, finance, evidence, běžný IT provoz
Zpracování dat (DAQ systémy)
DAQ = Data Acquisition data z experimentů, měření, kontrolních systémů řídicích systému Řádově stovky TB dat z jednoho měření !!! ELI = možnost až 6 experimentů současně ELI = 4 laserové dráhy různého typu ELI = zdroj dat v éře „BigData“
Agenda DAQ system
Data acquisition introduction Data acquisition devices Data acquisition server structure Data acquisition online processing Global level data acquisition Local level data acquisition
Data acquisition introduction Instrumentation
Data acquisition - DAQ
Data acquisition introduction Instrumentation
Data acquisition - DAQ
Frame rate up to ~kHz level High volume of data High speed throughput is required Data tame-stamping
Data acquisition structure Data acquisition input devices
Matrix detectors CCD CMOS Scalar detectors Digitizers …
Network based interfaces
GigE Visision 10GigE Visison (formally GigE Vision ver 2.0)
Local bus interfaces
USB 3.0 Metallic cable Optical fiber PCIe CoaXPress CameraLink PXI/PXIe CameraLink TCA
Data acquisition structure Data acquisition input devices
Matrix detectors CCD CMOS Scalar detectors Digitizers …
Network based interfaces
GigE Visision 10GigE Visison (formally GigE Vision ver 2.0)
Local bus interfaces
USB 3.0 Metallic cable Optical fiber PCIe CoaXPress CameraLink PXI/PXIe CameraLink TCA
Global level DAQ structure – hierarchical data storage
Online data view
NFS/CIFS mounts
DAQ server
Hierarchical data storage
High speed disks - Tier 1 Low speed disks - Tier 2 Tape library – Tier 3
DAQ server
Hierarchical data storage
High speed disks - Tier 1 Low speed disks - Tier 2 Tape library – Tier 3
DAQ server
Hierarchical data storage
High speed disks - Tier 1 Low speed disks - Tier 2 Tape library – Tier 3
Multinode DAQ server
Shared DRAM - Tier0
DAQ server
Hierarchical data storage High speed disks - Tier 1 Low speed disks - Tier 2 Tape library – Tier 3
Multinode DAQ server
Shared DRAM - Tier0
Network interface
At minimum 4xSFP+ 40Gb interface
DAQ server – FPGA & Timestamping FPGA online processing
FPGA card in PCIe DDR RAM buffer 10Gb SFP+ interfaces
FPGA IP cores & SW
OpenCL support DDR memory IP core 10Gb ethernet IP core PCIe IP core
DAQ server – FPGA & Timestamping FPGA online processing
FPGA card in PCIe DDR RAM buffer 10Gb SFP+ interfaces
FPGA IP cores & SW
OpenCL support DDR memory IP core 10Gb ethernet IP core PCIe IP core
Time stamping support
Timing system based card Available only for facility level devices PTP support Suitable for user computer or devices
Global level DAQ - interfaces
LC connector
MPO/MTP connector
Full duplex two fibers Standard in data networks and in ELI
Twelve fibers in one connector Used in service HUBs
FC connector
Standard for instrumentation
Local DAQ
Based on rack mount computer with PCIe interface
Amount of PCIe x8 interfaces should be available Optical fiber based NIC for DAQ network At the moment 4x SFP+ 10Gb/s NIC card (40 Gb/s overall throughput) Some SFP+ cages may be connected locally during local control More than one NIC may be inserted when needed and PCIe slots are available Digitizers Other “standard” measurement cards are available on the market Big amount of cheap DDR RAM is available Front access to devices in PCIe slots must be provided
Local DAQ - interfaces
PCIe
PXI/PXIe
Standard interface Data acquisition cards available Optical network is supported USB 3.0 is supported Timing and triggering not supported by the bus NI proprietary interface – open? Data acquisition and general measurement cards available General measurement cards CPU cards are very expensive and have not enough memory for high speed data storage Optical network not supported USB 3.0 interface not supported Backplane supports timing and triggering
μTCA
Advanced telecommunication architecture – open, very flexible, complicated Data acquisition and general cards available Optical network is supported USB 3.0 is supported Backplane supports timing and triggering Becomes to be popular in facilities (DESY, Elettra, SLAC etc.)
Local DAQ - interfaces
Based on rack mount computer with PCIe interface
Amount of PCIe x8 interfaces should be available Optical fiber based NIC for DAQ network At the moment 4x SFP+ 10Gb/s NIC card (40 Gb/s overall throughput) Some SFP+ cages may be connected locally during local control More than one NIC may be inserted when needed and PCIe slots are available Digitizers Other “standard” measurement cards are available on the market Big amount of cheap DDR RAM is available Front access to devices in PCIe slots must be provided
These requirements lead to server based computer
ELI – vlastní datové uložiště (vývoj) Komerční uložiště X Vlastní uložiště cena, rozšiřitelnost, morální udržitelnost, doba dodání -
4U/5U velikost pro montáž do rack skříně, duální napájení Možnosti – klasické disky/SSD disky/kombinace – dle požadavku na IOPs 45 disků 3,5“, hrubá kapacita až 270 TB (450 TB na přelomu 2015/2016) Možnost FC, iSCSI a InfiniBand (FDR/QDR) rozhraní dle potřeb
Institute of Physics ASCR, v. v. i. Na Slovance 2 182 21 Prague 8 Czech Republic
[email protected] www.eli-beams.eu
Děkujeme za pozornost a pozvání sdružení Kontakt: Fyzikální ústav AV ČR v.v.i., Na Slovance 2, 182 21 Praha 8 OC Harfa, Českomoravská 2420/15, 190 00 Praha 9 (do září 2015) Pavel Bastle,
[email protected] Jaroslav Charfreitag,
[email protected]
