Документ взят из кэша поисковой машины. Адрес оригинального документа : http://acat02.sinp.msu.ru/presentations/gasthuber/presentation.tgif.pdf Дата изменения: Thu Jun 27 00:39:32 2002 Дата индексирования: Mon Oct 1 20:34:44 2012 Кодировка:

Providing GRID Data Services TODAY
or: connect an existing data service fabric to a GRID

DESY

Patrick Fuhrmann, Martin Gasthuber Rich Wellner

DESY/Hamburg Fermi

DCache to GRID, DESY/Fermi

ACAT 2002 June 24-28


Initial Remarks

- We only talk about the data management, data access aspect of GRID computing - the 'Fabric' here is an existing data management (services) fabric running @ DESY and Fermi - We are not in deep touch to any of the current (HEP) GRID projects - Fermi has (active) contacts to Globus @ Argonne

- Goal connect existing (data) fabric to applications (or vice versa) through a GRID middleware

DCache to GRID, DESY/Fermi

ACAT 2002 June 24-28


the MAP

GRID enabled Application
high activity area

GRID middleware
low activity area

Fabric

DCache to GRID, DESY/Fermi

ACAT 2002 June 24-28


The existing data services - what is it

- DESY and Fermi use the same architecture and mostly the same components - for the nameservice: PNFS - for the tape service: Enstore (Fermi), OSM (Desy) - for the disk caching: dCache (collaboration DESY/Fermi) - Providing - single uniform (and scalable) namespace for all files in the system - independent of file location and multiplicity - use NFS v2 protocol to access - efficient, scalable internal architecture - simple, scalable tape services - direct access possible - distributed disk cache - dramatic reduction of tape load - direct - random access from application - using shared lib preload - flexible management / data flow steering scale in admin - policy based data placement

DCache to GRID, DESY/Fermi

ACAT 2002 June 24-28


Perfectly Normal FS Perfectly Normal FS * supports 8 layers per file entry. - Top layer appears as regular File Entry but denies I/O - Others can are hidden but can be accessed by 'spooky' filenames * supports inheritable directory tags. * supports wormholes. * uses eventhandler for 'remove' and 'move'. * is unaware of HSM or dCache.
DCache to GRID, DESY/Fermi ACAT 2002 June 24-28


dCache (1) - Failsafe - reconnect, re-stage - hidden to application - Data placement policy expressed by: - network topology (source, destination) include masks - storage group (set of physical tape media) - PNFS directory tag - Costs (CPU Load & (free) Space) - Sticky bit - pin files on cache pools - Thread safe access library (.a and .so) - Rules for pool selection - always fallbacks available - Secure/delegated management and administration - ACLs - Kerberos based authentication - ssh secured login (i.e. just use the ssh application) - WEB monitoring - ROOT interface available (initial version) - room for performance enhancements
DCache to GRID, DESY/Fermi ACAT 2002 June 24-28


dCache (2) - Pool to Pool transfers - immediate read after write (file not on tape yet) - file replication on different pools - Authenticated/secure data access (control line) - Kerberos 5 (GSSAPI) - SSL - Configurable load limits - # of movers (active transfers to/from client) - # of HSM stores/restores - Pluggable Protocol Engines - HSM -> Enstore, OSM, ... - Client -> dCap, FTP, ... (GRID access protocols) - URL based access - no NFS mount required
- dc_open("dcap://dcachedoor.desy.de/pnfs.desy.de/zeus/...", O_READ)

- 100% made of Java (same code on Linux,IRIX,Solaris) - 100% free of experiment code (real generic)
DCache to GRID, DESY/Fermi ACAT 2002 June 24-28


Simple Picture

PNFS

DCache

Enstore OSM

NFS v2

DCap (TCP) simple TCP

Application
DCache to GRID, DESY/Fermi ACAT 2002 June 24-28


DCache to GRID, DESY/Fermi

ACAT 2002 June 24-28


DCache to GRID, DESY/Fermi

ACAT 2002 June 24-28


Redundancy

Write Pool

Read Pool

GB Backbone Giga Bit Switch Switch Fast Ether

Door I
DCache to GRID, DESY/Fermi

Door II
ACAT 2002 June 24-28


The dCap Library

Application Posix open/read/write/close applications libdCache.so libc.so native FS
Data
Namespace Operations

d C a c h e System Door Node

Mover Node Mover Node Mover Node NFS (2) Node

DCache to GRID, DESY/Fermi

ACAT 2002 June 24-28


externally enforced attraction destination determined attraction Super Cluster Caches
Super Cache CentralCache
(Experiment)

Cluster Caches
(WorkingGroup)

Tertiary Storage

OSM Enstore Eurostore

Topic Caches
DCache to GRID, DESY/Fermi

Host Cache
ACAT 2002 June 24-28


dCache
Application

I/O Door

PnfsMgr

Pool Pool

Pool Pool Pool

PoolMgr ssh Client

Administration
AclMgr KeyMgr

AdminDoor

Netscape

http:www-dCache.desy.de-admin

Security
User Admin Users Groups Pool Admin

dCache Wizard Administration

Monitoring
SnmpMgr HttpMgr

DCache to GRID, DESY/Fermi

ACAT 2002 June 24-28


DCache to GRID, DESY/Fermi

ACAT 2002 June 24-28


DCache to GRID, DESY/Fermi

ACAT 2002 June 24-28




And now - the GRID comes along - Fermi require data exchange with Institutes in the UK - already have Kerberized FTP access - What interface to implement ?? - beside GridFTP nothing else common (as we know) - Within our community we know about: 1. GridFTP (invented by Globus) 2. SRM (invented by LBL, JLab, Fermi) - GridFTP (with parallel streams) implemented - includes Kerberos authentification - GSI (certificate based) in work

DCache to GRID, DESY/Fermi

ACAT 2002 June 24-28


SRM - Storage Resource Manager - SRM v1 implementation in progress - Pre-Allocation (semi persistent) requires more work - hope for less internal changes in dCache - Demo for next GGF scheduled - JLab has done already for JASMine - LBL has done already for HPSS - SRM v1 noticed by Globus - SRM v2 definition nearly completed - by LBL, JLab, Fermi, CERN-EDG - We still have a few concerns - lets see - see http://sdm.lbl.gov/srm/documents/joint.docs/
SRM.v2.0.joint.func.design.doc

DCache to GRID, DESY/Fermi

ACAT 2002 June 24-28


GridFTP - GridFTP (with parallel streams) implemented - includes Kerberos authentification - GSI (certificate based) in work - major problem with FTP port management - need changes for clean/scalable implementation - GridFTP v2 in preparation (too long) - maybe intermitten v1.5 definition - joint effort with Globus required - Demo at next GGF scheduled

DCache to GRID, DESY/Fermi

ACAT 2002 June 24-28


Conclusions - need real running experience (SRM) - to get confident in SRM completeness - optimize implementation (on both sides) - GridFTP needs some 'adjustments' - More (concurrent) interfaces might help - before its too late

http://www-dcache.desy.de http://www-pnfs.desy.de http://http://www-hppc.fnal.gov/enstore
DCache to GRID, DESY/Fermi ACAT 2002 June 24-28