Abstract
Besides computation intensive tasks, the Grid also facilitates sharing and processing very large databases and file systems that are distributed over multiple resources and administrative domains. Although accessing data in the Grid is supported by various lower level tools, end-users find it difficult to utilise these solutions directly. High level environments, such as Grid portal and workflow solutions provide little or no support for data access and manipulation. Workflow systems are widely utilised in Grid computing to automate computational tasks. Unfortunately, the ways of feeding data into these workflows is limited and in most cases requires additional tools and manual intervention. This paper describes how data can be fed into computational workflows from heterogeneous data sources. The P-GRADE Grid portal and workflow engine have been integrated with the SDSC Storage Resource Broker (SRB) in order to access SRB data resources as inputs and outputs of workflow components. The solution automates data interaction in computational workflows allowing users to seamlessly access and process data stored in SRB resources. The implemented solution also enables the seamless interoperation of SRB, SRM (Storage Resource Manager) and GridFTP file catalogues.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Mario Antonioletti et. al.: The design and implementation of Grid database services in OGSA-DAI, Concurrency and Computation: Practice and Experience, Volume 17, Issue 2-4, Pages 357 - 376, Special Issue: Grids and Web Services for e-Science, 2005 John Wiley & Sons, Ltd.
Arcot Rajasekar et. al. Storage Resource Broker - Managing Distributed Data in a Grid, Computer Society of India Journal, Special Issue on SAN, Vol. 33, No. 4, pp. 42-54, Oct 2003.
D. Churches et.al: Programming Scientific and Distributed Workflow with Triana Services. Grid Workflow 2004, Concurrency and Computation: Practice and Experience, Vol 18, Issue 10, August 2006, pp 1021-1037, ISSN 1532-0626.
T. Oinn, M. Addis, J. Ferris, D. Marvin, M. Greenwood, T. Carver, M. R. Pocock, A. Wipat and P. Li. Taverna: a tool for the composition and enactment of bioinformatics workflows, Bioinformatics, Vol. 20 no. 17, 2004, pages 3045-3054.
P. Kacsuk and G. Sipos: Multi-Grid, Multi-User Workflows in the P-GRADE Grid Portal, Journal of Grid Computing Vol. 3. No. 3-4., 2005, Springer, 1570-7873, pp 221-238
The UK National Grid Service Website, http://www.ngs.ac.uk/
The EGEE web page, http://public.eu-egee.org/
W. Allcock, J. Bester, J. Bresnahan, A. Chervenak, L. Liming, S. Tuecke: GridFTP: Proto-col Extension to FTP for the Grid, March 2001, http://wwwfp.mcs.anl.gov/dsl/GridFTP-ProtocolRFCDraft.pdf
The Open Science Grid Website, http://www.opensciencegrid.org/
The P-GRADE portal Website, http://www.lpds.sztaki.hu/pgportal/
T. Delaittre, T. Kiss, A. Goyeneche, G. Terstyanszky, S.Winter, P. Kacsuk: GEMLCA: Running Legacy Code Applications as Grid Services, Journal of Grid Computing Vol. 3. No. 1-2. June 2005, Springer Science + Business Media B.V.
T. Delaitre, A.Goyeneche, T.Kiss, G.Z. Terstyanszky, N. Weingarten, P. Maselino, A. Gourgoulis, S.C. Winter: Traffic Simulation in P-Grade as a Grid Service, Conf. Proc. of the DAPSYS 2004 Conference, pp 129-136, ISBN 0-387-23094-7, September 19-22, 2004, Budapest, Hungary.
SRB project homepage http://www.sdsc.edusrbindex.phpMain-Page.
P. Kacsuk, T. Kiss, G. Sipos, Solving the Grid Interoperability Problem by P-GRADE Portal at Workflow Level, Conf. Proc. of the Grid-Enabling Legacy Applications and Supporting End Users Workshop, within the framework of the 15th IEEE International Symposium on High Performance Distributed Computing , HPDC15, Paris, France, pp 3-7, June 19-23, 2006
D. Meredith, M. Maniopoulou, A. Richards, M. Mineter: A JSDL Application Repository and Artefact Sharing Portal for Heterogeneous Grids and the NGS, Proceedings of the UK e-Science All Hands Meeting 2007, Nottingham, UK, 10th-13th September 2007, pp 110-118, ISBN 978-0-9553988-3-4.
A. Sim, A. Soshani editors, Storage Resource Manager Interface Specification version 2.2,09.05.2007, http://www.ogf.orgPublic-Comment-DocsDocuments2007-10OGFGSM-SRMv2.2.pdf.
JasonNovotny, Ramil Manansala, Thien Nguyen: BIRN PortalOverview, Portals & Portlets2006,17-18July2006, Edinburgh, UKhttp://www.nesc.ac.uk/action/esi/download.cfm?index=3246.
The National Center for Microscopy and Imaging Research (NCMIR) - SRB portlet http://ncmir.ucsd.edu/Software/srbportlet.htm.
NGS P-GRADE portal: https://grid-portal.cpc.wmin.ac.uk:8080/gridsphere/gridsphere.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer Science+Business Media, LLC
About this chapter
Cite this chapter
Kiss, T., Tudose, A., Terstyanszky, G., Kacsuk, P., Sipos, G. (2008). Utilizing Heterogeneous Data Sources in Computational Grid Workflows. In: Making Grids Work. Springer, Boston, MA. https://doi.org/10.1007/978-0-387-78448-9_18
Download citation
DOI: https://doi.org/10.1007/978-0-387-78448-9_18
Publisher Name: Springer, Boston, MA
Print ISBN: 978-0-387-78447-2
Online ISBN: 978-0-387-78448-9
eBook Packages: Computer ScienceComputer Science (R0)