We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
Supporting dynamic pipeline changes using Class-Based Object Versioning in Astro-WISE.
- Authors
Mwebaze, Johnson; Boxhoorn, Danny; Rai, Idris; Valentijn, Edwin
- Abstract
Understanding the difference between data objects is a major problem especially in a scientific collaboration which allows scientists to collectively reuse data, modify and adapt scripts developed by their peers to process data while publishing the results to a centralized data store. Although data provenance has been significantly studied to address the origins of a data item, it does not however address changes made to the source code. Systems often appear as a large number of modules each containing hundreds of lines of code. It is, in general, not obvious which parts of source code contributed to the change in data object. The paper introduces the Class-Based Object Versioning framework, which overcomes some of the shortcomings of popular versioning systems (e.g. CVS, SVN) in maintaining data and code provenance information in scientific computing environments. The framework automatically identifies and captures useful fine-grained changes in the data and code of scripts that perform scientific experiments so that important information about intermediate stages (i.e. unrecorded changes in experiment parameters and procedures) can be identified and analyzed. The benefits of such a system include querying specific methods and code attributes for data items of interest, finding missing gaps of data lineage and implicit storage of intermediate data.
- Subjects
PIPELINE computers; INFORMATION storage &; retrieval systems; COOPERATIVE research; ASTRONOMY experiments; DATA mining; ASTRONOMY databases; ELECTRONIC data processing; ASTRONOMY
- Publication
Experimental Astronomy, 2013, Vol 35, Issue 1/2, p157
- ISSN
0922-6435
- Publication type
Article
- DOI
10.1007/s10686-011-9270-1