WP24: JRA2 – Demonstrator of a Photon Science Analysis Service (DaaS)

Objectives

Building up demonstrators for remote data analysis for a small number of archetypal experiments. The demonstrators will build on the HPC platforms of each participating institute. The demonstrators will be cloud based in those institutes where cloud technology is deployed. In the other institutes the demonstrator will run on standard HPC hardware. A web portal will ensure users to have a common user experience.
Assembly of integrated and deployable applications to reduce both the Research Infrastructure efforts as well as accelerate the scientific process. Applications will be implemented as deployable packages, as pre-configured virtual machines or as containers.
For remote data analysis, particularly important for experiments with very high data volumes, the system will provide access through a user portal implementing a standard authentication system (building on the UMBRELLA system realised through the PaNdata and CRISP FP7 projects) for secure access. The Umbrella attribute authority will be extended to allow users to add required attributes in a self-service way.
Solving use cases for both industrial and non-industrial experiments. Testbeds will be developed for solving the issues users face in exporting large data sets and obtaining access to the necessary CPU resources and appropriate software.
How cloud infrastructures and/or supercomputing centers can be exploited for these use cases will be studied. DESY and ESRF are official “end users” in the HNScienceCloud project contributing photon science specific use case, thereby validating the compatibility of generic cloud services and specific application framework deployment.
Profiling of applications as part of the provenance data furthermore permits to provide users with cost estimates and means for users to select cloud costs efficiently, thereby lowering the barriers to use of Research Infrastructures for researchers across Europe.
Strong linked to a number of other WP in CALIPSOplus:
- WP2 – NA1 – User tools for access and data management
- WP4 – N3 – European Light Sources for Industrial Innovation plus (ELSIIplus) working together on industrial innovation
- WP5 – NA4 – Striving for Sustainability of Photon Science in Europe especially ESUO to get feedback from different user communities

Tasks

Task #	Task name	Task leader
Task 24.1	Design a platform which satisfies the requirements for Remote Data Analysis as a Service. The design will take into account the current situation at the partner institutes. Study approaches which increase re-usability of solutions like component based software engineering and development of scientific/algorithmic libraries. Evaluate different container concepts as deployment strategies, like classical installation procedures, shipping of virtual machines and/or Docker containers.
Task 24.2	Collect and compare offline data analysis requirements and existing software solutions from users of at least three different sites. Analyse which of these use cases and/or solutions are suitable candidates for providing harmonised solutions which can be transferred from one site to another site. This is done for the use case domains selected i.e. SAXS, tomography, diffraction (including MX).
Task 24.3	Implementation DaaS on the local platform at each site with remote access to compute and storage resources using dedicated hardware resources supplied by each site.
Task 24.4	Design and implementation of Remote Data Access Analysis as a Service portal demonstrator
Task 24.5	Package and deploy data analysis packages for at least two use cases / site
Task 24.6	Extend and deploy the Umbrella authentication system as the standard authentication system mechanism to fully support different non-web-based authentication mechanisms to be able to service a broader range of facilities (eduGain + JISC Assent) with differing needs and capabilities.
Task 24.7	Test data analysis software for Use Cases with test data and users