Data Jockey - Multi-tiered Storage Systems for HPC centers (2016 - 2020)

Overview

Investigated the design and implementation of data movement and management middle-ware for multi-tiered storage systems seen in HPC centers.

• Developed abstractions, algorithms and techniques to help manage data location and movement across multiple tiers of storage within a HPC data center
• Designed and implemented a prototype middle-ware called Data Jockey
• Published details of the design, implementation, and findings in IPDPS'19

Technology

• Python - Django REST framework, Celery, Websocket,
• RabbitMQ, MariaDB
• OLCF - Torque Moab, RHEA cluster, HPSS, Lustre

Publications

• Shin et al., “Data Jockey: Automatic Data Management for HPC Multi-tiered Storage Systems”, IPDPS'19