PrimeHub Store
Last updated
Last updated
PrimeHub store is the central storage of PrimeHub and its backend is object storage. We use MinIO as the object storage solution. It supports launching as a standalone object store server or a gateway to connect the most popular cloud object storage solutions (e.g. AWS s3, google cloud GCS, azure blob...)
Unlike user volume, group volume, and data volume, which are designed for storing user's data, PrimeHub store is designed for storing both system and user data.
Central Storage for PrimeHub
Supports s3-compatible REST API
Allows pods to mount PrimeHub store through PVC
Does not provide a new volume type to connect to object storage.
Log Persistence: Currently, the job submission log is to retrieve the log from a pod. With the PrimeHub store, we can collect the log and store them in the PrimeHub store. The log can still be accessible even the pod is deleted.
PrimeHub File System (PHFS): new shared storage for groups.
MinIO is the key component of the Primehub Store. We support three persistence backend
MinIO Standalone
MinIO Gateway for S3
MinIO Gateway for GCS
MinIO hides the different implementation for different persistence backend and provides a consistent way for other PrimeHub components to access the PrimeHub store.
GraphQL and other PrimeHub services use MinIO s3-compatible REST API to access the PrimeHub store.
csi-rclone implements Container Storage Interface (CSI) plugin that allows using rclone mount as storage backend. We use it to mount PrimeHub store in the user Pods. It allows users to access PrimeHub store data by the file system.
As the PrimeHub is installed, there is also a csi-rclone provisioned PVC is also created. Usually, the PVC name is primehub-store.
The folder structure of the PrimeHub store is defined as follows.
The top-level folder is PrimeHub store relative features.
