PrimeHub
v4.1
v4.1
  • Introduction
  • Installation
  • Tiers and Licenses
  • End-to-End Tutorial
    • 1 - MLOps Introduction and Scoping the Project
    • 2 - Train and Manage the Model
    • 3 - Compare, Register and Deploy the Model
    • 4 - Build the Web Application
    • 5 - Summary
  • User Guide
    • User Portal
    • Notebook
      • Notebook Tips
      • Advanced Settings
      • PrimeHub Notebook Extension
      • Submit Notebook as Job
    • Jobs
      • Job Artifacts
      • Tutorial
        • (Part1) MNIST classifier training
        • (Part2) MNIST classifier training
        • (Advanced) Use Job Submission to Tune Hyperparameters
        • (Advanced) Model Serving by Seldon
        • Job Artifacts Simple Usecase
    • Models
      • Manage and Deploy Model
      • Model Management Configuration
    • Deployments
      • Pre-packaged servers
        • TensorFlow server
        • PyTorch server
        • SKLearn server
        • Customize Pre-packaged Server
        • Run Pre-packaged Server Locally
      • Package from Language Wrapper
        • Model Image for Python
        • Model Image for R
        • Reusable Base Image
      • Prediction APIs
      • Model URI
      • Tutorial
        • Model by Pre-packaged Server
        • Model by Pre-packaged Server (PHFS)
        • Model by Image built from Language Wrapper
    • Shared Files
    • Datasets
    • Apps
      • Label Studio
      • MATLAB
      • MLflow
      • Streamlit
      • Tutorial
        • Create Your Own App
        • Create an MLflow server
        • Label Dataset by Label Studio
        • Code Server
    • Group Admin
      • Images
      • Settings
    • Generate an PrimeHub API Token
    • Python SDK
    • SSH Server Feature
      • VSCode SSH Notebook Remotely
      • Generate SSH Key Pair
      • Permission Denied
      • Connection Refused
    • Advanced Tutorial
      • Labeling the data
      • Notebook as a Job
      • Custom build the Seldon server
      • PrimeHub SDK/CLI Tools
  • Administrator Guide
    • Admin Portal
      • Create User
      • Create Group
      • Assign Group Admin
      • Create/Plan Instance Type
      • Add InfuseAI Image
      • Add Image
      • Build Image
      • Gitsync Secret for GitHub
      • Pull Secret for GitLab
    • System Settings
    • User Management
    • Group Management
    • Instance Type Management
      • NodeSelector
      • Toleration
    • Image Management
      • Custom Image Guideline
    • Volume Management
      • Upload Server
    • Secret Management
    • App Settings
    • Notebooks Admin
    • Usage Reports
  • Reference
    • Jupyter Images
      • repo2docker image
      • RStudio image
    • InfuseAI Images List
    • Roadmap
  • Developer Guide
    • GitHub
    • Design
      • PrimeHub File System (PHFS)
      • PrimeHub Store
      • Log Persistence
      • PrimeHub Apps
      • Admission
      • Notebook with kernel process
      • JupyterHub
      • Image Builder
      • Volume Upload
      • Job Scheduler
      • Job Submission
      • Job Monitoring
      • Install Helper
      • User Portal
      • Meta Chart
      • PrimeHub Usage
      • Job Artifact
      • PrimeHub Apps
    • Concept
      • Architecture
      • Data Model
      • CRDs
      • GraphQL
      • Persistence Storages
      • Persistence
      • Resources Quota
      • Privilege
    • Configuration
      • How to configure PrimeHub
      • Multiple Jupyter Notebook Kernels
      • Configure SSH Server
      • Configure Job Submission
      • Configure Custom Image Build
      • Configure Model Deployment
      • Setup Self-Signed Certificate for PrimeHub
      • Chart Configuration
      • Configure PrimeHub Store
    • Environment Variables
Powered by GitBook
On this page
  1. Developer Guide
  2. Concept

Data Model

PreviousArchitectureNextCRDs

PrimeHub does not have its database. The data is stored in keycloak's object and kubernetes object. The following diagram is a brief view of the data.

  • Keycloak: We store user and group data. And assign a group to specific roles to bind this group to images, volumes, and instance types.

  • Kubernetes: The definition of image, volume, instance type is stored as CRD (Custom Resource Definition) objects respectively. And the secret is stored by the kubernetes builtin secret object.

Resources

Users

A user in PrimeHub corresponds to a user in Keycloak's primehub realm.

When a user is created via Admin UI, under the hood a user is created in Keycloak via API.

When a user is connected to existing groups via Admin UI, a user is assigned to groups in Keycloak via API.

  • Attributes

    • locale

    • personalDiskQuota, when a user is created, there is a 1G volume allocated by default for the user.

  • Role

    • If a user has admin privileges, it must have realm-admin in realm-management Client Roles.

Groups

A group in PrimeHub represents a group in Keycloak correspondingly.

When a group is created via Admin UI, a group data is created in Keycloak via API.

  • Attributes A group can have attributes that are stored in Keycloak such as canUseCpu, cpuQuota, gpuQuota, diskQuota, displayName and so on.

  • Role Mappings In Keycloak, a group can be assigned Realm Roles, all of the members in a group inherit the same realm roles by default.

  • Members A member who is a user in a group.

A hidden Group from Admin UI, everyone, which every created user must join is created for PrimeHub by default. The purpose is that when the flag Global is enabled in instance type/image/volume via Admin UI, there is a corresponding realm role, <type>:xxx, assigned to the group everyone.

Instance Types / Images / Volumes

When instance type, image or volume is created via Admin UI, there are two things done at the background,

  • A CRD object is created via Kubernetes.

    A CRD object stores settings which are configured via Admin UI such as Basic Info, Toleration and NodeSelector and so on. You can use commands below to list all of the stored CRDs.

      kubectl -n hub get instancetype
      kubectl -n hub get image
      kubectl -n hub get dataset
  • A dedicated realm-role is also created correspondingly in Keycloak.

    A name of Realm Role comes with a prefix which indicates instance type, image, or volume.

    • it:xxx, prefix it represents instance type.

    • img:xxx, prefix img represents image.

    • ds:[rw:]xxx, prefix ds represents volume, rw: read-write, read-only by default. A volume is connected to an existing group via Admin UI it is assigned to a group correspondingly in Keycloak via API.

    For advanced usage, we can add a prefix to these roles. For example, it:cpu-only with prefix cluster1 is cluster1:it:cpu-only. Please refer to helm customization relative document.

    Once a/an instance type/image/volume is connected to existing groups, the dedicated Realm Role is assigned to the groups.

    Therefore, all of the members(a.k.a users) of the group can see it from the menu Server Options of JupyterHub spawner. That also explains the relation between the flag Global and the hidden group everyone..

Secrets

When a secret is created via Admin UI, the settings are stored as a secret in Kubernetes. You can use command below to list them.

kubectl get secret -n hub