PrimeHub
v4.1
v4.1
  • Introduction
  • Installation
  • Tiers and Licenses
  • End-to-End Tutorial
    • 1 - MLOps Introduction and Scoping the Project
    • 2 - Train and Manage the Model
    • 3 - Compare, Register and Deploy the Model
    • 4 - Build the Web Application
    • 5 - Summary
  • User Guide
    • User Portal
    • Notebook
      • Notebook Tips
      • Advanced Settings
      • PrimeHub Notebook Extension
      • Submit Notebook as Job
    • Jobs
      • Job Artifacts
      • Tutorial
        • (Part1) MNIST classifier training
        • (Part2) MNIST classifier training
        • (Advanced) Use Job Submission to Tune Hyperparameters
        • (Advanced) Model Serving by Seldon
        • Job Artifacts Simple Usecase
    • Models
      • Manage and Deploy Model
      • Model Management Configuration
    • Deployments
      • Pre-packaged servers
        • TensorFlow server
        • PyTorch server
        • SKLearn server
        • Customize Pre-packaged Server
        • Run Pre-packaged Server Locally
      • Package from Language Wrapper
        • Model Image for Python
        • Model Image for R
        • Reusable Base Image
      • Prediction APIs
      • Model URI
      • Tutorial
        • Model by Pre-packaged Server
        • Model by Pre-packaged Server (PHFS)
        • Model by Image built from Language Wrapper
    • Shared Files
    • Datasets
    • Apps
      • Label Studio
      • MATLAB
      • MLflow
      • Streamlit
      • Tutorial
        • Create Your Own App
        • Create an MLflow server
        • Label Dataset by Label Studio
        • Code Server
    • Group Admin
      • Images
      • Settings
    • Generate an PrimeHub API Token
    • Python SDK
    • SSH Server Feature
      • VSCode SSH Notebook Remotely
      • Generate SSH Key Pair
      • Permission Denied
      • Connection Refused
    • Advanced Tutorial
      • Labeling the data
      • Notebook as a Job
      • Custom build the Seldon server
      • PrimeHub SDK/CLI Tools
  • Administrator Guide
    • Admin Portal
      • Create User
      • Create Group
      • Assign Group Admin
      • Create/Plan Instance Type
      • Add InfuseAI Image
      • Add Image
      • Build Image
      • Gitsync Secret for GitHub
      • Pull Secret for GitLab
    • System Settings
    • User Management
    • Group Management
    • Instance Type Management
      • NodeSelector
      • Toleration
    • Image Management
      • Custom Image Guideline
    • Volume Management
      • Upload Server
    • Secret Management
    • App Settings
    • Notebooks Admin
    • Usage Reports
  • Reference
    • Jupyter Images
      • repo2docker image
      • RStudio image
    • InfuseAI Images List
    • Roadmap
  • Developer Guide
    • GitHub
    • Design
      • PrimeHub File System (PHFS)
      • PrimeHub Store
      • Log Persistence
      • PrimeHub Apps
      • Admission
      • Notebook with kernel process
      • JupyterHub
      • Image Builder
      • Volume Upload
      • Job Scheduler
      • Job Submission
      • Job Monitoring
      • Install Helper
      • User Portal
      • Meta Chart
      • PrimeHub Usage
      • Job Artifact
      • PrimeHub Apps
    • Concept
      • Architecture
      • Data Model
      • CRDs
      • GraphQL
      • Persistence Storages
      • Persistence
      • Resources Quota
      • Privilege
    • Configuration
      • How to configure PrimeHub
      • Multiple Jupyter Notebook Kernels
      • Configure SSH Server
      • Configure Job Submission
      • Configure Custom Image Build
      • Configure Model Deployment
      • Setup Self-Signed Certificate for PrimeHub
      • Chart Configuration
      • Configure PrimeHub Store
    • Environment Variables
Powered by GitBook
On this page
  • Prerequisites
  • User Journey
  • Design
  1. Developer Guide
  2. Design

Job Artifact

Allows users to store the job output and can be downloaded from the job UI.

Prerequisites

PrimeHub store and PHFS should be enabled

User Journey

Create artfiacts from job

  1. Create a job with the command

    mkdir -p artifacts/sub
    echo "hello" > artifacts/test.txt
    echo "hello" > artifacts/sub/test.txt
  2. Go to the newly created job detail page.

  3. Wait for the job completed

  4. Go to the Artifacts tab, and we will see all the two artifacts we just created

Artifact retention

  1. After the job completed for 7 days (the default artifact retention)

  2. Go to the job detail page.

  3. Go to the Artifacts tab

  4. We will see no artifacts anymore. The artifacts are automatically deleted.

Design

Artifact Copy

  1. Use a new run-job.sh script to run the job pod

  2. In this script, it will run the command and then copy the files from /home/jovyan/artifacts to /phfs/jobArtifacts/<jobname>/

  3. At the same time, check if the artifacts folder execced the maximum size and maximum file counts

  4. The phjob resource in GraphQL API provides a new artifact field. The resolver would iterate all the files under /groups/<group>/jobArtifacts/<jobname>

Artifact Retention

  1. In the run-job.sh script, store the expiration time to /phfs/jobArtifacts/<jobname>/.primehub/metadata/expiredAt

  2. In the GraphQL, run the cleanup script periodically (per day)

    for group in store groups folder
     for job in `groups/${group}/jobArtifacts`
         expiredAt = get(`${job}/.metadata/expiredAt`)
         if (now > expiredAt)
           removeAll(job)
  3. In the GraphQL, Provide an additional GraphQL mutation to run the cleanup

Components

Controller

PhJob Reconciliation

  1. Use the configmap to mount the run-job.sh script

  2. Add new env vars to the job pod for artifact settings

  3. Run the job by the new run-job.sh script

Artifact copy script

  1. User copy the artifacts to /home/jovyan/artifacts (or create symbolic link)

  2. Once the job is completed (or failed), the script will copy files from /home/jovyan/artifacts (can be symbolic link or a real folder) to /phfs/jobArtifacts/<jobname>/

  3. The script also checks if the file size exceeds the limit.

  4. The script also stores expiration time (UNIX time in second) at /phfs/jobArtifacts/<jobname>/.primehub/metadata/expiredAt

GraphQL

  1. In the original phjob type, add artifact field

    query {
      phJob(where: {id: "job-202009250153-34s0ti"}) {
        id
        artifact {
          prefix
          items {
            name
            size
            lastModified
          }
        }
      }
    }
  2. Delete outdated artifacts folder manually

  3. Provide GraphQL API to delete outdated artifacts folder manually

    mutation {
      cleanupPhJobArtifact
    }

Client

  1. Add additional artifact tab in the job detail page.

  2. The tab is only enabled when the job is completed.

  3. When the tab is clicked, query the GraphQL artifact resource.

  4. Show the artifact list. It should include the path and file size.

  5. Click the path to download the artifact.

PreviousPrimeHub UsageNextPrimeHub Apps