Model by Pre-packaged Server (PHFS)
Last updated
Last updated
In this tutorial, we will show how to deploy a model trained from Jupyter Notebook via PHFS storage. We deploy a MNIST model by TensorFlow2 pre-packaged server.
Remember to enable model deployment in your group, contact your admin if it is not enabled yet.
Remember to enable PHFS Storage, contact your admin if it is not enabled yet.
PHFS, currently, supports writing files sequentially only; within this limitation, writing model files in HDF5 format directly into PHFS will cause the error, Problems closing file (file write failed: ...) since HDF5 uses seek while writing.
In this case, we suggest this step: writing HDF5 files into user home directory directly rather than PHFS, then copying files to PHFS for the preparation of model deployments
Go to Notebooks.
Then we are in spawner page, here we choose the instance type with configuration (CPU: 1 / Memory: 2 G / GPU: 0).
Also, we choosing the TensorFlow 2.2 as our runtime image and clicking on Start Notebook button, e.g. infuseai/docker-stacks:tensorflow-notebook-v2-2-1-a7f9696a
Once the notebook is started, running this example notebook to train and save a simple MNIST model.
If we want to deploy a model trained from Notebook, just move it to the phfs directory.
Alternatively, you can upload files to the phfs directory using the Shared Files feature in the User Portal:
Then, back to User Portal and select Deployments.
In the model deployment list page, now clicking on Create Deployment button.
Fill in the Deployment name field with quickstart-mnist
Select the Model Image field with TensorFlow2 server; This is a pre-packaged model server image that can serve TensorFlow 2 model.
Fill in the Model URI field with phfs:///TF-MNIST-Model; This path is included the trained model in the PHFS Storage.
In the Resources,
choose the instance type, here we use the one with configuration (CPU: 0.5 / Memory: 1 G / GPU: 0)
leave Replicas as default (1)
Click on Deploy button, then we will be redirected to model deployment list page. Wait for a while and click on Refresh button to check our model is deployed or not.
When the deployment is deployed successfully, we can click on cell to check its detail.
We can view some detailed information in detail page, now let's test our deployed model! Copy the endpoint URL and replace the ${YOUR_ENDPOINT_URL} in the following block.
Then copy the entire block to the terminal for execution. This Curl example is from PrimeHub model deployment example, and we are sending ndarray as request data.
Example of request data
Example of response data (it predicts the number is 7 as the highest probability is in index-7)
Congratulations! We have trained a model in Notebook and directly deployed it as an endpoint service that can respond requests anytime from everywhere.
(Advanced) We went through a simple MNIST example by sending ndarray data to the deployed model. Next, we can also try to send an exact image file to the deployed model.
Follow previous tutorial steps but with Model Image to be infuseai/tensorflow2-prepackaged_rest:v0.4.3.
(starting from v0.4.3, the TensorFlow2 model server image hosted by InfuseAI can handle the image file input)
Using this Curl example,
Example of request data
Example of response data
For the completed model deployment feature introduction, see Model Deployment.
For the customized pre-packaged server instruction, see Pre-packaged servers.
