(Advanced) Use Job Submission to Tune Hyperparameters

For typical machine learning algorithms, there are many hyperparameters to tune. Finding the right hyperparameters can be a very time consuming job. However, with Job Submission, you can let it run the job for you. All you have to do is simply check the results when the job is complete.

  1. Let's create another Python file tune_dropout.py in JupyterLab first:

     import tensorflow as tf
     import argparse
     import numpy as np
     parser = argparse.ArgumentParser(description='Process some integers.')
     parser.add_argument('--dropout', type=float, default=0.2)
     args = parser.parse_args()
     mnist = tf.keras.datasets.mnist
     (x_train, y_train),(x_test, y_test) = mnist.load_data()
     x_train, x_test = x_train / 255.0, x_test / 255.0
     model = tf.keras.models.Sequential([
       tf.keras.layers.Flatten(input_shape=(28, 28)),
       tf.keras.layers.Dense(512, activation=tf.nn.relu),
       tf.keras.layers.Dense(10, activation=tf.nn.softmax)
     model.fit(x_train, y_train, epochs=5)
     model.evaluate(x_test, y_test)
     prob = model.predict(x_test)
     prediction = np.argmax(prob, 1)
     equality = np.equal(prediction, y_test)
     accuracy = np.mean(equality)
     filename = 'dropout_{}.txt'.format(args.dropout)
     with open(filename, 'w') as f:
         print(accuracy, file=f)

    The main difference from the last version is that we are now output testing dataset's accuracy into a text file.

    Please note: we are not splitting dataset into training, validation, testing sub-datasets. For convenience, we are only using this dataset to tune the hyperparameters.

  2. Select the Job Submission tab again and select same group, instance type and image. But this time, type the following command; replace <group name>:

     cd /project/<group name>/
     for i in $(seq 1 9)
       dropout=`awk "BEGIN {print $i/10}"`
       echo "dropout rate: "$dropout
       python -u tune_dropout.py --dropout $dropout
       acc=`cat dropout_$dropout.txt`
       echo $acc
       better=`awk "BEGIN { print ($acc > $bestAcc) ? \"YES\" : \"NO\" }"`
       if [ "$better" = "YES" ]
         echo "Current best accuracy is: "$bestAcc
         echo "Current best dropout rate is: "$bestRate
     echo "Best accuracy is: "$bestAcc
     echo "Best dropout rate is: "$bestRate

    Here we write shell script loops through a 0.1 to 0.9 dropout rate. At the end of this command, the best accuracy and dropout rate is printed out.

    If you prefer, rather than writing in a shell script, you can create another Python file and wrap the original code as a function and write a for loop in Python. We can submit this new Python file as a job to find out best dropout rate.

  3. After your job has succeeded, you can view the best dropout rate for your model.

    Your results may not match this image below

In the future, we will provide even better hyperparameters tuning functions. Stay tuned!

Last updated