Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update notebooks with settings file #99

Closed
wants to merge 2 commits into from
Closed

Update notebooks with settings file #99

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
79f6c5b
Update notebooks with settings file
mehrinkiani Feb 6, 2024
e9542e0
Merge branch 'main' into update-notebooks-settingsfile
mehrinkiani Feb 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 85 additions & 35 deletions notebooks/keras_fashion_mnist.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"modelscan, version 0.5.0\n"
"modelscan, version 0.5.1\n"
]
}
],
Expand Down Expand Up @@ -48,6 +48,37 @@
"from utils.tensorflow_fashion_mnist_model import train_model, get_predictions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Optional Settings File for ModelScan\n",
"\n",
"ModelScan scan settings can be configured using a settings file. \n",
"\n",
"- To create a configurable settings file use: `modelscan create-settings-file` (creates a `modelscan-settings.toml` file in current directory). \n",
"\n",
"- The location and name of a settings file can also be specified with `modelscan create-settings-file -l ../path-to/settings-file.toml` \n",
"\n",
"Configurations:\n",
"- A settings file would allow enabling/disabling particular scans such as H5LambdaDetectScan, and PickleUnsafeOpScan. \n",
"\n",
"- A settings file can also be configured to set particular severity level (CRITICAL, HIGH, MEDIUM, or LOW) for unsafe operators. \n",
"\n",
"- Reporting format for ModelScan results can also be specified in a settings file.\n",
"\n",
"If you would like to create a configurable settings file, uncomment the code in the next cell and run it. "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"#!modelscan create-settings-file -l my-modelscan-settings.toml"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand All @@ -59,43 +90,43 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"1875/1875 [==============================] - 2s 1ms/step - loss: 0.4986 - accuracy: 0.8258\n",
"1875/1875 [==============================] - 1s 726us/step - loss: 0.4961 - accuracy: 0.8248\n",
"Epoch 2/10\n",
"1875/1875 [==============================] - 2s 800us/step - loss: 0.3741 - accuracy: 0.8649\n",
"1875/1875 [==============================] - 1s 689us/step - loss: 0.3736 - accuracy: 0.8641\n",
"Epoch 3/10\n",
"1875/1875 [==============================] - 1s 772us/step - loss: 0.3391 - accuracy: 0.8766\n",
"1875/1875 [==============================] - 1s 718us/step - loss: 0.3351 - accuracy: 0.8782\n",
"Epoch 4/10\n",
"1875/1875 [==============================] - 1s 755us/step - loss: 0.3125 - accuracy: 0.8852\n",
"1875/1875 [==============================] - 1s 716us/step - loss: 0.3147 - accuracy: 0.8829\n",
"Epoch 5/10\n",
"1875/1875 [==============================] - 2s 965us/step - loss: 0.2955 - accuracy: 0.8908\n",
"1875/1875 [==============================] - 1s 713us/step - loss: 0.2960 - accuracy: 0.8908\n",
"Epoch 6/10\n",
"1875/1875 [==============================] - 2s 905us/step - loss: 0.2808 - accuracy: 0.8963\n",
"1875/1875 [==============================] - 1s 715us/step - loss: 0.2797 - accuracy: 0.8963\n",
"Epoch 7/10\n",
"1875/1875 [==============================] - 2s 1ms/step - loss: 0.2696 - accuracy: 0.9004\n",
"1875/1875 [==============================] - 1s 715us/step - loss: 0.2672 - accuracy: 0.9008\n",
"Epoch 8/10\n",
"1875/1875 [==============================] - 2s 895us/step - loss: 0.2587 - accuracy: 0.9039\n",
"1875/1875 [==============================] - 1s 723us/step - loss: 0.2580 - accuracy: 0.9044\n",
"Epoch 9/10\n",
"1875/1875 [==============================] - 2s 925us/step - loss: 0.2479 - accuracy: 0.9068\n",
"1875/1875 [==============================] - 1s 733us/step - loss: 0.2470 - accuracy: 0.9067\n",
"Epoch 10/10\n",
"1875/1875 [==============================] - 2s 893us/step - loss: 0.2406 - accuracy: 0.9112\n",
"313/313 [==============================] - 0s 555us/step - loss: 0.3496 - accuracy: 0.8790\n",
"1875/1875 [==============================] - 1s 700us/step - loss: 0.2406 - accuracy: 0.9097\n",
"313/313 [==============================] - 0s 480us/step - loss: 0.3465 - accuracy: 0.8782\n",
"\n",
"Model trained with test accuracy: 0.8790000081062317\n"
"Model trained with test accuracy: 0.8781999945640564\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/mehrinkiani/mambaforge/envs/py310/lib/python3.10/site-packages/keras/src/engine/training.py:3000: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.\n",
"/Users/mehrinkiani/mambaforge/envs/modelscan/lib/python3.10/site-packages/keras/src/engine/training.py:3000: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.\n",
" saving_api.save_model(\n"
]
}
Expand All @@ -119,16 +150,16 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1/1 [==============================] - 0s 42ms/step\n",
"1/1 [==============================] - 0s 34ms/step\n",
"\n",
"The model predicts: ['Ankle boot', 'Pullover', 'Trouser'] with probabilities: [ 98.948 99.926 100. ]\n",
"The model predicts: ['Ankle boot', 'Pullover', 'Trouser'] with probabilities: [ 95.851 99.163 100. ]\n",
"\n",
"The true labels are ['Ankle boot', 'Pullover', 'Trouser']\n"
]
Expand All @@ -153,15 +184,22 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Scan the Safe Model\n",
"# Scan Safe Model\n",
"\n",
"\n",
"The scan results include information on the files scanned, and any issues if found. For the safe model scanned, modelscan finds no model serialization attacks, as expected.\n",
"\n",
"### ModelScan Settings File\n",
"- If you have created a settings file with default name and location (`modelscan-settings.toml`) it would automatically be used when scanning a model. \n",
"\n",
"- If you want to use a specific name and/or location of settings file, please specify it with `--settings-file` as outlined in the next cell. \n",
"\n",
"The scan results include information on the files scanned, and any issues if found. For the safe model scanned, modelscan finds no model serialization attacks, as expected."
"- If you have not created any settings file, ModelScan will scan using default settings. "
]
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 7,
"metadata": {},
"outputs": [
{
Expand All @@ -170,15 +208,19 @@
"text": [
"No settings file detected at /Users/mehrinkiani/Documents/modelscan/notebooks/modelscan-settings.toml. Using defaults. \n",
"\n",
"Scanning /Users/mehrinkiani/Documents/modelscan/notebooks/KerasModels/safe_model.h5 using modelscan.scanners.H5LambdaDetectScan model scan\n",
"\n",
"\u001b[34m--- Summary ---\u001b[0m\n",
"\n",
"\u001b[32m No issues found! 🎉\u001b[0m\n"
"\u001b[32m No issues found! 🎉\u001b[0m\n",
"\n",
"\u001b[34m--- Skipped --- \u001b[0m\n",
"\n",
"Total skipped: \u001b[1;36m1\u001b[0m - run with --show-skipped to see the full list.\n"
]
}
],
"source": [
"#!modelscan -p ./KerasModels/safe_model.h5 --settings-file my-modelscan-settings.toml\n",
"!modelscan -p ./KerasModels/safe_model.h5"
]
},
Expand All @@ -193,7 +235,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 8,
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -222,7 +264,7 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -244,7 +286,7 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 10,
"metadata": {},
"outputs": [
{
Expand All @@ -255,9 +297,9 @@
"aws_secret_access_key=<aws_secret_key>\n",
"aws_access_key_id=<access_key_id>\n",
"aws_secret_access_key=<aws_secret_key>\n",
"1/1 [==============================] - 0s 35ms/step\n",
"1/1 [==============================] - 0s 26ms/step\n",
"\n",
"The model predicts: ['Ankle boot', 'Pullover', 'Trouser'] with probabilities: [ 98.948 99.926 100. ]\n",
"The model predicts: ['Ankle boot', 'Pullover', 'Trouser'] with probabilities: [ 95.851 99.163 100. ]\n",
"\n",
"The true labels are ['Ankle boot', 'Pullover', 'Trouser']\n"
]
Expand All @@ -284,16 +326,23 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Scan the Unsafe Model\n",
"# Scan Unsafe Model\n",
"\n",
"The scan results include information on the files scanned, and any issues if found. In this case, a critical severity level issue is found in the unsafe model scanned. \n",
"\n",
"modelscan also outlines the found operator(s) and module(s) deemed unsafe. "
"ModelScan also outlines the found operator(s) and module(s) deemed unsafe. \n",
"\n",
"### ModelScan Settings File\n",
"- If you have created a settings file with default name and location (`modelscan-settings.toml`) it would automatically be used when scanning a model. \n",
"\n",
"- If you want to use a specific name and/or location of settings file, please specify it with `--settings-file` as outlined in the next cell. \n",
"\n",
"- If you have not created any settings file, ModelScan will scan using default settings."
]
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 11,
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -327,6 +376,7 @@
}
],
"source": [
"#!modelscan -p KerasModels/unsafe_model.h5 --settings-file my-modelscan-settings.toml\n",
"!modelscan -p KerasModels/unsafe_model.h5"
]
},
Expand All @@ -349,7 +399,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 12,
"metadata": {},
"outputs": [
{
Expand All @@ -359,7 +409,7 @@
"No settings file detected at /Users/mehrinkiani/Documents/modelscan/notebooks/modelscan-settings.toml. Using defaults. \n",
"\n",
"Scanning /Users/mehrinkiani/Documents/modelscan/notebooks/KerasModels/unsafe_model.h5 using modelscan.scanners.H5LambdaDetectScan model scan\n",
"\u001b[1m{\u001b[0m\u001b[32m\"modelscan_version\"\u001b[0m: \u001b[32m\"0.5.0\"\u001b[0m, \u001b[32m\"timestamp\"\u001b[0m: \u001b[32m\"2024-01-25T17:56:36.776966\"\u001b[0m, \n",
"\u001b[1m{\u001b[0m\u001b[32m\"modelscan_version\"\u001b[0m: \u001b[32m\"0.5.1\"\u001b[0m, \u001b[32m\"timestamp\"\u001b[0m: \u001b[32m\"2024-02-06T11:00:43.503049\"\u001b[0m, \n",
"\u001b[32m\"input_path\"\u001b[0m: \n",
"\u001b[32m\"/Users/mehrinkiani/Documents/modelscan/notebooks/KerasModels/unsafe_model.h5\"\u001b[0m, \n",
"\u001b[32m\"total_issues\"\u001b[0m: \u001b[1;36m1\u001b[0m, \u001b[32m\"summary\"\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m\"total_issues_by_severity\"\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m\"LOW\"\u001b[0m: \u001b[1;36m0\u001b[0m, \u001b[32m\"MEDIUM\"\u001b[0m: \n",
Expand Down Expand Up @@ -389,7 +439,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.10.13 ('py310')",
"display_name": "Python 3.10.13 ('modelscan')",
"language": "python",
"name": "python3"
},
Expand All @@ -408,7 +458,7 @@
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "bd638e2064d9001d4ca93bc8e56e039dad230900dd235e8a6196f1614960903a"
"hash": "4dc76e75b8580a33bdc5a5a988b23a4143f70625528e602e356063408d11834c"
}
}
},
Expand Down
Loading
Loading