diff --git a/network-intrusion-detection-part-1/meta.toml b/network-intrusion-detection-part-1/meta.toml
deleted file mode 100644
index d5acf21c..00000000
--- a/network-intrusion-detection-part-1/meta.toml
+++ /dev/null
@@ -1,8 +0,0 @@
-[meta]
-title="Part 1 or Real-time threat Detection - This notebook demonstrates the application of SingleStoreDB's similarity search to create a system 
-for identifying infrequent occurrences, a common requirement in fields such as cybersecurity 
-and fraud detection where only a small percentage of events are potentially malicious.
-"
-icon="browser"
-tags=["iot", "cybersecurity", "training", "vectordb"]
-destinations=["spaces"]
\ No newline at end of file
diff --git a/network-intrusion-detection-part-1/notebook.ipynb b/network-intrusion-detection-part-1/notebook.ipynb
deleted file mode 100644
index b403636a..00000000
--- a/network-intrusion-detection-part-1/notebook.ipynb
+++ /dev/null
@@ -1 +0,0 @@
-{"cells":[{"cell_type":"markdown","id":"93ad2bda-e101-4aad-a83b-45f84560597c","metadata":{"language":"python"},"source":"<div id=\"singlestore-header\" style=\"display: flex; background-color: rgba(235, 249, 245, 0.25); padding: 5px;\">\n    <div id=\"icon-image\" style=\"width: 90px; height: 90px;\">\n        <img width=\"100%\" height=\"100%\" src=\"https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/common/images/header-icons/browser.png\" />\n    </div>\n    <div id=\"text\" style=\"padding: 5px; margin-left: 10px;\">\n        <div id=\"badge\" style=\"display: inline-block; background-color: rgba(0, 0, 0, 0.15); border-radius: 4px; padding: 4px 8px; align-items: center; margin-top: 6px; margin-bottom: -2px; font-size: 80%\">SingleStore Notebooks</div>\n        <h1 style=\"font-weight: 500; margin: 8px 0 0 4px;\">IT Threat Detection, Part 1</h1>\n    </div>\n</div>"},{"cell_type":"markdown","id":"4d343fe9-0c6f-4cf7-bf04-02524cdb5879","metadata":{"language":"python"},"source":"This notebook demonstrates the application of SingleStoreDB's similarity search to create a system for identifying infrequent occurrences, a common requirement in fields such as cybersecurity and fraud detection where only a small percentage of events are potentially malicious.\n\nIn this instance, we aim to construct a network intrusion detection system. These systems continuously monitor incoming and outgoing network traffic, generating alerts when potential threats are detected. We'll utilize a combination of a deep learning model and similarity search to identify and classify network intrusion traffic.\n\nOur initial step involves a dataset of labeled traffic events, distinguishing between benign and malicious events, by transforming them into vector embeddings. These vector embeddings serve as comprehensive mathematical representations of network traffic events. SingleStoreDB's built-in similarity-search algorithms allow us to measure the similarity between different network events. To generate these embeddings, we'll leverage a deep learning model based on recent academic research.\n\nSubsequently, we'll apply this dataset to search for the most similar matches when presented with new, unseen network events. We'll retrieve these matches along with their corresponding labels. This process enables us to classify the unseen events as either **benign** or **malicious** by propagating the labels of the matched events. It's essential to note that intrusion detection is a complex classification task, primarily because malicious events occur infrequently. The similarity search service plays a crucial role in identifying relevant historical labeled events, thus enabling the identification of these rare events while maintaining a low rate of false alarms.\n\n## Install Dependencies"},{"cell_type":"code","execution_count":4,"id":"c649045f-0a53-4c49-88cb-0351e872d68c","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:11:32.025758Z","iopub.status.busy":"2023-10-04T15:11:32.025506Z","iopub.status.idle":"2023-10-04T15:12:02.177962Z","shell.execute_reply":"2023-10-04T15:12:02.177338Z","shell.execute_reply.started":"2023-10-04T15:11:32.025741Z"},"language":"python","tags":[],"trusted":true},"outputs":[],"source":"!pip3 install tensorflow keras pandas --upgrade --quiet"},{"cell_type":"code","execution_count":5,"id":"f3bb35f9-67ea-4a23-b713-888746494baf","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:12:06.378457Z","iopub.status.busy":"2023-10-04T15:12:06.378188Z","iopub.status.idle":"2023-10-04T15:12:07.773000Z","shell.execute_reply":"2023-10-04T15:12:07.772488Z","shell.execute_reply.started":"2023-10-04T15:12:06.378438Z"},"language":"python","scrolled":true,"tags":[],"trusted":true},"outputs":[],"source":"import os\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n\nimport pandas as pd\nimport tensorflow.keras.backend as K\nfrom tensorflow import keras\nfrom tensorflow.keras.models import Model"},{"cell_type":"markdown","id":"01fa2a0b-0213-4399-8481-adc1734cecf0","metadata":{"language":"python"},"source":"We'll define a Python context manager called `clear_memory()` using the **contextlib** module. This context manager will be used to clear memory by running Python's garbage collector (`gc.collect()`) after a block of code is executed."},{"cell_type":"code","execution_count":6,"id":"b7b5a0c6-aef9-4fea-91d9-8777e75d5c1f","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:12:13.931036Z","iopub.status.busy":"2023-10-04T15:12:13.930471Z","iopub.status.idle":"2023-10-04T15:12:13.934758Z","shell.execute_reply":"2023-10-04T15:12:13.933574Z","shell.execute_reply.started":"2023-10-04T15:12:13.931013Z"},"language":"python","trusted":true},"outputs":[],"source":"import contextlib\nimport gc\n\n@contextlib.contextmanager\ndef clear_memory():\n    try:\n        yield\n    finally:\n        gc.collect()"},{"cell_type":"markdown","id":"6eaabcc0-3bfa-4c67-86cd-8b3a42150a6f","metadata":{"language":"python"},"source":"We'll will incorporate portions of code from [research work](https://github.com/Colorado-Mesa-University-Cybersecurity/DeepLearning-IDS). To begin, we'll clone the repository required for data preparation."},{"cell_type":"code","execution_count":7,"id":"1d06bd7a-f3ad-4c3a-a085-d8583d20e2bc","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:12:20.727187Z","iopub.status.busy":"2023-10-04T15:12:20.726917Z","iopub.status.idle":"2023-10-04T15:12:23.906216Z","shell.execute_reply":"2023-10-04T15:12:23.905573Z","shell.execute_reply.started":"2023-10-04T15:12:20.727172Z"},"language":"python","tags":[],"trusted":true},"outputs":[],"source":"!git clone -q https://github.com/Colorado-Mesa-University-Cybersecurity/DeepLearning-IDS.git"},{"cell_type":"markdown","id":"8c323aab-899d-4156-8cfe-98f0fdbb2147","metadata":{"language":"python"},"source":"## Data Preparation\n\nThe datasets we'll utilize comprise two types of network traffic:\n\n1. Benign (normal)\n2. Malicious (attack)\n\nstemming from various network attacks. Our focus will be solely on web-based attacks. These web attacks fall into three common categories:\n\n1. Cross-site scripting (BruteForce-XSS)\n2. SQL-Injection (SQL-Injection)\n3. Brute force attempts on administrative and user passwords (BruteForce-Web)\n\nThe original data was collected over a span of two days.\n\n### Download Data\n\nWe'll proceed by downloading data for two specific dates:\n\n1. February 22, 2018\n2. February 23, 2018\n\nThese files will be retrieved and saved to the current directory. Our intention is to use one of these dates for training and generating vectors, while the other will be reserved for testing purposes."},{"cell_type":"code","execution_count":8,"id":"47a639f2-1f03-4972-8212-316887cc1c73","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:12:32.057896Z","iopub.status.busy":"2023-10-04T15:12:32.057596Z","iopub.status.idle":"2023-10-04T15:12:43.448204Z","shell.execute_reply":"2023-10-04T15:12:43.447571Z","shell.execute_reply.started":"2023-10-04T15:12:32.057874Z"},"language":"python","tags":[],"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":"Thursday-22-02-2018 100%[===================>] 364.91M  57.0MB/s    in 5.8s    \nFriday-23-02-2018_T 100%[===================>] 365.10M  78.5MB/s    in 4.7s    \n"}],"source":"!wget \"https://cse-cic-ids2018.s3.ca-central-1.amazonaws.com/Processed%20Traffic%20Data%20for%20ML%20Algorithms/Thursday-22-02-2018_TrafficForML_CICFlowMeter.csv\" -q --show-progress\n!wget \"https://cse-cic-ids2018.s3.ca-central-1.amazonaws.com/Processed%20Traffic%20Data%20for%20ML%20Algorithms/Friday-23-02-2018_TrafficForML_CICFlowMeter.csv\" -q --show-progress"},{"cell_type":"markdown","id":"21103ba6-036d-4525-9e5a-073ec515aba9","metadata":{"execution":{"iopub.execute_input":"2023-09-27T15:17:45.604442Z","iopub.status.busy":"2023-09-27T15:17:45.604182Z","iopub.status.idle":"2023-09-27T15:17:45.607166Z","shell.execute_reply":"2023-09-27T15:17:45.606518Z","shell.execute_reply.started":"2023-09-27T15:17:45.604422Z"},"language":"python"},"source":"### Review Data"},{"cell_type":"code","execution_count":14,"id":"c17e283b-8629-483a-9631-69392b3b872e","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:12:49.959451Z","iopub.status.busy":"2023-10-04T15:12:49.959246Z","iopub.status.idle":"2023-10-04T15:12:55.095007Z","shell.execute_reply":"2023-10-04T15:12:55.094492Z","shell.execute_reply.started":"2023-10-04T15:12:49.959434Z"},"language":"python","tags":[],"trusted":true},"outputs":[{"data":{"text/plain":"Label\nBenign              1048009\nBrute Force -Web        362\nBrute Force -XSS        151\nSQL Injection            53\nName: count, dtype: int64"},"execution_count":14,"metadata":{},"output_type":"execute_result"}],"source":"with clear_memory():\n    data = pd.read_csv('Friday-23-02-2018_TrafficForML_CICFlowMeter.csv')\n\ndata.Label.value_counts()"},{"cell_type":"markdown","id":"176bc1e1-822e-4e22-a5fa-0dd03d6ecf04","metadata":{"language":"python"},"source":"### Clean Data\n\nWe'll run a cleanup script from the previously downloaded GitHub repo."},{"cell_type":"code","execution_count":17,"id":"af4f1d28-cba5-4547-a2bd-a34b200fc261","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:13:01.860976Z","iopub.status.busy":"2023-10-04T15:13:01.860721Z","iopub.status.idle":"2023-10-04T15:13:47.515247Z","shell.execute_reply":"2023-10-04T15:13:47.514670Z","shell.execute_reply.started":"2023-10-04T15:13:01.860959Z"},"language":"python","tags":[],"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":"cleaning Friday-23-02-2018_TrafficForML_CICFlowMeter.csv\ntotal rows read = 1048576\nall done writing 1042868 rows; dropped 5708 rows\n"}],"source":"!python DeepLearning-IDS/data_cleanup.py \"Friday-23-02-2018_TrafficForML_CICFlowMeter.csv\" \"result23022018\""},{"cell_type":"markdown","id":"15fd1f07-d8a1-4ab5-a36c-5a08f1f043e9","metadata":{"language":"python"},"source":"We'll now review the cleaned data from the previous step."},{"cell_type":"code","execution_count":18,"id":"d893178a-6bcc-4a16-89ef-f3559adad14c","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:13:58.399760Z","iopub.status.busy":"2023-10-04T15:13:58.399474Z","iopub.status.idle":"2023-10-04T15:14:03.115597Z","shell.execute_reply":"2023-10-04T15:14:03.115048Z","shell.execute_reply.started":"2023-10-04T15:13:58.399739Z"},"language":"python","tags":[],"trusted":true},"outputs":[{"data":{"text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Dst Port</th>\n      <th>Protocol</th>\n      <th>Timestamp</th>\n      <th>Flow Duration</th>\n      <th>Tot Fwd Pkts</th>\n      <th>Tot Bwd Pkts</th>\n      <th>TotLen Fwd Pkts</th>\n      <th>TotLen Bwd Pkts</th>\n      <th>Fwd Pkt Len Max</th>\n      <th>Fwd Pkt Len Min</th>\n      <th>...</th>\n      <th>Fwd Seg Size Min</th>\n      <th>Active Mean</th>\n      <th>Active Std</th>\n      <th>Active Max</th>\n      <th>Active Min</th>\n      <th>Idle Mean</th>\n      <th>Idle Std</th>\n      <th>Idle Max</th>\n      <th>Idle Min</th>\n      <th>Label</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>22</td>\n      <td>6</td>\n      <td>1.519374e+09</td>\n      <td>1532698</td>\n      <td>11</td>\n      <td>11</td>\n      <td>1179</td>\n      <td>1969</td>\n      <td>648</td>\n      <td>0</td>\n      <td>...</td>\n      <td>32</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0.0</td>\n      <td>0.000000e+00</td>\n      <td>0</td>\n      <td>0</td>\n      <td>Benign</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>500</td>\n      <td>17</td>\n      <td>1.519374e+09</td>\n      <td>117573855</td>\n      <td>3</td>\n      <td>0</td>\n      <td>1500</td>\n      <td>0</td>\n      <td>500</td>\n      <td>500</td>\n      <td>...</td>\n      <td>8</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>58786927.5</td>\n      <td>2.375324e+07</td>\n      <td>75583006</td>\n      <td>41990849</td>\n      <td>Benign</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>500</td>\n      <td>17</td>\n      <td>1.519374e+09</td>\n      <td>117573848</td>\n      <td>3</td>\n      <td>0</td>\n      <td>1500</td>\n      <td>0</td>\n      <td>500</td>\n      <td>500</td>\n      <td>...</td>\n      <td>8</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>58786924.0</td>\n      <td>2.375325e+07</td>\n      <td>75583007</td>\n      <td>41990841</td>\n      <td>Benign</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>22</td>\n      <td>6</td>\n      <td>1.519374e+09</td>\n      <td>1745392</td>\n      <td>11</td>\n      <td>11</td>\n      <td>1179</td>\n      <td>1969</td>\n      <td>648</td>\n      <td>0</td>\n      <td>...</td>\n      <td>32</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0.0</td>\n      <td>0.000000e+00</td>\n      <td>0</td>\n      <td>0</td>\n      <td>Benign</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>500</td>\n      <td>17</td>\n      <td>1.519374e+09</td>\n      <td>89483474</td>\n      <td>6</td>\n      <td>0</td>\n      <td>3000</td>\n      <td>0</td>\n      <td>500</td>\n      <td>500</td>\n      <td>...</td>\n      <td>8</td>\n      <td>4000364.0</td>\n      <td>0.0</td>\n      <td>4000364</td>\n      <td>4000364</td>\n      <td>21370777.5</td>\n      <td>1.528092e+07</td>\n      <td>41989576</td>\n      <td>7200485</td>\n      <td>Benign</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 80 columns</p>\n</div>","text/plain":"   Dst Port  Protocol     Timestamp  Flow Duration  Tot Fwd Pkts  \\\n0        22         6  1.519374e+09        1532698            11   \n1       500        17  1.519374e+09      117573855             3   \n2       500        17  1.519374e+09      117573848             3   \n3        22         6  1.519374e+09        1745392            11   \n4       500        17  1.519374e+09       89483474             6   \n\n   Tot Bwd Pkts  TotLen Fwd Pkts  TotLen Bwd Pkts  Fwd Pkt Len Max  \\\n0            11             1179             1969              648   \n1             0             1500                0              500   \n2             0             1500                0              500   \n3            11             1179             1969              648   \n4             0             3000                0              500   \n\n   Fwd Pkt Len Min  ...  Fwd Seg Size Min  Active Mean  Active Std  \\\n0                0  ...                32          0.0         0.0   \n1              500  ...                 8          0.0         0.0   \n2              500  ...                 8          0.0         0.0   \n3                0  ...                32          0.0         0.0   \n4              500  ...                 8    4000364.0         0.0   \n\n   Active Max  Active Min   Idle Mean      Idle Std  Idle Max  Idle Min  \\\n0           0           0         0.0  0.000000e+00         0         0   \n1           0           0  58786927.5  2.375324e+07  75583006  41990849   \n2           0           0  58786924.0  2.375325e+07  75583007  41990841   \n3           0           0         0.0  0.000000e+00         0         0   \n4     4000364     4000364  21370777.5  1.528092e+07  41989576   7200485   \n\n    Label  \n0  Benign  \n1  Benign  \n2  Benign  \n3  Benign  \n4  Benign  \n\n[5 rows x 80 columns]"},"execution_count":18,"metadata":{},"output_type":"execute_result"}],"source":"with clear_memory():\n    data_23_cleaned = pd.read_csv('result23022018.csv')\n\ndata_23_cleaned.head()"},{"cell_type":"code","execution_count":19,"id":"d98eee6b-6a68-4fb6-b418-1599daa9e4ff","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:14:09.241483Z","iopub.status.busy":"2023-10-04T15:14:09.241237Z","iopub.status.idle":"2023-10-04T15:14:09.283891Z","shell.execute_reply":"2023-10-04T15:14:09.283370Z","shell.execute_reply.started":"2023-10-04T15:14:09.241467Z"},"language":"python","tags":[],"trusted":true},"outputs":[{"data":{"text/plain":"Label\nBenign              1042301\nBrute Force -Web        362\nBrute Force -XSS        151\nSQL Injection            53\nName: count, dtype: int64"},"execution_count":19,"metadata":{},"output_type":"execute_result"}],"source":"data_23_cleaned.Label.value_counts()"},{"cell_type":"markdown","id":"1b466f2f-ee4f-43ba-b1e6-0cb3380c9125","metadata":{"language":"python"},"source":"## Load Model\n\nIn this section, we'll load a pre-trained model that has been trained on data collected from the same date.\n\nThere are slight modifications to the original model, specifically, altering the number of classes. Initially, the model was designed to classify into four categories:\n\n1. Benign\n2. BruteForce-Web\n3. BruteForce-XSS\n4. SQL-Injection\n\nOur modified model has been adjusted to classify into just two categories:\n\n1. Benign\n2. Attack\n\n<div class=\"alert alert-block alert-warning\">\n    <b class=\"fa fa-solid fa-exclamation-circle\"></b>\n    <div>\n        <p><b>Action Required</b></p>\n        <p>The ZIP file is hosted on a Google Drive.</p>\n        <p>Using the <b>Edit Firewall</b> button in the top right, add the following to the SingleStoreDB Cloud notebook firewall, one-by-one:\n            <ul style=\"list-style: none;\">\n                <li><b>drive.google.com</b></li>\n                <li><b>*.googleapis.com</b></li>\n                <li><b>*.googleusercontent.com</b></li>\n            </ul>\n        </p>\n    </div>\n</div>"},{"cell_type":"code","execution_count":20,"id":"4074a769-b16b-4543-b0db-518c7f95f205","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:14:18.247758Z","iopub.status.busy":"2023-10-04T15:14:18.247504Z","iopub.status.idle":"2023-10-04T15:14:20.107308Z","shell.execute_reply":"2023-10-04T15:14:20.106660Z","shell.execute_reply.started":"2023-10-04T15:14:18.247741Z"},"language":"python","tags":[],"trusted":true},"outputs":[],"source":"!wget -q -O it_threat_model.zip \"https://drive.google.com/uc?export=download&id=1ahr5dYlhuxS56M6helUFI0yIxxIoFk9o\" \n!unzip -q it_threat_model.zip"},{"cell_type":"code","execution_count":21,"id":"7c77cf96-e3f7-4bed-9f7e-913f851367d8","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:14:26.353287Z","iopub.status.busy":"2023-10-04T15:14:26.352961Z","iopub.status.idle":"2023-10-04T15:14:26.688942Z","shell.execute_reply":"2023-10-04T15:14:26.688380Z","shell.execute_reply.started":"2023-10-04T15:14:26.353263Z"},"language":"python","tags":[],"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":"Model: \"sequential\"\n_________________________________________________________________\n Layer (type)                Output Shape              Param #   \n=================================================================\n dense (Dense)               (None, 128)               10240     \n                                                                 \n dense_1 (Dense)             (None, 64)                8256      \n                                                                 \n dense_2 (Dense)             (None, 1)                 65        \n                                                                 \n=================================================================\nTotal params: 18561 (72.50 KB)\nTrainable params: 18561 (72.50 KB)\nNon-trainable params: 0 (0.00 Byte)\n_________________________________________________________________\n"}],"source":"with clear_memory():\n    model = keras.models.load_model('it_threat_model')\n\nmodel.summary()"},{"cell_type":"code","execution_count":22,"id":"ed66c137-1023-4dce-b9ea-5e742099302e","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:14:34.082125Z","iopub.status.busy":"2023-10-04T15:14:34.081856Z","iopub.status.idle":"2023-10-04T15:14:34.224177Z","shell.execute_reply":"2023-10-04T15:14:34.223680Z","shell.execute_reply.started":"2023-10-04T15:14:34.082108Z"},"language":"python","tags":[],"trusted":true},"outputs":[],"source":"with clear_memory():\n    # Use the first layer\n    layer_name = 'dense'\n    intermediate_layer_model = Model(\n        inputs = model.input,\n        outputs = model.get_layer(layer_name).output\n    )"},{"cell_type":"markdown","id":"24d3bfbb-78ee-4a8b-a429-84f988915b6d","metadata":{"language":"python"},"source":"## Upload Data to SingleStoreDB\n\n### Prepare Data\nWe'll use a method for defining item IDs that aligns with the event's label."},{"cell_type":"code","execution_count":23,"id":"ae1fc38e-f532-430a-99cd-f3e154343bd4","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:14:41.159100Z","iopub.status.busy":"2023-10-04T15:14:41.158708Z","iopub.status.idle":"2023-10-04T15:16:14.701123Z","shell.execute_reply":"2023-10-04T15:16:14.700506Z","shell.execute_reply.started":"2023-10-04T15:14:41.159081Z"},"language":"python","tags":[],"trusted":true},"outputs":[{"name":"stdout","output_type":"stream","text":"32590/32590 [==============================] - 26s 797us/step\n"},{"name":"stderr","output_type":"stream","text":"100%|██████████| 1042867/1042867 [00:50<00:00, 20810.99it/s]\n"}],"source":"from tqdm import tqdm\nitems_to_upload = []\n\nwith clear_memory():\n    model_res = intermediate_layer_model.predict(K.constant(data_23_cleaned.iloc[:,:-1]))\n    \n    for i, res in tqdm(zip(data_23_cleaned.iterrows(), model_res), total = len(model_res)):\n        benign_or_attack = i[1]['Label'][:3]\n        items_to_upload.append((benign_or_attack + '_' + str(i[0]), res.tolist()))"},{"cell_type":"markdown","id":"e4b3f74e-efea-499e-b25a-74316fcf2395","metadata":{"language":"python"},"source":"We'll store the data in a Pandas DataFrame."},{"cell_type":"code","execution_count":29,"id":"0177714c-62b7-4d3e-a93b-2fd467ff1352","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:17:45.778441Z","iopub.status.busy":"2023-10-04T15:17:45.778183Z","iopub.status.idle":"2023-10-04T15:17:47.525268Z","shell.execute_reply":"2023-10-04T15:17:47.524802Z","shell.execute_reply.started":"2023-10-04T15:17:45.778425Z"},"language":"python","tags":[],"trusted":true},"outputs":[{"data":{"text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>ID</th>\n      <th>Model_Results</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Ben_0</td>\n      <td>[0.0, 0.0, 0.0, 125628656.0, 0.0, 0.0, 5421442...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Ben_1</td>\n      <td>[0.0, 0.0, 0.0, 356751744.0, 1190461440.0, 0.0...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Ben_2</td>\n      <td>[0.0, 0.0, 0.0, 356751680.0, 1190461440.0, 0.0...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Ben_3</td>\n      <td>[0.0, 0.0, 0.0, 125515856.0, 0.0, 0.0, 5432884...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Ben_4</td>\n      <td>[0.0, 0.0, 0.0, 26214912.0, 698683840.0, 0.0, ...</td>\n    </tr>\n  </tbody>\n</table>\n</div>","text/plain":"      ID                                      Model_Results\n0  Ben_0  [0.0, 0.0, 0.0, 125628656.0, 0.0, 0.0, 5421442...\n1  Ben_1  [0.0, 0.0, 0.0, 356751744.0, 1190461440.0, 0.0...\n2  Ben_2  [0.0, 0.0, 0.0, 356751680.0, 1190461440.0, 0.0...\n3  Ben_3  [0.0, 0.0, 0.0, 125515856.0, 0.0, 0.0, 5432884...\n4  Ben_4  [0.0, 0.0, 0.0, 26214912.0, 698683840.0, 0.0, ..."},"execution_count":29,"metadata":{},"output_type":"execute_result"}],"source":"with clear_memory():\n    df = pd.DataFrame(items_to_upload, columns=['ID', 'Model_Results'])\n\ndf.head()"},{"cell_type":"markdown","id":"f52639f6-82c8-427f-aa7b-9859d6580eae","metadata":{"language":"python"},"source":"Now we'll convert the vectors to a binary format, ready to store in SingleStoreDB."},{"cell_type":"code","execution_count":30,"id":"ec8a2d26-98cf-4f1a-9e13-e197cd65040c","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:17:53.975325Z","iopub.status.busy":"2023-10-04T15:17:53.975066Z","iopub.status.idle":"2023-10-04T15:17:57.811281Z","shell.execute_reply":"2023-10-04T15:17:57.810738Z","shell.execute_reply.started":"2023-10-04T15:17:53.975308Z"},"language":"python","trusted":true},"outputs":[],"source":"import struct\n\ndef data_to_binary(data: list[float]):\n    format_string = 'f' * len(data)\n    return struct.pack(format_string, *data)\n\nwith clear_memory():\n    df['Model_Results'] = df['Model_Results'].apply(data_to_binary)"},{"cell_type":"markdown","id":"e267e6a4-3e70-4411-96cd-94da3f1f6011","metadata":{"language":"python"},"source":"We'll check the DataFrame."},{"cell_type":"code","execution_count":31,"id":"770cfe1e-4d55-43a8-a5e0-36da2525ca25","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:18:02.459555Z","iopub.status.busy":"2023-10-04T15:18:02.459286Z","iopub.status.idle":"2023-10-04T15:18:02.466500Z","shell.execute_reply":"2023-10-04T15:18:02.465962Z","shell.execute_reply.started":"2023-10-04T15:18:02.459538Z"},"language":"python","trusted":true},"outputs":[{"data":{"text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>ID</th>\n      <th>Model_Results</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Ben_0</td>\n      <td>b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Ben_1</td>\n      <td>b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Ben_2</td>\n      <td>b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Ben_3</td>\n      <td>b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Ben_4</td>\n      <td>b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...</td>\n    </tr>\n  </tbody>\n</table>\n</div>","text/plain":"      ID                                      Model_Results\n0  Ben_0  b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...\n1  Ben_1  b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...\n2  Ben_2  b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...\n3  Ben_3  b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...\n4  Ben_4  b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00..."},"execution_count":31,"metadata":{},"output_type":"execute_result"}],"source":"df.head()"},{"cell_type":"markdown","id":"8131ad6c-6d50-49db-845c-457e9957ddb9","metadata":{"language":"python"},"source":"### Create Database and Table"},{"cell_type":"code","execution_count":34,"id":"8940168f-072f-42cd-8855-4c6b6829421d","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:18:54.107461Z","iopub.status.busy":"2023-10-04T15:18:54.107206Z","iopub.status.idle":"2023-10-04T15:19:02.426902Z","shell.execute_reply":"2023-10-04T15:19:02.426343Z","shell.execute_reply.started":"2023-10-04T15:18:54.107444Z"},"language":"sql","tags":[],"trusted":true},"outputs":[{"data":{"text/html":"<table>\n    <thead>\n        <tr>\n        </tr>\n    </thead>\n    <tbody>\n    </tbody>\n</table>","text/plain":"++\n||\n++\n++"},"execution_count":34,"metadata":{},"output_type":"execute_result"}],"source":"%%sql\n%%sql\nDROP DATABASE IF EXISTS siem_log_kafka_demo;\n\nCREATE DATABASE IF NOT EXISTS siem_log_kafka_demo;\n\nUSE siem_log_kafka_demo;\n\nDROP TABLE IF EXISTS model_results_demo;\n\nCREATE TABLE IF NOT EXISTS model_results (\n    id TEXT,\n    Model_Results BLOB\n);"},{"cell_type":"markdown","id":"b6390a0c-d60a-4876-85bd-56e17deabeeb","metadata":{"language":"python"},"source":"### Get Connection Details\n\n<div class=\"alert alert-block alert-warning\">\n    <b class=\"fa fa-solid fa-exclamation-circle\"></b>\n    <div>\n        <p><b>Action Required</b></p>\n        <p>Select the database from the drop-down menu at the top of this notebook. It updates the <b>connection_url</b> which is used by SQLAlchemy to make connections to the selected database.</p>\n    </div>\n</div>"},{"cell_type":"code","execution_count":36,"id":"8208b373-1973-4925-8d8a-cd43ba13e7d7","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:19:22.359227Z","iopub.status.busy":"2023-10-04T15:19:22.358972Z","iopub.status.idle":"2023-10-04T15:19:22.363216Z","shell.execute_reply":"2023-10-04T15:19:22.362608Z","shell.execute_reply.started":"2023-10-04T15:19:22.359211Z"},"language":"python","tags":[],"trusted":true},"outputs":[],"source":"from sqlalchemy import *\n\ndb_connection = create_engine(connection_url)"},{"cell_type":"markdown","id":"d97c6886-0ea9-47b5-938b-03ada717b394","metadata":{"execution":{"iopub.execute_input":"2023-09-27T15:37:17.351200Z","iopub.status.busy":"2023-09-27T15:37:17.350962Z","iopub.status.idle":"2023-09-27T15:37:17.354770Z","shell.execute_reply":"2023-09-27T15:37:17.353988Z","shell.execute_reply.started":"2023-09-27T15:37:17.351170Z"},"language":"python"},"source":"### Store DataFrame"},{"cell_type":"code","execution_count":37,"id":"e70b91a0-fdb9-46da-8c0d-5acc947799be","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:19:28.729452Z","iopub.status.busy":"2023-10-04T15:19:28.729169Z","iopub.status.idle":"2023-10-04T15:22:34.401289Z","shell.execute_reply":"2023-10-04T15:22:34.400783Z","shell.execute_reply.started":"2023-10-04T15:19:28.729435Z"},"language":"python","trusted":true},"outputs":[],"source":"with clear_memory():\n    df.to_sql(\n        'model_results',\n        con = db_connection,\n        if_exists = 'append',\n        index = False,\n        chunksize = 1000\n    )"},{"cell_type":"markdown","id":"559928bd-10a0-4880-805a-23444d7d72a3","metadata":{"language":"python"},"source":"### Check Stored Data"},{"cell_type":"code","execution_count":47,"id":"f4cb8e7f-362f-4477-8600-e5f39ec197b8","metadata":{"execution":{"iopub.execute_input":"2023-10-04T15:22:52.838607Z","iopub.status.busy":"2023-10-04T15:22:52.838359Z","iopub.status.idle":"2023-10-04T15:22:53.253409Z","shell.execute_reply":"2023-10-04T15:22:53.252952Z","shell.execute_reply.started":"2023-10-04T15:22:52.838591Z"},"language":"sql","trusted":true},"outputs":[{"data":{"text/html":"<table>\n    <thead>\n        <tr>\n            <th>ID</th>\n            <th>Model_Results</th>\n        </tr>\n    </thead>\n    <tbody>\n        <tr>\n            <td>Ben_764632</td>\n            <td>[0, 0, 0, 161398336, 0, 0, 91465440, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 186428320, 0, 0, 0, 167306864, 0, 0, 277207904, 0, 92328576, 73124928, 0, 0, 0, 95751136, 0, 0, 0, 0, 0, 0, 0, 0, 230162768, 273622432, 511405120, 0, 0, 0, 0, 0, 0, 0, 0, 0, 145775152, 106490400, 373456928, 0, 0, 0, 211604256, 30848250, 0, 0, 0, 0, 326004800, 0, 0, 0, 0, 13625428, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 248507264, 0, 121489904, 196521904, 0, 2331058, 0, 0, 234076784, 247954704, 0, 0, 16321682, 0, 0, 0, 343808992, 0, 0, 0, 74993352, 0, 0, 59710728, 0, 0, 89274704, 0, 174431776, 107296112, 0, 0, 134864096, 0, 0]</td>\n        </tr>\n    </tbody>\n</table>","text/plain":"+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n|     ID     |                                                                                                                                                                                                                                                                                                                    Model_Results                                                                                                                                                                                                                                                                                                                     |\n+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n| Ben_764632 | [0, 0, 0, 161398336, 0, 0, 91465440, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 186428320, 0, 0, 0, 167306864, 0, 0, 277207904, 0, 92328576, 73124928, 0, 0, 0, 95751136, 0, 0, 0, 0, 0, 0, 0, 0, 230162768, 273622432, 511405120, 0, 0, 0, 0, 0, 0, 0, 0, 0, 145775152, 106490400, 373456928, 0, 0, 0, 211604256, 30848250, 0, 0, 0, 0, 326004800, 0, 0, 0, 0, 13625428, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 248507264, 0, 121489904, 196521904, 0, 2331058, 0, 0, 234076784, 247954704, 0, 0, 16321682, 0, 0, 0, 343808992, 0, 0, 0, 74993352, 0, 0, 59710728, 0, 0, 89274704, 0, 174431776, 107296112, 0, 0, 134864096, 0, 0] |\n+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+"},"execution_count":47,"metadata":{},"output_type":"execute_result"}],"source":"%%sql\n%%sql\nUSE siem_log_kafka_demo;\n\nSELECT ID, JSON_ARRAY_UNPACK(Model_Results) AS Model_Results\nFROM model_results\nLIMIT 1;"}],"metadata":{"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.11.4"},"singlestore_connection":{"connectionID":"a98658e6-5e5d-4be9-be2e-9fa993172504","defaultDatabase":""},"singlestore_row_limit":300},"nbformat":4,"nbformat_minor":5}
\ No newline at end of file