03_infer_on_provided_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this competition, we don't really have a validation set. The competition organizers where open about the fact that the provided data we can use to evaluate our model comes from a different distribution than the test set!\n",
    "\n",
    "As a side note - it is really great that the competition organizers opted to take the route of transparency. Not being open about how you sample what might look like as the validation set, or going about it in a 'tricky' way, is a sure route to hurting the competition (or any ML project in general).\n",
    "\n",
    "So we know we don't have a validation set. We understand where the train set comes from, so that will give us some idea on what we can do to improve the preformance of our model.\n",
    "\n",
    "Still, the data provided is the closest thing we have to a validation set, at least for now. Let's put our model to the test and see what we can learn.\n",
    "\n",
    "Also, we could be importing from our library the AudioDataset, metrics, and so forth. But we will not have access to this on Kaggle. Thus the rules of engagement will be that we need to define everything we will need in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import torch\n",
    "from fastai2.vision.all import *\n",
    "import soundfile as sf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean, std = (-6.132126808166504e-05, 0.04304003225515465)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "classes = pd.read_pickle('data/classes.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_arch = lambda: nn.Sequential(*[\n",
    "    Lambda(lambda x: x.unsqueeze(1)),\n",
    "    ConvLayer(1, 16, ks=64, stride=2, ndim=1),\n",
    "    ConvLayer(16, 16, ks=8, stride=8, ndim=1),\n",
    "    ConvLayer(16, 32, ks=32, stride=2, ndim=1),\n",
    "    ConvLayer(32, 32, ks=8, stride=8, ndim=1),\n",
    "    ConvLayer(32, 64, ks=16, stride=2, ndim=1),\n",
    "    ConvLayer(64, 128, ks=8, stride=2, ndim=1),\n",
    "    ConvLayer(128, 256, ks=4, stride=2, ndim=1),\n",
    "    ConvLayer(256, 256, ks=4, stride=4, ndim=1),\n",
    "    Flatten(),\n",
    "    LinBnDrop(5120, 512, p=0.25, act=nn.ReLU()),\n",
    "    LinBnDrop(512, 512, p=0.25, act=nn.ReLU()),\n",
    "    LinBnDrop(512, 256, p=0.25, act=nn.ReLU()),\n",
    "    LinBnDrop(256, len(classes)),\n",
    "    nn.Sigmoid()\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = get_arch()\n",
    "model.load_state_dict(torch.load('data/models/first_model.pth'))\n",
    "model.cuda()\n",
    "model.eval();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_df = pd.read_csv('data/example_test_audio_summary.csv')\n",
    "test_df = pd.read_csv('data/test.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename_seconds</th>\n",
       "      <th>birds</th>\n",
       "      <th>filename</th>\n",
       "      <th>seconds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BLKFR-10-CPL_20190611_093000_5</td>\n",
       "      <td>gockin mouchi westan</td>\n",
       "      <td>BLKFR-10-CPL</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BLKFR-10-CPL_20190611_093000_10</td>\n",
       "      <td>gockin westan</td>\n",
       "      <td>BLKFR-10-CPL</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BLKFR-10-CPL_20190611_093000_15</td>\n",
       "      <td>gockin westan</td>\n",
       "      <td>BLKFR-10-CPL</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BLKFR-10-CPL_20190611_093000_20</td>\n",
       "      <td>mouchi</td>\n",
       "      <td>BLKFR-10-CPL</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>BLKFR-10-CPL_20190611_093000_25</td>\n",
       "      <td>mouchi</td>\n",
       "      <td>BLKFR-10-CPL</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  filename_seconds                 birds      filename  \\\n",
       "0   BLKFR-10-CPL_20190611_093000_5  gockin mouchi westan  BLKFR-10-CPL   \n",
       "1  BLKFR-10-CPL_20190611_093000_10         gockin westan  BLKFR-10-CPL   \n",
       "2  BLKFR-10-CPL_20190611_093000_15         gockin westan  BLKFR-10-CPL   \n",
       "3  BLKFR-10-CPL_20190611_093000_20                mouchi  BLKFR-10-CPL   \n",
       "4  BLKFR-10-CPL_20190611_093000_25                mouchi  BLKFR-10-CPL   \n",
       "\n",
       "   seconds  \n",
       "0        5  \n",
       "1       10  \n",
       "2       15  \n",
       "3       20  \n",
       "4       25  "
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_df.filename = valid_df.filename_seconds.apply(lambda fn: '_'.join(fn.split('_')[:-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_df.loc[valid_df.filename == 'BLKFR-10-CPL_20190611_093000'].filename = 'BLKFR-10-CPL_20190611_093000.pt540'\n",
    "valid_df.loc[valid_df.filename == 'ORANGE-7-CAP_20190606_093000'].filename = 'ORANGE-7-CAP_20190606_093000.pt623'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>site</th>\n",
       "      <th>row_id</th>\n",
       "      <th>seconds</th>\n",
       "      <th>audio_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>site_1</td>\n",
       "      <td>site_1_0a997dff022e3ad9744d4e7bbf923288_5</td>\n",
       "      <td>5</td>\n",
       "      <td>0a997dff022e3ad9744d4e7bbf923288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>site_1</td>\n",
       "      <td>site_1_0a997dff022e3ad9744d4e7bbf923288_10</td>\n",
       "      <td>10</td>\n",
       "      <td>0a997dff022e3ad9744d4e7bbf923288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>site_1</td>\n",
       "      <td>site_1_0a997dff022e3ad9744d4e7bbf923288_15</td>\n",
       "      <td>15</td>\n",
       "      <td>0a997dff022e3ad9744d4e7bbf923288</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     site                                      row_id  seconds  \\\n",
       "0  site_1   site_1_0a997dff022e3ad9744d4e7bbf923288_5        5   \n",
       "1  site_1  site_1_0a997dff022e3ad9744d4e7bbf923288_10       10   \n",
       "2  site_1  site_1_0a997dff022e3ad9744d4e7bbf923288_15       15   \n",
       "\n",
       "                           audio_id  \n",
       "0  0a997dff022e3ad9744d4e7bbf923288  \n",
       "1  0a997dff022e3ad9744d4e7bbf923288  \n",
       "2  0a997dff022e3ad9744d4e7bbf923288  "
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_df.loc[valid_df.birds.isna(), 'birds'] = ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(153, 4)"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "items = [(row.birds.split(), f'data/example_test_audio/{row.filename}.mp3', row.seconds) for idx, row in valid_df.iterrows()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "SAMPLE_RATE = 32_000\n",
    "\n",
    "from torch.utils.data import Dataset\n",
    "\n",
    "class AudioDataset(Dataset):\n",
    "    def __init__(self, items, classes, mean=None, std=None):\n",
    "        self.items = items\n",
    "        self.vocab = classes\n",
    "        self.do_norm = (mean and std)\n",
    "        self.mean = mean\n",
    "        self.std = std\n",
    "    def __getitem__(self, idx):\n",
    "        cls, path, offset = self.items[idx]\n",
    "        x, _ = sf.read(path, SAMPLE_RATE*5, start=offset)\n",
    "        if self.do_norm: x = self.normalize(x)\n",
    "        return x.astype(np.float32), self.one_hot_encode(cls)\n",
    "    def normalize(self, x):\n",
    "        return (x - self.mean) / self.std\n",
    "    def one_hot_encode(self, cls):\n",
    "        one_hot = np.zeros((len(self.vocab)))\n",
    "        for bird in cls:\n",
    "            if bird == 'unk': continue\n",
    "            bird_idx = self.vocab.index(bird)\n",
    "            one_hot[bird_idx] = 1\n",
    "        return one_hot\n",
    "    def __len__(self):\n",
    "        return len(self.items)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_ds = AudioDataset(items, classes, mean=mean, std=std)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "153"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(valid_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "Error opening 'data/example_test_audio/BLKFR-10-CPL_20190611_093000.pt540.mp3': File contains data in an unknown format.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-112-d6a63dc47d49>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mdl\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mDataLoader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalid_ds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m128\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0mbatch\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mdl\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m     \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py\u001b[0m in \u001b[0;36m__next__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    343\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    344\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__next__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 345\u001b[0;31m         \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_next_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    346\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_num_yielded\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    347\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_dataset_kind\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0m_DatasetKind\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mIterable\u001b[0m \u001b[0;32mand\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m\\\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py\u001b[0m in \u001b[0;36m_next_data\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    383\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_next_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    384\u001b[0m         \u001b[0mindex\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_next_index\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# may raise StopIteration\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 385\u001b[0;31m         \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_dataset_fetcher\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfetch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# may raise StopIteration\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    386\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_pin_memory\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    387\u001b[0m             \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpin_memory\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpin_memory\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py\u001b[0m in \u001b[0;36mfetch\u001b[0;34m(self, possibly_batched_index)\u001b[0m\n\u001b[1;32m     42\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mfetch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpossibly_batched_index\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     43\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mauto_collation\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 44\u001b[0;31m             \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midx\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0midx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mpossibly_batched_index\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     45\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     46\u001b[0m             \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpossibly_batched_index\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m     42\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mfetch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpossibly_batched_index\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     43\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mauto_collation\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 44\u001b[0;31m             \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midx\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0midx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mpossibly_batched_index\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     45\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     46\u001b[0m             \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpossibly_batched_index\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-109-30a583155e6f>\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, idx)\u001b[0m\n\u001b[1;32m     12\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__getitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m         \u001b[0mcls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpath\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moffset\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0midx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 14\u001b[0;31m         \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mSAMPLE_RATE\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstart\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moffset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     15\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdo_norm\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnormalize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mastype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfloat32\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mone_hot_encode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcls\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/soundfile.py\u001b[0m in \u001b[0;36mread\u001b[0;34m(file, frames, start, stop, dtype, always_2d, fill_value, out, samplerate, channels, format, subtype, endian, closefd)\u001b[0m\n\u001b[1;32m    255\u001b[0m     \"\"\"\n\u001b[1;32m    256\u001b[0m     with SoundFile(file, 'r', samplerate, channels,\n\u001b[0;32m--> 257\u001b[0;31m                    subtype, endian, format, closefd) as f:\n\u001b[0m\u001b[1;32m    258\u001b[0m         \u001b[0mframes\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_prepare_read\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstop\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mframes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    259\u001b[0m         \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mframes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malways_2d\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfill_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/soundfile.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, file, mode, samplerate, channels, subtype, endian, format, closefd)\u001b[0m\n\u001b[1;32m    625\u001b[0m         self._info = _create_info_struct(file, mode, samplerate, channels,\n\u001b[1;32m    626\u001b[0m                                          format, subtype, endian)\n\u001b[0;32m--> 627\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_file\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_open\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfile\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode_int\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mclosefd\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    628\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0missuperset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'r+'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mseekable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    629\u001b[0m             \u001b[0;31m# Move write position to 0 (like in Python file objects)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/soundfile.py\u001b[0m in \u001b[0;36m_open\u001b[0;34m(self, file, mode_int, closefd)\u001b[0m\n\u001b[1;32m   1180\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Invalid file: {0!r}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1181\u001b[0m         _error_check(_snd.sf_error(file_ptr),\n\u001b[0;32m-> 1182\u001b[0;31m                      \"Error opening {0!r}: \".format(self.name))\n\u001b[0m\u001b[1;32m   1183\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mmode_int\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0m_snd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSFM_WRITE\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1184\u001b[0m             \u001b[0;31m# Due to a bug in libsndfile version <= 1.0.25, frames != 0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/soundfile.py\u001b[0m in \u001b[0;36m_error_check\u001b[0;34m(err, prefix)\u001b[0m\n\u001b[1;32m   1353\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0merr\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1354\u001b[0m         \u001b[0merr_str\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_snd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msf_error_number\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1355\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprefix\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0m_ffi\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstring\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merr_str\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'utf-8'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'replace'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1356\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1357\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: Error opening 'data/example_test_audio/BLKFR-10-CPL_20190611_093000.pt540.mp3': File contains data in an unknown format."
     ]
    }
   ],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "dl = DataLoader(valid_ds, batch_size=128)\n",
    "\n",
    "for batch in dl:\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh no! Turns out `soundfile` is unable to read mp3 files! Switching to librosa... But that means we need to go back and use librosa everywhere now, and also save the train files in an mp3 format. Or even better, figure out how to use the train files without concatenating them together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "import librosa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/librosa/core/audio.py:161: UserWarning: PySoundFile failed. Trying audioread instead.\n",
      "  warnings.warn('PySoundFile failed. Trying audioread instead.')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([0.01870877, 0.02787147, 0.03086149, ..., 0.0399581 , 0.03838276,\n",
       "       0.0199724 ], dtype=float32)"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "librosa.load('data/example_test_audio/BLKFR-10-CPL_20190611_093000.pt540.mp3', sr=SAMPLE_RATE)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [],
   "source": [
    "SAMPLE_RATE = 32_000\n",
    "\n",
    "from torch.utils.data import Dataset\n",
    "\n",
    "class AudioDataset(Dataset):\n",
    "    def __init__(self, items, classes, mean=None, std=None):\n",
    "        self.items = items\n",
    "        self.vocab = classes\n",
    "        self.do_norm = (mean and std)\n",
    "        self.mean = mean\n",
    "        self.std = std\n",
    "    def __getitem__(self, idx):\n",
    "        cls, path, offset = self.items[idx]\n",
    "        x, _ = librosa.load(path, sr=SAMPLE_RATE, duration=5, offset=offset)\n",
    "        if self.do_norm: x = self.normalize(x)\n",
    "        return x.astype(np.float32), self.one_hot_encode(cls)\n",
    "    def normalize(self, x):\n",
    "        return (x - self.mean) / self.std\n",
    "    def one_hot_encode(self, cls):\n",
    "        one_hot = np.zeros((len(self.vocab)))\n",
    "        for bird in cls:\n",
    "            if bird in ['unk', 'whhwoo', 'squirrel', 'mouqua', 'hawo']: continue\n",
    "            bird_idx = self.vocab.index(bird)\n",
    "            one_hot[bird_idx] = 1\n",
    "        return one_hot\n",
    "    def __len__(self):\n",
    "        return len(self.items)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_ds = AudioDataset(items, classes, mean=mean, std=std)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, the files in test_audio have different sampling rate than 32_000... AFAICT, neither librosa or sondfile can output mp3 files. This means every time we load each examples, librosa will perform resampling...\n",
    "\n",
    "I guess we can live with this bit of slowness here and hope this will be much faster on kaggle where hopefully we will not need to peform resampling on the fly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "import multiprocessing\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=UserWarning)\n",
    "\n",
    "dl = DataLoader(valid_ds, batch_size=128, num_workers=multiprocessing.cpu_count() // 4)\n",
    "\n",
    "preds = []\n",
    "targs = []\n",
    "\n",
    "for batch in dl:\n",
    "    with torch.no_grad():\n",
    "        preds.append(model(batch[0].cuda()).cpu().detach())\n",
    "    targs.append(batch[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = torch.cat(preds)\n",
    "targs = torch.cat(targs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 310,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preds_to_tp_fp_fn(preds, targs):\n",
    "    positives = (preds >= 0.5)\n",
    "    true_positives = positives[targs == 1]\n",
    "    false_positives = positives[targs != 1]\n",
    "    negatives = ~positives\n",
    "    false_negatives = negatives[targs == 1]\n",
    "    return true_positives.sum(), false_positives.sum(), false_negatives.sum()\n",
    "\n",
    "def precision(preds, targs):\n",
    "    tp, fp, fn = preds_to_tp_fp_fn(preds, targs)\n",
    "    return (tp.float() / (tp + fp)).item()\n",
    "\n",
    "def recall(preds, targs):\n",
    "    tp, fp, fn = preds_to_tp_fp_fn(preds, targs)\n",
    "    return (tp.float() / (tp + fn)).item()\n",
    "\n",
    "def f1(preds, targs, eps=1e-8):\n",
    "    prec = precision(preds, targs)\n",
    "    rec = recall(preds, targs)\n",
    "    return 2 * (prec * rec) / (prec + rec + eps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 311,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 311,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall(preds, targs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 312,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 312,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision(preds, targs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 313,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 313,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1(preds, targs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 314,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0), tensor(48), tensor(106))"
      ]
     },
     "execution_count": 314,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds_to_tp_fp_fn(preds, targs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This doesn't seem like stellar performance from our model!\n",
    "\n",
    "We could have some bug in our training code. We could have an issue in how we calculate our metrics. But my guess is that the model cheated during training through how we sampled the validation set. Maybe it was not fitting to the calls at all but was learning some other charactericts of the audio files?\n",
    "\n",
    "Nonehteless, this is good. Making a pass through the entire pipeline all the way to predictions we have identified quite a few things that were not apparent from start and hopefully we will be able to use this information to iterate on the design.\n",
    "\n",
    "Time to move the weights to kaggle and create the piece for outputting the predictions there."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}