fastai-resnet18-color-coded-and-focalloss.ipynb

{"cells":[{"metadata":{"_uuid":"d35bf05d1ae58d483224e5bff58e83d1d206384e"},"cell_type":"markdown","source":"# Fastai conform dataset with resnet18"},{"metadata":{"_uuid":"65d39b3d7c4d3e2e102f9f70928cd02003cfc28c"},"cell_type":"markdown","source":"I created this kernel to get to know the fastai dataformats a bit better.\nI currently get 0.87 score, which is not too high but Ithink it should be easy to improve my approach.\n\nI had to fix a tiny bug in the lr_find2 so I'll import my hotfix of fastai. The problem was that some augmentations changed the shape of an image if it only has a single channel.\nFor now the runtime of 9hrs on kernels is the main bottleneck and im not sure how to fix this,\nI can currently train 12 cycles with 5k images per category. If anyone has Ideas for improvements go ahead.\n\nMy next Ideas would be trying some different augmentations and image sizes.\nAlso an Idea would be tuning the learning hyperparameters for the 1 cycle learning policy as described here:\n    \n1. [blog post by Sylvain Gugger summarizing the following papers](https://sgugger.github.io/the-1cycle-policy.html#the-1cycle-policy)\n2. [original papers by leslie smith on hyperparameter tuning](https://arxiv.org/pdf/1803.09820.pdf) \n3. [ and Superconvergence, the 1 cycle policy learning](https://arxiv.org/pdf/1708.07120.pdf)\n"},{"metadata":{"_uuid":"afcd459ceccf0651f9cf094cd4922722d812cad3"},"cell_type":"markdown","source":"# Imports and definition of necessary functions"},{"metadata":{"trusted":true,"_uuid":"6edc5fc8bb94b2098436379e7ebbf0313380f168","_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"!mkdir fastai_lib;cd fastai_lib;git clone https://github.com/rpauli/fastai.git;ln -s fastai_lib/fastai/old/fastai/ ../","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:00.859938Z","start_time":"2018-11-07T10:14:00.804947Z"},"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_kg_hide-input":false,"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport ast\nimport json\n\nimport numpy as np\nimport os\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.utils import shuffle\n\nfrom fastai.imports import *\n\nfrom fastai.transforms import *\nfrom fastai.conv_learner import *\nfrom fastai.model import *\nfrom fastai.dataset import *\nfrom fastai.sgdr import *\nfrom fastai.plots import *\n\n%load_ext autoreload\n%autoreload","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c5db859ebc939a785a68461a7d203b7d85b6e790"},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f805cf01889949f467042da603288c640cc02ac"},"cell_type":"markdown","source":"### Thanks to  [Belugas Kernel](https://www.kaggle.com/gaborfodor/greyscale-mobilenet-lb-0-892/notebook)"},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:04.070876Z","start_time":"2018-11-07T10:14:04.064523Z"},"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"\ndef f2cat(filename: str) -> str:\n    return filename.split('.')[0]\n\ndef list_all_categories():\n    files = os.listdir(os.path.join(path_base, path_train))\n    return sorted([f2cat(f) for f in files], key=str.lower)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7085e025348fc4afe84ec820ff9e1d7051a9ab5d"},"cell_type":"markdown","source":"## Define apk Metric and FocalLoss"},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:04.515215Z","start_time":"2018-11-07T10:14:04.495670Z"},"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"def calc_apk(pred,actual,k=3):\n    \"\"\"\n    pred 2D array, \n    axis 0 is predicted items\n    axis 1 is probabilities per label\n    \"\"\"\n    pred=to_np(pred)\n    actual=to_np(actual)\n    top_idxs=top_k_preds(pred,k)\n    idx_0=(np.where(actual==1)[1][:,None]-top_idxs)\n    idx_correct=idx_0==0\n    weighted_scores=1/np.arange(1,k+1)\n    score_per_data=weighted_scores[np.argmax(idx_correct,axis=1)]*np.max(idx_correct,axis=1)\n    mean_score=np.mean(score_per_data)\n    return mean_score\n\ndef top_k_preds(pred,k):\n    top_idxs=np.argsort(-to_np(pred),axis=1)[:,:k]\n    return top_idxs\n\ndef acc_metric(preds,actual):\n    return accuracy_score(np.where(actual==1)[1],np.argmax(to_np(preds),axis=1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1b79623863d47d5324a382b02158f442990bdfdd"},"cell_type":"code","source":"class FocalLoss(nn.Module):\n    def __init__(self, alpha=1, gamma=1, reduce=True):\n        super(FocalLoss, self).__init__()\n        self.alpha = alpha\n        self.gamma = gamma\n        self.reduce = reduce\n\n    def forward(self, inputs, targets):\n        \n        BCE_loss = F.binary_cross_entropy(inputs, targets, reduce=False)\n        pt = torch.exp(-BCE_loss)\n        F_loss = self.alpha * (1-pt)**self.gamma * BCE_loss\n\n        if self.reduce:\n            return torch.mean(F_loss)\n        else:\n            return F_loss","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"},"cell_type":"markdown","source":"## Read in image from raw stroke [(again thanks @Beluga)](https://www.kaggle.com/gaborfodor/greyscale-mobilenet-lb-0-892/notebook) \nThis version color codes the strokes from red to green in order, and the blue channel is the stroke velocity."},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:05.019771Z","start_time":"2018-11-07T10:14:04.995442Z"},"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"def draw_cv2(raw_strokes, size=256, lw=6, time_color=True):\n    img = np.zeros((BASE_SIZE, BASE_SIZE,3), np.float32)\n    t_max=max(1,len(raw_strokes))\n    \n    velos=[np.sqrt(np.sum(np.square(np.diff(stroke)),axis=0)).astype(int) for stroke in raw_strokes]\n    velo_min=np.min([np.min(velo) for velo in velos])\n    velo_max=max(1,np.max([np.max(velo) for velo in velos]))\n    for t, stroke in enumerate(raw_strokes):\n        velo=velos[t]\n        velo=(velo)*0.9/(velo_max)+0.1\n        velo*=255\n        for i in range(len(stroke[0]) - 1):\n            try:\n                color = (255 - int(t*255./t_max), int(t*255./t_max), int(velo[i])) if time_color else 255    \n            except:\n                print(t,t_max,t*255./t_max,int(t*255./t_max),int(velo[i]))\n            _ = cv2.line(img, (stroke[0][i], stroke[1][i]),\n                         (stroke[0][i + 1], stroke[1][i + 1]), color, lw)\n    img/=255\n    if size != BASE_SIZE:\n        return cv2.resize(img, (size, size))\n    else:\n        return img","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3da25e6d2c07752aadc533cadf0aed3e6281dfff"},"cell_type":"markdown","source":"## Define Dataset and Dataloader using the fastai library data format"},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:05.882891Z","start_time":"2018-11-07T10:14:05.731800Z"},"_kg_hide-input":false,"_uuid":"7cf63ae2881b782e041c09f5d5f4715695323189","trusted":true},"cell_type":"code","source":"class DoodleDataset(Dataset):\n    \"\"\"Adapted the fastai Dataset class for my needs, mainly the get_x function\"\"\"\n    def __init__(self, x, y, transform=None,sz=64):\n        self.x = x\n        self.y = y\n        self.transform = transform\n        self.n = self.get_n()\n        self.c = self.get_c()\n        self.sz = sz\n    @abstractmethod\n    def get_x(self, i):\n        return draw_cv2(json.loads(self.x[i]),self.sz)\n    @abstractmethod\n    def get_y(self, i):\n        y= np.zeros(self.c,dtype=np.float)\n        y[int(self.y[i])]=1\n        return y\n    @property\n    def is_multi(self): return True\n    @property\n    def is_reg(self):return False\n    #this flag is set to remove the output sigmoid that allows log(sigmoid) optimization\n    #of the numerical stability of the loss function\n    \n    def get1item(self, idx):\n        x,y = self.get_x(idx),self.get_y(idx)\n        return self.get(self.transform, x, y)\n\n\n    def __getitem__(self, idx):\n        if isinstance(idx,slice):\n            xs,ys = zip(*[self.get1item(i) for i in range(*idx.indices(self.n))])\n            return np.stack(xs),ys\n        return self.get1item(idx)\n\n    def __len__(self): return self.n\n\n    def get(self, tfm, x, y):     \n        return (x,y) if tfm is None else tfm(x,y)    \n    \n    @abstractmethod\n    def get_n(self):\n        return len(self.y)\n\n    @abstractmethod\n    def get_c(self):\n        return len(np.unique(self.y))\n\n    @abstractmethod\n    def get_sz(self):\n        return self.sz\n\n\n    @property\n    def is_multi(self):\n        \"\"\"Returns true if this data set contains multiple labels per sample.\"\"\"\n        return True\n\n\n\nclass DoodleImageData(ImageData):\n\n    def get_dl(self, ds, shuffle):\n        if ds is None: return None\n        return DataLoader(ds, batch_size=self.bs, shuffle=shuffle,\n            num_workers=self.num_workers, pin_memory=False)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"659a1a69241d010450c72d40cdc669322669da63"},"cell_type":"code","source":"def show_img(ims, idx, figsize=(5,5), normed=True, ax=None):\n    if ax is None: fig,ax = plt.subplots(figsize=figsize)\n    if normed: ims = np.rollaxis(to_np(ims),1,4);ims*=0.07367;ims+=0.09587;\n    else:      ims = np.rollaxis(to_np(ims),1,4)\n    ax.imshow(np.clip(ims,0,1)[idx,:,:,:])\n    ax.axis('off')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"77963a1a5e50952ed51986974b5d1a029da13e26"},"cell_type":"markdown","source":"## Defining the learn object based on the CNN objects from the fastai library"},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:06.384079Z","start_time":"2018-11-07T10:14:06.238478Z"},"_kg_hide-input":false,"_uuid":"df0ff323db0fc2e913157b6eff721485b5bb350d","trusted":true},"cell_type":"code","source":"class DoodleConvnetBuilder(ConvnetBuilder):\n    def __init__(self, f, c, is_multi, is_reg, ps=None, xtra_fc=None, xtra_cut=0, custom_head=None, pretrained=True):\n        self.f,self.c,self.is_multi,self.is_reg,self.xtra_cut = f,c,is_multi,is_reg,xtra_cut\n        if xtra_fc is None: xtra_fc = [512]\n        if ps is None: ps = [0.25]*len(xtra_fc) + [0.5]\n        self.ps,self.xtra_fc = ps,xtra_fc\n\n        if f in model_meta: cut,self.lr_cut = model_meta[f]\n        else: cut,self.lr_cut = 0,0\n        cut-=xtra_cut\n        layers = cut_model(f(pretrained), cut)\n        \n        layers[0] = nn.Conv2d(1,64,kernel_size=(7,7),stride=(2,2),padding=(3, 3), bias=False)\n        \n        self.nf = model_features[f] if f in model_features else (num_features(layers)*2)\n        if not custom_head: layers += [AdaptiveConcatPool2d(), Flatten()]\n        self.top_model = nn.Sequential(*layers)\n\n        n_fc = len(self.xtra_fc)+1\n        if not isinstance(self.ps, list): self.ps = [self.ps]*n_fc\n\n        if custom_head: fc_layers = [custom_head]\n        else: fc_layers = self.get_fc_layers()\n        self.n_fc = len(fc_layers)\n        self.fc_model = to_gpu(nn.Sequential(*fc_layers))\n        if not custom_head: apply_init(self.fc_model, kaiming_normal)\n        self.model = to_gpu(nn.Sequential(*(layers+fc_layers)))\n        \n\n        \nclass DoodleConvLearner(ConvLearner):\n    @classmethod\n    def pretrained(cls, f, data, ps=None, xtra_fc=None, xtra_cut=0, custom_head=None, precompute=False,\n                   pretrained=True, **kwargs):\n        models = DoodleConvnetBuilder(f, data.c, data.is_multi, data.is_reg,\n            ps=ps, xtra_fc=xtra_fc, xtra_cut=xtra_cut, custom_head=custom_head, pretrained=pretrained)\n        return cls(data, models, precompute, **kwargs)\n\n    @classmethod\n    def lsuv_learner(cls, f, data, ps=None, xtra_fc=None, xtra_cut=0, custom_head=None, precompute=False,\n                  needed_std=1.0, std_tol=0.1, max_attempts=10, do_orthonorm=False, **kwargs):\n        models = DoodleConvnetBuilder(f, data.c, data.is_multi, data.is_reg,\n            ps=ps, xtra_fc=xtra_fc, xtra_cut=xtra_cut, custom_head=custom_head, pretrained=False)\n        convlearn=cls(data, models, precompute, **kwargs)\n        convlearn.lsuv_init()\n        return convlearn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"916b3b7cb2ea0a6995844bded846c4484f8e502b"},"cell_type":"code","source":"def get_stats(X_trn,X_val,y_trn,y_val,N=100):\n    bs=340\n    num_workers=4 # Apprently 2 cpus per kaggle node\n    sz=64\n    aug_tfms = None\n    #dummy transformations, i subtract mean 0 and devide by 1 so it shouldnt change the stats\n    tfms = tfms_from_stats((np.array([0.,0.,0.]), np.array([1.,1.,1.])), sz=sz, aug_tfms=aug_tfms,tfm_y=TfmType.NO)\n    datasets=DoodleImageData.get_ds(DoodleDataset,(X_trn,y_trn),(X_val, y_val),tfms=tfms,sz=sz)\n    doodleimageloader=DoodleImageData('.',datasets,bs,num_workers,enc.classes_)\n    mn,var=[],[]\n    for i,(x,y) in tqdm(enumerate(iter(doodleimageloader.trn_dl))):\n        batch=to_np(x).swapaxes(1,3).reshape([-1,3])\n        mn.append(np.mean(batch,axis=0))\n        var.append(np.var(batch,axis=0))\n        if i>N:\n            break\n    channel_mean,channel_var=np.mean(mn,axis=0),np.mean(var,axis=0)\n    return channel_mean,channel_var","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:17.689887Z","start_time":"2018-11-07T10:14:17.573794Z"},"_uuid":"c87c2006d5a57fe76540d8434e03b33c879ee9f5","trusted":true},"cell_type":"code","source":"def get_set(N_cat,N_rows,N_skip):\n    if N_cats < len(list_all_categories()):\n        df = pd.concat([pd.read_csv(f'{path_base}/{path_train}/{cat}.csv',usecols=['drawing','word','key_id','recognized'],nrows=N_rows,skiprows=np.arange(1,N_skip+1)) for cat in  tqdm(np.random.choice(list_all_categories(),N_cats,replace=False))])\n    elif  N_cats == len(list_all_categories()):\n        df = pd.concat([pd.read_csv(f'{path_base}/{path_train}/{cat}.csv',usecols=['drawing','word','key_id','recognized'],nrows=N_rows,skiprows=np.arange(1,N_skip+1)) for cat in  tqdm(list_all_categories())])\n    else:\n        print('Somethings wrong')\n    df_cleaned=df[df.recognized==True].drop(['recognized'],axis=1)\n    #df_cleaned=df.drop(['recognized'],axis=1)\n    df_cleaned.word=df_cleaned.word.str.replace(' ','_')\n    return df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3a8938cd90625daf95fc38f07d9eb27952f08614"},"cell_type":"markdown","source":"# Preprocessing"},{"metadata":{"_uuid":"6f99af1802645835200f07223679c5f859ce243f"},"cell_type":"markdown","source":"I can currently only run about 8k samples with the 9h time limit on kaggle, a good size for the validation set seems to be 100 samples per category from what a read of other kernels and the discussion. I will also disregard any unregognized and later see if it makes a difference. In some cases there were class impurities where the doodle was missclassified, I hope those would drop out for regognized=True"},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:16.694299Z","start_time":"2018-11-07T10:14:16.675499Z"},"_uuid":"d3231e3f9d45c843f0e2bf73ead2b14d01f484a3","trusted":true},"cell_type":"code","source":"path_base='../input'\npath_train='train_simplified'\n# path_base='.'\n# path_train='train'","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:16.694299Z","start_time":"2018-11-07T10:14:16.675499Z"},"_uuid":"d3231e3f9d45c843f0e2bf73ead2b14d01f484a3","trusted":true},"cell_type":"code","source":"np.random.seed(0)\n#subsample for now\nN_cats=len(list_all_categories())\nN_rows_train=10000 #I can currently do about 8k samples which is not nearly enough\nN_rows_val=100 # This is a common value among kagglers 100 items per category to validate\nN_cats","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f027dfa4672715e2b105593b5134515fa36a6a23"},"cell_type":"markdown","source":"### Read in subset of row and categories"},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:17.689887Z","start_time":"2018-11-07T10:14:17.573794Z"},"_uuid":"c87c2006d5a57fe76540d8434e03b33c879ee9f5","trusted":true},"cell_type":"code","source":"df_val=get_set(N_cats,N_rows_val,0)\ndf_trn=get_set(N_cats,N_rows_train,N_rows_val)\ndf_val=df_val.set_index('key_id')\ndf_trn=df_trn.set_index('key_id')\ndf_val.index.intersection(df_trn.index)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"39117dd1a75d2b035c5d46dc9afd957650212c5a"},"cell_type":"markdown","source":"### load in test data"},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:19.460912Z","start_time":"2018-11-07T10:14:19.457949Z"},"_uuid":"3bc9e78b86770d11e856e767dd8780258e493ced","_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"test_file='test_simplified.csv'\ndf_test=pd.read_csv(f'{path_base}/{test_file}')\ndf_test.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"04abb02e7303880a63916059500e2c5cdcaeddc0"},"cell_type":"markdown","source":"\n## Ordinal encoding of labels and splitting into train and validation set\nAlso shuffle samples, apparently sklearn.utils.shuffle is the fastest method [ apparently sklearn.utils.shuffle is the fastest method](https://stackoverflow.com/questions/29576430/shuffle-dataframe-rows)"},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:20.565736Z","start_time":"2018-11-07T10:14:20.556156Z"},"_uuid":"8b7eec228a32f9e6fb23ea186fc3ef4489290339","trusted":true},"cell_type":"code","source":"enc=LabelEncoder()\nenc.fit(df_trn.word)\ndf_trn=shuffle(df_trn)\ndf_val=shuffle(df_val)\ny_trn=enc.transform(df_trn.word)\ny_val=enc.transform(df_val.word)\nX_trn=df_trn.drawing.values\nX_val=df_val.drawing.values","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:20.565736Z","start_time":"2018-11-07T10:14:20.556156Z"},"_uuid":"8b7eec228a32f9e6fb23ea186fc3ef4489290339","trusted":true},"cell_type":"code","source":"bs=256\nnum_workers=4 # Apprently 2 cpus per kaggle node\nBASE_SIZE=256\nsz=128\narch=resnet18","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"28b250c4c60109ce56ee91244d858f3c9f58b25b"},"cell_type":"markdown","source":"## Fill datasets and corresponding dataloader and model architecture (Resnet18)\nThe Image statistics of the doodle will be wildly different from imagenet, so I load in N batches of doodles and calculate channel mean and variance for proper normalization"},{"metadata":{"trusted":true,"_uuid":"7d98ad0eddde92c33fa2a4414f08c795cadbd25f","scrolled":true},"cell_type":"code","source":"channel_mean,channel_var=get_stats(X_trn,X_val,y_trn,y_val,50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a409028765813d8916564a5a36d431820682e143"},"cell_type":"code","source":"channel_mean,channel_var","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:21.791105Z","start_time":"2018-11-07T10:14:21.781495Z"},"_uuid":"f71f6426a22e212036226b6eeb7586ad23886fa1","trusted":true},"cell_type":"code","source":"aug_tfms = transforms_side_on+[RandomRotate(10)]\ntfms = tfms_from_stats((channel_mean, channel_var), sz=sz, aug_tfms=aug_tfms)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5c14024f33ca894beb4cd45ecc45af8ea360e5a1"},"cell_type":"markdown","source":"# Load Datasets in fastai format"},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:22.542510Z","start_time":"2018-11-07T10:14:22.534602Z"},"trusted":true,"_uuid":"5b53db851550fca3fcbc46673b9052d9ed777d4f"},"cell_type":"code","source":"datasets=DoodleImageData.get_ds(DoodleDataset,(X_trn,y_trn),(X_val, y_val),tfms=tfms,sz=sz,test=df_test.drawing.values)\ndatasets_warmup=DoodleImageData.get_ds(DoodleDataset,(X_trn[:200*bs],y_trn[:200*bs]),(X_val, y_val),tfms=tfms,sz=sz,test=df_test.drawing.values)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:23.258185Z","start_time":"2018-11-07T10:14:23.253558Z"},"_uuid":"f71f6426a22e212036226b6eeb7586ad23886fa1","trusted":true},"cell_type":"code","source":"doodleplot=DoodleImageData('.',datasets_warmup,8,num_workers,enc.classes_)\ndoodle_warmup=DoodleImageData('.',datasets_warmup,bs,num_workers,enc.classes_)\ndoodleimageloader=DoodleImageData('.',datasets,bs,num_workers,enc.classes_)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"860a8f5f1d8e1709149eca1084b243894ab51b9a"},"cell_type":"markdown","source":"#### Some images,time information is encoded in the brightness\nSome of these drawings are absolutely hilarious"},{"metadata":{"trusted":true,"_uuid":"6097e47d05a060ee0d3ea57443e2359101b358c5","_kg_hide-output":false,"_kg_hide-input":true},"cell_type":"code","source":"idx=0\nbatches = [next(iter(doodleplot.trn_dl)) for i in range(8)]\nfig, axes = plt.subplots(1,8, figsize=(18,9))\nfor i,(x,y) in enumerate(batches):\n    show_img(x,idx, ax=axes.flat[i])\n    axes.flat[i].set_title(enc.inverse_transform(np.where(y==1)[1][idx]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b9c663373d615d71f53ebdd70674352078d845bc"},"cell_type":"markdown","source":"#### An image and some augmentations,"},{"metadata":{"trusted":true,"_uuid":"80aba90f1343925ccbf83ce28494294455a9c254","_kg_hide-output":false,"_kg_hide-input":true},"cell_type":"code","source":"idx=2\nbatches = [next(iter(doodleplot.aug_dl)) for i in range(8)]\nfig, axes = plt.subplots(1,8, figsize=(18,9))\nfor i,(x,y) in enumerate(batches):\n    show_img(x,idx, ax=axes.flat[i])\n    axes.flat[i].set_title(enc.inverse_transform(np.where(y==1)[1][idx]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"852bdcc2d62eddbe79b097e90796f3346a36771f"},"cell_type":"markdown","source":"# Start learning stuff!"},{"metadata":{"_uuid":"057b3a29b70417c62f66e1c92325dd1aca92bbce"},"cell_type":"markdown","source":"This is the learner object, it has all the fancy learning rate finding and cyclicle learning rate with momentum methods."},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:24.734838Z","start_time":"2018-11-07T10:14:24.349279Z"},"_uuid":"f71f6426a22e212036226b6eeb7586ad23886fa1","_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"learn=ConvLearner.pretrained(arch,doodle_warmup,metrics=[acc_metric,calc_apk])\nlearn.crit=FocalLoss()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ff2a58906656ab08dbd5e6796833c54019872b4c"},"cell_type":"markdown","source":"## Logarithmic sweep to check for a good learning rate"},{"metadata":{"_uuid":"eccad6edf0235b71c88b6acf086b4d07d2453d8f"},"cell_type":"markdown","source":"An Appropriate learning rate should be on a slope but before the minimum.\n[(Great Explanation)](https://towardsdatascience.com/understanding-learning-rates-and-how-it-improves-performance-in-deep-learning-d0d4059c1c10)\nAn important thing to know is, in the original paper, the minimum is mentioned as the best value.\nTo smooth the curve a running average is used in the fastai library, so we have to take an earlier value. Something like minimum/10 works well for me."},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:24.734838Z","start_time":"2018-11-07T10:14:24.349279Z"},"_uuid":"f71f6426a22e212036226b6eeb7586ad23886fa1","_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"learn.lr_find(end_lr=1000,start_lr=0.0001)","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T10:14:24.734838Z","start_time":"2018-11-07T10:14:24.349279Z"},"_uuid":"f71f6426a22e212036226b6eeb7586ad23886fa1","_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"learn.sched.plot(n_skip_end=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e33f10f84b88358db5a03c6968d82c327b504dec"},"cell_type":"markdown","source":"## For now the layers are frozen except the last one to not unlearn all the tuned weights\n"},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T09:27:27.329421Z","start_time":"2018-11-07T09:26:15.314970Z"},"_uuid":"078f243806441ec41c5b3069a02c3aed957288ee","scrolled":true,"trusted":true},"cell_type":"code","source":"lr=10\nlearn.fit(lr,1,cycle_len=1,use_clr_beta = (10,10,0.95,0.85))","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T09:27:58.857618Z","start_time":"2018-11-07T09:27:58.693446Z"},"trusted":true,"_uuid":"464a0b14503c1c7ac4e5731a3bddfcf4aab88ddd"},"cell_type":"code","source":"learn.sched.plot_loss()\nlearn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"26dc61651f22b651287980ecae2928f41fc57371"},"cell_type":"markdown","source":"## Unfreeze network and check again if lr is appropriate"},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T09:28:10.489664Z","start_time":"2018-11-07T09:28:10.445986Z"},"trusted":true,"_uuid":"837f7303210a80730f1dc6bc6368d40bb7d80012"},"cell_type":"code","source":"learn.save('freeze')","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T09:28:10.739392Z","start_time":"2018-11-07T09:28:10.688417Z"},"trusted":true,"_uuid":"febd04c9f5acaa7fe1b876eb78b238221b9cb6dd"},"cell_type":"code","source":"learn.load('freeze')","execution_count":null,"outputs":[]},{"metadata":{"ExecuteTime":{"end_time":"2018-11-07T09:28:11.019771Z","start_time":"2018-11-07T09:28:11.014292Z"},"_uuid":"8db1b8831f4a1ff5b5c80646cd9e17fc227b03f7","trusted":true},"cell_type":"code","source":"learn.unfreeze()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"baa58b8a7fbb715f1cbca1e6ab401211c18ca82f"},"cell_type":"code","source":"learn.lr_find(end_lr=10000,start_lr=0.0001)\nlearn.sched.plot(n_skip_end=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dcd3f3630012b1d52c60aac08bc2508a6dbda063"},"cell_type":"markdown","source":"## Learning rate for earlier layers is supposed to be lower"},{"metadata":{"_uuid":"cdd6100ca5b89b932cbb57f865cda1df28e03a89"},"cell_type":"markdown","source":"The learning rate from the layers is the optimal learning rate from the find_lr method. We also decrease the learning rate for earlier layers so all the learned features from the resnet18 are not unlearned."},{"metadata":{"_uuid":"ec9b44a2ec6b5ae74fc8251bfb4f8b1b17e8e3bf","trusted":true},"cell_type":"code","source":"lr=100\nlrs=[lr/9,lr/3,lr]\nlearn.set_data(doodleimageloader)\nlearn.load('freeze')\nlearn.unfreeze()\nlearn.fit(lrs,1,cycle_len=1,use_clr_beta = (10,10,0.95,0.85))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c599495413194083f89845dc12bb6ad061d28000","trusted":true},"cell_type":"code","source":"learn.sched.plot_loss()\n\nlearn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"12a1757d8168a73813c267a0bdc898db91a12532"},"cell_type":"code","source":"learn.save('unfreeze')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fb98568164f440e5dedce49af5d5a0096c03180b"},"cell_type":"code","source":"learn.load('unfreeze')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"877d1304b0229c3db661e715b441f746ebc3cbf0"},"cell_type":"markdown","source":"### Predit the validation data using TTA\nHere for every image we want to predict on, n_augs images are augmented form the original image.\nWe can then compare the predictions on for example the image and the image flipped / roated / slightly different crop/ lighting/stretched etc. \nFor now only the diherdral and rotations are used. THis gives a nice extra percent or two when compared to the auc above after training where not TTA is used. \nI also test if mean or max is better to use on the image and its augments but it can't conclude anything yet.\n"},{"metadata":{"_uuid":"574b0397190f84e7fb497efe94168be86764a5fe","trusted":true},"cell_type":"code","source":"log_preds_test,y_test=learn.TTA(is_test=True,n_aug=16)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"40457af4047465634e9b4df5844e6fe2d38e0281"},"cell_type":"code","source":"mean_test_preds=np.mean(np.exp(log_preds_test),0)\nmax_test_preds=np.max(np.exp(log_preds_test),0)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f01f50c8f24445baca4860ec7845df24d87d23ae"},"cell_type":"code","source":"def prepare_submission(preds,fname):\n    preds_transformed=enc.inverse_transform(top_k_preds(preds,3))\n    preds_joined=[' '.join(words) for words in preds_transformed]\n    df_test['word']=preds_joined\n    df_test.loc[:,['key_id','word']].to_csv(fname,index=None)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"01d5b272b7951944322f158abca088e5270ca69d"},"cell_type":"markdown","source":"## I add the score to the name of the file so I can later plot the leaderboard score versus my validation score\nIn the fastai course Jeremy mentions that if you have a monotonic relation between validation and LB score the way you set up your validation set matches what the test set consists of."},{"metadata":{"trusted":true,"_uuid":"958ecacccd97670b255755689d6f2a38bced0e53"},"cell_type":"code","source":"prepare_submission(mean_test_preds,f'submission_mean.csv')\nprepare_submission(max_test_preds,f'submission_max.csv')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cf5e2d4f0202726592ad540749f4c40636c56e67"},"cell_type":"code","source":"!ls","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a01ad8a136a8f9cca0b76d3080c1ba237fd7c904"},"cell_type":"code","source":"!rm fastai","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c2b603d992957f2f097a085b921ea207719c7e71"},"cell_type":"code","source":"!rm -r fastai_lib/","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"latex_envs":{"LaTeX_envs_menu_present":true,"autocomplete":true,"bibliofile":"biblio.bib","cite_by":"apalike","current_citInitial":1,"eqLabelWithNumbers":true,"eqNumInitial":1,"hotkeys":{"equation":"Ctrl-E","itemize":"Ctrl-I"},"labels_anchors":false,"latex_user_defs":false,"report_style_numbering":false,"user_envs_cfg":false},"toc":{"nav_menu":{},"number_sections":true,"sideBar":true,"skip_h1_title":false,"title_cell":"Table of Contents","title_sidebar":"Contents","toc_cell":false,"toc_position":{"height":"calc(100% - 180px)","left":"10px","top":"150px","width":"165px"},"toc_section_display":true,"toc_window_display":true},"varInspector":{"cols":{"lenName":16,"lenType":16,"lenVar":40},"kernels_config":{"python":{"delete_cmd_postfix":"","delete_cmd_prefix":"del ","library":"var_list.py","varRefreshCmd":"print(var_dic_list())"},"r":{"delete_cmd_postfix":") ","delete_cmd_prefix":"rm(","library":"var_list.r","varRefreshCmd":"cat(var_dic_list()) "}},"types_to_exclude":["module","function","builtin_function_or_method","instance","_Feature"],"window_display":false},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}