diff --git a/README.md b/README.md index 6223031..60ffc62 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ A multi-part seminar series on Large Language Models (LLMs). ![Session 1](images/home_page/Large%20Language%20Models.png) -

Large Language Models Full Topic List

+

Click here for Large Language Models Full Topic List

## ✨ [Emergence, Fundamentals and Landscape of LLMs](session_1) @@ -34,14 +34,12 @@ Explore diverse applications of Large Language Models (LLMs) and the frameworks Coming soon... -## ✨ Training and Evaluating LLMs On Custom Datasets +## ✨ [Training and Evaluating LLMs On Custom Datasets](session_4) Delve into the intricacies of training and evaluating Large Language Models (LLMs) on your custom datasets. Gain insights into optimizing performance, fine-tuning, and assessing model effectiveness tailored to your specific data. ![Session 4](images/home_page/Session%204.png) -Coming soon... - ## ✨ Optimizing LLMs For Inference and Deployment Techniques Learn techniques to optimize Large Language Models (LLMs) for efficient inference. Explore strategies for seamless deployment, ensuring optimal performance in real-world applications. @@ -50,7 +48,7 @@ Learn techniques to optimize Large Language Models (LLMs) for efficient inferenc Coming soon... -## ✨ Open Challanges With LLMs +## ✨ Open Challenges With LLMs Delve into the dichotomy of small vs large LLMs, navigating production challenges, addressing research hurdles, and understanding the perils associated with the utilization of pretrained LLMs. Explore the evolving landscape of challenges within the realm of Large Language Models. diff --git a/images/session_1/part_3_landscape_of_llms/Large Language Models-challanges.png b/images/session_1/part_3_landscape_of_llms/Large Language Models-challenges.png similarity index 100% rename from images/session_1/part_3_landscape_of_llms/Large Language Models-challanges.png rename to images/session_1/part_3_landscape_of_llms/Large Language Models-challenges.png diff --git a/images/session_4/part_2_finetuning_lms_to_human_preferences/base_attn.png b/images/session_4/part_2_finetuning_lms_to_human_preferences/base_attn.png new file mode 100644 index 0000000..0f7dffd Binary files /dev/null and b/images/session_4/part_2_finetuning_lms_to_human_preferences/base_attn.png differ diff --git a/images/session_4/part_2_finetuning_lms_to_human_preferences/kl0_attn.png b/images/session_4/part_2_finetuning_lms_to_human_preferences/kl0_attn.png new file mode 100644 index 0000000..57b136c Binary files /dev/null and b/images/session_4/part_2_finetuning_lms_to_human_preferences/kl0_attn.png differ diff --git a/images/session_4/part_2_finetuning_lms_to_human_preferences/kl0_plot.png b/images/session_4/part_2_finetuning_lms_to_human_preferences/kl0_plot.png new file mode 100644 index 0000000..e76055d Binary files /dev/null and b/images/session_4/part_2_finetuning_lms_to_human_preferences/kl0_plot.png differ diff --git a/images/session_4/part_2_finetuning_lms_to_human_preferences/kl1_attn.png b/images/session_4/part_2_finetuning_lms_to_human_preferences/kl1_attn.png new file mode 100644 index 0000000..d5a2956 Binary files /dev/null and b/images/session_4/part_2_finetuning_lms_to_human_preferences/kl1_attn.png differ diff --git a/images/session_4/part_2_finetuning_lms_to_human_preferences/rl_outputs.png b/images/session_4/part_2_finetuning_lms_to_human_preferences/rl_outputs.png new file mode 100644 index 0000000..2a204f5 Binary files /dev/null and b/images/session_4/part_2_finetuning_lms_to_human_preferences/rl_outputs.png differ diff --git a/images/site/infocusp_logo_blue.png b/images/site/infocusp_logo_blue.png new file mode 100644 index 0000000..121623f Binary files /dev/null and b/images/site/infocusp_logo_blue.png differ diff --git a/images/site/infocusp_logo_blue.svg b/images/site/infocusp_logo_blue.svg deleted file mode 100644 index 39cc1bc..0000000 --- a/images/site/infocusp_logo_blue.svg +++ /dev/null @@ -1,5 +0,0 @@ - - - --> - - diff --git a/mkdocs.yaml b/mkdocs.yaml index 886ddaf..290715d 100644 --- a/mkdocs.yaml +++ b/mkdocs.yaml @@ -5,7 +5,7 @@ docs_dir: . site_dir: ../site theme: name: material - # logo: assets/icons8-code-64.png + logo: images/site/infocusp_logo_blue.png palette: primary: white features: @@ -22,6 +22,7 @@ theme: markdown_extensions: - toc: permalink: true + toc_depth: 3 - pymdownx.highlight: anchor_linenums: true line_spans: __span @@ -41,6 +42,8 @@ extra_css: extra: generator: false social: + - icon: fontawesome/solid/globe + link: https://infocusp.com - icon: fontawesome/brands/linkedin link: https://in.linkedin.com/company/infocusp - icon: fontawesome/brands/github @@ -51,4 +54,6 @@ extra: link: https://twitter.com/_infocusp plugins: - search - - same-dir \ No newline at end of file + - same-dir + - mkdocs-jupyter: + ignore_h1_titles: True \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index 8be8bd5..df62450 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,5 @@ mkdocs>=1.2.2 mkdocs-material>=7.1.11 mkdocs-static-i18n>=0.18 -mkdocs-same-dir>=0.1.3 \ No newline at end of file +mkdocs-same-dir>=0.1.3 +mkdocs-jupyter>=0.16.1 \ No newline at end of file diff --git a/session_1/README.md b/session_1/README.md index eeb5e1c..484b5ba 100644 --- a/session_1/README.md +++ b/session_1/README.md @@ -6,6 +6,7 @@ Covers important building blocks of what we call an LLM today, where they came from, etc. and then we'll dive into the deep universe that has sprung to life around these LLMs. This session is aimed to help: + * People who are new to LLMs * People who have just started working on them * People who are working on different use cases surrounding LLMs and need a roadmap. diff --git a/session_1/part_3_landscape_of_llms/README.md b/session_1/part_3_landscape_of_llms/README.md index 23b2e28..da6b995 100644 --- a/session_1/part_3_landscape_of_llms/README.md +++ b/session_1/part_3_landscape_of_llms/README.md @@ -465,9 +465,9 @@ -## Challanges with LLMs +## Challenges with LLMs -![Challanges with LLMs](./../../images/session_1/part_3_landscape_of_llms/Large%20Language%20Models-challanges.png) +![Challenges with LLMs](./../../images/session_1/part_3_landscape_of_llms/Large%20Language%20Models-challenges.png)
diff --git a/session_4/README.md b/session_4/README.md new file mode 100644 index 0000000..6421874 --- /dev/null +++ b/session_4/README.md @@ -0,0 +1,34 @@ +# Session 4 - Training and Evaluating LLMs On Custom Datasets + +

Session 4

+ +This session aims to equip you with the knowledge to train Large Language Models (LLMs) by exploring techniques like unsupervised pretraining and supervised fine-tuning with various preference optimization methods. It will also cover efficient fine-tuning techniques, retrieval-based approaches, and language agent fine-tuning. Additionally, the session will discuss LLM training frameworks and delve into evaluation methods for LLMs, including evaluation-driven development and using LLMs for evaluation itself. + +This session is aimed to help: + +* People who are already familiar basics of LLMs and Transformers +* People who already knows how to use pre-trained LLMs prompt engineering and RAG +* People who want train or finetune their own LLMs on custom data. +* People who want to lear how to evaluate LLMs + +## Outline + +### Part 1: Training Foundational LLMs + +Coming soon... + +### Part 2: [Finetuning LMs To Human Preferences](part_2_finetuning_lms_to_human_preferences/RLHF.ipynb) + +#### Details + +* Date: 14 March, 2024 +* Speaker: [Abhor Gupta](https://in.linkedin.com/in/abhor-gupta-565386145) +* Location: [Infocusp Innovations LLP](https://www.infocusp.com/) + +#### Material + +* Recording: TODO + +### Part 3: LLM Training Frameworks + +Coming soon... diff --git a/session_4/part_2_finetuning_lms_to_human_preferences/RLHF.ipynb b/session_4/part_2_finetuning_lms_to_human_preferences/RLHF.ipynb new file mode 100644 index 0000000..3d261b9 --- /dev/null +++ b/session_4/part_2_finetuning_lms_to_human_preferences/RLHF.ipynb @@ -0,0 +1,1535 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "321adccc", + "metadata": {}, + "source": [ + "# RLHF - An Independent Illustration" + ] + }, + { + "cell_type": "markdown", + "id": "b5085966", + "metadata": {}, + "source": [ + "Reinforcement learning from human feedback (RLHF) is a transformative technique that enables us to fine-tune large language models (LLMs) or transformer-based models for improved alignment with our intended goals. This approach goes beyond the standard techniques that train LLMs on massive volumes of text data. RLHF uses human feedback to teach LLMs how to better adhere to our preferences and values." + ] + }, + { + "cell_type": "markdown", + "id": "119f232e", + "metadata": {}, + "source": [ + "There are several very well written blogs on the topic - [here](https://medium.com/towards-generative-ai/reward-model-training-2209d1befb5f), [here](https://medium.com/@madhur.prashant7/rlhf-reward-model-ppo-on-llms-dfc92ec3885f) and [here](https://huggingface.co/blog/rlhf). I am especially fond of the one written by Chip Huyen [here](https://huyenchip.com/2023/05/02/rlhf.html). **The intention behind writing this is to understand RLHF using a simple and _mostly_ self-contained implementation to solve a demonstrative problem.** Let it be sufficiently trivial that we may open up our model and visually observe some effects of RLHF using different techniques. " + ] + }, + { + "cell_type": "markdown", + "id": "01c86337", + "metadata": {}, + "source": [ + "## Overview" + ] + }, + { + "cell_type": "markdown", + "id": "c9b868df", + "metadata": {}, + "source": [ + "Let us first go over some basics." + ] + }, + { + "cell_type": "markdown", + "id": "514c9112", + "metadata": {}, + "source": [ + "### Training LLMs" + ] + }, + { + "cell_type": "markdown", + "id": "ab3275a8", + "metadata": {}, + "source": [ + "Can't help but love this _beautiful_ depiction of RLHF among the broader spectrum of an LLM's training, by twitter.com/anthrupad. \n", + "\n", + "![](https://huyenchip.com/assets/pics/rlhf/2-shoggoth.jpg)" + ] + }, + { + "cell_type": "markdown", + "id": "5fffa9f4", + "metadata": {}, + "source": [ + "The image above shows the different methods of training an LLM, their \"size\" in terms of the space of outputs they represent and their \"quality\" in terms of their social acceptance to humans:" + ] + }, + { + "cell_type": "markdown", + "id": "e728ca93", + "metadata": {}, + "source": [ + "1. **Unsupervised Learning**: _Train an LLM on a massive corpus of raw text; this teaches the LLM a language - the fundamentals of its structure, grammer and relationship between words._ In terms of its objective, the LLM is trained to predict the next word in a context. \n", + "But! Though an LLM may know a language, it doesn't necessarily know how to converse. In its space of outputs, it is aware of what it _can_ do, but not necessarily what it _should_ do. It is like the Shoggoth, massive but ugly. \n", + "2. **Supervised Fine-tuning**: _Tailor the LLM to specific tasks like translation, question-answering, or summarization._ Here, the LLM is trained on a set of input-output pairs demonstrating the task. \n", + "Following the example from the point above, this is akin to teaching the LLM how to converse. Its output space here is refined to answer in specific ways, perhaps with domain specific know-how, or in accordance to a particular task. This is like the deformed human face, you'll accept it but it's not necessarily very pleasing. \n", + "3. **RLHF**: _Refine the LLM's output to better align with human values, preferences, and safety guidelines._ The training here involves giving feedback signals on the LLM's output to guide it to some desired behavior. \n", + "Following the same context from (1) and (2), after it has learnt language and knows how to converse, it learns to adhere to the social norms. Within its output space, this is the refinement that narrows down the conversation ability of the LLM to answer in a way that pleases its general reader - ethical speech, truthful statements, intelligent use of vocabulary etc. It is the smiley face that you want to talk to. :)" + ] + }, + { + "cell_type": "markdown", + "id": "f50c1d47", + "metadata": {}, + "source": [ + "For the scope of this notebook, we will only be exploring RLHF. Supervised training will be a part - though it is more a requirement for the sake of thoroughness, than an intented guide on the topic. Therefore, I'll be using a very simple supervised pretraining that is closer to the description of supervised fine-tuning above, than unsupervised learning." + ] + }, + { + "cell_type": "markdown", + "id": "4d8b0884", + "metadata": {}, + "source": [ + "### RLHF components" + ] + }, + { + "cell_type": "markdown", + "id": "de807a3d", + "metadata": {}, + "source": [ + "A complete RLHF pipeline requires the following components:" + ] + }, + { + "cell_type": "markdown", + "id": "f5a40a2d", + "metadata": {}, + "source": [ + "1. **A pre-trained base model**: We begin with a pre-trained LLM. This is a powerful language model that has already learned the intricacies of language structure from vast amounts of text data. This may be followed by supervised fine-tuning to attune the LLM to a specific task like question-answering or summarization. \n", + "2. **Training a reward model from human feedback**: We then create a \"reward model\" specifically designed to capture human preferences. This involves gathering human feedback on various LLM outputs, where people rate the responses based on their desired qualities like helpfulness, safety, and adherence to instructions. By analyzing this feedback, the reward model learns to assign scores to future LLM responses, essentially mimicking human judgment.\n", + "3. **Fine tuning using Reinforcement Learning**: Finally, we put the LLM and the reward model to work together. When presented with a prompt, the LLM generates multiple potential responses. Each response is then analyzed by the reward model, which assigns a score based on its learned understanding of human preferences. Finally, a reinforcement learning algorithm like PPO uses these scores to fine-tune the LLM's internal parameters. Responses that received higher scores become more likely to be generated in the future, while those with lower scores are gradually downplayed. This iterative process progressively aligns the LLM's outputs with human expectations and values." + ] + }, + { + "cell_type": "markdown", + "id": "549a5801", + "metadata": {}, + "source": [ + "This pipeline effectively utilizes human feedback to bridge the gap between raw LLM capabilities and human-desired outcomes. It allows us to shape these powerful language models into not just masters of language, but also responsible and valuable tools aligned with our needs." + ] + }, + { + "cell_type": "markdown", + "id": "9709c5d5", + "metadata": {}, + "source": [ + "### Applications of RLHF" + ] + }, + { + "cell_type": "markdown", + "id": "d0881a0b", + "metadata": {}, + "source": [ + "The most prevelant example of RLHF being applied in AI is for text generation to align chatbots with human preferences ([InstructGPT](https://openai.com/research/instruction-following), [ChatGPT](https://openai.com/blog/chatgpt), [Gemini](https://blog.google/technology/ai/google-gemini-ai/) are famous examples). Similarly, RLHF has seen application in image generation as well ([ImageReward](https://arxiv.org/abs/2304.05977), [DPOK](https://arxiv.org/abs/2305.16381)). Though limited, some research groups have also explored its application in games ([DeepMind](https://deepmind.google/discover/blog/learning-through-human-feedback/) and [OpenAI](https://openai.com/research/learning-from-human-preferences)). \n", + "\n", + "Even though currently the applications of RLHF in AI are limited, the scope for RLHF is much wider. \n", + "\n", + "Do you use e-commerce websites like Amazon? Do you use Uber for requesting cabs? Do you use Google Maps for deciding which restaurant, bar or hospital to visit? You must have seen ratings for products, or people, or services, or food. You likely would have given some yourself. These are all examples of human feedback. And when these affect the product or service to comform to user satisfaction, it is also a form of RLHF. \n", + "\n", + "Take, for instance, cooking robots are a thing now ([Moley](https://www.moley.com/), [Nymble](https://www.eatwithnymble.com/)). For the food that is cooked by the robots based on some recipe, the recipe can be adjusted for duration of cooking, quantity of spices etc for user preference based on their feedback. Self-driving cars are also real now ([Waymo](https://waymo.com/), [Tesla](https://www.tesla.com/support/autopilot)). Based on customer's feedback, the ride be adjusted to be faster/slower, less jerky, smoother maneuverability." + ] + }, + { + "cell_type": "markdown", + "id": "bd574943", + "metadata": {}, + "source": [ + "In the next section, we will establish a small toy problem to solve using a tiny LLM. Then we will dive into each of the RLHF concepts in detail along with some code to establish an implementational understanding as well some nice visualizations to complement our findings." + ] + }, + { + "cell_type": "markdown", + "id": "581953f1", + "metadata": {}, + "source": [ + "## Problem Statement" + ] + }, + { + "cell_type": "markdown", + "id": "0f11a388", + "metadata": {}, + "source": [ + "To keep the scale of things simple, let us work with a \"language\" of numbers. Our vocabulary consist of digits 0-9 as well as a special digit 10 that separates our input and output. \n", + "\n", + "Typically for large LLMs, the language training is followed by a task specialisation training like question-answering before moving on to RLHF. To keep things simple, we avoid differentiating between language training and task specialisation and do a supervised training once to get our base model. " + ] + }, + { + "cell_type": "markdown", + "id": "fee373ac", + "metadata": {}, + "source": [ + "### Language (supervised learning)" + ] + }, + { + "cell_type": "markdown", + "id": "a4b32e14", + "metadata": {}, + "source": [ + "The structure of the language is that for the current output to be generated, one of the last four digits (in the whole sequence of input+output) is chosen and its increment modulo 10 is outputted.\n", + "\n", + "Given $a_1, a_2, ..., a_{n-1}, a_n$, $\\forall n > 4$,\n", + "$$a'_{n+1} \\sim \\{a_{n-3}, a_{n-2}, a_{n-1}, a_{n}\\}$$\n", + "$$a_{n+1} = (a'_{n+1} + 1)\\ \\%\\ 10$$" + ] + }, + { + "cell_type": "markdown", + "id": "75fc02dd", + "metadata": {}, + "source": [ + "For example, \n", + "Input: 4, 5, 9, 1\n", + "\n", + "Generation: \n", + "4, 5, 9, 1, 10, **6** \n", + "4, 5, 9, 1, 10, 6, **2** \n", + "4, 5, 9, 1, 10, 6, 2, **7** \n", + "4, 5, 9, 1, 10, 6, 2, 7, **8**\n", + "\n", + "The underlined digits are the sampling set for the output. The digit marked in red is the digit sampled. Finally, the digit bolded at the end of the sequence is generated as the increment of the sampled digit." + ] + }, + { + "cell_type": "markdown", + "id": "cf00c517", + "metadata": {}, + "source": [ + "### Alignment (RLHF)" + ] + }, + { + "cell_type": "markdown", + "id": "605ee56b", + "metadata": {}, + "source": [ + "Now we need a \"preference\" to incorporate. Mathematically, we wish to adjust the probability of some generation rule learned during the previous step.\n", + "\n", + "For this toy example, let's change only the generation of the first output token to be the increment of the last input token.\n", + "\n", + "That is, given $a_1, a_2, ..., a_{n-1}, a_n$, $\\forall n > 4$, \n", + "$$\n", + " a_{n+1}= \n", + "\\begin{cases}\n", + " a_n + 1,& \\text{if } {a_{n+1}} = y_1\\\\\n", + " (a'_{n+1} + 1)\\ \\%\\ 10, & \\text{otherwise}\n", + "\\end{cases}\n", + "$$\n", + "\n", + "Where $y_1$ is the first output token." + ] + }, + { + "cell_type": "markdown", + "id": "3d1e59b8", + "metadata": {}, + "source": [ + "Considering the same example as above, \n", + "Input: 4, 5, 9, 1\n", + "\n", + "Generation: \n", + "4, 5, 9, 1, 10, **2** (notice the change in underline) \n", + "4, 5, 9, 1, 10, 2, **2** \n", + "4, 5, 9, 1, 10, 2, 2, **3\n", + "** \n", + "4, 5, 9, 1, 10, 2, 2, 3, **4**\n", + "\n", + "For the first generation step, the model will only sample from the last digit and continue the original rule thereafter. Notice that except for the first generation step, the rest of the outputs are sampled from the same positions as the earlier example, yet the entire sequence has changed. Thus, though we wish to preserve the rule for the rest of the outputs, the actual outputs are not independent of the first output. Therefore, this small change in the output of a single token can have a cascading effect and lead to very different generations. \n", + "\n", + "This is in line with the spirit of RLHF, where say, if we want to reduce the toxicity, reducing the probability of toxic words will have a cascading effect and we do not need to affect the probability of several thousands of unrelated words." + ] + }, + { + "cell_type": "markdown", + "id": "f4178080", + "metadata": {}, + "source": [ + "## Code and Commentary" + ] + }, + { + "cell_type": "markdown", + "id": "151d002f", + "metadata": {}, + "source": [ + "Before RLHF: _Enough talking, show me the code!_ \n", + "After RLHF: _You've explained what we're doing here well enough. We would like to move on to the implementational details._ \n", + "\n", + "😉" + ] + }, + { + "cell_type": "markdown", + "id": "beca59c0", + "metadata": {}, + "source": [ + "### Supervised pre-training" + ] + }, + { + "cell_type": "markdown", + "id": "a2ba0080", + "metadata": {}, + "source": [ + "First we learn the language using supervised learning. I'm using Karpathy's [minGPT](https://github.com/karpathy/minGPT/tree/master) for the LLM and supervised training. " + ] + }, + { + "cell_type": "markdown", + "id": "338dbd44", + "metadata": {}, + "source": [ + "![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/rlhf/pretraining.png)\n", + "\n", + "Source: [HuggingFace - RLHF blog](https://huggingface.co/blog/rlhf)" + ] + }, + { + "cell_type": "markdown", + "id": "08b9ae9e", + "metadata": {}, + "source": [ + "_Imports --_" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "991285e6", + "metadata": {}, + "outputs": [], + "source": [ + "import torch\n", + "from torch.utils.data import Dataset\n", + "from torch.utils.data.dataloader import DataLoader\n", + "from mingpt.utils import set_seed\n", + "import numpy as np\n", + "set_seed(3407)\n", + "\n", + "device = 'cuda' if torch.cuda.is_available() else 'cpu' " + ] + }, + { + "cell_type": "markdown", + "id": "d6bde156", + "metadata": {}, + "source": [ + "_Hyperparams for size of vocab and length of input --_" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "125ad91b", + "metadata": {}, + "outputs": [], + "source": [ + "VOCAB_SIZE = 10\n", + "INPUT_SIZE = 4" + ] + }, + { + "cell_type": "markdown", + "id": "af08cdc7", + "metadata": {}, + "source": [ + "_Class for generating training pairs for supervised language learning --_" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "2bab82cc", + "metadata": {}, + "outputs": [], + "source": [ + "class SupervisedDataset(Dataset):\n", + " \"\"\" \n", + " Problem: Look at last 4 digits and sample one of them to output its increment. \n", + " \n", + " Input: 3 8 1 4 \n", + " Possible ouputs: 2 2 3 5 || 5 9 6 0 || 2 9 5 3 etc\n", + " \n", + " Which will feed into the transformer concatenated as:\n", + " input: 3 8 1 4 S 2 2 3\n", + " output: I I I I 2 2 3 5\n", + " where I is \"ignore\", and S is the separation token\n", + " \"\"\"\n", + "\n", + " def __init__(self):\n", + " self.EOS = VOCAB_SIZE\n", + " \n", + " def __len__(self):\n", + " return 10000 # ...\n", + " \n", + " def get_vocab_size(self):\n", + " return VOCAB_SIZE+1 # normal vocab + serparation token\n", + "\n", + " def __getitem__(self, idx):\n", + " inputs = torch.randint(VOCAB_SIZE, size=(INPUT_SIZE,), dtype=torch.long)\n", + " ouptputs = []\n", + " \n", + " # Create input output pairs\n", + " inp = np.random.randint(VOCAB_SIZE, size=(INPUT_SIZE,)).tolist()\n", + " sol = []\n", + " for i in range(INPUT_SIZE):\n", + " sol.append((np.random.choice(inp[i:] + sol) + 1)%10)\n", + " \n", + " # concatenate the problem specification and the solution\n", + " cat = torch.Tensor(inp + [self.EOS] + sol).long()\n", + "\n", + " # the inputs to the transformer will be the offset sequence\n", + " x = cat[:-1].clone()\n", + " y = cat[1:].clone()\n", + "\n", + " # we only want to predict at output locations, mask out the loss at the input locations\n", + " y[:INPUT_SIZE] = -1\n", + " return x, y" + ] + }, + { + "cell_type": "markdown", + "id": "a7f95daa", + "metadata": {}, + "source": [ + "_Looking at one sample --_" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "ca079295", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(tensor([ 4, 3, 7, 6, 10, 8, 4, 8]),\n", + " tensor([-1, -1, -1, -1, 8, 4, 8, 5]))" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Supervised dataset\n", + "st_dataset = SupervisedDataset()\n", + "st_dataset[0]" + ] + }, + { + "cell_type": "markdown", + "id": "0b27620d", + "metadata": {}, + "source": [ + "_Create a GPT instance --_" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a6e0d707", + "metadata": {}, + "outputs": [], + "source": [ + "from mingpt.model import GPT\n", + "\n", + "def get_model(block_size, vocab_size, output_size=None):\n", + " '''\n", + " block_size = length of input\n", + " vocab_size = digits allowed\n", + " output_size = length of output\n", + " '''\n", + " if output_size is None:\n", + " output_size = vocab_size\n", + " model_config = GPT.get_default_config()\n", + " model_config.model_type = 'gpt-nano'\n", + " model_config.vocab_size = vocab_size\n", + " model_config.block_size = block_size\n", + " model_config.output_size = output_size\n", + " model = GPT(model_config)\n", + " return model\n", + "\n", + "st_model = get_model(INPUT_SIZE*2, st_dataset.get_vocab_size())" + ] + }, + { + "cell_type": "markdown", + "id": "aa9a1014", + "metadata": {}, + "source": [ + "Set up training --" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d8e8906b", + "metadata": {}, + "outputs": [], + "source": [ + "# create a Trainer object\n", + "from mingpt.trainer import Trainer\n", + "\n", + "train_config = Trainer.get_default_config()\n", + "train_config.learning_rate = 5e-4 # the model we're using is so small that we can go a bit faster\n", + "train_config.max_iters = 5000\n", + "train_config.num_workers = 0\n", + "trainer = Trainer(train_config, st_model, st_dataset)" + ] + }, + { + "cell_type": "markdown", + "id": "33ace95c", + "metadata": {}, + "source": [ + "Training --" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "11318c89", + "metadata": {}, + "outputs": [], + "source": [ + "def batch_end_callback(trainer):\n", + " if trainer.iter_num % 100 == 0:\n", + " print(f\"iter_dt {trainer.iter_dt * 1000:.2f}ms; iter {trainer.iter_num}: train loss {trainer.loss.item():.5f}\")\n", + "trainer.set_callback('on_batch_end', batch_end_callback)\n", + "\n", + "trainer.run()" + ] + }, + { + "cell_type": "markdown", + "id": "d5effd22", + "metadata": {}, + "source": [ + "Loss stabilizes around 1.25. It cannot go lower because we don't have fixed outputs. We are trying to have a probability distribution such that multiple outputs have equal probability of being sampled." + ] + }, + { + "cell_type": "markdown", + "id": "10fb240f", + "metadata": {}, + "source": [ + "Now let us give the model a random input and see what the model has learned to generate as the next token --" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "fb947447", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Input: tensor([[ 0, 2, 8, 6, 10]])\n", + "Possible outputs: tensor([1, 3, 7, 9])\n" + ] + } + ], + "source": [ + "x, _ = st_dataset[0]\n", + "x = x[:INPUT_SIZE+1].reshape(1, -1)\n", + "print(\"Input:\", x)\n", + "print(\"Possible outputs:\", torch.arange(11)[torch.nn.Softmax(dim=-1)(st_model(x)[0])[0, -1] > 0.1])" + ] + }, + { + "cell_type": "markdown", + "id": "beb3aaa1", + "metadata": {}, + "source": [ + "Works like a charm!" + ] + }, + { + "cell_type": "markdown", + "id": "b76344b3", + "metadata": {}, + "source": [ + "Let's save the model too. We'll need to load it later before we start the RL training --" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "53b8edb4", + "metadata": {}, + "outputs": [], + "source": [ + "# Save model weights\n", + "torch.save(st_model.state_dict(), \"models/minimal_RLHF_basic_supervised.pt\")" + ] + }, + { + "cell_type": "markdown", + "id": "0626b1f5", + "metadata": {}, + "source": [ + "### Training a reward model" + ] + }, + { + "cell_type": "markdown", + "id": "ba50428c", + "metadata": {}, + "source": [ + "Now we will train a reward model. \n", + "\n", + "The data required to train the reward model is collected as preferences in the format: \n", + "\\, \\, \\ \n", + "\n", + "The accepted and rejected responses are simply two difference generations by the supervised training model with human labels marking their preference among the two. " + ] + }, + { + "cell_type": "markdown", + "id": "48414a18", + "metadata": {}, + "source": [ + "![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/rlhf/reward-model.png)\n", + "\n", + "Source: [HuggingFace - RLHF blog](https://huggingface.co/blog/rlhf)" + ] + }, + { + "cell_type": "markdown", + "id": "a6d93d11", + "metadata": {}, + "source": [ + "I don't have the money to hire humans to do this labeling and neither the time myself to do it. :) \n", + "So here's a dataset class that'll generate the required data for us--" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "9f9ea8dc", + "metadata": {}, + "outputs": [], + "source": [ + "class PreferenceDataset(Dataset):\n", + " \"\"\"\n", + " Same as MyDataset, except this has output as where x and y are input output from MyDataset. y' is the output that is sampled \n", + " from preferred distribution - and is preferred over y. \n", + " \"\"\"\n", + " def __init__(self, dataset):\n", + " self.dataset = dataset\n", + " \n", + " def __len__(self):\n", + " return len(self.dataset)\n", + " \n", + " def get_vocab_size(self):\n", + " return self.dataset.get_vocab_size()\n", + " \n", + " def __getitem__(self, idx):\n", + " x, y = self.dataset[idx]\n", + " \n", + " _x = x[:INPUT_SIZE+1]\n", + " _y_reject = torch.concat([y[-INPUT_SIZE:], torch.Tensor([11]).long()])\n", + " _y_accept = _y_reject.clone()\n", + " \n", + " # Replace first element with increment of last digit in input\n", + " _y_accept[0] = (_x[INPUT_SIZE-1] + 1) % 10\n", + " if _y_accept[0] == _y_reject[0]:\n", + " _y_reject[0] = (_y_accept[0] - np.random.randint(1, 10)) % 10\n", + " \n", + " return _x, _y_accept, _y_reject\n", + " " + ] + }, + { + "cell_type": "markdown", + "id": "8994ee3f", + "metadata": {}, + "source": [ + "Let's look at one datapoint in this dataset --" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "b5b1f23a", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(tensor([ 8, 0, 3, 4, 10]),\n", + " tensor([ 5, 4, 5, 5, 11]),\n", + " tensor([ 1, 4, 5, 5, 11]))" + ] + }, + "execution_count": 17, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "pf_dataset = PreferenceDataset(st_dataset)\n", + "pf_dataset[0]" + ] + }, + { + "cell_type": "markdown", + "id": "a1bdf280", + "metadata": {}, + "source": [ + "The first tensor is the \\, the second is the \\ and the last is the \\. Notice the \\ and \\ only differ in their first digits. Unlike a usual RLHF pipeline where the pretrained model would be used to generate the outputs to be ranked for preference, here we artifically generate the data to look like this for our convenience." + ] + }, + { + "cell_type": "markdown", + "id": "5338f3c2", + "metadata": {}, + "source": [ + "Finally, it is time to train the reward model. For this we use the following loss function:\n", + "\n", + "$$loss = -log(\\sigma(R_{acc} - R_{rej}))$$\n", + "\n", + "Where $\\sigma$ is the sigmoid function, $R_{acc}$ and $R_{rej}$ are the rewards obtained by passing the \\ and \\ through the reward model. The intuition behind the loss function is to increase the difference between the rewards of the two types of responses. This becomes clear by looking at the training plot below." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9997e06a", + "metadata": {}, + "outputs": [], + "source": [ + "import tqdm\n", + "\n", + "# Hyperparams\n", + "epochs = 40\n", + "batch_size = 64\n", + "rm_lr = 1e-4\n", + "acc_list = []\n", + "rej_list = []\n", + "\n", + "# Dataloader\n", + "train_loader = DataLoader(pf_dataset, shuffle=False, batch_size=batch_size)\n", + "\n", + "# Optimizer\n", + "reward_model = get_model(block_size=INPUT_SIZE*2+2, vocab_size=pf_dataset.get_vocab_size()+1, output_size=1)\n", + "rm_opt = torch.optim.Adam(reward_model.parameters(), lr=rm_lr)\n", + "\n", + "# Training\n", + "reward_model.train()\n", + "for ep_i in tqdm.tqdm(range(epochs)):\n", + " for b_i, batch in enumerate(train_loader):\n", + " inp, acc, rej = batch\n", + " \n", + " # Get rewards\n", + " r_acc = reward_model(torch.concat([inp, acc], dim=-1))[0][:, -1, 0]\n", + " r_rej = reward_model(torch.concat([inp, rej], dim=-1))[0][:, -1, 0]\n", + " \n", + " # Loss and backprop\n", + " loss = -torch.log(torch.nn.Sigmoid()(r_acc-r_rej)).mean()\n", + " rm_opt.zero_grad()\n", + " loss.backward()\n", + " rm_opt.step()\n", + " \n", + " # Save for plotting\n", + " acc_list.append(r_acc.mean().detach().item())\n", + " rej_list.append(r_rej.mean().detach().item())\n", + " \n", + "# print(ep_i, np.mean(acc_list[-20:]), np.mean(rej_list[-20:]))" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "a37b9c0f", + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "data": { + "text/plain": [ + "Text(0, 0.5, 'Reward (moving average)')" + ] + }, + "execution_count": 19, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjUAAAHHCAYAAABHp6kXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAACS3ElEQVR4nOzdd3hT1RsH8G/Ske69d0tLy96UVWbZiKDiABEQURBElgoCskSmOBAZIsOfLFFBZCN770LLKJS2tLSle+8m9/fHbW5yk7RN2qTpeD/P0yd3nHtzcqvNyznvOUfAMAwDQgghhJB6TqjvChBCCCGEaAMFNYQQQghpECioIYQQQkiDQEENIYQQQhoECmoIIYQQ0iBQUEMIIYSQBoGCGkIIIYQ0CBTUEEIIIaRBoKCGEEIIIQ0CBTWEkGoRCARYvHixvqtRJR8fH4wfP75a16rzGWNjYyEQCLBjx45qvUdtqi+/M0Kqi4IaQrRsx44dEAgE3I+hoSHc3d0xfvx4JCQk6Lt6hBDSYBnquwKENFRLly6Fr68vioqKcO3aNezYsQOXLl1CREQETExM9F090ggVFhbC0JD+7JOGi/7rJkRHBg8ejI4dOwIAPvjgAzg4OGDVqlU4dOgQ3nzzTT3Xrmr5+fkwNzfXdzVIBYqKimBsbAyhUP0GdwqmSUNH3U+E1JKQkBAAwLNnz3jHHz9+jDfeeAN2dnYwMTFBx44dcejQIe58VlYWDAwM8OOPP3LH0tLSIBQKYW9vD4ZhuONTpkyBi4sLt3/x4kWMGjUKXl5eEIlE8PT0xMyZM1FYWMirw/jx42FhYYFnz55hyJAhsLS0xJgxYwAAxcXFmDlzJhwdHWFpaYnhw4fjxYsXan3mc+fOQSAQ4I8//sCSJUvg7u4OS0tLvPHGG8jOzkZxcTFmzJgBJycnWFhYYMKECSguLubdo6ysDMuWLUOTJk0gEong4+ODL7/8UqkcwzD4+uuv4eHhATMzM/Tp0wcPHjxQWa+srCzMmDEDnp6eEIlE8Pf3x6pVqyCRSNT6XOqo6vcKABkZGZgzZw5atWoFCwsLWFlZYfDgwbh37x6vnPQ57t27FwsWLIC7uzvMzMyQk5PD/e4SEhIwYsQIWFhYwNHREXPmzIFYLObdRzGnZvHixRAIBIiKisL48eNhY2MDa2trTJgwAQUFBbxrCwsLMX36dDg4OHD/HSQkJFCeDqlTqKWGkFoSGxsLALC1teWOPXjwAN27d4e7uzvmzp0Lc3Nz/PHHHxgxYgT++usvjBw5EjY2NmjZsiUuXLiA6dOnAwAuXboEgUCAjIwMPHz4EC1atADABjHS4AkA9u/fj4KCAkyZMgX29va4ceMG1q9fjxcvXmD//v28+pWVlWHgwIHo0aMH1q5dCzMzMwBsK9Pvv/+O0aNHo1u3bjhz5gyGDh2q0WdfsWIFTE1NMXfuXERFRWH9+vUwMjKCUChEZmYmFi9ezHXR+fr64quvvuKu/eCDD7Bz50688cYbmD17Nq5fv44VK1bg0aNHOHDgAFfuq6++wtdff40hQ4ZgyJAhuHPnDgYMGICSkhJeXQoKCtCrVy8kJCTgo48+gpeXF65cuYJ58+YhKSkJ33//vUafTRV1fq8AEB0djYMHD2LUqFHw9fVFcnIyNm/ejF69euHhw4dwc3Pj3XfZsmUwNjbGnDlzUFxcDGNjYwCAWCzGwIEDERwcjLVr1+K///7Dt99+iyZNmmDKlClV1vfNN9+Er68vVqxYgTt37mDr1q1wcnLCqlWruDLjx4/HH3/8gbFjx6JLly44f/68xv8dEKJzDCFEq7Zv384AYP777z8mNTWViY+PZ/7880/G0dGREYlETHx8PFe2X79+TKtWrZiioiLumEQiYbp168YEBARwx6ZOnco4Oztz+7NmzWJ69uzJODk5MRs3bmQYhmHS09MZgUDA/PDDD1y5goICpfqtWLGCEQgEzPPnz7lj48aNYwAwc+fO5ZUNCwtjADAff/wx7/jo0aMZAMyiRYsqfRZnz55lADAtW7ZkSkpKuOPvvPMOIxAImMGDB/PKd+3alfH29lZ6/w8++IBXbs6cOQwA5syZMwzDMExKSgpjbGzMDB06lJFIJFy5L7/8kgHAjBs3jju2bNkyxtzcnHny5AnvnnPnzmUMDAyYuLg47pg6nzEmJoYBwGzfvp07pu7vtaioiBGLxUr3E4lEzNKlS7lj0ufo5+en9DuV/u7kyzMMw7Rr147p0KED75ji51m0aBEDgHn//fd55UaOHMnY29tz+7dv32YAMDNmzOCVGz9+vFrPiJDaQt1PhOhIaGgoHB0d4enpiTfeeAPm5uY4dOgQPDw8ALBdD2fOnMGbb76J3NxcpKWlIS0tDenp6Rg4cCCePn3KjZYKCQlBcnIyIiMjAbAtMj179kRISAguXrwIgG29YRiG11JjamrKbefn5yMtLQ3dunUDwzC4e/euUp0V/1V/9OhRAOBaiKRmzJih0bN47733YGRkxO0HBweDYRi8//77vHLBwcGIj49HWVkZ7/1nzZrFKzd79mwAwJEjRwAA//33H0pKSvDJJ59AIBBUWs/9+/cjJCQEtra23DNPS0tDaGgoxGIxLly4oNFnU6TJ71UkEnE5MWKxGOnp6bCwsEBgYCDu3LmjdO9x48bxfqfyJk+ezNsPCQlBdHS0WnVWdW16ejpycnIAAMePHwcAfPzxx7xyn3zyiVr3J6S2UPcTITqyYcMGNG3aFNnZ2di2bRsuXLgAkUjEnY+KigLDMFi4cCEWLlyo8h4pKSlwd3fnApWLFy/Cw8MDd+/exddffw1HR0esXbuWO2dlZYU2bdpw18fFxeGrr77CoUOHkJmZybt3dnY2b9/Q0JALuKSeP38OoVCIJk2a8I4HBgZq9Cy8vLx4+9bW1gAAT09PpeMSiQTZ2dmwt7fn3t/f359XzsXFBTY2Nnj+/DlXTwAICAjglXN0dOR19wHA06dPcf/+fTg6Oqqsa0pKikafTZEmv1eJRIIffvgBP//8M2JiYng5MPb29krX+fr6qryfiYmJ0uextbVV+p1XRPH3I31mmZmZsLKy4n4Piu+v+HshRN8oqCFERzp37syNfhoxYgR69OiB0aNHIzIyEhYWFlxS6pw5czBw4ECV95B+abi5ucHX1xcXLlyAj48PGIZB165d4ejoiE8//RTPnz/HxYsX0a1bN96//Pv374+MjAx88cUXCAoKgrm5ORISEjB+/HilpFj5VgNtMzAw0Og4I5f8DIDX+lJTEokE/fv3x+eff67yfNOmTWt8f0C93+s333yDhQsX4v3338eyZctgZ2cHoVCIGTNmqExarqiVpqLnqC51fw+E1HUU1BBSCwwMDLBixQr06dMHP/30E+bOnQs/Pz8AgJGREUJDQ6u8R0hICC5cuABfX1+0bdsWlpaWaNOmDaytrXH8+HHcuXMHS5Ys4cqHh4fjyZMn2LlzJ9577z3u+KlTp9Sut7e3NyQSCZ49e8ZrnZF2g+ma9P2fPn2KZs2acceTk5ORlZUFb29vrhzAtsJInysApKamKrVWNGnSBHl5eWo98+rQ5Pf6559/ok+fPvj11195x7OysuDg4KCT+lWH9PcQExPDaw2LiorSY60IUUY5NYTUkt69e6Nz5874/vvvUVRUBCcnJ/Tu3RubN29GUlKSUvnU1FTefkhICGJjY7Fv3z6uO0ooFKJbt25Yt24dSktLefk00n99y/9rm2EY/PDDD2rXefDgwQDAG04OQCsjhNQxZMgQle+3bt06AOBG34SGhsLIyAjr16/nfV5V9XzzzTdx9epVnDhxQulcVlYWl89TXZr8Xg0MDJRaQ/bv31/nZp6Wtjj9/PPPvOPr16/XR3UIqRC11BBSiz777DOMGjUKO3bswOTJk7Fhwwb06NEDrVq1wqRJk+Dn54fk5GRcvXoVL1684M1XIg1YIiMj8c0333DHe/bsiWPHjkEkEqFTp07c8aCgIDRp0gRz5sxBQkICrKys8Ndff6mdZwEAbdu2xTvvvIOff/4Z2dnZ6NatG06fPl1r/0Jv06YNxo0bhy1btiArKwu9evXCjRs3sHPnTowYMQJ9+vQBAG5elhUrVmDYsGEYMmQI7t69i2PHjim1eHz22Wc4dOgQhg0bhvHjx6NDhw7Iz89HeHg4/vzzT8TGxta4lUTd3+uwYcOwdOlSTJgwAd26dUN4eDh27drFa22qCzp06IDXX38d33//PdLT07kh3U+ePAGg3e5BQmqCghpCatFrr72GJk2aYO3atZg0aRKaN2+OW7duYcmSJdixYwfS09Ph5OSEdu3a8eZqAdjkXCcnJ6SkpKBHjx7ccWmw07lzZ14ispGREf79919Mnz4dK1asgImJCUaOHIlp06bxkomrsm3bNjg6OmLXrl04ePAg+vbtiyNHjigl+erK1q1b4efnhx07duDAgQNwcXHBvHnzsGjRIl65r7/+GiYmJti0aRPOnj2L4OBgnDx5UmkuFTMzM5w/fx7ffPMN9u/fj99++w1WVlZo2rQplixZwiUx14S6v9cvv/wS+fn52L17N/bt24f27dvjyJEjmDt3bo3roG2//fYbXFxcsGfPHhw4cAChoaHYt28fAgMDaaZiUmcIGMoEI4QQUg1hYWFo164dfv/9d24GakL0iXJqCCGEVElxaQ2AzVkSCoXo2bOnHmpEiDLqfiKEEFKl1atX4/bt2+jTpw8MDQ1x7NgxHDt2DB9++GGtdUUSUhXqfiKEEFKlU6dOYcmSJXj48CHy8vLg5eWFsWPHYv78+TA0pH8fk7qBghpCCCGENAiUU0MIIYSQBoGCGkIIIYQ0CI2qI1QikSAxMRGWlpY0WRQhhBBSTzAMg9zcXLi5uVW6Rl2jCmoSExMpS58QQgipp+Lj4+Hh4VHh+UYV1FhaWgJgH4qVlZWea0MIIYQQdeTk5MDT05P7Hq9IowpqpF1OVlZWFNQQQggh9UxVqSOUKEwIIYSQBoGCGkIIIYQ0CBTUEEIIIaRBaFQ5NeoSi8UoLS3VdzUaHCMjIxgYGOi7GoQQQhooCmrkMAyDly9fIisrS99VabBsbGzg4uJC8wQRQgjROgpq5EgDGicnJ5iZmdEXrxYxDIOCggKkpKQAAFxdXfVcI0IIIQ0NBTXlxGIxF9DY29vruzoNkqmpKQAgJSUFTk5O1BVFCCFEqyhRuJw0h8bMzEzPNWnYpM+XcpYIIYRoGwU1CqjLSbfo+RJCCNEVCmoIIYQQ0iBQUEMIIYSQBoGCmgZg/PjxEAgEEAgEMDIygq+vLz7//HMUFRXpu2qEEEJIraHRTw3EoEGDsH37dpSWluL27dsYN24cBAIBVq1ape+qEUIIqackEgYMAANh/ciHpJaaBkIkEsHFxQWenp4YMWIEQkNDcerUKbWu/eKLL9C0aVOYmZnBz88PCxcuVBqd9O+//6JTp04wMTGBg4MDRo4cyZ0rLi7GF198AU9PT4hEIvj7++PXX3/V6ucjhBBS++b8eQ9NvjyKpOxCfVdFLfWqpSYhIQFffPEFjh07hoKCAvj7+2P79u3o2LGj1t+LYRgUloq1fl91mBoZ1GiUUEREBK5cuQJvb2+1yltaWmLHjh1wc3NDeHg4Jk2aBEtLS3z++ecAgCNHjmDkyJGYP38+fvvtN5SUlODo0aPc9e+99x6uXr2KH3/8EW3atEFMTAzS0tKqXX9CCCH6l55XjL/vJAAApvx+BwendtdzjapWb4KazMxMdO/eHX369MGxY8fg6OiIp0+fwtbWVifvV1gqRvOvTujk3lV5uHQgzIw1+9UcPnwYFhYWKCsrQ3FxMYRCIX766Se1rl2wYAG37ePjgzlz5mDv3r1cULN8+XK8/fbbWLJkCVeuTZs2AIAnT57gjz/+wKlTpxAaGgoA8PPz06juhBBC6haJhEGHr//j9iMSsisse+DuC8zcdw+7PwhGN3+H2qhehepNULNq1Sp4enpi+/bt3DFfX1891qhu6dOnDzZu3Ij8/Hx89913MDQ0xOuvv67Wtfv27cOPP/6IZ8+eIS8vD2VlZbCysuLOh4WFYdKkSSqvDQsLg4GBAXr16qWVz0EIIUS/fOYeUTpWJmFUlr39PAMz990DAIzeeh0LhzXHxB76+26uN0HNoUOHMHDgQIwaNQrnz5+Hu7s7Pv744wq/bAE216O4uJjbz8nJUfv9TI0M8HDpwBrVubpMjTRfPsDc3Bz+/v4AgG3btqFNmzb49ddfMXHixEqvu3r1KsaMGYMlS5Zg4MCBsLa2xt69e/Htt9/K6lO+vIHKulZyjhBCSP1yKzajwnMlZRIYG/JTcV/feJW3/92pJ3oNaupNonB0dDQ2btyIgIAAnDhxAlOmTMH06dOxc+fOCq9ZsWIFrK2tuR9PT0+1308gEMDM2FAvPzWddVcoFOLLL7/EggULUFhYeXKXNPdm/vz56NixIwICAvD8+XNemdatW+P06dMqr2/VqhUkEgnOnz9fozoTQgjRvxsKQc0nff257aYLjqHTclmXVG6R8nI3ecVlyC8u010Fq1BvghqJRIL27dvjm2++Qbt27fDhhx9i0qRJ2LRpU4XXzJs3D9nZ2dxPfHx8LdZYv0aNGgUDAwNs2LCh0nIBAQGIi4vD3r178ezZM/z44484cOAAr8yiRYuwZ88eLFq0CI8ePUJ4eDg3VNzHxwfjxo3D+++/j4MHDyImJgbnzp3DH3/8obPPRgghRDfuxmUBAD7q5YfYlUMxe0AgWrlbc+dTc4sRm5YPn7lH0GrxSe74jS/7Kd1DH+pNUOPq6ormzZvzjjVr1gxxcXEVXiMSiWBlZcX7aSwMDQ0xbdo0rF69Gvn5+RWWGz58OGbOnIlp06ahbdu2uHLlChYuXMgr07t3b+zfvx+HDh1C27Zt0bdvX9y4cYM7v3HjRrzxxhv4+OOPERQUhEmTJlX6noQQQtS38dwz7L1R8XedtjAMw3U/DWzhwh1XzKfpvfac0rVOViZo52WDJo7mMDXWPIVCWwQMw6jO/qljRo8ejfj4eFy8eJE7NnPmTFy/fh1XrlxR6x45OTmwtrZGdna2UoBTVFSEmJgY+Pr6wsTERKt1JzL0nAkhRH0LD0bgf9fYlIA9k7qgaxP7Kq9hGAZHwpPQ1tMGHrZmar/Xvptx+OKvcADAk68Hc/kzh+4lYvqeuxVeN71fAGb1b6r2+1RHZd/f8upNS83MmTNx7do1fPPNN4iKisLu3buxZcsWTJ06Vd9VI4QQQnRCGtAAwI2YipN45f17PwnTdt9Fj1VnUSqWqP1eZx6nAACau1rxEoJfae2KEzN6YvPYDiqvm9EvQO330LV6E9R06tQJBw4cwJ49e9CyZUssW7YM33//PcaMGaPvqtVp33zzDSwsLFT+DB48WN/VI4SQRiu3qBTZhcrJtlIShW6f/x4lq3Xf4xFJ3HZ4JfPLKJL224xo58Y7LhAIEOhiiYEtXDC+mw8AICTAAT+83Ra3F4RCWIeWUKg3Q7oBYNiwYRg2bJi+q1GvTJ48GW+++abKczQcmxBCNMMwDBYdegCGAZYMb1HtL/R78Vl4dcNlAMD/JnZGSICjUpndCnk04QnZ+P3ac7zbpfLZ4gtKZLPh34jJQHsv9SapzcgvAYBKu6wWD2+BRa80r/EoXV2pV0EN0ZydnR3s7Oz0XQ1CCGkQzkWm4rerbJfQsNauCParOsdFFWlAAwBjf72BZ98MUVo0csHBCKXrFhyMwMAWLnC0FFV475g02UCNyJe5atdJGtTYmRtXWq6uBjRAPep+IoQQQvTt33uJ3PZ1NXNc5DEMo3LG3k/33kVxmZjrcpJfQHLla61488U8flnxRLIZ+SV4nl7A7WuyEGVaHjtZrX0VQU1dRkENIYQQoqY7cZnc9tOUPI2v33ND9Xxph+8nIXDBcaw+EcmWuy7rehrRzh2f9JUl4yZlFVV4/6fJ/JaZtLwStepVKpYgp4idNM/eouJWoLqOghpCCCGkAquOP4bP3COY9/d9ZBeUIlauFeSZBkFNdmEpfOYewZcHwrljl77ogx/ebssrt+n8MxSVihH2gk3wHdLKBSZGBjA2FGJ4GzaBN7eSGXtTy1tbrEzY7BJp60tSdiHaLzvFG00lL7O860koAGxMjdT+XHUNBTWEEEKIColZhdh47hkAtoXl+IMk3vnotDzcjM2Az9wjeHPzVVW3QFGpGOtORqLNkpO84/snd4WHrRlebeuudM2g7y8gJo0NmN4NliUFW5QHKnlFlQQ1uWwQ09yNncslq6AU919koeuKM8jIL8HC8jyd/12NxbpTT7juLmmLjq2ZcZ0azaQpCmoIIYQQFf4JS+TtLz/yCAAwoLkzDIQCFJVKMGoTG8zciMnAtN13eOVLxRIELTyOH89EKd27o7dsRNKcAfyJ62LTCxCfwebCBLnKJpqzFLFBjao1l6SkQU1TZ0sYGbDBydJ/H/LKpOcVY+E/D/Dj6afcWk/SJGF7i/qbTwNQUENqQCAQ4ODBg/quBiGEqKVULMHVZ+kQK8z/Ulgihs/cI+i15iwK5YZDrzr+mFdOmnPS3tsW7jbKU2Icvs9vyfn25BOlMp/09cfT5YN5I4im9Q1A7MqhuLuwP6+shcgQtmayriBLaUuNQvfT1ovR2HA2CgzDcEGNs5UJ/J0sAQC3nmfyyi8/+ojbTi9voUnPZ6+rauRTXUdBTQMwfvx4CAQCCAQCGBkZwdfXF59//jmKiipOJtOGpKQkmsCPEFJvbDgbhXd+uYYfTj/lHZe2VjxPL8DvKnJOOvvwp8XwsTeHq7XqZV6kK1THZxRg0/ln3PG3OnriydeDMXtAIIwMVH/12pobw1xu3SQvOzNe8GPBtdTIgpqk7EJ8feQR1pyIRMD8Y1xOjaOFqMI6/n0ngdvOKmSDGq6lxrz+JgkDFNQ0GIMGDUJSUhKio6Px3XffYfPmzVi0aFG17lVaWnHTpjwXFxeIRPX7fwBCSP1RXCZGck4R/glLqHRYsyolZRJ8/x8bzPyoENQ8kZvLRdqKUVAiCxy2ju/IK983yAkSuWUTZ/dvys0bE5mcix2XYxCy+ix3/vaCUKx6ozVv6YGKFJfJljXwczTnnbM0YVtt5BOFHyTInkOZhOGGcztaimAu4k9Ft+zVFkrvJ22ZkrbYUEsNqRNEIhFcXFzg6emJESNGIDQ0FKdOnaryutjYWAgEAuzbtw+9evWCiYkJdu3aBQDYunUrmjVrBhMTEwQFBeHnn3/mXUvdT4QQXZFIGFyLTse16HR8eSAcJWUS9Fh1FsHfnMane8Mw6PuLEEsYXldMWSXrHD2RG+psZCDgdUFFKgyDLioVIyGTzWmxNDGElYkRLs/ti+auVjg0rTuMDYVIKe/mAYAAZwsEubBdPY+ScrBYLoelraeNRkOk3+nsxW37OVrwzkkTheVzar4+ws+XkU6852gpwtLhsiAmduVQtPdWnlk4v7g8qGkgOTU0o3BFGAYoLai6nC4YmQE1mLExIiICV65cgbd35VNpy5s7dy6+/fZbtGvXjgtsvvrqK/z0009o164d7t69i0mTJsHc3Bzjxo2rdt0IIY1b32/PIT2vBOc/6w0bs4q/QENWn0VClmziuKISMZcvItXky6MAgB0TOmH89pvc8diVQ5Xut0tu3pdSMYOolDwElgci8jPwAkBmQQniMti//9LcGXcbUxz9NIQrs39yV3RefhoAMKilKy5FpeHiU+Ds41Tevf6e0q3Cz6hKe28bbti1vxM/qLFUMfpJfoi5PEdLEWzNjXnPItDZUqmctEUqI7/+T7wHUFBTsdIC4Bu3qsvpwpeJgLF51eXkHD58GBYWFigrK0NxcTGEQiF++uknta+fMWMGXnvtNW5/0aJF+Pbbb7ljvr6+ePjwITZv3kxBDSGkWs5GpiA6lQ0g2i49pTL4kJIPaADg77sJFZQEL6ABgPbLTiEjvwQPlw6EmTH7NZejsHBkfEYBF9RIAxiprIJSbiHIZnKjj+Q5WZrw6t+kvFVFuuiksYEQT5ZrnnM4oq07frv6HC5WJugb5MQ7Zyliu5/y5PJ2pF5p48bNdiwQqO5GMjQQ4ru32mDmvnt4v7svtl2OQT4X1Ehbaup3SgEFNQ1Enz59sHHjRuTn5+O7776DoaEhXn/9dbWv79hR1mecn5+PZ8+eYeLEiZg0aRJ3vKysDNbW1lqtNyGk8ZigEHwUl4khMjRQKlfZkGV1SL+gVx17jCWvtgQgC1wMhGzXkzRoKpRrAXKwECEtrxiZBSV4nMR2SbVwUx3UKOrorZBM7FDxopCVEQgEOPBxd5XnLE34icL7bspmJ36roycX1JgbG1aYjDyynQdGtvPAlgtsEnNBsRgbzkbhZiw7Qqq+59RQUFMRIzO2xURf760hc3Nz+Puza4Ns27YNbdq0wa+//oqJEyeqfb1UXh476dMvv/yC4OBgXjkDA+U/QIQQUpWiUrHSscSsIvg6KLdKJ2XLRm762JvhRWYhyspzYOYNDkJxmQTrTikPl+7WxB5XnqVz+zuvPlcKarr62eNSVBoWHXqALReisX1CJwBswOBpZ4q0vGLkFpXh+IOXAIAgF/WCGlcb/kgj+WUNtMVCbki3RMJweUKdfezQ0l1WT8Uh36pIk4hT84p5rWD1vfuJEoUrIhCwXUD6+KnhCqhCoRBffvklFixYgMJC9Rczk3J2doabmxuio6Ph7+/P+/H19a1R3QghjdMhuYUgna3YLo6sAtXrEkmDmiAXS5z7rA+87GX/0HOyEmFK7yZK17za1g0TuvP/PhmXt1ZkF5Qiu7z7KdhX1qKSkFWIM49TAACetmawLl8eQL5bJ8hVOQ9FFcVgoGeAo1rXacJCbjRTbnEZN//MF4ODePlJIQEOVd7LrHzo+ItM/neEA3U/kbpo1KhR+Oyzz7BhwwbMmTNH4+uXLFmC6dOnw9raGoMGDUJxcTFu3bqFzMxMzJo1Swc1JoTUV3nFZWi56AQAYGhrV3zcuwmsTIzgaScLRi4+TQMAtPGwhphhkJxTjCyFPBeppPKuIek8K27WplwuTicfOxgZCLl8FrGEgVjCwNhQCLGEgUDAjvMAAMPyGXWlrTQOFiIEOPOTby+V18vHwQwGQjYIepYqSxxW90teIBBgYg9f/HopBgBgbab99ZNMjAxgbCBEiViCg3cTuG62Vu5sWkDMiiHILS6DlUnV721S3u2XnMOfz8yWWmpIXWRoaIhp06Zh9erVyM/Pr/oCBR988AG2bt2K7du3o1WrVujVqxd27NhBLTWEEJ7iMjEX0ADAkftJGPrjJYSsPssNm2YYhsv3mDu4Gdcikl2gOqhJzpHNigsA3f1lLQ+KM/kaCAXc/C8GQgH2TuqC+UOaAQAKSsQoLBFzQY23vRm87PjdXZei2KDGxcqUWwRSutK14uijqnw+KBDfjmqDa/P6aXSdJqRdUNIWJgDc5xcIBGoFNPLXFMjNoPzHR121VU29oZaaBmDHjh0qj8+dOxdz586t9FofHx8wDKPy3OjRozF69GiV54qL2T86Fhaa/U9PCGlYjke8rPDcokMR+HpEK17XUzsvG9iYsq0BqrqfikrFiM+UTSAHAB+E+CItrxhd/Ox5M+yqEuxnj86+dlhzMhIlZRKk5xfjeQb7DzsvOzOlCe2kbM2MUFDKftHfe5EFgM3n0YTI0ACvd/DQ6BpNWZoYIiO/hJs0762OntW6j+JEgJ197dDZ166C0vUHtdQQjeXk5GDPnj0QCoUICgrSd3UIIWooLBFj5M+X8e3JSJXnU3KKsOFsFErKlCewYxi2iyc1txjXo9MRn1HAze3y6d4wrtyeSV1410lnt32anMcdMzEy4Lpm5LufGIbBwoMRCFp4HH/efgGADUIAwMhAiIXDmqN/c2e1PqtAIIBDeTdKel4JlyPjZWcGEyMDLH21BRcwSdmaG3OtHKVi9h96ASrmddE3aV5NdPnz99Iw8JJSHB0lbT2r76ilpoH75ptv8M0336g8FxISgmPHjml8z0WLFmH37t1YtWoVPDx0+68SQoh29F57Fsk5xbgbl4UPevgp5Xx0/oadSG7NiUhM6+OPn85GYc+kLvB1MEeXFaeV7mdmbIB/psqGHq96vRW6NrFH7MqhuBKVhtFbryMqhQ1mXpbnbXw2MBAAYFP+BZol1/10KSqNm3ROyt1WedFIddlZGCMxuwjp+cXYc4Md+iwNkt7r6gNnKxN89L/bXHlHS5FSQOdkWfeSZqXDutOkazxVs46KLTUmRg1jZCsFNQ3c5MmT8eabb6o8Z2pavT8Y3333Hb777ruaVIsQUouY8sRcqQHfn8ew1m6YP6QZhEKBUhf0T2ejAABLDz/EoyTVaywVlIixVq7V561Osun9W3vaAGBHMWUXlOJhInsPl/IcGZvygCpbrqVGVTdW2/L7VId0YcbTj2S5J/JBkmLuiYOFCNei03nH6mLrhYWIXyePagZ+xgotNSI11qWqDyioaeDs7OxgZ1f/+0kJIcrKxBLEZRQorRH0PD0fQoGAG30kXddHKjmnGL9eikGgiyXe7OiptPyAVGJWIYQCQKI67Q4nHrCz5w5vw5993UJkCGtTI2QXliI+swBPU9jEW+nsvKpyamxUjBaSzgZcHdIARn55BG+5rhorU/69HS1EGNrKFdsvx3LH6mJQI22pkaruZHmKLTUU1BBCCNGr5l+dQIlYgraeNjhY3hWUlleM/t9dQEmZBMG+dlgwtDlvxWl5YfFZeLOjJ5cfIzIU8laJzi0qhY+DOaJT8/HruI5o6mzJW31aSlUXiL25MbILS/H9f0+4HJVm5XO+qMqpeZnNBlbvd/fF3fhMvNqmZsvUeNoq55q4WlfSUmNpDAb86M3Fmj+hXl2gGNTYVrJ+VmUUW2rUWUG8PqCgRoFEUvEqr6Tm6PkSoh3xGQUoKV+VOiw+C2ViCb74Kxx/3XnBlbkek4FXfrqEsV3YxW17BzriclQaF2REp7I5L9KgpmsTe3z/VlvkFpVhwHcXUFgq5uaH8XO0gKedGWJXDkVJmQRNF8jy8cYEy7qepOzMjRGdlo//yrt/nK1E3MglGxVDulNy2byb5m5W+OqV5jV9PNwEf1L/zerF21dsGTIzNoSFiD/rsXRIeV0iPwEfoLqFSx3KLTWUU9OgGBsbQygUIjExEY6OjjA2Nq5y6CBRH8MwKCkpQWpqKoRCIYyN6/cET4ToWmxaPj76322828ULY7v6KJ1/qJDr8uftF7yARp40AbeJowV2TOiMiIRsDFt/CU+S+UGNr4M5bMyMYWNmDB8Hcy6fRiAA3OSWAZD/QrQUGSp1fwHK3SLvy832K539lt9SwwY1LloKJAyE/L/finPOKAYHgGwOGKnqtoLokqVcC5OZsUG1gxHF0U/U/dTACIVC+Pr6IikpCYmJelrzqREwMzODl5cXhMKG8T8QIdrmM/cIb3/hPw/wdmcvpS+heIWVpef+Hc7bDypfgfrxy1zumDSnRdo1k5FfgqJSMTdKyU9uHSZvOzMuqHGxMlH68tz5fmecfpSMzwepntZBsUtKvitH2rqQVVACiYSBUCjgRki5WGtnxFEfuRWuh7ZyVTqv6h+tip9RMTCqC+QDr5oEXUotNUYN428yBTVyjI2N4eXlhbKyMojFyouvkZoxMDCAoaEhtYARosIfN+Px+V/3VZ67Fp2OEIW1hO7GZQEAHCyMkZbHTwR+rZ071r3VFgzDwHfeUe54/2bsPC9WpoYwMzZAQYkYiVmFOF0+O62vg6w1Qz4IUTWzbq+mjujVtOL1jRRbXORzWKQJuBIGyCspg6FQwK087aSllhorEyPsnhSMbZdi8P3bbSstq2pRzbrKSi6ocbCoflCj2DJD3U8NlEAggJGREYyM6l7WOyGk4ZBIGFx5lo7WntYoEzMVBjQAMPbXGwCAre91RCsPa2QXluJcJBuI/DymA97cfJUre/6z3vC2Z7+kBQIBrszti/7rzmNK7yZcgq5AIICbjSmiUvJwKzaTu1Y+eJFvaWnupt5K1fKcFZJs5bujTIwMYGIkRFGpBNkFpdwK3GbGBrBU0S1UXd2aOKBbk4oXd9w/uSvWnIjE4ldaaO09dU2+28zNpvrz+Ci2/FGiMCGEECUpuUW4F5+N0GZOlbZKHg5PwvQ9d9GzqSNv5Wh5Xw1rjqWHH3L7H/x2i3few9YUnXxscWdhf7yy/hLGd/PhAhopNxtTPFg6SOne7uVBjXwwJZ9cK58kG+Ck+cy6ii01TRRae2xMjfGytAhZBaXIKy7jrqnNltxOPnb1br0jU7lJ8hR/15owEApgIBRw63NRTg0hhDQCDxNz4GQlUmu15mvR6Xh7yzUAwPR+AZjVv2mFZb//7wkA4MKTVGTLzdfS3ssGez/sCmNDIYpKxbygRpGXnRkEAgHszI1xeW5fdT8SANX/ypcPKKQ5OQDQpIL1kioj333lbmOqctTOy5wiZBaUcKtN14XRRl8MCsKq448xom3NhpTrinxStmkNZwE2MRQiv3wNqYYS1DSMT0EIIdWUX1yGXy5EIy69QOlcREI2hvx4ER2//g+3n2equJpPumYRAPx4+qnS+eScIjxPzwfDMEjMKuSO33uRDQA4ObMn/v64O9cVYGJkgNmVBEaDWrpUWaeKuNvwA4jp/QJ4+0EulmjqbAEXKxMEuWje/SQf1CjOrSJ/LK+4TC5JWP9BzfhuPtj6XkesfqONvquikou1CQLKW73aednU6F4iuaCooQQ11FJDCGnw4jMKcDziJYa3deO1BjAMgxaLTgAAlh99hE/6+mP2gEAwDIP4jEKcfJjMlf3rzgt08Lat8n3kpeQUcYmv+cVlCC5fX+n7t9qiqJQ/Z5PIUMh9Wcn7pF8AJvTwxc4rsVhzgl2W4IMevvC2N8OYYG91H4ESxXWVFFuVDA2EODStBwSC6iWRyufGFKtYJFM6W3B+cRmSy4MaJyv9r7VkamyAUDUXztSXbeM74Xl6AXoEVJwvpA4TuUBG1EDWfmoYoRkhhFTiywPhWH70ET7YyeakSCQMdlyOwcpjj3nl1p+JQmpuMXbfiEPPNWd5rS0vMgtRFWmLg9TxBy+5RRL/eyQLkGbsC1O61rO8K0kVC5EhJvbw5bYXDGuOsV19IKzBkGMDuWkVpvXxV1nGxKj686AIBAK80YFd8Pa39zsrnZd2R8kHNdqao6ah87Qzq3FAA/AXsaSWGkIIqYMkEga/XY1FWy9btPW0QUFJGS4+TQMAhCdkIzGrEN1Wnqnw+pn7wnApKk3p+IsM5e4psYRBky+P4tW2bvj+rbZIKp9A7rX27vj7TgK++ucBvvrnAUKbOcG/imTbps7KrTTyTIwMELtyaKVlNDGwhaw1YvaAiru4amLtqDZYO0p1N465iP1CzS8Ra33iPaIe+RFPDWVId8MIzQghBGx30sw/wrD434cYseEyUnOLcfheEq/M6uOPla7bOKY9t60qoAHYlpoFB8PR5ZvTXD7MxJ03AQD/hCXiwN0ElJRJIBAAfeUmfgOA/x6l4OLTVKV79gmUzfPi51B5UKNtIkM2SIpdOVQvc0fxu5/YdZ8Uh4ET3WqILTUN41MQQhq9eX/fh++8o/gnTDYj+KDvLygl+B4MU54xfFBLF+yZ1KXS+5eIJfj9Whxe5hRh0/lnAIBzkbJA5ddLMQAABwsR2nkp5948SMxROjarfyC3HVBFS01DI+1+yisu49Z9opaa2mUiN4uwSQOZUbhhfApCSKNWKpZgz414pePp+SV4/JINJga14I8U8rQzxdDWrvh3Wg8IBAKlJGAfezP8O60HHi4dqPRlm18s5u4rJQ1aXK1N4F7BpGiKKyM3c7XEd2+1wagOHhjYovojmeoj8/KgJj6jAKViBgKB6tW+ie7IdzlR9xMhhGhBYYkYkvIJwABg1fHH8Jl7BO9tu6H2PQ7eTajwnHS49IAW/BEtq15vjQ2j26OVhzUANr9AvtuoR4ADWnlYw8zYEK4Kw58Tswqx+3qcyvdzLe9CUVwVGmCDmFMzewIAPunrD0MDIUa288CaUW14XQGNgTSn5nl5rpKtmbHSLLdEt0S80U8N49k3jE9BCKmX1p2MRLOvjuP98tyUkjIJNp5ju3YuPElFpNxijKrcjM3A8/R8XJbLg7kyty9iVw5F6/JgBQCMDARKw3RVDc+e3KsJty2/qrSbNb/lJTotj1sEclKIL++ca3lZfycLxK4cihWvteLOtXS3RoCzJWJXDsXsAYFozMzLc2peZLD5SYqrehPdk/u3hMpVy+ujhvEpCCH1jljC4MczUQDY3JQNZ6O4eVikbsRmILB8ZtvcolK0WnwSAPDk68EY9MMFRKfmA5AtkPhx7ybcTLmt3K1xv7yVxtfBnLegorOVSGVze2dfO/w0uh3yisp4M7e6KiSwJucUc8mtI9t5YPf1OG5mVjeFVp3+zZ0xr3wFbfl7NnbS7qcSMTvk3a4GK06T6imTyOYPqunsxHUFBTWEEJ1aeyISDxKzser11ohKzcPoX66juasVNsiNOAKgFNAAwK5rzzG2CzvB3LTdd7nj3/33hAtoACC7sBQAeK0zrT2sses6uy3t2rm/eAAy8krgU8mqzMNaK0+PrzhRnZSBUAB/Jwt09LHD+Sds0rCLQquOg4UI73X1xsWnaXiljWuF79vYSLufpGzNaRHh2lYiNymiPkbA6QIFNYQQjZWUSXA2MgU9/B24f3Gr8iKzAD+dZVtj9t6Mx7pT7HpHD5Ny8HP5cVUWDmuOZYcf4vHLXEgkDMITsrmgAQDXRaWojacNt92zqWy4dNvy41YmRrwWG3V525upPm5nBmNDIdcVBQBuKoYlL321pcbv2dAp/ndD3U+1r0TFTM/1HeXUEEI0tvHcM3z0v9sY+fNlXpKvosdJspyYv+684J3bf/uFYnEYGwgRs2IIxgR7cceepuThBxXrKAHsoojyXOVaSVytTfF2J08AwNQKZsxVV0iAI4JcLNHd3x4hcjO5Slee9rSTvW8LN2ul64kyxRwOCmpqX0NJDpbX8D4RIUSnJBIGWy9FAwCeJOchJj2/wrKxcueeq1gwEmDzW6R8HNilAkyMDNDd3x4AmwycXr6K87DWrrzFEV2sTHByZk9MCvHF42WDlO698vXWiF05tMarPxsZCHF8Rk/s+qAL715NynNkNr3bAU6WImx9ryNMjRtGboKumSk8J1vKqal173X1AcCuDN9Q1NugZuXKlRAIBJgxY4a+q0JIo9Jt5RnkFpVx+8nZRRWWjVUR8HRUGHUkP3+Mh62ZXDk7AMC16HREls8JM7N/U3jKlXG0FKGpsyXmD21ea0Oi5d/fv7ylxsbMGDfmh9b5hRDrEksRv5WNWmpqX/9mztj1QTB+/yBY31XRmnoZ1Ny8eRObN29G69at9V0VQhqcUrEExWVilefyisuUFm1U3AfYxN15f4fjxINkpXP9Fb743+7shUBnS9iaGeG7t9pyx6VDrg/fT0JRqQTmxgbwtTfnjS56WUlApSte9rKuJn8Vq2oT9ViaGMJAbkFOCmpqn1AoQHd/B27Jioag3gU1eXl5GDNmDH755RfY2irPM0EIqb5bsRkImH8MgQuOcyOK5E0qX+UaAIa0YltYYtKUW2Pe33ETe27EITW3WOlcH4V1kdysTXFiZk/c/WoANzQbAIJc+QtAtnC3hlAo4LXmdGtir+Yn055mrlYA2GnlmzhWPIqKVE4oFMBG7vdNQQ3RhnoX1EydOhVDhw5FaGholWWLi4uRk5PD+yGEVOydX65x21efpSMjvwQ9V5+Fz9wjuBefhavR6QCAEW3duITYeLnVq4tKxSgqFSuttyS/bEAThblarM1Uj0ZysjThjTryKx+G7SN3bOGw5hp9Pm0IcrHCpnc74NdxnWBZjZFURMZKLqihnBqiDfUqqNm7dy/u3LmDFStWqFV+xYoVsLa25n48PT11XENC6qcXmQWYuS8MpWLZSKbzT1LQftkpxJUHLa9uuMydWzCsORdwvMgsRG5RKX67GouOX/+H/t+dV7r/3g+7wMRIiNBmzjAQCvDfrF5wshRxSwZUJNBZ1lojXc5gRDt3BLlYYmIPXxjqaVr9QS1d0N3foeqCpFLy3U/2FhTUkJqrNx1p8fHx+PTTT3Hq1CmYmKg3kmHevHmYNWsWt5+Tk0OBDSHlGIbB21uuITW3GNEqupBULRAJsK0uDhYiLmE2PrOAm+kXYPNuFHnameHm/FBuFl9/JwvcmF91a2sHb1ucfMjm5UgDHBszYxyfUXkwROqHhMxCbruhzGhL9KveBDW3b99GSkoK2reXzUIqFotx4cIF/PTTTyguLoaBAf9/CpFIBJGIVn0lRBHDMHiQmIPrMRlVlu0X5IQzkSlgyhtxXmvvDgDwKJ9lV7pcQEX+N7EzAFSrq6aVu2zOlwAny0pKkvpIukQC0HBmtCX6VW+6n/r164fw8HCEhYVxPx07dsSYMWMQFhamFNAQ0pgNW38RPnOPqJwx9PSjZPjOO4ph6y+pvPamQgvK1nEd8XFv2UKPb3dmJ8azMzdWmmtEPgemmasVYlcORUiAI6qrm78DFgxthpWvtaow94bUX8tHtISBUICVcot+ElIT9aalxtLSEi1b8qcaNzc3h729vdJxQhoKhmEQnZYPbzsztfNHXmQWICKBTYrfcSUGH/aUBSSFJWJMlBvBJG/la624gCUkwAEXn6bhk77+EAgE+LRfU3jbm2NAc2fYlCd0CgQCeNia4kkyu0TAqA4emDekGdovOwUAsNTSqr8fhPhp5T6k7nm7sxde7+ABIz3lRpGGp94ENYQ0Rruux2HBwQi828ULX4+o/F+zOUWlyC0qw9Nk2TpEiVn8eVwiErOVrtv3YReIGQbdmsgSX3dO6IzY9Hz4lo84MjYU4s2OyvloxXItQT0CHHjDcoP97Kr4dISAAhqiVfU6qDl37py+q0CITkkXffz9WhyWvdoSAoEA2YWlyCkshaedrKunqFSM1uXJukNbyVaCfpaax7tftNz+6+09MCM0gHcfKaFQAD/HqieWk0/u7FE+GujwJz1wPOJljddbIoQQTdXroIaQhiy/uAyJcjPmvswpgqu1KdoskY00WjuqDd7o4IHHL2ULRx4JT+K2IxKywTAMl4S5+QK7ZtN7Xb21snL0xnc7oM/ac5japwnsLdik/Jbu1mjpTos6EkJqHwU1hNQR55+kYua+MGwe2wGdfOzwNIXfyhKTlo/0vBLesTn77+G1du54mKh6YsnMglKci0zFhB030dLdirteflHImvB1MEfsyqFauRchhNQUdWYSUke8v+MmMvJLMGrTVQDsQo7yYtMKVM4n8yw1Dw+TlHNlpCbsuAkAiEjI4ZY+GN7GXVvVJoSQOoOCGkL0JDGrEP+EJYBhGDAMA7FENptvcZkYK4895pVf/O8DZOSxc8IMauGCzr5sIu7d+CxecrCU/NIEipo600KMhJCGh7qfCNGTbivPAACMDYRKOSjPUpRbZErKJEgqz7FxtWHXRboRk4EbMRncUgbyjk4PQZulJ5WOt3K3ponOCCENErXUEFLL+n57Dj5zj3D7Jx68xAOFnJjzT1K57d8nBgMAHC1FXOKwm7UpfMqHW/95+wWSc9jj695sAwDYOKY9rM2MEL54ABYOa86bmdfFWr1lRgghpL6hlhpCalFxmRjRqfxWmNyiMt5K1wCw6jjb9dTD3wGtPdmAJDW3GP/eSwTABiZtPGy48tKeq5Ht3PFaew/uuKWJESb28MWDxGyEJ7B5N24U1BBCGihqqSGkFqkapRSTns91KynqHegIKxVrJvk6mMPLXnl+mYq6lT6Sm1W4hRsNtyaENEwU1BBSi6StJQCwYGgzAEB8RgHiM9mWmtn9m/LKDy6fSO+NDh684/5ObKJvSIBsFuBg34pn8A10scS1ef2wfXwnjOroUWE5QgipzyioIaQWSbuePurph/e7+8LSxBClYganHiYDAAKc+StRS0cwLR/JnyjPpHwm3/9NDMbgli4IdLbEpnc7VPreLtYm6BPkREnChJAGi3JqCKklDMPgr9svALAtJ0KhAMG+9vjvUTJXxs3GBFfm9sXMfWH4tjzpFwBEhrLlCIQKMcnGKoIZQghpLKilhhAtEksYbDgbxbW8yItNL0BucRkAoHv5OklBLvyWGV8Hc7jZmGLfR13hYcvPmTk1syc+GxiIZ98M0VHtCSGkfqOWGkK06PSjZKw5EQkA2Pl+Z/Rq6siduxuXCQBo5moFZyt2BJI0N0bKUkVSsFSAs6VS9xQhhBCZarXUlJaWIj4+HpGRkcjIyNB2nQipt27Gyv5/WHnsMWLS8vHTmacoLhNj1h/3AAB+5fPLAGw3lNT6d9rVXkUJIaQBUrulJjc3F7///jv27t2LGzduoKSkhFv918PDAwMGDMCHH36ITp066bK+hNRZJWUS/HIxhtt/lJSDPmvPAQDWnnzCHZd2PQFs99OoDh64E5eJwS1daq2uhBDSEKnVUrNu3Tr4+Phg+/btCA0NxcGDBxEWFoYnT57g6tWrWLRoEcrKyjBgwAAMGjQIT58+1XW9Cal18mszAcAXf95H6LrzSMktQkFJGZouOMadc7YSVXgf+eHZAoEAa0a1wenZvWFoQCluhBBSE2q11Ny8eRMXLlxAixYtVJ7v3Lkz3n//fWzatAnbt2/HxYsXERAQoNWKEqJP/4Ql4NO9YWjiaI7Ts3ujVCzBvlvxAID9t15g47lnXFk/R3M0cbRQmSzsZm0CY0MKXgghRBfUCmr27Nmj1s1EIhEmT55cowoRUhd9ujcMAPAsNR/JOUWQMLJWm93X45BXPqoJAA583B3rTkZy+yJDIYrLJACADj4VT5BHCCGkZqr9T8aoqCicOHEChYWFANg5OAhpiBKzCnn7919kIzmnmNtPkDt/ZW5fWJsawbV80jwAcLMxxa/jOsLa1AhfDArUfYUJIaSR0jioSU9PR2hoKJo2bYohQ4YgKSkJADBx4kTMnj1b6xUkpLYwDIOX2UVKAfpH/7vN23+eno+UHOW1muYNDoJbeTDjKrdoZFNnC/Rr5ox7iwYozT1DCCFEezQOambOnAlDQ0PExcXBzEz2B/qtt97C8ePHtVo5QmrTkn8fosuK09gqN4KJYRhuvSbpTL4xaflIzi1Wur6p3BwybnItNe29bHVUY0IIIfI0DmpOnjyJVatWwcODvyheQEAAnj9/rrWKEVJbGIZBfEYBdlyJBQAsP/qIO5ciF7wsHNYcABvUqGqp8ZZbNVs+qOnoQ0ENIYTUBo1nFM7Pz+e10EhlZGRAJKp4GCshddWhe4lcIrBUREI2HC1F+OVCNHesjacNAODKs3S4yHUvScl3LblZm+CVNm4oE0vQxsNGF9UmhBCiQOOgJiQkBL/99huWLVsGgJ1nQyKRYPXq1ejTp4/WK0iIrm0+H610bNj6S7z95q5WaOZixe1fj1aeSVt+qLZAIKAZggkhpJZpHNSsXr0a/fr1w61bt1BSUoLPP/8cDx48QEZGBi5fvqyLOhKidam5xVh2+CHe7uwJG7OK11uS+rhPE5gaG6CFmxUeJObwRjwB7LBtQggh+qXxX+KWLVviyZMn6NGjB1599VXk5+fjtddew927d9GkSRNd1JEQreu0/D8cupeIz/bfx0sV+TGKpAtT+tib847PH9IMxoZCnP+MWikJIUTfqrVKt7W1NebPn6/tuhBSK9Ly+HPMGJQPa1owtBm87c0x6bdbAIAf3m6LT/eGYVQHD271bC97fj7Z8LZumNTTr5ZqTgghpDIaBzX3799XeVwgEMDExAReXl6UMEzqDImEgVA6FrvclWfpvH2xhIGBUIDx3XxgaCDEsU9DYGduDGcrE7za1p1X1kDAv5edubFuKk4IIURjGgc1bdu2haD8D7t0kjKB3B96IyMjvPXWW9i8eTNMTJRHiBBSW84/ScW4bTcQ4GSBY5+G4FFSLlq4WWH6nrtKZd1sTLgFJZu5WimdlxrYwgU/nY3i9o1oEUpCCKkzNP6LfODAAQQEBGDLli24d+8e7t27hy1btiAwMBC7d+/Gr7/+ijNnzmDBggW6qC8hVcouLAXDMDh4NwEA8DQlDxN23MQrP13CnD/vceXc5eaSkd+uTEv3igMeQggh+qVxS83y5cvxww8/YODAgdyxVq1awcPDAwsXLsSNGzdgbm6O2bNnY+3atVqtLCFS2YWl+PLvcHw+KBDecsm716LT8faWaxjfzQfyPUUXn6YBAP6+k8Ad+6SvP+b+HQ4AcLNWL6gRCATo2dQRF56kolsTey18EkIIIdqicVATHh4Ob29vpePe3t4ID2e/INq2bcutCUWILiw59ABHwpNwJDwJh6Z1h62ZMTztzPD7NXZW6x1XYtHFr+IVsV9r5w5fB1kwpGoyvYpsfa8jDt1LRGgzp+p/AEIIIVqncfdTUFAQVq5ciZKSEu5YaWkpVq5ciaCgIABAQkICnJ2dtVdLQuRIJAz+vitrcRn+02WErD7LLXcg9fhlboX3aOpiyQtqNFlo0thQiDc6eMDGjJKECSGkLtG4pWbDhg0YPnw4PDw80Lp1awBs641YLMbhw4cBANHR0fj444+1W1NCAJSKJfjlovIMwADwIrMQSdmyOWeyCkorvE+QiyUcLUUwNzZAfokY/k4WWq8rIYSQ2qVxUNOtWzfExMRg165dePLkCQBg1KhRGD16NCwt2VWKx44dq91aElJu/Zko/Hj6qcpzl6LSeAtQAoC5sQHe6+aD1NxiRL7M5Vbc7uHvAIFAgC3vdcSLzAJ0okUnCSGk3qvW5HuWlpaYPHmytutCiEopuUXovPw0xgR7Ydf1OO74itdawdlKhP9dfY6zkanYfyte6Vpve3N8Mai8WzSrEN1XnsH+yV254dvd/R1q50MQQgjRuWoFNQDw8OFDxMXF8XJrAGD48OE1rhQh8jadY7ub5AMaAOjf3BkOFiIkZBXhbGQq7sRlKV3raccfth27cqhO60oIIUR/NA5qoqOjMXLkSISHh0MgEChNwCcWi7VbQ9LoZOSXIL+4DMaGQiw4GIGrCjMAA8DJmT3hYMHOXN3Egb8eU7cm9tyswS5WNAEkIYQ0FhqPfvr000/h6+uLlJQUmJmZ4cGDB7hw4QI6duyIc+fO6aCKpLF5Y9MV9Fl7Dh/+dgunHiYjr7iMd753oCOaOlty+4pJvn2DZEOtHS1pyQ5CCGksNA5qrl69iqVLl8LBwQFCoRBCoRA9evTAihUrMH36dF3UkTQihSViRKfmo0zC4N6LbJVl/Bz4QYxi4DKynWy9Jh+FVhxCCCENl8ZBjVgs5kY5OTg4IDExEQA7+V5kZKR2a0canZc5RVWW8VZYKVsgEKCjNzt66bOBgbC3EGHZqy3wfndfDGrhopN6EkIIqXs0zqlp2bIl7t27B19fXwQHB2P16tUwNjbGli1b4Ofnp4s6kkYiv7gMO6/EqjznbW+G5+nsxHoetspLGmwY0x7XotPxSms3AMDYrj66qiYhhJA6SuOgZsGCBcjPzwcALF26FMOGDUNISAjs7e2xb98+rVeQNB5Df7yI2PQC3jFnKxG2vtcJHramaLfsFADA00559l9nKxO82tZd6TghhJDGQ+OgRn4hS39/fzx+/BgZGRmwtbXlRkARUh2KAQ0AMAzQysMaALBlbAdkFpTwkoQJIYQQKY1yakpLS2FoaIiIiAjecTs7O50HNCtWrECnTp1gaWkJJycnjBgxgnJ46rnYtHz4zD0Cn7lHkJRdyB3v4C2b3dfeQpYEPKCFC97q5FWrdSSEEFJ/aBTUGBkZwcvLSy9z0Zw/fx5Tp07FtWvXcOrUKZSWlmLAgAFcVxipXx4kZqP32nPc/o7yXBoPW1P8NaUbVr/eGi5WJlgyvIV+KkgIIaTeETDS2fPU9Ouvv+Lvv//G//73P9jZ2emqXlVKTU2Fk5MTzp8/j549e6p1TU5ODqytrZGdnQ0rKysd15BUxmfuEd6+h60pXmQWIrSZM7aO66inWhFCCKmL1P3+1jin5qeffkJUVBTc3Nzg7e0Nc3P+PCB37tzRvLbVkJ3NzmGiz8CKVE++wmR6ALvCNgA0c6V8GUIIIdWjcVAzYsQIHVRDMxKJBDNmzED37t3RsmXLCssVFxejuFi2anNOTk5tVI9U4WZsBrc9KcQXv1yM4faDXKgFjRBCSPVoHNQsWrRIF/XQyNSpUxEREYFLly5VWm7FihVYsmRJLdWKVCUiIRuvbbyCkjIJACDIxRLzhzbHzdhMhMVnAQC6+FHLGyGEkOrROKcGALKysvDnn3/i2bNn+Oyzz2BnZ4c7d+7A2dkZ7u66nStk2rRp+Oeff3DhwgX4+vpWWlZVS42npyfl1OhB95VnkJBVyDs2b3AQPurVBIlZhTj9OAXvBnvRtACEEEKU6Cyn5v79+wgNDYW1tTViY2MxadIk2NnZ4e+//0ZcXBx+++23GlW8IgzD4JNPPsGBAwdw7ty5KgMaABCJRBCJaEFDfbsVm6EU0ADAyPZsAOxmY4qxXbxru1qEEEIaGI3Xfpo1axbGjx+Pp0+fwsTEhDs+ZMgQXLhwQauVkzd16lT8/vvv2L17NywtLfHy5Uu8fPkShYXKX5akbnmQqJzL1NzVCk6WJipKE0IIIdWjcUvNzZs3sXnzZqXj7u7uePnypVYqpcrGjRsBAL179+Yd3759O8aPH6+z9yU19yiJDWqm9mmCOQMCkZ5fAgcLakEjhBCiXRoHNSKRSOUooidPnsDR0VErlVKlGqk/pA6IzyjA3pvxAIAWbtYQCAQU0BBCCNEJjbufhg8fjqVLl6K0tBQAIBAIEBcXhy+++AKvv/661itI6ieGYSCRMHh7yzXuWAs3Ss4mhBCiOxoHNd9++y3y8vLg5OSEwsJC9OrVC/7+/rC0tMTy5ct1UUdSzzAMg7c2X0M3hRFPnrbKq2sTQggh2qJx95O1tTVOnTqFS5cu4f79+8jLy0P79u0RGhqqi/qReig1rxg35CbYA4Df3u8MoZCGaxNCCNEdjYOa+Ph4eHp6okePHujRo4cu6kTquehU/iKjrT2s0bOp7vKtCCGEEKAa3U8+Pj7o1asXfvnlF2RmZuqiTqSeUwxqXK1p6DYhhBDd0ziouXXrFjp37oylS5fC1dUVI0aMwJ9//smbuZc0btGpefquAiGEkEZI46CmXbt2WLNmDeLi4nDs2DE4Ojriww8/hLOzM95//31d1JHUM1svxfD2X2/voaeaEEIIaUyqtfaTojt37mDixIm4f/8+xGKxNuqlE+quHUGqr6hUjKCFxwEA345qg2A/O7jbmNKaToQQQqpN3e9vjVtqpF68eIHVq1ejbdu26Ny5MywsLLBhw4bq3o40EC8yZUO4R7Zzh4etGQU0hBBCaoXGo582b96M3bt34/LlywgKCsKYMWPwzz//wNubFiQkQFI2G9Q0dbagIdyEEEJqlcZBzddff4133nkHP/74I9q0aaOLOpF6LCmrCADgam2q55oQQghpbDQOauLi4qg7gVRIOoOwmw0N4yaEEFK7NA5qpAFNQUEB4uLiUFJSwjvfunVr7dSM1DtXnqXhh9NPAQDuNtRSQwghpHZpHNSkpqZi/PjxOH78uMrzdXn0E9Gd3KJSjP7lOrfvRkENIYSQWqbx6KcZM2YgOzsb169fh6mpKY4fP46dO3ciICAAhw4d0kUdST1w8kEybz/IhYbME0IIqV0at9ScOXMG//zzDzp27AihUAhvb2/0798fVlZWWLFiBYYOHaqLepI67mFSDre9bERLNHejoIYQQkjt0rilJj8/H05OTgAAW1tbpKamAgBatWqFO3fuaLd2pN54kpwLAFjxWiuM7ULD+wkhhNQ+jYOawMBAREZGAgDatGmDzZs3IyEhAZs2bYKrq6vWK0jqh4tP0wAATZ0t9VwTQgghjZXG3U+ffvopkpKSAACLFi3CoEGDsGvXLhgbG2PHjh3arh+pBx7JdT0FuVBQQwghRD80DmreffddbrtDhw54/vw5Hj9+DC8vLzg4OGi1cqR+uPg0lds2F2n8nxQhhBCiFTX+BjIzM0P79u21URdST4UnsC01s/s31XNNCCGENGbVXtCSEKkHidkAgFYe1nquCSGEkMaMghpSI1kFJYhOzQcAtHCjoIYQQoj+UFBDauR4xEsAgKXIEI6WIj3XhhBCSGNGQQ2pkWepeQCo64kQQoj+aZwofP/+fZXHBQIBTExM4OXlBZGI/sXeWMRnsKtyD2zhoueaEEIIaew0Dmratm3LrdStipGREd566y1s3rwZJiYmNaocqfuSc4sAAC7W9LsmhBCiXxp3Px04cAABAQHYsmULwsLCEBYWhi1btiAwMBC7d+/Gr7/+ijNnzmDBggW6qC+pY1JyigGA8mkIIYToncYtNcuXL8cPP/yAgQMHcsdatWoFDw8PLFy4EDdu3IC5uTlmz56NtWvXarWypG5hGAapuWxQ40RBDSGEED3TuKUmPDwc3t7KCxZ6e3sjPDwcANtFJV1KgTRcKbnFKBFLAAAOFhTUEEII0S+Ng5qgoCCsXLkSJSUl3LHS0lKsXLkSQUFBAICEhAQ4Oztrr5akTpKuzC0yFMLEyEDPtSGEENLYadz9tGHDBgwfPhweHh5o3bo1ALb1RiwW4/DhwwCA6OhofPzxx9qtKalznqcXAAC6NrHXc00IIYSQagQ13bp1Q0xMDHbt2oUnT54AAEaNGoXRo0fD0pJdoXns2LHarSWpkxKz2OHc3nZmeq4JIYQQUs0FLS0tLTF58mRt14XUM0nZ7HBuVxtTPdeEEEIIqWZQ8/TpU5w9exYpKSmQSCS8c1999ZVWKkbqvoTylhpXmqOGEEJIHaBxUPPLL79gypQpcHBwgIuLC28iPoFAQEFNIyLtfnKnlhpCCCF1gMZBzddff43ly5fjiy++0EV9SD0hljB4Wd795EZBDSGEkDpA4yHdmZmZGDVqlC7qQuqR1NxilEkYGAgFNPEeIYSQOkHjoGbUqFE4efKkLupC6pG4DHY4t6OFCIYGtNg7IYQQ/dO4+8nf3x8LFy7EtWvX0KpVKxgZGfHOT58+XWuVI3VLQUkZ1pyIxPA2bnh363UAQE5RqZ5rRQghhLAEDMMwmlzg6+tb8c0EAkRHR9e4UrqSk5MDa2trZGdnw8rKSt/VqXeW/PsA2y/H8o61cLPCkekh+qkQIYSQRkHd72+NW2piYmJqVDFSf118mqZ0bErvJnqoCSGEEKKMkiGI2gzkhu8DwNQ+TTCstZueakMIIYTwqdVSM2vWLCxbtgzm5uaYNWtWpWXXrVunlYqRuicpm52X5sSMnmjqbMGbo4gQQgjRN7WCmrt376K0tJTbrgh9yTVc6XnFyCkqg0AAuNua0u+aEEJInaNWUHP27FmV2/qwYcMGrFmzBi9fvkSbNm2wfv16dO7cWa91agykSyI4WYpgIarW6hqEEEKITmmcU/P777+joKBAF3Wp0r59+zBr1iwsWrQId+7cQZs2bTBw4ECkpKTopT6NiXT2YBcrWueJEEJI3aRxUDNz5kw4OTlh9OjROHr0KMRisS7qpdK6deswadIkTJgwAc2bN8emTZtgZmaGbdu21VodGqvkHDaocaaghhBCSB2lcVCTlJSEvXv3QiAQ4M0334SrqyumTp2KK1eu6KJ+nJKSEty+fRuhoaHcMaFQiNDQUFy9elXlNcXFxcjJyeH9EPUVlohx+3kGJBIGL8uDGhdakZsQQkgdpXFyhKGhIYYNG4Zhw4ahoKAABw4cwO7du9GnTx94eHjg2bNnuqgn0tLSIBaL4ezszDvu7OyMx48fq7xmxYoVWLJkiU7q0xi8t+06bsZmYlhrVy4xmIIaQgghdVWN5qkxMzPDwIEDMXjwYAQEBCA2NlZL1dKOefPmITs7m/uJj4/XyfsUFxfi+r9bwUhqrytO19LyijEmYRluiibjcfhN/HsvEQDl1BBCCKm7qjWMRdpCs2vXLpw+fRqenp5455138Oeff2q7fhwHBwcYGBggOTmZdzw5ORkuLi4qrxGJRBCJdLuCtFgsRtKqzgiWxOG2yBQdBozRyn3LSksRfuMMciKOwcjEHEFDPoGdo+rPqQtPo2MwwoDtUpxieAizSz8GALjbmNZaHQghhBBNaNxS8/bbb8PJyQkzZ86En58fzp07h6ioKCxbtgxBQUG6qCMAwNjYGB06dMDp06e5YxKJBKdPn0bXrl119r5VMTAwQLJrHwCA2bXvIRZLanQ/sViMs/t/QvLy5mh36k30StqObjE/QfRTG4Rtm47irCRtVLtCSS+iERlxB5LDskkWWwtkS2P4Oprr9P0JIYSQ6tJ4QcsxY8ZgzJgxGDhwIAwMDHRVL5X27duHcePGYfPmzejcuTO+//57/PHHH3j8+LFSro0qulrQMjc9AUbr28AEpbht1gPCliPg7t8WTgEdAblJ6uIf3UBK3FOU5KXBJOMxiktKUQRjGJblwzPrJiyZXBijFBZg54TJhRniLdrCoiAeXpL48mPmiOu/BS26D9Na/ZOe3kXyld1wjDsKd/ELlWW29bkOJ1srWhaBEEJIrVP3+1vjoEbffvrpJ27yvbZt2+LHH39EcHCwWtfqcpXuu3uXod3jtbxjV8xD0erj35D3MhppB75Aq7zLat0rD6aIavohWr72BQxNzCERS3Dl+C443vwWgYhBCWOAWz4fwbv/ZLh7eENSVoqM1CSk3D+BotjbKCnKQ5m5C9w6vgJn70CYmZkjNTEGKTEPITQyhpGlMywtzZGZnICS89+iTfFt7r3FjABiCJFpYAemw/twubmKPRE4BHhnj9aeFyGEEKIunQY158+fx9q1a/Ho0SMAQPPmzfHZZ58hJCSk+jWuBboMagDg6e0zyLq+Cxbp9xFY9hRCAYMXcIILkwZDgQRljBDPjfxQYmiJdHN/GIjMYcoUQiIpgygwFKZOTZAnNoC3jz+srW2U7p+Xn4enm8agXe45AGwAkiqwhz2TCSNB9ZOUSxkDhJl0Qm6TV9Cyz1uwsbGFkYEQAqEQ+KEtkFne/TTuMOCr4ncsLgMMaJZhQgghuqGzoOb333/HhAkT8Nprr6F79+4AgMuXL+PAgQPYsWMHRo8eXbOa65Cugxp50dcOwfX4BzBFMQDgjqgzLF9ZgYCWHWt0X0Yixu1/foLNw9/hX/qEdy5e4IZntt1hYmkPo9QI+OXfg60gFwBQxBghzsALBhDDWpIFA6YMEggRZ90JziO+hptfc9VvWJQDrPRkt63cgVkP+edv/AKc+BIYtQMIGlqjz0YIIYSoorOgplmzZvjwww8xc+ZM3vF169bhl19+4Vpv6qLaDGoAIC06DM+v/gVj72C07D5U64tA5qXEIvlFNCycvGHv7AlDI2OlMvmFxcjNz4etlTlExtUcCXZwKhD2O7s9ZC3Q6QM2V0giBpbascct3YDZdfd3TwghpP7SWVAjEonw4MED+Pv7845HRUWhZcuWKCoqql6Na0FtBzUNRmIYsKWXbH/QSqDLFODmVuDIbNnxRVm8xGhCCCFEG9T9/tZ4SLenpydvWLXUf//9B09PT01vR+oD+yb8/Yf/sK8Jd/jHz62onfoQQgghKmic3Tl79mxMnz4dYWFh6NatGwA2p2bHjh344YcftF5BUgeILPn7ceVrbRVl84+fXwU4NAVavVE79SKEEELkVGv004EDB/Dtt99y+TPNmjXDZ599hldffVXrFdQm6n6qgTPLgds7gPwU2THXtkBSmHLZxdnKxwghhJBqarDz1NQEBTVasNha+VjPz4ELq2X7814ot+4QQggh1aSznBp5eXl5yMnJ4f2QBu7dv5WPeXfj76c9US5DCCGE6JjGQU1MTAyGDh0Kc3NzWFtbw9bWFra2trCxsYGtra0u6kjqEv9+wKSz/GMeHYGQObL9jBgQQgghtU3jROF3330XDMNg27ZtcHZ21vrcK6QesPPl74ssgX4LgZxE4N5u2QzEhBBCSC3SOKi5d+8ebt++jcDAQF3Uh9QHpnItcu3GyralwU5GbK1WhxBCCAGqEdR06tQJ8fHxFNQ0douzgZwkwNxRdszag33NSdBPnQghhDRqGgc1W7duxeTJk5GQkICWLVvCyMiId75169Zaqxyp46xc+fum5UsmFGXVelUIIYQQjYOa1NRUPHv2DBMmTOCOCQQCMAwDgUAAsbj6q0WTes7Uhn0tzNRrNQghhDROGgc177//Ptq1a4c9e/ZQojDhk+baZMYCJQWAsZleq0MIIaRx0Tioef78OQ4dOqS0oCUhsHCSbV/7Geg5p+KyhBBCiJZpPE9N3759ce/ePV3UhdR3praASyt2+8wy4PlV/daHEEJIo6JxS80rr7yCmTNnIjw8HK1atVJKFB4+fLjWKkfqocFrgO2D2O3tg4CJpwDPzvqtEyGEkEZB47WfhMKKG3fqeqIwrf1UCxgGWGIj2zcQAQtTKixOCCGEVEXd72+NW2okEkmNKkYaOMXEcXExUFoIGJnqpz6EEEIajRotaEmISt2m8/fTn+mnHoQQQhoVtYKavXv3qn3D+Ph4XL58udoVIg1A/6XAZLn/BrLi9FcXQgghjYZaQc3GjRvRrFkzrF69Go8ePVI6n52djaNHj2L06NFo37490tPTtV5RUo8IBIBLS9n+pXX6qwshhJBGQ62cmvPnz+PQoUNYv3495s2bB3Nzczg7O8PExASZmZl4+fIlHBwcMH78eERERMDZ2VnX9Sb1gdAQkJQB7h31XRNCCCGNgNqJwsOHD8fw4cORlpaGS5cu4fnz5ygsLISDgwPatWuHdu3aVToyijRCIbOB86vYZGFCCCFExzQe/eTg4IARI0booCqkwbF0YV9vbWPzbESW+q0PIYSQBo2aVoju+ITItv94DyjO019dCCGENHgU1BDdcQiQbT87A6xwB56c1F99CCGENGgU1BDdCp7M3989CsijGYYJIYRoHwU1RLdajFQ+tjYAoJmpCSGEaBkFNUS3XFqrPh53pXbrQQghpMFTa/TTrFmz1L7hunU00RqRY2wGfJkEGBgDv70KPL/EHt8xFFicrd+6EUIIaVDUCmru3r3L279z5w7KysoQGBgIAHjy5AkMDAzQoUMH7deQ1H/GZuzrhCPAt0FAbhK7zzDKC2ASQggh1aRWUHP27Flue926dbC0tMTOnTtha2sLAMjMzMSECRMQEhJS0S0IYU06C6wLYrd3vQE4BAIDlgFCA/3WixBCSL0nYBiG0eQCd3d3nDx5Ei1atOAdj4iIwIABA5CYmKjVCmpTTk4OrK2tkZ2dDSsrK31Xp3GSiIGldsrHF6YDBhrPBUkIIaQRUPf7W+NE4ZycHKSmpiodT01NRW5urqa3I42N0AAIHKp8/PfXar8uhBBCGhSNg5qRI0diwoQJ+Pvvv/HixQu8ePECf/31FyZOnIjXXqMvJqIGY3PlYzHngbKS2q8LIYSQBkPj9v5NmzZhzpw5GD16NEpLS9mbGBpi4sSJWLNmjdYrSBqg5sOB8D+Uj+cmArY+tV4dQgghDYNGOTVisRiXL19Gq1atYGxsjGfPngEAmjRpAnNzFf/6rmMop6aOYBjg6SnAoyNgYgMsZRPOMfYg0KSPPmtGCCGkDlL3+1vjRGETExM8evQIvr6+Na5kbaOgpo5abC3bNjIH5tfdZHNCCCG1T2eJwi1btkR0dHSNKkdIhUrzgfgb+q4FIYSQekjjoObrr7/GnDlzcPjwYSQlJSEnJ4f3Q4jGxh7k7z+nJRQIIYRoTuPuJ6FQFgcJ5GaDZRgGAoEAYrFYe7XTMup+qqMU565xaw98eLbi8oQQQhoVdb+/NR79JD+7MCFaoTibsJGpfupBCCGkXtM4qOnVq5cu6lGp2NhYLFu2DGfOnMHLly/h5uaGd999F/Pnz4exsXGt14fowLDvgcMz2O0i6sYkhBCiuWrPS19QUIC4uDiUlPAnTGvdunWNK6Xo8ePHkEgk2Lx5M/z9/REREYFJkyYhPz8fa9eu1fr7ET3oOAFwbw9s7gmkPQEKMoDUx8DxucArPwJubfVdQ0IIIXWcxjk1qampmDBhAo4dO6byfG3l1KxZswYbN27UaCQW5dTUcQUZwGoVUwWYOwKfRdV+fQghhNQJOhvSPWPGDGRlZeH69eswNTXF8ePHsXPnTgQEBODQoUM1qrQmsrOzYWenYmFEOcXFxTQ6qz4xtVV9PD8VOLUICP+zdutDCCGkXtG4++nMmTP4559/0LFjRwiFQnh7e6N///6wsrLCihUrMHSoisUKtSwqKgrr16+vsutpxYoVWLJkic7rQ7REbjSdksvfs68tRionFhNCCCGoRktNfn4+nJycAAC2trbcit2tWrXCnTt3NLrX3LlzIRAIKv15/Pgx75qEhAQMGjQIo0aNwqRJkyq9/7x585Cdnc39xMfHa1Q/Ugf90pedgbggQ981IYQQUsdo3FITGBiIyMhI+Pj4oE2bNti8eTN8fHywadMmuLq6anSv2bNnY/z48ZWW8fPz47YTExPRp08fdOvWDVu2bKny/iKRCCKRSKM6ET376CKwOaTi80lh7OufE4D3/qmVKhFCCKkfNA5qPv30UyQlJQEAFi1ahEGDBmHXrl0wNjbGjh07NLqXo6MjHB0d1SqbkJCAPn36oEOHDti+fTtvEkDSgLi2BhZn89eDUsXQRLZdWggsdwF6zwN6z9Vt/QghhNRZGo9+UlRQUIDHjx/Dy8sLDg4O2qoXT0JCAnr37g1vb2/s3LkTBgaynAoXFxe170Ojn+qR9GfA0c/Yod773lU+798feLc8cXhDFyD1Ebu9IBUwpLmLCCGkIdHZjMLR0dG8LiEzMzO0b9++erVU06lTpxAVFYWoqCh4eHjwztUwJiN1lX0TYOzf7PYnd4Cke2yXk1TeS9m2NKABgORwwL1D7dSREEJInaJxH46/vz+8vLwwduxY/Prrr4iK0v38IePHjwfDMCp/SCNg3wTwDOYfy0thXxMUktPv/l47dSKEEFLnaBzUxMfHY8WKFTA1NcXq1avRtGlTeHh4YMyYMdi6dasu6kgIYKnQzZifyi6EmZPIP54ZW2tVIoQQUrdoHNS4u7tjzJgx2LJlCyIjIxEZGYnQ0FD88ccf+Oijj3RRR0LYuWlGbAS6TGX3GQkb2GQ9Z/cF5f8p07pRhBDSaGmcU1NQUIBLly7h3LlzOHfuHO7evYugoCBMmzYNvXv31kEVCSnXdjT7GvEnkJcM5L4Erm9mj/n2AqLPAtk0FxEhhDRWGgc1NjY2sLW1xZgxYzB37lyEhITA1raC6e0J0QUTGzao2aJixfi8ZGDbYEBkCbz1O42EIoSQRkTj7qchQ4ZALBZj79692Lt3L/bv348nT57oom6EqJYWqXwsdDFgZM5ux10Bnp4AvnYElrsBxXm1Wj1CCCH6oXFQc/DgQaSlpeH48ePo2rUrTp48iZCQEC7XhhCds/NTPubWlv1RVJoP3Nis6xoRQgipA6o9LW+rVq3QvXt3dO3aFZ06dUJKSgr27dunzboRotpHF/j7I8uXzHBrp7p8/A3d1ocQQkidoHFQs27dOgwfPhz29vYIDg7Gnj170LRpU/z111/c4paE6JTIEvg8hp2Ub8JxoM1b7PHWb6ouX5zLzlC82AZ49G+tVZMQQkjt0niZhE6dOqFXr17o3bs3QkJCYG1dxRo9dQgtk9AIJN5lc2h2DpMds/IAcl7I9mc9Aqzcar9uhBBCqkXd7+8ar/1Un1BQ04ikRrLDvW/9CkAAQOE/88XZ+qgVIYSQalD3+7taOTUXL17Eu+++i65duyIhIQEA8L///Q+XLl2qXm0J0TbHQGDot4CxBZQCGgCIPlfbNSKEEKJjGgc1f/31FwYOHAhTU1PcvXsXxcXFAIDs7Gx88803Wq8gIdUmEAAOTVWf++1VYLE1EEuBOCGENBQaBzVff/01Nm3ahF9++QVGRkbc8e7du+POnTuVXEmIHiiOiOo1l7+/Yyjw6DAb4Kyn1b0JIaQ+0zioiYyMRM+ePZWOW1tbIysrSxt1IkR7bLz4+33mKZfZVz6/UnoU24LTeNLMCCGkQdE4qHFxcUFUVJTS8UuXLsHPT8WkaITok0dH2faH52SvTQepLh99Dtj3ro4rRQghRBc0DmomTZqETz/9FNevX4dAIEBiYiJ27dqFOXPmYMqUKbqoIyHV59MDGHsA+PC8rCvKrR0weh9gIFJ9zePDtVc/QgghWqPxgpZz586FRCJBv379UFBQgJ49e0IkEmHOnDn45JNPdFFHQmqmSV/VxzuMr3gJhaJswKT+zMFECCGkGi01AoEA8+fPR0ZGBiIiInDt2jWkpqZi2bJlKCws1EUdCdEN5+aybfcOwJi/AEtXdn+lFyAu1U+9CCGEVEu1134yNjZG8+bN0blzZxgZGWHdunXw9fXVZt0I0S0nhaAmIBSwdJEdW+ZQ+3UihBBSbWoHNcXFxZg3bx46duyIbt264eDBgwCA7du3w9fXF9999x1mzpypq3oSon0OAbLtNm+zr12m8suUldRefQghhNSI2jk1X331FTZv3ozQ0FBcuXIFo0aNwoQJE3Dt2jWsW7cOo0aNgoGBgS7rSoh2mdgATfoBBWmAUwv2WOtRgMgC2FMe5Kxvz7biPDwIvH8C8Oqir9oSQgipgtpBzf79+/Hbb79h+PDhiIiIQOvWrVFWVoZ79+5BIBDoso6E6IZAAIz9W/l44GDZdnY8+wMA2wbSmlGEEFKHqd399OLFC3TowM642rJlS4hEIsycOZMCGtIwDf9J9fHYy7VbD0IIIWpTO6gRi8UwNjbm9g0NDWFhYaGTShGid+3Hqj6+YwhwYj67nf4MeHELOL+GXWYhbHft1Y8QQogSAcOoNye8UCjE4MGDIRKxE5b9+++/6Nu3L8zNzXnl/v5bRXN+HaHu0uWEAGCXSzi9FJCUAVd+5J+b/xJY7qJ8Tf+lgNAQ6PIx271FCCGkxtT9/lY7qJkwYYJab7x9+3b1aqgHFNSQapNIgBtbgONfqFf+7T1A0BAg4TaQlwoEVrAsAyGEkCppPahpCCioITW20oudbbgqTQcBQ9cB35XPhePcCmj/HhD8oW7rRwghDZC639/VnnyPkEZpskKi8MAVwMT/gIXpgJFcV+yT48BtuVbL5HDg2GfAf0tqp56EENIIUVBDiCZsPIF5CeyimINWAV0/Bjw7AQaGwJcJwKxHsrJJ95Wvv7ROts0wbKIxTfBHCCFaQUENIZoSWQAfngO6TOYfFwgAKzfApTW7//SE6utL8tnXu78DW/sBp77SWVUJIaQxoaCGEG3zDObvdxgP9JBbQiT6PPt6aBr7en1jrVSLEEIaOgpqCNE2l5b8/dDF7I9jELufGQvkp/PLFOeyI6yeXwFKi2qhkoQQ0vBQUEOItrUdI9v26ASY2rLbzV9lX/9bpNw1deorYF0zYPtg4OK3tVNPQghpYGhINyG6kv4MsPYADNkJK3F1A3DiS/Wula4xJS4Fjs8F7P2BLlN0U09CCKnjaEg3Ifpm30QW0ABsbo0i/1DV1xblsK9XfgRubmUDm+JcrVeREEIaEgpqCKktxubsvDby2r2ruuzTk+zr6aWyYy/DgZwkdp2pvWOUr4m5CFzbyA4Vl5f+DPhrEpD6pPp1J4SQesBQ3xUgpFHp+jEQMAD4iV3xHu4d2Llt9o0FTG2AqP/Y439NBKzc+dfuGgWU5LHbjw8DxXns8HIAyIoDdg5jt40t+AtynvqKLf/oELAgWWcfjRBC9I2CGkJqm4M/8OZv7LaNF/s66TT7uthaVm67wnpR0oBGKi2SDYoAIOme7PjjI/yg5nn5LMhlNKqKENKwUfcTIfrQ/FXZaCh5H5xW/x4pcrMXZ8XLtrPjgchjwNqmQNRpoDCr2tUkhJD6hIIaQuoSj46ykU9SrUYBw39SLisf1GRE84/veRvISwZ+fw2AXI5NWbFWq0sIIXUJBTWE1EUWzrJtj05sd9J7/7D7dk3Y1+eXgYIMdls+wGHEFd83+4V260kIIXUIBTWE1EVjD8q2m49gX/16s604I8qXVUi8C6z2ZbuaUsuDGiOzyu+bFae9OkYeAw5OpRmQCSF1BiUKE1IXOTcHvspkF8kUCPjnXFvz9/e8zb4KjYCA/sDDf5TvZ2zBJhonR7DBkeI9NZWfJntfE2tg0Dc1ux8hhGgBtdQQUlcJhaqDDyNT1RP5SUqB9u/J9rt9orx9cgGwxAZIus/u/zsDWOnNTzTm7icB0qKU570BgNhLsu30KOD5VX4XGCGE6EG9C2qKi4vRtm1bCAQChIWF6bs6hOjHKz+wXVFdp8mONekH+PUFIGBzckKXAjMi2B/vbvzrN4cAsZeB29uBoizg2s/K7/HvJ+x8Okts2eUaXtxmh5xf2wgUyC3ImXCbHX7+cxe2S4oQQvSk3gU1n3/+Odzc3PRdDULqhoHLgS+eA5MvA2P/Zlt3FmcBc56w2zae7I+tr/K1O4bItq/9zAY58u7+Xr7BABF/AfvHsbvH5wK5SbJyBWmy7ds7tfGpCCGkWupVTs2xY8dw8uRJ/PXXXzh2jP5FSAgAdiZiU5vKy1i5saOminOB/BTVZeSDHEWZz/ldYfHXVZer6N6EEFIL6k1Qk5ycjEmTJuHgwYMwM6tihEe54uJiFBfL5uXIycnRVfUIqduEBsDki2w3kpEpsNy18qHfitKjgIJM2X7MBdXlEm6zuTVOzdS7b14KkPKQTV4mhJAaqhfdTwzDYPz48Zg8eTI6duyo9nUrVqyAtbU19+Pp6anDWhJSxxmbsy068iuHA8DQbwGfkMqvDf8DKFFzlXD5RTgrU1oIrA0AfnsViDyu3jWEEFIJvQY1c+fOhUAgqPTn8ePHWL9+PXJzczFv3jyN7j9v3jxkZ2dzP/HxKkZ4ENIYya8O3nYMMP4wm3i8OBuY/xIwMmfPKS7bIDCo+J5ty+8ZeZRNKP5nauV1WN9Btr3nLfa1KFv1aCtCCFGDgGH09xckNTUV6enplZbx8/PDm2++iX///RcCuT59sVgMAwMDjBkzBjt3qpecmJOTA2tra2RnZ8PKyqpGdSekXisrBo7OAXx6Aq1HKZ8vKWAXwDSz4y+yCbCLaCbcZreHrgOOzAL8Q4FXfgS+a84vuyhL9bB0iRhYasc/1ukD4OZWwKUVMPmS8jWEkEZL3e9vvQY16oqLi+PlwyQmJmLgwIH4888/ERwcDA8PD7XuQ0ENIdVweCZwaxu77dIamHQWiLsKeHdnR1jJ2z8eeHBAtv/GdqDla7L93JeAgTEbFO16o+L3XJgOGFSR8hdzETj7DfDaZtlq57q23BUoLQCmXAUcg4CltuzxGeG1VwdCGiF1v7/rRaKwlxf/j4WFhQUAoEmTJmoHNISQarL3l22/sY0NNnwryMEZtYP92RQCvLzPzltj5cYGMU7NgP+N5Jc3dwTs/JRHU4X/AbQdXXm9dg5jX79vpbwIqC7E32ADGgC49B3QYZzs3JnlbHBFCNGrepEoTAjRo/blX97uHQCHAPWuCV3MvibcArYNBE58qRzQAMCbv6lOUk4Mq/z+4jL+fm3MZpzyULYtEPJnYc6M1f37E0KqVC+DGh8fHzAMg7Zt2+q7KoQ0fCILtiVk0hn1r3Frx75mRFdezrsb0G6MbN+6vFX2xmbg6OdA8gO2FUS6aGbkMeDYF8pBxPMrlb8PwwB5qZWXuf8HsMaffU9V5Fc4z4oDchNl+/HXKr83IaRW1MughhBSx5nZsZP9KfLuIVuBfNBK9tXOD+g2nX1941dZ2RubgY3dgAurgeXO7FDxPW8D1zexyzfIe6YQcCXcBnYMA+7tY/fPrwbW+gOPj1Zc5+PzgPxUYHNP1ed5Qc1zIDeZf16iwbw/hBCdqBc5NYSQesi9A5DxjN0WCIFeX7A/AoFyDsyAZewPABiasCOvFF38VvmYkTlQmg88OQ4U57GtSgBwYgEQdwWIvQh4dwXOla8ifngmEKRi5uSyYtlyD5Iy5fMAkJMgt53Ib6kBgCcnVN+bEFJrqKWGEKIbLi1l29NuAb3nqh7erWhBMvD2HvXeo90YwNCUDUQeHACenGSPJ96VlbmxRbZtXcHAguMKc2Dd369cJkduvSswyt1Ue99hh7//9qp6dSeEaB0FNYQQ3fDsItu289Ps2sDBsu1x/wIDvlZdzrkl4NOd3T40Ddg9Cji5EBDLlkfBlfWybVWtMIWZwK1f+cfOLlcuJ7+IJyDLF2o+gn88+pzquhJCdI6CGkKIbngFA2/+j51IT50WGnnSLqrF2YBvT6DbJ8Bbu4CAAfxygUPYwEbenZ0AI1F937SngETh3Gq53B9bH/Y1M4ZtdXl2lt0vygZK8thtxfcLmc0uNSFPXMq+PjwEXKeh3oTUFgpqCCG603w4O0OwNjQbBozZz+blSFk4Au7t+eWKyvN1HJoq36M0n03yLcwCwvawQYv8wp6K3V7/G8G+pj0tfz8XwDGQX8bSlZ0NeVGWbBmJhDts4vAfY4FjnwMvwzX4oISQ6qJEYUJI/dLzc3bSvoD+7H7TwarL+fYEpt0Eos8D5g7A3x8ByeHsnDaHpgEFCku0zI4ELJyV7yO/TIRjIGDjzT9vZs++CgSA0BAQi4Hnl4BwubyciL+1F9wRQipELTWEkPrFwBDoPEnWVWRoDMx+Anx4jl/OrbwFx68X4NwCcC5flyr6nHJAE7oEsHRhA5PKgg/HINn7SskvFdG7vBXp8o/AzV9kx1Mjle9VlCObb6cgA9gzGnh0uOL3JoRUiVpqCCH1n6Uz+9Okr2zOGmkCsZRTM/b1hooclx4zZNuTLwGZz4EfWiuXMzYHbOVaavxD+eedywOioiz+8cgjyvda6cm+WnsB2XGycrWx5ENdUFIA3N4OBA3jP1NCaoCCGkJIwzH2AHBrO9uNpNii4qCQC+PUAhj7N2Bio3wfW282v2bvO/zjzYfzu6icW/DPN+lbcd0KMwHT8gUwSwpkx6UBjZREorxQaEN0aBoQ8RdwdxfwcRUzQhOipkbwfw4hpFHpOEH1JHg+Pfj7IzeyXU5GJqrv03QQ0PkjYPBqoMVIoO9CdvkHKzd2tXIA6DGTf42BIeDXW7b/2i+y4ezxN4HsBODfGcCjQxXXX3HouLaIS4E7/wPy09jJBpPus8tH6EvEX+xrSgXLUhBSDQKG0ed/1bVL3aXLCSENVPQ5dnI8l1ZsN1NtODAFuLebHbV193f+zMQVqaoLKuYCsPMVdnt6GGDnW3l5hgGW2Mj2A4eyXV1vbAdubWNnXlbnfbUl7SnwU0fZ/mfPAGOLigNM0uip+/1NLTWEkMbDrzf7xV1bAQ0gm8X4/Cr1AhqAP2GgInGpLKABgB/bVn0/xYVFpTk+f06QBTQAO9Rd1+78jx/QAMCaJrIcI0JqgIIaQgjRJVXz5UhZucu2P4+RbZ9cwObWPP0PyFcYqRW2q+r33NidHYoeV756eMJt9eq6ypu9rjiv8nJ3d7F1FFewTlZFxKVsLo3KcyVsgjap3D9TgQ3B7Og5bbn0HbDMCSjJ19499YSCGkII0aUWI1Qf9wkBZj0EPr0HfJXJrmw+aJXs/LHPgV2vA2v8gB/bAbtGsUHEv58q3yvpnmw77SmQHMFubxvIBil3f9eszpfWVXyOYYB/PmZbkw6rqEtl10mHsFfkx3bq368xyk9nf5epj4HbO7Rzz6Js4L/F7NIiRz/Tzj31iIIaQgjRJQMjYPhPsv1xh9kusPHlc9LY+shGO3WZDFiXd8PIz3OTEQ08PQkss5cd6z5DNtoq9jK7cvhia2DbIOU6xJzXrM4Xv604iTg/TbYdfaHqe8VeYuu1xEbWcgSwc/7YePHLys/uTJSlPpZtV7QUiKai5f7bCNsFPD6qnfvqCQU1hBCia+3eBXp/CfRfBviGVF62/Tj17tl3AWDpxm6fnA+sK5+HpyCt8mtU8ejEvsovPJr7UnVZ+SHo1u6qy8jbMVS2Le16avkGMPU64N+fX9bMoer7NWbFubLt0kIt3VOhG0txGoN6hoIaQgjRNYGAnW24+/Sqy3p14e/LDxGX8uzCtgB5lCfcqvpXu4uKyQM7fcAuAmruCMyQW4/qje3s2lXT78qOpT5SXb8suaCmqsTnilp7pJ9RMcgqSJMtBkqUFWbKthUneKwu+UCJex8t3VsPKKghhJC6xFthJuT3/lEeah1f3o3T7BVU6KML7HBveaa2wDt7gM+i2K4f6UroNp6yldSDhrGvqpZ2AID0Z7LtnER24c6KyH8Jy5O2RpnZsflE8+VahR79W/H9GosXt/ndQlKFGXLbFTxbTalKDtbWvfWAghpCCKlLhEJgwjF2+0O5L7aWb8i2O0xgX83kcmwANlh47xAwP5kNUux8gQUpwNB1/MChMtLlJE4vBY59webDRB6TnU+Ua82RlAF5KexIrWdnlL8Ms1+wr+aOwMRT7Ha36ex6XfKf18hUtl9WpF49GypxGbC1L/DbcHZUkjz556ut1hRpUBM8Rdadqa1WID2gZRIIIaSu8e6m3Drz+lb2C78gHRiyhj0mEACzHsnyaQL6swt4yjMUAZ0mqv/edk3Y19IC4PomdnvP2+yQ86w44LHCopvJD4B1QbL9L2Jly0Fklg9Tt/YEPDtXPrlfp0lscvTLCPXr2hBJZ1oG2AVO5Wet5gU1cq02NSENaozNARNrIDeRHRFVT1FLDSGE1AcCAfD2LuD942w+jZSVG9s6M3i1rOuoJlq+pvp43FVgh9z9Te3Y112v88ut8mFbG56eAv54jz3mGIQqOZWXubZBo+pqJPIY2/JU0bw90eeBxDDdvb865AMXc8eKz2m7+8nYDDC1Kb93lnburQcU1BBCSH3n1wsI/kiWF1MThiLVx/eOBkrkkkq9u1V8j2X2wC657jIzu6rfV37kk6rkVamibDZYqs48LXveZl9/UbHwaPYLtstnSy+gVI9dYPmpsm3FbiD555JfySg3TZRKgxoLtqUGoJYaQgghDciUq+zr23vYbiFF448qByqzK0gsBtg8mqo0HSjbDt+vfD43GdjSG1jpBTz8h52E8MWtqu9bEcXAJfqcbDv+GvQm7YlsW7HFRH6m5+Ic7SxIKm2pMTKTrVhfj3NqKKghhBDC59yczX8JGgL0X8I/99ovgE93fpKykTm74vmiLOV7jd4PWDpX/Z7yycI3tvLPlRQA3zblJykDwB9yc/pIJGzX0mJr1V/2BQo5KIrD0eOuyrb1uVxDTqJsW7GLSb6lhpEAJVUsZ6EOaVAjopYaQgghDZ2xOfDxdXa76SCg9ZvsducPZWVm3GdfBQJg7AHZ8Xf2AU0HqP9e0lYhc7mA6fQy4BtX1eVzXsgCmKj/ZMefnVYuqzhEXToyizsv10KSl6xefRU9/Q+48UvV5SrCMEB6lGxfscWkRKFbThvrP0kDI2OLBpFTQ6OfCCGEVM4pSHnkkpWb6tFMTfoC026zuTk2Gq683WIEOwIq5gI7/w3DABfXKpez9ZWNrLq0DgiZDaQ8kJ1Pugf4h/KvUWyZUQxq5IeSVxXUvIwANnUHXvmRHel1biXw5m+ypOmEO8DIjZXfQxHDsPk88oFMWRHbTWZkwu4r5hoV5wBQY1bnyiiOfgKopYYQQgjhOPhrHtAAgGsb2fYaf+D5Zf759uPYQOrTMNmxiPKWobtyq5ermjgwO15hXy6oOfYF8PK+bL+iJSKkNpVPkPjvdOCPsWxA9VMH2fl7uyu/XpWMaP7CpILyr2f5IEeaUyMsH/2mTkvN0/+Aqxsqzr+hnBpCCCFEB0SWsu3CDHY0ktT4o8DwH2X7H5xhX5PD2Xya9Keyc6mPlb/E42+wr8LyDgppkFOULZuPRyovBXh2ls3PUVzgUd01lxRzeKoiH9B0nSZrNZF2BZWVsCtpA2wrGaC8bpMicSnbenTiS+DRIdVlSmj0EyGEEKIbASpycLy6scnJ8lxaybb/GMs/l3SPXRV8sTUQc7F8VuTy4KR1+bBuaUvN5R+U3y81EvjfCHZ77zv89a6qasXh7vG46jLypN1jLV8HBi6XTWAoTRaWTwq29mBfqwo+5JecSLijfJ5h5HJqzBtETg0FNYQQQuqOt3crL//Qc45yOUNjwKJ8VJX8LMcGxvxyOxUmJPTtyb5Kg5onJ9lX+wA2FwgAihWChcjjsu20p1CLpkFNdnlQY1WeI6PYFSRtlTEykwU8VQU18nXITVI+X1YsWwyVcmoIIYQQLTMwAuZE8Y95dFJddug6/v7gNYC4pPL7e5bfK/sFm4QrTTAedwiwcFR9jfzcMdJuq8AhwOu/sknK8jp/xL5K83pyEtk5cA5M5g/XVpRTHmRJW2EUW02kScKadBPJr9mVFa98Xn4xS2NzfiCljTlw9ICCGkIIIXWLUAi0fVe2b2KlupxXV9m2gQgI/lB1OQD48Bzw2TNZS0hZIRB7kW2pMLYELF0BkcL7hC5mX2/KDdOWBjXWHkCrN4B+XwEzwtljo/fLusVSHrLdXuuaAb+9CtzbA/ynMOePvCpbasq7iUSWsqCmqpwa+eRnxURpQNb1ZGgKCA1k9xWX1NuFRSmoIYQQUvcM+w745E7li2Ca2wNTbwB9FgBflA/xnl3eqmJkxk4G+HkM8FUG4NYOMHdgh5o7li8AKl3KwcqVnWNHcZmJwKGy7ejyFdOl3VbWcqO7bLzYejYdALi0ZI/FXFCub0IlMyBLc2qqaqmRD2oqaqnJjGUDKt79E9n8HPlh4fLDuaX3NjSVla+HKKghhBBS9xgaA/ZNqi7nGAj0+kz2xWzpzAYY85PYIMXMjm2FULxGnmdn5fua2ACOTWX70pFY2QrdRIqcmsuGYytKj2Ln35F3fz87HD0vhX9fpURhuaBG2qJUUVDzQxv+voExwIjZxUZXN2FHRQHKQY1AIHvm6c9U37uOo6CGEEJI4zJ4NX+/zWjZ9vsn2P1PykcLGZrIzjEM8OImu21dwTw8hiLAxlu236Qvf5blkwtl26eXAX9/APzzMQCGbSWRJkkrdT9p0FIjncdGSr4+4mLg2OfstvxswlIZ0ezr7lGq713HUVBDCCGkcbF0BhamAw6BbNeRe3vZOa8u7GzA0qUaZkTIzp1cAEjK2O3KJheUHwLedDAb2Ehd28DOmRNzQXm2ZDM7WReYUveTXADCBTVyOTXSxN6yYv49X/9VuWXq1jb2VbGlBgBavyXbrsmSD3pCQQ0hhJDGx8AQ+Og8u66V/GKaiuRHRF2TW/rA3Knia175Xrbd/j32ddRO2bFf+wM7X1G+Tn4pB3VbavJS2fyZJTbAy3Ag6T4gKQVM7djArdUb/EkNpYpz5YIaM9nxod/Kto/OYe+9uVe9GQ1FQQ0hhJDGyciU/4VekRHlMw4z5fkwgUPZEVoVaf8em9ezOFu2blOz4bJgpCKubWXbXEtNeU4NF9RYyEaDFWUDMedl12zqAWSU58I4t2ADNwCw91d+r8znqrufhAbKw9STwmRrbQHs6uiLrYH/Flf+efSAghpCCCGkMgH9+fvSgEMTQiEwNw4QKQQ27d4FxvwJeHcH3tgmOy5tqZF2P+WVz2Rs5iC3hEImcH8f/36Zseyrna/sWJeP2VwdxyDAuZWsnKruJ4CfYyT15AQbyCy2Bh4eZI9d+g4oKajwI+sDrdJNCCGEVMbcgb8v7VKqjlkP2W4mhgGurGdX+hYaqAicykc/5aewZaWjrmy8APPyLjFxMfD0JP866bIPtnJBjbEZ8Hl5AvAf49j1srKeVxzU2Pkp1/v4XNWf5/kVICBU9Tk9oJYaQgghpCryI6Y8g6t/H5EFm7jrFASM2KA83FzKzE62nR0PxF9nty1d2CDEWEWeDACUlrecODVTfd62fCRU5nOgtIKgRigEXitPEnbvWPnn2fU6G3TVkZwbCmoIIYSQqgR/xE7m91WG8iR9uiAfaESdlm3b+rCvli6VX19R4CUd3n1jM3+FbkWt32RzgvovrbquS2zYnxQN17vSAQpqCCGEEHUIBBW3rOhCk37s6+EZ7KuNtyyYkXZHAUCrUWxejtQHZ/gtPfKkE+8BsnwdxZYaeV5d+PvSBOhPVKz6fX5VxfepJfUqqDly5AiCg4NhamoKW1tbjBgxQt9VIoQQQnRDMSFZusI4wK5dJeXWjs3JkQYcHh0qvme7MbLt5PI5eKRJyaoIDYB5CcC7fwNfZcqO2zcBeink2Tz4W3menFpWb4Kav/76C2PHjsWECRNw7949XL58GaNHq8jQJoQQQhoC6YrfUj0/k20PkZu4r8N49e8pspStfSVdfbyq0VwiC8C/n/Iw9j7z2CBqttwq5uuaAWVVrJSuQ/Vi9FNZWRk+/fRTrFmzBhMnTuSON2/eXI+1IoQQQnRIcU0qW7nlDjpPAhwC2DloKus+UkU6sorbr6CrSl2WzuyQ9OeXgYJ04PomoPv0mt2zmupFS82dO3eQkJAAoVCIdu3awdXVFYMHD0ZERESl1xUXFyMnJ4f3QwghhNQLAgHb5TPpDDs7sCK/3hUvrFmZuKv8fdc2qstposME2XZNhrzXUL0IaqKj2fH1ixcvxoIFC3D48GHY2tqid+/eyMjIqPC6FStWwNramvvx9KxkrQ5CCCGkrhEKAfcOstmBteGt//H3RSpGP2mq5evAm78Bn0VXb3JCLdFrUDN37lwIBIJKfx4/fgyJRAIAmD9/Pl5//XV06NAB27dvh0AgwP79+yu8/7x585Cdnc39xMfH19ZHI4QQQuqmZq8Aw9ezsxuPP6qdewqFQPNXZQuB6olec2pmz56N8ePHV1rGz88PSUlJAPg5NCKRCH5+foiLi6voUohEIohEIq3UlRBCCGkw2r+n124iXdFrUOPo6AhHR8cqy3Xo0AEikQiRkZHo0aMHAKC0tBSxsbHw9vau4mpCCCGENAb1YvSTlZUVJk+ejEWLFsHT0xPe3t5Ys2YNAGDUqFF6rh0hhBBC6oJ6EdQAwJo1a2BoaIixY8eisLAQwcHBOHPmDGxtbau+mBBCCCENnoBh6sgqVLUgJycH1tbWyM7OhpWVlb6rQwghhBA1qPv9XS+GdBNCCCGEVIWCGkIIIYQ0CBTUEEIIIaRBoKCGEEIIIQ0CBTWEEEIIaRAoqCGEEEJIg0BBDSGEEEIaBApqCCGEENIgUFBDCCGEkAaBghpCCCGENAj1Zu0nbZCuCJGTk6PnmhBCCCFEXdLv7apWdmpUQU1ubi4AwNPTU881IYQQQoimcnNzYW1tXeH5RrWgpUQiQWJiIiwtLSEQCLR235ycHHh6eiI+Pp4WylSBnk/l6PlUjp5P5ej5VI6eT9XqwzNiGAa5ublwc3ODUFhx5kyjaqkRCoXw8PDQ2f2trKzq7H8QdQE9n8rR86kcPZ/K0fOpHD2fqtX1Z1RZC40UJQoTQgghpEGgoIYQQgghDQIFNVogEomwaNEiiEQifVelTqLnUzl6PpWj51M5ej6Vo+dTtYb0jBpVojAhhBBCGi5qqSGEEEJIg0BBDSGEEEIaBApqCCGEENIgUFBDCCGEkAaBghot2LBhA3x8fGBiYoLg4GDcuHFD31XSugsXLuCVV16Bm5sbBAIBDh48yDvPMAy++uoruLq6wtTUFKGhoXj69CmvTEZGBsaMGQMrKyvY2Nhg4sSJyMvL45W5f/8+QkJCYGJiAk9PT6xevVrXH00rVqxYgU6dOsHS0hJOTk4YMWIEIiMjeWWKioowdepU2Nvbw8LCAq+//jqSk5N5ZeLi4jB06FCYmZnByckJn332GcrKynhlzp07h/bt20MkEsHf3x87duzQ9cersY0bN6J169bc5F5du3bFsWPHuPON+dmosnLlSggEAsyYMYM71pif0eLFiyEQCHg/QUFB3PnG/GykEhIS8O6778Le3h6mpqZo1aoVbt26xZ1vNH+jGVIje/fuZYyNjZlt27YxDx48YCZNmsTY2NgwycnJ+q6aVh09epSZP38+8/fffzMAmAMHDvDOr1y5krG2tmYOHjzI3Lt3jxk+fDjj6+vLFBYWcmUGDRrEtGnThrl27Rpz8eJFxt/fn3nnnXe489nZ2YyzszMzZswYJiIigtmzZw9jamrKbN68ubY+ZrUNHDiQ2b59OxMREcGEhYUxQ4YMYby8vJi8vDyuzOTJkxlPT0/m9OnTzK1bt5guXbow3bp1486XlZUxLVu2ZEJDQ5m7d+8yR48eZRwcHJh58+ZxZaKjoxkzMzNm1qxZzMOHD5n169czBgYGzPHjx2v182rq0KFDzJEjR5gnT54wkZGRzJdffskYGRkxERERDMM07mej6MaNG4yPjw/TunVr5tNPP+WON+ZntGjRIqZFixZMUlIS95Oamsqdb8zPhmEYJiMjg/H29mbGjx/PXL9+nYmOjmZOnDjBREVFcWUay99oCmpqqHPnzszUqVO5fbFYzLi5uTErVqzQY610SzGokUgkjIuLC7NmzRruWFZWFiMSiZg9e/YwDMMwDx8+ZAAwN2/e5MocO3aMEQgETEJCAsMwDPPzzz8ztra2THFxMVfmiy++YAIDA3X8ibQvJSWFAcCcP3+eYRj2eRgZGTH79+/nyjx69IgBwFy9epVhGDZwFAqFzMuXL7kyGzduZKysrLhn8vnnnzMtWrTgvddbb73FDBw4UNcfSetsbW2ZrVu30rORk5ubywQEBDCnTp1ievXqxQU1jf0ZLVq0iGnTpo3Kc4392TAM+3eyR48eFZ5vTH+jqfupBkpKSnD79m2EhoZyx4RCIUJDQ3H16lU91qx2xcTE4OXLl7znYG1tjeDgYO45XL16FTY2NujYsSNXJjQ0FEKhENevX+fK9OzZE8bGxlyZgQMHIjIyEpmZmbX0abQjOzsbAGBnZwcAuH37NkpLS3nPKCgoCF5eXrxn1KpVKzg7O3NlBg4ciJycHDx48IArI38PaZn69N+bWCzG3r17kZ+fj65du9KzkTN16lQMHTpU6XPQMwKePn0KNzc3+Pn5YcyYMYiLiwNAzwYADh06hI4dO2LUqFFwcnJCu3bt8Msvv3DnG9PfaApqaiAtLQ1isZj3PwoAODs74+XLl3qqVe2TftbKnsPLly/h5OTEO29oaAg7OzteGVX3kH+P+kAikWDGjBno3r07WrZsCYCtv7GxMWxsbHhlFZ9RVZ+/ojI5OTkoLCzUxcfRmvDwcFhYWEAkEmHy5Mk4cOAAmjdvTs+m3N69e3Hnzh2sWLFC6Vxjf0bBwcHYsWMHjh8/jo0bNyImJgYhISHIzc1t9M8GAKKjo7Fx40YEBATgxIkTmDJlCqZPn46dO3cCaFx/oxvVKt2E1IapU6ciIiICly5d0ndV6pTAwECEhYUhOzsbf/75J8aNG4fz58/ru1p1Qnx8PD799FOcOnUKJiYm+q5OnTN48GBuu3Xr1ggODoa3tzf++OMPmJqa6rFmdYNEIkHHjh3xzTffAADatWuHiIgIbNq0CePGjdNz7WoXtdTUgIODAwwMDJSy7JOTk+Hi4qKnWtU+6Wet7Dm4uLggJSWFd76srAwZGRm8MqruIf8edd20adNw+PBhnD17Fh4eHtxxFxcXlJSUICsri1de8RlV9fkrKmNlZVXn/7gbGxvD398fHTp0wIoVK9CmTRv88MMP9GzAdqGkpKSgffv2MDQ0hKGhIc6fP48ff/wRhoaGcHZ2bvTPSJ6NjQ2aNm2KqKgo+u8HgKurK5o3b8471qxZM66LrjH9jaagpgaMjY3RoUMHnD59mjsmkUhw+vRpdO3aVY81q12+vr5wcXHhPYecnBxcv36dew5du3ZFVlYWbt++zZU5c+YMJBIJgoODuTIXLlxAaWkpV+bUqVMIDAyEra1tLX2a6mEYBtOmTcOBAwdw5swZ+Pr68s536NABRkZGvGcUGRmJuLg43jMKDw/n/WE5deoUrKysuD9YXbt25d1DWqY+/vcmkUhQXFxMzwZAv379EB4ejrCwMO6nY8eOGDNmDLfd2J+RvLy8PDx79gyurq703w+A7t27K00h8eTJE3h7ewNoZH+j9Z2pXN/t3buXEYlEzI4dO5iHDx8yH374IWNjY8PLsm8IcnNzmbt37zJ3795lADDr1q1j7t69yzx//pxhGHa4oI2NDfPPP/8w9+/fZ1599VWVwwXbtWvHXL9+nbl06RITEBDAGy6YlZXFODs7M2PHjmUiIiKYvXv3MmZmZnVquGBFpkyZwlhbWzPnzp3jDTstKCjgykyePJnx8vJizpw5w9y6dYvp2rUr07VrV+68dNjpgAEDmLCwMOb48eOMo6OjymGnn332GfPo0SNmw4YN9WLY6dy5c5nz588zMTExzP3795m5c+cyAoGAOXnyJMMwjfvZVER+9BPDNO5nNHv2bObcuXNMTEwMc/nyZSY0NJRxcHBgUlJSGIZp3M+GYdhpAAwNDZnly5czT58+ZXbt2sWYmZkxv//+O1emsfyNpqBGC9avX894eXkxxsbGTOfOnZlr167pu0pad/bsWQaA0s+4ceMYhmGHDC5cuJBxdnZmRCIR069fPyYyMpJ3j/T0dOadd95hLCwsGCsrK2bChAlMbm4ur8y9e/eYHj16MCKRiHF3d2dWrlxZWx+xRlQ9GwDM9u3buTKFhYXMxx9/zNja2jJmZmbMyJEjmaSkJN59YmNjmcGDBzOmpqaMg4MDM3v2bKa0tJRX5uzZs0zbtm0ZY2Njxs/Pj/ceddX777/PeHt7M8bGxoyjoyPTr18/LqBhmMb9bCqiGNQ05mf01ltvMa6uroyxsTHj7u7OvPXWW7w5WBrzs5H6999/mZYtWzIikYgJCgpitmzZwjvfWP5GCxiGYfTTRkQIIYQQoj2UU0MIIYSQBoGCGkIIIYQ0CBTUEEIIIaRBoKCGEEIIIQ0CBTWEEEIIaRAoqCGEEEJIg0BBDSGEEEIaBApqCCGNikAgwMGDB/VdDUKIDlBQQwipNePHj4dAIFD6GTRokL6rRghpAAz1XQFCSOMyaNAgbN++nXdMJBLpqTaEkIaEWmoIIbVKJBLBxcWF9yNd4VcgEGDjxo0YPHgwTE1N4efnhz///JN3fXh4OPr27QtTU1PY29vjww8/RF5eHq/Mtm3b0KJFC4hEIri6umLatGm882lpaRg5ciTMzMwQEBCAQ4cOcecyMzMxZswYODo6wtTUFAEBAUpBGCGkbqKghhBSpyxcuBCvv/467t27hzFjxuDtt9/Go0ePAAD5+fkYOHAgbG1tcfPmTezfvx///fcfL2jZuHEjpk6dig8//BDh4eE4dOgQ/P39ee+xZMkSvPnmm7h//z6GDBmCMWPGICMjg3v/hw8f4tixY3j06BE2btwIBweH2nsAhJDq0/eKmoSQxmPcuHGMgYEBY25uzvtZvnw5wzDsaueTJ0/mXRMcHMxMmTKFYRiG2bJlC2Nra8vk5eVx548cOcIIhULm5cuXDMMwjJubGzN//vwK6wCAWbBgAbefl5fHAGCOHTvGMAzDvPLKK8yECRO084EJIbWKcmoIIbWqT58+2LhxI++YnZ0dt921a1feua5duyIsLAwA8OjRI7Rp0wbm5ubc+e7du0MikSAyMhICgQCJiYno169fpXVo3bo1t21ubg4rKyukpKQAAKZMmYLXX38dd+7cwYABAzBixAh069atWp+VEFK7KKghhNQqc3Nzpe4gbTE1NVWrnJGREW9fIBBAIpEAAAYPHoznz5/j6NGjOHXqFPr164epU6di7dq1Wq8vIUS7KKeGEFKnXLt2TWm/WbNmAIBmzZrh3r17yM/P585fvnwZQqEQgYGBsLS0hI+PD06fPl2jOjg6OmLcuHH4/fff8f3332PLli01uh8hpHZQSw0hpFYVFxfj5cuXvGOGhoZcMu7+/fvRsWNH9OjRA7t27cKNGzfw66+/AgDGjBmDRYsWYdy4cVi8eDFSU1PxySefYOzYsXB2dgYALF68GJMnT/5/+3aMokgUhWH070SwYhFqBYKGYugCzATNzQUxMTFxCboMKzMx0AW4B0P3YKKREzQ0TNYw09M9xTnhC4pb2cerW2m32xmNRrnf77lcLlksFp+ab7PZpN/vp9fr5fl85ng8fkQV8LOJGuCfOp1OKcvyt7NOp5Pr9Zrk/c+kqqoyn89TlmX2+3263W6SpCiKnM/nLJfLDAaDFEWRyWSS7Xb78azZbJbH45HdbpfVapVWq5XpdPrp+RqNRtbrdW63W5rNZobDYaqq+gtvDny1t9fr9fruIQCS992Ww+GQ8Xj83aMA/yE7NQBALYgaAKAW7NQAP4av4cCfcFMDANSCqAEAakHUAAC1IGoAgFoQNQBALYgaAKAWRA0AUAuiBgCoBVEDANTCL9BTmLkoRTIeAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "from matplotlib import pyplot as plt\n", + "import numpy as np\n", + "\n", + "def moving_average(a, n=10):\n", + " ret = np.cumsum(a, dtype=float)\n", + " ret[n:] = ret[n:] - ret[:-n]\n", + " return ret[n - 1:] / n\n", + "\n", + "plt.plot(moving_average(acc_list, 20), label=\"R_acc\")\n", + "plt.plot(moving_average(rej_list, 20), label=\"R_rej\")\n", + "plt.legend()\n", + "plt.title(f\"Reward model learning\")\n", + "plt.xlabel(\"Epochs\")\n", + "plt.ylabel(\"Reward (moving average)\")" + ] + }, + { + "cell_type": "markdown", + "id": "de9a4613", + "metadata": {}, + "source": [ + "Let's take a look at the kind of rewards the model generates for a fixed sequence and different values at the first position of the output." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "b93643d8", + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0 || -2.0374112129211426\n", + "1 || 4.983941078186035\n", + "2 || -0.8424915075302124\n", + "3 || -6.84158182144165\n", + "4 || -6.4326276779174805\n", + "5 || -6.820979118347168\n", + "6 || -6.832927703857422\n", + "7 || -6.83734130859375\n", + "8 || -6.613164901733398\n", + "9 || -4.2613844871521\n", + "10 || -6.424010753631592\n" + ] + } + ], + "source": [ + "i = 0\n", + "for i in range(11):\n", + " print(i, \"||\", reward_model(torch.Tensor([[ 6, 1, 7, 0, 10, i, 5, 5, 6, 11]]).long())[0][0, -1, 0].item())" + ] + }, + { + "cell_type": "markdown", + "id": "7c723d96", + "metadata": {}, + "source": [ + "Given the input sequence [6, 1, 7, 0], the first output token should be last_token+1 = 0 + 1 = 1. All other generations are \"wrong\", so the reward model gives positive reward for token 1 and negative rewards for others." + ] + }, + { + "cell_type": "markdown", + "id": "ff7bfac9", + "metadata": {}, + "source": [ + "### RL fine-tuning" + ] + }, + { + "cell_type": "markdown", + "id": "667455cf", + "metadata": {}, + "source": [ + "Finally, we have arrived at our final training that will use our reward model to update the supervised learning model using reinforecement learning. " + ] + }, + { + "cell_type": "markdown", + "id": "ef6e4095", + "metadata": {}, + "source": [ + "![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/rlhf/rlhf.png)\n", + "\n", + "Source: [HuggingFace - RLHF blog](https://huggingface.co/blog/rlhf)" + ] + }, + { + "cell_type": "markdown", + "id": "cb10079e", + "metadata": {}, + "source": [ + "Following is a function for calculating logprob given a model and outputs. This'll help us calculate loss for RL and KL Divergence. More details on these a few code blocks below --" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "e3c8d54d", + "metadata": {}, + "outputs": [], + "source": [ + "from torch.distributions import Categorical\n", + "\n", + "def get_logprob(agent, outputs):\n", + " '''\n", + " Get the logprobs for outputs acc. to agent's policy.\n", + " \n", + " Args:\n", + " agent: Actor network (or reference)\n", + " outputs: output ids\n", + " Shape = (sequence, tokens)\n", + " \n", + " returns \n", + " logprob of outputs acc to agent's policy\n", + " Shape = (sequence, tokens)\n", + " '''\n", + " logits = agent(outputs[:, :-1])[0][:, -INPUT_SIZE:, :]\n", + " logprob = Categorical(logits=logits).log_prob(outputs[:, -INPUT_SIZE:])\n", + " return logprob" + ] + }, + { + "cell_type": "markdown", + "id": "207bfe14", + "metadata": {}, + "source": [ + "Hyperparameters --" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "fa705304", + "metadata": {}, + "outputs": [], + "source": [ + "# Hyperparams\n", + "epochs = 100\n", + "actor_lr = 1e-5\n", + "critic_lr = 1e-4\n", + "train_actor_iter = 4 # Train the networks this many times per epoch\n", + "train_critic_iter = 4 \n", + "clip_ratio = 0.2 # PPO Clip\n", + "gamma = 0.99 # Discount factor\n", + "kl_beta = 1 # KL coeff for reward\n", + "save = False\n", + "\n", + "# For plotting\n", + "rew_list = []\n", + "kl_list = []" + ] + }, + { + "cell_type": "markdown", + "id": "add83ee2", + "metadata": {}, + "source": [ + "Here we set up our models and optimizers. We typically need 3 models for RLHF training:" + ] + }, + { + "cell_type": "markdown", + "id": "60f32e29", + "metadata": {}, + "source": [ + "1. Actor: This is the LLM that we will fine-tune using reinforcement learning(RL). It is initialised as a copy of the pretrained model. \n", + "2. Reference: To prevent the actor's output distribution (or \"policy\" in RL terms) from diverging too much from the pretrained model's distribution, we need to apply some constraint on the distance/difference of the two distributions. For this, we keep this reference model which is a frozen copy of the pretrained model to calculate KL divergence during our RL training. \n", + "3. Critic: The critic network is also a copy of the base LLM but with the last layer replaced with a single output. This is used to estimate the value function, which is a component required to calculate the actor's loss.\n", + "\n", + "In our simple problem statement, the rewards are given at the end of the sequence. Therefore, we don't need to estimate the value function and hence, don't train a critic network. For more information, see this [answer](https://stats.stackexchange.com/questions/380123/reinforcement-learning-what-is-the-logic-behind-actor-critic-methods-why-use) on StackExchange." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1920d08f", + "metadata": {}, + "outputs": [], + "source": [ + "# Actor\n", + "actor = get_model(block_size=INPUT_SIZE*2, vocab_size=st_dataset.get_vocab_size()) \n", + "actor.load_state_dict(torch.load(\"models/minimal_RLHF_basic_supervised.pt\")) # Load ST model from disk\n", + "# Reference\n", + "reference = get_model(block_size=INPUT_SIZE*2, vocab_size=st_dataset.get_vocab_size()) \n", + "reference.load_state_dict(torch.load(\"models/minimal_RLHF_basic_supervised.pt\")) # Clone of actor\n", + "\n", + "# Optimizers\n", + "actor_opt = torch.optim.AdamW(actor.parameters(), lr=actor_lr)\n", + "\n", + "# Set models to train/eval\n", + "reference.eval()\n", + "reward_model.eval()\n", + "actor.train()" + ] + }, + { + "cell_type": "markdown", + "id": "11b4372a", + "metadata": {}, + "source": [ + "At last, we come to our main RL training. We use PPO, a famous RL algorithm for fine-tuning our model along with a KL divergence penalty. " + ] + }, + { + "cell_type": "markdown", + "id": "eaff1744", + "metadata": {}, + "source": [ + "**PPO:** \n", + "The main idea behind PPO is to induce stability in the training process by preventing large updates. \n", + "\n", + "Let's look at the PPO loss:\n", + "\n", + "$$L = \\text{min}\\biggl( \\frac{\\pi_{k+1} (a|s)}{\\pi_{k} (a|s)} R, \\text{ clip}\\Bigl(\\frac{\\pi_{k+1} (a|s)}{\\pi_{k} (a|s)}, 1-\\epsilon, 1+\\epsilon\\Bigr) R\\biggr)$$\n", + "\n", + "Where $\\pi_k$ represents the policy at $k$'th training step, R is reward and $\\epsilon$ is a hyperparameter for clipping the policy update. I have partly modified the loss to prevent too many new ideas at once for beginners. This version is sufficient for our current case. To learn more about the PPO loss, look at [SpinningUp](https://spinningup.openai.com/en/latest/algorithms/ppo.html) and [Eric's article](https://medium.com/analytics-vidhya/coding-ppo-from-scratch-with-pytorch-part-1-4-613dfc1b14c8).\n", + "\n", + "The PPO loss looks complicated but is fairly straightforward. To understand what the PPO loss does, consider the two cases: \n", + "1. R is positive \n", + "2. R is negative \n", + "\n", + "**Case 1**: R is positive. \n", + "Then the loss reduces to \n", + "$$L = \\text{min}\\biggl( \\frac{\\pi_{k+1} (a|s)}{\\pi_{k} (a|s)}, 1+\\epsilon \\biggr)R$$\n", + "So if the policy at the next training step is _increasing_ too far from the previous step, we clip it to $1+\\epsilon$. \n", + "\n", + "**Case 2**: R is negative.\n", + "Then the loss reduces to \n", + "$$L = \\text{max}\\biggl( \\frac{\\pi_{k+1} (a|s)}{\\pi_{k} (a|s)}, 1-\\epsilon \\biggr)|R|$$\n", + "So if the policy at the next training step is _decreasing_ too far from the previous step, we clip it to $1-\\epsilon$. " + ] + }, + { + "cell_type": "markdown", + "id": "9f77c733", + "metadata": {}, + "source": [ + "**KL Divergence:** \n", + "[KL divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence) is a measure of difference between two distributions. We use KL divergence penalty to ensure our actor's policy (probability distribution over next tokens) does not stray too far from the reference model's policy. For two distributions P and Q defined on the same sample space, $X$, the KL divergence is given by: \n", + "\n", + "$$D_{KL}(P||Q) = \\sum_{x \\in X} P(x) log\\biggl(\\frac{P(x)}{Q(x)}\\biggr)$$" + ] + }, + { + "cell_type": "markdown", + "id": "597c8ce5", + "metadata": {}, + "source": [ + "The final reward used for PPO is then a linear combination of the scalar output from the reward model $R_{RM}$ and the value of KL divergence $R_{KL}$ with a hyperparameter $\\beta$.\n", + "\n", + "$$R = R_{RM} - \\beta R_{KL}$$" + ] + }, + { + "cell_type": "markdown", + "id": "28f6b4b4", + "metadata": {}, + "source": [ + "Both of PPO and KL divergence are crucial components to the RLHF training due to the inherent fragile nature of RLHF. Mainly, the issue lies with the reward model and the fact that it _cannot_ completely capture human preferences. The data used to train the reward model is generated using the base LLM's policy. Therefore, if the actor diverges too far from the base policy and the reward model is asked to give feedback for samples that do not come from the training distribution, we cannot predict the behaviour of the reward model. In fact, this exact issue often leads to adversarial training (see [Deepak's article](https://medium.com/@prdeepak.babu/reward-hacking-in-large-language-models-llms-c57abbc0cde7) on Reward Hacking in LLMs). This issue is avoided by taking small steps in PPO and using a KL penalty to prevent moving too far from base policy." + ] + }, + { + "cell_type": "markdown", + "id": "4b9594c0", + "metadata": {}, + "source": [ + "Now we have the RL code --" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a8f6c590", + "metadata": {}, + "outputs": [], + "source": [ + "# Dataloader - we use the same as reward model for now since we only need the inputs and it's in the correct format for what we want in RLHF training\n", + "# Can't use the supervised training's dataloader directly since the input has some part of output concatenated in that\n", + "ft_dataloader = train_loader\n", + "\n", + "# Train\n", + "for ep in range(epochs):\n", + " for b_i, batch in enumerate(ft_dataloader): \n", + " # Get some inputs from supervised dataset (only inputs - we don't care about the ground truths anymore)\n", + " inp, _, __ = batch\n", + "\n", + " # Generate output sequence\n", + " out = actor.generate(inp, max_new_tokens=INPUT_SIZE, do_sample=True) # Not sampling since good and bad in our problem is fairly deterministic, otherwise prefer to sample.\n", + " start_logprob = get_logprob(actor, out).detach()\n", + " start_logprob = start_logprob.sum(-1)\n", + "\n", + " # Reward\n", + " rew_out = torch.concat([out, torch.Tensor([[11]]*out.shape[0])], dim=-1).long() # Add [CLS] = 11\n", + " rew = reward_model(rew_out)[0][:, -1, 0]\n", + " rew_list.append(rew.mean().item())\n", + " \n", + " # Actor train loop\n", + " for _iter_actor in range(train_actor_iter):\n", + " # Get logprobs\n", + " cur_logprob = get_logprob(actor, out)\n", + " ref_logprob = get_logprob(reference, out)\n", + " cur_logprob = cur_logprob.sum(dim=-1) # Summing because we don't have rewards for each timestep\n", + " ref_logprob = ref_logprob.sum(dim=-1)\n", + "\n", + " # KL and reward update\n", + " kl_div = (cur_logprob - ref_logprob).detach()\n", + " rew = rew - kl_beta * kl_div\n", + "\n", + " # PPO loss\n", + " ratio = torch.exp(cur_logprob - start_logprob)\n", + " clip_rat = torch.clamp(ratio, 1-clip_ratio, 1+clip_ratio)\n", + " actor_loss = -(torch.min(ratio * rew, clip_rat * rew)).mean()\n", + "\n", + " # Update actor\n", + " actor_opt.zero_grad()\n", + " actor_loss.backward(retain_graph=True)\n", + " actor_opt.step()\n", + "\n", + " # Save kl div for plotting\n", + " kl_list.append(kl_div.mean().item())\n", + "\n", + " # Eval\n", + " if ep % 1 == 0 and b_i % 50 == 0:\n", + " print(f\"Epoch={ep} -- batch={b_i} || \" + \\\n", + " f\"Reward={round(rew_list[-1], 2)} || \" + \\\n", + " f\"KLD={round(kl_list[-1], 2)} || \" + \\\n", + " f\"actor loss={round(actor_loss.item(), 2)}\")\n", + " print(out[0])\n", + " print(\"#\"*100)" + ] + }, + { + "cell_type": "markdown", + "id": "2441630c", + "metadata": {}, + "source": [ + "Output is ommitted for brevity. Here is an image of some of the outputs: \n", + "![](../../../images/session_4/part_2_finetuning_lms_to_human_preferences/rl_outputs.png)" + ] + }, + { + "cell_type": "markdown", + "id": "a9091014", + "metadata": {}, + "source": [ + "Save model --" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "9dcc1b12", + "metadata": {}, + "outputs": [], + "source": [ + "import datetime, os, json\n", + "\n", + "save = True\n", + "\n", + "# RUN TO SAVE MODEL\n", + "folder = f\"models/08_min_rlhf_basic_{datetime.datetime.now().__str__()}\"\n", + "os.makedirs(folder, exist_ok=True)\n", + "\n", + "torch.save(reward_model, f\"{folder}/reward_nodel.pt\")\n", + "torch.save(reference, f\"{folder}/reference.pt\")\n", + "torch.save(actor, f\"{folder}/actor.pt\")\n", + "\n", + "with open(f\"{folder}/config.json\", 'w') as f:\n", + " json.dump({\n", + " \"epochs\": epochs,\n", + " \"actor_lr\": actor_lr,\n", + " \"critic_lr\": critic_lr,\n", + " \"train_actor_iter\": train_actor_iter,\n", + " \"train_critic_iter\": train_critic_iter,\n", + " \"clip_ratio\": clip_ratio,\n", + " \"gamma\": gamma,\n", + " \"kl_beta\": kl_beta,\n", + " }, f)" + ] + }, + { + "cell_type": "markdown", + "id": "d7706531", + "metadata": {}, + "source": [ + "Plot rewards --" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "d187c707", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAj4AAAHHCAYAAAC/R1LgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB4x0lEQVR4nO3dd3hTZRsG8Dvde9EJlLbQQhlllSG7QKUMQVDAjyWgIiDIRkCWIktEBBUZKkNlyRCRKaNM2aVsyuygUEop3Ts53x+HpA1NR9qkaZv7d125SM583qTkPHnPOySCIAggIiIi0gMGug6AiIiIqKww8SEiIiK9wcSHiIiI9AYTHyIiItIbTHyIiIhIbzDxISIiIr3BxIeIiIj0BhMfIiIi0htMfIiIiEhvMPEh0nMSiQRffPGFxo43bNgweHp6aux4RZ0rICAg33KJRIINGzaUSQxF2bBhAyQSCS5duqTrUIgITHyICiS/YMkfRkZGqFatGoYNG4bo6Ghdh1cpLVy4ELt37y6z8w0bNgxWVlb5ll+7dg2Ojo7w9PREeHg4ACAgIAANGjQos9gKs3//fo0mq9rw9OlTTJ8+HR07doS1tTUkEgmOHz+u67CImPgQFWXevHn4/fffsXr1anTr1g1//PEHOnTogIyMDF2HVumUdeKjyo0bN9C5c2dYWloiODi4zGqv1LF//358+eWXug6jUGFhYfj6668RHR0NPz8/XYdDpGCk6wCIyrtu3bqhWbNmAICPPvoIjo6O+Prrr7Fnzx70799fx9EVLTU1FZaWlroOo0K4efMmOnXqBHNzcwQHB8PLy0vXIVVY/v7+ePHiBRwcHLBjxw7069dP1yERAWCND5Ha2rVrBwB48OCB0vI7d+6gb9++cHBwgJmZGZo1a4Y9e/Yo1ickJMDQ0BDff/+9YllcXBwMDAxQpUoVCIKgWD569Gi4uroqXp86dQr9+vVDjRo1YGpqCnd3d0ycOBHp6elKMchv3Tx48ADdu3eHtbU1Bg0aBADIzMzExIkT4eTkBGtra/Tq1QuPHz8uVpmPHz8OiUSCbdu24fPPP4erqyssLS3Rq1cvREVFFbl/amoqJk+eDHd3d5iamqJOnTpYunSpUpklEglSU1OxceNGxe3FYcOGFSs+Tbh9+zY6d+4MU1NTBAcHo2bNmho9flpaGkaOHIkqVarAxsYG77//Pl6+fJlvuwMHDqBdu3awtLSEtbU1evTogZs3byrWDxs2DCtXrgQApVuxckuXLkXr1q1RpUoVmJubw9/fHzt27NBoWYrD2toaDg4OZX5eoqKwxodITfI2H/b29oplN2/eRJs2bVCtWjVMnz4dlpaW+PPPP9G7d2/s3LkTffr0gZ2dHRo0aICTJ09i3LhxAIDTp09DIpEgPj4et27dQv369QGIiY48wQKA7du3Iy0tDaNHj0aVKlVw4cIF/PDDD3j8+DG2b9+uFF9OTg6CgoLQtm1bLF26FBYWFgDE2qo//vgDAwcOROvWrXHs2DH06NFDrbIvWLAAEokE06ZNQ2xsLJYvX47AwECEhobC3Nxc5T6CIKBXr14IDg7Ghx9+iMaNG+PQoUOYOnUqoqOj8d133wEAfv/9d3z00Udo0aIFPv74YwBArVq11IqvpMLCwtCpUycYGRkhODhYK+cdO3Ys7Ozs8MUXXyAsLAyrVq1CRESEIqkExPdg6NChCAoKwtdff420tDSsWrUKbdu2xZUrV+Dp6YmRI0fiyZMnOHz4MH7//fd851mxYgV69eqFQYMGISsrC1u3bkW/fv2wd+/eIj/vtLQ0pKWlFVkWQ0NDpb9/ogpFICKV1q9fLwAQjhw5Ijx//lyIiooSduzYITg5OQmmpqZCVFSUYtvOnTsLfn5+QkZGhmKZTCYTWrduLfj4+CiWjRkzRnBxcVG8njRpktC+fXvB2dlZWLVqlSAIgvDixQtBIpEIK1asUGyXlpaWL75FixYJEolEiIiIUCwbOnSoAECYPn260rahoaECAOGTTz5RWj5w4EABgDB37txC34vg4GABgFCtWjUhKSlJsfzPP/8UACjFOnToUMHDw0Pxevfu3QIAYf78+UrH7Nu3ryCRSIT79+8rlllaWgpDhw4tNJa8hg4dKnTo0CHfcgDC+vXri7W/sbGx4ObmJlStWlW4e/dugdt26NBBqF+/frFjk5P/Hfn7+wtZWVmK5UuWLBEACH///bcgCIKQnJws2NnZCSNGjFDaPyYmRrC1tVVaPmbMGKGgr+/X/1aysrKEBg0aCJ06dSoy1rlz5woAinzk/XyLY/v27QIAITg4WK39iLSBt7qIihAYGAgnJye4u7ujb9++sLS0xJ49e1C9enUAQHx8PI4dO4b+/fsjOTkZcXFxiIuLw4sXLxAUFIR79+4peoG1a9cOz549Q1hYGACxZqd9+/Zo164dTp06BUCsBRIEQanGJ29tSmpqKuLi4tC6dWsIgoArV67ki3n06NFKr/fv3w8AipomuQkTJqj1Xrz//vuwtrZWvO7bty/c3NwUx1dl//79MDQ0zHfuyZMnQxAEHDhwQK0YNE0qlSIuLg4ODg5wdHTU2nk+/vhjGBsbK16PHj0aRkZGivfu8OHDSEhIwIABAxR/Q3FxcTA0NETLli0RHBxcrPPk/Vt5+fIlEhMT0a5dO4SEhBS57/vvv4/Dhw8X+di0aZOapScqP3iri6gIK1euRO3atZGYmIh169bh5MmTMDU1Vay/f/8+BEHA7NmzMXv2bJXHiI2NRbVq1RTJzKlTp1C9enVcuXIF8+fPh5OTE5YuXapYZ2Njg0aNGin2j4yMxJw5c7Bnz5587UISExOVXhsZGSmSMrmIiAgYGBjku4VTp04dtd4LHx8fpdcSiQTe3t6K23+qREREoGrVqkoJEwDUrVtXsV6XzM3N8csvv2DQoEHo0aMHDh8+rJXG4K+/d1ZWVnBzc1O8d/fu3QMAdOrUSeX+NjY2xTrP3r17MX/+fISGhiIzM1OxPG87oILUrFlT422biMobJj5ERWjRooWiV1fv3r3Rtm1bDBw4EGFhYbCysoJMJgMATJkyBUFBQSqP4e3tDQCoWrUqvLy8cPLkSXh6ekIQBLRq1QpOTk4YP348IiIicOrUKbRu3RoGBmKFrFQqxZtvvon4+HhMmzYNvr6+sLS0RHR0NIYNG6Y4v5ypqaliXyqe//3vf3j58iU++eQTvPPOO/jnn39gYmJSpjHIP8fff/9dqWG7nJFR0V/Xp06dQq9evdC+fXv89NNPcHNzg7GxMdavX4/NmzcXuX9KSgpSUlKK3M7Q0BBOTk5FbkdUHjHxIVKDoaEhFi1ahI4dO+LHH3/E9OnTFb+QjY2NERgYWOQx2rVrh5MnT8LLywuNGzeGtbU1GjVqBFtbWxw8eBAhISFKY7Rcv34dd+/excaNG/H+++8rlh8+fLjYcXt4eEAmk+HBgwdKtTzyW27FJa+VkBMEAffv30fDhg0LPfeRI0eQnJysVOtz584dxXq54tRKaMvo0aMRHx+PWbNmYfDgwdi6datGE8h79+6hY8eOitcpKSl4+vQpunfvDiC3Ibezs3ORf0cFvU87d+6EmZkZDh06pFQruX79+mLFuHTp0mKND+Th4VFoLR9RecafhURqCggIQIsWLbB8+XJkZGTA2dkZAQEBWLNmDZ4+fZpv++fPnyu9bteuHcLDw7Ft2zbFrS8DAwO0bt0ay5YtQ3Z2tlL7HkNDQwBQ6votCAJWrFhR7Ji7desGAEpd6QFg+fLlxT4GAPz2229ITk5WvN6xYweePn2qOL4q3bt3h1QqxY8//qi0/LvvvoNEIlHa19LSEgkJCWrFpEkzZ87ExIkTsX37dowcOVKjx167di2ys7MVr1etWoWcnBxF+YOCgmBjY4OFCxcqbSeX9+9Ifivu9ffK0NAQEokEUqlUsSw8PLzYg0KyjQ/pA9b4EJXA1KlT0a9fP2zYsAGjRo3CypUr0bZtW/j5+WHEiBGoWbMmnj17hrNnz+Lx48e4evWqYl95UhMWFoaFCxcqlrdv3x4HDhyAqakpmjdvrlju6+uLWrVqYcqUKYiOjoaNjQ127typcgyYgjRu3BgDBgzATz/9hMTERLRu3RpHjx7F/fv31Sq3g4MD2rZti+HDh+PZs2dYvnw5vL29MWLEiAL36dmzJzp27IiZM2ciPDwcjRo1wr///ou///4bEyZMUGp35O/vjyNHjmDZsmWK24ItW7ZUK8bS+vbbb/Hy5Uv88ssvcHBwwNdff61Y9/z5c8yfPz/fPl5eXorxkgqSlZWFzp07o3///ggLC8NPP/2Etm3bolevXgDENjyrVq3CkCFD0LRpU/zvf/+Dk5MTIiMjsW/fPrRp00aRPPr7+wMQG6sHBQXB0NAQ//vf/9CjRw8sW7YMXbt2xcCBAxEbG4uVK1fC29sb165dK7Lsmm7jI3+v5OMQ/f777zh9+jQAYNasWRo7D5FadNehjKh8k3dDvnjxYr51UqlUqFWrllCrVi0hJydHEARBePDggfD+++8Lrq6ugrGxsVCtWjXhrbfeEnbs2JFvf2dnZwGA8OzZM8Wy06dPCwCEdu3a5dv+1q1bQmBgoGBlZSU4OjoKI0aMEK5evZqv2/bQoUMFS0tLleVJT08Xxo0bJ1SpUkWwtLQUevbsKURFRanVnX3Lli3CjBkzBGdnZ8Hc3Fzo0aOHUnd6eQyvd3dOTk4WJk6cKFStWlUwNjYWfHx8hG+++UaQyWRK2925c0do3769YG5uLgAosmu7Jrqzq3q/cnJyhN69ewsAhEWLFgmCIHZnRwHduzt37lzgOeR/RydOnBA+/vhjwd7eXrCyshIGDRokvHjxIt/2wcHBQlBQkGBrayuYmZkJtWrVEoYNGyZcunRJKb5PP/1UcHJyEiQSiVLX9l9//VXw8fERTE1NBV9fX2H9+vWKbuplraD3i5ce0iWJIOSpPyciUuH48ePo2LEjtm/fjr59++o6HIVhw4YhPDw83+SXEokE69evL9ORn4moYmAbHyIiItIbTHyIiIhIbzDxISIiIr3BNj5ERESkN1jjQ0RERHqDiQ8RERHpDb0awFAmk+HJkyewtrbW6dD4REREVHyCICA5ORlVq1Yt9VQyepX4PHnyBO7u7roOg4iIiEogKioK1atXL9Ux9CrxkU+QGBUVBRsbGx1HQ0RERMWRlJQEd3d3pYmOS0qvEh/57S0bGxsmPkRERBWMJpqpsHEzERER6Q0mPkRERKQ3mPgQERGR3mDiQ0RERHqDiQ8RERHpDSY+REREpDeY+BAREZHeYOJDREREeoOJDxEREekNJj5ERESkN5j4EBERkd5g4kNERER6g4kPERHpnYxsqa5D0LgcqUzXIVQITHyIiKhUZDJB8XzV8QcY8uv5Ai/C1x4n4MTd51qNJy0rBzeiEyEIgsr1v58Nh+/sg1gZfF9j58yWytBm8TH8eOxevnUvU7NQb85BrDnxACmZOSr3FwQBsckZJT7/gn230HjeYYTHpSrFFBWfVuJjvi4tKyffe5r3s68omPgQUYUXEvkSntP3YeN/4cXaPiYxAwlpWYVu8/hlGv44F6GRmoHSXhwuhsej/+qzaDb/CC5HvAQgluFJQjrC41Lx58UoxbZZOTJ4Tt8Hz+n71KoByMyRYsWRe7j2OKHY+8hkAjyn70PNz/fjUVwqZDIBXx+8g1P34rA79Iliu9ikDFwKj4fn9H3o9eMZDF13ASGRLzHpz1B8tPES4lIyi33OvOdecvAODt2Mybdu5l838NYPp7H11fuSkS3FhUfxkL76HFafeAgA+OZQWL59g+/EYtnhu4ptAeB5ciYO3Ywp9HMc+ftlRCekY+m/d/MlB02+Ooy0LCkWHbiDoO9Oqtx/yaEwtFhwFFsvRGLk75dwPCxWaX1MYgZm776B+7HJKvf/+dQjpGTmIGDpccX/A5+ZB9BuSbBSovnd4bv45tAdxev0LPFzv/dMPG5WjgzxqeL/DXk5Lke8xKdbrqDenEPotuIUniam42liOgasPYean+/H539dx5BfzyMxLbvA96c8kQgFpcSVUFJSEmxtbZGYmAgbGxtdh0NEanqZmgU7C2NIJBIAwLE7zzDvn1sIf5H7qzZ8cY8C978RnYh3Vv2HrBwxIQid8ybsLExUbtv0q8OIT83Cp528MblLHZXbRCekw83GDHGpmXiakIFG7nb5tll78gEW7r+DXZ+0RtMa9gCA74/ew88nH+LMjE6wMTMustye0/cpnrs7mCN4cgC8Zx7It92hCe1xKSIeM/+6AQBwtTHDjtGtUN3eQlx/MwYjf7+MCYE+cLQyxazdN7BuWDN08nXBquMP8PVB8YK4c3Qr+Hs4QBAExXudlpWDfqvPwtLUCDlSGQa19EDEi1R8f0ysNWnjXQU/DfRHo3n/AgC8HC1xYHw7vEjNQpvFx4os49fv+mHazuuo52YDU2MDfP1uQxgaSHD7aRL+vfkM896uDzsLE5y+F4d/b8XAxswYP76qsQlf3AOn7j3HzSdJGNm+Jrxm7Fcc9/a8rpi4LRQHb8Zg0pu1Ma6zj9L7uffTtqjjao0/L0Xhn6tPcO5hvGLdpVmBuB6diOHrLwIAqtmZ48z0TkpxP01Mx75rT7Hi6D0kZ+TW5jxc2B0GBhLcj01B4LITSvt093PFkr6NsPVCJMJikvHl2/VRb86hfO9Jz0ZV8W2/RjAxMkCfn87gSmQC7CyMcWX2m4hOSMeFR/GwtzRBC08H1J+rvP+Z6Z2U3vdTn3XEsTuxmLvnJgBgejdf/BR8HzWdrBAalaD4zB69qjEa29Ebmy9EYmT7mlh04A6KY3o3X4zqUKtY26pLk9dvJj5EpBPhcalIy5KiXtXi/V/cd+0pxmwOEfd9ldzkvYApjpsn8REEATeik1DL2RLmxoZo8tVhJLz2q1R+kX+d/Ng1HS1xbEoAAEAqExCfmoXLES9x91kylh2+q7TP7jFt0KCqDQ7dfIaYpAx82NZLKcZHi7pDIpEolhkZSHB/YXelY9x+moQb0Yno618dR2/H4qPfLuWL7bcPWuD9dRdUv1EqbP34DbTwdEDNz/erXB++uAc6fXscD5/n3iZpWsMOIZEJAIB+/tVR180G8/beKvAcdd1scPtpUr7lU4PqqKxZKYl3m1bHzpDH+ZZPDKyN747cVbFHfmuG+GPk75dLFUf9qjbwq2aLuT3ro+6cgyq3sTEzwodta2L9f4/y/c2p68rsN9Hkq8OlOkZZqedmg20j34B1MRJ6dTDxKSEmPlRZyaunXW3NFMsO3ojByuD7WP6/xqjlZKVYnpKZAytTIwBiG4AF+27j9tMkLHzHT2k7dc9vYmQAQRDw8e+XkS2VYf2w5oragteFRL7EOz/9BwDoWt8Vq4f4K9YJgoDULCkuhcfjj3MR+KZvI9hbmqD5giN4npx7S+TvMW3w9sozBca0sI8f0rJyMH/f7SLjvzu/myJ+uby1BiX1w4Am+HTLFaVlrycC8kQtM0eKkIgEDPj5HACghZcDLjyKR1mY2b0udl2JVpm4UMFqu1jh7rMUXYdR7uwY1QrNPPP/mCgNJj4lxMSHKit5DcI/Y9vC3tIYlyNeYvzWUABAC08H/DmqFQBg0C/ncOb+C/wwoAl6NqqKaTuuYdul3PYh8hoJdUzYegW7Q59g9eCmaFDNFm2/DgYAzO/dAHP33MQfH7ZENTtzfLXvFjKypfi8e110W3FK6RiBdV1w5PYz/DmyFfZee4LfzkYo1slvL3y08RKO3H6m9ntT3n3Zqz4EQcAX/xRcm1IWTIwMFLcAiUpD/kNCk5j4lBATH6pMUjNz8M/VJ/D3sMebBTSYlOvrXx07LivfIlg5sKni1pHce83c8XXfhvn2z9vWI69L4fHou/qs4nX9qja4+YS1BkQVWa9GVbHn6hOV61YNaorRm0JUrpMrrJ1dSWny+s1eXUQ6JggCxm+9gr9Do5WW33qShL3XxC+fuJRMPE/OxK+nH6Hj0uPIkcpQf+4hTN91vcikB0C+pAdAvqQHALZdikJmjhTH7jzDsTvPsOVCJBbtvw2vGfvhOX0f/rryGI/iUjH5z6u49yxZKekBwKSHcGFmZ+z6pLXWz+PuYI61eW6R5tW1viv6+lcv0XFPfdaxNGHB2sxI5fKBLWvA29kKhya0L9XxC3PjyyBM7+aLzSNaKpY5WZuqfZyaTpaK50H1XRTPzY0N0amus+L1rXlBeLSoO6Z19VUsm/d2fbXPV9ZUf0JEVKT41CyYvro9YG+Zv2fQ/utP4eVoCV9Xa2TmyGBmbKhYd/NJIpLSc/DHuQikZeUgOOw5/g59grcbV1Ns0/178XbQjegkrD7xQOnYjb78V0ulAurMUt1YEwAmbruqeK6qkWlF4mhlgriUwru0lzcPF3ZX2UD5/oJuOHU/DhIArrZm+CskGmtOPtTIOdcO8cfHeRoDX5oViJYLjyq6e9dwsEDkq7Fi7i/oBiNDAzhbmykdo4WnA7aNfCNfu6kfBjRBdEI6Fhez15BcSy8HbBsp3r5V1fB3af9GOBH2XGXCL7ft4zcw4OdzyNtD/Yue9eDuYIHVg/0x6o/iN4B2szVD/2buyMiRYlqQr8rPaM5b9ZS+A153dU4XRY84ALjzVVf4zi74/+KpzzoiJTMH7g4WaDD3ENp4V4GVqVG+XlWfBNTClypuo+4c3RrRCekYt+UKLszsjLMPXmD81lCM6+SNvPeBJr5ZG2uGNFPa9+CEdpDKBFiYiCnE6IBaeJaUAZkg4P1WngXGXF4w8SEqgf8exGHgz+cVr69/0QVWpkaKBOfa4wR88qo6eEALd+y+8gT/TmwPdwcLPElIR4/vT6s8bnxqFkIiXir15Hk96QGA1KzKN+psSd35qiuiE9LR+dvcLsPtfBxx6l5cvm2b1LBDVo4M5saG2D6qFW4+ScJbP6j+LNTVydcZPfzcEFjXBX9deVyqNjtGBhLkyAQE1XfBoZtiu6YV/2sMAwMJbMyMkJSRgyqWJhjSygMftasJI0MDdKyT+0t8RncbvNfcHafvx6GanTk+3Cj+Pf0woAkcrUwxbusVpYbiBQmeEgAvR0uMbF8Ta04+hEcVCzhameL+gm44c/8FfN2scf1xIoZvELt7Gxnm3kRwsDRRjAezbeQbkEgk+PpdPxy4EYPjYeK4Mj0bVYUgCMiRynD3WYri9sq3/Rph8vbcJPvsjE5wtTHDg+cpSEzPRh3X3Fsdqn50WJkaoUt9F/h72KO6vTnaeDsiKT1bqaG7oYEEDxf1UOp1N7S1JwCxlmNYa09seDUezurB/qjnZoOtFyPx03Hl/4+mRgY4+VlHGOcpe/CUAKw7/QhpWVJce5yA0QG1lJKea190wd0Y5RpTWwtjmBoZIDNHhuXvNYaZsSEuzQrEtotRaO7pgP5rcrfdPaYN3B0sFK9V3Vr6Z2xbPE1MR103G5WJj7+HPfw97NGrUVUAwNuNqyGgjjNszY1x9sELrDgqDsRoa56/d5ava/5bTV/0Kv81PXJs40OkpnWnH6ns1vt246r4O1Rsc9PE3Q6/nH6Ub5ug+i6o6WSFVcfzJzOkvgV9GmBQSw8AQOtFR/EkURz59sikDkpjp2z6qCVqOVnBxcYUggBIJIBEIkFiWrbSr+zXfdjWCzZmxvjuyF0sebchFh+8g3eaVFP6bN9pWg3zezdQ/PrNS1V3e7nT0zoqGoLndeerrjAzNkR6lhTmJoaKY1yaFQhHK1NkS2VIz5YWa/yfvHKkMkViEhqVgN4qesQNb+OJ3Vei8TItG339q2Npv0aKdQW18xIEAUsOhcGriiX6N3dXLE/NzMGvpx+hu58rvJ2tlfa5EZ0IGzNj1KiSe/GWyQTU/Hw/6rhY49DE9tgV8hiT/hSTn7D5XWFqVHBtydjNIdh77SkAoIefG1YOaqqy/PKxj0wMDRAy501YmRop3t93mlTDsvcaK+3zKC4VGdlS1HXLvV7k/UxL25ZFJhMw+NfzeLdpdbzrXx3JGdl4FJeKhtXtlLZLycxBg1fj9Fyd20VlMlKY42GxqGJpiiHrziMhLVuR0BYkLCYZQcvFW+i353WFuUnB731ZYePmEmLiQ6Uhkwn4YONFxa/ViqizrzOO3oktesNSOjejM95YdBQA0KpmFZx9+KJUx9sxqlW+9kQAMK6zDya9WVt8vuWKosbg0aLuitsqhQ1SmJUjQ+1Z+QcCBIC23o7446OWKtf9evoRFu6/jYPj28HHxVrlNgDQb/V/uBj+UuU6+UXz55MPsWD/bUXcrycXz5MzkZqZA89CLlQlcfBGDGo6WaKanTksTY3wIiUTDpYmEASxTZmTtanaPfw0KT1LijcWHYWLjSn+ndihWPskZ2TD0sQIBgYFx52eJUW2TKZIHLdciMTG/8Lxy9BmioEeC5P3h482GvEWJCYxA8aGElSxUr/NTl7ZUplS7VRB27z1/Wm42pph4wctSnU+TWHiU0JMfOh1VyJfYuzmK5jZoy66+7kVuu3fodGKLuIV1eVZgfCffyTf8pNTO6L9N/lrH4pjWGtPfBJQCy0WHlUsy3sBP3D9aZG9QOTCF/dAWEwyLoTHw9vJSjGmzcWZgbgenYCMbBnsLIwVtxk/7+6Lj9uLbRpikzIwbec1DH7DA53ruhR4jtfJf8Fv/KAF7C2M0Xf1WXza0RufdvYp9jEKki2VITE9GyuO3IOxoQHWncmtKZJfNLdciMSMXdeVlpEoLSsHxoYGRV6oy1piejZszIx0mhhqm0wmFJpAljVNXr/Zxof02ojfLiMuJROfbArB6IBaSr0TXrcnVHX3zvLi5pdBOPvgBZIzs5UaIcvNfqsebF6rIp8YWBvjOntDIpHgn7Ft0fNH1e1dTn3WEe2WqE6Muvu5wdnGDNXtzfH4ZToAKF0QiltNLh9UsY6rNeq4irUoRyd3QEpGDpysTdHJV0xmQiJza1CGtfZSPHe2McP64er/Oj31WUdk5kgVt2Ouf9Gl0Nsq6jA2NICjlSm+6t0AABD2LAln7r9AS6/cwd38qtlq5FyVkarbh+WBureaKqLylPRoWvn8qyLSInn7huXvNVaaHHHV8QcYHVALNmbGuPcsGWHPkrFo/x38NKgpGrnblcktInWtG9YM3/57F2M7esPS1AiB9Vzw4HnuSLIzuvni93MR2PdpO9haGOebPFGe9ACAr5vqWzanPusIdwcLbB7RErtCovP1lGnmIc4/dWxyAD7ZdBkj2tVUWt+6liPaeFdBE3d73I9NwcE8k0r2aOiGD9p44eHzFHT0dcbrVI0k3cTdDqM61EJtFyuNDJKWt5EoAI0lPaps+uiNfMsaVLPFz+83Q1U7MxV7EJGm8VYX6Z3CGpyO7+yDVSce5BvBNnxxj0L3K41RHWphZ8jjYvey6bj0OABxxF95L5TXbb8UBTsLE7xZL/8tn8IaZzb84hCS8ky0qGobqUyAgQRYfuQeHK1NMeQNjyLjlpPJBDx4ngJnazP8eysGQQ1c1W6kS0T6h7e6iF4jb7B3OSIe7vYWcLZR/etZ3r22IPIunK8rbDyQ0tj4QQt0qO2EjGypoutsXnnna9oztg28HC0x+c3auPU0CYMLSTj6NXMvcJ2c/6uamrwuzXoTyRnZKtsByRm+qgKf+KpRsToMDCSKxsDFiZGISNNY40MV3rXHCei3+izaejsqbkcVNOeUJud7+qCNl1JjVblVg5riwI0YPH6ZppjdWpV2Po74/UOx11BwWCyGrxfHQulSzwX/3hJjPDC+Hc4+eIH0bCnGdPTWSNxR8Wk4cfc5+jWrXuBtncS0bCzYfwv/a1EDTWvkT5CIiMoSa3yI8pi9+wYyc2RKbXCO3YlF+9pOSr1BEtKyNDrJ5Zye9TCusze+PhiGLRciFctbezuim58bTt17jiG/XlAslw8CJ/dpp9xeQ/Lkws3WDGvfb4bDt54hLSsHdd1slMYQ0QR3B4tCa4sAcTC1JX0bFboNEVFFxMSHKjypikrLDzdegp2FMeb2rIeA2s6wtzRB43mHVexdOnYWJlj0jh+auNvhs53XAIgjuQLiODDvt/LAb2cj8NOgpuju56aU+LTI07PH1twYV+d2Ueyrqm0OERGVXvkaHIHKpSuRL7Hu9CPIZGVzV3TT+Qise23U4+SMbCw7fBf3niXn2z6tgOkbEtLEbt1NvjqMxPTsQs/5Rc96asU4MVC5fUtTDzvFc5NXtUwSiQTz3m6AR4u6FzlGECAmP4XN5UNERKXHGh8qUp+f/gMA1HK2QofaToh4kYqp269hdEAtlV2QSyMrR4aZf90AALzVyA3O1mY4HhaLYa/av3x/9F6+XkbysWMKU9iknltGvIH7sfkTqoKoGmSulpMV3m5cFY5WpvnGv8jb1ujBwu54FJeKWk6aHYWXiIiKhzU+VGxXXg0cN3XHNVwIj1dMTFhaG/8Lh+f0fThw/SkycnJrbxLTxFoaedIjdyk8Ht1WnILv7AP468pjZEuVu56r642aDthyIapY21qbqv6tIJFIsOJ/TTD7rcJrjgwNJPB2tqrUI74SEZVnTHyo2OR3uuLyjDeTroFZwufuuQkAGL0pBJnZuUmMTAC2XYzMt33f1Wdx+2kSMrJlmLjtKkrSL9EkT6NniURS6IR9eV2cFaj+yYiIqNzgrS5SkvOq9sRIxdw48pEPUjJzB7jruuIkOtZxhp2FMY7ejkVr7yoY2soTVe3MFdvIZ5lWZdP5CKXXGdm5idThWzFYfya8xGUpzLEpHfDZjmto7ik2MJ7fuwH2XRdnd27mYY9LEWLt1gdtvOBqa4qF++/A38OebXCIiCo4juNDCjKZgC7LT0IqE3BkUgfFQHXykX7HdKyFEe1qFqt31B8ftkQLLwf8fOohvjkUhjVD/BFU31WxPi4lE80KGSRPW5a82xD9m+cfOC9HKoP3THGW7tWD/dHY3Q6utrmDID5JSIeTtWm5myyRiEgfcBwf0orkzBzcjxXneQqJfKmoDZGTCSh2l/DBv55XmrRy5O+XFY2CV594gMUH7mgw8uJTlfQAyjVcUpmglPQAUKrBIiKiios/X0nRTT3v/FQzdl3HkVvPMOK3S7nbqVk5+Hpvq8G/nMfak9pLepb208yAezbm/D1ARFRZVajEJzo6GoMHD0aVKlVgbm4OPz8/XLp0qegdSSEtKwcfbbykmHvqUng8Gs37F39ejFJqX3M/NgUf/XYJh2/ljnS85sTDfMdTx+n7cVi4X3s1PX39q6tcvqBPA/z2QQvc+DKo0P2X9muED9t6oa23ozbCIyKicqDCJD4vX75EmzZtYGxsjAMHDuDWrVv49ttvYW/PeYTUsfbkQxy5/QxTtl8FAIz6IwTJGTn4bOe1Igf5K89m9agLAHiroThQYGDd3PGFLj6KR/vaTrAqoCu6XF//6pj9Vj12NSciqsQqTJ3+119/DXd3d6xfv16xzMvLS4cRVUzReW4/xadmITkjN9l564fTugipWCa/WRvfHr5b4PqP2tUEAHzTtxH6+leHj4s1jtw+BgDokqdRNRER6bcKk/js2bMHQUFB6NevH06cOIFq1arhk08+wYgRI3QdWoWRLZXh76tPFK+bfqX5uau0pV5V5Vb864c3R0RcKupVtVUaBdncxBABdcTanoEtayDiRSq6NWDiQ0REogqT+Dx8+BCrVq3CpEmT8Pnnn+PixYsYN24cTExMMHToUJX7ZGZmIjMzd7C9pKSksgq3XAr45rhSA+aKpNNrU2N0rOMM1Cl8n4V9/LQYERERVUQVpo2PTCZD06ZNsXDhQjRp0gQff/wxRowYgdWrVxe4z6JFi2Bra6t4uLur7spc2d1+moTUzBxEJxQ9p5UuTH6zdqHrF/bxg0QiwbUvusDRygQ/DWpaRpEREVFlU2ESHzc3N9SrpzwPUt26dREZmX9KA7kZM2YgMTFR8YiKKt58TJXJwRsx6LbiFHqvPFOm553SpfBkxtUmd5wcU+PC/ww9q1gAAGzMjHFp1pvFmumciIhIlQqT+LRp0wZhYWFKy+7evQsPD48C9zE1NYWNjY3SQ9+M+uMyAODeq4EJNeGr3g3wbRFj5nzUria2ffwGHizsjiV9G6KuW+57b2VqhHOfd1ba/t2muV3RX583y5TTRBARkYZUmDY+EydOROvWrbFw4UL0798fFy5cwNq1a7F27Vpdh1YupWXl4N4zzSU7eQ15Q0w2/7n2BMfDnqvcxszYEC1rVgEA9G/mjv7N3BGblIGHcal449VyOUMDAyzp2xA7Q8SxhUa2r4npu64r1psz8SEiIg2pMIlP8+bN8ddff2HGjBmYN28evLy8sHz5cgwaNEjXoZVLw9ZdxIXweI0fd++nbRXPVw/2x6l7cWjjXQX15hwqcl9nGzM457nF9WFbLxy5/Qz9mlWHoYEE52Z0RkjkSwTVd8WZBy/wz6seaGZF3AojIiIqLk5SWknJJxbVlEeLuhc6sF+OVIb+a84iJDIBABTzchVFEASVx/3yn5uKmdmDpwTku/1FRET6Q5PXb/6UrmQexaWi07fHNXa8QxPa48aXQUWOZmxkaIAV/2uCno2qYs/YNsU+fkHHlY/ADABur00YSkREVFIV5lYXFc+cv2/g4fNUtfbpUNsJJ+6KbXV6NqqquMW0Y1Qr1HG1LvZx3B0s8MOAJmqduyD+Hg4Y39kHrrZmMGMbHyIi0hAmPpVEfGoWMrKlSM+SFr3xa9YPa46HcSk4HvYcQ1p5YHxnH8QkZqCZp4MWIi2+iUWM70NERKQuJj6VREmnn6jnZgMDAwm8na3h7SzW7ng7W8Hb2UqT4REREZULbONTCeRISz4Nxc9Dm2kwEiIiovKNNT6VQFp28W9vBdZ1xs/vN0N6thQWJvz4iYhIv7DGpxJQp13PkduxkEgkTHqIiEgvMfGpBM49fFHo+obVbcsoEiIiovKNiU8lIO+KXpC/xxR/XB0iIqLKjIlPJdClnmuh6/MOEtikhp2WoyEiIiq/mPhUCsWfdeSrtxtoMQ4iIqLyjS1cK4FVxx8UuM731cjL64c1x7OkDDSoxvY+RESkv5j4VAJXHyfmWzYh0AcftasJMyOxUq+jr3NZh0VERFTuMPGp4GKTM1QunxDI6R6IiIhexzY+FVz0y3Rdh0BERFRhMPGp4MxN8s9c3s+/ug4iISIiKv+Y+FRwRgaSfMu+6ddIB5EQERGVf0x8Krg/Lz3WdQhEREQVBhOfCiw2OQNrTz7UdRhEREQVBhOfCiw5I0fXIRAREVUoTHwqsPyte4CZ3euWeRxEREQVBcfxqcAyc2RKr+981RVmxvl7eREREZGINT4V1K0nSZjz9w2lZUx6iIiICscanwqq+/endB0CERFRhcManwroeXKmrkMgIiKqkJj4VDBSmYDlR+7qOgwiIqIKiYlPBbPz8mNsOh+Zb3mj6rY6iIaIiKhiYeJTwaw8fl/l8m0jW5VxJERERBUPE58KJuJFWr5l7zatzh5dRERExcDEpxLYGcL5uoiIiIqDiQ8RERHpDSY+lUDPRlV1HQIREVGFwMSnEpgY6KPrEIiIiCoEJj6VQE0nK12HQEREVCEw8alAsqWyojciIiKiAjHxqUBuRCfmW3ZhZmcdREJERFQxMfGpQOb8fTPfMmdrMx1EQkREVDEx8alAkjOydR0CERFRhcbEpwLJzGEbHyIiotJg4lOBMPEhIiIqHSY+FYidhbHS6yY17HQTCBERUQVVosQnOzsbUVFRCAsLQ3x8vKZjogJ0a+AKAJBIgC971ccv7zfTcUREREQVS7ETn+TkZKxatQodOnSAjY0NPD09UbduXTg5OcHDwwMjRozAxYsXtRmr3vv9bAQAoJ2PE4a29kQVK1MdR0RERFSxFCvxWbZsGTw9PbF+/XoEBgZi9+7dCA0Nxd27d3H27FnMnTsXOTk56NKlC7p27Yp79+5pO269lJSRAwA4efe5jiMhIiKqmIyKs9HFixdx8uRJ1K9fX+X6Fi1a4IMPPsDq1auxfv16nDp1Cj4+nD9Kk3aFPNZ1CERERBWeRBAEQddBlJWkpCTY2toiMTERNjY2ug5HLZ7T9ymet/B0wJ+jWukwGiIiorKjyet3iXt13b9/H4cOHUJ6ejoAQI/yJ517v7WHrkMgIiKqkNROfF68eIHAwEDUrl0b3bt3x9OnTwEAH374ISZPnqzxACk/CSS6DoGIiKhCUjvxmThxIoyMjBAZGQkLCwvF8vfeew8HDx7UaHBEREREmlSsxs15/fvvvzh06BCqV6+utNzHxwcREREaC4wKJmGFDxERUYmoXeOTmpqqVNMjFx8fD1NTjitDRERE5ZfaiU+7du3w22+/KV5LJBLIZDIsWbIEHTt21GhwpFo9t4rVI42IiKi8UPtW15IlS9C5c2dcunQJWVlZ+Oyzz3Dz5k3Ex8fjzJkz2oiRANRyssSD56n4qncDeDpa6jocIiKiCkntGp8GDRrg7t27aNu2Ld5++22kpqbinXfewZUrV1CrVi1txEgApDJxuIB6btY6joSIiKjiUrvGBwBsbW0xc+ZMTcdChQh/kabrEIiIiCo8tROfa9euqVwukUhgZmaGGjVqsJGzhiVlZCueXwp/CX8PBx1GQ0REVHGpnfg0btwYklf9qeWjNUvy9K82NjbGe++9hzVr1sDMzExDYeq3/+7HKZ77VbPVYSREREQVm9ptfP766y/4+Phg7dq1uHr1Kq5evYq1a9eiTp062Lx5M3799VccO3YMs2bN0ka8emnUHyGK5y1rVtFhJERERBWb2jU+CxYswIoVKxAUFKRY5ufnh+rVq2P27Nm4cOECLC0tMXnyZCxdulSjwRJgaMDRC4mIiEpK7Rqf69evw8Mj/ySZHh4euH79OgDxdph8Di8iIiKi8kLtxMfX1xeLFy9GVlaWYll2djYWL14MX19fAEB0dDRcXFw0FyURERGRBqid+KxcuRJ79+5F9erVERgYiMDAQFSvXh179+7FqlWrAAAPHz7EJ598ovFg81q8eDEkEgkmTJig1fPoWsSLVF2HQEREVGmo3candevWePToETZt2oS7d+8CAPr164eBAwfC2locXG/IkCGajfI1Fy9exJo1a9CwYUOtnqc8WH8mXNchEBERVRolGsDQ2toao0aN0nQsxZKSkoJBgwbh559/xvz583USQ1na8F+4rkMgIiKqNEqU+ADArVu3EBkZqdTWBwB69epV6qAKM2bMGPTo0QOBgYFFJj6ZmZnIzMxUvE5KStJqbNo2OoBTghAREZWG2onPw4cP0adPH1y/fh0SiSTfIIZSqVSzEeaxdetWhISE4OLFi8XaftGiRfjyyy+1Fk9ZG9WeiQ8REVFpqN24efz48fDy8kJsbCwsLCxw8+ZNnDx5Es2aNcPx48e1EKIoKioK48ePx6ZNm4o9IvSMGTOQmJioeERFRWktvrJgY17iCjoiIiJCCWp8zp49i2PHjsHR0REGBgYwMDBA27ZtsWjRIowbNw5XrlzRRpy4fPkyYmNj0bRpU8UyqVSKkydP4scff0RmZiYMDQ2V9jE1Na1U84blnRqEiIiI1Kd24iOVShW9txwdHfHkyRPUqVMHHh4eCAsL03iAcp07d1YMkCg3fPhw+Pr6Ytq0afmSHiIiIqLXqZ34NGjQAFevXoWXlxdatmyJJUuWwMTEBGvXrkXNmjW1ESMAsSdZgwYNlJZZWlqiSpUq+ZZXRu4O5roOgYiIqMJTO/GZNWsWUlPFQfXmzZuHt956C+3atUOVKlWwbds2jQdIopFs2ExERFRqaic+eScn9fb2xp07dxAfHw97e/syb4OizcbU5YWbrRmeJmagXlUbXYdCRERU4anVqys7OxtGRka4ceOG0nIHBwc2vNUS+btqxFnZiYiISk2txMfY2Bg1atTQ6lg9pCxHJo6TZGSg9sgDRERE9Bq1r6YzZ87E559/jvj4eG3EQ6+RyhMfQ9b4EBERlZbabXx+/PFH3L9/H1WrVoWHhwcsLS2V1oeEhGgsOMqt8THkrS4iIqJSUzvx6d27txbCoILkSGUA2MaHiIhIE9ROfObOnauNOKgArPEhIiLSnBK1mE1ISMAvv/yCGTNmKNr6hISEIDo6WqPBUW4bH2NDNm4mIiIqLbVrfK5du4bAwEDY2toiPDwcI0aMgIODA3bt2oXIyEj89ttv2ohTLwmCwBofIiIiDVK7GmHSpEkYNmwY7t27pzRLevfu3XHy5EmNBqfv5LU9ANv4EBERaYLaic/FixcxcuTIfMurVauGmJgYjQRFopw8iQ9rfIiIiEpP7cTH1NQUSUlJ+ZbfvXsXTk5OGgmKRHlrfNjGh4iIqPTUvpr26tUL8+bNQ3Z2NgBAIpEgMjIS06ZNw7vvvqvxAPXZy7QsxXPW+BAREZWe2onPt99+i5SUFDg7OyM9PR0dOnSAt7c3rK2tsWDBAm3EqLc+2nhJ8dyQc6ERERGVmtq9umxtbXH48GGcPn0a165dQ0pKCpo2bYrAwEBtxKfX7sQkK54bsMaHiIio1NROfKKiouDu7o62bduibdu22oiJiIiISCvUvtXl6emJDh064Oeff8bLly+1ERMRERGRVqid+Fy6dAktWrTAvHnz4Obmht69e2PHjh3IzMzURnxEREREGqN24tOkSRN88803iIyMxIEDB+Dk5ISPP/4YLi4u+OCDD7QRo14SBKHojYiIiEgtJR4cRiKRoGPHjvj5559x5MgReHl5YePGjZqMTa/lHcOHiIiINKPEic/jx4+xZMkSNG7cGC1atICVlRVWrlypydj0Wg4THyIiIo1Tu1fXmjVrsHnzZpw5cwa+vr4YNGgQ/v77b3h4eGgjPr3FxIeIiEjz1E585s+fjwEDBuD7779Ho0aNtBETAciRynQdAhERUaWjduITGRkJCUcR1rqo+HRdh0BERFTpqJ34yJOetLQ0REZGIisrS2l9w4YNNROZnlsZfF/XIRAREVU6aic+z58/x7Bhw3Dw4EGV66VSaamDIiApI1vx/L1m7jqMhIiIqPJQu1fXhAkTkJiYiPPnz8Pc3BwHDx7Exo0b4ePjgz179mgjRr2UkZ2bQC5+10+HkRAREVUeatf4HDt2DH///TeaNWsGAwMDeHh44M0334SNjQ0WLVqEHj16aCNOvfMoLlXxnG2qiIiINEPtGp/U1FQ4OzsDAOzt7fH8+XMAgJ+fH0JCQjQbnR57mZZd9EZERESkFrUTnzp16iAsLAwA0KhRI6xZswbR0dFYvXo13NzcNB6gvrIxU7syjoiIiIqg9tV1/PjxePr0KQBg7ty56Nq1KzZt2gQTExNs2LBB0/HprTfruWJnyGN81rWOrkMhIiKqNNROfAYPHqx47u/vj4iICNy5cwc1atSAo6OjRoPTZ7JXk5QaG5R4VhEiIiJ6Tanvp1hYWKBp06aaiIXykE9SamDAhs1ERESawuqEckqe+Bgx8SEiItIYJj7lFGt8iIiINI+JTzklFVjjQ0REpGlMfMopeY2PIQcvJCIi0hi1Gzdfu3ZN5XKJRAIzMzPUqFEDpqampQ5M3/FWFxERkeapnfg0bty40CkUjI2N8d5772HNmjUwMzMrVXD6TN6d3ZB1ckRERBqj9mX1r7/+go+PD9auXYvQ0FCEhoZi7dq1qFOnDjZv3oxff/0Vx44dw6xZs7QRr95Q1PjwVhcREZHGqF3js2DBAqxYsQJBQUGKZX5+fqhevTpmz56NCxcuwNLSEpMnT8bSpUs1Gqw+UbTx4a0uIiIijVG7xuf69evw8PDIt9zDwwPXr18HIN4Ok09rQSWjuNXFGh8iIiKNUTvx8fX1xeLFi5GVlaVYlp2djcWLF8PX1xcAEB0dDRcXF81FqYfYuJmIiEjz1L7VtXLlSvTq1QvVq1dHw4YNAYi1QFKpFHv37gUAPHz4EJ988olmI9UjKZk5CIlMAMAaHyIiIk1SO/Fp3bo1Hj16hE2bNuHu3bsAgH79+mHgwIGwtrYGAAwZMkSzUeqZL/fcVDxnGx8iIiLNKdEkpdbW1hg1apSmY6FXgsNiFc95q4uIiEhzSpT43Lt3D8HBwYiNjYVMJlNaN2fOHI0Eps9etWsGwCkriIiINEntxOfnn3/G6NGj4ejoCFdXV6XBDCUSCRMfDciT93AcHyIiIg1SO/GZP38+FixYgGnTpmkjHgIg5KnyYRsfIiIizVG7O/vLly/Rr18/bcRCr7xMy1Y855QVREREmqP2ZbVfv374999/tRELqcBbXURERJqj9q0ub29vzJ49G+fOnYOfnx+MjY2V1o8bN05jwRFvdREREWmS2onP2rVrYWVlhRMnTuDEiRNK6yQSCRMfDWONDxERkeaonfg8evRIG3FQHhJJbpd21vgQERFpDpvOlkP2FiaK50x8iIiINKdYNT6TJk3CV199BUtLS0yaNKnQbZctW6aRwPRZfGruBLDpWVIdRkJERFS5FCvxuXLlCrKzsxXPCyJhe5RSy5Yqj4T9IjVTR5EQERFVPsVKfIKDg1U+J83bd+2p0uvWtRx1FAkREVHlo3Ybnz/++ANpaWnaiIUA3I5JUnptZmyoo0iIiIgqH7UTn4kTJ8LZ2RkDBw7E/v37IZWyDYomZWbLit6IiIiISkTtxOfp06fYunUrJBIJ+vfvDzc3N4wZMwb//fefNuLTO76u1roOgYiIqNJSO/ExMjLCW2+9hU2bNiE2NhbfffcdwsPD0bFjR9SqVUsbMeoVozyTc9mYqT3MEhERERWiVOP4WFhYICgoCN26dYOPjw/Cw8M1FFZ+ixYtQvPmzWFtbQ1nZ2f07t0bYWFhWjufrpgZ534kZ6Z30mEkRERElU+JEp+0tDRs2rQJ3bt3R7Vq1bB8+XL06dMHN2/e1HR8CidOnMCYMWNw7tw5HD58GNnZ2ejSpQtSU1O1dk5dyJGKQza38a4CazPjIrYmIiIidah9L+V///sf9u7dCwsLC/Tv3x+zZ89Gq1attBGbkoMHDyq93rBhA5ydnXH58mW0b99e6+cvK/JxfIwMOKg2ERGRpqmd+BgaGuLPP/9EUFAQDA1119U6MTERAODg4FDgNpmZmcjMzB0AMCkpqcBty4scmVjjY2zIwSCJiIg0Te3EZ9OmTdqIQy0ymQwTJkxAmzZt0KBBgwK3W7RoEb788ssyjKx0wuNSMWPXdQCs8SEiItKGEl1dT5w4gZ49e8Lb2xve3t7o1asXTp06penYCjRmzBjcuHEDW7duLXS7GTNmIDExUfGIiooqowhLZszmEMVzQ9b4EBERaVyJRm4ODAyEhYUFxo0bh3HjxsHc3BydO3fG5s2btRGjkrFjx2Lv3r0IDg5G9erVC93W1NQUNjY2So/y7FFcbkNtY87KTkREpHFq3+pasGABlixZgokTJyqWjRs3DsuWLcNXX32FgQMHajRAOUEQ8Omnn+Kvv/7C8ePH4eXlpZXz6FJanpnY847nQ0RERJqh9tX14cOH6NmzZ77lvXr1wqNHjzQSlCpjxozBH3/8gc2bN8Pa2hoxMTGIiYlBenq61s6pS6/P0k5ERESlp3bi4+7ujqNHj+ZbfuTIEbi7u2skKFVWrVqFxMREBAQEwM3NTfHYtm2b1s6pS3+HPtF1CERERJWO2re6Jk+ejHHjxiE0NBStW7cGAJw5cwYbNmzAihUrNB6gnCAIWjs2ERER6Qe1E5/Ro0fD1dUV3377Lf78808AQN26dbFt2za8/fbbGg+QiIiISFNKNAtmnz590KdPH03HQkRERKRVpZr+OyUlBTKZciPc8t5lvKKY1aOurkMgIiKqdNRu3Pzo0SP06NEDlpaWsLW1hb29Pezt7WFnZwd7e3ttxKiX3GzNdR0CERFRpaN2jc/gwYMhCALWrVsHFxcXSCQcaE8bOIwPERGR5qmd+Fy9ehWXL19GnTp1tBEPvWLAhJKIiEjj1K5XaN68ebmf86oyYE0aERGR5qld4/PLL79g1KhRiI6ORoMGDWBsbKy0vmHDhhoLTp9diojHm/VcdB0GERFRpaJ24vP8+XM8ePAAw4cPVyyTSCQQBAESiQRSqbSQvam4snM4YCMREZGmqZ34fPDBB2jSpAm2bNnCxs1aZG7C1s1ERESapnbiExERgT179sDb21sb8dAr9hYmug6BiIio0lG7WqFTp064evWqNmKhPOq5cSBIIiIiTVO7xqdnz56YOHEirl+/Dj8/v3yNm3v16qWx4PRZa29HXYdARERU6aid+IwaNQoAMG/evHzr2LhZMwwN2G6KiIhIG9ROfF6fm4s078D4droOgYiIqFJi16FyIluam1CmZbHWjIiISBuKlfhs3bq12AeMiorCmTNnShyQvsrMyZv45OgwEiIiosqrWInPqlWrULduXSxZsgS3b9/Otz4xMRH79+/HwIED0bRpU7x48ULjgVZ2UmnugIWmRqyIIyIi0oZitfE5ceIE9uzZgx9++AEzZsyApaUlXFxcYGZmhpcvXyImJgaOjo4YNmwYbty4ARcXTrWgrqw8t7q8HK10GAkREVHlVezGzb169UKvXr0QFxeH06dPIyIiAunp6XB0dESTJk3QpEkTGBiwpqKk8rbxcbDk4IVERETaoHavLkdHR/Tu3VsLoei3rFdtfKxM1f5IiIiIqJhYRVNOHLn9DACQksmGzaRHBA1Nxpv+ErixC8jJ1MzxiKjSYuJTTszfl7/ROJHGyaTAsfnAg2DtHD8rFXh4HJBmF71tdgbw0xvAn++X/rx7JwE7hgM7Pij9sYioUmPiQ1TeZWcAf48F7h9Vbz9pNhD/UHnZ1a3AyW+A33sX/zgZSUBmsup1idHAvilA3D3x9Y4PgN/eBo4vVr19Zkru86jzwPM7wK2/gZwssfYn8rzyuaTZwJNQIO/AqYfnAl/YAvePAKFbxOc3d4nr7uwFtg4Cnt3K3f7Cz8D1HcpxXN8B7BqpHI+6XjwQz/2FLbCyZcHbSbOBZzfz124JguZqvDSpPMZEpEESQdCfv/KkpCTY2toiMTERNjblaxLQSdtCsetKNAAgfHEPHUdDGnViiZi0vP0j4OiTu/zYAuDkEmDKfcDKqeD9D34OnFspPv8iMXd5VioQ8R/w5AoQvEBc9s7PgFd7QGIIHJgK3PxL+Vj1++Quy3ssQRBjtKsBmFoDJhaAma2YKP01UtxmzkvAwEC8rZT0FHCpJ1705T4NAX5omvt60m3A2g2QvJqCZc+nQMhvQPelgIkl8PgicGmduM61IRBzLXffKffFxGjboNxlTnWB52rUjHZfCtz+B3h0Qnw9/irwfRPxXE9DxWVmdkBGgvi8xzKg+Yfi84xEYHU7ICECGH5AfF8khsCt3YBPF6BKLeWyA4BbI+BlRO7x3BoBQ3YDS7zE14FfAM0+FMue+BhY0RCo1xvov1H5ODKZ+D7nfR36B+DWWHxPGg0ATIvZ8/O/H4CLvwCNBwHtp+Z+FtIcIDUWsKmqvP3Ta2Li2vFzoMUI1cfMyQKMKlgHjGc3gUvrgQ6fAVbOwPMwYF1XYPh+wLlu7nYyKWBgqLs4SyP6MnD/GNB2AmBoXOTmaslOF/9PWLtq9rhq0OT1m4lPOTF95zVsvRiFCYE+mBBYW9fhVH4vHogXtVqd1NtPJhO/YEwsAJf6yuuk2cDeCWJy8d8P4i2f13WeK144IQCr2+Yu/yIRuLIJuPgz0P838YvGsTYgywG+chK3z2viLeC7eurFri6fIODeoYLXt50InP5OuzGUtTrdgbD9RW/X6wcxkSsJ66pi8vPiVS3ZZ48Ac3sgM0m8IP/6JlD/HaD3T8CL+8p/J3IzY4Btg8VaryreQIN3gWrNgCNzgdhbQJMhwJXflfdxrA2kJ4gJT+2uwN2D4vLeq4DGA8Xk96c3xFo4ABhzETj6pViLZuUq/l2u65J7vNH/AQ41gUOfA+5vAD5v5iZ5AAAJMPIkYO8JGFuIxw07APj1BSydAEMTIO4ukB4P2FQTk8m4+8Dl9UDrceI2eRPAZ7eANe0AY0ugaiNg6D9AVpp4HGmWGOuVP4DJd8TkPS95kupQS0xkv83zHftFotg2bO8kMcEEAP/hwFvfATHXxc/Gzl35eOGnxZrJOt3yfzbhp8XyGpmJPw4Kc2m9+Dl9cCg3WclKBe7sAxr0zS3/te3A0XmAgyfQdz1gqWISa3kZuy0Rz+1YG/BopbyNIAAbXv2wHvoPkPRETGaizouJtamV+B0nywaMTHP3W9EYePkI+OBfoEae2s2I/4D13YBanYHu34ifoZaUeeIzadKkYh9w2bJlpQpIm8pz4uM5fR8A4MO2Xpj9lpYvaJVZdjpgaKr8hSknCGItQ04msPEtcVm/jUD93srbRV0UvwiaDhF/wctyxF+J5g7AQrfc7fr/JtbGNB4IdJoJfGGHfAlKcbUcBZxfXbJ9iUqr4f+Aa8Ufob9MfXxC/D+bN+kCgLq9gNt7AAMj8ZGTkX/f6ZFizeXrtXN5zYwB1ncHnoQoL39jTG5Na78NYtIUc12sQZNvO+q0mJi++6uYSEaHAPEPco8x+qyY/NzZB+z6GGjwDtB2EuDgBaTEAkt9lM858xmwoBjj4PXbCNR7G0iNE2uLI84C67vm327kScClATDPQXzdeHBucgcJ8n1f1XtbvPUMiMl9QhRQpyvw82s/ENtPFWtL/52pvLzTbLHW1Ny+6DKoqcwTn44dOyq9DgkJQU5ODurUqQMAuHv3LgwNDeHv749jx46VKiBtqgiJT103m/I3SWlWmljNaeNW8DbSHLEaXZfVxPJfH4DybRy5yxuAf8YXfozPHr32q5WIqBRqdwPuHtB1FGVrbkLubVUN0eT1u1iDxgQH5/YAWbZsGaytrbFx40bY24tZ3cuXLzF8+HC0a1fOLtgVkGF5bG6+pr1YLT/hutjW4XU5WWIVubkd8NFRsQ2Iub3qP/ykp2LV6uvrIs8BFo6Ao3f+fdLigYfB4m2A2l2Bak2V18c/FNtu5LU2AHCuJ94ukCsq6QGY9FQk7/0h/j2uaV+643i0EW8/HpmrmbiI8tK3pAcAUp7ptD1QUdS+zH777bdYtGiRIukBAHt7e8yfPx/ffvutRoPTR4YazpI1Qt4W4d6/4r8yKbB9GPDHu0DqC2Bzf7F6N/oy8OikmDxs7i/ekz7ypXiLKf4RcGoZsMxX7FW0rhuw4S0xqUl6AqwLAn70F28tfWELrM1Ty7jES+wtdOJr4OeOwLlVwCJ3YPcY4MGx/EkPIDb4Dd2U2+umoF5GuhS0EPB9S/PHHbQj/7LAL8VfYbOe557TyRf4/EnRx5t0RzymW2PV68dcKGmkosl3gXGhua/bTxVr7GbF5i6bGQOMOgP0fnU70MgcqNtTbEA88abY4LdWZ+Xj9lkjNpKeHae8vMXHQN91YtmnRYgNXNtOANpNKVn8c+LFhtNG5rnLJtwABmwr2fFUqduz4HUtR+U+N7cHmg4F3vgEmPpALP/cBKD/7wXuXqTmHym//jRE9XZ515f2b4IqtpRnuo6gUGo3bra2tsY///yDgIAApeXBwcHo1asXkpML6PZaDlSEW12N3e2we0wbHUfzyqllYoNBOcc6YrsXQQYcnqO589hUB5Ie51/efITYUHhDd82dS1NMrICsArpCj78m9tgxMsttdzBsv3I5HGoB415dQFTd6x99VmzcaWYLhJ8Uu7TXfZWw7PgQuPEqufk0RPyScW2o3NPn8BzA1l1Mbh4cE3vp5O3pIc0BDF9V+KbFi2WxdRdrz6LOA7tHi+tc/IDRp3P3y0wW481KFRuaArm3FTf2FBPfd38Fdr7qHVX/HaDfeuDvMWIj17ADYoIs12E60HGG+PyfCcD17cAnZ3NrFuPuiYm2s2/uPgX1vMnJEtsn1OwgtsnKS96+q2qTwnu8xD8Sz33zL7EBsq272CjXr2/uNukvxc9/cQ2xEXH3JeLyrDTxM8/bvizxsdj248B0sUY09XnuurdXAvcOiz3FXufVIbc32vt/A1WbAovdxdjqvS02nu+2BGg5MjduI9P8vbRUCT8t/oh499f8f3cAMOOxOKyARyvxPZBLjgEsncXynV0p3lp+YzQQexvY/yppHLwL8H6VgAqC+Pe/oJBf/vIy3NkPbB1QdOyfPRLb2fzWq/DtPjoG/FJEx4VJd8QfYnnZ1gAa9AHOrCg6lsLU7Sn2KMyr2YfApV8L3qdB39z/1+rybCf23Ns9qvDtuswH/p2lvGzqQ+CbmrmvW4wUa+Rfb3M45Z74/2rvpNxy1O8jNkT/Oc8PVXm7Kg3Taa+u999/H6dOncK3336LFi1aAADOnz+PqVOnol27dti4cWMRR9Cd8pr4SGUCan0u9iRp7mmP7aNaa+9kiY/FC0zjAWJPEEC88Jnbi1/KZ1eKvZUa9i+8QWBlZ24vXtT++155uZmt+GvezAaIuiB2wW40ULzgLXQTv4AG7xAbY0oMAUEKGBgDyU9ze2H59Rdre/J2YZdJgf1TxV4iNTvmJiUFEQSN30NXkp0ujt/zes+aosiTkg1vAeGn8t/rz0gUL5TuLYGkaLE3j3y9IIi9c/L2JqksstMBSJQbrsoTxsxksRfS6eXi7bamQ4Fe34u3diUGykMgaNrZlWKvrE6zxIEthx/M3xOoKHf2AVsHis8/fyr2eMxLJgMi/xO/V7JSxQa5VRvnP07wQjEhm/FYrD0TZOLfkiArOFmNuwf82Ex87uQLNPof4P0m4NogN5kaESzeHhcEMUEL+V1MOhv2E/+f/jtLHAKibk/l/1cyqfi5mVqJg2zKG/02HwH4DxUT/ZajgGYfiD8uqjdX7tW0dZDYI07us0eAhYP4o+OrKuKyRgOBq5vF53NeAvNe3Umx9xRrM43MgGvbxIQsMfLVPgOAq1vE56+3Zcz7na2qnSMA/BoERJ0Tn8v/f8qk4g8aeTvO9d2BiDPi83pvAwEzcrv9S3OA5X5AvV5At6/FZdr+PoKOE5+0tDRMmTIF69atQ3a2ODqrkZERPvzwQ3zzzTewtLQs4gi6U14Tn4xsKXxni11LW9Wsgi0fv6GdE73+q+qLRHEgt50f5m+A1+Bd4MZO7cRRXjjXE/9T138HSH4ijl8CADOic2tPstLEpEUT3TTPrBC73r4xuvTHooopdIvY1mzyHfEi+DpptubHYNE2QQB+bC4mNq+PSaTvEqLE2/jVm4m9QOWyM8Sk1tBYrGF1aSB+51z4WbwtP3inmBxKs8VeZF7tX9W2GYp/NwlR4o8SYzPl813bDuz6SPz+7rtOdUwyqTi2WMP+BX+vhZ/JraEecQyo5l/qt6K0dJb4SKVSnDlzBn5+fjAxMcGDB2K3vVq1apXrhEeuvCY+CWlZaDzvMACgnY8jfv+wkFFg1SGTir84arQW2+CsC1JeP/yg6i6Q+qDhe8A7a5WXCYL4nhVV20JEpC2lrT3RxCCMeWulPg3R6vg8xVXmvbrkDA0N0aVLF9y+fRteXl5o2LBhqU5Ooinbc0esNTLQYHXhdw3EmgxAvH3yutIkPVYuZduA7bNHYruV1wdlA8TqdUMT8T/7jZ3iSMNvLQf2jBXXDz8IQBAbUcvbnlRtmv84EgmTHiLSrdLeMtLEkCKGRsDA7eKgmuUg6dE0tb/lGzRogIcPH8LLi91+NUU+MzsA9GxUjMaJBUl6Ko766j9MvD+cnKfHzsNSTEqZd5oDABi0U2zAeGqp2C4AEHvHfO2Rf9/2U8W2Mskx4oif8nvQZrbAyFPiverji4A35+U2mHZ/Q2ygeuLV/eP3NonVu299J9779u4sNlJVxa9vbkNUS0fx11PeNgt2HsCDo7lTExARUX61uxS9TQWlduIzf/58TJkyBV999RX8/f3z3eIqT7eQKqI+TaqVfOfg+UBiFHDsq9IFYWAsDlku1/P73MQnb+PFvMmHSZ4eRU6+wJjzYkNGea8Q+1dJ0dhLYuNGeTISMF18AOJcTGdWiAmOYnRR5LaFMDQG2qvR5VjVcPLuzcUHERHpJbUTn+7dxQZPvXr1giRPlZwgCJBIJJBKpZqLTg9JSlrNmRgtjoFTEr5vid1qpZlig7gG74r3eP9bAdQMEHswqeohUKOV2ODOroZYNTrnJZASk9ul1kRFuy9Hn4J7qdTukvsr440xuV1K7VTUJBEREZWA2olP3lGcqZzIzgBWtRK7ChfHF4nAzd3A9qHi6x7fivNQPbue2/bF0AhoN7nw45hYigO3Gb6aqdnAoHjjiBSHlTNQp4c4T5amjklERHpP7cSnQ4cO2oiDSkomK96kdnJdX41gXL834P1YHENEPrR4Sbosvj5mh6ZIJMCAzdo5NhER6a0Sd2FJS0tDZGQksrKylJazp1cZ26mikW7/34HIs8C5PPNUQSI2Um4+IneRqbX4ICIi0hNqJz7Pnz/H8OHDceCA6onX2MZHPTKZWuNH5oq6CPwaqHpdvV6AtVtu4jM7ruINikZERKQFak9SOmHCBCQkJOD8+fMwNzfHwYMHsXHjRvj4+GDPnj3aiLFSyylJ4pMQWXDSI1e9mTicevelTHqIiIheUbvG59ixY/j777/RrFkzGBgYwMPDA2+++SZsbGywaNEi9OjRQxtxVlo5MlnxNxYE4Eu74m0rkeTOo0JEREQASlDjk5qaCmdnceZje3t7PH8uzjbs5+eHkJAQzUanB7KluTU+24qao+vEksLXV28hjrZJREREKqld41OnTh2EhYXB09MTjRo1wpo1a+Dp6YnVq1fDzc1NGzFWajnS3Bqf5p4qJi2UO7cKOL6w4PXv/po7YjERERGppHbiM378eDx9+hQAMHfuXHTt2hWbNm2CiYkJNmzYoOn4Kr3nKZkAAAMJYFDYPF0Hp6te/vZKcZBB2+qaD46IiKiSUTvxGTx4sOK5v78/IiIicOfOHdSoUQOOjo4aDU4fdF1+CgBQaBvn1wcmNLUFunwF+PYQ56MiIiKiYlE78Xn48CFq1qypeG1hYYGmTVXMdE2akZUGLK6hvGzSTY6/Q0REVAJqJz7e3t6oXr06OnTogICAAHTo0AHe3t7aiI0AYNNr7XaaDGbSQ0REVEJq9+qKiorCokWLYG5ujiVLlqB27dqoXr06Bg0ahF9++UUbMVZaD56nFL6BNAeIOKO87O2V2guIiIioklM78alWrRoGDRqEtWvXIiwsDGFhYQgMDMSff/6JkSNHaiPGSutlalbBK0M3A4uqKS9TNUM6ERERFZvat7rS0tJw+vRpHD9+HMePH8eVK1fg6+uLsWPHIiAgQAshVl57rz0teOXu0cqvfbpoNxgiIiI9oHbiY2dnB3t7ewwaNAjTp09Hu3btYG9vr43YKr2MbDXmNRvEgQmJiIhKS+3Ep3v37jh9+jS2bt2KmJgYxMTEICAgALVr19ZGfJWaRFLIuD1ERESkcWq38dm9ezfi4uJw8OBBtGrVCv/++y/atWunaPtDxce8h4iIqGypXeMj5+fnh5ycHGRlZSEjIwOHDh3Ctm3bsGnTJk3GV6k1qGqrekVOZtkGQkREpCfUrvFZtmwZevXqhSpVqqBly5bYsmULateujZ07dyomLKXiiUspIMHZ8UHucyMzYNTpsgmIiIioklO7xmfLli3o0KEDPv74Y7Rr1w62tgXUWlCRlh2+q3rFnb25z6fcBcz4HhMREWmC2onPxYsXtREHAcCRL4HTy5SXmXCUZiIiIk1R+1YXAJw6dQqDBw9Gq1atEB0dDQD4/fffcfq09m/JrFy5Ep6enjAzM0PLli1x4cIFrZ+zzLye9ACAQYk+IiIiIlJB7avqzp07ERQUBHNzc1y5cgWZmWI7lcTERCxcuFDjAea1bds2TJo0CXPnzkVISAgaNWqEoKAgxMbGavW8ZcEGqfkX9l1f9oEQERFVYmonPvPnz8fq1avx888/w9jYWLG8TZs2CAkJ0Whwr1u2bBlGjBiB4cOHo169eli9ejUsLCywbt06rZ63LMw1/k15wSfngAbv6CYYIiKiSkrtxCcsLAzt27fPt9zW1hYJCQmaiEmlrKwsXL58GYGBgYplBgYGCAwMxNmzZ1Xuk5mZiaSkJKVHeVRLEo13DU8pL3Suq5tgiIiIKjG1Ex9XV1fcv38/3/LTp0+jZs2aGglKlbi4OEilUri4uCgtd3FxQUxMjMp9Fi1aBFtbW8XD3d1da/GVhGcVCwDAUdOpyism3NBBNERERJWf2onPiBEjMH78eJw/fx4SiQRPnjzBpk2bMGXKFIwePbroA5ShGTNmIDExUfGIiooq2wBUDUQoCMDLCODmbhxP7Y1ws4HK6y2dALvylaARERFVFmp3Z58+fTpkMhk6d+6MtLQ0tG/fHqamppgyZQo+/fRTbcQIAHB0dIShoSGePXumtPzZs2dwdXVVuY+pqSlMTU21FpMSaTYgkwJGpsD51YDEADjwGWBqA7z1HeDXV9zm8Fzg3MqCj9NmfNnES0REpIckgiAIJdkxKysL9+/fR0pKCurVqwcrKyukp6fD3Nxc0zEqtGzZEi1atMAPP/wAAJDJZKhRowbGjh2L6dOnF7l/UlISbG1tkZiYCBsbG80FlpMFzHcq/XECZgAdpnESLyIiojw0ef0u8SAxJiYmqFevHlq0aAFjY2MsW7YMXl5epQqmKJMmTcLPP/+MjRs34vbt2xg9ejRSU1MxfPhwrZ63SGvyN/YukYDpTHqIiIi0qNi3ujIzM/HFF1/g8OHDMDExwWeffYbevXtj/fr1mDlzJgwNDTFx4kRtxor33nsPz58/x5w5cxATE4PGjRvj4MGD+Ro8l7nnt0u029acAMzO+QCb6l1Ei66c2Z6IiEjbin2ra9q0aVizZg0CAwPx33//4fnz5xg+fDjOnTuHzz//HP369YOhoaG24y0Vrd3q+kLNubSc60M66gxqfb5f3L1nPQxro93aMiIioopKk9fvYtf4bN++Hb/99ht69eqFGzduoGHDhsjJycHVq1ch0ffbM3NeAlnJgLEFIMgASAAjk9z1oVsAaxfAsz1gKL7l0hyZYrWJUflOGImIiCqLYic+jx8/hr+/PwCgQYMGMDU1xcSJE5n0AOJ8WoXNoN54QL5FR2/n9k4zNuR7SEREVBaK3bhZKpXCxCS3FsPIyAhWVlZaCaqyS8nMwehNudN7dPR11mE0RERE+qPYNT6CIGDYsGGKcXEyMjIwatQoWFpaKm23a9cuzUZYCZ1/+ELptZWp2sMpERERUQkU+4o7dOhQpdeDBw/WeDD64sONl5Re824hERFR2Sh24rN+/XptxqHXjAxKPJwSERERqYFX3HLA0IBVPkRERGWBiQ8RERHpDSY+REREpDeY+BAREZHeYOJTxoo5QwgRERFpAROfMrb4wB1dh0BERKS3mPiUsTUnH+o6BCIiIr3FxIeIiIj0BhMfHdv0UUtdh0BERKQ3mPjo0NJ+jdDG21HXYRAREekNJj5lSCpT7tHVwtNBR5EQERHpJyY+Zejj35QnJ+UUXURERGWLl94ydPROrNJrztFFRERUtpj4lJEcqUzXIRAREek9Jj5lZMmhsHzLZBzEmYiIqEwx8Skja1UMXOhqY6aDSIiIiPQXEx8deadpNbbxISIiKmNMfHTEhbU9REREZY6JTxlpVbOK0uvG7na6CYSIiEiPMfEpI3VcrRXPjQ0l6FLPRYfREBER6ScmPmXk1tMkxfP+zdwhkbB9DxERUVlj4lNGLjyKVzw3YNJDRESkE0x8dEAAB/AhIiLSBSY+OsBBnImIiHSDiY8O9PBz03UIREREeomJTxlr5+OItj6Oug6DiIhILzHxKQPSPJNyNalhr8NIiIiI9BsTnzLg98UhxfNsNvAhIiLSGSY+WnY54iXSsqSK17WcrHQYDRERkX5j4qNl7676T+l1r0ZVdRQJERERMfEpYyZGfMuJiIh0hVdhIiIi0htMfIiIiEhvMPEhIiIivcHEh4iIiPQGEx8iIiLSG0x8iIiISG8w8SEiIiK9YaTrACqruJRMrAy+r+swiIiIKA8mPlrSbP4Rpde1Xazw9bsNdRQNERERAUx8yswXPetzZnYionJMKpUiOztb12HoJWNjYxgaGpbJuZj4lBFDA4muQyAiIhUEQUBMTAwSEhJ0HYpes7Ozg6urKyQS7V4vmfho2L5rT7H1YmS+5UaGbEdORFQeyZMeZ2dnWFhYaP3CS8oEQUBaWhpiY2MBAG5ublo9HxMfDRuzOUTlciPW+BARlTtSqVSR9FSpUkXX4egtc3NzAEBsbCycnZ21etuL1RBlxMiQiQ8RUXkjb9NjYWGh40hI/hlou50VE58yYmTAt5qIqLzi7S3dK6vPgFfjMsIaHyIi0lfHjx+HRCIpFw3ImfiUEbbxISIi0j0mPhqSlpWDvqv+K3A9e3UREZE2ZWVl6TqEchFDUXg11pAtF6JwKeJlgetZ40NERJoUEBCAsWPHYsKECXB0dERQUBBu3LiBbt26wcrKCi4uLhgyZAji4uIAAHv37oWdnR2kUikAIDQ0FBKJBNOnT1cc86OPPsLgwYMBAC9evMCAAQNQrVo1WFhYwM/PD1u2bCkyBgDYv38/ateuDXNzc3Ts2BHh4eFl8I4UDxMfDcnIlha6nokPEVHFIAgC0rJydPIQBEGtWDdu3AgTExOcOXMGixcvRqdOndCkSRNcunQJBw8exLNnz9C/f38AQLt27ZCcnIwrV64AAE6cOAFHR0ccP35ccbwTJ04gICAAAJCRkQF/f3/s27cPN27cwMcff4whQ4bgwoULBcawevVqREVF4Z133kHPnj0RGhqKjz76SCm50jWO46Mhp+/FFbqevbqIiCqG9Gwp6s05pJNz35oXBAuT4l+afXx8sGTJEgDA/Pnz0aRJEyxcuFCxft26dXB3d8fdu3dRu3ZtNG7cGMePH0ezZs1w/PhxTJw4EV9++SVSUlKQmJiI+/fvo0OHDgCAatWqYcqUKYpjffrppzh06BD+/PNPtGjRQmUMAPD555+jVq1a+PbbbwEAderUwfXr1/H111+X7E3RMF6NNeTswxeFrmevLiIi0jR/f3/F86tXryI4OBhWVlaKh6+vLwDgwYMHAIAOHTrg+PHjEAQBp06dwjvvvIO6devi9OnTOHHiBKpWrQofHx8A4uCOX331Ffz8/ODg4AArKyscOnQIkZGRBcYAALdv30bLli2VlrVq1UrjZS8p1vhoQHGqJk2NmGMSEVUE5saGuDUvSGfnVoelpaXieUpKCnr27KmyZkU+DURAQADWrVuHq1evwtjYGL6+vggICMDx48fx8uVLRW0PAHzzzTdYsWIFli9fDj8/P1haWmLChAn5GjDnjaEiYOKjAecexqtcPjqgFg7fegZbc2P26iIiqiAkEolat5vKi6ZNm2Lnzp3w9PSEkZHq+OXtfL777jtFkhMQEIDFixfj5cuXmDx5smLbM2fO4O2331Y0dpbJZLh79y7q1atXaBx169bFnj17lJadO3euNEXTqApxNQ4PD8eHH34ILy8vmJubo1atWpg7d2656TZ3PTpB5XJna1McmtAeO0aVnyo+IiKqnMaMGYP4+HgMGDAAFy9exIMHD3Do0CEMHz5c0ZPL3t4eDRs2xKZNmxSNmNu3b4+QkBDcvXtXqcbHx8cHhw8fxn///Yfbt29j5MiRePbsWZFxjBo1Cvfu3cPUqVMRFhaGzZs3Y8OGDdoocolUiMTnzp07kMlkWLNmDW7evInvvvsOq1evxueff67r0AAAUln+ZW28q+B/zWvA0EDCodCJiEjrqlatijNnzkAqlaJLly7w8/PDhAkTYGdnB4M8HWw6dOgAqVSqSHwcHBxQr149uLq6ok6dOortZs2ahaZNmyIoKAgBAQFwdXVF7969i4yjRo0a2LlzJ3bv3o1GjRph9erVSg2udU0iqNt3rpz45ptvsGrVKjx8+LDY+yQlJcHW1haJiYmwsbHRWCwrg+/jm0NhSsvCF/fQ2PGJiEg7MjIy8OjRI3h5ecHMzEzX4ei1wj4LTV6/K95NzFcSExPh4OBQ6DaZmZnIzMxUvE5KStJKLBU0dyQiItI7FeJW1+vu37+PH374ASNHjix0u0WLFsHW1lbxcHd310o8VaxMtXJcIiIi0iydJj7Tp0+HRCIp9HHnzh2lfaKjo9G1a1f069cPI0aMKPT4M2bMQGJiouIRFRWllXI4MfEhIiKqEHR6q2vy5MkYNmxYodvUrFlT8fzJkyfo2LEjWrdujbVr1xZ5fFNTU5iaaj8pkfJWFxERUYWg08THyckJTk5Oxdo2OjoaHTt2hL+/P9avX6/UQl3X2MaHiIioYqgQjZujo6MREBAADw8PLF26FM+fP1esc3V11WFkIlXd2YmIiKj8qRCJz+HDh3H//n3cv38f1atXV1pXHmpbZOUgBiIiIipa+blfVIhhw4ZBEASVj/KAiQ8REVHFUCESn/JOKmPiQ0REVBEw8dEA5j1ERFTWAgICMGHCBJXrhg0bVqzpJfQREx8NkDHzISIiqhAqROPm8m76rmuK57bmxpgY6KPDaIiIiKggrPHRgLwVPldmv4lhbbx0FwwREemlffv2wdbWFps2bdJ1KOUaa3w0zMBAousQiIioNAQByE7TzbmNLQCJ+teRzZs3Y9SoUdi8eTPeeustHD58WAvBVQ5MfIiIiPLKTgMWVtXNuT9/AphYqrXLypUrMXPmTPzzzz/o0KGDlgKrPJj4EBERVVA7duxAbGwszpw5g+bNm+s6nAqBiQ8REVFexhZizYuuzq2GJk2aICQkBOvWrUOzZs0gKcFtMn3DxIeIiCgviUTt2026UqtWLXz77bcICAiAoaEhfvzxR12HVO4x8dEAiURsC0dERFTWateujeDgYAQEBMDIyAjLly/XdUjlGhMfDfjl/Wb44p+b+LZfY12HQkREeqhOnTo4duyYouaHCsbERwM613VB57ouug6DiIj0yPHjx5Ve161bF8+ePdNNMBUIBzAkIiIivcHEh4iIiPQGEx8iIiLSG0x8iIiISG8w8SEiIiK9wcSHiIj0nsDB2HSurD4DJj5ERKS3jI2NAQBpaTqajZ0U5J+B/DPRFo7jQ0REesvQ0BB2dnaIjY0FAFhYWHC+qzImCALS0tIQGxsLOzs7rQ/AyMSHiIj0mqurKwAokh/SDTs7O8VnoU1MfIiISK9JJBK4ubnB2dkZ2dnZug5HLxkbG5fZVBtMfIiIiCDe9uI8V5UfGzcTERGR3mDiQ0RERHqDiQ8RERHpDb1q4yMfHCkpKUnHkRAREVFxya/bmhjkUK8Sn+TkZACAu7u7jiMhIiIidSUnJ8PW1rZUx5AIejROt0wmw5MnT2Btba3RAaqSkpLg7u6OqKgo2NjYaOy45Y0+lFMfygjoRzn1oYyAfpRTH8oI6Ec5S1pGQRCQnJyMqlWrwsCgdK109KrGx8DAANWrV9fa8W1sbCrtH2te+lBOfSgjoB/l1IcyAvpRTn0oI6Af5SxJGUtb0yPHxs1ERESkN5j4EBERkd5g4qMBpqammDt3LkxNTXUdilbpQzn1oYyAfpRTH8oI6Ec59aGMgH6UszyUUa8aNxMREZF+Y40PERER6Q0mPkRERKQ3mPgQERGR3mDiQ0RERHqDiY8GrFy5Ep6enjAzM0PLli1x4cIFXYek0qJFi9C8eXNYW1vD2dkZvXv3RlhYmNI2GRkZGDNmDKpUqQIrKyu8++67ePbsmdI2kZGR6NGjBywsLODs7IypU6ciJydHaZvjx4+jadOmMDU1hbe3NzZs2KDt4qm0ePFiSCQSTJgwQbGsspQxOjoagwcPRpUqVWBubg4/Pz9cunRJsV4QBMyZMwdubm4wNzdHYGAg7t27p3SM+Ph4DBo0CDY2NrCzs8OHH36IlJQUpW2uXbuGdu3awczMDO7u7liyZEmZlA8ApFIpZs+eDS8vL5ibm6NWrVr46quvlObrqWjlPHnyJHr27ImqVatCIpFg9+7dSuvLsjzbt2+Hr68vzMzM4Ofnh/3795dJObOzszFt2jT4+fnB0tISVatWxfvvv48nT55UqHIW9VnmNWrUKEgkEixfvlxpeXkvI1C8ct6+fRu9evWCra0tLC0t0bx5c0RGRirWl6vvXYFKZevWrYKJiYmwbt064ebNm8KIESMEOzs74dmzZ7oOLZ+goCBh/fr1wo0bN4TQ0FChe/fuQo0aNYSUlBTFNqNGjRLc3d2Fo0ePCpcuXRLeeOMNoXXr1or1OTk5QoMGDYTAwEDhypUrwv79+wVHR0dhxowZim0ePnwoWFhYCJMmTRJu3bol/PDDD4KhoaFw8ODBMi3vhQsXBE9PT6Fhw4bC+PHjFcsrQxnj4+MFDw8PYdiwYcL58+eFhw8fCocOHRLu37+v2Gbx4sWCra2tsHv3buHq1atCr169BC8vLyE9PV2xTdeuXYVGjRoJ586dE06dOiV4e3sLAwYMUKxPTEwUXFxchEGDBgk3btwQtmzZIpibmwtr1qwpk3IuWLBAqFKlirB3717h0aNHwvbt2wUrKythxYoVFbac+/fvF2bOnCns2rVLACD89ddfSuvLqjxnzpwRDA0NhSVLlgi3bt0SZs2aJRgbGwvXr1/XejkTEhKEwMBAYdu2bcKdO3eEs2fPCi1atBD8/f2VjlHey1nUZym3a9cuoVGjRkLVqlWF7777rkKVsTjlvH//vuDg4CBMnTpVCAkJEe7fvy/8/fffStfB8vS9y8SnlFq0aCGMGTNG8VoqlQpVq1YVFi1apMOoiic2NlYAIJw4cUIQBPHLyNjYWNi+fbtim9u3bwsAhLNnzwqCIP4HMDAwEGJiYhTbrFq1SrCxsREyMzMFQRCEzz77TKhfv77Sud577z0hKChI20VSSE5OFnx8fITDhw8LHTp0UCQ+laWM06ZNE9q2bVvgeplMJri6ugrffPONYllCQoJgamoqbNmyRRAEQbh165YAQLh48aJimwMHDggSiUSIjo4WBEEQfvrpJ8He3l5Rbvm569Spo+kiqdSjRw/hgw8+UFr2zjvvCIMGDRIEoeKX8/WLSFmWp3///kKPHj2U4mnZsqUwcuRIjZZREPKXU5ULFy4IAISIiAhBECpeOQsq4+PHj4Vq1aoJN27cEDw8PJQSn4pWRkFQXc733ntPGDx4cIH7lLfvXd7qKoWsrCxcvnwZgYGBimUGBgYIDAzE2bNndRhZ8SQmJgIAHBwcAACXL19Gdna2Unl8fX1Ro0YNRXnOnj0LPz8/uLi4KLYJCgpCUlISbt68qdgm7zHk25TlezJmzBj06NEjXxyVpYx79uxBs2bN0K9fPzg7O6NJkyb4+eefFesfPXqEmJgYpRhtbW3RsmVLpXLa2dmhWbNmim0CAwNhYGCA8+fPK7Zp3749TExMFNsEBQUhLCwML1++1HYx0bp1axw9ehR3794FAFy9ehWnT59Gt27dKlU55cqyPLr+G35dYmIiJBIJ7OzsAFSOcspkMgwZMgRTp05F/fr1862vLGXct28fateujaCgIDg7O6Nly5ZKt8PK2/cuE59SiIuLg1QqVfqgAMDFxQUxMTE6iqp4ZDIZJkyYgDZt2qBBgwYAgJiYGJiYmCi+eOTylicmJkZleeXrCtsmKSkJ6enp2iiOkq1btyIkJASLFi3Kt66ylPHhw4dYtWoVfHx8cOjQIYwePRrjxo3Dxo0bleIs7G8zJiYGzs7OSuuNjIzg4OCg1nuhTdOnT8f//vc/+Pr6wtjYGE2aNMGECRMwaNAgpRgqejnlyrI8BW2ji++ujIwMTJs2DQMGDFBMXFkZyvn111/DyMgI48aNU7m+MpQxNjYWKSkpWLx4Mbp27Yp///0Xffr0wTvvvIMTJ04o4itP37t6NTs75RozZgxu3LiB06dP6zoUjYqKisL48eNx+PBhmJmZ6TocrZHJZGjWrBkWLlwIAGjSpAlu3LiB1atXY+jQoTqOTnP+/PNPbNq0CZs3b0b9+vURGhqKCRMmoGrVqpWqnPosOzsb/fv3hyAIWLVqla7D0ZjLly9jxYoVCAkJgUQi0XU4WiOTyQAAb7/9NiZOnAgAaNy4Mf777z+sXr0aHTp00GV4KrHGpxQcHR1haGiYr2X6s2fP4OrqqqOoijZ27Fjs3bsXwcHBqF69umK5q6srsrKykJCQoLR93vK4urqqLK98XWHb2NjYwNzcXNPFUXL58mXExsaiadOmMDIygpGREU6cOIHvv/8eRkZGcHFxqfBlBAA3NzfUq1dPaVndunUVvSjkcRb2t+nq6orY2Fil9Tk5OYiPj1frvdCmqVOnKmp9/Pz8MGTIEEycOFFRm1dZyilXluUpaJuyLK886YmIiMDhw4cVtT3y+CpyOU+dOoXY2FjUqFFD8V0UERGByZMnw9PTUxFbRS4jIF4HjYyMivw+Kk/fu0x8SsHExAT+/v44evSoYplMJsPRo0fRqlUrHUammiAIGDt2LP766y8cO3YMXl5eSuv9/f1hbGysVJ6wsDBERkYqytOqVStcv35d6T+r/AtL/offqlUrpWPItymL96Rz5864fv06QkNDFY9mzZph0KBBiucVvYwA0KZNm3xDEdy9exceHh4AAC8vL7i6uirFmJSUhPPnzyuVMyEhAZcvX1Zsc+zYMchkMrRs2VKxzcmTJ5Gdna3Y5vDhw6hTpw7s7e21Vj65tLQ0GBgof00ZGhoqfmVWlnLKlWV5dP03LE967t27hyNHjqBKlSpK6yt6OYcMGYJr164pfRdVrVoVU6dOxaFDhypFGQHxOti8efNCv4/K3bVFrabQlM/WrVsFU1NTYcOGDcKtW7eEjz/+WLCzs1NqmV5ejB49WrC1tRWOHz8uPH36VPFIS0tTbDNq1CihRo0awrFjx4RLly4JrVq1Elq1aqVYL+9y2KVLFyE0NFQ4ePCg4OTkpLLL4dSpU4Xbt28LK1eu1El3drm8vboEoXKU8cKFC4KRkZGwYMEC4d69e8KmTZsECwsL4Y8//lBss3jxYsHOzk74+++/hWvXrglvv/22ym7RTZo0Ec6fPy+cPn1a8PHxUepKm5CQILi4uAhDhgwRbty4IWzdulWwsLAos+7sQ4cOFapVq6bozr5r1y7B0dFR+OyzzypsOZOTk4UrV64IV65cEQAIy5YtE65cuaLozVRW5Tlz5oxgZGQkLF26VLh9+7Ywd+5cjXaBLqycWVlZQq9evYTq1asLoaGhSt9HeXsvlfdyFvVZvu71Xl0VoYzFKeeuXbsEY2NjYe3atcK9e/cU3cxPnTqlOEZ5+t5l4qMBP/zwg1CjRg3BxMREaNGihXDu3Dldh6QSAJWP9evXK7ZJT08XPvnkE8He3l6wsLAQ+vTpIzx9+lTpOOHh4UK3bt0Ec3NzwdHRUZg8ebKQnZ2ttE1wcLDQuHFjwcTERKhZs6bSOcra64lPZSnjP//8IzRo0EAwNTUVfH19hbVr1yqtl8lkwuzZswUXFxfB1NRU6Ny5sxAWFqa0zYsXL4QBAwYIVlZWgo2NjTB8+HAhOTlZaZurV68Kbdu2FUxNTYVq1aoJixcv1nrZ5JKSkoTx48cLNWrUEMzMzISaNWsKM2fOVLo4VrRyBgcHq/x/OHTo0DIvz59//inUrl1bMDExEerXry/s27evTMr56NGjAr+PgoODK0w5i/osX6cq8SnvZRSE4pXz119/Fby9vQUzMzOhUaNGwu7du5WOUZ6+dyWCkGcIVCIiIqJKjG18iIiISG8w8SEiIiK9wcSHiIiI9AYTHyIiItIbTHyIiIhIbzDxISIiIr3BxIeIiIj0BhMfIio3PD09sXz5cl2HQUSVGBMfIlKbRCIp9PHFF1+U6LgXL17Exx9/rNlg8wgICMCECRO0dnwiKv+MdB0AEVU8T58+VTzftm0b5syZozRJoZWVleK5IAiQSqUwMir668bJyUmzgRIRvYY1PkSkNldXV8XD1tYWEolE8frOnTuwtrbGgQMH4O/vD1NTU5w+fRoPHjzA22+/DRcXF1hZWaF58+Y4cuSI0nFfv9UlkUjwyy+/oE+fPrCwsICPjw/27NlTaGw//fQTfHx8YGZmBhcXF/Tt2xcAMGzYMJw4cQIrVqxQ1EyFh4cDAG7cuIFu3brBysoKLi4uGDJkCOLi4hTHDAgIwNixYzF27FjY2trC0dERs2fPRt4Zfwo6LxGVL0x8iEgrpk+fjsWLF+P27dto2LAhUlJS0L17dxw9ehRXrlxB165d0bNnT0RGRhZ6nC+//BL9+/fHtWvX0L17dwwaNAjx8fEqt7106RLGjRuHefPmISwsDAcPHkT79u0BACtWrECrVq0wYsQIPH36FE+fPoW7uzsSEhLQqVMnNGnSBJcuXcLBgwfx7Nkz9O/fX+nYGzduhJGRES5cuIAVK1Zg2bJl+OWXX4o8LxGVM2pPa0pElMf69esFW1tbxWv5TM6vz86sSv369YUffvhB8fr12asBCLNmzVK8TklJEQAIBw4cUHm8nTt3CjY2NkJSUpLK9R06dBDGjx+vtOyrr74SunTporQsKipKAKCYFb1Dhw5C3bp1BZlMpthm2rRpQt26dYt1XiIqP1jjQ0Ra0axZM6XXKSkpmDJlCurWrQs7OztYWVnh9u3bRdb4NGzYUPHc0tISNjY2iI2NVbntm2++CQ8PD9SsWRNDhgzBpk2bkJaWVujxr169iuDgYFhZWSkevr6+AIAHDx4otnvjjTcgkUgUr1u1aoV79+5BKpWW6LxEpBtMfIhIKywtLZVeT5kyBX/99RcWLlyIU6dOITQ0FH5+fsjKyir0OMbGxkqvJRIJZDKZym2tra0REhKCLVu2wM3NDXPmzEGjRo2QkJBQ4PFTUlLQs2dPhIaGKj3u3btX7NtVJTkvEekGEx8iKhNnzpzBsGHD0KdPH/j5+cHV1VXRuFiTjIyMEBgYiCVLluDatWsIDw/HsWPHAAAmJiaQSqVK2zdt2hQ3b96Ep6cnvL29lR55k7fz588r7Xfu3Dn4+PjA0NCwyPMSUfnBxIeIyoSPjw927dqF0NBQXL16FQMHDiyw5qak9u7di++//x6hoaGIiIjAb7/9BplMhjp16gAQe42dP38e4eHhiIuLg0wmw5gxYxAfH48BAwbg4sWLePDgAQ4dOoThw4crJUmRkZGYNGkSwsLCsGXLFvzwww8YP358sc5LROUHx/EhojKxbNkyfPDBB2jdujUcHR0xbdo0JCUlafQcdnZ22LVrF7744gtkZGTAx8cHW7ZsQf369QGIt9uGDh2KevXqIT09HY8ePYKnpyfOnDmDadOmoUuXLsjMzISHhwe6du0KA4Pc34bvv/8+0tPT0aJFCxgaGmL8+PGKwRaLOi8RlR8SQcgzEAUREeUTEBCAxo0bczoNokqAt7qIiIhIbzDxISIiIr3BW11ERESkN1jjQ0RERHqDiQ8RERHpDSY+REREpDeY+BAREZHeYOJDREREeoOJDxEREekNJj5ERESkN5j4EBERkd5g4kNERER64//fbTwJYCuhzAAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "from matplotlib import pyplot as plt\n", + "import numpy as np\n", + "\n", + "def moving_average(a, n=10):\n", + " ret = np.cumsum(a, dtype=float)\n", + " ret[n:] = ret[n:] - ret[:-n]\n", + " return ret[n - 1:] / n\n", + "\n", + "plt.plot(moving_average(rew_list, 20), label=\"reward\")\n", + "plt.plot(moving_average(kl_list, 20), label=\"kl\")\n", + "plt.legend()\n", + "plt.title(f\"Reward plot || KL beta = {kl_beta}\")\n", + "plt.xlabel(\"Train steps\")\n", + "plt.ylabel(\"Reward (moving average)\")\n", + "if save:\n", + " plt.savefig(f\"{folder}/plot.png\")\n", + "else:\n", + " plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "1465c909", + "metadata": {}, + "source": [ + "With our hyperparameters, we have maximized reward while maintaining a non-divergent KL. Looking at the outputs above, the model seems to behave the way we want it to. \n", + "\n", + "Hurray!" + ] + }, + { + "cell_type": "markdown", + "id": "6a6226b8", + "metadata": {}, + "source": [ + "## Explainability / Interpretability" + ] + }, + { + "cell_type": "markdown", + "id": "d70c9c26", + "metadata": {}, + "source": [ + "First I run some visualizations on the base LLM to set the stage. " + ] + }, + { + "cell_type": "markdown", + "id": "4877fde5", + "metadata": {}, + "source": [ + "### Base model's visualizations" + ] + }, + { + "cell_type": "markdown", + "id": "9da71d36", + "metadata": {}, + "source": [ + "Let's look at some probability plots over the output tokens for different input lengths. " + ] + }, + { + "cell_type": "code", + "execution_count": 124, + "id": "66501500", + "metadata": {}, + "outputs": [], + "source": [ + "x, y = torch.Tensor([[5, 7, 1, 5, 10]]).long(), torch.Tensor([[6, 2, 7, 3]]).long()" + ] + }, + { + "cell_type": "code", + "execution_count": 125, + "id": "244f1fda", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Text(0, 0.5, 'Probability')" + ] + }, + "execution_count": 125, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAABNEAAAEpCAYAAAC9VbStAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA4LUlEQVR4nO3de1xUdf7H8TcXGUQDLyheItGKNfOOaVimbfyictvMas0skVr7ZbCZVKalopWhXVzX8tJFrTVZ3dzuulqR2K+flOWtLC+5avozQUwFgwSD8/vDh1MTA2cYZjxz4PV8PObxcL6c853PmWXeG5/5nnOCDMMwBAAAAAAAAKBawVYXAAAAAAAAAAQ6mmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaPC7qVOnKigoSEFBQWratKnV5aCWjh8/7vzfLygoSM8884zVJQF1QibBnWbNmjl/L9LT060uBw0ImYSzZciQIc7fta5du1pdDgIQeQQ7u//++8/K7y9NNBt45ZVXFBQUpC+++MLqUiRJpaWlmjp1qnJzc2u135IlS7Rw4UKXsVGjRrk0aM48Onfu7HV9cXFxbucMCgrShRde6NWchw4d0oQJE3TllVfqnHPOUVBQUK2P/7f27dtXbZ3Lli3zet758+frlltu0XnnnaegoCCNGjWq2m2PHz+uu+++W61atVKTJk105ZVXatOmTS7bNGnSREuWLNFf//pXr2tC/UIm1Q6Z5NtM8samTZv0xz/+US1atFBERIS6du2qOXPmuGzz4osvasmSJXV+LZx9ZFLtkEn2yCRP7dixQ+PHj1fPnj11zjnnqG3btho8eLDbz8O4ceO0ZMmSOv3+oGbkUe2QRw03jzyVm5tb7XsfFBSk6dOnO7e94447tGTJEg0YMMDr1/NEqF9nR71UWlqqadOmSZIGDRrk8X63336723GHw6GXX37ZZSwqKsrr+mbPnq0ff/zRZey7777TpEmTdPXVV3s1586dOzVz5kxdeOGF6tatm/Ly8ryu77eGDx+u6667zmUsMTHR6/lmzpypEydOqG/fvjp06FC121VWVmrw4MHaunWrHnroIUVHR2vevHkaNGiQNm7c6Pw/rkaNGun222/Xvn37NG7cOK/rAvyFTGpYmVRb77//vq6//nr16tVLkydPVtOmTfWf//xH//d//+ey3Z/+9CdJp/8DDKgLMolMqomnmeSpl19+WQsXLtRNN92ke++9V0VFRXrhhRd06aWXavXq1UpKSnJuO3DgQOc+R44c8er1YC/kEXlUEyvzyFMXXXSR2y85lyxZovfff9/l9yQhIUEJCQn68MMPfdJgrA5NNFguNDS02qD2xpAhQ6qMPfHEE5KkESNGeDVnQkKCfvjhB7Vo0UIrVqzQLbfcUpcSXfTu3dunx79u3Trntxk1LWNdsWKF1q9fr9dff10333yzpNN/RMbHxyszM1PZ2dk+qwmwEzKp/mRScXGxRo4cqcGDB2vFihUKDmYBPuyHTCKTajJ8+HBNnTrV5VjuvPNOXXTRRZo6dapXf7QC1SGPyKOa+COPYmJi3L7n06ZN04UXXqhLLrmkTjV7g/+atKlRo0apadOmOnjwoIYMGaKmTZuqVatWevDBB1VRUeHc7szS02eeeUZ//etf1aFDBzVu3FgDBw7Utm3bXOYcNGiQ228oRo0apbi4OOd8rVq1knT6F/fMMsqpU6fW6XgqKipUXFxcpzlqkp2drY4dO6p///5e7X/OOeeoRYsWPq7qFyUlJSovL/fJXB06dFBQUJDpditWrFBMTIyGDh3qHGvVqpX+9Kc/6e2331ZZWZlP6kHDQCbVDplUlT8yKTs7WwUFBZo+fbqCg4NVUlKiysrKWs8D+yGTaodMqsoumZSQkFDlj++WLVtqwIAB2r59e53mhm+QR7VDHlVFHrnasGGDdu/e7XWjta5ootlYRUWFkpOT1bJlSz3zzDMaOHCgnn32Wb344otVtv373/+uOXPmKC0tTRMnTtS2bdv0+9//XgUFBbV6zVatWmn+/PmSpBtvvFFLlizRkiVLXD7QtVVaWqrIyEhFRUWpRYsWSktLq7Ksty42b96s7du367bbbvPZnL40bdo0NW3aVOHh4brkkkv0/vvvn5XX3bx5s3r37l3lW4e+ffuqtLRUu3btOit1oP4gkzxDJrnnj0z68MMPFRkZqYMHD+p3v/udmjZtqsjISI0ZM0YnT570VekIUGSSZ8gk9+yeSfn5+YqOjvbpnPAeeeQZ8sg98sjV0qVLJXm/WrGuOJ3Txk6ePKlhw4Zp8uTJkqR77rlHvXv31sKFCzVmzBiXbXfv3q1vv/1W7du3lyRdc8016tevn2bOnKlZs2Z5/JpNmjTRzTffrDFjxqh79+51Xs7atm1bjR8/Xr1791ZlZaVWr16tefPmaevWrcrNzVVoaN1/Ra3+kFUnODhYV199tW688Ua1b99ee/bs0axZs3TttdfqnXfe0eDBg/36+ocOHdIVV1xRZbxt27aSpO+//17dunXzaw2oX8gkz5BJ7vkjk7799lv9/PPPuuGGG3TXXXcpKytLubm5eu6553T8+HH94x//8EntCExkkmfIJPfsnEn/8z//o7y8PE2aNMkn86HuyCPPkEfukUe/qKio0PLly9W3b19dcMEFPpmztmii2dw999zj8nzAgAFuL7w3ZMgQZxBLp7vW/fr106pVq2oVxr6WlZXl8vzWW29VfHy8Hn30Ua1YsUK33nprneavrKzUsmXL1KtXL1100UV1msvXzjvvPK1Zs8Zl7I477lCXLl30wAMP+D2Mf/rpJzkcjirj4eHhzp8DtUUm1YxMqp4/MunHH39UaWmp7rnnHuedpoYOHary8nK98MILeuyxx7y+GC/sgUyqGZlUPbtm0uHDh3XbbbepY8eOGj9+fJ3mgm+RRzUjj6pHHv0iJydHBQUFeuSRR3wynzc4ndPGwsPDnee5n9G8eXMdO3asyrbuPgDx8fHat2+fv8rz2rhx4xQcHKwPP/ywznOtW7dOBw8eDLhvM6rTokULpaamaufOnV7fFcVTjRs3dnv+/Jnlu40bN/br66P+IZPMkUnV80cmndln+PDhLuNnThPx5R27EHjIJHNkUvXsmEklJSX6wx/+oBMnTujtt9+u8ULlOLvII3PkUfXIo18sXbpUISEhGjZsmE/m8wZNNBsLCQnx6XzVXdTw1xe8PBsaN26sli1b6ujRo3Wea+nSpQoODq4SDoEsNjZWknxy/DVp27at21spnxlr166dX18f9Q+ZZI5Mqp4/MunMPjExMS7jrVu3liS3f7yg/iCTzJFJ1bNbJpWXl2vo0KH68ssv9fbbb6tr165ezwXfI4/MkUfVI49O++mnn/Tmm28qKSmpSt1nE020BuLbb7+tMrZr1y7n3Vuk09+GHD9+vMp23333nctzT+4gUhcnTpzQkSNHqnxbU1tlZWX617/+pUGDBtmqIbRnzx5JqvPxm+nZs6c2bdpU5S4sn332mSIiIhQfH+/X10fDRiaRSb/lj0xKSEiQJB08eNBl/Pvvv5fk/2OCfZBJZNJv2SmTKisrNXLkSOXk5Cg7O1sDBw70ah4EBvKIPPot8ui0d955RydOnLB8tSJNtAbirbfecvmAbNiwQZ999pmuvfZa59j555+vHTt2qLCw0Dm2detW/e///q/LXBEREZLkNrhr4+TJkzpx4kSV8ccff1yGYeiaa66p0/yrVq3S8ePHLf+QVefX7/MZBw8e1KJFi9S9e3fnhSL95eabb1ZBQYHeeOMN59iRI0f0+uuv6/rrr3d73j3gK2RS4KmPmfSnP/1JkrRw4UKX8ZdfflmhoaEaNGhQnWpG/UEmBR4yyXN/+ctftHz5cs2bN69Od15EYCCPAg955Dl/5lF2drYiIiJ04403+nTe2uLGAg3EBRdcoMsvv1xjxoxRWVmZZs+erZYtW7pc4O/OO+/UrFmzlJycrLvuukuHDx/WggULdPHFF6u4uNi5XePGjdWlSxctX75c8fHxatGihbp27VrrZZr5+fnq1auXhg8frs6dO0uS1qxZo1WrVumaa67RDTfc4LL9mW9fPL0ewNKlS+VwOHTTTTdVu82gQYO0bt06GYZhOt8TTzwhSfr6668lSUuWLNEnn3wiSS53G5k6daqmTZumtWvX1hg+48eP13/+8x9dddVVateunfbt26cXXnhBJSUl+tvf/uay7SuvvKLU1FQtXrxYo0aNqrHOd999V1u3bpUknTp1Sl9++aWz9j/+8Y/q3r27pNNhfOmllyo1NVXffPONoqOjNW/ePFVUVGjatGmm7wdQF2SSe2SSZ5k0atQovfrqq9q7d6/LN/O/1atXL915551atGiRfv75Zw0cOFC5ubl6/fXXNXHiRFt92w3/IpPcI5OsyyRP36fZs2dr3rx5SkxMVEREhF577TWXn994441q0qRJje8JAgt55B55VL/yKDc3V1deeaUyMzM1derUGt8n6fQps//+97910003WX+9RwMBb/HixYYk4/PPP3eOpaSkGE2aNKmybWZmpvHr/1n37t1rSDKefvpp49lnnzViY2MNh8NhDBgwwNi6dWuV/V977TWjU6dORlhYmNGzZ09jzZo1RkpKitGhQweX7davX28kJCQYYWFhhiQjMzOz2vp/W9MZx44dM26//XbjggsuMCIiIgyHw2FcfPHFxpNPPmmUl5dX2T46Otq49NJLq32dXysqKjLCw8ONoUOH1rhdQkKC0aZNG4/mlFTt49ceeOABIygoyNi+fXuN82VnZxtXXHGF0apVKyM0NNSIjo42brzxRmPjxo1Vtn3uuecMScbq1atN60xJSam2zsWLF7tse/ToUeOuu+4yWrZsaURERBgDBw50+T37tV//LqFhI5NOI5Osy6SbbrrJaNy4sXHs2DHT1y8vLzemTp1qdOjQwWjUqJFxwQUXGH/961+r3V6SkZaWZjovAgeZdBqZVL8yydP3qabjkWTs3bu3yj4DBw40Lr74YtNaUXvk0WnkEXlklkfvvvuuIclYsGCBaZ2GYRgLFiwwJBnvvPOO6bbVfeZ8hSZaPRcIjY8zYVxYWGgcOXLEqzm+/vprQ5Lx3nvv+ayu4uJiIzQ01Hj++ed9NqdhGMYll1xi3HzzzT6d85ZbbjEuueQSn87pqcrKSqOwsNDYtGmT5b9LsD8yqXpkkudat25tPPjggz6d84cffjAKCwtpojUwZFL1yCTP+SOT/PE+FRcXG4WFhUb//v1pogUg8qh65JHn7JJHDz30kHHuuecaJ0+e9NmcP/74o1FYWGjceuutfm2icTonzppWrVqpSZMm+vHHH2u979q1a5WYmKjBgwf7rJ6PP/5Y7du31+jRo302Z3FxsbZu3apXX33VZ3MahqHc3Nwqy2HPlqKiIi6+jXqJTPKO1Zn09ddf66efftLDDz/s03k7deqkoqIin84J1AaZ5J36mEn+eJ8k6Y477tDbb78tSbr44ot9OjfqF/LIO+SR59auXavJkyf79Drcjz76qPP0Wn+exh5kGB6cVAzb2rdvnzp27Kinn35aDz74oCU17Nmzx3nnEi7kbD8///yzcnNznc/j4+N13nnnWVcQbI1MQqBat26dTp06Jen0Let/97vfWVwRzgYyCQ3Jl19+qcOHD0uSmjZtqksvvdTiivBr5BFQN7t27dL+/fsl+ff3l5Vo8LtOnTqpU6dOVpcBL4WGhiopKcnqMgCfIZPgji9vwQ7UBpmEs+XMRcqB6pBHsLP4+HjFx8f7/XUsXYn28ccf6+mnn9bGjRt16NAhvfnmmxoyZEiN++Tm5iojI0Nff/21YmNjNWnSJNO7XgAAAAAAAAB1EWzli5eUlKhHjx6aO3euR9vv3btXgwcP1pVXXqktW7bo/vvv15///GetWbPGz5UCAAAAAACgIQuYa6IFBQWZrkR7+OGHtXLlSm3bts05duutt+r48eNavXr1WagSAAAAAAAADZGtromWl5dX5dpMycnJuv/++6vdp6ysTGVlZc7nlZWVOnr0qFq2bKmgoCB/lQogQBiGoRMnTqhdu3YKDrZ08W2tkV9Aw0V2AbAru+YX2QU0bJ5ml62aaPn5+YqJiXEZi4mJUXFxsX766Sc1bty4yj5ZWVmaNm3a2SoRQIA6cOCAzj33XKvLqBXyCwDZBcCu7JZfZBcAyTy7bHU6Z3x8vFJTUzVx4kTn2KpVqzR48GCVlpa6baL99huFoqIinXfeeTpw4IAiIyN9egwAAk9xcbFiY2N1/PhxRUVFWV1OrZBfQMNFdgGwK7vmF9kFNGyeZpetVqK1adNGBQUFLmMFBQWKjIx020CTJIfDIYfDUWU8MjKSMAQaEDsuwye/AJBdAOzKbvlFdgGQzLPLPiepS0pMTFROTo7L2AcffKDExESLKgIAAAAAAEBDYGkT7ccff9SWLVu0ZcsWSdLevXu1ZcsW7d+/X5I0ceJEjRw50rn9Pffcoz179mj8+PHasWOH5s2bp3/+858aN26cFeUDAAAAAACggbC0ifbFF1+oV69e6tWrlyQpIyNDvXr10pQpUyRJhw4dcjbUJKljx45auXKlPvjgA/Xo0UPPPvusXn75ZSUnJ1tSPwAAAAAAABoGS6+JNmjQINV0X4NXXnnF7T6bN2/2Y1UAAAAAAACAK1tdEw0AAAAAAACwAk00AAAAAAAAwARNNAAAAAAAAMAETTQAAAAAAADABE00AAAAAAAAwARNNAAAAAAAAMAETTQAAAAAAADABE00AAAAAAAAwARNNAAAAAAAAMAETTQAAAAAAADABE00AAAAAAAAwARNNAAAAAAAAMAETTQAAAAAAADABE00AAAAAAAAwARNNAAAAAAAAMAETTQAAAAAAADABE00AAAAAAAAwARNNAAAAAAAAMAETTQAAAAAAADABE00AAAAAAAAwARNNAAAAAAAAMAETTQAAAAAAADABE00AAAAAAAAwARNNAAAAAAAAMAETTQAAAAAAADABE00AAAAAAAAwARNNAAAAAAAAMAETTQAAAAAAADABE00AAAAAAAAwARNNAAAAAAAAMAETTQAAAAAAADABE00AAAAAAAAwARNNAAAAAAAAMAETTQAAAAAAADABE00AAAAAAAAwITlTbS5c+cqLi5O4eHh6tevnzZs2FDj9rNnz9bvfvc7NW7cWLGxsRo3bpxOnjx5lqoFAAAAAABAQ2RpE2358uXKyMhQZmamNm3apB49eig5OVmHDx92u312drYmTJigzMxMbd++XQsXLtTy5cv1yCOPnOXKAQAAAAAA0JBY2kSbNWuWRo8erdTUVHXp0kULFixQRESEFi1a5Hb79evX67LLLtNtt92muLg4XX311Ro+fLjp6jUAAAAAAACgLixropWXl2vjxo1KSkr6pZjgYCUlJSkvL8/tPv3799fGjRudTbM9e/Zo1apVuu6666p9nbKyMhUXF7s8AMAOyC8AdkR2AbAjsguAJyxroh05ckQVFRWKiYlxGY+JiVF+fr7bfW677TY99thjuvzyy9WoUSOdf/75GjRoUI2nc2ZlZSkqKsr5iI2N9elxAIC/kF8A7IjsAmBHZBcAT1h+Y4HayM3N1ZNPPql58+Zp06ZNeuONN7Ry5Uo9/vjj1e4zceJEFRUVOR8HDhw4ixUDgPfILwB2RHYBsCOyC4AnQq164ejoaIWEhKigoMBlvKCgQG3atHG7z+TJk3XHHXfoz3/+sySpW7duKikp0d13361HH31UwcFVe4IOh0MOh8P3BwAAfkZ+AbAjsguAHZFdADxh2Uq0sLAwJSQkKCcnxzlWWVmpnJwcJSYmut2ntLS0SqMsJCREkmQYhv+KBQAAAAAAQINm2Uo0ScrIyFBKSor69Omjvn37avbs2SopKVFqaqokaeTIkWrfvr2ysrIkSddff71mzZqlXr16qV+/ftq9e7cmT56s66+/3tlMAwAAAAAAAHzN0ibasGHDVFhYqClTpig/P189e/bU6tWrnTcb2L9/v8vKs0mTJikoKEiTJk3SwYMH1apVK11//fWaPn26VYcAAAAAAACABiDIaGDnQRYXFysqKkpFRUWKjIy0uhwAflafPvP16VgA1Kw+fd7r07EAMFdfPvP15TgAeMbTz7yt7s4JAAAAAAAAWIEmGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGDCqyba2rVrfV0HAAAAAAAAELC8aqJdc801Ov/88/XEE0/owIEDvq4JAAAAAAAACCheNdEOHjyo9PR0rVixQp06dVJycrL++c9/qry83Nf1AQAAAAAAAJbzqokWHR2tcePGacuWLfrss88UHx+ve++9V+3atdN9992nrVu3+rpOAAAAAAAAwDJ1vrFA7969NXHiRKWnp+vHH3/UokWLlJCQoAEDBujrr7/2RY0AAAAAAACApbxuop06dUorVqzQddddpw4dOmjNmjV6/vnnVVBQoN27d6tDhw665ZZbfFkrAAAAAAAAYAmvmmh/+ctf1LZtW/33f/+34uPjtXnzZuXl5enPf/6zmjRpori4OD3zzDPasWOH6Vxz585VXFycwsPD1a9fP23YsKHG7Y8fP660tDS1bdtWDodD8fHxWrVqlTeHAQAAAAAAAHgk1JudvvnmGz333HMaOnSoHA6H222io6O1du3aGudZvny5MjIytGDBAvXr10+zZ89WcnKydu7cqdatW1fZvry8XP/1X/+l1q1ba8WKFWrfvr2+++47NWvWzJvDAAAAAAAAADziVRMtMzNT/fv3V2io6+4///yz1q9fryuuuEKhoaEaOHBgjfPMmjVLo0ePVmpqqiRpwYIFWrlypRYtWqQJEyZU2X7RokU6evSo1q9fr0aNGkmS4uLivDkEAAAAAAAAwGNenc555ZVX6ujRo1XGi4qKdOWVV3o0R3l5uTZu3KikpKRfigkOVlJSkvLy8tzu88477ygxMVFpaWmKiYlR165d9eSTT6qiosKbwwAAAAAAAAA84tVKNMMwFBQUVGX8hx9+UJMmTTya48iRI6qoqFBMTIzLeExMTLXXUtuzZ48++ugjjRgxQqtWrdLu3bt177336tSpU8rMzHS7T1lZmcrKypzPi4uLPaoPAKxGfgGwI7ILgB2RXQA8Uasm2tChQyVJQUFBGjVqlMv10CoqKvTll1+qf//+vq3wVyorK9W6dWu9+OKLCgkJUUJCgg4ePKinn3662iZaVlaWpk2b5reaAMBfyC8AdkR2AbAjsguAJ2p1OmdUVJSioqJkGIbOOecc5/OoqCi1adNGd999t1577TWP5oqOjlZISIgKCgpcxgsKCtSmTRu3+7Rt21bx8fEKCQlxjl100UXKz89XeXm5230mTpyooqIi5+PAgQMeHi0AWIv8AmBHZBcAOyK7AHiiVivRFi9eLOn0xfwffPBBj0/ddCcsLEwJCQnKycnRkCFDJJ1eaZaTk6P09HS3+1x22WXKzs5WZWWlgoNP9/927dqltm3bKiwszO0+Doej2juIAkAgI78A2BHZBcCOyC4AnvDqxgKZmZl1aqCdkZGRoZdeekmvvvqqtm/frjFjxqikpMR5t86RI0dq4sSJzu3HjBmjo0ePauzYsdq1a5dWrlypJ598UmlpaXWuBQAAAAAAAKiOxyvRevfurZycHDVv3ly9evVye2OBMzZt2uTRnMOGDVNhYaGmTJmi/Px89ezZU6tXr3bebGD//v3OFWeSFBsbqzVr1mjcuHHq3r272rdvr7Fjx+rhhx/29DAAAAAAAACAWvO4iXbDDTc4l7eeOf3SF9LT06s9fTM3N7fKWGJioj799FOfvT4AAAAAAABgxuMm2q/vflndnTABAAAAAACA+sira6IBAAAAAAAADYnHK9GaN29e43XQfu3o0aNeFwQAAAAAAAAEGo+baLNnz/ZjGQAAAAAAAEDg8riJlpKS4s86AAAAAAAAgIDlcROtuLhYkZGRzn/X5Mx2AAAAAAAAQH1Qq2uiHTp0SK1bt1azZs3cXh/NMAwFBQWpoqLCp0UCAAAAAAAAVvK4ifbRRx+pRYsWkqS1a9f6rSAAAAAAAAAg0HjcRBs4cKDbfwMAAAAAAAD1ncdNtN86duyYFi5cqO3bt0uSunTpotTUVOdqNQAAAAAAAKC+CPZmp48//lhxcXGaM2eOjh07pmPHjmnOnDnq2LGjPv74Y1/XCAAAAAAAAFjKq5VoaWlpGjZsmObPn6+QkBBJUkVFhe69916lpaXpq6++8mmRAAAAAAAAgJW8Wom2e/duPfDAA84GmiSFhIQoIyNDu3fv9llxAAAAAAAAQCDwqonWu3dv57XQfm379u3q0aNHnYsCAAAAAAAAAonHp3N++eWXzn/fd999Gjt2rHbv3q1LL71UkvTpp59q7ty5mjFjhu+rBAAAAAAAACzkcROtZ8+eCgoKkmEYzrHx48dX2e62227TsGHDfFMdAAAAAAAAEAA8bqLt3bvXn3UAAAAAAAAAAcvjJlqHDh38WQcAAAAAAAAQsDxuornzzTffaP/+/SovL3cZ/+Mf/1inogAAAAAAAIBA4lUTbc+ePbrxxhv11VdfuVwnLSgoSJJUUVHhuwoBAAAAAAAAiwV7s9PYsWPVsWNHHT58WBEREfr666/18ccfq0+fPsrNzfVxiQAAAAAAAIC1vFqJlpeXp48++kjR0dEKDg5WcHCwLr/8cmVlZem+++7T5s2bfV0nAAAAAAAAYBmvVqJVVFTonHPOkSRFR0fr+++/l3T65gM7d+70XXUAAAAAAABAAPBqJVrXrl21detWdezYUf369dNTTz2lsLAwvfjii+rUqZOvawQAAAAAAAAs5VUTbdKkSSopKZEkPfbYY/rDH/6gAQMGqGXLllq+fLlPCwQAAAAAAACs5lUTLTk52fnvCy64QDt27NDRo0fVvHlz5x06AQAAAAAAgPrCqybarx04cECSFBsbW+diAAAAAAAAgEDk1Y0Ffv75Z02ePFlRUVGKi4tTXFycoqKiNGnSJJ06dcrXNQIAAAAAAACW8mol2l/+8he98cYbeuqpp5SYmChJysvL09SpU/XDDz9o/vz5Pi0SAAAAAAAAsJJXTbTs7GwtW7ZM1157rXOse/fuio2N1fDhw2miAQAAAAAAoF7x6nROh8OhuLi4KuMdO3ZUWFhYXWsCAAAAAAAAAopXTbT09HQ9/vjjKisrc46VlZVp+vTpSk9P91lxAAAAAAAAQCDw+HTOoUOHujz/8MMPde6556pHjx6SpK1bt6q8vFxXXXWVbysELBQ3YWWd59g3Y7APKgEAAAAAAFbyuIkWFRXl8vymm25yeR4bG+ubigAAAAAAAIAA43ETbfHixf6sAwAAAAAAAAhYXl0T7YzCwkJ98skn+uSTT1RYWOj1PHPnzlVcXJzCw8PVr18/bdiwwaP9li1bpqCgIA0ZMsTr1wYAAAAAAADMeNVEKykp0Z133qm2bdvqiiuu0BVXXKF27drprrvuUmlpaa3mWr58uTIyMpSZmalNmzapR48eSk5O1uHDh2vcb9++fXrwwQc1YMAAbw4BAAAAAAAA8JhXTbSMjAytW7dO7777ro4fP67jx4/r7bff1rp16/TAAw/Uaq5Zs2Zp9OjRSk1NVZcuXbRgwQJFRERo0aJF1e5TUVGhESNGaNq0aerUqZM3hwAAAAAAAAB4zKsm2r/+9S8tXLhQ1157rSIjIxUZGanrrrtOL730klasWOHxPOXl5dq4caOSkpJ+KSg4WElJScrLy6t2v8cee0ytW7fWXXfd5U35AAAAAAAAQK14fGOBXystLVVMTEyV8datW9fqdM4jR46ooqKiylwxMTHasWOH230++eQTLVy4UFu2bPHoNcrKylRWVuZ8Xlxc7HF9AGAl8guAHZFdAOyI7ALgCa9WoiUmJiozM1MnT550jv3000+aNm2aEhMTfVbcb504cUJ33HGHXnrpJUVHR3u0T1ZWlqKiopyP2NhYv9UHAL5EfgGwI7ILgB2RXQA8EWQYhlHbnb766itdc801KisrU48ePSRJW7duVXh4uNasWaOLL77Yo3nKy8sVERGhFStWuNxhMyUlxXmdtV/bsmWLevXqpZCQEOdYZWWlpNOnge7cuVPnn3++yz7uvlGIjY1VUVGRIiMja3XcaHjiJqys8xz7Zgz2QSXwVnFxsaKiomz5mSe/gIaL7AJgV3bNL7ILaNg8zS6vTufs1q2bvv32Wy1dutR52uXw4cM1YsQINW7c2ON5wsLClJCQoJycHGcTrbKyUjk5OUpPT6+yfefOnfXVV1+5jE2aNEknTpzQ3/72N7ffFjgcDjkcjlocHQAEBvILgB2RXQDsiOwC4IlaN9FOnTqlzp0767333tPo0aPrXEBGRoZSUlLUp08f9e3bV7Nnz1ZJSYlSU1MlSSNHjlT79u2VlZWl8PBwde3a1WX/Zs2aSVKVcQAAAAAAAMBXat1Ea9Sokcu10Opq2LBhKiws1JQpU5Sfn6+ePXtq9erVzpsN7N+/X8HBXl26DQAAAAAAAPAJr07nTEtL08yZM/Xyyy8rNNSrKVykp6e7PX1TknJzc2vc95VXXqnz6wMAAAAAAAA18aoD9vnnnysnJ0fvv/++unXrpiZNmrj8/I033vBJcQAAAAAAAEAg8KqJ1qxZM910002+rgUAAAAAAAAISLVqolVWVurpp5/Wrl27VF5ert///veaOnVqre7ICQAAAAAAANhNra7YP336dD3yyCNq2rSp2rdvrzlz5igtLc1ftQEAAAAAAAABoVZNtL///e+aN2+e1qxZo7feekvvvvuuli5dqsrKSn/VBwAAAAAAAFiuVk20/fv367rrrnM+T0pKUlBQkL7//nufFwYAAAAAAAAEilpdE+3nn39WeHi4y1ijRo106tQpnxYFAAAAAADsIW7CyjrPsW/GYB9UAvhXrZpohmFo1KhRcjgczrGTJ0/qnnvuUZMmTZxjb7zxhu8qBAAAAAAAACxWqyZaSkpKlbHbb7/dZ8UAAAAAAAAAgahWTbTFixf7qw4AAAAAAAAgYNWqiYbAxnnoAAAAAAAA/lGru3MCAAAAAAAADRFNNAAAAAAAAMAETTQAAAAAAADABNdEAwAAAAIE17gFACBwsRINAAAAAAAAMEETDQAAAAAAADBBEw0AAAAAAAAwQRMNAAAAAAAAMEETDQAAAAAAADBBEw0AAAAAAAAwQRMNAAAAAAAAMEETDQAAAAAAADBBEw0AAAAAAAAwEWp1AQAAxE1YWec59s0Y7INKAAAAAMA9VqIBAAAAAAAAJliJBgBAPcTqPgAAAMC3WIkGAAAAAAAAmGAlGgAAAIB6i5W5AABfYSUaAAAAAAAAYIImGgAAAAAAAGCC0zkBAIBHOCUKAAAADRkr0QAAAAAAAAATrEQD6glWiAAAAADA2eePv8X4+y4wsRINAAAAAAAAMMFKNAAAYBm+ZYWd8fsLAEDDEhAr0ebOnau4uDiFh4erX79+2rBhQ7XbvvTSSxowYICaN2+u5s2bKykpqcbtAQAAAAAAgLqyvIm2fPlyZWRkKDMzU5s2bVKPHj2UnJysw4cPu90+NzdXw4cP19q1a5WXl6fY2FhdffXVOnjw4FmuHAAAAAAAAA2F5U20WbNmafTo0UpNTVWXLl20YMECRUREaNGiRW63X7p0qe6991717NlTnTt31ssvv6zKykrl5OSc5coBAAAAAADQUFh6TbTy8nJt3LhREydOdI4FBwcrKSlJeXl5Hs1RWlqqU6dOqUWLFm5/XlZWprKyMufz4uLiuhUNAGcJ+QXAjsgu1AXXmYNVyC4AnrB0JdqRI0dUUVGhmJgYl/GYmBjl5+d7NMfDDz+sdu3aKSkpye3Ps7KyFBUV5XzExsbWuW4AOBvILwB2RHYBsCOyC4AnLD+dsy5mzJihZcuW6c0331R4eLjbbSZOnKiioiLn48CBA2e5SgDwDvkFwI7ILgB2RHYB8ISlp3NGR0crJCREBQUFLuMFBQVq06ZNjfs+88wzmjFjhj788EN179692u0cDoccDodP6gWAs4n8AmBHZBcAOyK7AHjC0pVoYWFhSkhIcLkpwJmbBCQmJla731NPPaXHH39cq1evVp8+fc5GqQAAAAAAAGjALF2JJkkZGRlKSUlRnz591LdvX82ePVslJSVKTU2VJI0cOVLt27dXVlaWJGnmzJmaMmWKsrOzFRcX57x2WtOmTdW0aVPLjgMAAAAAACCQcQOXurG8iTZs2DAVFhZqypQpys/PV8+ePbV69WrnzQb279+v4OBfFszNnz9f5eXluvnmm13myczM1NSpU89m6QAAAAAAAGggLG+iSVJ6errS09Pd/iw3N9fl+b59+/xfEAAAAAAAAPArtr47JwAAAAAAAHA20EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATAREE23u3LmKi4tTeHi4+vXrpw0bNtS4/euvv67OnTsrPDxc3bp106pVq85SpQAAAAAAAGiILG+iLV++XBkZGcrMzNSmTZvUo0cPJScn6/Dhw263X79+vYYPH6677rpLmzdv1pAhQzRkyBBt27btLFcOAAAAAACAhsLyJtqsWbM0evRopaamqkuXLlqwYIEiIiK0aNEit9v/7W9/0zXXXKOHHnpIF110kR5//HH17t1bzz///FmuHAAAAAAAAA1FqJUvXl5ero0bN2rixInOseDgYCUlJSkvL8/tPnl5ecrIyHAZS05O1ltvveV2+7KyMpWVlTmfFxUVSZKKi4vrWH3gqSwrrfMc9fF9qQs7vad2qvVsOnNMhmFYXEntkV+1Ux/fl7rwx3tqlznrA7LLHvhM+J5d3tOumWvqPOe2acl1niMQ2TW/GlJ2+QPZZY/s8ue8dudxdhkWOnjwoCHJWL9+vcv4Qw89ZPTt29ftPo0aNTKys7NdxubOnWu0bt3a7faZmZmGJB48eDTwx4EDB3wTXGcR+cWDBw+yiwcPHnZ92C2/yC4ePHhI5tkVZBjWfUXw/fffq3379lq/fr0SExOd4+PHj9e6dev02WefVdknLCxMr776qoYPH+4cmzdvnqZNm6aCgoIq2//2G4XKykodPXpULVu2VFBQUJ2Pobi4WLGxsTpw4IAiIyPrPJ8/2aVWu9Qp2adWu9Qp+b5WwzB04sQJtWvXTsHBlp/BXivk12l2qVOyT612qVOyT61k1y/IrtPsUqdkn1rtUqfUsGu1a36RXb+wS612qVOyT612qVOyLrssPZ0zOjpaISEhVZpfBQUFatOmjdt92rRpU6vtHQ6HHA6Hy1izZs28L7oakZGRAf9LdoZdarVLnZJ9arVLnZJva42KivLJPGcb+eXKLnVK9qnVLnVK9qmV7CK7fssudUr2qdUudUoNt1Y75hfZVZVdarVLnZJ9arVLndLZzy5LvxoICwtTQkKCcnJynGOVlZXKyclxWZn2a4mJiS7bS9IHH3xQ7fYAAAAAAABAXVm6Ek2SMjIylJKSoj59+qhv376aPXu2SkpKlJqaKkkaOXKk2rdvr6ysLEnS2LFjNXDgQD377LMaPHiwli1bpi+++EIvvviilYcBAAAAAACAeszyJtqwYcNUWFioKVOmKD8/Xz179tTq1asVExMjSdq/f7/L+aj9+/dXdna2Jk2apEceeUQXXnih3nrrLXXt2tWS+h0OhzIzM6ss/Q1EdqnVLnVK9qnVLnVK9qrV7uzyXtulTsk+tdqlTsk+tdqlzvrALu+1XeqU7FOrXeqUqBVV2el9tkutdqlTsk+tdqlTsq5WS28sAAAAAAAAANiBfW6XAgAAAAAAAFiEJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJlodzZ07V3FxcQoPD1e/fv20YcMGq0tykZWVpUsuuUTnnHOOWrdurSFDhmjnzp1Wl+WRGTNmKCgoSPfff7/VpVRx8OBB3X777WrZsqUaN26sbt266YsvvrC6rCoqKio0efJkdezYUY0bN9b555+vxx9/XIFwP5GPP/5Y119/vdq1a6egoCC99dZbLj83DENTpkxR27Zt1bhxYyUlJenbb7+1pth6KNCzS7JvfgVydkn2yC+yC9Uhu/yH7PKNQM0vsst6gZ5fZJd/kF11F2j5RROtDpYvX66MjAxlZmZq06ZN6tGjh5KTk3X48GGrS3Nat26d0tLS9Omnn+qDDz7QqVOndPXVV6ukpMTq0mr0+eef64UXXlD37t2tLqWKY8eO6bLLLlOjRo3073//W998842effZZNW/e3OrSqpg5c6bmz5+v559/Xtu3b9fMmTP11FNP6bnnnrO6NJWUlKhHjx6aO3eu258/9dRTmjNnjhYsWKDPPvtMTZo0UXJysk6ePHmWK61/7JBdkj3zK5CzS7JPfpFdcIfs8h+yy3cCNb/ILmvZIb/ILt8ju3wj4PLLgNf69u1rpKWlOZ9XVFQY7dq1M7KysiysqmaHDx82JBnr1q2zupRqnThxwrjwwguNDz74wBg4cKAxduxYq0ty8fDDDxuXX3651WV4ZPDgwcadd97pMjZ06FBjxIgRFlXkniTjzTffdD6vrKw02rRpYzz99NPOsePHjxsOh8P4xz/+YUGF9Ysds8swAj+/Aj27DMM++UV2wR2yyz/ILt+yQ36RXWefHfOL7Ko7ssv3AiG/WInmpfLycm3cuFFJSUnOseDgYCUlJSkvL8/CympWVFQkSWrRooXFlVQvLS1NgwcPdnlvA8k777yjPn366JZbblHr1q3Vq1cvvfTSS1aX5Vb//v2Vk5OjXbt2SZK2bt2qTz75RNdee63FldVs7969ys/Pd/kdiIqKUr9+/QL682UHds0uKfDzK9CzS7JPfpFd+C2yy3/ILt+yY36RXf5l1/wiu+qO7PI/K/Ir1C+zNgBHjhxRRUWFYmJiXMZjYmK0Y8cOi6qqWWVlpe6//35ddtll6tq1q9XluLVs2TJt2rRJn3/+udWlVGvPnj2aP3++MjIy9Mgjj+jzzz/Xfffdp7CwMKWkpFhdnosJEyaouLhYnTt3VkhIiCoqKjR9+nSNGDHC6tJqlJ+fL0luP19nfgbv2DG7pMDPLztkl2Sf/CK78Ftkl3+QXb5nx/wiu/zLjvlFdvkG2eV/VuQXTbQGJC0tTdu2bdMnn3xidSluHThwQGPHjtUHH3yg8PBwq8upVmVlpfr06aMnn3xSktSrVy9t27ZNCxYsCLgw/Oc//6mlS5cqOztbF198sbZs2aL7779f7dq1C7hagZoEcn7ZJbsk++QX2YX6guzyDbtkl0R+oX4gu3yD7KqfOJ3TS9HR0QoJCVFBQYHLeEFBgdq0aWNRVdVLT0/Xe++9p7Vr1+rcc8+1uhy3Nm7cqMOHD6t3794KDQ1VaGio1q1bpzlz5ig0NFQVFRVWlyhJatu2rbp06eIydtFFF2n//v0WVVS9hx56SBMmTNCtt96qbt266Y477tC4ceOUlZVldWk1OvMZssvny07sll1S4OeXXbJLsk9+kV34LbLL98gu/7BjfpFd/mW3/CK7fIfs8j8r8osmmpfCwsKUkJCgnJwc51hlZaVycnKUmJhoYWWuDMNQenq63nzzTX300Ufq2LGj1SVV66qrrtJXX32lLVu2OB99+vTRiBEjtGXLFoWEhFhdoiTpsssuq3K75127dqlDhw4WVVS90tJSBQe7fsxDQkJUWVlpUUWe6dixo9q0aePy+SouLtZnn30WUJ8vO7JLdkn2yS+7ZJdkn/wiu/BbZJfvkV3+Ycf8Irv8yy75RXb5Htnlf5bkl19uV9BALFu2zHA4HMYrr7xifPPNN8bdd99tNGvWzMjPz7e6NKcxY8YYUVFRRm5urnHo0CHno7S01OrSPBKId1rZsGGDERoaakyfPt349ttvjaVLlxoRERHGa6+9ZnVpVaSkpBjt27c33nvvPWPv3r3GG2+8YURHRxvjx4+3ujTjxIkTxubNm43NmzcbkoxZs2YZmzdvNr777jvDMAxjxowZRrNmzYy3337b+PLLL40bbrjB6Nixo/HTTz9ZXLn92SG7DMPe+RWI2WUY9skvsgvukF3+R3bVXaDmF9llLTvkF9nle2SXbwRaftFEq6PnnnvOOO+884ywsDCjb9++xqeffmp1SS4kuX0sXrzY6tI8EqiB+O677xpdu3Y1HA6H0blzZ+PFF1+0uiS3iouLjbFjxxrnnXeeER4ebnTq1Ml49NFHjbKyMqtLM9auXev2dzMlJcUwjNO3K548ebIRExNjOBwO46qrrjJ27txpbdH1SKBnl2HYO78CNbsMwx75RXahOmSXf5FddReo+UV2WS/Q84vs8g+yq+4CLb+CDMMwfL++DQAAAAAAAKg/uCYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABg4v8BPO64xNsNXXIAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "fig, axes = plt.subplots(1, 4, sharey=True, figsize=(15, 3))\n", + "for i in range(4):\n", + " _x = torch.concat([x, y[:, :i]], dim=1)\n", + " axes[i].bar(np.arange(11), torch.softmax((reference(_x)[0][0, -1].detach()), dim=0).tolist())\n", + " axes[i].title.set_text(f\"Input {_x[0].tolist()}\")\n", + " axes[i].set_ylim([0, 1])\n", + "axes[0].set_ylabel(\"Probability\")" + ] + }, + { + "cell_type": "markdown", + "id": "9ef20486", + "metadata": {}, + "source": [ + "The supervised model learns an almost equal probability over the increment of last 4 tokens. This was our intended behaviour. \n", + "\n", + "The exact probabilities vary but are mostly within a some threshold of each other. For the first plot though, since the input contains two 5's, the probability of 6 is much higher than others. " + ] + }, + { + "cell_type": "markdown", + "id": "6952efbe", + "metadata": {}, + "source": [ + "Now, let's look at what the attention heads focus on. I'm using [BertViz](https://github.com/jessevig/bertviz) for visualizing attention weights." + ] + }, + { + "cell_type": "markdown", + "id": "166f77b7", + "metadata": {}, + "source": [ + "![](../../../images/session_4/part_2_finetuning_lms_to_human_preferences/base_attn.png)" + ] + }, + { + "cell_type": "markdown", + "id": "cd112409", + "metadata": {}, + "source": [ + "Here also, the model successfully learns to look at the last 4 input tokens while ignoring the separation token." + ] + }, + { + "cell_type": "markdown", + "id": "7d3b08ab", + "metadata": {}, + "source": [ + "### Fine tuned model with beta = 1" + ] + }, + { + "cell_type": "markdown", + "id": "07d8f884", + "metadata": {}, + "source": [ + "We look at the same visualizations for the fine tuned model with $\\beta = 1$." + ] + }, + { + "cell_type": "code", + "execution_count": 126, + "id": "81e9ae26", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Text(0, 0.5, 'Probability')" + ] + }, + "execution_count": 126, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAABNEAAAEpCAYAAAC9VbStAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA4NUlEQVR4nO3de1xUdf7H8TcXGUATLyheItGKzLxjEpZpG7+o3Mqs1swSqbVfBptJZVoqWhnaxXUtL13UWpPVze2uqxWJ/fpJWd5K85Krpj8TxLxgkGBwfn/4cGpi4AzDjGeOvJ6Px3k8mi/nfOcz08x7t898zzlBhmEYAgAAAAAAAFCtYKsLAAAAAAAAAAIdTTQAAAAAAADABE00AAAAAAAAwARNNAAAAAAAAMAETTQAAAAAAADABE00AAAAAAAAwARNNAAAAAAAAMAETTQAAAAAAADABE00AAAAAAAAwARNNPjdpEmTFBQUpKCgIDVq1MjqclBLR48edf77CwoK0nPPPWd1SUCdkElwp0mTJs7PRUZGhtXloB4hk3CmDBw40PlZ69y5s9XlIACRR7CzBx988Ix8fmmi2cBrr72moKAgffXVV1aXIkkqLS3VpEmTlJeXV6vjFi5cqHnz5rmMDR8+3KVBc3rr2LGj1/XFxcW5nTMoKEgXXnihV3MeOHBAY8eO1VVXXaVzzjlHQUFBtX79v7dnz55q61y8eLHX886ZM0e33XabzjvvPAUFBWn48OHV7nv06FHde++9atGihRo2bKirrrpK69evd9mnYcOGWrhwof761796XRPOLmRS7ZBJvs0kb6xfv1433nijmjVrpsjISHXu3FkzZ8502efll1/WwoUL6/xcOPPIpNohk+yRSZ7atm2bxowZo+7du+ucc85R69atNWDAALffh9GjR2vhwoV1+vygZuRR7ZBH9TePPJWXl1ftex8UFKQpU6Y4973rrru0cOFC9e3b1+vn80SoX2fHWam0tFSTJ0+WJPXv39/j4+6880634w6HQ6+++qrLWFRUlNf1zZgxQz/99JPL2Pfff6/x48frmmuu8WrO7du3a9q0abrwwgvVpUsX5efne13f7w0ZMkTXX3+9y1hSUpLX802bNk3Hjx9X7969deDAgWr3q6ys1IABA7Rp0yY98sgjio6O1uzZs9W/f3+tW7fO+T9cDRo00J133qk9e/Zo9OjRXtcF+AuZVL8yqbY+/PBD3XDDDerRo4cmTJigRo0a6T//+Y/+7//+z2W/P/3pT5JO/R8woC7IJDKpJp5mkqdeffVVzZs3T7fccovuv/9+HTt2TC+99JIuu+wyrVixQsnJyc59+/Xr5zzm0KFDXj0f7IU8Io9qYmUeeeriiy92+yPnwoUL9eGHH7p8ThISEpSQkKCPP/7YJw3G6tBEg+VCQ0OrDWpvDBw4sMrYU089JUkaOnSoV3MmJCToxx9/VLNmzbR06VLddtttdSnRRc+ePX36+levXu38NaOmZaxLly7VmjVr9Oabb+rWW2+VdOo/IuPj45WVlaWcnByf1QTYCZl09mRScXGxhg0bpgEDBmjp0qUKDmYBPuyHTCKTajJkyBBNmjTJ5bXcfffduvjiizVp0iSv/qMVqA55RB7VxB95FBMT4/Y9nzx5si688EJdeumldarZG/y/SZsaPny4GjVqpP3792vgwIFq1KiRWrRooYcfflgVFRXO/U4vPX3uuef017/+Ve3atVNERIT69eunzZs3u8zZv39/t79QDB8+XHFxcc75WrRoIenUB/f0MspJkybV6fVUVFSouLi4TnPUJCcnR+3bt1efPn28Ov6cc85Rs2bNfFzVr0pKSlReXu6Tudq1a6egoCDT/ZYuXaqYmBgNGjTIOdaiRQv96U9/0rvvvquysjKf1IP6gUyqHTKpKn9kUk5OjgoLCzVlyhQFBwerpKRElZWVtZ4H9kMm1Q6ZVJVdMikhIaHKf3w3b95cffv21datW+s0N3yDPKod8qgq8sjV2rVrtXPnTq8brXVFE83GKioqlJKSoubNm+u5555Tv3799Pzzz+vll1+usu/f//53zZw5U+np6Ro3bpw2b96sP/zhDyosLKzVc7Zo0UJz5syRJN18881auHChFi5c6PKFrq3S0lI1btxYUVFRatasmdLT06ss662LDRs2aOvWrbrjjjt8NqcvTZ48WY0aNVJ4eLguvfRSffjhh2fkeTds2KCePXtW+dWhd+/eKi0t1Y4dO85IHTh7kEmeIZPc80cmffzxx2rcuLH279+viy66SI0aNVLjxo01cuRInThxwlelI0CRSZ4hk9yzeyYVFBQoOjrap3PCe+SRZ8gj98gjV4sWLZLk/WrFuuJ0Ths7ceKEBg8erAkTJkiS7rvvPvXs2VPz5s3TyJEjXfbduXOnvvvuO7Vt21aSdO211yoxMVHTpk3T9OnTPX7Ohg0b6tZbb9XIkSPVtWvXOi9nbd26tcaMGaOePXuqsrJSK1as0OzZs7Vp0ybl5eUpNLTuH1Grv2TVCQ4O1jXXXKObb75Zbdu21a5duzR9+nRdd911eu+99zRgwAC/Pv+BAwd05ZVXVhlv3bq1JOmHH35Qly5d/FoDzi5kkmfIJPf8kUnfffedfvnlF91000265557lJ2drby8PL3wwgs6evSo/vGPf/ikdgQmMskzZJJ7ds6k//mf/1F+fr7Gjx/vk/lQd+SRZ8gj98ijX1VUVGjJkiXq3bu3LrjgAp/MWVs00Wzuvvvuc3nct29ftxfeGzhwoDOIpVNd68TERC1fvrxWYexr2dnZLo9vv/12xcfH6/HHH9fSpUt1++2312n+yspKLV68WD169NDFF19cp7l87bzzztPKlStdxu666y516tRJDz30kN/D+Oeff5bD4agyHh4e7vw7UFtkUs3IpOr5I5N++uknlZaW6r777nPeaWrQoEEqLy/XSy+9pCeeeMLri/HCHsikmpFJ1bNrJh08eFB33HGH2rdvrzFjxtRpLvgWeVQz8qh65NGvcnNzVVhYqMcee8wn83mD0zltLDw83Hme+2lNmzbVkSNHquzr7gsQHx+vPXv2+Ks8r40ePVrBwcH6+OOP6zzX6tWrtX///oD7NaM6zZo1U1pamrZv3+71XVE8FRER4fb8+dPLdyMiIvz6/Dj7kEnmyKTq+SOTTh8zZMgQl/HTp4n48o5dCDxkkjkyqXp2zKSSkhL98Y9/1PHjx/Xuu+/WeKFynFnkkTnyqHrk0a8WLVqkkJAQDR482CfzeYMmmo2FhIT4dL7qLmr42wtengkRERFq3ry5Dh8+XOe5Fi1apODg4CrhEMhiY2MlySevvyatW7d2eyvl02Nt2rTx6/Pj7EMmmSOTquePTDp9TExMjMt4y5YtJcntf7zg7EEmmSOTqme3TCovL9egQYP09ddf691331Xnzp29ngu+Rx6ZI4+qRx6d8vPPP+vtt99WcnJylbrPJJpo9cR3331XZWzHjh3Ou7dIp34NOXr0aJX9vv/+e5fHntxBpC6OHz+uQ4cOVfm1prbKysr0r3/9S/3797dVQ2jXrl2SVOfXb6Z79+5av359lbuwfPHFF4qMjFR8fLxfnx/1G5lEJv2ePzIpISFBkrR//36X8R9++EGS/18T7INMIpN+z06ZVFlZqWHDhik3N1c5OTnq16+fV/MgMJBH5NHvkUenvPfeezp+/LjlqxVpotUT77zzjssXZO3atfriiy903XXXOcfOP/98bdu2TUVFRc6xTZs26X//939d5oqMjJQkt8FdGydOnNDx48erjD/55JMyDEPXXnttneZfvny5jh49avmXrDq/fZ9P279/v+bPn6+uXbs6LxTpL7feeqsKCwv11ltvOccOHTqkN998UzfccIPb8+4BXyGTAs/ZmEl/+tOfJEnz5s1zGX/11VcVGhqq/v3716lmnD3IpMBDJnnuL3/5i5YsWaLZs2fX6c6LCAzkUeAhjzznzzzKyclRZGSkbr75Zp/OW1vcWKCeuOCCC3TFFVdo5MiRKisr04wZM9S8eXOXC/zdfffdmj59ulJSUnTPPffo4MGDmjt3ri655BIVFxc794uIiFCnTp20ZMkSxcfHq1mzZurcuXOtl2kWFBSoR48eGjJkiDp27ChJWrlypZYvX65rr71WN910k8v+p3998fR6AIsWLZLD4dAtt9xS7T79+/fX6tWrZRiG6XxPPfWUJGnLli2SpIULF+qzzz6TJJe7jUyaNEmTJ0/WqlWragyfMWPG6D//+Y+uvvpqtWnTRnv27NFLL72kkpIS/e1vf3PZ97XXXlNaWpoWLFig4cOH11jn+++/r02bNkmSTp48qa+//tpZ+4033qiuXbtKOhXGl112mdLS0vTtt98qOjpas2fPVkVFhSZPnmz6fgB1QSa5RyZ5lknDhw/X66+/rt27d7v8Mv97PXr00N1336358+frl19+Ub9+/ZSXl6c333xT48aNs9Wv3fAvMsk9Msm6TPL0fZoxY4Zmz56tpKQkRUZG6o033nD5+80336yGDRvW+J4gsJBH7pFHZ1ce5eXl6aqrrlJWVpYmTZpU4/sknTpl9t///rduueUW66/3aCDgLViwwJBkfPnll86x1NRUo2HDhlX2zcrKMn77r3X37t2GJOPZZ581nn/+eSM2NtZwOBxG3759jU2bNlU5/o033jA6dOhghIWFGd27dzdWrlxppKamGu3atXPZb82aNUZCQoIRFhZmSDKysrKqrf/3NZ125MgR48477zQuuOACIzIy0nA4HMYll1xiPP3000Z5eXmV/aOjo43LLrus2uf5rWPHjhnh4eHGoEGDatwvISHBaNWqlUdzSqp2+62HHnrICAoKMrZu3VrjfDk5OcaVV15ptGjRwggNDTWio6ONm2++2Vi3bl2VfV944QVDkrFixQrTOlNTU6utc8GCBS77Hj582LjnnnuM5s2bG5GRkUa/fv1cPme/9dvPEuo3MukUMsm6TLrllluMiIgI48iRI6bPX15ebkyaNMlo166d0aBBA+OCCy4w/vrXv1a7vyQjPT3ddF4EDjLpFDLp7MokT9+nml6PJGP37t1VjunXr59xySWXmNaK2iOPTiGPyCOzPHr//fcNScbcuXNN6zQMw5g7d64hyXjvvfdM963uO+crNNHOcoHQ+DgdxkVFRcahQ4e8mmPLli2GJOODDz7wWV3FxcVGaGio8eKLL/psTsMwjEsvvdS49dZbfTrnbbfdZlx66aU+ndNTlZWVRlFRkbF+/XrLP0uwPzKpemSS51q2bGk8/PDDPp3zxx9/NIqKimii1TNkUvXIJM/5I5P88T4VFxcbRUVFRp8+fWiiBSDyqHrkkefskkePPPKIce655xonTpzw2Zw//fSTUVRUZNx+++1+baJxOifOmBYtWqhhw4b66aefan3sqlWrlJSUpAEDBvisnk8//VRt27bViBEjfDZncXGxNm3apNdff91ncxqGoby8vCrLYc+UY8eOcfFtnJXIJO9YnUlbtmzRzz//rEcffdSn83bo0EHHjh3z6ZxAbZBJ3jkbM8kf75Mk3XXXXXr33XclSZdccolP58bZhTzyDnnkuVWrVmnChAk+vQ73448/7jy91p+nsQcZhgcnFcO29uzZo/bt2+vZZ5/Vww8/bEkNu3btct65hAs5288vv/yivLw85+P4+Hidd9551hUEWyOTEKhWr16tkydPSjp1y/qLLrrI4opwJpBJqE++/vprHTx4UJLUqFEjXXbZZRZXhN8ij4C62bFjh/bu3SvJv59fVqLB7zp06KAOHTpYXQa8FBoaquTkZKvLAHyGTII7vrwFO1AbZBLOlNMXKQeqQx7BzuLj4xUfH+/357F0Jdqnn36qZ599VuvWrdOBAwf09ttva+DAgTUek5eXp8zMTG3ZskWxsbEaP3686V0vAAAAAAAAgLoItvLJS0pK1K1bN82aNcuj/Xfv3q0BAwboqquu0saNG/Xggw/qz3/+s1auXOnnSgEAAAAAAFCfBcw10YKCgkxXoj366KNatmyZNm/e7By7/fbbdfToUa1YseIMVAkAAAAAAID6yFbXRMvPz69ybaaUlBQ9+OCD1R5TVlamsrIy5+PKykodPnxYzZs3V1BQkL9KBRAgDMPQ8ePH1aZNGwUHW7r4ttbIL6D+IrsA2JVd84vsAuo3T7PLVk20goICxcTEuIzFxMSouLhYP//8syIiIqock52drcmTJ5+pEgEEqH379uncc8+1uoxaIb8AkF0A7Mpu+UV2AZDMs8tWp3PGx8crLS1N48aNc44tX75cAwYMUGlpqdsm2u9/UTh27JjOO+887du3T40bN/bpa8DZp3NW3a+3t3lyig8qgbeKi4sVGxuro0ePKioqyupyaoX8AuovsguAXdk1v8guoH7zNLtstRKtVatWKiwsdBkrLCxU48aN3TbQJMnhcMjhcFQZb9y4MWEIU8GOyDrPwecsMNhxGT75BYDsAmBXdssvsguAZJ5d9jlJXVJSUpJyc3Ndxj766CMlJSVZVBEAAAAAAADqA0ubaD/99JM2btyojRs3SpJ2796tjRs3au/evZKkcePGadiwYc7977vvPu3atUtjxozRtm3bNHv2bP3zn//U6NGjrSgfAAAAAAAA9YSlTbSvvvpKPXr0UI8ePSRJmZmZ6tGjhyZOnChJOnDggLOhJknt27fXsmXL9NFHH6lbt256/vnn9eqrryolhWtOAQAAAAAAwH8svSZa//79VdN9DV577TW3x2zYsMGPVQEAAAAAAACubHVNNAAAAAAAAMAKNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAE5Y30WbNmqW4uDiFh4crMTFRa9eurXH/GTNm6KKLLlJERIRiY2M1evRonThx4gxVCwAAAAAAgPrI0ibakiVLlJmZqaysLK1fv17dunVTSkqKDh486Hb/nJwcjR07VllZWdq6davmzZunJUuW6LHHHjvDlQMAAAAAAKA+sbSJNn36dI0YMUJpaWnq1KmT5s6dq8jISM2fP9/t/mvWrNHll1+uO+64Q3Fxcbrmmms0ZMgQ09VrAAAAAAAAQF1Y1kQrLy/XunXrlJyc/GsxwcFKTk5Wfn6+22P69OmjdevWOZtmu3bt0vLly3X99ddX+zxlZWUqLi522QDADsgvAHZEdgGwI7ILgCcsa6IdOnRIFRUViomJcRmPiYlRQUGB22PuuOMOPfHEE7riiivUoEEDnX/++erfv3+Np3NmZ2crKirKucXGxvr0dQCAv5BfAOyI7AJgR2QXAE9YfmOB2sjLy9PTTz+t2bNna/369Xrrrbe0bNkyPfnkk9UeM27cOB07dsy57du37wxWDADeI78A2BHZBcCOyC4Angi16omjo6MVEhKiwsJCl/HCwkK1atXK7TETJkzQXXfdpT//+c+SpC5duqikpET33nuvHn/8cQUHV+0JOhwOORwO378AAPAz8guAHZFdAOyI7ALgCctWooWFhSkhIUG5ubnOscrKSuXm5iopKcntMaWlpVUaZSEhIZIkwzD8VywAAAAAAADqNctWoklSZmamUlNT1atXL/Xu3VszZsxQSUmJ0tLSJEnDhg1T27ZtlZ2dLUm64YYbNH36dPXo0UOJiYnauXOnJkyYoBtuuMHZTAMAAAAAAAB8zdIm2uDBg1VUVKSJEyeqoKBA3bt314oVK5w3G9i7d6/LyrPx48crKChI48eP1/79+9WiRQvdcMMNmjJlilUvAQAAAAAAAPWApU00ScrIyFBGRobbv+Xl5bk8Dg0NVVZWlrKyss5AZQAAAAAAAMAptro7JwAAAAAAAGAFmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACa+aaKtWrfJ1HQAAAAAAAEDA8qqJdu211+r888/XU089pX379vm6JgAAAAAAACCgeNVE279/vzIyMrR06VJ16NBBKSkp+uc//6ny8nJf1wcAAAAAAABYzqsmWnR0tEaPHq2NGzfqiy++UHx8vO6//361adNGDzzwgDZt2uTrOgEAAAAAAADL1PnGAj179tS4ceOUkZGhn376SfPnz1dCQoL69u2rLVu2+KJGAAAAAAAAwFJeN9FOnjyppUuX6vrrr1e7du20cuVKvfjiiyosLNTOnTvVrl073Xbbbb6sFQAAAAAAALCEV020v/zlL2rdurX++7//W/Hx8dqwYYPy8/P15z//WQ0bNlRcXJyee+45bdu2zXSuWbNmKS4uTuHh4UpMTNTatWtr3P/o0aNKT09X69at5XA4FB8fr+XLl3vzMgAAAAAAAACPhHpz0LfffqsXXnhBgwYNksPhcLtPdHS0Vq1aVeM8S5YsUWZmpubOnavExETNmDFDKSkp2r59u1q2bFll//Lycv3Xf/2XWrZsqaVLl6pt27b6/vvv1aRJE29eBgAAAAAAAOARr5poWVlZ6tOnj0JDXQ//5ZdftGbNGl155ZUKDQ1Vv379apxn+vTpGjFihNLS0iRJc+fO1bJlyzR//nyNHTu2yv7z58/X4cOHtWbNGjVo0ECSFBcX581LAAAAAAAAADzm1emcV111lQ4fPlxl/NixY7rqqqs8mqO8vFzr1q1TcnLyr8UEBys5OVn5+fluj3nvvfeUlJSk9PR0xcTEqHPnznr66adVUVHhzcsAAAAAAAAAPOLVSjTDMBQUFFRl/Mcff1TDhg09muPQoUOqqKhQTEyMy3hMTEy111LbtWuXPvnkEw0dOlTLly/Xzp07df/99+vkyZPKyspye0xZWZnKysqcj4uLiz2qDwCsRn4BsCOyC4AdkV0APFGrJtqgQYMkSUFBQRo+fLjL9dAqKir09ddfq0+fPr6t8DcqKyvVsmVLvfzyywoJCVFCQoL279+vZ599ttomWnZ2tiZPnuy3mgDAX8gvAHZEdgGwI7ILgCdqdTpnVFSUoqKiZBiGzjnnHOfjqKgotWrVSvfee6/eeOMNj+aKjo5WSEiICgsLXcYLCwvVqlUrt8e0bt1a8fHxCgkJcY5dfPHFKigoUHl5udtjxo0bp2PHjjm3ffv2efhqAcBa5BcAOyK7ANgR2QXAE7VaibZgwQJJpy7m//DDD3t86qY7YWFhSkhIUG5urgYOHCjp1Eqz3NxcZWRkuD3m8ssvV05OjiorKxUcfKr/t2PHDrVu3VphYWFuj3E4HNXeQRQAAhn5BcCOyC4AdkR2AfCEVzcWyMrKqlMD7bTMzEy98sorev3117V161aNHDlSJSUlzrt1Dhs2TOPGjXPuP3LkSB0+fFijRo3Sjh07tGzZMj399NNKT0+vcy0AAAAAAABAdTxeidazZ0/l5uaqadOm6tGjh9sbC5y2fv16j+YcPHiwioqKNHHiRBUUFKh79+5asWKF82YDe/fuda44k6TY2FitXLlSo0ePVteuXdW2bVuNGjVKjz76qKcvAwAAAAAAAKg1j5toN910k3N56+nTL30hIyOj2tM38/LyqowlJSXp888/99nzAwAAAAAAAGY8bqL99u6X1d0JEwAAAAAAADgbeXVNNAAAAAAAAKA+8XglWtOmTWu8DtpvHT582OuCAAAAAAAAgEDjcRNtxowZfiwDAAAAAAAACFweN9FSU1P9WQcAAAAAAAAQsDxuohUXF6tx48bOf67J6f0AAAAAAACAs0Gtrol24MABtWzZUk2aNHF7fTTDMBQUFKSKigqfFgkAAAAAAABYyeMm2ieffKJmzZpJklatWuW3ggAAAAAAAIBA43ETrV+/fm7/GQAAAAAAADjbedxE+70jR45o3rx52rp1qySpU6dOSktLc65WAwAAAAAAAM4Wwd4c9OmnnyouLk4zZ87UkSNHdOTIEc2cOVPt27fXp59+6usaAQAAAAAAAEt5tRItPT1dgwcP1pw5cxQSEiJJqqio0P3336/09HR98803Pi0SAAAAAAAAsJJXK9F27typhx56yNlAk6SQkBBlZmZq586dPisOAAAAAAAACAReNdF69uzpvBbab23dulXdunWrc1EAAAAAAABAIPH4dM6vv/7a+c8PPPCARo0apZ07d+qyyy6TJH3++eeaNWuWpk6d6vsqAQAAAAAAAAt53ETr3r27goKCZBiGc2zMmDFV9rvjjjs0ePBg31QHAAAAAAAABACPm2i7d+/2Zx0AAAAAAABAwPK4idauXTt/1gEAAAAAAAAELI+baO58++232rt3r8rLy13Gb7zxxjoVBQAAAAAAAAQSr5pou3bt0s0336xvvvnG5TppQUFBkqSKigrfVQgAAAAAAABYLNibg0aNGqX27dvr4MGDioyM1JYtW/Tpp5+qV69eysvL83GJAAAAAAAAgLW8WomWn5+vTz75RNHR0QoODlZwcLCuuOIKZWdn64EHHtCGDRt8XScAAAAAAABgGa9WolVUVOicc86RJEVHR+uHH36QdOrmA9u3b/dddQAAAAAAAEAA8GolWufOnbVp0ya1b99eiYmJeuaZZxQWFqaXX35ZHTp08HWNAAAAAAAAgKW8aqKNHz9eJSUlkqQnnnhCf/zjH9W3b181b95cS5Ys8WmBAAAAAAAAgNW8aqKlpKQ4//mCCy7Qtm3bdPjwYTVt2tR5h04AAAAAAADgbOFVE+239u3bJ0mKjY2tczEAAAAAAABAIPLqxgK//PKLJkyYoKioKMXFxSkuLk5RUVEaP368Tp486esaAQAAAAAAAEt5tRLtL3/5i9566y0988wzSkpKkiTl5+dr0qRJ+vHHHzVnzhyfFgkAAAAAAABYyasmWk5OjhYvXqzrrrvOOda1a1fFxsZqyJAhNNEAAAAAAABwVvHqdE6Hw6G4uLgq4+3bt1dYWFhdawIAAAAAAAACildNtIyMDD355JMqKytzjpWVlWnKlCnKyMjwWXEAAAAAAABAIPD4dM5Bgwa5PP7444917rnnqlu3bpKkTZs2qby8XFdffbVvKwQAAAAAAAAs5nETLSoqyuXxLbfc4vI4NjbWNxUBAAAAAAAAAcbjJtqCBQv8WQcAAAAAAAAQsLy6JtppRUVF+uyzz/TZZ5+pqKjI63lmzZqluLg4hYeHKzExUWvXrvXouMWLFysoKEgDBw70+rkBAAAAAAAAM1410UpKSnT33XerdevWuvLKK3XllVeqTZs2uueee1RaWlqruZYsWaLMzExlZWVp/fr16tatm1JSUnTw4MEaj9uzZ48efvhh9e3b15uXAAAAAAAAAHjMqyZaZmamVq9erffff19Hjx7V0aNH9e6772r16tV66KGHajXX9OnTNWLECKWlpalTp06aO3euIiMjNX/+/GqPqaio0NChQzV58mR16NDBm5cAAAAAAAAAeMyrJtq//vUvzZs3T9ddd50aN26sxo0b6/rrr9crr7yipUuXejxPeXm51q1bp+Tk5F8LCg5WcnKy8vPzqz3uiSeeUMuWLXXPPfd4Uz4AAAAAAABQKx7fWOC3SktLFRMTU2W8ZcuWtTqd89ChQ6qoqKgyV0xMjLZt2+b2mM8++0zz5s3Txo0bPXqOsrIylZWVOR8XFxd7XB8AWIn8AmBHZBcAOyK7AHjCq5VoSUlJysrK0okTJ5xjP//8syZPnqykpCSfFfd7x48f11133aVXXnlF0dHRHh2TnZ2tqKgo5xYbG+u3+gDAl8gvAHZEdgGwI7ILgCeCDMMwanvQN998o2uvvVZlZWXq1q2bJGnTpk0KDw/XypUrdckll3g0T3l5uSIjI7V06VKXO2ympqY6r7P2Wxs3blSPHj0UEhLiHKusrJR06jTQ7du36/zzz3c5xt0vCrGxsTp27JgaN25cq9eN+idu7LI6z7Fn6gAfVAJvFRcXKyoqypbfefILqL/ILgB2Zdf8IruA+s3T7PLqdM4uXbrou+++06JFi5ynXQ4ZMkRDhw5VRESEx/OEhYUpISFBubm5ziZaZWWlcnNzlZGRUWX/jh076ptvvnEZGz9+vI4fP66//e1vbn8tcDgccjgctXh1ABAYyC8AdkR2AbAjsguAJ2rdRDt58qQ6duyoDz74QCNGjKhzAZmZmUpNTVWvXr3Uu3dvzZgxQyUlJUpLS5MkDRs2TG3btlV2drbCw8PVuXNnl+ObNGkiSVXGAQAAAAAAAF+pdROtQYMGLtdCq6vBgwerqKhIEydOVEFBgbp3764VK1Y4bzawd+9eBQd7dek2AAAAAAAAwCe8Op0zPT1d06ZN06uvvqrQUK+mcJGRkeH29E1JysvLq/HY1157rc7PDwAAAAAAANTEqw7Yl19+qdzcXH344Yfq0qWLGjZs6PL3t956yyfFAQAAAAAAAIHAqyZakyZNdMstt/i6FgAAAAAAACAg1aqJVllZqWeffVY7duxQeXm5/vCHP2jSpEm1uiMnAAAAAAAAYDe1aqJNmTJFkyZNUnJysiIiIjRz5kwVFRVp/vz5/qoPAAAAAAAEsLixy+o8x56pA3xQCeBftbrt5d///nfNnj1bK1eu1DvvvKP3339fixYtUmVlpb/qAwAAAAAAACxXqyba3r17df311zsfJycnKygoSD/88IPPCwMAAAAAAAACRa2aaL/88ovCw8Ndxho0aKCTJ0/6tCgAAAAAAAAgkNTqmmiGYWj48OFyOBzOsRMnTui+++5Tw4YNnWNvvfWW7yoEAAAAAAAALFarJlpqamqVsTvvvNNnxQAAAAAAANgJN1aoP2rVRFuwYIG/6gAAAAAAAAACVq2aaAAAAAD8h9UMAAAErlrdWAAAAAAAAACoj1iJBgAAPMIKGQAAANRnrEQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATIRaXQAAAABgR3Fjl9V5jj1TB/igEgAAcCawEg0AAAAAAAAwQRMNAAAAAAAAMEETDQAAAAAAADDBNdEAAIBluKYUAAAA7IKVaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACAiVCrCwAAAL4XN3ZZnefYM3WADyoBAGuRhwAAX2ElGgAAAAAAAGCCJhoAAAAAAABgIiCaaLNmzVJcXJzCw8OVmJiotWvXVrvvK6+8or59+6pp06Zq2rSpkpOTa9wfAAAAAAAAqCvLr4m2ZMkSZWZmau7cuUpMTNSMGTOUkpKi7du3q2XLllX2z8vL05AhQ9SnTx+Fh4dr2rRpuuaaa7Rlyxa1bdvWglcAAAAAAAAQ+LhOZN1YvhJt+vTpGjFihNLS0tSpUyfNnTtXkZGRmj9/vtv9Fy1apPvvv1/du3dXx44d9eqrr6qyslK5ublnuHIAAAAAAADUF5auRCsvL9e6des0btw451hwcLCSk5OVn5/v0RylpaU6efKkmjVr5vbvZWVlKisrcz4uLi6uW9EAcIaQXwDsiOwCYEdkFwBPWLoS7dChQ6qoqFBMTIzLeExMjAoKCjya49FHH1WbNm2UnJzs9u/Z2dmKiopybrGxsXWuGwDOBPILgB2RXQDsiOwC4AnLT+esi6lTp2rx4sV6++23FR4e7nafcePG6dixY85t3759Z7hKAPAO+QXAjsguAHZEdgHwhKWnc0ZHRyskJESFhYUu44WFhWrVqlWNxz733HOaOnWqPv74Y3Xt2rXa/RwOhxwOh0/qBYAzifwCYEdkFwA7IrsAeMLSlWhhYWFKSEhwuSnA6ZsEJCUlVXvcM888oyeffFIrVqxQr169zkSpAAAAAAAAqMcsXYkmSZmZmUpNTVWvXr3Uu3dvzZgxQyUlJUpLS5MkDRs2TG3btlV2drYkadq0aZo4caJycnIUFxfnvHZao0aN1KhRI8teBwAAAAAAAM5eljfRBg8erKKiIk2cOFEFBQXq3r27VqxY4bzZwN69exUc/OuCuTlz5qi8vFy33nqryzxZWVmaNGnSmSwdAAAAAAAA9YTlTTRJysjIUEZGhtu/5eXluTzes2eP/wsCAAAAAAAAfiMgmmgAgPotbuyyOs+xZ+oAH1QCAAAAAO5ZemMBAAAAAAAAwA5oogEAAAAAAAAmaKIBAAAAAAAAJmiiAQAAAAAAACa4sQAAAACAgMCNZgAAgYyVaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAICJUKsLAAAAAAAAwK/ixi6r8xx7pg7wQSX4LVaiAQAAAAAAACZoogEAAAAAAAAmaKIBAAAAAAAAJmiiAQAAAAAAACZoogEAAAAAAAAmaKIBAAAAAAAAJmiiAQAAAAAAACZoogEAAAAAAAAmaKIBAAAAAAAAJmiiAQAAAAAAACZoogEAAAAAAAAmaKIBAAAAAAAAJgKiiTZr1izFxcUpPDxciYmJWrt2bY37v/nmm+rYsaPCw8PVpUsXLV++/AxVCgAAAAAAgPrI8ibakiVLlJmZqaysLK1fv17dunVTSkqKDh486Hb/NWvWaMiQIbrnnnu0YcMGDRw4UAMHDtTmzZvPcOUAAAAAAACoLyxvok2fPl0jRoxQWlqaOnXqpLlz5yoyMlLz5893u//f/vY3XXvttXrkkUd08cUX68knn1TPnj314osvnuHKAQAAAAAAUF+EWvnk5eXlWrduncaNG+ccCw4OVnJysvLz890ek5+fr8zMTJexlJQUvfPOO273LysrU1lZmfPxsWPHJEnFxcV1rB71QWVZaZ3n4LNmrdPvv2EYFldSe/Upv/iu+Z4/3lO7zHk2ILvsge+E7/Ge2p9d86s+ZZc/1PfvmZ2yq77/u6qOx9llWGj//v2GJGPNmjUu44888ojRu3dvt8c0aNDAyMnJcRmbNWuW0bJlS7f7Z2VlGZLY2Njq+bZv3z7fBNcZRH6xsbGRXWxsbHbd7JZfZBcbG5tknl1BhmHdTwQ//PCD2rZtqzVr1igpKck5PmbMGK1evVpffPFFlWPCwsL0+uuva8iQIc6x2bNna/LkySosLKyy/+9/UaisrNThw4fVvHlzBQUF1fk1FBcXKzY2Vvv27VPjxo3rPJ8/2aVWu9Qp2adWu9Qp+b5WwzB0/PhxtWnTRsHBlp/BXivk1yl2qVOyT612qVOyT61k16/IrlPsUqdkn1rtUqdUv2u1a36RXb+yS612qVOyT612qVOyLrssPZ0zOjpaISEhVZpfhYWFatWqldtjWrVqVav9HQ6HHA6Hy1iTJk28L7oajRs3DvgP2Wl2qdUudUr2qdUudUq+rTUqKson85xp5Jcru9Qp2adWu9Qp2adWsovs+j271CnZp1a71CnV31rtmF9kV1V2qdUudUr2qdUudUpnPrss/WkgLCxMCQkJys3NdY5VVlYqNzfXZWXabyUlJbnsL0kfffRRtfsDAAAAAAAAdWXpSjRJyszMVGpqqnr16qXevXtrxowZKikpUVpamiRp2LBhatu2rbKzsyVJo0aNUr9+/fT8889rwIABWrx4sb766iu9/PLLVr4MAAAAAAAAnMUsb6INHjxYRUVFmjhxogoKCtS9e3etWLFCMTExkqS9e/e6nI/ap08f5eTkaPz48Xrsscd04YUX6p133lHnzp0tqd/hcCgrK6vK0t9AZJda7VKnZJ9a7VKnZK9a7c4u77Vd6pTsU6td6pTsU6td6jwb2OW9tkudkn1qtUudErWiKju9z3ap1S51Svap1S51StbVaumNBQAAAAAAAAA7sM/tUgAAAAAAAACL0EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0ESro1mzZikuLk7h4eFKTEzU2rVrrS7JRXZ2ti699FKdc845atmypQYOHKjt27dbXZZHpk6dqqCgID344INWl1LF/v37deedd6p58+aKiIhQly5d9NVXX1ldVhUVFRWaMGGC2rdvr4iICJ1//vl68sknFQj3E/n00091ww03qE2bNgoKCtI777zj8nfDMDRx4kS1bt1aERERSk5O1nfffWdNsWehQM8uyb75FcjZJdkjv8guVIfs8h+yyzcCNb/ILusFen6RXf5BdtVdoOUXTbQ6WLJkiTIzM5WVlaX169erW7duSklJ0cGDB60uzWn16tVKT0/X559/ro8++kgnT57UNddco5KSEqtLq9GXX36pl156SV27drW6lCqOHDmiyy+/XA0aNNC///1vffvtt3r++efVtGlTq0urYtq0aZozZ45efPFFbd26VdOmTdMzzzyjF154werSVFJSom7dumnWrFlu//7MM89o5syZmjt3rr744gs1bNhQKSkpOnHixBmu9Oxjh+yS7JlfgZxdkn3yi+yCO2SX/5BdvhOo+UV2WcsO+UV2+R7Z5RsBl18GvNa7d28jPT3d+biiosJo06aNkZ2dbWFVNTt48KAhyVi9erXVpVTr+PHjxoUXXmh89NFHRr9+/YxRo0ZZXZKLRx991LjiiiusLsMjAwYMMO6++26XsUGDBhlDhw61qCL3JBlvv/2283FlZaXRqlUr49lnn3WOHT161HA4HMY//vEPCyo8u9gxuwwj8PMr0LPLMOyTX2QX3CG7/IPs8i075BfZdebZMb/Irroju3wvEPKLlWheKi8v17p165ScnOwcCw4OVnJysvLz8y2srGbHjh2TJDVr1sziSqqXnp6uAQMGuLy3geS9995Tr169dNttt6lly5bq0aOHXnnlFavLcqtPnz7Kzc3Vjh07JEmbNm3SZ599puuuu87iymq2e/duFRQUuHwGoqKilJiYGNDfLzuwa3ZJgZ9fgZ5dkn3yi+zC75Fd/kN2+ZYd84vs8i+75hfZVXdkl/9ZkV+hfpm1Hjh06JAqKioUExPjMh4TE6Nt27ZZVFXNKisr9eCDD+ryyy9X586drS7HrcWLF2v9+vX68ssvrS6lWrt27dKcOXOUmZmpxx57TF9++aUeeOABhYWFKTU11eryXIwdO1bFxcXq2LGjQkJCVFFRoSlTpmjo0KFWl1ajgoICSXL7/Tr9N3jHjtklBX5+2SG7JPvkF9mF3yO7/IPs8j075hfZ5V92zC+yyzfILv+zIr9ootUj6enp2rx5sz777DOrS3Fr3759GjVqlD766COFh4dbXU61Kisr1atXLz399NOSpB49emjz5s2aO3duwIXhP//5Ty1atEg5OTm65JJLtHHjRj344INq06ZNwNUK1CSQ88su2SXZJ7/ILpwtyC7fsEt2SeQXzg5kl2+QXWcnTuf0UnR0tEJCQlRYWOgyXlhYqFatWllUVfUyMjL0wQcfaNWqVTr33HOtLsetdevW6eDBg+rZs6dCQ0MVGhqq1atXa+bMmQoNDVVFRYXVJUqSWrdurU6dOrmMXXzxxdq7d69FFVXvkUce0dixY3X77berS5cuuuuuuzR69GhlZ2dbXVqNTn+H7PL9shO7ZZcU+Plll+yS7JNfZBd+j+zyPbLLP+yYX2SXf9ktv8gu3yG7/M+K/KKJ5qWwsDAlJCQoNzfXOVZZWanc3FwlJSVZWJkrwzCUkZGht99+W5988onat29vdUnVuvrqq/XNN99o48aNzq1Xr14aOnSoNm7cqJCQEKtLlCRdfvnlVW73vGPHDrVr186iiqpXWlqq4GDXr3lISIgqKystqsgz7du3V6tWrVy+X8XFxfriiy8C6vtlR3bJLsk++WWX7JLsk19kF36P7PI9sss/7JhfZJd/2SW/yC7fI7v8z5L88svtCuqJxYsXGw6Hw3jttdeMb7/91rj33nuNJk2aGAUFBVaX5jRy5EgjKirKyMvLMw4cOODcSktLrS7NI4F4p5W1a9caoaGhxpQpU4zvvvvOWLRokREZGWm88cYbVpdWRWpqqtG2bVvjgw8+MHbv3m289dZbRnR0tDFmzBirSzOOHz9ubNiwwdiwYYMhyZg+fbqxYcMG4/vvvzcMwzCmTp1qNGnSxHj33XeNr7/+2rjpppuM9u3bGz///LPFldufHbLLMOydX4GYXYZhn/wiu+AO2eV/ZFfdBWp+kV3WskN+kV2+R3b5RqDlF020OnrhhReM8847zwgLCzN69+5tfP7551aX5EKS223BggVWl+aRQA3E999/3+jcubPhcDiMjh07Gi+//LLVJblVXFxsjBo1yjjvvPOM8PBwo0OHDsbjjz9ulJWVWV2asWrVKrefzdTUVMMwTt2ueMKECUZMTIzhcDiMq6++2ti+fbu1RZ9FAj27DMPe+RWo2WUY9sgvsgvVIbv8i+yqu0DNL7LLeoGeX2SXf5BddRdo+RVkGIbh+/VtAAAAAAAAwNmDa6IBAAAAAAAAJmiiAQAAAAAAACZoogEAAAAAAAAmaKIBAAAAAAAAJmiiAQAAAAAAACZoogEAAAAAAAAmaKIBAAAAAAAAJmiiAQAAAAAAACZoogEAAAAAAAAmaKIBAAAAAAAAJmiiAQAAAAAAACZoogEAAAAAAAAm/h8MS3mBpryazwAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "fig, axes = plt.subplots(1, 4, sharey=True, figsize=(15, 3))\n", + "for i in range(4):\n", + " _x = torch.concat([x, y[:, :i]], dim=1)\n", + " axes[i].bar(np.arange(11), torch.softmax((actor(_x)[0][0, -1].detach()), dim=0).tolist())\n", + " axes[i].title.set_text(f\"Input {_x[0].tolist()}\")\n", + " axes[i].set_ylim([0, 1])\n", + "axes[0].set_ylabel(\"Probability\")" + ] + }, + { + "cell_type": "markdown", + "id": "5b37b059", + "metadata": {}, + "source": [ + "![](../../../images/session_4/part_2_finetuning_lms_to_human_preferences/kl1_attn.png)" + ] + }, + { + "cell_type": "markdown", + "id": "69f41ade", + "metadata": {}, + "source": [ + "The model learnt to change the distribution over the first output token only. For the first plot and attention figure, the model learns to focus on a single token and output its increment. As for the rest of the tokens, it retains a similar behavior as the base model." + ] + }, + { + "cell_type": "markdown", + "id": "dd7024d2", + "metadata": {}, + "source": [ + "### Fine tuned model with beta 0" + ] + }, + { + "cell_type": "markdown", + "id": "bb27d860", + "metadata": {}, + "source": [ + "As as additional exercise, we also look at how the model behaves when no KL divergence penalty is applied. \n", + "\n", + "I've run the training separately and compiled the results here." + ] + }, + { + "cell_type": "markdown", + "id": "554d9eab", + "metadata": {}, + "source": [ + "![](../../../images/session_4/part_2_finetuning_lms_to_human_preferences/kl0_plot.png)" + ] + }, + { + "cell_type": "markdown", + "id": "c77b1643", + "metadata": {}, + "source": [ + "The reward hits the max but the KL is diverging, which means that the policy we are learning is moving further away from the base distribution as the training goes on. This is a result of not applying a penalty to KL divergence." + ] + }, + { + "cell_type": "code", + "execution_count": 149, + "id": "54050769", + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "data": { + "text/plain": [ + "Text(0, 0.5, 'Probability')" + ] + }, + "execution_count": 149, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAABNEAAAEpCAYAAAC9VbStAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA4EUlEQVR4nO3de1hU5d7/8c8AMnhIPKB4iEQrMvOMaVim7XiicltmBzNLpbIngzKpTEtFK0M7mFkeOqiVydZyd9atGYk9bSnLU1mamZr+TBDzgEGCwfr94eXUxMAahhnXLHm/rmtdl3Oz1j3fmZhP+p17reUwDMMQAAAAAAAAgAqFWF0AAAAAAAAAEOxoogEAAAAAAAAmaKIBAAAAAAAAJmiiAQAAAAAAACZoogEAAAAAAAAmaKIBAAAAAAAAJmiiAQAAAAAAACZoogEAAAAAAAAmaKIBAAAAAAAAJmiiIeAmTpwoh8Mhh8OhevXqWV0Oqujw4cOu/34Oh0PPPPOM1SUB1UImwZMGDRq4fi9SU1OtLgc1CJmEU6V///6u37X27dtbXQ6CEHkEO7v//vtPye8vTTQbeO211+RwOPT1119bXYokqaioSBMnTlR2dnaVjluwYIHmzp3rNjZs2DC3Bs3JrW3btj7XFxsb63FOh8Ohc88916c59+3bpzFjxuiyyy7TGWecIYfDUeXX/3e7du2qsM5Fixb5PO/s2bN144036qyzzpLD4dCwYcMq3Pfw4cO666671KRJE9WtW1eXXXaZ1q9f77ZP3bp1tWDBAj333HM+14TTC5lUNWSSfzPJF+vXr9c111yjRo0aqU6dOmrfvr1mzJjhts/LL7+sBQsWVPu5cOqRSVVDJtkjk7y1detWjR49Wp07d9YZZ5yh5s2bq2/fvh4/D6NGjdKCBQuq9fuDypFHVUMe1dw88lZ2dnaF773D4dDkyZNd+952221asGCBevXq5fPzeSMsoLPjtFRUVKRJkyZJkvr06eP1cbfeeqvHcafTqVdffdVtLDIy0uf6pk+frt9++81t7Oeff9a4ceN0xRVX+DTnDz/8oKlTp+rcc89Vhw4dlJOT43N9fzdo0CBdffXVbmMJCQk+zzd16lQdPXpU3bt31759+yrcr6ysTH379tWmTZv00EMPKSoqSrNmzVKfPn20bt061/+4atWqpVtvvVW7du3SqFGjfK4LCBQyqWZlUlV9/PHH6tevn7p06aLx48erXr16+umnn/T//t//c9vvpptuknTiL2BAdZBJZFJlvM0kb7366quaO3eurr/+et1zzz06cuSIXnrpJV100UVavny5EhMTXfv27t3bdcyBAwd8ej7YC3lEHlXGyjzy1vnnn+/xS84FCxbo448/dvs9iY+PV3x8vD755BO/NBgrQhMNlgsLC6swqH3Rv3//cmNPPPGEJGnw4ME+zRkfH69ff/1VjRo10pIlS3TjjTdWp0Q3Xbt29evrX716tevbjMqWsS5ZskRr1qzR22+/rRtuuEHSiX9ExsXFKT09XZmZmX6rCbATMun0yaSCggINGTJEffv21ZIlSxQSwgJ82A+ZRCZVZtCgQZo4caLba7n99tt1/vnna+LEiT79oxWoCHlEHlUmEHkUHR3t8T2fNGmSzj33XF144YXVqtkX/G3SpoYNG6Z69epp79696t+/v+rVq6cmTZrowQcfVGlpqWu/k0tPn3nmGT333HNq1aqVateurd69e2vz5s1uc/bp08fjNxTDhg1TbGysa74mTZpIOvGLe3IZ5cSJE6v1ekpLS1VQUFCtOSqTmZmp1q1bq2fPnj4df8YZZ6hRo0Z+rupPhYWFKikp8ctcrVq1ksPhMN1vyZIlio6O1oABA1xjTZo00U033aT3339fxcXFfqkHNQOZVDVkUnmByKTMzEzl5eVp8uTJCgkJUWFhocrKyqo8D+yHTKoaMqk8u2RSfHx8uX98N27cWL169dKWLVuqNTf8gzyqGvKoPPLI3dq1a7V9+3afG63VRRPNxkpLS5WUlKTGjRvrmWeeUe/evfXss8/q5ZdfLrfvG2+8oRkzZiglJUVjx47V5s2b9Y9//EN5eXlVes4mTZpo9uzZkqTrrrtOCxYs0IIFC9w+0FVVVFSk+vXrKzIyUo0aNVJKSkq5Zb3VsWHDBm3ZskW33HKL3+b0p0mTJqlevXqKiIjQhRdeqI8//viUPO+GDRvUtWvXct86dO/eXUVFRdq2bdspqQOnDzLJO2SSZ4HIpE8++UT169fX3r17dd5556levXqqX7++RowYoWPHjvmrdAQpMsk7ZJJnds+k3NxcRUVF+XVO+I488g555Bl55G7hwoWSfF+tWF2czmljx44d08CBAzV+/HhJ0t13362uXbtq7ty5GjFihNu+27dv148//qiWLVtKkq688kr16NFDU6dO1bRp07x+zrp16+qGG27QiBEj1LFjx2ovZ23evLlGjx6trl27qqysTMuXL9esWbO0adMmZWdnKyys+r+iVn/IKhISEqIrrrhC1113nVq2bKkdO3Zo2rRpuuqqq/TBBx+ob9++AX3+ffv26dJLLy033rx5c0nSL7/8og4dOgS0BpxeyCTvkEmeBSKTfvzxR/3xxx+69tprdccddygjI0PZ2dl64YUXdPjwYf3rX//yS+0ITmSSd8gkz+ycSf/3f/+nnJwcjRs3zi/zofrII++QR56RR38qLS3V4sWL1b17d51zzjl+mbOqaKLZ3N133+32uFevXh4vvNe/f39XEEsnutY9evTQsmXLqhTG/paRkeH2+Oabb1ZcXJweffRRLVmyRDfffHO15i8rK9OiRYvUpUsXnX/++dWay9/OOussrVixwm3stttuU7t27fTAAw8EPIx///13OZ3OcuMRERGunwNVRSZVjkyqWCAy6bffflNRUZHuvvtu152mBgwYoJKSEr300kt67LHHfL4YL+yBTKocmVQxu2bS/v37dcstt6h169YaPXp0teaCf5FHlSOPKkYe/SkrK0t5eXl65JFH/DKfLzid08YiIiJc57mf1LBhQx06dKjcvp4+AHFxcdq1a1egyvPZqFGjFBISok8++aTac61evVp79+4Num8zKtKoUSMlJyfrhx9+8PmuKN6qXbu2x/PnTy7frV27dkCfH6cfMskcmVSxQGTSyWMGDRrkNn7yNBF/3rELwYdMMkcmVcyOmVRYWKh//vOfOnr0qN5///1KL1SOU4s8MkceVYw8+tPChQsVGhqqgQMH+mU+X9BEs7HQ0FC/zlfRRQ3/esHLU6F27dpq3LixDh48WO25Fi5cqJCQkHLhEMxiYmIkyS+vvzLNmzf3eCvlk2MtWrQI6PPj9EMmmSOTKhaITDp5THR0tNt406ZNJcnjP15w+iCTzJFJFbNbJpWUlGjAgAH65ptv9P7776t9+/Y+zwX/I4/MkUcVI49O+P333/Xuu+8qMTGxXN2nEk20GuLHH38sN7Zt2zbX3VukE9+GHD58uNx+P//8s9tjb+4gUh1Hjx7VgQMHyn1bU1XFxcX697//rT59+tiqIbRjxw5JqvbrN9O5c2etX7++3F1YvvzyS9WpU0dxcXEBfX7UbGQSmfR3gcik+Ph4SdLevXvdxn/55RdJgX9NsA8yiUz6OztlUllZmYYMGaKsrCxlZmaqd+/ePs2D4EAekUd/Rx6d8MEHH+jo0aOWr1akiVZDvPfee24fkLVr1+rLL7/UVVdd5Ro7++yztXXrVuXn57vGNm3apP/+979uc9WpU0eSPAZ3VRw7dkxHjx4tN/7444/LMAxdeeWV1Zp/2bJlOnz4sOUfsor89X0+ae/evZo3b546duzoulBkoNxwww3Ky8vTO++84xo7cOCA3n77bfXr18/jefeAv5BJwed0zKSbbrpJkjR37ly38VdffVVhYWHq06dPtWrG6YNMCj5kkvfuvfdeLV68WLNmzarWnRcRHMij4EMeeS+QeZSZmak6derouuuu8+u8VcWNBWqIc845R5dccolGjBih4uJiTZ8+XY0bN3a7wN/tt9+uadOmKSkpSXfccYf279+vOXPm6IILLlBBQYFrv9q1a6tdu3ZavHix4uLi1KhRI7Vv377KyzRzc3PVpUsXDRo0SG3btpUkrVixQsuWLdOVV16pa6+91m3/k9++eHs9gIULF8rpdOr666+vcJ8+ffpo9erVMgzDdL4nnnhCkvTdd99JkhYsWKDPP/9cktzuNjJx4kRNmjRJq1atqjR8Ro8erZ9++kmXX365WrRooV27dumll15SYWGhnn/+ebd9X3vtNSUnJ2v+/PkaNmxYpXV++OGH2rRpkyTp+PHj+uabb1y1X3PNNerYsaOkE2F80UUXKTk5Wd9//72ioqI0a9YslZaWatKkSabvB1AdZJJnZJJ3mTRs2DC9/vrr2rlzp9s383/XpUsX3X777Zo3b57++OMP9e7dW9nZ2Xr77bc1duxYW33bjcAikzwjk6zLJG/fp+nTp2vWrFlKSEhQnTp19Oabb7r9/LrrrlPdunUrfU8QXMgjz8ij0yuPsrOzddlllyk9PV0TJ06s9H2STpwy+5///EfXX3+99dd7NBD05s+fb0gyvvrqK9fY0KFDjbp165bbNz093fjrf9adO3cakoynn37aePbZZ42YmBjD6XQavXr1MjZt2lTu+DfffNNo06aNER4ebnTu3NlYsWKFMXToUKNVq1Zu+61Zs8aIj483wsPDDUlGenp6hfX/vaaTDh06ZNx6663GOeecY9SpU8dwOp3GBRdcYDz55JNGSUlJuf2joqKMiy66qMLn+asjR44YERERxoABAyrdLz4+3mjWrJlXc0qqcPurBx54wHA4HMaWLVsqnS8zM9O49NJLjSZNmhhhYWFGVFSUcd111xnr1q0rt+8LL7xgSDKWL19uWufQoUMrrHP+/Plu+x48eNC44447jMaNGxt16tQxevfu7fZ79ld//V1CzUYmnUAmWZdJ119/vVG7dm3j0KFDps9fUlJiTJw40WjVqpVRq1Yt45xzzjGee+65CveXZKSkpJjOi+BBJp1AJp1emeTt+1TZ65Fk7Ny5s9wxvXv3Ni644ALTWlF15NEJ5BF5ZJZHH374oSHJmDNnjmmdhmEYc+bMMSQZH3zwgem+FX3m/IUm2mkuGBofJ8M4Pz/fOHDggE9zfPfdd4Yk46OPPvJbXQUFBUZYWJjx4osv+m1OwzCMCy+80Ljhhhv8OueNN95oXHjhhX6d01tlZWVGfn6+sX79est/l2B/ZFLFyCTvNW3a1HjwwQf9Ouevv/5q5Ofn00SrYcikipFJ3gtEJgXifSooKDDy8/ONnj170kQLQuRRxcgj79kljx566CHjzDPPNI4dO+a3OX/77TcjPz/fuPnmmwPaRON0TpwyTZo0Ud26dfXbb79V+dhVq1YpISFBffv29Vs9n332mVq2bKnhw4f7bc6CggJt2rRJr7/+ut/mNAxD2dnZ5ZbDnipHjhzh4ts4LZFJvrE6k7777jv9/vvvevjhh/06b5s2bXTkyBG/zglUBZnkm9MxkwLxPknSbbfdpvfff1+SdMEFF/h1bpxeyCPfkEfeW7VqlcaPH+/X63A/+uijrtNrA3kau8MwvDipGLa1a9cutW7dWk8//bQefPBBS2rYsWOH684lXMjZfv744w9lZ2e7HsfFxemss86yriDYGpmEYLV69WodP35c0olb1p933nkWV4RTgUxCTfLNN99o//79kqR69erpoosusrgi/BV5BFTPtm3btHv3bkmB/f1lJRoCrk2bNmrTpo3VZcBHYWFhSkxMtLoMwG/IJHjiz1uwA1VBJuFUOXmRcqAi5BHsLC4uTnFxcQF/HktXon322Wd6+umntW7dOu3bt0/vvvuu+vfvX+kx2dnZSktL03fffaeYmBiNGzfO9K4XAAAAAAAAQHWEWPnkhYWF6tSpk2bOnOnV/jt37lTfvn112WWXaePGjbr//vt15513asWKFQGuFAAAAAAAADVZ0FwTzeFwmK5Ee/jhh7V06VJt3rzZNXbzzTfr8OHDWr58+SmoEgAAAAAAADWRra6JlpOTU+7aTElJSbr//vsrPKa4uFjFxcWux2VlZTp48KAaN24sh8MRqFIBBAnDMHT06FG1aNFCISGWLr6tMvILqLnILgB2Zdf8IruAms3b7LJVEy03N1fR0dFuY9HR0SooKNDvv/+u2rVrlzsmIyNDkyZNOlUlAghSe/bs0Zlnnml1GVVCfgEguwDYld3yi+wCIJlnl61O54yLi1NycrLGjh3rGlu2bJn69u2roqIij020v3+jcOTIEZ111lnas2eP6tev79fXACD4FBQUKCYmRocPH1ZkZKTV5VQJ+QXUXGQXALuya36RXUDN5m122WolWrNmzZSXl+c2lpeXp/r163tsoEmS0+mU0+ksN16/fn3CEKZixyyt9hy7pvT1QyWoLjsuwye/AJBdAOzKbvlFdgGQzLPLPiepS0pISFBWVpbb2MqVK5WQkGBRRQAAAAAAAKgJLG2i/fbbb9q4caM2btwoSdq5c6c2btyo3bt3S5LGjh2rIUOGuPa/++67tWPHDo0ePVpbt27VrFmz9NZbb2nUqFFWlA8AAAAAAIAawtIm2tdff60uXbqoS5cukqS0tDR16dJFEyZMkCTt27fP1VCTpNatW2vp0qVauXKlOnXqpGeffVavvvqqkpKSLKkfAAAAAAAANYOl10Tr06ePKruvwWuvvebxmA0bNgSwKgAAAAAAAMCdra6JBgAAAAAAAFiBJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABggiYaAAAAAAAAYIImGgAAAAAAAGCCJhoAAAAAAABgIszqAgAAAABAkmLHLK32HLum9PVDJQAAlMdKNAAAAAAAAMAEK9EAADgNsZoDAAAA8C9WogEAAAAAAAAmaKIBAAAAAAAAJmiiAQAAAAAAACYsb6LNnDlTsbGxioiIUI8ePbR27dpK958+fbrOO+881a5dWzExMRo1apSOHTt2iqoFAAAAAABATWRpE23x4sVKS0tTenq61q9fr06dOikpKUn79+/3uH9mZqbGjBmj9PR0bdmyRXPnztXixYv1yCOPnOLKAQAAAAAAUJNY2kSbNm2ahg8fruTkZLVr105z5sxRnTp1NG/ePI/7r1mzRhdffLFuueUWxcbG6oorrtCgQYNMV68BAAAAAAAA1WFZE62kpETr1q1TYmLin8WEhCgxMVE5OTkej+nZs6fWrVvnaprt2LFDy5Yt09VXX13h8xQXF6ugoMBtAwA7IL8A2BHZBcCOyC4A3rCsiXbgwAGVlpYqOjrabTw6Olq5ubkej7nlllv02GOP6ZJLLlGtWrV09tlnq0+fPpWezpmRkaHIyEjXFhMT49fXAQCBQn4BsCOyC4AdkV0AvGH5jQWqIjs7W08++aRmzZql9evX65133tHSpUv1+OOPV3jM2LFjdeTIEde2Z8+eU1gxAPiO/AJgR2QXADsiuwB4I8yqJ46KilJoaKjy8vLcxvPy8tSsWTOPx4wfP1633Xab7rzzTklShw4dVFhYqLvuukuPPvqoQkLK9wSdTqecTqf/XwAABBj5BcCOyC4AdkR2AfCGZSvRwsPDFR8fr6ysLNdYWVmZsrKylJCQ4PGYoqKico2y0NBQSZJhGIErFgAAAAAAADWaZSvRJCktLU1Dhw5Vt27d1L17d02fPl2FhYVKTk6WJA0ZMkQtW7ZURkaGJKlfv36aNm2aunTpoh49emj79u0aP368+vXr52qmAQAAAAAAAP5maRNt4MCBys/P14QJE5Sbm6vOnTtr+fLlrpsN7N69223l2bhx4+RwODRu3Djt3btXTZo0Ub9+/TR58mSrXgIAAAAAAABqAEubaJKUmpqq1NRUjz/Lzs52exwWFqb09HSlp6efgsoAAAAAAACAE2x1d04AAAAAAADACjTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM+NdFWrVrl7zoAAAAAAACAoOVTE+3KK6/U2WefrSeeeEJ79uzxd00AAAAAAABAUPGpibZ3716lpqZqyZIlatOmjZKSkvTWW2+ppKTE3/UBAAAAAAAAlvOpiRYVFaVRo0Zp48aN+vLLLxUXF6d77rlHLVq00H333adNmzb5u04AAAAAAADAMtW+sUDXrl01duxYpaam6rffftO8efMUHx+vXr166bvvvvNHjQAAAAAAAIClfG6iHT9+XEuWLNHVV1+tVq1aacWKFXrxxReVl5en7du3q1WrVrrxxhv9WSsAAAAAAABgCZ+aaPfee6+aN2+u//3f/1VcXJw2bNignJwc3Xnnnapbt65iY2P1zDPPaOvWraZzzZw5U7GxsYqIiFCPHj20du3aSvc/fPiwUlJS1Lx5czmdTsXFxWnZsmW+vAwAAAAAAADAK2G+HPT999/rhRde0IABA+R0Oj3uExUVpVWrVlU6z+LFi5WWlqY5c+aoR48emj59upKSkvTDDz+oadOm5fYvKSnR//zP/6hp06ZasmSJWrZsqZ9//lkNGjTw5WUAAAAAAAAAXvGpiZaenq6ePXsqLMz98D/++ENr1qzRpZdeqrCwMPXu3bvSeaZNm6bhw4crOTlZkjRnzhwtXbpU8+bN05gxY8rtP2/ePB08eFBr1qxRrVq1JEmxsbG+vAQAAAAAAADAaz6dznnZZZfp4MGD5caPHDmiyy67zKs5SkpKtG7dOiUmJv5ZTEiIEhMTlZOT4/GYDz74QAkJCUpJSVF0dLTat2+vJ598UqWlpb68DAAAAAAAAMArPq1EMwxDDoej3Pivv/6qunXrejXHgQMHVFpaqujoaLfx6OjoCq+ltmPHDn366acaPHiwli1bpu3bt+uee+7R8ePHlZ6e7vGY4uJiFRcXux4XFBR4VR8AWI38AmBHZBcAOyK7AHijSk20AQMGSJIcDoeGDRvmdj200tJSffPNN+rZs6d/K/yLsrIyNW3aVC+//LJCQ0MVHx+vvXv36umnn66wiZaRkaFJkyYFrCYACBTyC4AdkV0A7IjsAuCNKp3OGRkZqcjISBmGoTPOOMP1ODIyUs2aNdNdd92lN99806u5oqKiFBoaqry8PLfxvLw8NWvWzOMxzZs3V1xcnEJDQ11j559/vnJzc1VSUuLxmLFjx+rIkSOubc+ePV6+WgCwFvkFwI7ILgB2RHYB8EaVVqLNnz9f0omL+T/44INen7rpSXh4uOLj45WVlaX+/ftLOrHSLCsrS6mpqR6Pufjii5WZmamysjKFhJzo/23btk3NmzdXeHi4x2OcTmeFdxAFgGBGfgGwI7ILgB2RXQC84dONBdLT06vVQDspLS1Nr7zyil5//XVt2bJFI0aMUGFhoetunUOGDNHYsWNd+48YMUIHDx7UyJEjtW3bNi1dulRPPvmkUlJSql0LAAAAAAAAUBGvV6J17dpVWVlZatiwobp06eLxxgInrV+/3qs5Bw4cqPz8fE2YMEG5ubnq3Lmzli9f7rrZwO7du10rziQpJiZGK1as0KhRo9SxY0e1bNlSI0eO1MMPP+ztywAAAAAAAPCb2DFLqz3Hril9/VAJAs3rJtq1117rWt568vRLf0hNTa3w9M3s7OxyYwkJCfriiy/89vwAAAAAAACAGa+baH+9+2VFd8IEAAAAAAAATkc+XRMNAAAAAAAAqEm8XonWsGHDSq+D9lcHDx70uSAAAAAAAAAg2HjdRJs+fXoAywAAAAAAAACCl9dNtKFDhwayDgAAAAAAACBoed1EKygoUP369V1/rszJ/QAAAAAAAIDTQZWuibZv3z41bdpUDRo08Hh9NMMw5HA4VFpa6tciAQAAAAAAACt53UT79NNP1ahRI0nSqlWrAlYQAAAAAAAAEGy8bqL17t3b458BAAAAAACA053XTbS/O3TokObOnastW7ZIktq1a6fk5GTXajUAAAAAAADgdBHiy0GfffaZYmNjNWPGDB06dEiHDh3SjBkz1Lp1a3322Wf+rhEAAAAAAACwlE8r0VJSUjRw4EDNnj1boaGhkqTS0lLdc889SklJ0bfffuvXIgEAAAAAAAAr+bQSbfv27XrggQdcDTRJCg0NVVpamrZv3+634gAAAAAAAIBg4FMTrWvXrq5rof3Vli1b1KlTp2oXBQAAAAAAAAQTr0/n/Oabb1x/vu+++zRy5Eht375dF110kSTpiy++0MyZMzVlyhT/VwkAAAAAAABYyOsmWufOneVwOGQYhmts9OjR5fa75ZZbNHDgQP9UBwAAAAAAAAQBr5toO3fuDGQdAAAAAAAAQNDyuonWqlWrQNYBAAAAAAAABC2vm2iefP/999q9e7dKSkrcxq+55ppqFQUAAAAAAAAEE5+aaDt27NB1112nb7/91u06aQ6HQ5JUWlrqvwoBAAAAAAAAi4X4ctDIkSPVunVr7d+/X3Xq1NF3332nzz77TN26dVN2drafSwQAAAAAAACs5dNKtJycHH366aeKiopSSEiIQkJCdMkllygjI0P33XefNmzY4O86AQAAAAAAAMv4tBKttLRUZ5xxhiQpKipKv/zyi6QTNx/44Ycf/FcdAAAAAAAAEAR8WonWvn17bdq0Sa1bt1aPHj301FNPKTw8XC+//LLatGnj7xoBAAAAAAAAS/nURBs3bpwKCwslSY899pj++c9/qlevXmrcuLEWL17s1wIBAAAAAAAAq/nUREtKSnL9+ZxzztHWrVt18OBBNWzY0HWHTgAAAAAAAOB04VMT7a/27NkjSYqJial2MQAAAAAAAEAw8unGAn/88YfGjx+vyMhIxcbGKjY2VpGRkRo3bpyOHz/u7xoBAAAAAAAAS/m0Eu3ee+/VO++8o6eeekoJCQmSpJycHE2cOFG//vqrZs+e7dciAQAAAAAAACv51ETLzMzUokWLdNVVV7nGOnbsqJiYGA0aNIgmGgAAAAAAAE4rPp3O6XQ6FRsbW268devWCg8Pr25NAAAAAAAAQFDxqYmWmpqqxx9/XMXFxa6x4uJiTZ48WampqX4rDgAAAAAAAAgGXp/OOWDAALfHn3zyic4880x16tRJkrRp0yaVlJTo8ssv92+FAAAAAAAAgMW8bqJFRka6Pb7++uvdHsfExPinIgAAAAAAACDIeN1Emz9/fiDrAAAAAAAAAIKWT9dEOyk/P1+ff/65Pv/8c+Xn5/s8z8yZMxUbG6uIiAj16NFDa9eu9eq4RYsWyeFwqH///j4/NwAAAAAAAGDGpyZaYWGhbr/9djVv3lyXXnqpLr30UrVo0UJ33HGHioqKqjTX4sWLlZaWpvT0dK1fv16dOnVSUlKS9u/fX+lxu3bt0oMPPqhevXr58hIAAAAAAAAAr/nUREtLS9Pq1av14Ycf6vDhwzp8+LDef/99rV69Wg888ECV5po2bZqGDx+u5ORktWvXTnPmzFGdOnU0b968Co8pLS3V4MGDNWnSJLVp08aXlwAAAAAAAAB4zacm2r///W/NnTtXV111lerXr6/69evr6quv1iuvvKIlS5Z4PU9JSYnWrVunxMTEPwsKCVFiYqJycnIqPO6xxx5T06ZNdccdd/hSPgAAAAAAAFAlXt9Y4K+KiooUHR1dbrxp06ZVOp3zwIEDKi0tLTdXdHS0tm7d6vGYzz//XHPnztXGjRu9eo7i4mIVFxe7HhcUFHhdHwBYifwCYEdkFwA7IrsAeMOnlWgJCQlKT0/XsWPHXGO///67Jk2apISEBL8V93dHjx7VbbfdpldeeUVRUVFeHZORkaHIyEjXFhMTE7D6AMCfyC8AdkR2AbAjsguAN3xqok2fPl3//e9/deaZZ+ryyy/X5ZdfrpiYGK1Zs0bPP/+81/NERUUpNDRUeXl5buN5eXlq1qxZuf1/+ukn7dq1S/369VNYWJjCwsL0xhtv6IMPPlBYWJh++umncseMHTtWR44ccW179uyp+gsGAAuQXwDsiOwCYEdkFwBv+HQ6Z4cOHfTjjz9q4cKFrtMuBw0apMGDB6t27dpezxMeHq74+HhlZWWpf//+kqSysjJlZWUpNTW13P5t27bVt99+6zY2btw4HT16VM8//7zHbwucTqecTmcVXh0ABAfyC4AdkV0A7IjsAuCNKjfRjh8/rrZt2+qjjz7S8OHDq11AWlqahg4dqm7duql79+6aPn26CgsLlZycLEkaMmSIWrZsqYyMDEVERKh9+/Zuxzdo0ECSyo0DAAAAAAAA/lLlJlqtWrXcroVWXQMHDlR+fr4mTJig3Nxcde7cWcuXL3fdbGD37t0KCfHprFMAAAAAAADAL3w6nTMlJUVTp07Vq6++qrAwn6Zwk5qa6vH0TUnKzs6u9NjXXnut2s8PAAAAAAAAVManDthXX32lrKwsffzxx+rQoYPq1q3r9vN33nnHL8UBAAAAAAAAwcCnJlqDBg10/fXX+7sWAAAAAAAAIChVqYlWVlamp59+Wtu2bVNJSYn+8Y9/aOLEiVW6IycAAAAAAABgN1W6Yv/kyZP1yCOPqF69emrZsqVmzJihlJSUQNUGAAAAAAAABIUqNdHeeOMNzZo1SytWrNB7772nDz/8UAsXLlRZWVmg6gMAAAAAAAAsV6Um2u7du3X11Ve7HicmJsrhcOiXX37xe2EAAAAAAABAsKhSE+2PP/5QRESE21itWrV0/PhxvxYFAAAAAAAABJMq3VjAMAwNGzZMTqfTNXbs2DHdfffdqlu3rmvsnXfe8V+FAAAAAAAAgMWq1EQbOnRoubFbb73Vb8UAAAAAAAAAwahKTbT58+cHqg4AAAAAAAAgaFXpmmgAAAAAAABATUQTDQAAAAAAADBBEw0AAAAAAAAwQRMNAAAAAAAAMEETDQAAAAAAADBBEw0AAAAAAAAwQRMNAAAAAAAAMEETDQAAAAAAADBBEw0AAAAAAAAwQRMNAAAAAAAAMBFmdQEAAAAAAAAIvNgxS6s9x64pff1QiT2xEg0AAAAAAAAwQRMNAAAAAAAAMEETDQAAAAAAADBBEw0AAAAAAAAwQRMNAAAAAAAAMEETDQAAAAAAADBBEw0AAAAAAAAwQRMNAAAAAAAAMEETDQAAAAAAADBBEw0AAAAAAAAwQRMNAAAAAAAAMEETDQAAAAAAADBBEw0AAAAAAAAwQRMNAAAAAAAAMEETDQAAAAAAADARFE20mTNnKjY2VhEREerRo4fWrl1b4b6vvPKKevXqpYYNG6phw4ZKTEysdH8AAAAAAACguixvoi1evFhpaWlKT0/X+vXr1alTJyUlJWn//v0e98/OztagQYO0atUq5eTkKCYmRldccYX27t17iisHAAAAAABATWF5E23atGkaPny4kpOT1a5dO82ZM0d16tTRvHnzPO6/cOFC3XPPPercubPatm2rV199VWVlZcrKyjrFlQMAAAAAAKCmCLPyyUtKSrRu3TqNHTvWNRYSEqLExETl5OR4NUdRUZGOHz+uRo0aefx5cXGxiouLXY8LCgqqVzQAnCLkFwA7IrsA2BHZBcAblq5EO3DggEpLSxUdHe02Hh0drdzcXK/mePjhh9WiRQslJiZ6/HlGRoYiIyNdW0xMTLXrBoBTgfwCYEdkFwA7IrsAeMPy0zmrY8qUKVq0aJHeffddRUREeNxn7NixOnLkiGvbs2fPKa4SAHxDfgGwI7ILgB2RXQC8YenpnFFRUQoNDVVeXp7beF5enpo1a1bpsc8884ymTJmiTz75RB07dqxwP6fTKafT6Zd6AeBUIr8A2BHZBcCOyC4A3rB0JVp4eLji4+Pdbgpw8iYBCQkJFR731FNP6fHHH9fy5cvVrVu3U1EqAAAAAAAAajBLV6JJUlpamoYOHapu3bqpe/fumj59ugoLC5WcnCxJGjJkiFq2bKmMjAxJ0tSpUzVhwgRlZmYqNjbWde20evXqqV69epa9DgAAAAAAAJy+LG+iDRw4UPn5+ZowYYJyc3PVuXNnLV++3HWzgd27dysk5M8Fc7Nnz1ZJSYluuOEGt3nS09M1ceLEU1k6AAAAAAAAagjLm2iSlJqaqtTUVI8/y87Odnu8a9euwBcEAAAAAAAA/IWt784JAAAAAAAAnAo00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATNNEAAAAAAAAAEzTRAAAAAAAAABM00QAAAAAAAAATYVYXAAAA7CF2zNJqz7FrSl8/VAIAAACceqxEAwAAAAAAAEywEg0AAAAAACCIcAZAcGIlGgAAAAAAAGCClWgAAABAkGDlAQAAwYuVaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAmaaAAAAAAAAICJMKsLAAAAAIBAiR2ztNpz7JrS1w+VAADsjpVoAAAAAAAAgAmaaAAAAAAAAIAJmmgAAAAAAACACZpoAAAAAAAAgAluLAAAACzDBb8BAABgF0GxEm3mzJmKjY1VRESEevToobVr11a6/9tvv622bdsqIiJCHTp00LJly05RpQAAAAAAAKiJLG+iLV68WGlpaUpPT9f69evVqVMnJSUlaf/+/R73X7NmjQYNGqQ77rhDGzZsUP/+/dW/f39t3rz5FFcOAAAAAACAmsLy0zmnTZum4cOHKzk5WZI0Z84cLV26VPPmzdOYMWPK7f/888/ryiuv1EMPPSRJevzxx7Vy5Uq9+OKLmjNnzimtHQBqGk69AwAAgJ3x91lUh6VNtJKSEq1bt05jx451jYWEhCgxMVE5OTkej8nJyVFaWprbWFJSkt577z2P+xcXF6u4uNj1+MiRI5KkgoKCalaPmqCsuKjac/C7Zq2T779hGBZXUnXBmF+B+ky0T19R7Xk3T0qq9hynk0D8t7LLnKcDsqvmqumfCXLG/uyaX2RX9djp73J2yZlAZReZ6JnX2WVYaO/evYYkY82aNW7jDz30kNG9e3ePx9SqVcvIzMx0G5s5c6bRtGlTj/unp6cbktjY2Gr4tmfPHv8E1ylEfrGxsZFdbGxsdt3sll9kFxsbm2SeXQ7DsO4rgl9++UUtW7bUmjVrlJCQ4BofPXq0Vq9erS+//LLcMeHh4Xr99dc1aNAg19isWbM0adIk5eXlldv/798olJWV6eDBg2rcuLEcDke1X0NBQYFiYmK0Z88e1a9fv9rzBZJdarVLnZJ9arVLnZL/azUMQ0ePHlWLFi0UEmL5ZSCrhPw6wS51Svap1S51Svaplez6E9l1gl3qlOxTq13qlGp2rXbNL7LrT3ap1S51Svap1S51StZll6Wnc0ZFRSk0NLRc8ysvL0/NmjXzeEyzZs2qtL/T6ZTT6XQba9Cgge9FV6B+/fpB/0t2kl1qtUudkn1qtUudkn9rjYyM9Ms8pxr55c4udUr2qdUudUr2qZXsIrv+zi51Svap1S51SjW3VjvmF9lVnl1qtUudkn1qtUud0qnPLku/GggPD1d8fLyysrJcY2VlZcrKynJbmfZXCQkJbvtL0sqVKyvcHwAAAAAAAKguy+/OmZaWpqFDh6pbt27q3r27pk+frsLCQtfdOocMGaKWLVsqIyNDkjRy5Ej17t1bzz77rPr27atFixbp66+/1ssvv2zlywAAAAAAAMBpzPIm2sCBA5Wfn68JEyYoNzdXnTt31vLlyxUdHS1J2r17t9v5qD179lRmZqbGjRunRx55ROeee67ee+89tW/f3pL6nU6n0tPTyy39DUZ2qdUudUr2qdUudUr2qtXu7PJe26VOyT612qVOyT612qXO04Fd3mu71CnZp1a71ClRK8qz0/tsl1rtUqdkn1rtUqdkXa2W3lgAAAAAAAAAsAP73C4FAAAAAAAAsAhNNAAAAAAAAMAETTQAAAAAAADABE00AAAAAAAAwARNtGqaOXOmYmNjFRERoR49emjt2rVWl+QmIyNDF154oc444ww1bdpU/fv31w8//GB1WV6ZMmWKHA6H7r//fqtLKWfv3r269dZb1bhxY9WuXVsdOnTQ119/bXVZ5ZSWlmr8+PFq3bq1ateurbPPPluPP/64guF+Ip999pn69eunFi1ayOFw6L333nP7uWEYmjBhgpo3b67atWsrMTFRP/74ozXFnoaCPbsk++ZXMGeXZI/8IrtQEbIrcMgu/wjW/CK7rBfs+UV2BQbZVX3Bll800aph8eLFSktLU3p6utavX69OnTopKSlJ+/fvt7o0l9WrVyslJUVffPGFVq5cqePHj+uKK65QYWGh1aVV6quvvtJLL72kjh07Wl1KOYcOHdLFF1+sWrVq6T//+Y++//57Pfvss2rYsKHVpZUzdepUzZ49Wy+++KK2bNmiqVOn6qmnntILL7xgdWkqLCxUp06dNHPmTI8/f+qppzRjxgzNmTNHX375perWraukpCQdO3bsFFd6+rFDdkn2zK9gzi7JPvlFdsETsitwyC7/Cdb8IrusZYf8Irv8j+zyj6DLLwM+6969u5GSkuJ6XFpaarRo0cLIyMiwsKrK7d+/35BkrF692upSKnT06FHj3HPPNVauXGn07t3bGDlypNUluXn44YeNSy65xOoyvNK3b1/j9ttvdxsbMGCAMXjwYIsq8kyS8e6777oel5WVGc2aNTOefvpp19jhw4cNp9Np/Otf/7KgwtOLHbPLMII/v4I9uwzDPvlFdsETsiswyC7/skN+kV2nnh3zi+yqPrLL/4Ihv1iJ5qOSkhKtW7dOiYmJrrGQkBAlJiYqJyfHwsoqd+TIEUlSo0aNLK6kYikpKerbt6/bextMPvjgA3Xr1k033nijmjZtqi5duuiVV16xuiyPevbsqaysLG3btk2StGnTJn3++ee66qqrLK6scjt37lRubq7b70BkZKR69OgR1J8vO7BrdknBn1/Bnl2SffKL7MLfkV2BQ3b5lx3zi+wKLLvmF9lVfWRX4FmRX2EBmbUGOHDggEpLSxUdHe02Hh0dra1bt1pUVeXKysp0//336+KLL1b79u2tLsejRYsWaf369frqq6+sLqVCO3bs0OzZs5WWlqZHHnlEX331le677z6Fh4dr6NChVpfnZsyYMSooKFDbtm0VGhqq0tJSTZ48WYMHD7a6tErl5uZKksfP18mfwTd2zC4p+PPLDtkl2Se/yC78HdkVGGSX/9kxv8iuwLJjfpFd/kF2BZ4V+UUTrQZJSUnR5s2b9fnnn1tdikd79uzRyJEjtXLlSkVERFhdToXKysrUrVs3Pfnkk5KkLl26aPPmzZozZ07QheFbb72lhQsXKjMzUxdccIE2btyo+++/Xy1atAi6WoHKBHN+2SW7JPvkF9mF0wXZ5R92yS6J/MLpgezyD7Lr9MTpnD6KiopSaGio8vLy3Mbz8vLUrFkzi6qqWGpqqj766COtWrVKZ555ptXleLRu3Trt379fXbt2VVhYmMLCwrR69WrNmDFDYWFhKi0ttbpESVLz5s3Vrl07t7Hzzz9fu3fvtqiiij300EMaM2aMbr75ZnXo0EG33XabRo0apYyMDKtLq9TJz5BdPl92YrfskoI/v+ySXZJ98ovswt+RXf5HdgWGHfOL7Aosu+UX2eU/ZFfgWZFfNNF8FB4ervj4eGVlZbnGysrKlJWVpYSEBAsrc2cYhlJTU/Xuu+/q008/VevWra0uqUKXX365vv32W23cuNG1devWTYMHD9bGjRsVGhpqdYmSpIsvvrjc7Z63bdumVq1aWVRRxYqKihQS4v4xDw0NVVlZmUUVead169Zq1qyZ2+eroKBAX375ZVB9vuzILtkl2Se/7JJdkn3yi+zC35Fd/kd2BYYd84vsCiy75BfZ5X9kV+BZkl8BuV1BDbFo0SLD6XQar732mvH9998bd911l9GgQQMjNzfX6tJcRowYYURGRhrZ2dnGvn37XFtRUZHVpXklGO+0snbtWiMsLMyYPHmy8eOPPxoLFy406tSpY7z55ptWl1bO0KFDjZYtWxofffSRsXPnTuOdd94xoqKijNGjR1tdmnH06FFjw4YNxoYNGwxJxrRp04wNGzYYP//8s2EYhjFlyhSjQYMGxvvvv2988803xrXXXmu0bt3a+P333y2u3P7skF2GYe/8CsbsMgz75BfZBU/IrsAju6ovWPOL7LKWHfKL7PI/sss/gi2/aKJV0wsvvGCcddZZRnh4uNG9e3fjiy++sLokN5I8bvPnz7e6NK8EayB++OGHRvv27Q2n02m0bdvWePnll60uyaOCggJj5MiRxllnnWVEREQYbdq0MR599FGjuLjY6tKMVatWefzdHDp0qGEYJ25XPH78eCM6OtpwOp3G5Zdfbvzwww/WFn0aCfbsMgx751ewZpdh2CO/yC5UhOwKLLKr+oI1v8gu6wV7fpFdgUF2VV+w5ZfDMAzD/+vbAAAAAAAAgNMH10QDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM0EQDAAAAAAAATNBEAwAAAAAAAEzQRAMAAAAAAABM/H9gAVp1hhOT1gAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "fig, axes = plt.subplots(1, 4, sharey=True, figsize=(15, 3))\n", + "kl0_actor = torch.load(\"models/kl_beta=0/actor.pt\")\n", + "for i in range(4):\n", + " _x = torch.concat([x, y[:, :i]], dim=1)\n", + " axes[i].bar(np.arange(11), torch.softmax((kl0_actor(_x)[0][0, -1].detach()), dim=0).tolist())\n", + " axes[i].title.set_text(f\"Input {_x[0].tolist()}\")\n", + " axes[i].set_ylim([0, 1])\n", + "axes[0].set_ylabel(\"Probability\")" + ] + }, + { + "cell_type": "markdown", + "id": "cd3a185d", + "metadata": {}, + "source": [ + "The first output is correct but the rest are all messed up. Let's look at the attention heads." + ] + }, + { + "cell_type": "markdown", + "id": "d9d6a87a", + "metadata": {}, + "source": [ + "![kl0_attn](../../../images/session_4/part_2_finetuning_lms_to_human_preferences/kl0_attn.png)" + ] + }, + { + "cell_type": "markdown", + "id": "b2abf30f", + "metadata": {}, + "source": [ + "The model seems to have learnt to not look at tokens before the last input token. We can see this in the output probability plots too, that the tokens before 5 do not have high probability.\n", + "That is only for the input though. The effect of sampling from the output on the probability distribution is harder to interpret since we do not have a way to visualize how the weights affect the probability during the forward pass." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.16" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/stylesheets/extra.css b/stylesheets/extra.css index 858b297..21c5ba1 100644 --- a/stylesheets/extra.css +++ b/stylesheets/extra.css @@ -1,3 +1,7 @@ .md-grid { max-width: 1520px; - } \ No newline at end of file + } + +.md-header { + margin-top: 10px; +} \ No newline at end of file