Updates

tutorials/semantic_chunking.ipynb

@@ -79,15 +79,15 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 65,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Number of chunks: 4790\n",
-      "Time taken: 2.1546314590086695\n"
+      "Number of chunks: 6148\n",
+      "Time taken: 2.2917541670030914\n"
      ]
     }
    ],
@@ -116,22 +116,22 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 64,
+   "execution_count": 66,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      " Having disentangled his leg, he rose. \n",
+      " ”    “I can’t think what the servants are about,” said the countess, turning  to her husband. \n",
       "\n",
-      " Prince Andrew arrived at Bennigsen’s quarters—a country gentleman’s  house of moderate size, situated on the very banks of the river. Neither  Bennigsen nor the Emperor was there, but Chernýshev, the Emperor’s  aide-de-camp, received Bolkónski and informed him that the Emperor,  accompanied by General Bennigsen and Marquis Paulucci, had gone a second  time that day to inspect the fortifications of the Drissa camp, of the  suitability of which serious doubts were beginning to be felt. Chernýshev was sitting at a window in the first room with a French novel  in his hand. This room had probably been a music room; there was still  an organ in it on which some rugs were piled, and in one corner stood  the folding bedstead of Bennigsen’s adjutant. This adjutant was also  there and sat dozing on the rolled-up bedding, evidently exhausted by  work or by feasting. Two doors led from the room, one straight on into  what had been the drawing room, and another, on the right, to the study. Through the first door came the sound of voices conversing in German  and occasionally in French. In that drawing room were gathered, by  the Emperor’s wish, not a military council (the Emperor preferred  indefiniteness), but certain persons whose opinions he wished to know in  view of the impending difficulties. It was not a council of war, but,  as it were, a council to elucidate certain questions for the Emperor  personally. To this semicouncil had been invited the Swedish General  Armfeldt, Adjutant General Wolzogen, Wintzingerode (whom Napoleon had  referred to as a renegade French subject), Michaud, Toll, Count Stein  who was not a military man at all, and Pfuel himself, who, as Prince  Andrew had heard, was the mainspring of the whole affair. Prince Andrew  had an opportunity of getting a good look at him, for Pfuel arrived soon  after himself and, in passing through to the drawing room, stopped a  minute to speak to Chernýshev. At first sight, Pfuel, in his ill-made uniform of a Russian general,  which fitted him badly like a fancy costume, seemed familiar to Prince  Andrew, though he saw him now for the first time. \n",
+      " He was received in the best houses not merely as a doctor, but  as an equal. \n",
       "\n",
-      " She got up quickly just as Nicholas entered, almost ran to the door  which was hidden by curtains, struck her head against it, and rushed  from the room with a moan either of pain or sorrow. Pierre gazed at the door through which she had disappeared and did not  understand why he suddenly felt all alone in the world. Princess Mary roused him from his abstraction by drawing his attention  to her nephew who had entered the room. At that moment of emotional tenderness young Nicholas’ face, which  resembled his father’s, affected Pierre so much that when he had kissed  the boy he got up quickly, took out his handkerchief, and went to the  window. He wished to take leave of Princess Mary, but she would not let  him go. “No, Natásha and I sometimes don’t go to sleep till after two, so please  don’t go. Go downstairs, we will come immediately. ”    Before Pierre left the room Princess Mary told him: “This is the first  time she has talked of him like that. ”            CHAPTER XVII    Pierre was shown into the large, brightly lit dining room; a few minutes  later he heard footsteps and Princess Mary entered with Natásha. Natásha  was calm, though a severe and grave expression had again settled on her  face. They all three of them now experienced that feeling of awkwardness  which usually follows after a serious and heartfelt talk. It is  impossible to go back to the same conversation, to talk of trifles is  awkward, and yet the desire to speak is there and silence seems like  affectation. They went silently to table. The footmen drew back the  chairs and pushed them up again. Pierre unfolded his cold table napkin  and, resolving to break the silence, looked at Natásha and at Princess  Mary. They had evidently both formed the same resolution; the eyes of  both shone with satisfaction and a confession that besides sorrow life  also has joy. “Do you take vodka, Count? ” asked Princess Mary, and those words  suddenly banished the shadows of the past. “Now tell us about yourself,”  said she. “One hears such improbable wonders about you. ”    “Yes,” replied Pierre with the smile of mild irony now habitual to him. “They even tell me wonders I myself never dreamed of! Mary Abrámovna  invited me to her house and kept telling me what had happened, or ought  to have happened, to me. \n",
+      " “That’s enough, Natásha,” said Sónya. \n",
       "\n",
-      " ” Murat again interrupted him, “with all my heart I  wish the Emperors may arrange the affair between them, and that the war  begun by no wish of mine may finish as quickly as possible! ” said he,  in the tone of a servant who wants to remain good friends with another  despite a quarrel between their masters. And he went on to inquiries about the Grand Duke and the state of his  health, and to reminiscences of the gay and amusing times he had spent  with him in Naples. Then suddenly, as if remembering his royal dignity,  Murat solemnly drew himself up, assumed the pose in which he had stood  at his coronation, and, waving his right arm, said:    “I won’t detain you longer, General. I wish success to your mission,”  and with his embroidered red mantle, his flowing feathers, and his  glittering ornaments, he rejoined his suite who were respectfully  awaiting him. Balashëv rode on, supposing from Murat’s words that he would very soon  be brought before Napoleon himself. But instead of that, at the next  village the sentinels of Davout’s infantry corps detained him as  the pickets of the vanguard had done, and an adjutant of the corps  commander, who was fetched, conducted him into the village to Marshal  Davout. CHAPTER V    Davout was to Napoleon what Arakchéev was to Alexander—though not a  coward like Arakchéev, he was as precise, as cruel, and as unable to  express his devotion to his monarch except by cruelty. In the organism of states such men are necessary, as wolves are  necessary in the organism of nature, and they always exist, always  appear and hold their own, however incongruous their presence and their  proximity to the head of the government may be. This inevitability alone  can explain how the cruel Arakchéev, who tore out a grenadier’s mustache  with his own hands, whose weak nerves rendered him unable to face  danger, and who was neither an educated man nor a courtier, was able to  maintain his powerful position with Alexander, whose own character was  chivalrous, noble, and gentle. Balashëv found Davout seated on a barrel in the shed of a peasant’s hut,  writing—he was auditing accounts. \n",
+      " Pierre did not catch what they were saying, but knew they were talking  about him. He reddened and turned away. “Well, now to the health of handsome women! ” said Dólokhov, and  with a serious expression, but with a smile lurking at the corners of  his mouth, he turned with his glass to Pierre. “Here’s to the health of lovely women, Peterkin—and their  lovers! Pierre, with downcast eyes, drank out of his glass without looking at  Dólokhov or answering him. The footman, who was distributing leaflets  with Kutúzov’s cantata, laid one before Pierre as one of the  principal guests. He was just going to take it when Dólokhov, leaning  across, snatched it from his hand and began reading it. Pierre looked  at Dólokhov and his eyes dropped, the something terrible and monstrous  that had tormented him all dinnertime rose and took possession of him. He leaned his whole massive body across the table. “How dare you take it? Hearing that cry and seeing to whom it was addressed, Nesvítski and the  neighbor on his right quickly turned in alarm to Bezúkhov. \n",
       "\n",
-      " “I expect you have guessed that Prince Vasíli has  not come and brought his pupil with him” (for some reason Prince  Bolkónski referred to Anatole as a “pupil”) “for the sake of my  beautiful eyes. Last night a proposition was made me on your account  and, as you know my principles, I refer it to you. ”    “How am I to understand you, mon père? ” said the princess, growing  pale and then blushing. ” cried her father angrily. “Prince Vasíli  finds you to his taste as a daughter-in-law and makes a proposal to you  on his pupil’s behalf. That’s how it’s to be understood! ”    “I do not know what you think, Father,” whispered the princess. Leave me out of the question. I’m not going to  get married. That’s what I want to know. ”    The princess saw that her father regarded the matter with disapproval,  but at that moment the thought occurred to her that her fate would be  decided now or never. She lowered her eyes so as not to see the gaze  under which she felt that she could not think, but would only be able to  submit from habit, and she said: “I wish only to do your will, but if  I had to express my own desire. ” She had no time to finish. The old  prince interrupted her. “He will take you with your  dowry and take Mademoiselle Bourienne into the bargain. She’ll be the  wife, while you. He saw the effect these words had produced on his  daughter. She lowered her head and was ready to burst into tears. “Now then, now then, I’m only joking! “Remember this,  Princess, I hold to the principle that a maiden has a full right to  choose. Only remember that your life’s happiness  depends on your decision. ”    “But I do not know, Father! ”    “There’s no need to talk! He receives his orders and will marry you  or anybody; but you are free to choose. Go to your room, think it  over, and come back in an hour and tell me in his presence: yes or no. I know you will pray over it. Well, pray if you like, but you had better  think it over. \n",
+      " The  whole street was full of clouds of black smoke. Tongues of flame here  and there broke through that cloud. A great number of people crowded in  front of the conflagration. In the middle of the street stood a French  general saying something to those around him. Pierre, accompanied by the  maid, was advancing to the spot where the general stood, but the French  soldiers stopped him. ” * cried a voice. ”      “This way, uncle,” cried the girl. “We’ll pass through the side street,  by the Nikúlins’! ”    Pierre turned back, giving a spring now and then to keep up with her. She ran across the street, turned down a side street to the left, and,  passing three houses, turned into a yard on the right. “It’s here, close by,” said she and, running across the yard, opened a  gate in a wooden fence and, stopping, pointed out to him a small wooden  wing of the house, which was burning brightly and fiercely. One of its  sides had fallen in, another was on fire, and bright flames issued from  the openings of the windows and from under the roof. \n",
       "\n"
      ]
     }
@@ -154,14 +154,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 55,
+   "execution_count": 67,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Time taken: 1.4906437500030734\n"
+      "Time taken: 1.5817922909918707\n"
      ]
     }
    ],
@@ -188,7 +188,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 58,
+   "execution_count": 68,
    "metadata": {},
    "outputs": [
     {