11-6-moral-foundation-dict.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Moral Foundation Dictionary \n",
    "\n",
    "- MoralStrength的主页：https://github.com/oaraque/moral-foundations\n",
    "- 相关文献介绍：https://www.douban.com/group/topic/284123430/\n",
    "- 原文献 Oscar Araque, Lorenzo Gatti, Kyriaki Kalimeri (2020) MoralStrength: Exploiting a moral lexicon and embedding similarity for moral foundations prediction, Knowledge-Based Systems,Volume 191,105184 https://doi.org/10.1016/j.knosys.2019.105184\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## MoralStrength: Exploiting a moral lexicon and embedding similarity for moral foundations prediction\n",
    "\n",
    "Abstract: \n",
    "\n",
    "Moral rhetoric plays a fundamental role in how we perceive and interpret the information we receive, greatly influencing our decision-making process. Especially when it comes to controversial social and political issues, our opinions and attitudes are hardly ever based on evidence alone. \n",
    "- The Moral Foundations Dictionary (MFD) was developed to operationalize moral values in the text. \n",
    "In this study, we present MoralStrength, a lexicon of approximately 1,000 lemmas, obtained as an extension of the Moral Foundations Dictionary, based on WordNet synsets. \n",
    "- Moreover, for each lemma it provides with a crowdsourced numeric assessment of Moral Valence, indicating the strength with which a lemma is expressing the specific value. \n",
    "- We evaluated the predictive potentials of this moral lexicon, defining three utilization approaches of increased complexity, ranging from lemmas’ statistical properties to a deep learning approach of word embeddings based on semantic similarity. \n",
    "- Logistic regression models trained on the features extracted from MoralStrength, significantly outperformed the current state-of-the-art, reaching an F1-score of 87.6% over the previous 62.4% (p-value <0.01), and an average F1-Score of 86.25% over six different datasets. \n",
    "\n",
    "Such findings pave the way for further research, allowing for an in-depth understanding of moral narratives in text for a wide range of social issues.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-01T10:09:19.509743Z",
     "start_time": "2023-12-01T10:09:16.624226Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: moralstrength in /opt/anaconda3/lib/python3.9/site-packages (0.2.13)\n",
      "Requirement already satisfied: scikit-learn in /opt/anaconda3/lib/python3.9/site-packages (from moralstrength) (1.3.0)\n",
      "Requirement already satisfied: spacy in /opt/anaconda3/lib/python3.9/site-packages (from moralstrength) (3.7.2)\n",
      "Requirement already satisfied: pandas in /opt/anaconda3/lib/python3.9/site-packages (from moralstrength) (1.3.4)\n",
      "Requirement already satisfied: gsitk in /opt/anaconda3/lib/python3.9/site-packages (from moralstrength) (0.2.5)\n",
      "Requirement already satisfied: numpy in /opt/anaconda3/lib/python3.9/site-packages (from moralstrength) (1.22.4)\n",
      "Requirement already satisfied: beautifulsoup4 in /opt/anaconda3/lib/python3.9/site-packages (from gsitk->moralstrength) (4.10.0)\n",
      "Requirement already satisfied: gensim in /opt/anaconda3/lib/python3.9/site-packages (from gsitk->moralstrength) (4.3.1)\n",
      "Requirement already satisfied: pyyaml in /opt/anaconda3/lib/python3.9/site-packages (from gsitk->moralstrength) (6.0)\n",
      "Requirement already satisfied: nltk in /opt/anaconda3/lib/python3.9/site-packages (from gsitk->moralstrength) (3.6.5)\n",
      "Requirement already satisfied: pytreebank in /opt/anaconda3/lib/python3.9/site-packages (from gsitk->moralstrength) (0.2.7)\n",
      "Requirement already satisfied: lxml in /opt/anaconda3/lib/python3.9/site-packages (from gsitk->moralstrength) (4.6.3)\n",
      "Requirement already satisfied: pytest in /opt/anaconda3/lib/python3.9/site-packages (from gsitk->moralstrength) (6.2.4)\n",
      "Requirement already satisfied: soupsieve>1.2 in /opt/anaconda3/lib/python3.9/site-packages (from beautifulsoup4->gsitk->moralstrength) (2.2.1)\n",
      "Requirement already satisfied: scipy>=1.7.0 in /opt/anaconda3/lib/python3.9/site-packages (from gensim->gsitk->moralstrength) (1.7.1)\n",
      "Requirement already satisfied: smart-open>=1.8.1 in /opt/anaconda3/lib/python3.9/site-packages (from gensim->gsitk->moralstrength) (5.2.1)\n",
      "Requirement already satisfied: click in /opt/anaconda3/lib/python3.9/site-packages (from nltk->gsitk->moralstrength) (8.0.3)\n",
      "Requirement already satisfied: joblib in /opt/anaconda3/lib/python3.9/site-packages (from nltk->gsitk->moralstrength) (1.3.1)\n",
      "Requirement already satisfied: regex>=2021.8.3 in /opt/anaconda3/lib/python3.9/site-packages (from nltk->gsitk->moralstrength) (2021.8.3)\n",
      "Requirement already satisfied: tqdm in /opt/anaconda3/lib/python3.9/site-packages (from nltk->gsitk->moralstrength) (4.62.3)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /opt/anaconda3/lib/python3.9/site-packages (from pandas->moralstrength) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2017.3 in /opt/anaconda3/lib/python3.9/site-packages (from pandas->moralstrength) (2021.3)\n",
      "Requirement already satisfied: six>=1.5 in /opt/anaconda3/lib/python3.9/site-packages (from python-dateutil>=2.7.3->pandas->moralstrength) (1.16.0)\n",
      "Requirement already satisfied: attrs>=19.2.0 in /opt/anaconda3/lib/python3.9/site-packages (from pytest->gsitk->moralstrength) (21.2.0)\n",
      "Requirement already satisfied: iniconfig in /opt/anaconda3/lib/python3.9/site-packages (from pytest->gsitk->moralstrength) (1.1.1)\n",
      "Requirement already satisfied: packaging in /opt/anaconda3/lib/python3.9/site-packages (from pytest->gsitk->moralstrength) (21.0)\n",
      "Requirement already satisfied: pluggy<1.0.0a1,>=0.12 in /opt/anaconda3/lib/python3.9/site-packages (from pytest->gsitk->moralstrength) (0.13.1)\n",
      "Requirement already satisfied: py>=1.8.2 in /opt/anaconda3/lib/python3.9/site-packages (from pytest->gsitk->moralstrength) (1.10.0)\n",
      "Requirement already satisfied: toml in /opt/anaconda3/lib/python3.9/site-packages (from pytest->gsitk->moralstrength) (0.10.2)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /opt/anaconda3/lib/python3.9/site-packages (from packaging->pytest->gsitk->moralstrength) (3.0.4)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/anaconda3/lib/python3.9/site-packages (from scikit-learn->moralstrength) (2.2.0)\n",
      "Requirement already satisfied: thinc<8.3.0,>=8.1.8 in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (8.2.1)\n",
      "Requirement already satisfied: setuptools in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (58.0.4)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (2.0.10)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (3.0.12)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (1.0.10)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (2.26.0)\n",
      "Requirement already satisfied: weasel<0.4.0,>=0.1.0 in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (0.3.4)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (3.0.9)\n",
      "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (1.1.2)\n",
      "Requirement already satisfied: jinja2 in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (2.11.3)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (3.3.0)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (2.0.8)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (2.4.8)\n",
      "Requirement already satisfied: typer<0.10.0,>=0.3.0 in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (0.9.0)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (2.5.2)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /opt/anaconda3/lib/python3.9/site-packages (from spacy->moralstrength) (1.0.5)\n",
      "Requirement already satisfied: typing-extensions>=4.6.1 in /opt/anaconda3/lib/python3.9/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy->moralstrength) (4.8.0)\n",
      "Requirement already satisfied: pydantic-core==2.14.5 in /opt/anaconda3/lib/python3.9/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy->moralstrength) (2.14.5)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /opt/anaconda3/lib/python3.9/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy->moralstrength) (0.6.0)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/anaconda3/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy->moralstrength) (2.0.4)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/anaconda3/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy->moralstrength) (1.26.7)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/anaconda3/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy->moralstrength) (3.2)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/anaconda3/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy->moralstrength) (2021.10.8)\n",
      "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /opt/anaconda3/lib/python3.9/site-packages (from thinc<8.3.0,>=8.1.8->spacy->moralstrength) (0.1.4)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /opt/anaconda3/lib/python3.9/site-packages (from thinc<8.3.0,>=8.1.8->spacy->moralstrength) (0.7.11)\n",
      "Requirement already satisfied: cloudpathlib<0.17.0,>=0.7.0 in /opt/anaconda3/lib/python3.9/site-packages (from weasel<0.4.0,>=0.1.0->spacy->moralstrength) (0.16.0)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /opt/anaconda3/lib/python3.9/site-packages (from jinja2->spacy->moralstrength) (1.1.1)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install moralstrength"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-01T10:15:54.485073Z",
     "start_time": "2023-12-01T10:15:50.992157Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 手动安装en_core_web_sm\n",
    "moralstrength依赖spacy，后者需要安装en_core_web_sm\n",
    "\n",
    "- 网址 https://spacy.io/models/en\n",
    "- 找到最新版本 https://github.com/explosion/spacy-models/releases/tag/en_core_web_sm-3.7.1\n",
    "- 选择下载 en_core_web_sm-3.7.1.tar.gz\n",
    "\n",
    "> pip install /path_to_it/en_core_web_sm-3.7.1.tar.gz\n",
    "\n",
    "![image.png](img/spacy.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-01T10:25:47.618941Z",
     "start_time": "2023-12-01T10:25:47.615970Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from moralstrength import lexicon_use\n",
    "\n",
    "lexicon_use.select_version(\"latest\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-01T10:26:01.324370Z",
     "start_time": "2023-12-01T10:26:01.319632Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'care': 8.799999999999999,\n",
       " 'fairness': nan,\n",
       " 'loyalty': nan,\n",
       " 'authority': nan,\n",
       " 'purity': nan}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "moralstrength.word_moral_annotations('care')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-01T10:26:06.228617Z",
     "start_time": "2023-12-01T10:26:06.215971Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.9/site-packages/sklearn/base.py:347: InconsistentVersionWarning: Trying to unpickle estimator LogisticRegression from version 1.1.3 when using version 1.3.0. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:\n",
      "https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations\n",
      "  warnings.warn(\n",
      "/opt/anaconda3/lib/python3.9/site-packages/sklearn/base.py:347: InconsistentVersionWarning: Trying to unpickle estimator CountVectorizer from version 1.1.3 when using version 1.3.0. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:\n",
      "https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.9982871348452281"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"PLS help #HASHTAG's family. No one prepares for this. They are in need of any assistance you can offer\"  \n",
    "\n",
    "moralstrength.string_moral_value(text, moral='care')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-12-01T10:26:11.003182Z",
     "start_time": "2023-12-01T10:26:10.984111Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   care  fairness  loyalty  authority  purity\n",
      "0   NaN       NaN    8.875     5.1250     NaN\n",
      "1   NaN       NaN    8.875     6.9625     NaN\n",
      "2   NaN       NaN      NaN        NaN     NaN\n",
      "3   NaN       9.0      NaN        NaN     NaN\n",
      "4   NaN       NaN      NaN        NaN     9.0\n",
      "5   8.8       NaN      NaN        NaN     NaN\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.9/site-packages/spacy/pipeline/lemmatizer.py:211: UserWarning: [W108] The rule-based lemmatizer did not find POS annotation for one or more tokens. Check that your pipeline includes components that assign token.pos, typically 'tagger'+'attribute_ruler' or 'morphologizer'.\n",
      "  warnings.warn(Warnings.W108)\n"
     ]
    }
   ],
   "source": [
    "from moralstrength.moralstrength import estimate_morals\n",
    "\n",
    "texts = '''My dog is very loyal to me.\n",
    "My cat is not loyal, but understands my authority.\n",
    "He did not want to break the router, he was fixing it.\n",
    "It is not fair! She cheated on the exams.\n",
    "Are you pure of heart? Because I am sure not.\n",
    "Will you take care of me? I am sad.'''\n",
    "\n",
    "texts = texts.split('\\n')\n",
    "\n",
    "result = estimate_morals(texts, process=True) # set to false if text is alredy pre-processed\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Chinese Moral Foundation Dictionary 2.0\n",
    "\n",
    "Chinese Moral Foundation Dictionary 2.0 for Python. This project was inspired by https://github.com/civictechlab/cmfd\n",
    "\n",
    "### Introduction\n",
    "The Chinese Moral Foundation Dictionary 2.0 (C-MFD 2.0) is a dictionary to assist the automated moral intuition detection and analysis in the Chinese language context. Starting from the existing Chinese translation of the English MFD, two experts selected additional Chinese moral concepts and used word2vec to fetch related words from an extensive Chinese dictionary. Four experts went through four-rounds of coding, followed by the validation from 202 crowd coders. The CMFD identifies not only the classic five moral foundations but also several potentially novel moral foundation candidates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-24T06:59:27.040471Z",
     "start_time": "2023-11-24T06:59:24.381430Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: cmfd in /opt/anaconda3/lib/python3.9/site-packages (0.2.1)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install cmfd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-27T02:24:29.619291Z",
     "start_time": "2023-11-27T02:24:29.614930Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```Python\n",
    "import cmfd\n",
    "\n",
    "result = cmfd.moral_quantity(text_test, duplicate=False, with_word=True)\n",
    "print(result)  \n",
    "```\n",
    "\n",
    "URLError: <urlopen error [Errno 61] Connection refused>\n",
    "\n",
    "```Python\n",
    "import pandas as pd\n",
    "df = pd.read_csv(r\"https://raw.githubusercontent.com/CivicTechLab/CMFD/main/cmfd_civictech.csv\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-27T02:16:44.012645Z",
     "start_time": "2023-11-27T02:16:43.943437Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\"./data/cmfd_civictech.csv\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-27T02:17:39.167047Z",
     "start_time": "2023-11-27T02:17:37.866226Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Building prefix dict from the default dictionary ...\n",
      "Dumping model to file cache /var/folders/8b/hhnbt0nd4zsg2qhxc28q23w80000gn/T/jieba.cache\n",
      "Loading model cost 1.067 seconds.\n",
      "Prefix dict has been built successfully.\n"
     ]
    }
   ],
   "source": [
    "import jieba\n",
    "\n",
    "moral_dict = {}\n",
    "for word in set(df['chinese'].tolist()):\n",
    "    jieba.add_word(word)\n",
    "\n",
    "for category, df_item in df.groupby(by=['foundation']):\n",
    "    moral_dict[category] = df_item['chinese'].tolist()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-24T06:59:29.111823Z",
     "start_time": "2023-11-24T06:59:29.108759Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "text_test = \"\"\" 她一见他就倾心， 但他却偏要为非作歹。 结果两人败俱伤， 她心碎， 他惨遭报应。 他本来是一个英勇战斗的将军， 但因为一纸空文， 被诬陷入狱。 再加上尔虞我诈的诡计， 他最终丧失了自己的荣誉和尊严。 他曾认为自己是无所不能， 却不知道一念之间， 一切都会化为乌有。 如果他当初没有欺骗她， 如果他当初没有撕毁那张契约， 也许他们现在还在幸福的生活中。 现在， 他身陷囹圄， 只能思念那些曾经的日子。 他懊悔不已， 只能悔恨自己的软弱和自私。 他希望能有机会改正错误， 重新取回自己的荣誉， 但现实却是那么残酷。\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-27T02:18:03.144201Z",
     "start_time": "2023-11-27T02:18:03.130945Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def get_moral_dict():\n",
    "    return moral_dict\n",
    "\n",
    "\n",
    "def moral_quantity(text, duplicate=True, normalize=True, with_word=False):\n",
    "    \"\"\"\n",
    "     Calculate the number or proportion of moral dictionaries in Chinese text\n",
    "    :param text: chinese text\n",
    "    :param duplicate: keep repeated moral words\n",
    "    :param normalize: calculate the ratio\n",
    "    :param with_word: output with moral words\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    if isinstance(text, str):\n",
    "        # The total number of matched moral words\n",
    "        word_total = 0\n",
    "        # dict of moral word\n",
    "        moral_word = {}\n",
    "        # Store the number corresponding to the moral vocabulary\n",
    "        moral_num = {}\n",
    "        if duplicate:\n",
    "            for key in moral_dict.keys():\n",
    "                moral_word[key] = []\n",
    "            for word in jieba.cut(text):\n",
    "                for key in moral_dict.keys():\n",
    "                    if word in moral_dict[key]:\n",
    "                        moral_word[key].append(word)\n",
    "        else:\n",
    "            for key in moral_dict.keys():\n",
    "                moral_word[key] = set()\n",
    "\n",
    "            for word in jieba.cut(text):\n",
    "                for key in moral_dict.keys():\n",
    "                    if word in moral_dict[key]:\n",
    "                        moral_word[key].add(word)\n",
    "\n",
    "        for key in moral_word.keys():\n",
    "            word_total += len(moral_word[key])\n",
    "\n",
    "        if word_total == 0:\n",
    "            return None\n",
    "\n",
    "        if normalize:\n",
    "            for key in moral_word.keys():\n",
    "                moral_num[key] = len(moral_word[key]) / word_total\n",
    "        else:\n",
    "            moral_num[key] = len(moral_word[key])\n",
    "        if with_word:\n",
    "            moral_word_num = {}\n",
    "            for key in moral_dict.keys():\n",
    "                moral_word_num[key] = {}\n",
    "                moral_word_num[key]['num'] = moral_num[key]\n",
    "                moral_word_num[key]['word'] = \";\".join(moral_word[key])\n",
    "            return moral_word_num\n",
    "        else:\n",
    "            return moral_num\n",
    "    return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-27T02:18:14.441256Z",
     "start_time": "2023-11-27T02:18:14.360834Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'altr': 0.0,\n",
       " 'auth': 0.26666666666666666,\n",
       " 'care': 0.26666666666666666,\n",
       " 'dili': 0.0,\n",
       " 'fair': 0.26666666666666666,\n",
       " 'general': 0.06666666666666667,\n",
       " 'libe': 0.0,\n",
       " 'loya': 0.06666666666666667,\n",
       " 'mode': 0.0,\n",
       " 'resi': 0.0,\n",
       " 'sanc': 0.06666666666666667,\n",
       " 'wast': 0.0}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "moral_quantity(text_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "幻灯片",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}