#
tiktoken
Here are 62 public repositories matching this topic...
Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion
github pdf repository text doi ipynb pull-request arxiv papers pmid sci-hub youtube-transcript-api llm tiktoken
-
Updated
Dec 19, 2024 - Python
An elegant LLM chat UI forked from chatgpt-demo of @anse-app. Index site at https://free-chat.asia
-
Updated
Dec 19, 2024 - Svelte
Analysis of OpenAI's ChatGPT
chat analysis plugins openai language-model gpt-4 chatgpt text-davinci-002-render tiktoken text-davinci-002-render-sha gpt-35-turbo-0301 gpt-35-turbo chat-markup-language cl100kbase text-davinci-002-plugins gpt-4-0613 gpt-35-turbo-16k-0613 gpt-4-32k-0613 gpt-35-turbo-1106
-
Updated
Jan 28, 2024 - Jupyter Notebook
This project implements token calculation for OpenAI's gpt-4 and gpt-3.5-turbo model, specifically using `cl100k_base` encoding.
encoding ai csharp tokens openai gpt4 chatgpt langchain tiktoken gpt35turbo cl100kbase tiktoken-sharp p50kbase langchain-dotnet
-
Updated
Dec 16, 2024 - C#
The ChatGPT Long Term Memory package is a powerful tool designed to empower your projects with the ability to handle a large number of simultaneous users and external sources.
python redis memory vector chatbot context embeddings openai text-summarization datastore similarity-search text-retrieval long-term-memory gpt-3 chatgpt chatgpt-api tiktoken llama-index gpt-35-turbo embedding-similarity
-
Updated
Jul 31, 2023 - Python
A custom GPT based on [Zero To Hero](https://karpathy.ai/zero-to-hero.html) utilizing tiktoken with the intent to augment AI Transformer-model education and reverse engineer GPT models from scratch.
machine-learning ai hack chatbot reverse-engineering transformers hacking artificial-intelligence cybersecurity gpt cyber transformer-architecture gpt-2 transformer-models chatgpt chatgpt-api chatgpt-app tiktoken generative-pre-trained-transformer
-
Updated
Jun 19, 2023 - Python
Predict the exact openai token usage of functions
tokenizer tokens openai gpt-4 chatgpt tiktoken gpt-35-turbo function-calling openai-functions openai-function-call
-
Updated
Sep 20, 2023 - Python
🧰 The AutoTokenizer that TikToken always needed -- Load any tokenizer with TikToken now! ✨
-
Updated
Dec 11, 2024 - Python
📐 GPT token estimation and context size utilities without a full tokenizer
-
Updated
Nov 29, 2024 - TypeScript
PHP port for openai/tiktoken (most)
-
Updated
Aug 25, 2024 - PHP
Improve this page
Add a description, image, and links to the tiktoken topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the tiktoken topic, visit your repo's landing page and select "manage topics."