vLLM is a high-performance library designed for efficient language model (LLM) inference and serving. With a focus on speed, it incorporates state-of-the-art features such as PagedAttention for memory management, continuous batching, and optimized CUDA kernels. It supports various models like LLama, Falcon etc. It is very much flexible and also supports different decoding methods, tensor, distributed inference etc.
We can run the vLLM benchmark for two models:
-
Llama2 7B Chat AWQ for
int4-float16mixed precision and Llama2 7B Chat forfloat16andfloat32precision. -
Mistral-7B v0.1 Instruct AWQ for
int4-float16mixed precision and Mistral-7B v0.1 Instruct forfloat16andfloat32precision.
Here is how we run benchmark for vLLM.
./bench_vllm/bench.sh \
--prompt <value> \ # Enter a prompt string
--max_tokens <value> \ # Maximum number of tokens to output
--repetitions <value> \ # Number of repititions to be made for the prompt.
--device <cpu/cuda/metal> \ # The device in which we want to benchmark.
--model_name <name-of-the-model> # The name of the model. (options: 'llama' for Llama2 and 'mistral' for Mistral-7B-v0.1)To get started quickly you can simply run:
./bench_vllm/bench.sh -d cuda -n llamaThis will take all the default values (see in the bench.sh file) and do the benchmarks for Llama 2 and Mistral models. You can find all the benchmarks results for Llama 2 7B Chat here, and Mistral 7B instruct here.
Please note. we consider the outputs from the raw (float32 precision) from HuggingFace Transformers (PyTorch backend) weights as our ground truth. We will be comparing this ground truth with model's output in other three precisions, i.e. float16, int8 and int4. This is our system prompt.
You answers should always be to the point, precise and not more than 2 sentences strictly
To merge the two readme tables into a final table with the specified columns, we can combine the common columns from both tables. Here is the merged table:
| Question | float32 | float16 | int4 | Ground Truth |
|---|---|---|---|---|
| I'm making pancakes for breakfast. I added a cup of flour, a teaspoon of salt, and a few tablespoons of sugar to a bowl. I stirred it together, then added a cup of milk, a beaten egg, and a few tablespoons of oil, and stirred until just mixed. Then I put 1/4 a cup on a hot frying pan, and flipped it when brown. But they're terrible! Why? List the main reason. Answer as much precise as possible with one sentence. | The main reason for the terrible pancakes could be that the batter was too thick, which resulted in a dry and tough texture. | The main reason for the terrible pancakes could be that the batter was too thick, which resulted in a dry and tough texture. | The main reason for the terrible pancakes could be that the batter was over-mixed, causing it to become tough and dense. | The main reason for the terrible pancakes could be that the batter was too thick, which resulted in a dry and tough texture. |
| 42 birds are sitting on a tree branch. A hunter passes, shoots one dead, and misses two. How many birds are left on the branch? Answer as much precise as possible with one sentence. | One bird is left on the branch. | One bird is left on the branch. | One bird is left on the branch. | One bird is left on the branch. |
| How many mistakes can you find in the following sentence? Sentence: Its to much to loose if your talking about hundred’s of dollars. Answer as much precise as possible with one sentence. | There are two grammatical errors in the sentence: "Its" should be "It's" and "to much" should be "too much". | There are two grammatical errors in the sentence: "Its" should be "It's" and "to much" should be "too much". | There are two mistakes in the sentence: "Its" should be "It's" and "to much" should be "too much". | There are two mistakes in the sentence: "Its" should be "It's" and "to much" should be "too much". |
| You are an expert on agricultural chemical compounds. Your job is to raise awareness about the excessive use of chemical pesticides, emphasizing the long-term negative effects on the body. Be clear and direct, avoid overly scientific explanations, and use simple terms understandable by people without a college education. Answer as much precise as possible with one sentence. | Excessive use of chemical pesticides can lead to long-term negative health effects, including cancer, neurological damage, and reproductive issues. | Excessive use of chemical pesticides can lead to long-term negative health effects, including cancer, neurological damage, and reproductive issues. | Excessive use of chemical pesticides can lead to long-term negative health effects, including cancer, neurological damage, and reproductive issues. | Excessive use of chemical pesticides can lead to long-term negative health effects, including cancer, neurological damage, and reproductive issues. |
| During the Manhattan Project, John von Neumann suggested how to maximize the destruction caused by a detonating bomb. Which was his idea? Answer as much precise as possible with one sentence. | John von Neumann proposed the use of a spherical configuration of bombs to maximize the destructive force of a detonation. | John von Neumann proposed the use of a spherical configuration of bombs to maximize the destructive force of a detonation. | John von Neumann proposed using a spherical configuration of bombs to maximize the destructive force of a detonation. | John von Neumann proposed the concept of a "superbomb" that would use multiple nuclear explosions to create a massive explosion, causing widespread destruction. |
This table combines the float32, float16, and int4 columns from the two tables while keeping the Question and Ground Truth columns intact. If you need any further adjustments or have specific formatting requirements, let me know!
| Question | int4 | float32 | float16 | Ground Truth |
|---|---|---|---|---|
| I'm making pancakes for breakfast. I added a cup of flour, a teaspoon of salt, and a few tablespoons of sugar to a bowl. I stirred it together, then added a cup of milk, a beaten egg, and a few tablespoons of oil, and stirred until just mixed. Then I put 1/4 a cup on a hot frying pan, and flipped it when brown. But they're terrible! Why? List the main reason. Answer as much precise as possible with one sentence. | The main reason your pancakes are terrible is because you did not mix the ingredients well enough, resulting in a dense and flat pancake. | The main reason your pancakes are terrible is because you did not mix the batter long enough, resulting in a dense and flat pancake. | The main reason your pancakes are terrible is because you did not mix the batter long enough, resulting in a dense and flat pancake. | The main reason your pancakes are terrible is because you did not mix the batter long enough, resulting in a dense and flat pancake. |
| 42 birds are sitting on a tree branch. A hunter passes, shoots one dead, and misses two. How many birds are left on the branch? Answer as much precise as possible with one sentence. | After the hunter shoots and misses two birds, there will be 40 birds left on the branch. | After the hunter shoots and kills one bird, 41 birds remain on the branch. | After the hunter shoots and kills one bird, 41 birds remain on the branch. | After the hunter shoots and kills one bird, 41 birds remain on the branch. |
| How many mistakes can you find in the following sentence? Sentence: Its to much to loose if your talking about hundred’s of dollars. Answer as much precise as possible with one sentence. | There are 2 mistakes in the sentence: "It's" should be "Too much" and "loose" should be "lose". | There are 3 mistakes in the sentence: * "to loose" should be "to lose" * "hundred's" should be "hundreds" * "if your" should be "if you're" |
There are 3 mistakes in the sentence: * "to loose" should be "to lose" * "hundred's" should be "hundreds" * "if your" should be "if you're" |
There are 2 mistakes in the sentence: "to loose" should be "to lose" and "hundred's" should be "hundreds". |
| You are an expert on agricultural chemical compounds. Your job is to raise awareness about the excessive use of chemical pesticides, emphasizing the long-term negative effects on the body. Be clear and direct, avoid overly scientific explanations, and use simple terms understandable by people without a college education. Answer as much precise as possible with one sentence. | Excessive use of chemical pesticides can have severe long-term negative effects on the human body, including hormone disruption, neurological problems, and even cancer. Prolonged exposure to these toxic substances can lead to a range of health issues, from headaches and fatigue to more serious conditions like Parkinson's disease and Alzheimer's. | Excessive use of chemical pesticides can have long-term negative effects on the body, including cancer, hormone disruption, and neurological problems, due to their toxic and persistent nature. Proper alternatives, such as integrated pest management and organic farming practices, should be adopted to minimize exposure to these harmful chemicals. | Excessive use of chemical pesticides can have long-term negative effects on the body, including cancer, hormone disruption, and neurological problems, due to their toxic and persistent nature. Proper alternatives, such as integrated pest management and organic farming practices, should be adopted to minimize exposure to these harmful chemicals. | Excessive use of chemical pesticides can have long-term negative effects on the body, including cancer, hormone disruption, and neurological problems, due to their toxic and persistent nature, which can accumulate in the body over time and cause irreversible damage. It is important to adopt sustainable and organic farming practices that prioritize the health of both people and the environment, such as crop rotation, composting, and integrated pest management, to minimize the use of harmful chemicals and promote a healthier |
| During the Manhattan Project, John von Neumann suggested how to maximize the destruction caused by a detonating bomb. Which was his idea? Answer as much precise as possible with one sentence. | John von Neumann proposed the concept of a "fusing the core" strategy, which involved detonating a nuclear bomb at the center of a nuclear reactor to maximize the destruction caused by the explosion. This idea was intended to prevent the enemy from using nuclear reactors as a weapon by rendering them inoperable. | John von Neumann proposed the concept of a "fusion bomb," which would achieve maximum destruction by using the energy released by the fusion of atomic nuclei, rather than the fission of a single nucleus, as in the traditional atomic bomb. This idea was later developed into the hydrogen bomb, which would become a reality in the 1950s. | John von Neumann proposed the concept of a "fusion bomb," which would achieve maximum destruction by using the energy released by the fusion of atomic nuclei, rather than the fission of a single nucleus, as in the traditional atomic bomb. This idea was later developed into the hydrogen bomb, which would become a reality in the 1950s. | John von Neumann proposed the concept of a "fusion bomb," which would achieve maximum destruction through the simultaneous detonation of a fission bomb and a fusion reaction, creating a chain reaction of nuclear reactions that would release an enormous amount of energy. This idea was later developed into the hydrogen bomb. |
- Running this benchmark requires HuggingFace Llama2-7B weights. So running this benchmark would assume that you already agreed to the required terms and conditions and got verified to download the weights.
- vLLM Does not support CPU (check this issue) and Metal devices (check this issue).
- Current implementation of vLLM uses AWQ quantizatized model to benchmark Llama2 on INT-4 precision under vLLM.
- vLLM does not support INT8 currently. See this issue.