deploy: 2548822

_next/data/yuHU6dymGkcGnsEn7YhW4/blog/2024-05-17-category-hard.json

@@ -0,0 +1 @@
+{"pageProps":{"frontmatter":{"title":"Introducing New Chatbot Arena Category: Hard Prompts","author":"Tianle Li, Wei-Lin Chiang","date":"May 17, 2024","previewImg":"/images/blog/category_hard/preview.png"},"content":"\nWe are thrilled to introduce a new category on Chatbot Arena: Hard Prompts. \n\n[Motivations]\n\nThrough our [Arena-Hard](https://lmsys.org/blog/2024-04-19-arena-hard/) pipeline, we have identified a collection of high-quality prompts from existing Chatbot Arena battles. Each user prompt is evaluated against the 7 Key Criteria defined in Table X, using Llama-3-70B-Instruct as judge. The 7 Key Criteria are:\n\n<table style=\"width:100%; border-collapse: collapse; border: 1px solid black;\">\n  <tr style=\"background-color: black; color: white;\">\n    <!-- <th style=\"border: 1px solid black; padding: 10px; text-align: left;\">7 Key \"Hardness\" Criteria</th> -->\n  </tr>\n  <tr>\n    <td style=\"border: 1px solid black; padding: 10px; text-align: left;\"><strong>1. Specificity:</strong> Does the prompt ask for a specific output?</td>\n  </tr>\n  <tr>\n    <td style=\"border: 1px solid black; padding: 10px; text-align: left;\"><strong>2. Domain Knowledge:</strong> Does the prompt cover one or more specific domains?</td>\n  </tr>\n  <tr>\n    <td style=\"border: 1px solid black; padding: 10px; text-align: left;\"><strong>3. Complexity:</strong> Does the prompt have multiple levels of reasoning, components, or variables?</td>\n  </tr>\n  <tr>\n    <td style=\"border: 1px solid black; padding: 10px; text-align: left;\"><strong>4. Problem-Solving:</strong> Does the prompt directly involve the AI to demonstrate active problem-solving skills?</td>\n  </tr>\n  <tr>\n    <td style=\"border: 1px solid black; padding: 10px; text-align: left;\"><strong>5. Creativity:</strong> Does the prompt involve a level of creativity in approaching the problem?</td>\n  </tr>\n  <tr>\n    <td style=\"border: 1px solid black; padding: 10px; text-align: left;\"><strong>6. Technical Accuracy:</strong> Does the prompt require technical accuracy in the response?</td>\n  </tr>\n  <tr>\n    <td style=\"border: 1px solid black; padding: 10px; text-align: left;\"><strong>7. Real-world Application:</strong> Does the prompt relate to real-world applications?</td>\n  </tr>\n</table>\n\nA Hardness Score is then calculated using the how many criteria are satisfied. Prompts that satisfy 6 or more of these hardness criteria are then designated as part of the \"Hard\" category and featured on a dedicated leaderboard. We present the distribution of the criteria and hardness score in Figure 1 and 2. We also present several example prompts with labeled criteria in [Example Section](#example).\n\n<img src=\"/images/blog/category_hard/key_criteria_breakdown.png\" style=\"display:block; margin-top: auto; margin-left: auto; margin-right: auto; margin-bottom: auto; width: 85%\"></img>\n<p style=\"color:gray; text-align: center;\">Figure 1. The percentage of each criteria within 1 million Chatbot Arena data.</p>\n\n<img src=\"/images/blog/category_hard/hardness_breakdown.png\" style=\"display:block; margin-top: auto; margin-left: auto; margin-right: auto; margin-bottom: auto; width: 85%\"></img>\n<p style=\"color:gray; text-align: center;\">Figure 2. The percentage of prompts with different hardness score within 1 million Chatbot Arena data.</p>\n\nWe are launching the Hard Prompts category for All Languages and English Only, but we are working to expand this offering to other languages as well. For viewing of the full leaderboard, check out (link).\n\nThe results from the Hard Prompts (English) category, as shown in Table X, reveal some notable ranking differences. Specifically, we observe that the Llama-3-8B-Instruct model, which had previously performed on par with GPT-4-0314 on the general English leaderboard, has seen a significant drop in ranking within the Hard Prompts (English) category. This suggests that the Llama-3-8B-Instruct may struggle with the increased complexity and difficulty of the prompts in this new specialized category. We also observe improvement in performance among top proprietary models, such as GPT-4-Turbo, Claude-3-Opus, Claude-3-Sonnet, and GPT-4.\n\n<img src=\"/images/blog/category_hard/elo_comparison_1.png\" style=\"display:block; margin-top: auto; margin-left: auto; margin-right: auto; margin-bottom: auto; width: 85%\"></img>\n<p style=\"color:gray; text-align: center;\">Figure 3. Comparison between Chatbot Arena Category English vs Hard Prompts (English). We set gpt-4-0314 as anchor when computing elo.</p>\n\n<img src=\"/images/blog/category_hard/elo_comparison_2.png\" style=\"display:block; margin-top: auto; margin-left: auto; margin-right: auto; margin-bottom: auto; width: 85%\"></img>\n<p style=\"color:gray; text-align: center;\">Figure 4. Comparison between Chatbot Arena Category English vs Hard Prompts (English). We set mixtral-8x7b-instruct-v0.1 as anchor when computing elo.</p>\n\n## Future\nWe are committed to continually enhancing the Chatbot Arena experience for our users. We look forward to seeing how the latest advancements in language models perform on these challenging prompts, and to sharing these insights with the broader community.\n\n## Citation\n```\n@misc{arenacategoryhard2024,\n    title = {Introducing New Chatbot Arena Category: Hard Prompts},\n    url = {https://lmsys.org/blog/2024-05-17-category-hard/},\n    author = {Tianle Li, Wei-Lin Chiang},\n    month = {May},\n    year = {2024}\n}\n```\n\n## Example\nWe present 10 examples of user prompt with criteria labeled by Llama-3-70B-Instruct, in increasing hardness. The labeled criteria are inside the bracket.\n\n**Prompt 1:**\n\n[None]\n\nhello\n\n\n**Prompt 2:**\n\n[Real World]\n\nwhat is cake\n\n\n**Prompt 3:**\n\n[Creativity, Real World]\n\nHow to pickup a girl?\n\n\n**Prompt 4:**\n\n[Specificity, Creativity, Real World]\n\nwriten ten different sentences that end with word \"apple\"\n\n\n**Prompt 5:**\n\n[Specificity, Creativity, Real World]\n\nWriting prompt: write the start of a short story / a man with an iphone is transported back to 1930s USA. \n\n\n**Prompt 6:** \n\n[Specificity, Domain Knowledge, Complexity, Problem-solving, Technical Accuracy, Real World]\n\ntell me how to make a hydroponic nutrient solution at home to grow lettuce with precise amount of each nutrient\n\n\n**Prompt 7:** \n\n[Specificity, Domain Knowledge, Complexity, Problem-solving, Technical Accuracy, Real World]\n\nSolve the integral $\\int_{-\\infty}^{+\\infty} exp(-x^2) dx $ step-by-step with detailed explanation\n\n\n**Prompt 8:** \n\n[Specificity, Domain Knowledge, Complexity, Problem-solving, Technical Accuracy, Real World]\n\nwrite me GLSL code which can gennrate at least 5 colors and 2 waves of particles cross each other\t\n\n\n**Prompt 9:**\n\n[Specificity, Domain Knowledge, Complexity, Problem-solving, Technical Accuracy, Real World]\n\nMy situation is this: I’m setting up a server running at home Ubuntu to run an email server and a few other online services. As we all know, for my email to work reliably and not get blocked I need to have an unchanging public IP address. Due to my circumstances I am not able to get a static IP address through my ISP or change ISPs at the moment.\n\nThe solution I have found is to buy a 4G SIM card with a static IP (from an ISP that offers that), which I can then use with a USB dongle. However this 4G connection costs me substantially per MB to use.\n\nBut. Mail is the only server that needs a static IP address. For everything else using my home network connection and updating my DNS records with DDNS would be fine. I have tested this setup previously for other services and it has worked.\n\nSo. I was wondering. Would it in theory be possible to: connect the server to two network interfaces at the same time and route traffic depending on destination port. I.e. all outgoing connections to ports 25, 465, 587, and possibly 993 should be sent through the 4G dongle interface (enx344b50000000) and all other connections sent over eth0. Similarly, the server should listen for incoming connections on the same ports on enx344b50000000 and listen on all other ports (if allowed by ufw) on eth0.\n\nI would then need DNS records from mail.mydomain.tld —> <4g static public IP> and mydomain.tld —> <home public IP> (updated with DDNS, and NAT configured on my home router).\n\nComputers on the internet would then be able to seamlessly connect to these two IP addresses, not “realising” that they are in fact the same machine, as long as requests to mail.mydomain.tld are always on the above mentioned ports.\n\nQuestion: Is this possible? Could it be a robust solution that works the way I hope? Would someone be able to help me set it up?\n\nI have come across a few different guides in my DuckDuckGo-ing, I understand it has to do with setting a mark in iptables and assigning them to a table using ip route. However I haven't managed to get it to work yet, and many of these guides are for VPNs and they all seem to be slightly different to each other. So I thought I would ask about my own specific use case\n\n\n**Prompt 10:** \n\n[Specificity, Domain Knowledge, Complexity, Problem-solving, Creativity, Technical Accuracy, Real World]\n\nWrite me a python script for the foobar problem, but make it so that if read aloud, each pair of lines rhymes. (i.e. lines 1/2 rhyme, 3/4 rhyme and so on)","slug":"2024-05-17-category-hard"},"__N_SSG":true}