You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
```# create datasetdataset=pb.DatasetLoader.load_dataset("mmlu",local_path=localpathconfig.MMLU_PATH) #还有mrpc的实验logging.info(f"数据集: mmlu")
# 确保数据集包含足够的数据iflen(dataset) >=1000:
# 选择前1000条记录validation_dataset=dataset[:1000]
else:
validation_dataset=datasetprompts= [
"In relation to the multiple-choice question on {}, please provide the accurate answer by choosing 'A', 'B', 'C', or 'D'",
"For each multiple-choice question about {}, identify the correct answer by selecting 'A', 'B', 'C', or 'D'",
"Answer the subsequent multiple-choice question about {} by picking the right option among 'A', 'B', 'C', or 'D'",
"As an expert in {}, respond to the following multiple-choice question by selecting 'A', 'B', 'C', or 'D'",
"Considering your familiarity with {}, attend to the following multiple-choice question by picking 'A', 'B', 'C', or 'D'",
"As someone well-versed in {}, please address the multiple-choice question below by selecting 'A', 'B', 'C', or 'D'"
]
try:
model_t5=LLMModel(model='google/flan-t5-large', temperature=0.5)
print("语句执行成功,模型已成功加载。")
exceptExceptionase:
print("语句执行失败,以下是错误信息:")
print(str(e))
# define the projection function required by the output processdefproj_func(pred):
mapping= {
"a": 0,
"b": 1,
"c": 2,
"d": 3
}
pred_lower=pred.lower() # 将输入转换为小写ifpred_lowerinmapping:
returnmapping[pred_lower]
else:
logging.info(f"ERROR OUT: {pred}") # 记录到日志文件return-1# define the evaluation function required by the attackdefeval_func(prompt, validation_dataset, model):
logging.info(f"Prompt: {prompt}") # 记录到日志文件preds= []
labels= []
fordintqdm(validation_dataset, desc="process"):
input_text=pb.InputProcess.basic_format(prompt.replace("{}", "{content}"), d)
raw_output=model(input_text) #是有回答的output=pb.OutputProcess.cls(raw_output, proj_func) #将输出结果映射到1 0 -1preds.append(output)
labels.append(d["label"])
returnpb.Eval.compute_cls_accuracy(preds, labels)
# define the unmodifiable words in the prompt# for example, the labels "positive" and "negative" are unmodifiable, and "content" is modifiable because it is a placeholder# if your labels are enclosed with '', you need to add \' to the unmodifiable words (due to one feature of textattack)unmodifiable_words= ['A', 'B', 'C', 'D', 'A\'', 'B\'', 'C\'', 'D\'', 'a', 'b', 'c', 'd', 'a\'', 'b\'', 'c\'', 'd\'']
# print all supported attacksprint(Attack.attack_list())
The text was updated successfully, but these errors were encountered:
The text was updated successfully, but these errors were encountered: