Store the image moderation and text moderation logs #3478

BabyChouSr · 2024-08-15T04:58:58Z

Right now, we don't store the text moderation and image moderation info when it can be very helpful.

…m-sys#3413) Co-authored-by: Wei-Lin Chiang <[email protected]> Co-authored-by: Wei-Lin Chiang <[email protected]> Co-authored-by: simon-mo <[email protected]>

infwinston

quick first pass. overall looks good to me!

infwinston · 2024-08-16T05:39:54Z

fastchat/serve/moderation/moderator.py

@@ -0,0 +1,167 @@
+import datetime


Super awesome to see we implement this abstraction!

fastchat/serve/gradio_block_arena_anony.py

infwinston

thanks @BabyChouSr left some comments

fastchat/serve/gradio_block_arena_named.py

infwinston · 2024-08-20T05:23:03Z

fastchat/serve/gradio_block_arena_named.py

-def flash_buttons():
+def flash_buttons(dont_show_vote_buttons: bool = False):
+    if dont_show_vote_buttons:
+        yield [no_change_btn] * 4 + [enable_btn] * 2


this actually ends up breaking the ui - need to keep yield + return pattern

fastchat/serve/gradio_block_arena_vision_anony.py

infwinston · 2024-08-20T05:30:19Z

fastchat/serve/gradio_block_arena_vision_anony.py

+            + [disable_btn] * 4
+            + [no_change_btn] * 3


why x4 + x3 vs x7 before

fastchat/serve/gradio_block_arena_vision_named.py

fastchat/serve/gradio_web_server.py

…ion-log

infwinston

Thanks @BabyChouSr ! this is super awesome. took a pass and left some comments. I think the main discussion item is what data should we log? maybe we don't need to log the entire moderation output which will take lots of space

infwinston · 2024-08-27T04:18:01Z

fastchat/serve/gradio_block_arena_anony.py

@@ -342,18 +352,45 @@ def bot_response_multi(
    request: gr.Request,
 ):
    logger.info(f"bot_response_multi (anony). ip: {get_ip(request)}")
+    states = [state0, state1]
+
+    if states[0] is None or states[0].skip_next:


maybe we can use this variable

states[0].content_moderator.text_flag

infwinston · 2024-10-11T21:33:53Z

fastchat/serve/gradio_web_server.py

@@ -151,6 +154,7 @@ def dict(self):
            {
                "conv_id": self.conv_id,
                "model_name": self.model_name,
+                "moderation": self.content_moderator.conv_moderation_responses,


I'm worried this would make our logs too huge... do we need to store the complete result or just moderation: True | False

the log increased probably 3x larger from

{"tstamp": 1728682280.9245, "type": "chat", "model": "chatgpt-4o-latest", "gen_params": {"temperature": 0.7, "top_p": 0.7, "max_new_tokens": 1024}, "start": 1728682277.5957, "finish": 1728682280.9245, "state": {"template_name": "gpt-4-turbo-2024-04-09", "system_message": "You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture.\nKnowledge cutoff: 2023-11\nCurrent date: 2024-10-11\n\nImage input capabilities: Enabled\nPersonality: v2", "roles": ["user", "assistant"], "messages": [["user", "heyy"], ["assistant", "Hey! How\u2019s it going?"]], "offset": 0, "conv_id": "79aac635c1924185ad5e2d2c07d626a7", "model_name": "chatgpt-4o-latest"}

to

{"tstamp": 1728682280.9245, "type": "chat", "model": "chatgpt-4o-latest", "gen_params": {"temperature": 0.7, "top_p": 0.7, "max_new_tokens": 1024}, "start": 1728682277.5957, "finish": 1728682280.9245, "state": {"template_name": "gpt-4-turbo-2024-04-09", "system_message": "You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture.\nKnowledge cutoff: 2023-11\nCurrent date: 2024-10-11\n\nImage input capabilities: Enabled\nPersonality: v2", "roles": ["user", "assistant"], "messages": [["user", "heyy"], ["assistant", "Hey! How\u2019s it going?"]], "offset": 0, "conv_id": "79aac635c1924185ad5e2d2c07d626a7", "model_name": "chatgpt-4o-latest", "moderation": [{"text_moderation": {"response": {"harassment": 1.1623426871665288e-05, "harassment_threatening": 4.7560683924530167e-07, "hate": 3.7377474200184224e-06, "hate_threatening": 3.018905303520114e-08, "illicit": null, "illicit_violent": null, "self_harm": 5.3270043281372637e-05, "self_harm_instructions": 3.869533247780055e-05, "self_harm_intent": 9.72322523011826e-05, "sexual": 0.0004021718923468143, "sexual_minors": 3.3023070500348695e-06, "violence": 1.4770392908758367e-06, "violence_graphic": 2.4935059173003538e-06, "self-harm": 5.3270043281372637e-05, "sexual/minors": 3.3023070500348695e-06, "hate/threatening": 3.018905303520114e-08, "violence/graphic": 2.4935059173003538e-06, "self-harm/intent": 9.72322523011826e-05, "self-harm/instructions": 3.869533247780055e-05, "harassment/threatening": 4.7560683924530167e-07}, "flagged": false}, "nsfw_moderation": {"flagged": false}, "csam_moderation": {"flagged": false}}, {"text_moderation": {"response": {"harassment": 3.410908902878873e-05, "harassment_threatening": 3.6979138258175226e-06, "hate": 2.1581441615126096e-05, "hate_threatening": 3.484581512225304e-08, "illicit": null, "illicit_violent": null, "self_harm": 4.098457338841399e-06, "self_harm_instructions": 5.236282163423311e-07, "self_harm_intent": 9.65906565397745e-07, "sexual": 0.0002804335963446647, "sexual_minors": 7.323227464439697e-07, "violence": 5.0677666877163574e-05, "violence_graphic": 9.602602403901983e-06, "self-harm": 4.098457338841399e-06, "sexual/minors": 7.323227464439697e-07, "hate/threatening": 3.484581512225304e-08, "violence/graphic": 9.602602403901983e-06, "self-harm/intent": 9.65906565397745e-07, "self-harm/instructions": 5.236282163423311e-07, "harassment/threatening": 3.6979138258175226e-06}, "flagged": false}, "nsfw_moderation": {"flagged": false}, "csam_moderation": {"flagged": false}}]}, "ip": "76.102.1.74"}

done. it looks something like this now

{"moderation": [{"text_moderation": {"flagged": false}, "nsfw_moderation": {"flagged": false}, "csam_moderation": {"flagged": false}}, {"text_moderation": {"flagged": false}, "nsfw_moderation": {"flagged": false}, "csam_moderation": {"flagged": false}}], "has_csam_image": false}, "ip": "67.170.233.8"}

fastchat/serve/gradio_block_arena_vision_anony.py

infwinston · 2024-10-15T08:57:59Z

fastchat/serve/gradio_block_arena_named.py

@@ -301,14 +346,19 @@ def bot_response_multi(
            break


-def flash_buttons():
+def flash_buttons(show_vote_buttons: bool = True):


sorry could you say more what's this for?

We shouldn't flash vote buttons if the text fails the moderation test. essentially, people shouldn't be able to vote if it fails since there will be no output

fastchat/serve/gradio_block_arena_named.py

fastchat/serve/monitor/monitor.py

fastchat/serve/gradio_web_server.py

infwinston · 2024-10-15T09:14:03Z

fastchat/serve/gradio_block_arena_anony.py

 )

 logger = build_logger("gradio_web_server_multi", "gradio_web_server_multi.log")

 num_sides = 2
 enable_moderation = False
+use_remote_storage = False


why do we need this be global variable? also should it be globally False?

I think globally False is the correct decision because we have a set bucket where we place images and not everyone will do it that way - i think that having it default False makes it so anyone can run this without google cloud storage.

infwinston and others added 8 commits July 5, 2024 00:17

update

a71e3c6

Use Reka Python SDK and add script for benchmarking and add send_btn (l…

68023e1

…m-sys#3413) Co-authored-by: Wei-Lin Chiang <[email protected]> Co-authored-by: Wei-Lin Chiang <[email protected]> Co-authored-by: simon-mo <[email protected]>

Store text and image moderation logs

cb4da0d

Update moderation

605add3

Run formatter

4492299

Show vote button

2723660

Fix pylint

51f9a0d

Fix pylint

38a1360

BabyChouSr marked this pull request as ready for review August 16, 2024 05:14

BabyChouSr requested a review from infwinston August 16, 2024 05:23

infwinston reviewed Aug 16, 2024

View reviewed changes

BabyChouSr added 2 commits August 16, 2024 06:35

Save bad images

e10d11b

Address comments

5159d3b

infwinston reviewed Aug 20, 2024

View reviewed changes

BabyChouSr and others added 12 commits August 27, 2024 03:07

Save moderation info per turn

dba425f

Change states

d289be9

Clean up

7911ecd

Get rid of previous moderation response

1527aac

Rename

36c67da

Enable vision arena across all tabs (lm-sys#3483)

1ccbe8b

Merge branch 'main' into moderation-log

b11f710

Merge branch 'main' into moderation-log

571f39e

Merge remote-tracking branch 'fastchat/operation-202407' into moderat…

3555d01

…ion-log

Format

fe45c6f

Merge with unified vision arena

a2200e4

Fix edge case

c90b8fc

BabyChouSr changed the base branch from operation-202407 to main October 6, 2024 03:36

BabyChouSr added 3 commits October 8, 2024 00:55

Merge

807b66f

Merge

24ce7b7

Merge

a25bd4d

BabyChouSr added 2 commits October 8, 2024 01:19

Fix

c6c284e

Format

87b6390

BabyChouSr requested a review from infwinston October 8, 2024 01:20

infwinston reviewed Oct 15, 2024

View reviewed changes

fastchat/serve/monitor/monitor.py Outdated Show resolved Hide resolved

infwinston reviewed Oct 15, 2024

View reviewed changes

fastchat/serve/gradio_web_server.py Show resolved Hide resolved

infwinston reviewed Oct 15, 2024

View reviewed changes

BabyChouSr added 7 commits November 28, 2024 18:44

Merge branch 'main' into moderation-log

4c9c98f

Address comments

37f3a0c

Save only flag

d7a152a

Format

2ef314b

Reset moderaiton flags

5b1fa5e

Address comments

add072b

Format

2f9d4e2

BabyChouSr requested a review from infwinston November 28, 2024 20:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store the image moderation and text moderation logs #3478

Store the image moderation and text moderation logs #3478

BabyChouSr commented Aug 15, 2024

infwinston left a comment

infwinston Aug 16, 2024 •

edited

Loading

infwinston left a comment

infwinston Aug 20, 2024

BabyChouSr Aug 27, 2024

infwinston Aug 20, 2024

infwinston left a comment •

edited

Loading

infwinston Aug 27, 2024

infwinston Oct 11, 2024

BabyChouSr Nov 28, 2024

infwinston Oct 15, 2024

BabyChouSr Nov 28, 2024

infwinston Oct 15, 2024 •

edited

Loading

BabyChouSr Nov 28, 2024

Store the image moderation and text moderation logs #3478

Are you sure you want to change the base?

Store the image moderation and text moderation logs #3478

Conversation

BabyChouSr commented Aug 15, 2024

infwinston left a comment

Choose a reason for hiding this comment

infwinston Aug 16, 2024 • edited Loading

Choose a reason for hiding this comment

infwinston left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

infwinston left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

infwinston Oct 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

infwinston Aug 16, 2024 •

edited

Loading

infwinston left a comment •

edited

Loading

infwinston Oct 15, 2024 •

edited

Loading