Skip to content

Latest commit

 

History

History
95 lines (66 loc) · 3.23 KB

index.org

File metadata and controls

95 lines (66 loc) · 3.23 KB
Raw

Why my Project is pointless

Natural Language Understanding

  • A subtask of Natural Language Understanding that deal with machine reading

comprehension.

Natural Language Understanding

  • Consider the sentence - Charlotte is in North Carolina.
    • If a program can successfuly carry out NLU, you’d expect it to comprehend that Charlotte is in fact in North Carolina.
    • How do you know if your program comprehended the meaning successfully?
      • You ask it a question!
        • Where is Charlotte?
        • Is Charlotte in NC?

Natural Language Understanding

  • Generative Language Models can potentially implicitly solve this problem, without any task-specific training.

An example.

  • Book a spot for three at Maid-Rite Sandwich Shop in Antigua and Barbuda. I’ll book a table for you at […]
  • Book a spot for three at Maid-Rite Sandwich Shop in Antigua and Barbuda. Which place? […]
  • Book a spot for three at Maid-Rite Sandwich Shop in Antigua and Barbuda. I’ll book a table for […]

An example.

  • Book a spot for three at Maid-Rite Sandwich Shop in Antigua and Barbuda. I’ll book a table for you at Maid-Rite Sandwich
  • Book a spot for three at Maid-Rite Sandwich Shop in Antigua and Barbuda. Which place? Maid-Rite Sandwich
  • Book a spot for three at Maid-Rite Sandwich Shop in Antigua and Barbuda. I’ll book a table for three

RoBERTa

RoBERTa iterates on BERT’s pretraining procedure, including training the model longer, with bigger batches over more data; removing the next sentence prediction objective; training on longer sequences; and dynamically changing the masking pattern applied to the training data.

RoBERTa

  • Bi-directional masked language model.

What I intended to do -

  • Book a spot for three at Eve’s Pizzeria. Which place? <mask>

What I intended to do -

  • Book a spot for three at Eve’s Pizzeria. Which place? <mask>
  • Book a spot for three at Eve’s Pizzeria. Which place? Eve’s <mask>
  • Book a spot for three at Eve’s Pizzeria. Which place? Eve’s Pizzeria

What actually happens -

  • <s> Book a spot for three at Eve’s Pizzeria. Which place? </s>

What actually happens -

  • <s> Book a spot for three at Eve’s Pizzeria. Which place? </s></s>
  • <s> Book a spot for three at Eve’s Pizzeria. Which place? </s></s></s></s></s></s></s></s>

How to get it to work -

  • Don’t use a birdirectional masked language model.
  • Examples of GPT2, a uni-directional transformer model trained for the same task -
  • Do not use dynamic masking, as the orignal paper did, instead mask only the final answer.
    • What the paper did -
      • Book a <mask> for three <mask> Eve’s Pizzeria. Which place? Eve’s Pizzeria.
    • Instead, do -
      • Book a spot for three at Eve’s Pizzeria. Which place? <mask> <mask>

Any questions?