{
    "componentChunkName": "component---src-templates-post-tsx",
    "path": "/electra/",
    "result": {"data":{"logo":null,"markdownRemark":{"html":"<p>Below is just about everything you’ll need to style in the theme. Check the source code to see the many embedded elements within paragraphs.</p>\n<hr>\n<h1>ELECTRA</h1>\n<ul>\n<li><a href=\"https://openreview.net/forum?id=r1xMH1BtvB\">paper</a> / <a href=\"https://github.com/google-research/electra\">code</a></li>\n<li>ICLR 2020</li>\n</ul>\n<h2>Abstract</h2>\n<ul>\n<li>BERT에서 제안한 Masked Language Modeling(MLM)은 좋은 성능을 보여줬지만, 전체 데이터의 15%만을 마스킹해서 학습 효율 측면에서 좋지 않음.</li>\n<li>ELECTRA는 <strong>모델의 성능</strong>과 함께 <strong>학습의 효율성</strong>도 개선할 수 있는 방법을 제안함.</li>\n<li>Replaced Token Detection(RTD)라는 새로운 pre-training 태스크 제안.</li>\n<li>ELECTRA의 장점은 Small 모델에서 두드러짐. 1개의 GPU로 4일만 학습한 모델로 계산량이 30배인 GPT를 능가.</li>\n</ul>\n<h2>Replaced Token Detection (RTD)</h2>\n<img src=\"https://blog.pingpong.us/images/2020.05.08.electra/figure2.png\" width=\"700\">\n<ul>\n<li>Generator: BERT의 MLM\n<ul>\n<li>입력된 인풋 중 15%의 토큰을 [MASK]로 가림</li>\n<li>[MASK]로 가려진 인풋의 원래 토큰을 예측</li>\n</ul>\n</li>\n<li>Discriminator\n<ul>\n<li>입력 토큰 시퀀스에 대해서 각 토큰이 original인지 replaced인지 이진 분류로 학습</li>\n<li>학습 과정\n<ol>\n<li>Generator에서 마스킹 된 입력 토큰들을 예측</li>\n<li>마스킹할 위치의 토큰에 대해 generator가 예측했던 softmax 분포에서 높은 순위의 토큰 중 하나로 치환 (1위: cooked, 2위: ate, 3위: … 이였으면 MLM은 cooked를 선택하지만 해당 과정에서 ate를 가져오는 방식)</li>\n</ol>\n<ul>\n<li>Original input : [the, chef, cooked, the, meal]</li>\n<li>Input for generator : [[MASK], chef, [MASK], the, meal]</li>\n<li>Input for discriminator : [the, chef, ate, the, meal]</li>\n</ul>\n<ol start=\"3\">\n<li>치환된 입력에 대해 discriminator는 원래 입력과 동일한지 치환된 것인지를 이진 분류로 예측</li>\n</ol>\n</li>\n</ul>\n</li>\n</ul>\n<h2>Training Algorithm</h2>\n<ul>\n<li>Jointly 학습\n<ul>\n<li>Generator와 Discriminator를 동시에 학습시키는 방법</li>\n</ul>\n</li>\n<li>Two-stage 학습\n<ol>\n<li>Generator만 MLM으로 N 스텝동안 학습</li>\n<li>뒤이어 해당 모델을 Discriminator로 N 스텝동안 학습시키는 방식 (이때 Generator의 웨이트는 고정)</li>\n</ol>\n</li>\n<li>Adversarial 학습\n<ul>\n<li>Adversarial training을 모사해서 학습시키는 방식 (jointly보다 좋지 않아서 자세히 안 봤습니다.)</li>\n</ul>\n</li>\n</ul>\n<h2>Result</h2>\n<h3>Performance &#x26; Efficiency</h3>\n<img src=\"https://blog.pingpong.us/images/2020.05.08.electra/figure1.png\" width=\"700\">\n<ul>\n<li>다른 모델들에 비해 매우 빠르게 성능이 향상되는 것을 볼 수 있음</li>\n<li>그럼에도 불구하고, 기존 BERT보다 더 좋은 성느을 기록함.</li>\n</ul>\n<h3>Weight Sharing</h3>\n<ul>\n<li>Generator와 discriminator는 모두 트랜스포머의 인코더 구조.</li>\n<li>그렇기 때문에 3가지 선택사항이 생김.\n<ol>\n<li>Generator, Discriminator가 서로 독립적으로 학습 (83.5)</li>\n<li>임베딩 레이어의 웨이트만 공유 (84.3)</li>\n<li>모든 레이어의 웨이트를 공유 (84.4)</li>\n</ol>\n</li>\n<li>결과적으로 모든 웨이트를 공유하는 것이 가장 좋은 성능을 보임.</li>\n</ul>\n<h3>Training Algorithm</h3>\n<img src=\"https://blog.pingpong.us/images/2020.05.08.electra/figure3.png\" width=\"700\">\n<ul>\n<li>Jointly 방식이 가장 성능이 좋았음 (왼쪽은 discriminator와 generator의 사이즈에 따른 실험)</li>\n</ul>\n<h2>Conclusion</h2>\n<ul>\n<li>더 효율적이고 효과도 좋은 Replaced Token Detection (RTD) 제안</li>\n<li>메인 아이디어는 Generator가 만들어 낸 질 좋은 negative sample로 학습함으로써 더 적은 리소스로 모델을 더욱 견고하게 만드는 것.</li>\n</ul>","htmlAst":{"type":"root","children":[{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"Below is just about everything you’ll need to style in the theme. Check the source code to see the many embedded elements within paragraphs."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"hr","properties":{},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"h1","properties":{},"children":[{"type":"text","value":"ELECTRA"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"a","properties":{"href":"https://openreview.net/forum?id=r1xMH1BtvB"},"children":[{"type":"text","value":"paper"}]},{"type":"text","value":" / "},{"type":"element","tagName":"a","properties":{"href":"https://github.com/google-research/electra"},"children":[{"type":"text","value":"code"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"ICLR 2020"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Abstract"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"BERT에서 제안한 Masked Language Modeling(MLM)은 좋은 성능을 보여줬지만, 전체 데이터의 15%만을 마스킹해서 학습 효율 측면에서 좋지 않음."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"ELECTRA는 "},{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"모델의 성능"}]},{"type":"text","value":"과 함께 "},{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"학습의 효율성"}]},{"type":"text","value":"도 개선할 수 있는 방법을 제안함."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Replaced Token Detection(RTD)라는 새로운 pre-training 태스크 제안."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"ELECTRA의 장점은 Small 모델에서 두드러짐. 1개의 GPU로 4일만 학습한 모델로 계산량이 30배인 GPT를 능가."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Replaced Token Detection (RTD)"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://blog.pingpong.us/images/2020.05.08.electra/figure2.png","width":700},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Generator: BERT의 MLM\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"입력된 인풋 중 15%의 토큰을 [MASK]로 가림"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"[MASK]로 가려진 인풋의 원래 토큰을 예측"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Discriminator\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"입력 토큰 시퀀스에 대해서 각 토큰이 original인지 replaced인지 이진 분류로 학습"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"학습 과정\n"},{"type":"element","tagName":"ol","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Generator에서 마스킹 된 입력 토큰들을 예측"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"마스킹할 위치의 토큰에 대해 generator가 예측했던 softmax 분포에서 높은 순위의 토큰 중 하나로 치환 (1위: cooked, 2위: ate, 3위: … 이였으면 MLM은 cooked를 선택하지만 해당 과정에서 ate를 가져오는 방식)"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Original input : [the, chef, cooked, the, meal]"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Input for generator : [[MASK], chef, [MASK], the, meal]"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Input for discriminator : [the, chef, ate, the, meal]"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ol","properties":{"start":3},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"치환된 입력에 대해 discriminator는 원래 입력과 동일한지 치환된 것인지를 이진 분류로 예측"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Training Algorithm"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Jointly 학습\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Generator와 Discriminator를 동시에 학습시키는 방법"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Two-stage 학습\n"},{"type":"element","tagName":"ol","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Generator만 MLM으로 N 스텝동안 학습"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"뒤이어 해당 모델을 Discriminator로 N 스텝동안 학습시키는 방식 (이때 Generator의 웨이트는 고정)"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Adversarial 학습\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Adversarial training을 모사해서 학습시키는 방식 (jointly보다 좋지 않아서 자세히 안 봤습니다.)"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Result"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Performance & Efficiency"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://blog.pingpong.us/images/2020.05.08.electra/figure1.png","width":700},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"다른 모델들에 비해 매우 빠르게 성능이 향상되는 것을 볼 수 있음"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"그럼에도 불구하고, 기존 BERT보다 더 좋은 성느을 기록함."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Weight Sharing"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Generator와 discriminator는 모두 트랜스포머의 인코더 구조."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"그렇기 때문에 3가지 선택사항이 생김.\n"},{"type":"element","tagName":"ol","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Generator, Discriminator가 서로 독립적으로 학습 (83.5)"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"임베딩 레이어의 웨이트만 공유 (84.3)"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"모든 레이어의 웨이트를 공유 (84.4)"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"결과적으로 모든 웨이트를 공유하는 것이 가장 좋은 성능을 보임."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Training Algorithm"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://blog.pingpong.us/images/2020.05.08.electra/figure3.png","width":700},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Jointly 방식이 가장 성능이 좋았음 (왼쪽은 discriminator와 generator의 사이즈에 따른 실험)"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Conclusion"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"더 효율적이고 효과도 좋은 Replaced Token Detection (RTD) 제안"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"메인 아이디어는 Generator가 만들어 낸 질 좋은 negative sample로 학습함으로써 더 적은 리소스로 모델을 더욱 견고하게 만드는 것."}]},{"type":"text","value":"\n"}]}],"data":{"quirksMode":false}},"excerpt":"Below is just about everything you’ll need to style in the theme. Check the source code to see the many embedded elements within paragraphs…","fields":{"readingTime":{"text":"4 min read"}},"frontmatter":{"title":"Sooftware NLP - Electra Paper Review","userDate":"23 September 2020","date":"2020-09-23T10:00:00.000Z","tags":["nlp","paper"],"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","backgroundColor":"#f8f8f8","images":{"fallback":{"src":"/static/7ccdb9951c9d362c8a3548e8c2a87231/59ccb/electra.png","srcSet":"/static/7ccdb9951c9d362c8a3548e8c2a87231/c68af/electra.png 750w,\n/static/7ccdb9951c9d362c8a3548e8c2a87231/87f65/electra.png 1080w,\n/static/7ccdb9951c9d362c8a3548e8c2a87231/d464a/electra.png 1366w,\n/static/7ccdb9951c9d362c8a3548e8c2a87231/59ccb/electra.png 1920w","sizes":"100vw"},"sources":[{"srcSet":"/static/7ccdb9951c9d362c8a3548e8c2a87231/9fb02/electra.webp 750w,\n/static/7ccdb9951c9d362c8a3548e8c2a87231/cd76f/electra.webp 1080w,\n/static/7ccdb9951c9d362c8a3548e8c2a87231/b7397/electra.webp 1366w,\n/static/7ccdb9951c9d362c8a3548e8c2a87231/507b8/electra.webp 1920w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.4479166666666667}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"children":[{"gatsbyImageData":{"layout":"fullWidth","backgroundColor":"#282838","images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/0d6f4/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/248f9/soohwan.png 40w,\n/static/a9e6b445142b247ee4cfa66155398bb2/fd435/soohwan.png 80w,\n/static/a9e6b445142b247ee4cfa66155398bb2/0d6f4/soohwan.png 120w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/e7f45/soohwan.webp 40w,\n/static/a9e6b445142b247ee4cfa66155398bb2/589ec/soohwan.webp 80w,\n/static/a9e6b445142b247ee4cfa66155398bb2/71a38/soohwan.webp 120w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6833333333333333}}]}}]}},"relatedPosts":{"totalCount":50,"edges":[{"node":{"id":"6529d72e-80e7-5d61-a672-d66276a3f641","excerpt":"MoE(Mixture of Experts) 기초부터 DeepSeek 혁신까지 작년 이맘때쯤 DeepSeek-V3가 저비용으로 엄청난 성능을 보이면서 화제가 되었습니다. 그 핵심 기술인 MoE(Mixture of Experts…","frontmatter":{"title":"MoE(Mixture of Experts) 기초부터 DeepSeek 혁신까지","date":"2026-01-15T01:11:55.000Z"},"fields":{"readingTime":{"text":"8 min read"},"slug":"/developed-moe/"}}},{"node":{"id":"8b49d3ef-4ce3-568c-9d71-240ff17fc3e0","excerpt":"BERT는 사실 Diffusion 모델이였다?! 최근 굉장히 흥미로운 글을 읽게되어 공유합니다. 원문 : link BERT와 Diffusion이 같은 방식이다?! NLP 연구자들에게 BERT는 너무 익숙한 모델입니다. 201…","frontmatter":{"title":"BERT는 사실 Diffusion 모델이였다?!","date":"2025-10-21T12:00:00.000Z"},"fields":{"readingTime":{"text":"8 min read"},"slug":"/bert_diffusion/"}}},{"node":{"id":"7c7e1676-ea84-58de-970c-ddec9aa10660","excerpt":"RLHF는 수다쟁이를 만든다?! (Does RLHF Breed Verbose Chatterboxes?!) RLHF(Reinforcement Learning from Human Feedback)는 OpenAI의 ChatGPT…","frontmatter":{"title":"RLHF는 수다쟁이를 만든다?! (Does RLHF Breed Verbose Chatterboxes?!)","date":"2024-03-13T01:11:55.000Z"},"fields":{"readingTime":{"text":"7 min read"},"slug":"/rlhf-vervosity/"}}},{"node":{"id":"2b9d3e22-1796-5fab-878d-5941d2e76e9d","excerpt":"LLM Paper Abstract - 2023.12 LLM…","frontmatter":{"title":"LLM Paper Abstract - 2023.12","date":"2024-01-05T10:00:00.000Z"},"fields":{"readingTime":{"text":"5 min read"},"slug":"/llm-abs-202312/"}}},{"node":{"id":"f43fa33b-917b-5c00-8462-937d99592ad7","excerpt":"What it MoE? (Mixture of Experts) 현존 최강 LLM인 GPT-4에서 “MoE (Mixture of Experts)” 방식을 채택하여 사용하고 있다고 알려졌는데요, 최근 AI계의 뜨거운 감자 Mistral AI…","frontmatter":{"title":"What is MoE? (Mixture of Experts)","date":"2023-12-22T01:11:55.000Z"},"fields":{"readingTime":{"text":"9 min read"},"slug":"/moe/"}}}]}},"pageContext":{"slug":"/electra/","prev":{"excerpt":"wav2vec 2.0 : A Framework for Self-Supervised Learning of Speech Representations Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael…","frontmatter":{"title":"Sooftware Speech - Wav2vec 2.0 : A Framework for Self-Supervised Learning of Speech Representations","tags":["speech","paper"],"date":"2020-09-12T10:00:00.000Z","draft":false,"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAYAAAC0VX7mAAAACXBIWXMAAAsTAAALEwEAmpwYAAACT0lEQVQoz22SWU9TARCF709UgoYgLYWABlECKgqhKISy2cTEJ38Aib4ZTDREjCYk2JSdtpSlC3ahe0u3e3v3+5l7+6iTnJc5M5OZc0bgP2FZJmDDwjJNLEOhVi0jSSKqInNzk6VcylEs5lGVTrfHNLAsC8FurlQqpFIpZFlGkroF1TYUGmBaEE13qLc1xI5GrS46fKJgoJo4SJbM7lBAUGSZdDpNMpmkWquhKCp6K0E1u082HkAuH5G6+k09d8Btdp9cfAe1FiIR3UEsnTjIxAJot2F0rYPw77kgR5fgtA/1wIVx5ILQIJ1gP6Wf9yj/uk9jt8/JWadurBM3hNzoxx6UZro7UNM0FEVxNLAHdpIbKNFFmiEf6uUa7YgP6WyFyuEimV0vYmQZ6cyHdrWOcrFKJ7qMeuVHFYsItm4XF+eEIxHK5TKNRgNNN5xtY9dRmmILRW5zeX3u5P5kEhQqRTA0YqlLdF2jWsuTLWUdXmi328RiMeKJOPlCnlar7RDRVIvdSAHQ+bZXot5skS2JbB/m7JvYPqpxU7qlIap8DeYwja6ZgiRJjimZTMYxxY5MVcf/Hd5tw1YY5jbh8zF8+CHh/VRlK2wxvwkbAfgYhPkvcJCyDTAQbNFM00TXdQf23wX2jlnzv8e34mdy+jWLSys8mZplaGScicnn9HvGcHtGeDg+hWt4jMGhUWa8S+i6gaBpqvOHxWKRer2OYRjs7wV5u77KvHeO2ZlpFhbeMPF0HNfAA54/m8IzOEBvbw+Pxx4xOjJMz907vHr5wlnoL6u+uueManMzAAAAAElFTkSuQmCC"},"images":{"fallback":{"src":"/static/ef0fb6095c0dcc78d73cfdf24bc9ac11/038c6/wav2vec2.png","srcSet":"/static/ef0fb6095c0dcc78d73cfdf24bc9ac11/e6cc4/wav2vec2.png 750w,\n/static/ef0fb6095c0dcc78d73cfdf24bc9ac11/038c6/wav2vec2.png 842w","sizes":"100vw"},"sources":[{"srcSet":"/static/ef0fb6095c0dcc78d73cfdf24bc9ac11/79bef/wav2vec2.webp 750w,\n/static/ef0fb6095c0dcc78d73cfdf24bc9ac11/daf06/wav2vec2.webp 842w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.5106888361045131}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAADjElEQVQ4y23Oy48TdQDA8TExEaTttvSBfdHddru7bWc6bWd+8+i0u23n0SdlBRQXZGHDgiiLDwgqXjCIJEaC8cLBmHgwUU9EYzRqQuLJi/4NJB6NF+8evgbOHr7XT76SM5ix7s4QnRHnnm9yZzvBTIswN8JcP5bC67ZoWi7CdtFsn5bloZo+dSNAMYbIIqCmD6mJIVXdR3J6I7r9IQ3T59pZlW+uxzndCbPdC/HZXoq3TikowsfqBOjOmFZ7SrM9p2kfRbXn1O0jKNbsSbJ5BMnu9Gl3+qiiz4OPSvz0QRLHFGwOZH6+neLHT5YRbRerM0J0JrTas/8FZXNKzZgg2es+ZtvDdbs8+iHJl+9lWVP7OG2br26k+PvXJCfmFqoYIpwhTXtMwxpTN0coxhjFmCKbE2rGmKoYPQY9qi2fyzsq//4e5sNXi5TlLqbd5t6lPH/9kuDTG2usqgGi7dO0hzSsIXUzQDaGKGKMLEZUxZCKCJB02+PUpMH9vQwP7ybYnSxyfNrjwumA3dEiD94v8cfnK7xzWsbpujQtH9X0qBs+iuEjC5+a7lPVPSqai9SyelyYFbjkh5hqC3TkKOutLON2nkYpgtuMcT6IszdPslo8SLlcpmUH1A0XRQyQhUtNuFQ1l4o2QFI1h3OTVXbcg+grIfSVBRqlEGopgrYSRi0e4KiIse1mScSeopA/iLADmoZL3Rgg631q+oCq1qeq95FOzhQefpzn7eMpBmqcoBVj04pzwkky0eNMRYIdL8F3twusLe2nslrE6g5pGX1U8/FhD1nvUW1tUJK7SH9+n+DR1zF2vQSvH0lz88wiV45V2J1WuHqsxLXNAnuzNJdfWmK5/BylUgFhb6DZAxR9g7rpIos+Wttja2uE9M9vYb69k+DqfJErWzLOuoVqdKlpDo4jODms4zlFckuHObx4iHwuRi6fYqmYY201x/LqMrnFDNl8gRvvukhNdT9394rcOlNFMxrUhYFmW9imTEsp0mhWKC2nWSokyWSiFA7HSKcXiEb34fayyLVniUQk8rmn2drKIqlqiNcuVrh2ymQS1Nh8sc/GoMEbl7u8eb5D1yqTzkRJJsOkkiGymQVShyLE4yHWKnEq1RjpTIhmI8r2y3mkyloI38uha1n6bpVXrkw5+4LD/S/2uHnrIopaJhZ9hmQqwqHkAXJP8BDhyD46ZoWdMxOGcx/PNfECk/8AbxXdRjRliPoAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/34f77/soohwan.png 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/a94f6/soohwan.png 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png 1148w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/38420/soohwan.webp 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7470d/soohwan.webp 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/b5ef6/soohwan.webp 1148w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6829268292682927}}}}]},"fields":{"readingTime":{"text":"5 min read"},"layout":"","slug":"/wav2vec2/"}},"next":{"excerpt":"RoBERTa paper / code Abstract BERT를 제대로 학습시키는 법을 제안 BERT는 엄청난 모델이지만, Original BERT 논문에서 하이퍼파라미터에 대한 실험이 제대로 진행되지 않음 BERT…","frontmatter":{"title":"Sooftware NLP - RoBERTa Paper Review","tags":["nlp","paper"],"date":"2020-10-11T10:00:00.000Z","draft":false,"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAASCAYAAABb0P4QAAAACXBIWXMAAAsTAAALEwEAmpwYAAADkUlEQVQ4y62Uy28aVxTG5w9tt5Uqtdv2D2irLqIqqbJIu+hLSjdeVUljx8aY18wwMMM8mGGAwWDAQIIB85gH+AG/iuukirJtj3R0zz3fdz6du7iflCQJ/V6PxWLBPna73X9KaX49I1ss0h0N+T9CSu63DE7PmNlVotst4SomDCNWYUgY7uuEMErEuRL1mjBcv8Pec9aswpjN5gapEnRJPXpM+tfnvNBqHOV1XistUqU+R3Kdw4LO37kKh4UKx6rNy6whekeyz4nWF+eRrPMiY1O2mkiGVSXTUEi7ClkrR6Ujk7UH6G2QvQsU74SzSoZ05ZhT4zUpPYXeypJ3mxQbW2SvjewekTYtDLePZLotjrU2Z0aPjHVJutJB9uaUghty1QmpcotTo8tJqcWR2iRtdDkzL8hXJ4JTcKec6i1OK0Mq7gWSXeug1NeUgg2l4BatuUGtR+JutO+wumD1wN5nH4z2jlJwhxZsKDYStOZaCO81jGr7QTDvrZC9BQVvSc6Zofghf2UDnv5+yKNnBzz97RU//vKCn/5Mcaj2BF5wF8i1UNSqv0L2Y4xq50FQ9iO0ZiK2KjZikTnnmr8yTQ6Oq7zMt4XQodIla0/R23diq/12xXqM1ojeCbaRrFqHjLMk68zJOgvS5oycM0f1I+RahOLHKH6CXIvFUM5dkbGvBfeBE5Jzl+TcCMPtIDlOmU7hS7rFrzmXv2JW+RzN8pADsMwMo8InNEtPBPZW+4x68QmX6he0S99TaIBVOWasfkpN/wPdGyGZlol+9hg9+wxb/plG8Qfy5jm5OmimSTX7Lbp6gFV4RvnkO0ryAa78BLP4nFztjqJews1+g6q+QrN7SMM3U8reALP+BrP+Ft0bUm1c4vht7PpA3O3mW4FV/BF2cIVRG1Lxh1T9Dk5ziOGPMLw+/cEIaXd3y9jUmXgOk5rDUFO4T2LxL28XM7qFDLNWk6nnMNAUlhfnDEoqk7onOJvZhG4+w7J/8fCXN1FMT1HolcpclssMFJXF1YQdsByNOD9N0zcthuUy3XyegW3TVxTeOA7bPWc4JEilmDSDB8G95dxvt2x3O25ub4mThDiO2dvaer0WJnF3fy/w97nebEjWa8Hbc8IoEn0h+LH9LJdLptOp8Mf9wO4d8cOIoojr62uurq6Yz+fc3Nz866XSh+a4j73Z+r5Pq9ViPL566H/EGY/H1Ot1giCg2+2K17wX/Ad7CQ6hmGCGbAAAAABJRU5ErkJggg=="},"images":{"fallback":{"src":"/static/26b5f2f1a6d4d031c6cf36eac285256a/1be58/roberta.png","srcSet":"/static/26b5f2f1a6d4d031c6cf36eac285256a/5dae1/roberta.png 750w,\n/static/26b5f2f1a6d4d031c6cf36eac285256a/35cd7/roberta.png 1080w,\n/static/26b5f2f1a6d4d031c6cf36eac285256a/1be58/roberta.png 1134w","sizes":"100vw"},"sources":[{"srcSet":"/static/26b5f2f1a6d4d031c6cf36eac285256a/76436/roberta.webp 750w,\n/static/26b5f2f1a6d4d031c6cf36eac285256a/ce7b4/roberta.webp 1080w,\n/static/26b5f2f1a6d4d031c6cf36eac285256a/03f03/roberta.webp 1134w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.8835978835978836}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAADjElEQVQ4y23Oy48TdQDA8TExEaTttvSBfdHddru7bWc6bWd+8+i0u23n0SdlBRQXZGHDgiiLDwgqXjCIJEaC8cLBmHgwUU9EYzRqQuLJi/4NJB6NF+8evgbOHr7XT76SM5ix7s4QnRHnnm9yZzvBTIswN8JcP5bC67ZoWi7CdtFsn5bloZo+dSNAMYbIIqCmD6mJIVXdR3J6I7r9IQ3T59pZlW+uxzndCbPdC/HZXoq3TikowsfqBOjOmFZ7SrM9p2kfRbXn1O0jKNbsSbJ5BMnu9Gl3+qiiz4OPSvz0QRLHFGwOZH6+neLHT5YRbRerM0J0JrTas/8FZXNKzZgg2es+ZtvDdbs8+iHJl+9lWVP7OG2br26k+PvXJCfmFqoYIpwhTXtMwxpTN0coxhjFmCKbE2rGmKoYPQY9qi2fyzsq//4e5sNXi5TlLqbd5t6lPH/9kuDTG2usqgGi7dO0hzSsIXUzQDaGKGKMLEZUxZCKCJB02+PUpMH9vQwP7ybYnSxyfNrjwumA3dEiD94v8cfnK7xzWsbpujQtH9X0qBs+iuEjC5+a7lPVPSqai9SyelyYFbjkh5hqC3TkKOutLON2nkYpgtuMcT6IszdPslo8SLlcpmUH1A0XRQyQhUtNuFQ1l4o2QFI1h3OTVXbcg+grIfSVBRqlEGopgrYSRi0e4KiIse1mScSeopA/iLADmoZL3Rgg631q+oCq1qeq95FOzhQefpzn7eMpBmqcoBVj04pzwkky0eNMRYIdL8F3twusLe2nslrE6g5pGX1U8/FhD1nvUW1tUJK7SH9+n+DR1zF2vQSvH0lz88wiV45V2J1WuHqsxLXNAnuzNJdfWmK5/BylUgFhb6DZAxR9g7rpIos+Wttja2uE9M9vYb69k+DqfJErWzLOuoVqdKlpDo4jODms4zlFckuHObx4iHwuRi6fYqmYY201x/LqMrnFDNl8gRvvukhNdT9394rcOlNFMxrUhYFmW9imTEsp0mhWKC2nWSokyWSiFA7HSKcXiEb34fayyLVniUQk8rmn2drKIqlqiNcuVrh2ymQS1Nh8sc/GoMEbl7u8eb5D1yqTzkRJJsOkkiGymQVShyLE4yHWKnEq1RjpTIhmI8r2y3mkyloI38uha1n6bpVXrkw5+4LD/S/2uHnrIopaJhZ9hmQqwqHkAXJP8BDhyD46ZoWdMxOGcx/PNfECk/8AbxXdRjRliPoAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/34f77/soohwan.png 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/a94f6/soohwan.png 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png 1148w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/38420/soohwan.webp 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7470d/soohwan.webp 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/b5ef6/soohwan.webp 1148w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6829268292682927}}}}]},"fields":{"readingTime":{"text":"4 min read"},"layout":"","slug":"/roberta/"}},"primaryTag":"nlp"}},
    "staticQueryHashes": ["3170763342","3229353822"]}