{
    "componentChunkName": "component---src-templates-post-tsx",
    "path": "/luna/",
    "result": {"data":{"logo":null,"markdownRemark":{"html":"<h1>Luna: Linear Unified Nested Attention</h1>\n<ul>\n<li>USC + CMU + Facebook AI</li>\n<li>2021.06</li>\n<li><a href=\"https://github.com/XuezheMax/fairseq-apollo\">code</a></li>\n</ul>\n<h2>Abstract</h2>\n<ul>\n<li>\n<p>트랜스포머의 Multi Headed Self Attention은 시퀀스가 길어질수록 메모리, 시간복잡도가 quadratic함</p>\n</li>\n<li>\n<p>이를 linear하게 바꾸기 위한 시도로 Luna (Linear Unified Nested Attention) 을 제안</p>\n</li>\n<li>\n<p>핵심은 인풋을 고정 길이로 변환한다는 점과 attention을 2개로 분리한다는 점.</p>\n</li>\n<li>\n<p>Positional Embedding을 별도의 고정 길이 query로 뺌.</p>\n</li>\n<li>\n<p>속도는 Performer랑 비슷한데 시퀀스가 길어질수록 좀 더 효과적이고 메모리도 적게 쓴다고 함.</p>\n</li>\n<li>\n<p>정확도 성능은 경쟁력이 높은 편.</p>\n</li>\n</ul>\n<h2>Attention</h2>\n<h3>Traditional attention mechanism:</h3>\n<img src=\"https://user-images.githubusercontent.com/42150335/127510622-5af08d8d-771c-4bb9-8e9e-53bb3f4cf85e.png\" height=\"70\">\n<ul>\n<li>X: Query sequence, C: Context sequence</li>\n<li>ω: activation function (usually softmax)</li>\n<li>Q = XW<sub>Q</sub>, K = CW<sub>K</sub>, V = CW<sub>V</sub></li>\n<li>Self attention에서는 X == C</li>\n<li>공간 &#x26; 시간복잡도: O(nm) (n: X의 시퀀스 길이, m: C의 시퀀스 길이)</li>\n</ul>\n<h2>Linear Unified Nested Attention (LUNA)</h2>\n<img src=\"https://user-images.githubusercontent.com/42150335/127514004-0ca02332-8ef5-4563-b9b4-bd9bf6bb72b9.png\" height=\"400\">\n<ul>\n<li>Goal: Attention mechanism’s complexity <strong>quadratic => linear</strong></li>\n</ul>\n<h3>Luna (Pack and Unpack Attention)</h3>\n<ul>\n<li>이 어텐션의 핵심은 어텐션을 2개로 쪼개는 것.</li>\n<li>O(ln) (l은 P의 길이)</li>\n<li>Pack Attention:\n<ul>\n<li>Query를 learnable paramter인 P로 대체 (고정 길이, 길이에 대한 실험 있음)</li>\n<li>상대적으로 더 짧은 Query를 놓음으로써 complexity를 줄임</li>\n<li>P-contextual: 첫 레이어의 P는 learnable parameter이며 다음 레이어로 전달</li>\n<li>P-non-contextual: 각 레이어마다 P를 학습하고 다음 레이어로 넘기지 않음</li>\n<li>contextual &#x26; non-contextual에 대한 실험은 뒤에 있음</li>\n</ul>\n</li>\n</ul>\n<img src=\"https://user-images.githubusercontent.com/42150335/127603299-44234ff8-1735-4dc2-95b0-bc8f66bfbbde.png\" height=\"60\">\n<ul>\n<li>Unpack Attention:\n<ul>\n<li>Pack Attention의 결과인 <code class=\"language-text\">packed context</code>를 Key와 Value로 사용. Query는 기존 query.</li>\n</ul>\n</li>\n</ul>\n<img src=\"https://user-images.githubusercontent.com/42150335/127603352-dfb741df-ec40-4b49-8bdf-b158e55b771e.png\" height=\"70\">\n<ul>\n<li>Luna Attention:</li>\n</ul>\n<img src=\"https://user-images.githubusercontent.com/42150335/127603409-554fa551-e89d-46ee-aec4-9a471a5cdc8f.png\" height=\"70\">\n<ul>\n<li>Luna Layer</li>\n</ul>\n<img src=\"https://user-images.githubusercontent.com/42150335/127603246-192380b0-fde6-4941-b060-1bf639d3b8e7.png\" height=\"100\">\n<h2>Discussion</h2>\n<h3>Relation to Linformer</h3>\n<ul>\n<li>Linformer와 비슷한 포지션. 그럼 뭐가 더 나은가?\n<ul>\n<li>Linformer는 인풋 시퀀스가 모두 고정 길이를 가져야하는데, Luna는 various length가 가능 (projection matrix 때문에 linformer는 길이가 고정이여야함)</li>\n<li>결정적으로 Linformer보다 성능이 좋음</li>\n</ul>\n</li>\n</ul>\n<h2>Experiment</h2>\n<h3>Long-Context Sequence Modeling</h3>\n<h4><strong>Score</strong></h4>\n<img src=\"https://user-images.githubusercontent.com/42150335/127603947-40c6b9f0-63d7-475c-8b6d-cefdfbe5ab59.png\" height=\"500\">\n<h4><strong>Training Speed &#x26; Memory</strong></h4>\n<img src=\"https://user-images.githubusercontent.com/42150335/127604072-79facf8c-7f84-4e9d-bd1e-38b3322458d0.png\" height=\"500\">\n<ul>\n<li>인풋 길이가 길어지면 Luna가 Linformer보다 더 빠름</li>\n<li>메모리 사용량에서 Luna가 Linformer를 포함한 다른 모델들보다 경쟁력이 있음</li>\n</ul>\n<h3>NLU Task (Masked Language Modeling for Large-Scale Pretraining)</h3>\n<img src=\"https://user-images.githubusercontent.com/42150335/127604344-7074edf7-ede2-4140-a6e8-320ed711d5f3.png\" height=\"300\">\n<ul>\n<li>BERT 방식으로 pre-training 후 파인튜닝 했을 때 성능 비교</li>\n<li>RoBERTa와도 비견될만큼 좋은 성능을 보임</li>\n</ul>\n<h3>Machine Translation</h3>\n<img src=\"https://user-images.githubusercontent.com/42150335/127604608-822e1fd6-00a1-472b-801e-24cda16efa5f.png\" height=\"250\">\n<h3>Abbrebiation Study (contextual &#x3C;-> non-contextual)</h3>\n<img src=\"https://user-images.githubusercontent.com/42150335/127604743-ae8289cc-d420-41b0-9ff1-5604ef770b4c.png\" height=\"150\">\n<ul>\n<li>P를 각 레이어의 파라미터로 둘지, 위층으로 넘김으로써 context 정보를 넘겨줄지에 대한 실험</li>\n<li>결론: 다음 층으로 넘겨주는게 좋더라.</li>\n</ul>","htmlAst":{"type":"root","children":[{"type":"element","tagName":"h1","properties":{},"children":[{"type":"text","value":"Luna: Linear Unified Nested Attention"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"USC + CMU + Facebook AI"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"2021.06"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"a","properties":{"href":"https://github.com/XuezheMax/fairseq-apollo"},"children":[{"type":"text","value":"code"}]}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Abstract"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"트랜스포머의 Multi Headed Self Attention은 시퀀스가 길어질수록 메모리, 시간복잡도가 quadratic함"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"이를 linear하게 바꾸기 위한 시도로 Luna (Linear Unified Nested Attention) 을 제안"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"핵심은 인풋을 고정 길이로 변환한다는 점과 attention을 2개로 분리한다는 점."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"Positional Embedding을 별도의 고정 길이 query로 뺌."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"속도는 Performer랑 비슷한데 시퀀스가 길어질수록 좀 더 효과적이고 메모리도 적게 쓴다고 함."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"정확도 성능은 경쟁력이 높은 편."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Attention"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Traditional attention mechanism:"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/127510622-5af08d8d-771c-4bb9-8e9e-53bb3f4cf85e.png","height":70},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"X: Query sequence, C: Context sequence"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"ω: activation function (usually softmax)"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Q = XW"},{"type":"element","tagName":"sub","properties":{},"children":[{"type":"text","value":"Q"}]},{"type":"text","value":", K = CW"},{"type":"element","tagName":"sub","properties":{},"children":[{"type":"text","value":"K"}]},{"type":"text","value":", V = CW"},{"type":"element","tagName":"sub","properties":{},"children":[{"type":"text","value":"V"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Self attention에서는 X == C"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"공간 & 시간복잡도: O(nm) (n: X의 시퀀스 길이, m: C의 시퀀스 길이)"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Linear Unified Nested Attention (LUNA)"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/127514004-0ca02332-8ef5-4563-b9b4-bd9bf6bb72b9.png","height":400},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Goal: Attention mechanism’s complexity "},{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"quadratic => linear"}]}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Luna (Pack and Unpack Attention)"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"이 어텐션의 핵심은 어텐션을 2개로 쪼개는 것."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"O(ln) (l은 P의 길이)"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Pack Attention:\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Query를 learnable paramter인 P로 대체 (고정 길이, 길이에 대한 실험 있음)"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"상대적으로 더 짧은 Query를 놓음으로써 complexity를 줄임"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"P-contextual: 첫 레이어의 P는 learnable parameter이며 다음 레이어로 전달"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"P-non-contextual: 각 레이어마다 P를 학습하고 다음 레이어로 넘기지 않음"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"contextual & non-contextual에 대한 실험은 뒤에 있음"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/127603299-44234ff8-1735-4dc2-95b0-bc8f66bfbbde.png","height":60},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Unpack Attention:\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Pack Attention의 결과인 "},{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"packed context"}]},{"type":"text","value":"를 Key와 Value로 사용. Query는 기존 query."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/127603352-dfb741df-ec40-4b49-8bdf-b158e55b771e.png","height":70},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Luna Attention:"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/127603409-554fa551-e89d-46ee-aec4-9a471a5cdc8f.png","height":70},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Luna Layer"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/127603246-192380b0-fde6-4941-b060-1bf639d3b8e7.png","height":100},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Discussion"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Relation to Linformer"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Linformer와 비슷한 포지션. 그럼 뭐가 더 나은가?\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Linformer는 인풋 시퀀스가 모두 고정 길이를 가져야하는데, Luna는 various length가 가능 (projection matrix 때문에 linformer는 길이가 고정이여야함)"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"결정적으로 Linformer보다 성능이 좋음"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Experiment"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Long-Context Sequence Modeling"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h4","properties":{},"children":[{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"Score"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/127603947-40c6b9f0-63d7-475c-8b6d-cefdfbe5ab59.png","height":500},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"h4","properties":{},"children":[{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"Training Speed & Memory"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/127604072-79facf8c-7f84-4e9d-bd1e-38b3322458d0.png","height":500},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"인풋 길이가 길어지면 Luna가 Linformer보다 더 빠름"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"메모리 사용량에서 Luna가 Linformer를 포함한 다른 모델들보다 경쟁력이 있음"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"NLU Task (Masked Language Modeling for Large-Scale Pretraining)"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/127604344-7074edf7-ede2-4140-a6e8-320ed711d5f3.png","height":300},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"BERT 방식으로 pre-training 후 파인튜닝 했을 때 성능 비교"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"RoBERTa와도 비견될만큼 좋은 성능을 보임"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Machine Translation"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/127604608-822e1fd6-00a1-472b-801e-24cda16efa5f.png","height":250},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Abbrebiation Study (contextual <-> non-contextual)"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/42150335/127604743-ae8289cc-d420-41b0-9ff1-5604ef770b4c.png","height":150},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"P를 각 레이어의 파라미터로 둘지, 위층으로 넘김으로써 context 정보를 넘겨줄지에 대한 실험"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"결론: 다음 층으로 넘겨주는게 좋더라."}]},{"type":"text","value":"\n"}]}],"data":{"quirksMode":false}},"excerpt":"Luna: Linear Unified Nested Attention USC + CMU + Facebook AI 2021.06 code Abstract 트랜스포머의 Multi Headed Self Attention…","fields":{"readingTime":{"text":"4 min read"}},"frontmatter":{"title":"Sooftware NLP - Luna: Linear Unified Nested Attention","userDate":"3 July 2021","date":"2021-07-03T23:46:37.121Z","tags":["nlp","paper"],"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","backgroundColor":"#f8f8f8","images":{"fallback":{"src":"/static/540ac0a1a4457065fef9e262f9962c83/af6c5/luna.png","srcSet":"/static/540ac0a1a4457065fef9e262f9962c83/a5b02/luna.png 750w,\n/static/540ac0a1a4457065fef9e262f9962c83/ace3b/luna.png 1080w,\n/static/540ac0a1a4457065fef9e262f9962c83/58836/luna.png 1366w,\n/static/540ac0a1a4457065fef9e262f9962c83/af6c5/luna.png 1920w","sizes":"100vw"},"sources":[{"srcSet":"/static/540ac0a1a4457065fef9e262f9962c83/6f4ad/luna.webp 750w,\n/static/540ac0a1a4457065fef9e262f9962c83/9ec3b/luna.webp 1080w,\n/static/540ac0a1a4457065fef9e262f9962c83/51e4d/luna.webp 1366w,\n/static/540ac0a1a4457065fef9e262f9962c83/3c39a/luna.webp 1920w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.5192708333333333}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"children":[{"gatsbyImageData":{"layout":"fullWidth","backgroundColor":"#282838","images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/0d6f4/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/248f9/soohwan.png 40w,\n/static/a9e6b445142b247ee4cfa66155398bb2/fd435/soohwan.png 80w,\n/static/a9e6b445142b247ee4cfa66155398bb2/0d6f4/soohwan.png 120w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/e7f45/soohwan.webp 40w,\n/static/a9e6b445142b247ee4cfa66155398bb2/589ec/soohwan.webp 80w,\n/static/a9e6b445142b247ee4cfa66155398bb2/71a38/soohwan.webp 120w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6833333333333333}}]}}]}},"relatedPosts":{"totalCount":50,"edges":[{"node":{"id":"6529d72e-80e7-5d61-a672-d66276a3f641","excerpt":"MoE(Mixture of Experts) 기초부터 DeepSeek 혁신까지 작년 이맘때쯤 DeepSeek-V3가 저비용으로 엄청난 성능을 보이면서 화제가 되었습니다. 그 핵심 기술인 MoE(Mixture of Experts…","frontmatter":{"title":"MoE(Mixture of Experts) 기초부터 DeepSeek 혁신까지","date":"2026-01-15T01:11:55.000Z"},"fields":{"readingTime":{"text":"8 min read"},"slug":"/developed-moe/"}}},{"node":{"id":"8b49d3ef-4ce3-568c-9d71-240ff17fc3e0","excerpt":"BERT는 사실 Diffusion 모델이였다?! 최근 굉장히 흥미로운 글을 읽게되어 공유합니다. 원문 : link BERT와 Diffusion이 같은 방식이다?! NLP 연구자들에게 BERT는 너무 익숙한 모델입니다. 201…","frontmatter":{"title":"BERT는 사실 Diffusion 모델이였다?!","date":"2025-10-21T12:00:00.000Z"},"fields":{"readingTime":{"text":"8 min read"},"slug":"/bert_diffusion/"}}},{"node":{"id":"7c7e1676-ea84-58de-970c-ddec9aa10660","excerpt":"RLHF는 수다쟁이를 만든다?! (Does RLHF Breed Verbose Chatterboxes?!) RLHF(Reinforcement Learning from Human Feedback)는 OpenAI의 ChatGPT…","frontmatter":{"title":"RLHF는 수다쟁이를 만든다?! (Does RLHF Breed Verbose Chatterboxes?!)","date":"2024-03-13T01:11:55.000Z"},"fields":{"readingTime":{"text":"7 min read"},"slug":"/rlhf-vervosity/"}}},{"node":{"id":"2b9d3e22-1796-5fab-878d-5941d2e76e9d","excerpt":"LLM Paper Abstract - 2023.12 LLM…","frontmatter":{"title":"LLM Paper Abstract - 2023.12","date":"2024-01-05T10:00:00.000Z"},"fields":{"readingTime":{"text":"5 min read"},"slug":"/llm-abs-202312/"}}},{"node":{"id":"f43fa33b-917b-5c00-8462-937d99592ad7","excerpt":"What it MoE? (Mixture of Experts) 현존 최강 LLM인 GPT-4에서 “MoE (Mixture of Experts)” 방식을 채택하여 사용하고 있다고 알려졌는데요, 최근 AI계의 뜨거운 감자 Mistral AI…","frontmatter":{"title":"What is MoE? (Mixture of Experts)","date":"2023-12-22T01:11:55.000Z"},"fields":{"readingTime":{"text":"9 min read"},"slug":"/moe/"}}}]}},"pageContext":{"slug":"/luna/","prev":{"excerpt":"Ray: multi-processing library…","frontmatter":{"title":"Ray: multi-processing library","tags":["toolkit"],"date":"2021-07-03T10:00:00.000Z","draft":false,"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAGCAYAAADDl76dAAAACXBIWXMAAAsTAAALEwEAmpwYAAAA+0lEQVQY042RsUoDURBF7zNqrGysrBSVTTIzSyRvNouBqIWkEwSbxUYE/8AfsPIn7PJWO639HfuwI/kAV0zWkCAYLxyY6nAvA/wkGBAKIK8ItoXcNvFW4haAj2MoNaFCU5gX+J2Z0Nz0tjOEIqs9vqOd9jd8HG8rNVZUqK5Ca8q8rsy1JcKJtI5g1wiWYTh6wMvnTTvpJUrNSy+SeKErFeqp0ECZ3fKGoVhFsCMEu8BwdI/X8qRz2ImUWqkXPvbCpyqsKkTVfKdM/5o8QG7nePoAyhLdaA/pwQ480wyd4w/h5CGukgPPY7eb3SFp7LukFX2LXFf6C2I/N/kLonRoV2yUhgcAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/60ac3ec5a068a489c5ce524ee4097082/5e0b4/ray.png","srcSet":"/static/60ac3ec5a068a489c5ce524ee4097082/87157/ray.png 750w,\n/static/60ac3ec5a068a489c5ce524ee4097082/0107c/ray.png 1080w,\n/static/60ac3ec5a068a489c5ce524ee4097082/92c3b/ray.png 1366w,\n/static/60ac3ec5a068a489c5ce524ee4097082/5e0b4/ray.png 1920w","sizes":"100vw"},"sources":[{"srcSet":"/static/60ac3ec5a068a489c5ce524ee4097082/6ae24/ray.webp 750w,\n/static/60ac3ec5a068a489c5ce524ee4097082/02909/ray.webp 1080w,\n/static/60ac3ec5a068a489c5ce524ee4097082/5981b/ray.webp 1366w,\n/static/60ac3ec5a068a489c5ce524ee4097082/707e4/ray.webp 1920w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.3125}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAADjElEQVQ4y23Oy48TdQDA8TExEaTttvSBfdHddru7bWc6bWd+8+i0u23n0SdlBRQXZGHDgiiLDwgqXjCIJEaC8cLBmHgwUU9EYzRqQuLJi/4NJB6NF+8evgbOHr7XT76SM5ix7s4QnRHnnm9yZzvBTIswN8JcP5bC67ZoWi7CdtFsn5bloZo+dSNAMYbIIqCmD6mJIVXdR3J6I7r9IQ3T59pZlW+uxzndCbPdC/HZXoq3TikowsfqBOjOmFZ7SrM9p2kfRbXn1O0jKNbsSbJ5BMnu9Gl3+qiiz4OPSvz0QRLHFGwOZH6+neLHT5YRbRerM0J0JrTas/8FZXNKzZgg2es+ZtvDdbs8+iHJl+9lWVP7OG2br26k+PvXJCfmFqoYIpwhTXtMwxpTN0coxhjFmCKbE2rGmKoYPQY9qi2fyzsq//4e5sNXi5TlLqbd5t6lPH/9kuDTG2usqgGi7dO0hzSsIXUzQDaGKGKMLEZUxZCKCJB02+PUpMH9vQwP7ybYnSxyfNrjwumA3dEiD94v8cfnK7xzWsbpujQtH9X0qBs+iuEjC5+a7lPVPSqai9SyelyYFbjkh5hqC3TkKOutLON2nkYpgtuMcT6IszdPslo8SLlcpmUH1A0XRQyQhUtNuFQ1l4o2QFI1h3OTVXbcg+grIfSVBRqlEGopgrYSRi0e4KiIse1mScSeopA/iLADmoZL3Rgg631q+oCq1qeq95FOzhQefpzn7eMpBmqcoBVj04pzwkky0eNMRYIdL8F3twusLe2nslrE6g5pGX1U8/FhD1nvUW1tUJK7SH9+n+DR1zF2vQSvH0lz88wiV45V2J1WuHqsxLXNAnuzNJdfWmK5/BylUgFhb6DZAxR9g7rpIos+Wttja2uE9M9vYb69k+DqfJErWzLOuoVqdKlpDo4jODms4zlFckuHObx4iHwuRi6fYqmYY201x/LqMrnFDNl8gRvvukhNdT9394rcOlNFMxrUhYFmW9imTEsp0mhWKC2nWSokyWSiFA7HSKcXiEb34fayyLVniUQk8rmn2drKIqlqiNcuVrh2ymQS1Nh8sc/GoMEbl7u8eb5D1yqTzkRJJsOkkiGymQVShyLE4yHWKnEq1RjpTIhmI8r2y3mkyloI38uha1n6bpVXrkw5+4LD/S/2uHnrIopaJhZ9hmQqwqHkAXJP8BDhyD46ZoWdMxOGcx/PNfECk/8AbxXdRjRliPoAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/34f77/soohwan.png 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/a94f6/soohwan.png 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png 1148w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/38420/soohwan.webp 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7470d/soohwan.webp 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/b5ef6/soohwan.webp 1148w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6829268292682927}}}}]},"fields":{"readingTime":{"text":"3 min read"},"layout":"","slug":"/ray/"}},"next":{"excerpt":"[REVIEW] 크래프톤웨이 (Krafton Way) 배틀그라운드, 테라 등의 게임개발사로 유명한 KRAFTON…","frontmatter":{"title":"[REVIEW] 크래프톤웨이 (Krafton Way)","tags":["book","review"],"date":"2021-07-10T10:00:00.000Z","draft":false,"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAeCAYAAAAsEj5rAAAACXBIWXMAAAsTAAALEwEAmpwYAAAGb0lEQVRIx42TC0yUVxbH78w3yJtBumldE9Os2WZjLSLVrZbW1e7WRDdr6Wop1QpdjSgFsQir8kbNVqGCgJZG23QNrNQXqwLCICjIU+jyEpjhMeowMLyUh4ZVNgX9bb5voLVmN+FLfjnnfvfkf8+551whoi2IuAFE/H3E/iFE3D0rynoEsX/Y+j9+ai9+aj9u0OrLcTKxAyhaIqYXdWgTUmAZ0o4S1MHVClJQBdK2QqSAIqTtJUiBN5BCapGCqpCCb6IO+R4pqPJH1LubETF9CHFgBOmNADQr9yC96o0kW6+ditUs24G0yBfNm0Fo3H3QzH8H6b00NF470bh/gI1XMJqlW5CWfIJ6YxbiwDBCHHyA2vNjNO9EoPHciGb5djQevmje2okquAr1hhNI65LQLPFDetkLlfdxVO/GoZm/ErFkK2KxP9LvIxEhDYg4WTC2D9W2IoTPaYTvWYTPGcTGbMSHZxFBtQi/PMSmS4j1p1D5X0LaWoAUUIwIuI6L32n89ibzTUYWUSev4BZjQIjoXuxizbya1MXrqWaWH+/GI8XMinQzvz7UyaJkE17Hu3jtaA/2sV2IfXcQUd3Yb9exYNlqdgVuJSYmlpjQ7fjsSUOISAu/TBwgKGeUyMJREktH2JU7QmLpKKtPDSv+4ZIRPrkwzJyEAURED+JvY8x562O0NoLlXm+zYf0GVq1cxfp1a+Uu92Gztx3tzgqcdtXwwp567ELrmffXKrRhddiG1vFCWA2O4U3YxJgRUT2IA6PMXbObBS+/iPf7f2bz5s0se30xK97zs46NiOhCvasREaZHhBlQhbchPmtGFT7l79YjZCuLRfcg4od58dMcFnsuxWPhAl6Z/ytWv/1bNh2vlO/QYp0feVDlAY/rR8T2W/3Y53z58GgLqpgeRMw9VIHlOC7zZ/5v3NmyyYevL5VNZSiLzoheRIwFEdHHLw6aedM/Ht/QBLxWrGJHwDZCdwVPCc4QlSy2rxfvjD4SU9I5eSyJ/4yNUlVZQVRUFMVFxTMVtFiJ7sU+zszfL+QTGR6Cvq2TicknyF9yUhKDg4MzF1Syi+zntQQDJ79MZV9kNKWlpVgsFsbGxkhJTsZkMs1QMLoXtVJuH4H/aCXnQhZ/2bKVY8eO0dbeTn9/P5H79nLbaJxuylRjlNL+T7kyUT18d/0W6alHWPfHNaSlpvKvujoePnxISNCnNDc3/48uP7t+Ljv3ZBPnzpzmQ9+PeGN9CJH7D5GXl0OLXo+vzweUlJQg5JlSxVpQxfWhiu1VZuxZ1MrM9Siv5FTZXTK/+oLF3kG4fZROcNQhss+dpqyiHO8/raWwsEB+y0OIvf2Iz4yI8G5ExNBz3EfsGcXzcBu6y5eJj47DaWclYqOO6PhEGpoMVFbX8+4f1lCgu45Y8EUlCw+X4r4/l0WfF+CeXI7H0TIWHinHI6Ucz5QbvHKkmuCvTnEi0Y/de/xZkpDP0rRKcnTfYm6/QkPteXb4/46m2nOIiRYNEy02TBpmcf+mI5ZSB4yFTozW2im2U+fMSI09lhInjPkO3C5yorfCiZ7rtvSXOTDeYAt6NRO3VDxplRDoBdP80KjCVGhLy0UHhipsMeTY0XDeicEyW+4U2tOZ74Tpqh39pTbc1s2iPc+BkWoNT1uFgqwhMAge1UnczNLSKWegs6M9zw5jgSP3K2zR58hCdtxVBG1pvuhK62VHmrKdqflOS3mGC8YCOyZbpgX1gomWWYzUzGakdjaPGlwYq3Ph3/WujDdqedzozHiTjIviD9904+H3WkZrXRmsdGOo2lXZQy8pWYrpVBXaBYNls6g750brZVc6853pyJ+tZG7IccaQq8VU5ExXkQOGXCce16ug7adylQyVhUENnXZgsudx40t05M/DVDyHtjxZzBVj4Vy6rr2EsUDLbZ0WU/FcTMXzeGpwg25naJ/1zB3qBaMdwVSV/RNdThpnMvZzNfco1TcyGTZnMvEgH11uOnXVZ5l8kMv4/YuUXP2W3OwELmR9zrnMg7S3lkLP+zxtlgVbBeOmQIb6KuhoyuRC5l5u1XzDve5rPB7IZMB4gmt5h6ku+ZJh09f8MHSW4rxk8rPjuZIdz/mMMMydBdC91ir4Y/0GAR0Cumyh03o38mGytZRoGSxztN6X3M07Ety1sSLHt4mflzw9RzKTzT9fyzyqUzHeoGI69knLT0zHT+/9F02b6CYnFs8IAAAAAElFTkSuQmCC"},"images":{"fallback":{"src":"/static/6281efa490feec135db893f41708f052/b3219/krafton.png","srcSet":"/static/6281efa490feec135db893f41708f052/b3219/krafton.png 690w","sizes":"100vw"},"sources":[{"srcSet":"/static/6281efa490feec135db893f41708f052/6dfae/krafton.webp 690w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":1.4840579710144928}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAADjElEQVQ4y23Oy48TdQDA8TExEaTttvSBfdHddru7bWc6bWd+8+i0u23n0SdlBRQXZGHDgiiLDwgqXjCIJEaC8cLBmHgwUU9EYzRqQuLJi/4NJB6NF+8evgbOHr7XT76SM5ix7s4QnRHnnm9yZzvBTIswN8JcP5bC67ZoWi7CdtFsn5bloZo+dSNAMYbIIqCmD6mJIVXdR3J6I7r9IQ3T59pZlW+uxzndCbPdC/HZXoq3TikowsfqBOjOmFZ7SrM9p2kfRbXn1O0jKNbsSbJ5BMnu9Gl3+qiiz4OPSvz0QRLHFGwOZH6+neLHT5YRbRerM0J0JrTas/8FZXNKzZgg2es+ZtvDdbs8+iHJl+9lWVP7OG2br26k+PvXJCfmFqoYIpwhTXtMwxpTN0coxhjFmCKbE2rGmKoYPQY9qi2fyzsq//4e5sNXi5TlLqbd5t6lPH/9kuDTG2usqgGi7dO0hzSsIXUzQDaGKGKMLEZUxZCKCJB02+PUpMH9vQwP7ybYnSxyfNrjwumA3dEiD94v8cfnK7xzWsbpujQtH9X0qBs+iuEjC5+a7lPVPSqai9SyelyYFbjkh5hqC3TkKOutLON2nkYpgtuMcT6IszdPslo8SLlcpmUH1A0XRQyQhUtNuFQ1l4o2QFI1h3OTVXbcg+grIfSVBRqlEGopgrYSRi0e4KiIse1mScSeopA/iLADmoZL3Rgg631q+oCq1qeq95FOzhQefpzn7eMpBmqcoBVj04pzwkky0eNMRYIdL8F3twusLe2nslrE6g5pGX1U8/FhD1nvUW1tUJK7SH9+n+DR1zF2vQSvH0lz88wiV45V2J1WuHqsxLXNAnuzNJdfWmK5/BylUgFhb6DZAxR9g7rpIos+Wttja2uE9M9vYb69k+DqfJErWzLOuoVqdKlpDo4jODms4zlFckuHObx4iHwuRi6fYqmYY201x/LqMrnFDNl8gRvvukhNdT9394rcOlNFMxrUhYFmW9imTEsp0mhWKC2nWSokyWSiFA7HSKcXiEb34fayyLVniUQk8rmn2drKIqlqiNcuVrh2ymQS1Nh8sc/GoMEbl7u8eb5D1yqTzkRJJsOkkiGymQVShyLE4yHWKnEq1RjpTIhmI8r2y3mkyloI38uha1n6bpVXrkw5+4LD/S/2uHnrIopaJhZ9hmQqwqHkAXJP8BDhyD46ZoWdMxOGcx/PNfECk/8AbxXdRjRliPoAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/34f77/soohwan.png 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/a94f6/soohwan.png 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png 1148w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/38420/soohwan.webp 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7470d/soohwan.webp 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/b5ef6/soohwan.webp 1148w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6829268292682927}}}}]},"fields":{"readingTime":{"text":"2 min read"},"layout":"","slug":"/krafton/"}},"primaryTag":"nlp"}},
    "staticQueryHashes": ["3170763342","3229353822"]}