{ "componentChunkName": "component---src-templates-post-tsx", "path": "/developed-moe/", "result": {"data":{"logo":null,"markdownRemark":{"html":"

MoE(Mixture of Experts) 기초부터 DeepSeek 혁신까지

\n

작년 이맘때쯤 DeepSeek-V3가 저비용으로 엄청난 성능을 보이면서 화제가 되었습니다. 그 핵심 기술인 MoE(Mixture of Experts)에 대해서 어떻게 발전시켰는지 살펴보겠습니다.

\n

1. MoE란?

\n

핵심 아이디어

\n

기존 트랜스포머 구조처럼 모든 파라미터를 항상 사용하지말고, 입력에 따라 선택적으로 활성화해서 쓰자!

\n

부분적 활성화는 MoE의 핵심입니다. 기존 Dense Model처럼 모든 토큰 생성마다 모든 파라미터를 전부 사용하는게 아니라, 각 토큰별로 전문가(Expert)를 두고 선택적으로 활성화하자는 겁니다.

\n

2. 기본 MoE 구조

\n

Transformer에서 MoE 적용

\n\n \n \n \"load\n \n \n

문제 인식

\n

Expert가 자연스럽게 특화되는 건 좋지만, 문제가 생깁니다.

\n

256개의 Expert 중 N개의 Expert만 운 좋게 초반에 잘 맞추게 된다면, 해당하는 N개의 Expert들만 집중적으로 학습되게 됩니다.

\n

문제점:

\n\n

Auxiliary Loss 도입

\n

이러한 문제점을 해결하기 위해 Auxiliary Loss를 도입했습니다. Auxiliary Loss는 N개의 숫자가 있을 때 다양하게 선택이 되도록 학습되는 Loss입니다.

\n\n
total_loss = main_loss + λ * aux_loss = \"정답 맞추기\" + \"골고루 쓰기\"
\n

그래서 MoE 구조에서 처음에는 이렇게 Auxiliary Loss를 도입해서 load balancing 문제를 해결했습니다.

\n

Auxiliary Loss의 문제점

\n

위 Total Loss 수식에서 보듯이, 전체 로스에서 Auxiliary Loss가 더해지게 됩니다. 여기서 문제가 생기는데,

\n

딜레마:

\n\n

위와 같은 딜레마가 생기게 됩니다.

\n

7. DeepSeek의 혁신 ③: Auxiliary-Loss-Free

\n

핵심 아이디어

\n

꼭 Auxiliary Loss로 해야해? 라우팅만 골고루 하게 하면 되는거 아니야?

\n
\n

기존 방식에서는 router가 expert들을 골고루 선택하게 하기 위해 Auxiliary Loss를 도입했는데, DeepSeek에서는 이거를 bias control로 loss 없이 구현해냈습니다!

\n

매 N-step마다 expert들의 사용량을 체크하고, 상대적으로 많이 사용되는 expert의 bias는 -하고, 적게 사용되는 expert의 bias는 +하는 방식으로 컨트롤합니다.

\n\n \n \n \"auxiliary\n \n \n

이 단순한 방식으로 DeepSeek은 load balancing 문제를 풀었습니다.

\n

Auxiliary Loss 도입으로 인해 전체 학습에 방해가 되니, 이를 간단한 bias control로 gradient에 영향이 없도록 하여 성능을 개선했습니다!

\n

실제로 이러한 bias 방식을 도입했더니, auxiliary loss를 사용했을때랑 대비해서 expert들간의 불균형도가 조금 높아지긴 했지만, 모델 성능 자체는 더 좋아졌다고 합니다.

\n

이상으로 DeepSeek에서 MoE 구조를 어떻게 개선했는지 살펴봤습니다.

","htmlAst":{"type":"root","children":[{"type":"element","tagName":"h1","properties":{},"children":[{"type":"text","value":"MoE(Mixture of Experts) 기초부터 DeepSeek 혁신까지"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"작년 이맘때쯤 DeepSeek-V3가 저비용으로 엄청난 성능을 보이면서 화제가 되었습니다. 그 핵심 기술인 MoE(Mixture of Experts)에 대해서 어떻게 발전시켰는지 살펴보겠습니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"1. MoE란?"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"핵심 아이디어"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"기존 트랜스포머 구조처럼 모든 파라미터를 항상 사용하지말고, 입력에 따라 선택적으로 활성화해서 쓰자!"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"부분적 활성화"}]},{"type":"text","value":"는 MoE의 핵심입니다. 기존 Dense Model처럼 모든 토큰 생성마다 모든 파라미터를 전부 사용하는게 아니라, 각 토큰별로 전문가(Expert)를 두고 선택적으로 활성화하자는 겁니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"2. 기본 MoE 구조"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Transformer에서 MoE 적용"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"span","properties":{"className":["gatsby-resp-image-wrapper"],"style":"position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 1081px; "},"children":[{"type":"text","value":"\n "},{"type":"element","tagName":"a","properties":{"className":["gatsby-resp-image-link"],"href":"/static/e35ee3338f574bcbacd9958e6f7982af/0c139/ffn-moe.png","style":"display: block","target":"_blank","rel":["noopener"]},"children":[{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["gatsby-resp-image-background-image"],"style":"padding-bottom: 51%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAYAAAC0VX7mAAAACXBIWXMAAAsTAAALEwEAmpwYAAABzUlEQVQoz22S3W6jMBSE8/6vtJft7kW3arP5awI0ZBNogBAMCcE22Hwr07RKqx3pyJZ1PJpzZkZ93/O9HG7vDrWUpMcj3fXNnYei4FTX3GL0+fGGoOs6tNYY06GVGipfv7J7fqZcv1KGa6pNSPxnTBr4qGuP+z8a1AD2Sta27UDm4BpFrTicFdv1lnDhke327IINb2GEP1mwCyPyWnOsaowxjNosRSzmvI2fuQQ+3eXyKV9KSXKSBFnN42rLbJcTloborHkKYlbJiVVyxk9OxMV5mGx0riWbTYznh6SZoOvsVbMjVOzLC+lZElcN06igbHsKZRGqZRoJ4pMibzRRXtF1hpFKE7LJmL+PD5x8D6sVH1YoqUgrSS4t27Lmx3TBQVly1VNZuF8FLJMM0cJe1BinsG/2VG9PRN5PTLnCds3nyK3WiLLkKCpSz6d6WVCufJLpjMPihWq5JHtZkhcCIUqstc4UizEaKd3uLFqrYRcfO2xbjbU9to7Rhwk6n5P4v5DZDFPMUUcPaw1N07yb0vMVzmznsiN15Zx2FQRLkmTndGOMpBQHwtCjaaqh3xEOCv8X6o8s3iKKYh5+P3Jp5HvQ6wt3d/cEweuXYP8Deuj9tBFyijAAAAAASUVORK5CYII='); background-size: cover; display: block;"},"children":[]},{"type":"text","value":"\n "},{"type":"element","tagName":"img","properties":{"className":["gatsby-resp-image-image"],"alt":"ffn moe","title":"ffn moe","src":"/static/e35ee3338f574bcbacd9958e6f7982af/0c139/ffn-moe.png","srcSet":["/static/e35ee3338f574bcbacd9958e6f7982af/0eb09/ffn-moe.png 500w","/static/e35ee3338f574bcbacd9958e6f7982af/1263b/ffn-moe.png 1000w","/static/e35ee3338f574bcbacd9958e6f7982af/0c139/ffn-moe.png 1081w"],"sizes":"(max-width: 1081px) 100vw, 1081px","style":"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;","loading":"lazy","decoding":"async"},"children":[]},{"type":"text","value":"\n "}]},{"type":"text","value":"\n "}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"전통적인 Transformer의 FFN(Feed-Forward Network) 레이어를 MoE로 교체합니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h4","properties":{},"children":[{"type":"text","value":"Mixtral 8x7B 예시"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"8개의 experts (각 7B 파라미터)"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"토큰당 2개만 선택 (Top-2 routing)"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"총 47B 파라미터 중 활성 13B"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"FFN layer만 MoE로 교체, attention은 shared"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"3. DeepSeek의 혁신 ①: Fine-Grained Expert Segmentation"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h4","properties":{},"children":[{"type":"text","value":"문제 인식"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"기존 MoE(ex. Mixtral)의 한계:"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"text"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-text"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"8개의 expert 중 2개 선택\n→ 조합의 경우의 수: 28가지\n→ 제한적인 유연성"}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h4","properties":{},"children":[{"type":"text","value":"해결책 : Expert를 더 여러개로 나누기"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"span","properties":{"className":["gatsby-resp-image-wrapper"],"style":"position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 1200px; "},"children":[{"type":"text","value":"\n "},{"type":"element","tagName":"a","properties":{"className":["gatsby-resp-image-link"],"href":"/static/ede4e2c2c9e86e2000e11ebd764ba9a6/03ffe/finegrained-segmentation.jpg","style":"display: block","target":"_blank","rel":["noopener"]},"children":[{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["gatsby-resp-image-background-image"],"style":"padding-bottom: 49.8%; position: relative; bottom: 0; left: 0; background-image: url('data:image/jpeg;base64,/9j/2wBDAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQH/2wBDAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQH/wgARCAAKABQDASIAAhEBAxEB/8QAGQAAAgMBAAAAAAAAAAAAAAAAAAQCBQYJ/8QAFQEBAQAAAAAAAAAAAAAAAAAAAQD/2gAMAwEAAhADEAAAAe6aEKgNGPk//8QAGxAAAgIDAQAAAAAAAAAAAAAAAQQCAwAFERP/2gAIAQEAAQUCkw0LouNyzXw6taB6NARp1Z6h/8QAGBEAAgMAAAAAAAAAAAAAAAAAARCBodH/2gAIAQMBAT8Bk1i//8QAGxEAAQQDAAAAAAAAAAAAAAAAkQEQESFhodL/2gAIAQIBAT8BvAXo6hv/xAAkEAABAgYBBAMAAAAAAAAAAAABAgMAERITISJhBDEyQVFxgf/aAAgBAQAGPwJ1tKQpYTUlq62JIrWEu1W8VappVOXJBh5NCCG3HELcvtotSHiqbNJo22AzjtCDZHYSUOqcXckhILm2UzUDrxP3BwNntseW3v5/YcCRIBt0gDGaDn75jpyc6CP/xAAbEAEAAgMBAQAAAAAAAAAAAAABESEAMUFRYf/aAAgBAQABPyF0eUq7nZSh26AbyrUJrzZImiVLmvhhQ7IQTZCljJ3AYUARz5W1Lb7hACUgGqQgE7F/cZIpmVlqi35Wf//aAAwDAQACAAMAAAAQbC//xAAeEQACAAYDAAAAAAAAAAAAAAABEQAhMUFxoYGx0f/aAAgBAwEBPxBaOr85mzOMUs1TgAaEf//EABkRAQEBAQEBAAAAAAAAAAAAAAERITEAQf/aAAgBAgEBPxBpOQdlU+CqKvQaYO/AwrsLMKSwVgpysqV9/8QAGRABAQEBAQEAAAAAAAAAAAAAAREhAEFh/9oACAEBAAE/EHG7n0BLAq6R6fSd+l0RrWshHDyiuGtw4/bjaghM1mApHASXwVJVSeYwehwQQDSqet7qZ/QisAK4AGB3/9k='); background-size: cover; display: block;"},"children":[]},{"type":"text","value":"\n "},{"type":"element","tagName":"img","properties":{"className":["gatsby-resp-image-image"],"alt":"finegrained segmentation","title":"finegrained segmentation","src":"/static/ede4e2c2c9e86e2000e11ebd764ba9a6/03ffe/finegrained-segmentation.jpg","srcSet":["/static/ede4e2c2c9e86e2000e11ebd764ba9a6/953fe/finegrained-segmentation.jpg 500w","/static/ede4e2c2c9e86e2000e11ebd764ba9a6/0a251/finegrained-segmentation.jpg 1000w","/static/ede4e2c2c9e86e2000e11ebd764ba9a6/03ffe/finegrained-segmentation.jpg 1200w"],"sizes":"(max-width: 1200px) 100vw, 1200px","style":"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;","loading":"lazy","decoding":"async"},"children":[]},{"type":"text","value":"\n "}]},{"type":"text","value":"\n "}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"Expert 수를 늘리고, 선택의 경우의 수도 늘리되, 계산량은 유지:"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"text"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-text"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"[기존 Mixtral] \n8개 expert, 2개 선택 \n각 expert = 7B 파라미터\n → 14B 활성화 \n \n [DeepSeek Fine-Grained] \n 32개 expert, 8개 선택 (4배씩 증가) \n 각 expert = 1.75B 파라미터 \n → 14B 활성화 (동일!) \n → 조합의 경우의 수: 🆙"}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h4","properties":{},"children":[{"type":"text","value":"Multi-Head Attention과의 유사성"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"이 개념은 개인적으로 "},{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"Single-head Attention → Multi-head Attention"}]},{"type":"text","value":"과 유사하게 느껴집니다."}]},{"type":"text","value":"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"},{"type":"element","tagName":"table","properties":{},"children":[{"type":"element","tagName":"thead","properties":{},"children":[{"type":"element","tagName":"tr","properties":{},"children":[{"type":"element","tagName":"th","properties":{},"children":[{"type":"text","value":"구분"}]},{"type":"element","tagName":"th","properties":{},"children":[{"type":"text","value":"Single → Multi"}]},{"type":"element","tagName":"th","properties":{},"children":[{"type":"text","value":"Coarse → Fine-Grained"}]}]}]},{"type":"element","tagName":"tbody","properties":{},"children":[{"type":"element","tagName":"tr","properties":{},"children":[{"type":"element","tagName":"td","properties":{},"children":[{"type":"text","value":"나누기"}]},{"type":"element","tagName":"td","properties":{},"children":[{"type":"text","value":"1개 → 8개 head"}]},{"type":"element","tagName":"td","properties":{},"children":[{"type":"text","value":"8개 → 256개 expert"}]}]},{"type":"element","tagName":"tr","properties":{},"children":[{"type":"element","tagName":"td","properties":{},"children":[{"type":"text","value":"활성화"}]},{"type":"element","tagName":"td","properties":{},"children":[{"type":"text","value":"모두 사용"}]},{"type":"element","tagName":"td","properties":{},"children":[{"type":"text","value":"일부만 선택"}]}]},{"type":"element","tagName":"tr","properties":{},"children":[{"type":"element","tagName":"td","properties":{},"children":[{"type":"text","value":"효과"}]},{"type":"element","tagName":"td","properties":{},"children":[{"type":"text","value":"다양한 attention 패턴"}]},{"type":"element","tagName":"td","properties":{},"children":[{"type":"text","value":"다양한 expert 조합"}]}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"이처럼 기존의 덩어리를 더 작지만 여러개로 나누면서 "},{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"성능 향상"}]},{"type":"text","value":"이라는 효과를 얻었습니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"4. DeepSeek의 혁신 ②: Shared Expert Isolation"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h4","properties":{},"children":[{"type":"text","value":"문제 인식"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"Fine-grained로 expert를 잘게 쪼갰더니 새로운 문제 발생:"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"여러 expert가 같은 기본 지식을 중복해서 학습"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"파라미터 낭비.."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h4","properties":{},"children":[{"type":"text","value":"원인 :"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"각 expert가 독립적으로 학습하다보니 다들 공통 지식을 각자 배움"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"공통 지식을 배우다보니 특화된 지식을 배울 공간(파라미터)이 부족해진다는 단점"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h4","properties":{},"children":[{"type":"text","value":"해결책 : 공통 지식을 따로 빼자!"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"span","properties":{"className":["gatsby-resp-image-wrapper"],"style":"position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 779px; "},"children":[{"type":"text","value":"\n "},{"type":"element","tagName":"a","properties":{"className":["gatsby-resp-image-link"],"href":"/static/ac1e43f2a0e5d4f9f4c2d4acceed8a00/a3e85/shared-expert.png","style":"display: block","target":"_blank","rel":["noopener"]},"children":[{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["gatsby-resp-image-background-image"],"style":"padding-bottom: 65.4%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsTAAALEwEAmpwYAAACRUlEQVQ4y4WT30/TUBTH+Td99x8x8c34YEIwEAwILgY2F0eGD8agMcGHCS44XNut3VbSsrXd2rWsW3/uY1ocDIh6kntP7809n557zveuzOdzMlv4zMIoJJiG/FBqvP72kp2vG+zW1ukaCv+yjLGyDMu853m4rkuSQk+oU916RmXrOZ/evkDvtYniBMuyEEWRer3O6ekphmHcArNpARyPHYJgln8nwFBqoR18QCofYB5+5Eq/5Go65UJVOTo64rzRQBCEHD6bXcetLGCTyYRZENyk77guhiQiV97TKL/j4rCK2ekgtmVGwyGapjG/d92bDIMgyIFpHBL7Dmk4xfE8RrJMq1SiUSzSKZeZmSaqpuG6HkPLQmkJmNbwbg3jOEbtdRn7IZe1EsqrR3QrTxDaCqYkoVQqnJVKXFSrjLpdmpLEz18ivc+bKDuPGZsatuPm5cqBaZqSxHH+h+mghX78Blv4gjGyMUSR1t4e9UIBeX8fX9dpKTKGZdOvH6JWn+LZJiPbwXGc2y5fp5vm/mp2DfeDAK3RQCwUqO/u0isWkb6fcC6I/5TOTZcXzXHdMUkckeGtZpPm2hq11VU6m5voZ2coqpqfS9ME5inL8Xd0uABmOkyThHQ+x7dtnHYbtVbDUxTGpkmn2+V+zPKjWFleLIALy2obhyFuv08ax/i+T6fTeQB5cGXTNBkMBui6Tr/fx880GUV5DU82NjheX6exvY3ebtOS5f8DoyjKtZiNMAzzke2F2d5kQuj7BL6fv4bgj/j/BvwNDzrOP0Nj28gAAAAASUVORK5CYII='); background-size: cover; display: block;"},"children":[]},{"type":"text","value":"\n "},{"type":"element","tagName":"img","properties":{"className":["gatsby-resp-image-image"],"alt":"shared expert","title":"shared expert","src":"/static/ac1e43f2a0e5d4f9f4c2d4acceed8a00/a3e85/shared-expert.png","srcSet":["/static/ac1e43f2a0e5d4f9f4c2d4acceed8a00/0eb09/shared-expert.png 500w","/static/ac1e43f2a0e5d4f9f4c2d4acceed8a00/a3e85/shared-expert.png 779w"],"sizes":"(max-width: 779px) 100vw, 779px","style":"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;","loading":"lazy","decoding":"async"},"children":[]},{"type":"text","value":"\n "}]},{"type":"text","value":"\n "}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"Shared Expert"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"항상 활성화하는 Expert를 따로 둠으로써, 이 expert가 공통 지식을 담당→ 파라미터 효율 증가"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"나머지 Expert들의 특화도 증가 : 나머지 Expert들은 공통 지식을 제외한 특화 지식에만 집중되도록 학습"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"5. Expert는 어떻게 특화되는거지?"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"여기서 놀라운 사실인데, 따로 특화시키지 않습니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"text"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-text"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"연구자가 하는 것:\n❌ \"너는 수학 expert야!\"\n❌ \"너는 코드 expert야!\"\n❌ Expert별로 다른 데이터 주기\n\n실제로 하는 것:\n✅ 그냥 전체 데이터로 같이 학습\n✅ Router와 Experts를 end-to-end로 학습\n✅ 알아서 특화됨!"}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"랜덤하게 초기화 된 Expert들은 학습이 되면서 자연스럽게 수학에 강한 Expert, 코딩에 강한 Expert와 같이 나뉘어지게 됩니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"실제 Mixtral 8x7B 분석 결과:"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"text"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-text"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"Expert 0: 코드 관련 토큰에 강하게 반응 \nExpert 2: 수학 수식에 강함 \nExpert 5: 프랑스어에 특화 \nExpert 7: 범용적 (다양한 토큰)"}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"6. Load Balancing"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"span","properties":{"className":["gatsby-resp-image-wrapper"],"style":"position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 649px; "},"children":[{"type":"text","value":"\n "},{"type":"element","tagName":"a","properties":{"className":["gatsby-resp-image-link"],"href":"/static/31f9e3519f7670549d2a990711cb5e99/1bcec/load-balancing.png","style":"display: block","target":"_blank","rel":["noopener"]},"children":[{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["gatsby-resp-image-background-image"],"style":"padding-bottom: 71.8%; position: relative; bottom: 0; left: 0; background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAAA7DAAAOwwHHb6hkAAAC4ElEQVQ4y42TS28cVRCF/fMQ/4MNgh0IwRoWWSKEZIgEzAqEEEpC7AQciCNMHNvxY2b8mumxu6f7Prp7pqdf0495eD7UFyE7CyRqU1e36p66VefUGkAcx0ipUNrHEwqtfaTUuJ5ECIVtD0mTlMVsRpGm/JetVivWmoNSmjSbEo5T/nzVxxYRwo/J84JpUSOVT5pmLJdLkjhGex55npPnmfFZllGWpQE1gL4fMC0q4qzg52cdnuxa9IcjPDFCqAjbkeaR1pqL40POD/YIfc0o9BkFGiU9ho59C6iUz3K5YHPrmPc/afHexy2+uP+Usqqp6gVBOCaOE7Jpzub6lzjdDtcHf6B6bao8Jo8CfK3u/DAIiCYZZ5cejzYP2Pj1kJ4lTbtFWaP90ACmecbGV+t0nj/h6HGL3qtt5GCAc36KbfVvAcMwpGcJBteaS0tw5fgmeLq7Q2//Be2d3w1xSZbx8N6H3H/nLb57922s3S2WNzfkSYRW8hawYXU2m+PJMV//8JLttoccZcirAd3DE6yzDmmWEfqK1w+/5afPPuDBvY+wj15ws1pRTpv5qjdJyacFV8OAT9e3+eZxl7blswImaY3j+mRZTuBr4kAQSYeRd00SSpJAEikXJdxbwCAISJKcgRvy/fM+n/94wKOtDpYluOgJ+taQeDIxbQ0OnnG58wvW3m9M04jA6SH7XeRdQO0HVFXNUES0Nrq0nnbYO7lmPl9Q1nPDciObRnPCdTh//RcXhy/pnewzOGujpGQymdyRjf5H2FleYrshjhxTVDPqemZYVjowLP9rZRZTJBPycWAkc7NcvrkpTfVGOp6n2Ns/5eTkkqPjC9qdHqPRCCEldV3zf2ytSVyaCiuzarajCcIYrSN8f8JiMTfx2WzGYrGgrmrKoqAqS6qqoq4q45u7JmfNHQ4RQuDYDp7rmuF6notWAikFtm3jeR79Xt/EGz1OoogoikiShOl0anwUjc2+/w0I0QjJCAKqgwAAAABJRU5ErkJggg=='); background-size: cover; display: block;"},"children":[]},{"type":"text","value":"\n "},{"type":"element","tagName":"img","properties":{"className":["gatsby-resp-image-image"],"alt":"load balancing","title":"load balancing","src":"/static/31f9e3519f7670549d2a990711cb5e99/1bcec/load-balancing.png","srcSet":["/static/31f9e3519f7670549d2a990711cb5e99/0eb09/load-balancing.png 500w","/static/31f9e3519f7670549d2a990711cb5e99/1bcec/load-balancing.png 649w"],"sizes":"(max-width: 649px) 100vw, 649px","style":"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;","loading":"lazy","decoding":"async"},"children":[]},{"type":"text","value":"\n "}]},{"type":"text","value":"\n "}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h4","properties":{},"children":[{"type":"text","value":"문제 인식"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"Expert가 자연스럽게 특화되는 건 좋지만, 문제가 생깁니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"256개의 Expert 중 N개의 Expert만 운 좋게 초반에 잘 맞추게 된다면, 해당하는 N개의 Expert들만 집중적으로 학습되게 됩니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"문제점:"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"나머지 256 - N개의 expert 파라미터 낭비"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"N개의 expert에 과부하"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"다양성 X"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"256개로 나눈 의미가 없어짐"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Auxiliary Loss 도입"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"이러한 문제점을 해결하기 위해 Auxiliary Loss를 도입했습니다. Auxiliary Loss는 N개의 숫자가 있을 때 다양하게 선택이 되도록 학습되는 Loss입니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"Total Loss"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"text"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-text"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"total_loss = main_loss + λ * aux_loss = \"정답 맞추기\" + \"골고루 쓰기\""}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"그래서 MoE 구조에서 처음에는 이렇게 Auxiliary Loss를 도입해서 load balancing 문제를 해결했습니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Auxiliary Loss의 문제점"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"위 Total Loss 수식에서 보듯이, 전체 로스에서 Auxiliary Loss가 더해지게 됩니다. 여기서 문제가 생기는데,"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"딜레마:"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"λ 크게 설정: 골고루 쓰지만 성능 나쁨"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"λ 작게 설정: 성능 좋지만 불균형 심각"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"위와 같은 딜레마가 생기게 됩니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"7. DeepSeek의 혁신 ③: Auxiliary-Loss-Free"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h4","properties":{},"children":[{"type":"text","value":"핵심 아이디어"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"꼭 Auxiliary Loss로 해야해? 라우팅만 골고루 하게 하면 되는거 아니야?"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"hr","properties":{},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"기존 방식에서는 router가 expert들을 골고루 선택하게 하기 위해 Auxiliary Loss를 도입했는데, DeepSeek에서는 이거를 bias control로 loss 없이 구현해냈습니다!"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"매 N-step마다 expert들의 사용량을 체크하고, 상대적으로 많이 사용되는 expert의 bias는 -하고, 적게 사용되는 expert의 bias는 +하는 방식으로 컨트롤합니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"span","properties":{"className":["gatsby-resp-image-wrapper"],"style":"position: relative; display: block; margin-left: auto; margin-right: auto; max-width: 1200px; "},"children":[{"type":"text","value":"\n "},{"type":"element","tagName":"a","properties":{"className":["gatsby-resp-image-link"],"href":"/static/31078f1e5aaa98d931f3a37147bfe73e/03ffe/auxiliary-loss-free.jpg","style":"display: block","target":"_blank","rel":["noopener"]},"children":[{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["gatsby-resp-image-background-image"],"style":"padding-bottom: 62%; position: relative; bottom: 0; left: 0; background-image: url('data:image/jpeg;base64,/9j/2wBDAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQH/2wBDAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQH/wgARCAAMABQDASIAAhEBAxEB/8QAGQAAAgMBAAAAAAAAAAAAAAAAAAcDBAYJ/8QAFQEBAQAAAAAAAAAAAAAAAAAAAAH/2gAMAwEAAhADEAAAAe70a4pQ7zOlf//EABwQAAICAgMAAAAAAAAAAAAAAAIDAQQFFBUhM//aAAgBAQABBQJvs1O3C+gnJXZPkbglUNxp/8QAGREAAgMBAAAAAAAAAAAAAAAAEcEQUWHh/9oACAEDAQE/AXa6cEf/xAAZEQACAwEAAAAAAAAAAAAAAAAR4RAxYcH/2gAIAQIBAT8B5TQ0x//EACcQAAIBAgUDBAMAAAAAAAAAAAECIQMRABITMVEjQWEEIjJScXOB/9oACAEBAAY/AnzpmtpaYR7M1zJO2zdiWXKL2E4K1NWiqNGizUqjH9gsCDwsTvwomBb3S39Pc8nDdbYso6Pp9s1TmkfqPHjDWqic5PRoGRIM0zaZiwv2wrNWYk2PwogC6LAC0hH5ufOP/8QAGhABAQEAAwEAAAAAAAAAAAAAAREhADFBUf/aAAgBAQABPyFbYOM0BlpcfQAod9iWD3AMQRGm5wQbMFsBQtsexVdr3y2nZXY9FfctStcHBmxo8wkNjYXgy0yBqZnKqeiYAP/aAAwDAQACAAMAAAAQ3M//xAAaEQEBAAIDAAAAAAAAAAAAAAABEQBBITFh/9oACAEDAQE/EBAYF4KLA8yLtr1suf/EABkRAQEAAwEAAAAAAAAAAAAAAAERACExQf/aAAgBAgEBPxBrBWJQKVZd1oSh1XzDP//EABkQAQEAAwEAAAAAAAAAAAAAAAERACExQf/aAAgBAQABPxCOg3dXF1SfE8as86qrtLRGGNrPmTikyYgAAGpGtsdMDI71QeCoSpnOegkpIBiqLQExwQeHRCobTyv/2Q=='); background-size: cover; display: block;"},"children":[]},{"type":"text","value":"\n "},{"type":"element","tagName":"img","properties":{"className":["gatsby-resp-image-image"],"alt":"auxiliary loss free","title":"auxiliary loss free","src":"/static/31078f1e5aaa98d931f3a37147bfe73e/03ffe/auxiliary-loss-free.jpg","srcSet":["/static/31078f1e5aaa98d931f3a37147bfe73e/953fe/auxiliary-loss-free.jpg 500w","/static/31078f1e5aaa98d931f3a37147bfe73e/0a251/auxiliary-loss-free.jpg 1000w","/static/31078f1e5aaa98d931f3a37147bfe73e/03ffe/auxiliary-loss-free.jpg 1200w"],"sizes":"(max-width: 1200px) 100vw, 1200px","style":"width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0;","loading":"lazy","decoding":"async"},"children":[]},{"type":"text","value":"\n "}]},{"type":"text","value":"\n "}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"이 단순한 방식으로 DeepSeek은 load balancing 문제를 풀었습니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"Auxiliary Loss 도입으로 인해 전체 학습에 방해가 되니, 이를 간단한 bias control로 gradient에 영향이 없도록 하여 성능을 개선했습니다!"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"실제로 이러한 bias 방식을 도입했더니, auxiliary loss를 사용했을때랑 대비해서 expert들간의 불균형도가 조금 높아지긴 했지만, 모델 성능 자체는 더 좋아졌다고 합니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"이상으로 DeepSeek에서 MoE 구조를 어떻게 개선했는지 살펴봤습니다."}]}],"data":{"quirksMode":false}},"excerpt":"MoE(Mixture of Experts) 기초부터 DeepSeek 혁신까지 작년 이맘때쯤 DeepSeek-V3가 저비용으로 엄청난 성능을 보이면서 화제가 되었습니다. 그 핵심 기술인 MoE(Mixture of Experts…","fields":{"readingTime":{"text":"8 min read"}},"frontmatter":{"title":"MoE(Mixture of Experts) 기초부터 DeepSeek 혁신까지","userDate":"15 January 2026","date":"2026-01-15T01:11:55.000Z","tags":["nlp","mixtral"],"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","backgroundColor":"#f8f8f8","images":{"fallback":{"src":"/static/6d82ca0c08d95b33eb8b13cfb947a064/943f0/deepseek-moe.png","srcSet":"/static/6d82ca0c08d95b33eb8b13cfb947a064/f118f/deepseek-moe.png 750w,\n/static/6d82ca0c08d95b33eb8b13cfb947a064/7e80a/deepseek-moe.png 1080w,\n/static/6d82ca0c08d95b33eb8b13cfb947a064/29f78/deepseek-moe.png 1366w,\n/static/6d82ca0c08d95b33eb8b13cfb947a064/943f0/deepseek-moe.png 1400w","sizes":"100vw"},"sources":[{"srcSet":"/static/6d82ca0c08d95b33eb8b13cfb947a064/d2a19/deepseek-moe.webp 750w,\n/static/6d82ca0c08d95b33eb8b13cfb947a064/72f43/deepseek-moe.webp 1080w,\n/static/6d82ca0c08d95b33eb8b13cfb947a064/8992a/deepseek-moe.webp 1366w,\n/static/6d82ca0c08d95b33eb8b13cfb947a064/d652b/deepseek-moe.webp 1400w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6678571428571428}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"children":[{"gatsbyImageData":{"layout":"fullWidth","backgroundColor":"#282838","images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/0d6f4/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/248f9/soohwan.png 40w,\n/static/a9e6b445142b247ee4cfa66155398bb2/fd435/soohwan.png 80w,\n/static/a9e6b445142b247ee4cfa66155398bb2/0d6f4/soohwan.png 120w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/e7f45/soohwan.webp 40w,\n/static/a9e6b445142b247ee4cfa66155398bb2/589ec/soohwan.webp 80w,\n/static/a9e6b445142b247ee4cfa66155398bb2/71a38/soohwan.webp 120w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6833333333333333}}]}}]}},"relatedPosts":{"totalCount":50,"edges":[{"node":{"id":"6529d72e-80e7-5d61-a672-d66276a3f641","excerpt":"MoE(Mixture of Experts) 기초부터 DeepSeek 혁신까지 작년 이맘때쯤 DeepSeek-V3가 저비용으로 엄청난 성능을 보이면서 화제가 되었습니다. 그 핵심 기술인 MoE(Mixture of Experts…","frontmatter":{"title":"MoE(Mixture of Experts) 기초부터 DeepSeek 혁신까지","date":"2026-01-15T01:11:55.000Z"},"fields":{"readingTime":{"text":"8 min read"},"slug":"/developed-moe/"}}},{"node":{"id":"8b49d3ef-4ce3-568c-9d71-240ff17fc3e0","excerpt":"BERT는 사실 Diffusion 모델이였다?! 최근 굉장히 흥미로운 글을 읽게되어 공유합니다. 원문 : link BERT와 Diffusion이 같은 방식이다?! NLP 연구자들에게 BERT는 너무 익숙한 모델입니다. 201…","frontmatter":{"title":"BERT는 사실 Diffusion 모델이였다?!","date":"2025-10-21T12:00:00.000Z"},"fields":{"readingTime":{"text":"8 min read"},"slug":"/bert_diffusion/"}}},{"node":{"id":"7c7e1676-ea84-58de-970c-ddec9aa10660","excerpt":"RLHF는 수다쟁이를 만든다?! (Does RLHF Breed Verbose Chatterboxes?!) RLHF(Reinforcement Learning from Human Feedback)는 OpenAI의 ChatGPT…","frontmatter":{"title":"RLHF는 수다쟁이를 만든다?! (Does RLHF Breed Verbose Chatterboxes?!)","date":"2024-03-13T01:11:55.000Z"},"fields":{"readingTime":{"text":"7 min read"},"slug":"/rlhf-vervosity/"}}},{"node":{"id":"2b9d3e22-1796-5fab-878d-5941d2e76e9d","excerpt":"LLM Paper Abstract - 2023.12 LLM…","frontmatter":{"title":"LLM Paper Abstract - 2023.12","date":"2024-01-05T10:00:00.000Z"},"fields":{"readingTime":{"text":"5 min read"},"slug":"/llm-abs-202312/"}}},{"node":{"id":"f43fa33b-917b-5c00-8462-937d99592ad7","excerpt":"What it MoE? (Mixture of Experts) 현존 최강 LLM인 GPT-4에서 “MoE (Mixture of Experts)” 방식을 채택하여 사용하고 있다고 알려졌는데요, 최근 AI계의 뜨거운 감자 Mistral AI…","frontmatter":{"title":"What is MoE? (Mixture of Experts)","date":"2023-12-22T01:11:55.000Z"},"fields":{"readingTime":{"text":"9 min read"},"slug":"/moe/"}}}]}},"pageContext":{"slug":"/developed-moe/","prev":{"excerpt":"Throwback 2025 2020년 회고 글 2021년 회고 글 2023년 회고 글 2024년 회고 글 2025년 회고 글입니다! 202…","frontmatter":{"title":"Throwback 2025","tags":["record"],"date":"2026-01-01T10:00:00.000Z","draft":false,"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAIAAAAmMtkJAAAACXBIWXMAAAsTAAALEwEAmpwYAAADJElEQVQozwEZA+b8AHKmrYe8v4XCxJ2zpau6qrvHt5HGwZvEu53GxY62toy6t8DEsLvFssHXxsXayMvjzsHayLa/rKS3rn+srgBkoKjU1cOjvKzGzLvIw66tr56LvLWfwLWjysZ/traTwLzt3cba1L7F1MHK3Mne2sDU1r3WwqS7rJtyp6oAbKKnxNfOl7675ObcvtDFXnFyKB8waoaGsuDUkbW1kbuy0NHE8/Hiucy6r8Kv3NvKu8eztsKwo7aseaemAGmjp3OqnlOJlpqVk561p3J9e1VHTXaKioCtqIOino23sZaZk5+em9DMuMvArNfMu7S3sq3Fu77KwnKfowBhmqF8nZRckJdydX15cHB5e3d/mZeBfXZ9Z1xyioV3fnJTTFphYG24qpq8w6+vpJSYgnmIk4yUo5ltlZkAU4mTl5OMe3hxMF5zZ2VmhZCMdrm/bnx7solvhYmCfXx0enl2qZyLzLmb1cCexKuKq5Fyr5iFnnxva3p5AHKXooent32XpnKXqHmbpnuisoutuoSVl5GDc5aJeot5a4uFe5WZk7q0pqmup5GXk6CgnLaxopKRj3WFigBlp8RywfF3ud6N2f95sc6VxdjC5e7D3+bO2NWvurhbZHBveYdZd5F7maB7q7plm7CHuqyRpaaJlp5mfIoAda/Aqt3tn7nBy/DygaWsiqmwq8jIs9ngt9jWg5qZdmZlmI+CnqCV2tvQqsamnL6hnb6RjqyNmpmPhoN9AIm0vMfV2X+CicfTwdDQv8LHvKjCwUqJj4xvaZRkWkA3QU5vcZWslurg1bq1oO3q17m1lKCHTmtwamVaVgBsk5SFlIRxamqEeHmAj5Gnk4O9rpuCtrmcq6p7iYkSQXZAWXg/XXY5YodpeHp9hpBseYNNQUlEWXpDUGsAT3iAfHtssqWLN0NXFGeVopOQgn98U5OhhbW/do6KCSNNK0BcOEdnFEB1RklaYlpnTkpXGzRZBCVYKThXAFl2foN8cpmWeldmZzlTcrGvr6mnoGp6dYCRln99gCgzUDVAVzJDYCxFZT1IWjJKbDNKaTFFYSIzVTxDWM9ywkFQtIcUAAAAAElFTkSuQmCC"},"images":{"fallback":{"src":"/static/f6779a574676afa1bc96d6af9d6c9005/6e90d/2025.png","srcSet":"/static/f6779a574676afa1bc96d6af9d6c9005/ea847/2025.png 750w,\n/static/f6779a574676afa1bc96d6af9d6c9005/4a806/2025.png 1080w,\n/static/f6779a574676afa1bc96d6af9d6c9005/e2aad/2025.png 1366w,\n/static/f6779a574676afa1bc96d6af9d6c9005/6e90d/2025.png 1536w","sizes":"100vw"},"sources":[{"srcSet":"/static/f6779a574676afa1bc96d6af9d6c9005/57584/2025.webp 750w,\n/static/f6779a574676afa1bc96d6af9d6c9005/984df/2025.webp 1080w,\n/static/f6779a574676afa1bc96d6af9d6c9005/4a276/2025.webp 1366w,\n/static/f6779a574676afa1bc96d6af9d6c9005/7db9c/2025.webp 1536w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6666666666666666}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAADjElEQVQ4y23Oy48TdQDA8TExEaTttvSBfdHddru7bWc6bWd+8+i0u23n0SdlBRQXZGHDgiiLDwgqXjCIJEaC8cLBmHgwUU9EYzRqQuLJi/4NJB6NF+8evgbOHr7XT76SM5ix7s4QnRHnnm9yZzvBTIswN8JcP5bC67ZoWi7CdtFsn5bloZo+dSNAMYbIIqCmD6mJIVXdR3J6I7r9IQ3T59pZlW+uxzndCbPdC/HZXoq3TikowsfqBOjOmFZ7SrM9p2kfRbXn1O0jKNbsSbJ5BMnu9Gl3+qiiz4OPSvz0QRLHFGwOZH6+neLHT5YRbRerM0J0JrTas/8FZXNKzZgg2es+ZtvDdbs8+iHJl+9lWVP7OG2br26k+PvXJCfmFqoYIpwhTXtMwxpTN0coxhjFmCKbE2rGmKoYPQY9qi2fyzsq//4e5sNXi5TlLqbd5t6lPH/9kuDTG2usqgGi7dO0hzSsIXUzQDaGKGKMLEZUxZCKCJB02+PUpMH9vQwP7ybYnSxyfNrjwumA3dEiD94v8cfnK7xzWsbpujQtH9X0qBs+iuEjC5+a7lPVPSqai9SyelyYFbjkh5hqC3TkKOutLON2nkYpgtuMcT6IszdPslo8SLlcpmUH1A0XRQyQhUtNuFQ1l4o2QFI1h3OTVXbcg+grIfSVBRqlEGopgrYSRi0e4KiIse1mScSeopA/iLADmoZL3Rgg631q+oCq1qeq95FOzhQefpzn7eMpBmqcoBVj04pzwkky0eNMRYIdL8F3twusLe2nslrE6g5pGX1U8/FhD1nvUW1tUJK7SH9+n+DR1zF2vQSvH0lz88wiV45V2J1WuHqsxLXNAnuzNJdfWmK5/BylUgFhb6DZAxR9g7rpIos+Wttja2uE9M9vYb69k+DqfJErWzLOuoVqdKlpDo4jODms4zlFckuHObx4iHwuRi6fYqmYY201x/LqMrnFDNl8gRvvukhNdT9394rcOlNFMxrUhYFmW9imTEsp0mhWKC2nWSokyWSiFA7HSKcXiEb34fayyLVniUQk8rmn2drKIqlqiNcuVrh2ymQS1Nh8sc/GoMEbl7u8eb5D1yqTzkRJJsOkkiGymQVShyLE4yHWKnEq1RjpTIhmI8r2y3mkyloI38uha1n6bpVXrkw5+4LD/S/2uHnrIopaJhZ9hmQqwqHkAXJP8BDhyD46ZoWdMxOGcx/PNfECk/8AbxXdRjRliPoAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/34f77/soohwan.png 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/a94f6/soohwan.png 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png 1148w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/38420/soohwan.webp 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7470d/soohwan.webp 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/b5ef6/soohwan.webp 1148w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6829268292682927}}}}]},"fields":{"readingTime":{"text":"16 min read"},"layout":"","slug":"/2025/"}},"next":{"excerpt":"Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models Paper Review 흥미로운 최신 AI Agent…","frontmatter":{"title":"Agentic Context Engineering (ACE) Paper Review","tags":["ai","agent"],"date":"2026-02-02T01:11:55.000Z","draft":false,"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAACXBIWXMAAAsTAAALEwEAmpwYAAAEz0lEQVQ4ywHEBDv7AAIFCwYMFwYNGQAJFQkUIiMuPRAfMhUoPiI3UBQtSSU8VxcxTRswSBksQxoqPQwaLAYRIAoQHAYLFAUIDQALDBIOEhsQFiELFSMTIjRmcn+BjJhab4RcdoxEa4o2Y4RdfJVeeI9YbYFpd4YlNEQNFCEaGB0QEBUKDRUAEhcgHyMsJSYuMTA3ECQ2CSU+V26BVHCIVHWOUH2ZToCdWYOeTW2HS2V6Fi9HLS02Sy8kXDQhPCgjGhkfAA0gNTE6SVVEQEhITCg/URo3Ti5PaEBogluFnGCPp2KRp0x9l0dnfWp1gE5DQXNEKnhEJXFOPEdARSAuQgAJJ0cxT2xoUkmEZFFYWl1iYFxIWGAeUm8pY4BbgphSe5E7aYNUWVx4UTybVSmTWTVlXFwzS2g4UW4ULUkADhwtFylAOz5LgGlgzpp057B+yriVU3iGY3V9v6NvxKJpbmpmeWBOgm1ekXZjQFlxSWeEY3SBTmqBJThPAColJZdbOLF3S8yXarB9Wqp2VbeKZHCRlE54hcGtetu2eG9sZ2VjYWZ4hH+LkRZil0JqhYl3a1pZXCYoMAApMj7ElHf7wHn9w3ScZz9UTUlucWpHkKB/rbi7v7p8k5hOdIRXZ29IcIoeW4QhV3lKV149OTxONy0uHhsAGik6P0FJfV9PqYd0rJ2Ti5mUSpCbYZCeqqmwvMXOiJujQHCCVXB7P2V4fYmM5tTIyry/b2BfRDk3VzUlABMoPi83QUZTZXOGoX2TqK6trVqUnUmAmmN4kYift4aUpjdke0FkeEJNW36Bhlh0jGp/l2VvhkdGTkhBQQAYHyomPVEyTWlYb4dXdYtrfY9gi5Q8iZ9WbYCBhZJvfYhBaHtKZ3heZ21yeIF8kaxygZxESVxINzI4JyIADx8xHS09KD5SVm2NeJKturq6UpamVoyhmZukZ4CatbS6b29xX3qKXYyrjo+RtausfYOWW1lpOywqXzUeACslKRQaKTdEW3yImai3x6Omqz1UYWFgXnl9hld3nGN5llhZYHhraSRSgUdjgpyTkauqtltecCEeJjglHgAnJy4UMksTNlRtfI6NpLYpS20BIkhMTFhQXGZCYXlLW2piX2J/VDxpQC9dR0NpT0d3Z203NUAYGCArHRgACx0xKUVZHEJdDzJRKUZfLEhjGkJlLUtoT2BsaHFxh3JgtW821oExzH89s2w7s249ml82b0UsgUwkOiEVABArQC5CUhU5UhRBXixSbS5Tbz5fdTxccUJcanZvZoxjSblyOPjAePa+gOqlZu6kWOWhaaNlP4hCED0nHgAEFSgSPloeTmwrW3kvaok+f5tAfJVUhZhtgYV4a198VDi9hFbwrHLRk2jRlWvXmmy1i3F1WEssHhwfHSMACRUlESU4FjJLFzdSJ0lgPWmDIFh4UY+pbJCcf2ZQUEQ+WktFYlJKcV1Te2decF1SZ1xaTUdLJh8eLCIiAAYPHA0bLAkfOA4nQBw2TRJKbTZ0k0FxiUdXX0ZISzlBS04vIFY0IVUwHzwoIDUrKzMjICgbGiUaGRMQFAACCBMDDR0FGC8DHj8FPGcWTG8SL0oLIDciJjIUIzQlIyZNLRxTLhkwHRgnHBk1KSYtHRgmFA4UERQEBQosoZ6A+zi13gAAAABJRU5ErkJggg=="},"images":{"fallback":{"src":"/static/42daa50b4ec0a0a56cae6e25c0e2d977/b5658/ace.png","srcSet":"/static/42daa50b4ec0a0a56cae6e25c0e2d977/f054e/ace.png 750w,\n/static/42daa50b4ec0a0a56cae6e25c0e2d977/b5658/ace.png 1024w","sizes":"100vw"},"sources":[{"srcSet":"/static/42daa50b4ec0a0a56cae6e25c0e2d977/4f03f/ace.webp 750w,\n/static/42daa50b4ec0a0a56cae6e25c0e2d977/67ded/ace.webp 1024w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":1}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAADjElEQVQ4y23Oy48TdQDA8TExEaTttvSBfdHddru7bWc6bWd+8+i0u23n0SdlBRQXZGHDgiiLDwgqXjCIJEaC8cLBmHgwUU9EYzRqQuLJi/4NJB6NF+8evgbOHr7XT76SM5ix7s4QnRHnnm9yZzvBTIswN8JcP5bC67ZoWi7CdtFsn5bloZo+dSNAMYbIIqCmD6mJIVXdR3J6I7r9IQ3T59pZlW+uxzndCbPdC/HZXoq3TikowsfqBOjOmFZ7SrM9p2kfRbXn1O0jKNbsSbJ5BMnu9Gl3+qiiz4OPSvz0QRLHFGwOZH6+neLHT5YRbRerM0J0JrTas/8FZXNKzZgg2es+ZtvDdbs8+iHJl+9lWVP7OG2br26k+PvXJCfmFqoYIpwhTXtMwxpTN0coxhjFmCKbE2rGmKoYPQY9qi2fyzsq//4e5sNXi5TlLqbd5t6lPH/9kuDTG2usqgGi7dO0hzSsIXUzQDaGKGKMLEZUxZCKCJB02+PUpMH9vQwP7ybYnSxyfNrjwumA3dEiD94v8cfnK7xzWsbpujQtH9X0qBs+iuEjC5+a7lPVPSqai9SyelyYFbjkh5hqC3TkKOutLON2nkYpgtuMcT6IszdPslo8SLlcpmUH1A0XRQyQhUtNuFQ1l4o2QFI1h3OTVXbcg+grIfSVBRqlEGopgrYSRi0e4KiIse1mScSeopA/iLADmoZL3Rgg631q+oCq1qeq95FOzhQefpzn7eMpBmqcoBVj04pzwkky0eNMRYIdL8F3twusLe2nslrE6g5pGX1U8/FhD1nvUW1tUJK7SH9+n+DR1zF2vQSvH0lz88wiV45V2J1WuHqsxLXNAnuzNJdfWmK5/BylUgFhb6DZAxR9g7rpIos+Wttja2uE9M9vYb69k+DqfJErWzLOuoVqdKlpDo4jODms4zlFckuHObx4iHwuRi6fYqmYY201x/LqMrnFDNl8gRvvukhNdT9394rcOlNFMxrUhYFmW9imTEsp0mhWKC2nWSokyWSiFA7HSKcXiEb34fayyLVniUQk8rmn2drKIqlqiNcuVrh2ymQS1Nh8sc/GoMEbl7u8eb5D1yqTzkRJJsOkkiGymQVShyLE4yHWKnEq1RjpTIhmI8r2y3mkyloI38uha1n6bpVXrkw5+4LD/S/2uHnrIopaJhZ9hmQqwqHkAXJP8BDhyD46ZoWdMxOGcx/PNfECk/8AbxXdRjRliPoAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/34f77/soohwan.png 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/a94f6/soohwan.png 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png 1148w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/38420/soohwan.webp 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7470d/soohwan.webp 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/b5ef6/soohwan.webp 1148w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6829268292682927}}}}]},"fields":{"readingTime":{"text":"13 min read"},"layout":"post","slug":"/ace/"}},"primaryTag":"nlp"}}, "staticQueryHashes": ["3170763342","3229353822"]}