{
    "componentChunkName": "component---src-templates-post-tsx",
    "path": "/deepspeed/",
    "result": {"data":{"logo":null,"markdownRemark":{"html":"<h1>DeepSpeed Usage</h1>\n<p>최근 인공지능 모델은 사이즈가 점점 커지는 추세입니다.</p>\n<img src=\"https://neurohive.io/wp-content/uploads/2020/02/rsz_ddad-scaled.png\" width=\"600\">  \n<p>다른 어떤 방법보다도 그냥 더 큰 모델, 더 많은 데이터를 집어넣으면 더 좋은 모델이 만들어지는게 현실인 것 같습니다.\n문제는 큰 모델은 학습시키기가 어렵다는 건데요, 많이 쓰이는 비싼 GPU A100이 메모리 40GB인데, 2B 넘어가는 모델은\n학습시에 GPU 1대에 올릴수가 없어서 Model Parallelism과 같은 방법으로 모델을 쪼개서 GPU에 올리거나 해야합니다.</p>\n<img src=\"https://xiandong79.github.io/downloads/ddl1.png\" width=\"400\">  \n<p>근데 이런 Model Parallelism을 직접 하나하나 구현하려면 굉장히 어렵습니다.\n하지만 고맙게도 Microsoft에서 DeepSpeed라는 오픈소스를 공개해주었습니다.</p>\n<p>DeepSpeed는 지나치게 큰 모델에 대한 트레이닝에 어려움을 겪는 개발자들을 위해 기존보다 10배 이상 큰 모델을 5배 이상의 속도로 게다가 코드 변화 거의 없이 학습 시킬 수 있는 혁명적인 오픈소스 라이브러리입니다.</p>\n<p>이번 포스트에서는 DeepSpeed 사용법에 대해 간단하게 기록해둘까 합니다.<br>\n예제 코드는 2.7B의 파라미터를 가진 GPT-Neo 2.7B 모델로 테스트 했습니다.</p>\n<p>Dataset 클래스 정의 등의 디테일한거는 스킵했습니다.</p>\n<h2>기존 PyTorch Code</h2>\n<div class=\"gatsby-highlight\" data-language=\"python\"><pre class=\"language-python\"><code class=\"language-python\"><span class=\"token keyword\">import</span> torch\n<span class=\"token keyword\">from</span> torch<span class=\"token punctuation\">.</span>utils<span class=\"token punctuation\">.</span>data <span class=\"token keyword\">import</span> DataLoader\n<span class=\"token keyword\">from</span> transformers <span class=\"token keyword\">import</span> GPTNeoForCausalLM<span class=\"token punctuation\">,</span> AutoTokenizer<span class=\"token punctuation\">,</span> LineByLineDataset\n\ndevice <span class=\"token operator\">=</span> torch<span class=\"token punctuation\">.</span>device<span class=\"token punctuation\">(</span><span class=\"token string\">'cuda'</span> <span class=\"token keyword\">if</span> torch<span class=\"token punctuation\">.</span>cuda<span class=\"token punctuation\">.</span>is_available<span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span> <span class=\"token keyword\">else</span> <span class=\"token string\">'cpu'</span><span class=\"token punctuation\">)</span>\n\ntokenizer <span class=\"token operator\">=</span> AutoTokenizer<span class=\"token punctuation\">.</span>from_pretrained<span class=\"token punctuation\">(</span><span class=\"token string\">'EleutherAI/gpt-neo-2.7B'</span><span class=\"token punctuation\">)</span>\nmodel <span class=\"token operator\">=</span> GPTNeoForCausalLM<span class=\"token punctuation\">.</span>from_pretrained<span class=\"token punctuation\">(</span><span class=\"token string\">'EleutherAI/gpt-neo-2.7B'</span><span class=\"token punctuation\">)</span>\n\ndataset <span class=\"token operator\">=</span> LineByLineDataset<span class=\"token punctuation\">(</span><span class=\"token punctuation\">.</span><span class=\"token punctuation\">.</span><span class=\"token punctuation\">.</span><span class=\"token punctuation\">)</span>\ndata_loader <span class=\"token operator\">=</span> DataLoader<span class=\"token punctuation\">(</span>dataset<span class=\"token operator\">=</span>dataset<span class=\"token punctuation\">,</span>\n                         num_workers<span class=\"token operator\">=</span>args<span class=\"token punctuation\">.</span>num_workers<span class=\"token punctuation\">,</span>\n                         batch_size<span class=\"token operator\">=</span>args<span class=\"token punctuation\">.</span>batch_size<span class=\"token punctuation\">,</span>\n                         drop_last<span class=\"token operator\">=</span><span class=\"token boolean\">False</span><span class=\"token punctuation\">)</span>\n\noptimizer <span class=\"token operator\">=</span> torch<span class=\"token punctuation\">.</span>optim<span class=\"token punctuation\">.</span>Adam<span class=\"token punctuation\">(</span>model<span class=\"token punctuation\">.</span>parameters<span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">,</span> lr<span class=\"token operator\">=</span>args<span class=\"token punctuation\">.</span>lr<span class=\"token punctuation\">)</span>\nscheduler <span class=\"token operator\">=</span> torch<span class=\"token punctuation\">.</span>optim<span class=\"token punctuation\">.</span>lr_scheduler<span class=\"token punctuation\">.</span>OneCycleLR<span class=\"token punctuation\">(</span>optimizer<span class=\"token operator\">=</span>optimizer<span class=\"token punctuation\">,</span>\n                                                max_lr<span class=\"token operator\">=</span>args<span class=\"token punctuation\">.</span>lr<span class=\"token punctuation\">,</span>\n                                                epochs<span class=\"token operator\">=</span>args<span class=\"token punctuation\">.</span>num_epochs<span class=\"token punctuation\">,</span>\n                                                steps_per_epoch<span class=\"token operator\">=</span><span class=\"token builtin\">len</span><span class=\"token punctuation\">(</span>data_loader<span class=\"token punctuation\">)</span><span class=\"token punctuation\">,</span>\n                                                anneal_strategy<span class=\"token operator\">=</span><span class=\"token string\">'linear'</span><span class=\"token punctuation\">)</span>\n\n<span class=\"token keyword\">for</span> step<span class=\"token punctuation\">,</span> batch <span class=\"token keyword\">in</span> <span class=\"token builtin\">enumerate</span><span class=\"token punctuation\">(</span>data_loader<span class=\"token punctuation\">)</span><span class=\"token punctuation\">:</span>\n    input_ids<span class=\"token punctuation\">,</span> attention_masks<span class=\"token punctuation\">,</span> labels <span class=\"token operator\">=</span> batch\n\n    input_ids <span class=\"token operator\">=</span> input_ids<span class=\"token punctuation\">.</span>to<span class=\"token punctuation\">(</span>device<span class=\"token punctuation\">)</span>\n    attention_masks <span class=\"token operator\">=</span> attention_masks<span class=\"token punctuation\">.</span>to<span class=\"token punctuation\">(</span>device<span class=\"token punctuation\">)</span>\n    labels <span class=\"token operator\">=</span> labels<span class=\"token punctuation\">.</span>to<span class=\"token punctuation\">(</span>device<span class=\"token punctuation\">)</span>\n\n    outputs <span class=\"token operator\">=</span> model<span class=\"token punctuation\">(</span>input_ids<span class=\"token operator\">=</span>input_ids<span class=\"token punctuation\">,</span> attention_mask<span class=\"token operator\">=</span>attention_masks<span class=\"token punctuation\">,</span> labels<span class=\"token operator\">=</span>labels<span class=\"token punctuation\">)</span>\n    loss <span class=\"token operator\">=</span> outputs<span class=\"token punctuation\">.</span>loss\n\n    optimizer<span class=\"token punctuation\">.</span>zero_grad<span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span>\n    loss<span class=\"token punctuation\">.</span>backward<span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span>\n    torch<span class=\"token punctuation\">.</span>nn<span class=\"token punctuation\">.</span>utils<span class=\"token punctuation\">.</span>clip_grad_norm_<span class=\"token punctuation\">(</span>model<span class=\"token punctuation\">.</span>parameters<span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">,</span> args<span class=\"token punctuation\">.</span>gradient_clip_val<span class=\"token punctuation\">)</span>\n    optimizer<span class=\"token punctuation\">.</span>step<span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span>\n    scheduler<span class=\"token punctuation\">.</span>step<span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span></code></pre></div>\n<h2>DeepSpeed Code</h2>\n<div class=\"gatsby-highlight\" data-language=\"python\"><pre class=\"language-python\"><code class=\"language-python\"><span class=\"token keyword\">import</span> torch\n<span class=\"token keyword\">import</span> deepspeed\n<span class=\"token keyword\">from</span> torch<span class=\"token punctuation\">.</span>utils<span class=\"token punctuation\">.</span>data <span class=\"token keyword\">import</span> DataLoader\n<span class=\"token keyword\">from</span> transformers <span class=\"token keyword\">import</span> GPTNeoForCausalLM<span class=\"token punctuation\">,</span> AutoTokenizer<span class=\"token punctuation\">,</span> LineByLineDataset\n\ntorch<span class=\"token punctuation\">.</span>distributed<span class=\"token punctuation\">.</span>init_process_group<span class=\"token punctuation\">(</span>backend<span class=\"token operator\">=</span><span class=\"token string\">\"nccl\"</span><span class=\"token punctuation\">)</span>\ndeepspeed<span class=\"token punctuation\">.</span>init_distributed<span class=\"token punctuation\">(</span><span class=\"token string\">\"nccl\"</span><span class=\"token punctuation\">)</span>\n\ndevice <span class=\"token operator\">=</span> torch<span class=\"token punctuation\">.</span>device<span class=\"token punctuation\">(</span><span class=\"token string\">'cuda'</span> <span class=\"token keyword\">if</span> torch<span class=\"token punctuation\">.</span>cuda<span class=\"token punctuation\">.</span>is_available<span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span> <span class=\"token keyword\">else</span> <span class=\"token string\">'cpu'</span><span class=\"token punctuation\">)</span>\n\ntokenizer <span class=\"token operator\">=</span> AutoTokenizer<span class=\"token punctuation\">.</span>from_pretrained<span class=\"token punctuation\">(</span><span class=\"token string\">'EleutherAI/gpt-neo-2.7B'</span><span class=\"token punctuation\">)</span>\nmodel <span class=\"token operator\">=</span> GPTNeoForCausalLM<span class=\"token punctuation\">.</span>from_pretrained<span class=\"token punctuation\">(</span><span class=\"token string\">'EleutherAI/gpt-neo-2.7B'</span><span class=\"token punctuation\">)</span>\n\nconfig <span class=\"token operator\">=</span> <span class=\"token punctuation\">{</span>\n    <span class=\"token string\">\"train_batch_size\"</span><span class=\"token punctuation\">:</span> <span class=\"token number\">1</span><span class=\"token punctuation\">,</span>\n    <span class=\"token string\">\"fp16\"</span><span class=\"token punctuation\">:</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token string\">\"enabled\"</span><span class=\"token punctuation\">:</span> <span class=\"token boolean\">True</span><span class=\"token punctuation\">,</span>\n    <span class=\"token punctuation\">}</span><span class=\"token punctuation\">,</span>\n    <span class=\"token string\">\"zero_optimization\"</span><span class=\"token punctuation\">:</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token string\">\"stage\"</span><span class=\"token punctuation\">:</span> <span class=\"token number\">2</span><span class=\"token punctuation\">,</span>\n        <span class=\"token string\">\"contiguous_gradients\"</span><span class=\"token punctuation\">:</span> <span class=\"token boolean\">True</span><span class=\"token punctuation\">,</span>\n        <span class=\"token string\">\"overlap_comm\"</span><span class=\"token punctuation\">:</span> <span class=\"token boolean\">True</span><span class=\"token punctuation\">,</span>\n        <span class=\"token string\">\"offload_optimizer\"</span><span class=\"token punctuation\">:</span> <span class=\"token punctuation\">{</span>\n            <span class=\"token string\">\"device\"</span><span class=\"token punctuation\">:</span> <span class=\"token string\">\"cpu\"</span><span class=\"token punctuation\">,</span>\n            <span class=\"token string\">\"pin_memory\"</span><span class=\"token punctuation\">:</span> <span class=\"token boolean\">True</span><span class=\"token punctuation\">,</span>\n            <span class=\"token string\">\"fast_init\"</span><span class=\"token punctuation\">:</span> <span class=\"token boolean\">True</span>\n        <span class=\"token punctuation\">}</span><span class=\"token punctuation\">,</span>\n    <span class=\"token punctuation\">}</span><span class=\"token punctuation\">,</span>\n    <span class=\"token string\">\"optimizer\"</span><span class=\"token punctuation\">:</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token string\">\"type\"</span><span class=\"token punctuation\">:</span> <span class=\"token string\">\"Adam\"</span><span class=\"token punctuation\">,</span>\n        <span class=\"token string\">\"params\"</span><span class=\"token punctuation\">:</span> <span class=\"token punctuation\">{</span>\n            <span class=\"token string\">\"lr\"</span><span class=\"token punctuation\">:</span> <span class=\"token number\">3e-05</span><span class=\"token punctuation\">,</span>\n            <span class=\"token string\">\"betas\"</span><span class=\"token punctuation\">:</span> <span class=\"token punctuation\">[</span>\n                <span class=\"token number\">0.9</span><span class=\"token punctuation\">,</span>\n                <span class=\"token number\">0.999</span>\n            <span class=\"token punctuation\">]</span><span class=\"token punctuation\">,</span>\n            <span class=\"token string\">\"eps\"</span><span class=\"token punctuation\">:</span> <span class=\"token number\">1e-8</span>\n        <span class=\"token punctuation\">}</span>\n    <span class=\"token punctuation\">}</span><span class=\"token punctuation\">,</span>\n<span class=\"token punctuation\">}</span>\n\ndataset <span class=\"token operator\">=</span> LineByLineDataset<span class=\"token punctuation\">(</span><span class=\"token punctuation\">.</span><span class=\"token punctuation\">.</span><span class=\"token punctuation\">.</span><span class=\"token punctuation\">)</span>\ndata_loader <span class=\"token operator\">=</span> DataLoader<span class=\"token punctuation\">(</span>dataset<span class=\"token operator\">=</span>dataset<span class=\"token punctuation\">,</span>\n                         num_workers<span class=\"token operator\">=</span>args<span class=\"token punctuation\">.</span>num_workers<span class=\"token punctuation\">,</span>\n                         batch_size<span class=\"token operator\">=</span>args<span class=\"token punctuation\">.</span>batch_size<span class=\"token punctuation\">,</span>\n                         drop_last<span class=\"token operator\">=</span><span class=\"token boolean\">False</span><span class=\"token punctuation\">)</span>\n\nmodel<span class=\"token punctuation\">,</span> optimizer<span class=\"token punctuation\">,</span> _<span class=\"token punctuation\">,</span> _ <span class=\"token operator\">=</span> deepspeed<span class=\"token punctuation\">.</span>initialize<span class=\"token punctuation\">(</span>model<span class=\"token operator\">=</span>model<span class=\"token punctuation\">,</span>\n                                              config_params<span class=\"token operator\">=</span>config<span class=\"token punctuation\">,</span>\n                                              model_parameters<span class=\"token operator\">=</span>model<span class=\"token punctuation\">.</span>parameters<span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span>\n\n<span class=\"token keyword\">for</span> step<span class=\"token punctuation\">,</span> batch <span class=\"token keyword\">in</span> <span class=\"token builtin\">enumerate</span><span class=\"token punctuation\">(</span>data_loader<span class=\"token punctuation\">)</span><span class=\"token punctuation\">:</span>\n    input_ids<span class=\"token punctuation\">,</span> attention_masks<span class=\"token punctuation\">,</span> labels <span class=\"token operator\">=</span> batch\n\n    input_ids <span class=\"token operator\">=</span> input_ids<span class=\"token punctuation\">.</span>to<span class=\"token punctuation\">(</span>device<span class=\"token punctuation\">)</span>\n    attention_masks <span class=\"token operator\">=</span> attention_masks<span class=\"token punctuation\">.</span>to<span class=\"token punctuation\">(</span>device<span class=\"token punctuation\">)</span>\n    labels <span class=\"token operator\">=</span> labels<span class=\"token punctuation\">.</span>to<span class=\"token punctuation\">(</span>device<span class=\"token punctuation\">)</span>\n\n    outputs <span class=\"token operator\">=</span> model<span class=\"token punctuation\">(</span>input_ids<span class=\"token operator\">=</span>input_ids<span class=\"token punctuation\">,</span> attention_mask<span class=\"token operator\">=</span>attention_masks<span class=\"token punctuation\">,</span> labels<span class=\"token operator\">=</span>labels<span class=\"token punctuation\">)</span>\n    loss <span class=\"token operator\">=</span> outputs<span class=\"token punctuation\">.</span>loss\n\n    model<span class=\"token punctuation\">.</span>backward<span class=\"token punctuation\">(</span>loss<span class=\"token punctuation\">)</span>\n    model<span class=\"token punctuation\">.</span>step<span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span></code></pre></div>\n<p>기존 코드에서 큰 변화 없이 몇 가지 사항만 수정해주면 deepspeed 적용이 가능합니다.</p>","htmlAst":{"type":"root","children":[{"type":"element","tagName":"h1","properties":{},"children":[{"type":"text","value":"DeepSpeed Usage"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"최근 인공지능 모델은 사이즈가 점점 커지는 추세입니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://neurohive.io/wp-content/uploads/2020/02/rsz_ddad-scaled.png","width":600},"children":[]},{"type":"text","value":"  \n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"다른 어떤 방법보다도 그냥 더 큰 모델, 더 많은 데이터를 집어넣으면 더 좋은 모델이 만들어지는게 현실인 것 같습니다.\n문제는 큰 모델은 학습시키기가 어렵다는 건데요, 많이 쓰이는 비싼 GPU A100이 메모리 40GB인데, 2B 넘어가는 모델은\n학습시에 GPU 1대에 올릴수가 없어서 Model Parallelism과 같은 방법으로 모델을 쪼개서 GPU에 올리거나 해야합니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"img","properties":{"src":"https://xiandong79.github.io/downloads/ddl1.png","width":400},"children":[]},{"type":"text","value":"  \n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"근데 이런 Model Parallelism을 직접 하나하나 구현하려면 굉장히 어렵습니다.\n하지만 고맙게도 Microsoft에서 DeepSpeed라는 오픈소스를 공개해주었습니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"DeepSpeed는 지나치게 큰 모델에 대한 트레이닝에 어려움을 겪는 개발자들을 위해 기존보다 10배 이상 큰 모델을 5배 이상의 속도로 게다가 코드 변화 거의 없이 학습 시킬 수 있는 혁명적인 오픈소스 라이브러리입니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"이번 포스트에서는 DeepSpeed 사용법에 대해 간단하게 기록해둘까 합니다."},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n예제 코드는 2.7B의 파라미터를 가진 GPT-Neo 2.7B 모델로 테스트 했습니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"Dataset 클래스 정의 등의 디테일한거는 스킵했습니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"기존 PyTorch Code"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"python"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-python"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-python"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" torch\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"from"}]},{"type":"text","value":" torch"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"utils"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"data "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" DataLoader\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"from"}]},{"type":"text","value":" transformers "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" GPTNeoForCausalLM"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" AutoTokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" LineByLineDataset\n\ndevice "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" torch"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"device"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'cuda'"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"if"}]},{"type":"text","value":" torch"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"cuda"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"is_available"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"else"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'cpu'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\ntokenizer "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" AutoTokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"from_pretrained"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'EleutherAI/gpt-neo-2.7B'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\nmodel "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" GPTNeoForCausalLM"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"from_pretrained"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'EleutherAI/gpt-neo-2.7B'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\ndataset "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" LineByLineDataset"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\ndata_loader "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" DataLoader"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"dataset"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"dataset"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n                         num_workers"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"num_workers"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n                         batch_size"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"batch_size"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n                         drop_last"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"False"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\noptimizer "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" torch"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"optim"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"Adam"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"model"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"parameters"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" lr"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"lr"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\nscheduler "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" torch"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"optim"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"lr_scheduler"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"OneCycleLR"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"optimizer"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"optimizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n                                                max_lr"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"lr"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n                                                epochs"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"num_epochs"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n                                                steps_per_epoch"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","builtin"]},"children":[{"type":"text","value":"len"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"data_loader"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n                                                anneal_strategy"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'linear'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"for"}]},{"type":"text","value":" step"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" batch "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"in"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","builtin"]},"children":[{"type":"text","value":"enumerate"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"data_loader"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":"\n    input_ids"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" attention_masks"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" labels "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" batch\n\n    input_ids "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" input_ids"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"to"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"device"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n    attention_masks "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" attention_masks"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"to"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"device"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n    labels "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" labels"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"to"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"device"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\n    outputs "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" model"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"input_ids"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"input_ids"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" attention_mask"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"attention_masks"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" labels"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"labels"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n    loss "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" outputs"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"loss\n\n    optimizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"zero_grad"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n    loss"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"backward"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n    torch"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"nn"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"utils"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"clip_grad_norm_"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"model"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"parameters"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"gradient_clip_val"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n    optimizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"step"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n    scheduler"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"step"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"DeepSpeed Code"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"python"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-python"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-python"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" torch\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" deepspeed\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"from"}]},{"type":"text","value":" torch"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"utils"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"data "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" DataLoader\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"from"}]},{"type":"text","value":" transformers "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" GPTNeoForCausalLM"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" AutoTokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" LineByLineDataset\n\ntorch"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"distributed"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"init_process_group"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"backend"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"nccl\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\ndeepspeed"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"init_distributed"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"nccl\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\ndevice "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" torch"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"device"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'cuda'"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"if"}]},{"type":"text","value":" torch"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"cuda"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"is_available"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"else"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'cpu'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\ntokenizer "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" AutoTokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"from_pretrained"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'EleutherAI/gpt-neo-2.7B'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\nmodel "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" GPTNeoForCausalLM"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"from_pretrained"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'EleutherAI/gpt-neo-2.7B'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\nconfig "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"{"}]},{"type":"text","value":"\n    "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"train_batch_size\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","number"]},"children":[{"type":"text","value":"1"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n    "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"fp16\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"{"}]},{"type":"text","value":"\n        "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"enabled\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"True"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n    "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"}"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n    "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"zero_optimization\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"{"}]},{"type":"text","value":"\n        "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"stage\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","number"]},"children":[{"type":"text","value":"2"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n        "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"contiguous_gradients\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"True"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n        "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"overlap_comm\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"True"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n        "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"offload_optimizer\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"{"}]},{"type":"text","value":"\n            "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"device\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"cpu\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n            "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"pin_memory\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"True"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n            "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"fast_init\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"True"}]},{"type":"text","value":"\n        "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"}"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n    "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"}"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n    "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"optimizer\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"{"}]},{"type":"text","value":"\n        "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"type\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"Adam\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n        "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"params\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"{"}]},{"type":"text","value":"\n            "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"lr\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","number"]},"children":[{"type":"text","value":"3e-05"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n            "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"betas\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"["}]},{"type":"text","value":"\n                "},{"type":"element","tagName":"span","properties":{"className":["token","number"]},"children":[{"type":"text","value":"0.9"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n                "},{"type":"element","tagName":"span","properties":{"className":["token","number"]},"children":[{"type":"text","value":"0.999"}]},{"type":"text","value":"\n            "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"]"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n            "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"eps\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","number"]},"children":[{"type":"text","value":"1e-8"}]},{"type":"text","value":"\n        "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"}"}]},{"type":"text","value":"\n    "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"}"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"}"}]},{"type":"text","value":"\n\ndataset "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" LineByLineDataset"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\ndata_loader "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" DataLoader"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"dataset"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"dataset"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n                         num_workers"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"num_workers"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n                         batch_size"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"batch_size"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n                         drop_last"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"False"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\nmodel"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" optimizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" _"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" _ "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" deepspeed"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"initialize"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"model"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"model"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n                                              config_params"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"config"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n                                              model_parameters"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"model"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"parameters"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"for"}]},{"type":"text","value":" step"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" batch "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"in"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","builtin"]},"children":[{"type":"text","value":"enumerate"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"data_loader"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":"\n    input_ids"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" attention_masks"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" labels "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" batch\n\n    input_ids "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" input_ids"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"to"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"device"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n    attention_masks "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" attention_masks"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"to"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"device"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n    labels "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" labels"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"to"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"device"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\n    outputs "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" model"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"input_ids"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"input_ids"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" attention_mask"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"attention_masks"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" labels"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"labels"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n    loss "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" outputs"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"loss\n\n    model"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"backward"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"loss"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n    model"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"step"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"기존 코드에서 큰 변화 없이 몇 가지 사항만 수정해주면 deepspeed 적용이 가능합니다."}]}],"data":{"quirksMode":false}},"excerpt":"DeepSpeed Usage…","fields":{"readingTime":{"text":"4 min read"}},"frontmatter":{"title":"Sooftware NLP - DeepSpeed Usage","userDate":"30 October 2021","date":"2021-10-30T10:00:00.000Z","tags":["nlp","parallelism","large-scale"],"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","backgroundColor":"#081868","images":{"fallback":{"src":"/static/9b0c4df957330fe4209b1e9f8e708409/67a35/deepspeed.png","srcSet":"/static/9b0c4df957330fe4209b1e9f8e708409/67a35/deepspeed.png 224w","sizes":"100vw"},"sources":[{"srcSet":"/static/9b0c4df957330fe4209b1e9f8e708409/f42a0/deepspeed.webp 224w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":1}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"children":[{"gatsbyImageData":{"layout":"fullWidth","backgroundColor":"#282838","images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/0d6f4/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/248f9/soohwan.png 40w,\n/static/a9e6b445142b247ee4cfa66155398bb2/fd435/soohwan.png 80w,\n/static/a9e6b445142b247ee4cfa66155398bb2/0d6f4/soohwan.png 120w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/e7f45/soohwan.webp 40w,\n/static/a9e6b445142b247ee4cfa66155398bb2/589ec/soohwan.webp 80w,\n/static/a9e6b445142b247ee4cfa66155398bb2/71a38/soohwan.webp 120w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6833333333333333}}]}}]}},"relatedPosts":{"totalCount":50,"edges":[{"node":{"id":"6529d72e-80e7-5d61-a672-d66276a3f641","excerpt":"MoE(Mixture of Experts) 기초부터 DeepSeek 혁신까지 작년 이맘때쯤 DeepSeek-V3가 저비용으로 엄청난 성능을 보이면서 화제가 되었습니다. 그 핵심 기술인 MoE(Mixture of Experts…","frontmatter":{"title":"MoE(Mixture of Experts) 기초부터 DeepSeek 혁신까지","date":"2026-01-15T01:11:55.000Z"},"fields":{"readingTime":{"text":"8 min read"},"slug":"/developed-moe/"}}},{"node":{"id":"8b49d3ef-4ce3-568c-9d71-240ff17fc3e0","excerpt":"BERT는 사실 Diffusion 모델이였다?! 최근 굉장히 흥미로운 글을 읽게되어 공유합니다. 원문 : link BERT와 Diffusion이 같은 방식이다?! NLP 연구자들에게 BERT는 너무 익숙한 모델입니다. 201…","frontmatter":{"title":"BERT는 사실 Diffusion 모델이였다?!","date":"2025-10-21T12:00:00.000Z"},"fields":{"readingTime":{"text":"8 min read"},"slug":"/bert_diffusion/"}}},{"node":{"id":"7c7e1676-ea84-58de-970c-ddec9aa10660","excerpt":"RLHF는 수다쟁이를 만든다?! (Does RLHF Breed Verbose Chatterboxes?!) RLHF(Reinforcement Learning from Human Feedback)는 OpenAI의 ChatGPT…","frontmatter":{"title":"RLHF는 수다쟁이를 만든다?! (Does RLHF Breed Verbose Chatterboxes?!)","date":"2024-03-13T01:11:55.000Z"},"fields":{"readingTime":{"text":"7 min read"},"slug":"/rlhf-vervosity/"}}},{"node":{"id":"2b9d3e22-1796-5fab-878d-5941d2e76e9d","excerpt":"LLM Paper Abstract - 2023.12 LLM…","frontmatter":{"title":"LLM Paper Abstract - 2023.12","date":"2024-01-05T10:00:00.000Z"},"fields":{"readingTime":{"text":"5 min read"},"slug":"/llm-abs-202312/"}}},{"node":{"id":"f43fa33b-917b-5c00-8462-937d99592ad7","excerpt":"What it MoE? (Mixture of Experts) 현존 최강 LLM인 GPT-4에서 “MoE (Mixture of Experts)” 방식을 채택하여 사용하고 있다고 알려졌는데요, 최근 AI계의 뜨거운 감자 Mistral AI…","frontmatter":{"title":"What is MoE? (Mixture of Experts)","date":"2023-12-22T01:11:55.000Z"},"fields":{"readingTime":{"text":"9 min read"},"slug":"/moe/"}}}]}},"pageContext":{"slug":"/deepspeed/","prev":{"excerpt":"Wandb (Weights & Bias) Image Log Wandb 라이브러리는 최근에 가장 편리하면서도 파워풀한 logging 라이브러리입니다. NLP에서 많이 쓰이는 PyTorch, PyTorch-Lightning, Huggingface…","frontmatter":{"title":"Sooftware ML - Wandb Image Log","tags":["toolkit","logging"],"date":"2021-10-13T22:00:00.000Z","draft":false,"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAAsTAAALEwEAmpwYAAABuklEQVQoz52Su2sVQRSHp7eUVIKVrYUIdnZ2Vmn8GwwpAio2YuGjkBSJBGKRxAhRLqKFSC4mBDFqTHxdTQiBSxQlUeTi7qI32WQf8/pkdnYXBQvjgbPMnpnzze+cOcIYg3NnFktlVdxaC8619Ges5fccMKX7HKG1/1lqthgZmGT8QoPga+STjUVpw9jcAtfGprkz96YGOvvciRi5McP1S/dpvf5YxIRSulhc7ZvgoDjJYXGKlaX1GpjkOUcvX2Ffbz8nBocw2qBL4KP2Gj2959h/6DTDt2Y9sFL4/u0Hbt98TGPqKVG4VQONNUyvvmRy/gnNdqtoSqWwE4c0Fp8x8fA5K5++eKArQSlFlECQU4I0xvje7WaKaAd2yj131uVYq9natQQxdeeV1gh3myvbJXZTXSrQ/kGMIcslYTch6KZkUhbAai9OMjo/En7GGUpKXLWibvDGd+Zn3vFidpnt7aR+tb+ZV2j/jJVf4WQ6u3i3yYHjZzhy7DzL65tFzN1YJf+rC2088MHqK/pHpzg7dI+NIPTAaj73AqzmKkkNcepH25q9K6uBDialZDNMaX9LiZOcPM8Kdf8D/AXrmDiJkhItEQAAAABJRU5ErkJggg=="},"images":{"fallback":{"src":"/static/fd6ffa741fe53de299a57e6a8852f68d/2add8/wandb_image.png","srcSet":"/static/fd6ffa741fe53de299a57e6a8852f68d/5a15a/wandb_image.png 750w,\n/static/fd6ffa741fe53de299a57e6a8852f68d/20c21/wandb_image.png 1080w,\n/static/fd6ffa741fe53de299a57e6a8852f68d/397a5/wandb_image.png 1366w,\n/static/fd6ffa741fe53de299a57e6a8852f68d/2add8/wandb_image.png 1920w","sizes":"100vw"},"sources":[{"srcSet":"/static/fd6ffa741fe53de299a57e6a8852f68d/d7d73/wandb_image.webp 750w,\n/static/fd6ffa741fe53de299a57e6a8852f68d/525de/wandb_image.webp 1080w,\n/static/fd6ffa741fe53de299a57e6a8852f68d/f312c/wandb_image.webp 1366w,\n/static/fd6ffa741fe53de299a57e6a8852f68d/42023/wandb_image.webp 1920w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.5359375000000001}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAADjElEQVQ4y23Oy48TdQDA8TExEaTttvSBfdHddru7bWc6bWd+8+i0u23n0SdlBRQXZGHDgiiLDwgqXjCIJEaC8cLBmHgwUU9EYzRqQuLJi/4NJB6NF+8evgbOHr7XT76SM5ix7s4QnRHnnm9yZzvBTIswN8JcP5bC67ZoWi7CdtFsn5bloZo+dSNAMYbIIqCmD6mJIVXdR3J6I7r9IQ3T59pZlW+uxzndCbPdC/HZXoq3TikowsfqBOjOmFZ7SrM9p2kfRbXn1O0jKNbsSbJ5BMnu9Gl3+qiiz4OPSvz0QRLHFGwOZH6+neLHT5YRbRerM0J0JrTas/8FZXNKzZgg2es+ZtvDdbs8+iHJl+9lWVP7OG2br26k+PvXJCfmFqoYIpwhTXtMwxpTN0coxhjFmCKbE2rGmKoYPQY9qi2fyzsq//4e5sNXi5TlLqbd5t6lPH/9kuDTG2usqgGi7dO0hzSsIXUzQDaGKGKMLEZUxZCKCJB02+PUpMH9vQwP7ybYnSxyfNrjwumA3dEiD94v8cfnK7xzWsbpujQtH9X0qBs+iuEjC5+a7lPVPSqai9SyelyYFbjkh5hqC3TkKOutLON2nkYpgtuMcT6IszdPslo8SLlcpmUH1A0XRQyQhUtNuFQ1l4o2QFI1h3OTVXbcg+grIfSVBRqlEGopgrYSRi0e4KiIse1mScSeopA/iLADmoZL3Rgg631q+oCq1qeq95FOzhQefpzn7eMpBmqcoBVj04pzwkky0eNMRYIdL8F3twusLe2nslrE6g5pGX1U8/FhD1nvUW1tUJK7SH9+n+DR1zF2vQSvH0lz88wiV45V2J1WuHqsxLXNAnuzNJdfWmK5/BylUgFhb6DZAxR9g7rpIos+Wttja2uE9M9vYb69k+DqfJErWzLOuoVqdKlpDo4jODms4zlFckuHObx4iHwuRi6fYqmYY201x/LqMrnFDNl8gRvvukhNdT9394rcOlNFMxrUhYFmW9imTEsp0mhWKC2nWSokyWSiFA7HSKcXiEb34fayyLVniUQk8rmn2drKIqlqiNcuVrh2ymQS1Nh8sc/GoMEbl7u8eb5D1yqTzkRJJsOkkiGymQVShyLE4yHWKnEq1RjpTIhmI8r2y3mkyloI38uha1n6bpVXrkw5+4LD/S/2uHnrIopaJhZ9hmQqwqHkAXJP8BDhyD46ZoWdMxOGcx/PNfECk/8AbxXdRjRliPoAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/34f77/soohwan.png 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/a94f6/soohwan.png 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png 1148w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/38420/soohwan.webp 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7470d/soohwan.webp 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/b5ef6/soohwan.webp 1148w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6829268292682927}}}}]},"fields":{"readingTime":{"text":"3 min read"},"layout":"","slug":"/wandb_image/"}},"next":{"excerpt":"광운대학교 SW 중심대학사업단에서 SW 전문가 특강 2021.11.04에 모교인 광운대학교 컴퓨터공학과 학생들을 대상으로 SW…","frontmatter":{"title":"광운대학교 SW 중심대학사업단에서 SW 전문가 특강","tags":["record","presentation"],"date":"2021-10-30T11:00:00.000Z","draft":false,"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAYAAAAywQxIAAAACXBIWXMAAAsTAAALEwEAmpwYAAACPklEQVQozyXMW0/aAABA4f6VbW4mRqdIW0CsrNVgECxyKXUoTieoUBwQLCoqoFwEWorZHz6L7uHkvH2CJIqsrfkISjIbsoy4uorsl1BEETUQYm9DIRvRMGNxzEyGXCqFoe6gBxV+rMuIy2ssfVviy6cFFj5/RXgHF78tIvl8hIJBwpsK4WCQTVEk4vMRkyTSoTA5VcVI6hh6ksyWSiIQRpNCbIgy4rqI3y/h84kIka0IIUkisLqCEpTZlCUUWUKVZba/rxBbWSbjFzlWNX7F4pzG4hztRElrUWIRDU2JsLWpsBEKI0sBhEw6Tc4wiG9r6GqEbFInl0ljZg1+6kkKsThXqTT11AG2eYhtmlT1fc4S++R1nUxsFz2qoUe32d+JIFStCrXra64rZY4SCUqGQa1Wp15v0Kg3sBtN7orn3GdTdE9P6JWKPJ6dYBcOqeQOOEvuUthTOY6r5HcVhF63Q6/bpf/So91sUDVyPLYfee526Hc7jAYDhq0b2kfH2Kksz5dXPFsWnfIF9xenVA8PKKYSXJpZrvI5BGc6xXUcZtMJb65Dr93mpdtjOhnjODM87w1v/pfpdMa4P8LpPDFpNZjYf3Bu60yaFV6bFq+tGq9tG2HuzZjPZ7x5//M8l37nidFw9IE47huOO2fqzhne3jGuW4xvavQr5wysEkPrnGH1jFH5mFEph/AOvKPefMbMc/FmLuPBCw+2zfh1ymTiMh67H39otrDLFg+1GvZFiVrepH5kcvO7wO15gU7R5B+UXHtz2Jqw6wAAAABJRU5ErkJggg=="},"images":{"fallback":{"src":"/static/aec57a2174fb337a19378819bf43bfb3/ae673/kw_teuc.png","srcSet":"/static/aec57a2174fb337a19378819bf43bfb3/9c370/kw_teuc.png 750w,\n/static/aec57a2174fb337a19378819bf43bfb3/9ca7e/kw_teuc.png 1080w,\n/static/aec57a2174fb337a19378819bf43bfb3/99ecf/kw_teuc.png 1366w,\n/static/aec57a2174fb337a19378819bf43bfb3/ae673/kw_teuc.png 1920w","sizes":"100vw"},"sources":[{"srcSet":"/static/aec57a2174fb337a19378819bf43bfb3/67368/kw_teuc.webp 750w,\n/static/aec57a2174fb337a19378819bf43bfb3/1f97d/kw_teuc.webp 1080w,\n/static/aec57a2174fb337a19378819bf43bfb3/ebf70/kw_teuc.webp 1366w,\n/static/aec57a2174fb337a19378819bf43bfb3/daa50/kw_teuc.webp 1920w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.4427083333333333}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAADjElEQVQ4y23Oy48TdQDA8TExEaTttvSBfdHddru7bWc6bWd+8+i0u23n0SdlBRQXZGHDgiiLDwgqXjCIJEaC8cLBmHgwUU9EYzRqQuLJi/4NJB6NF+8evgbOHr7XT76SM5ix7s4QnRHnnm9yZzvBTIswN8JcP5bC67ZoWi7CdtFsn5bloZo+dSNAMYbIIqCmD6mJIVXdR3J6I7r9IQ3T59pZlW+uxzndCbPdC/HZXoq3TikowsfqBOjOmFZ7SrM9p2kfRbXn1O0jKNbsSbJ5BMnu9Gl3+qiiz4OPSvz0QRLHFGwOZH6+neLHT5YRbRerM0J0JrTas/8FZXNKzZgg2es+ZtvDdbs8+iHJl+9lWVP7OG2br26k+PvXJCfmFqoYIpwhTXtMwxpTN0coxhjFmCKbE2rGmKoYPQY9qi2fyzsq//4e5sNXi5TlLqbd5t6lPH/9kuDTG2usqgGi7dO0hzSsIXUzQDaGKGKMLEZUxZCKCJB02+PUpMH9vQwP7ybYnSxyfNrjwumA3dEiD94v8cfnK7xzWsbpujQtH9X0qBs+iuEjC5+a7lPVPSqai9SyelyYFbjkh5hqC3TkKOutLON2nkYpgtuMcT6IszdPslo8SLlcpmUH1A0XRQyQhUtNuFQ1l4o2QFI1h3OTVXbcg+grIfSVBRqlEGopgrYSRi0e4KiIse1mScSeopA/iLADmoZL3Rgg631q+oCq1qeq95FOzhQefpzn7eMpBmqcoBVj04pzwkky0eNMRYIdL8F3twusLe2nslrE6g5pGX1U8/FhD1nvUW1tUJK7SH9+n+DR1zF2vQSvH0lz88wiV45V2J1WuHqsxLXNAnuzNJdfWmK5/BylUgFhb6DZAxR9g7rpIos+Wttja2uE9M9vYb69k+DqfJErWzLOuoVqdKlpDo4jODms4zlFckuHObx4iHwuRi6fYqmYY201x/LqMrnFDNl8gRvvukhNdT9394rcOlNFMxrUhYFmW9imTEsp0mhWKC2nWSokyWSiFA7HSKcXiEb34fayyLVniUQk8rmn2drKIqlqiNcuVrh2ymQS1Nh8sc/GoMEbl7u8eb5D1yqTzkRJJsOkkiGymQVShyLE4yHWKnEq1RjpTIhmI8r2y3mkyloI38uha1n6bpVXrkw5+4LD/S/2uHnrIopaJhZ9hmQqwqHkAXJP8BDhyD46ZoWdMxOGcx/PNfECk/8AbxXdRjRliPoAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/34f77/soohwan.png 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/a94f6/soohwan.png 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png 1148w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/38420/soohwan.webp 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7470d/soohwan.webp 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/b5ef6/soohwan.webp 1148w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6829268292682927}}}}]},"fields":{"readingTime":{"text":"1 min read"},"layout":"","slug":"/ssdc_2021_2/"}},"primaryTag":"nlp"}},
    "staticQueryHashes": ["3170763342","3229353822"]}