{ "componentChunkName": "component---src-templates-post-tsx", "path": "/tokenizer/", "result": {"data":{"logo":null,"markdownRemark":{"html":"

Tokenization

\n

문장에서 의미있는 단위로 나누는 작업을 토큰화라고 한다.

\n\n
어제 카페 갔었어 > 어, 제, 카, 페, 갔, 었, 어\n어제 카페 갔었는데요 > 어, 제, 카, 페, 갔, 었, 는, 데, 요
\n
\n\n
어제 카페 갔었어 > 어제, 카페, 갔었어\n어제 카페 갔었는데요 > 어제, 카페, 갔었는데요
\n
\n\n

서브워드 기반 토크나이저

\n

BPE(Byte-Pair Encoding)

\n\n
aaabdaaabac    # 가장 자주 등장하고 있는 바이트의 쌍(byte pair)은 'aa' , Z=aa\nZabdZabac      # Y=ab\nZYdZYac        # X=ZY\nXdXac          # 더 이상 병합할 바이트의 쌍이 없으므로 최종 결과로 하여 종료
\n
\n

자연어 처리에서의 BPE

\n\n
\n\n

\"image\"

\n\n

\"image\"

\n
\n

WordPiece

\n\n

\"image\"

\n

또 이 학생의 집에서 병든 소를 도축했던 35살 남성도 탄저병에 걸린 것으로 확인됐습니다.

\n
\n

['또', '이', '학생', '##의', '집', '##에', '##서', '병든', '소', '##를', '도축', '##했', '##던', '35', '##살', '남성', '##도', '탄', '##저', '##병', '##에', '걸린', '것', '##으로', '확인', '##됐', '##습', '##니다', '.']

\n
\n
\n

Unigram

\n\n
\n

SentencePiece

\n\n

또 이 학생의 집에서 병든 소를 도축했던 35살 남성도 탄저병에 걸린 것으로 확인됐습니다.

\n
\n

['▁또', '▁이', '▁학생', '의', '▁집에서', '▁병', '든', '▁소', '를', '▁', '도', '축', '했던', '▁35', '살', '▁남성', '도', '▁탄', '저', '병', '에', '▁걸린', '▁것으로', '▁확인', '됐', '습니다', '.']

\n
\n
\n

Reference

\n","htmlAst":{"type":"root","children":[{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Tokenization"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"문장에서 의미있는 단위로 나누는 작업을 "},{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"토큰화"}]},{"type":"text","value":"라고 한다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"문자 단위 토큰화"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"문자 단위로 토큰화를 하는 것이다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"한글 음절 수는 모두 11,172개이므로 알파벳, 숫자, 기호 등을 고려한다고 해도 단어 사전의 크기는 기껏해야 15,000개를 넘기 어렵다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"장점"}]},{"type":"text","value":" : 모든 문자를 포함시켜서 "},{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"UNK"}]},{"type":"text","value":"토큰이 잘 발생하지 않는다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"단점"}]},{"type":"text","value":" : 의미 있는 단위가 되기 어렵고, 상대적으로 시퀀스가 길어진다."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"text"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-text"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"어제 카페 갔었어 > 어, 제, 카, 페, 갔, 었, 어\n어제 카페 갔었는데요 > 어, 제, 카, 페, 갔, 었, 는, 데, 요"}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"단어 단위 토큰화"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"단어 단위로 토큰화를 하는 것이다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"장점"}]},{"type":"text","value":" : 공백으로 쉽게 분리할 수 있다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"단점"}]},{"type":"text","value":" : 모든 단어들을 다 포함시키기에는 단어 사전의 크기가 상당히 크다. 이는 메모리 문제를 야기!"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"text"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-text"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"어제 카페 갔었어 > 어제, 카페, 갔었어\n어제 카페 갔었는데요 > 어제, 카페, 갔었는데요"}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"strong","properties":{},"children":[{"type":"text","value":"서브워드 단위 토큰화"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"문자 단위와 단어 단위 토큰화의 중간에 있는 형태이다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"“자주 등장한 단어는 그대로 두고, 자주 등장하지 않은 단어는 의미있는 서브 워드 토큰들로 분절한다” 라는 원칙에 기반을 둔 알고리즘"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"단어 사전의 크기를 지나치게 늘리지 않으면서도 "},{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"UNK"}]},{"type":"text","value":" 문제를 해결할 수 있다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"희귀 단어, 신조어와 같은 문제를 완화시킬 수 있다."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"서브워드 기반 토크나이저"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"BPE(Byte-Pair Encoding)"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"BPE(Byte pair encoding) 알고리즘은 1994년에 제안된 데이터 압축 알고리즘"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"연속적으로 가장 많이 등장한 글자의 쌍을 찾아서 하나의 글자로 병합하는 방식을 수행"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"text"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-text"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"aaabdaaabac # 가장 자주 등장하고 있는 바이트의 쌍(byte pair)은 'aa' , Z=aa\nZabdZabac # Y=ab\nZYdZYac # X=ZY\nXdXac # 더 이상 병합할 바이트의 쌍이 없으므로 최종 결과로 하여 종료"}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"element","tagName":"a","properties":{"href":"https://arxiv.org/abs/1508.07909"},"children":[{"type":"text","value":"자연어 처리에서의 BPE"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"글자(charcter) 단위에서 점차적으로 단어 집합(vocabulary)을 만들어 내는 Bottom up 방식의 접근을 사용"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"BPE는 일반적으로 훈련 데이터를 단어 단위로 분절하는 Pre-tokenize 과정을 거쳐야한다."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"Pre-tokenize는 공백 단위나 규칙 기반으로 수행될 수 있다."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"Example)"},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"('hug', 10), ('pug', 5), ('pun', 12), ('bun', 4), ('hugs', 5)"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"Pre-tokenize를 거쳐서 나온 단어들이라 하고 여기서 정수 값은 각 단어가 얼마나 등장했는지를 나타내는 값이다."},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n이때 기본 사전은 [‘b’, ‘g’, ‘h’, ‘n’, ‘p’, ‘s’, ‘u’] 이다."},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n기본 사전을 기반으로 단어들을 쪼개면 다음과 같다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"('h' 'u' 'g', 10), ('p' 'u' 'g', 5), ('p' 'u' 'n', 12), ('b' 'u' 'n', 4), ('h' 'u' 'g' 's', 5)"}]},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n“hu”는 총 15번, “ug”는 총 20번이 나와 가장 많이 등장한 쌍은 “ug”가 되고 “u”와 “g”를 합친 “ug”를 사전에 새로 추가한다."},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n그럼 이때 기본 사전은 [‘b’, ‘g’, ‘h’, ‘n’, ‘p’, ‘s’, ‘u’, ‘ug’] 이다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"('h' 'ug', 10), ('p' 'ug', 5), ('p' 'u' 'n', 12), ('b' 'u' 'n', 4), ('h' 'ug' 's', 5)"}]},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n또 가장 많이 나온 쌍은 16번 등장한 “un”이므로, “un”을 사전에 추가한다. 그 다음은 15번 등장한 “hug”이므로 “hug”도 사전에 추가한다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"('hug', 10), ('p' 'ug', 5), ('p' 'un', 12), ('b' 'un', 4), ('hug' 's', 5)"}]},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n기본 사전은 [‘b’, ‘g’, ‘h’, ‘n’, ‘p’, ‘s’, ‘u’, ‘ug’, ‘un’, ‘hug’]가 됩니다.\n이렇게 처음에는 글자 단위였던 것이 의미있는 서브워드 토큰들로 분절할 수 있다."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"strong","properties":{},"children":[{"type":"element","tagName":"a","properties":{"href":"https://arxiv.org/pdf/1909.03341.pdf"},"children":[{"type":"text","value":"Byte-level BPE(BBPE)"}]}]}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/54731898/133186594-e4f0a5d8-65a2-4ba5-b6a2-09be7bdc6757.png","alt":"image"},"children":[]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"a","properties":{"href":"https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf"},"children":[{"type":"text","value":"GPT-2 논문"}]},{"type":"text","value":"에서 바이트를 사전의 기본 단위로 사용하는 트릭을 사용"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"GPT-2 모델은 256개의 기본 바이트 토큰과 "},{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":""}]},{"type":"text","value":" 토큰 그리고 50,000 개의 서브 워드를 더해 총 50,257 개의 단어 집합(vocabulary)을 가진다."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"256 바이트셋으로 모든 텍스트를 표현할 수 있다."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"UNK"}]},{"type":"text","value":" 문제없이 모든 텍스트를 분절할 수 있다."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"multilingual일 때, 언어들 사이에서 vocabulary 공유를 가장 많이 한다."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/54731898/133136459-f1b4fdbf-d9d4-4976-842e-c00cbf657624.png","alt":"image"},"children":[]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"WordPiece"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"a","properties":{"href":"https://arxiv.org/abs/1810.04805"},"children":[{"type":"text","value":"BERT"}]},{"type":"text","value":"에서 활용된 서브 워드 토크나이저 알고리즘"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"BPE와 마찬가지로 사전을 코퍼스 내 등장한 캐릭터들로 초기화 한 후, 사용자가 지정한 횟수 만큼 서브 워드를 병합하는 방식으로 훈련"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"하지만 WordPiece는 BPE와 같이 가장 많이 등장한 쌍을 병합하는 것이 아니라, 병합되었을 때 코퍼스의 Likelihood를 가장 높이는 쌍을 병합하게 된다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"즉, WordPiece에서는 코퍼스 내에서 “ug”가 등장할 확률을 “u”와 “g”가 각각 등장할 확률을 곱한 값으로 나눈 값이 다른 쌍보다 클 경우 해당 쌍을 병합하게 된다."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"img","properties":{"src":"https://user-images.githubusercontent.com/54731898/133140262-f0afdedc-e54e-4564-a88d-134163c0a219.png","alt":"image"},"children":[]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"또 이 학생의 집에서 병든 소를 도축했던 35살 남성도 탄저병에 걸린 것으로 확인됐습니다."}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"blockquote","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"['또', '이', '학생', '##의', '집', '##에', '##서', '병든', '소', '##를', '도축', '##했', '##던', '35', '##살', '남성', '##도', '탄', '##저', '##병', '##에', '걸린', '것', '##으로', '확인', '##됐', '##습', '##니다', '.']"}]}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Unigram"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"서브 워드에서 시작해 점차 사전을 줄여나가는 top-down 방식으로 진행"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"매 스텝마다 Unigram은 주어진 코퍼스와 현재 사전에 대한 Loss를 측정한다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"또한 각각의 서브 워드에 대해 해당 서브 워드가 코퍼스에서 제거되었을 때, Loss가 얼마나 증가하는지를 측정하여 Loss를 가장 조금 증가시키는 p 개 토큰을 제거한다. (p는 보통 전체 사전 크기의 10-20% 값으로 설정)"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"해당 과정을 사용자가 원하는 사전 크기를 지니게 될 때 까지 반복하게 되고, 기본 캐릭터들은 반드시 사전에서 제거되지 않고 유지되어야한다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"매번 같은 토큰 리스트를 반환하는 BPE, WordPiece와 달리 Unigram은 다양한 토큰 리스트가 생길 수 있다."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"SentencePiece"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"지금까지 살펴본 모든 방법들은 공백을 기준으로 단어를 분절할 수 없기 때문에 Pre-tokenize 과정을 필요로 했다."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"하지만 sentencepiece는 공백을 기준으로 단어를 분절할 수 있기 때문에 Pre-tokenize 과정이 필요없다."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"또한 디코딩 과정에서 모든 토큰들을 붙여준 후, 메타스페이스(”▁”)만 공백으로 바꿔주면 되기 때문에 원상복구가 가능하다는 특징이 있다."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"BPE 혹은 Unigram을 적용하여 사전을 구축하게 된다."}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"또 이 학생의 집에서 병든 소를 도축했던 35살 남성도 탄저병에 걸린 것으로 확인됐습니다."}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"blockquote","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"['▁또', '▁이', '▁학생', '의', '▁집에서', '▁병', '든', '▁소', '를', '▁', '도', '축', '했던', '▁35', '살', '▁남성', '도', '▁탄', '저', '병', '에', '▁걸린', '▁것으로', '▁확인', '됐', '습니다', '.']"}]}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n"},{"type":"element","tagName":"h2","properties":{},"children":[{"type":"text","value":"Reference"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"a","properties":{"href":"https://wikidocs.net/22592"},"children":[{"type":"text","value":"https://wikidocs.net/22592"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"a","properties":{"href":"https://huggingface.co/transformers/master/tokenizer_summary.html"},"children":[{"type":"text","value":"https://huggingface.co/transformers/master/tokenizer_summary.html"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"a","properties":{"href":"https://karter.io/huggingface"},"children":[{"type":"text","value":"https://karter.io/huggingface"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"a","properties":{"href":"https://ratsgo.github.io/nlpbook/docs/preprocess/bpe/"},"children":[{"type":"text","value":"https://ratsgo.github.io/nlpbook/docs/preprocess/bpe/"}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"element","tagName":"a","properties":{"href":"https://arxiv.org/pdf/1508.07909.pdf"},"children":[{"type":"text","value":"https://arxiv.org/pdf/1508.07909.pdf"}]}]},{"type":"text","value":"\n"}]}],"data":{"quirksMode":false}},"excerpt":"Tokenization 문장에서 의미있는 단위로 나누는 작업을 라고 한다. 문자 단위 토큰화 문자 단위로 토큰화를 하는 것이다. 한글 음절 수는 모두 11,172개이므로 알파벳, 숫자, 기호 등을 고려한다고 해도 단어 사전의 크기는 기껏해야 1…","fields":{"readingTime":{"text":"10 min read"}},"frontmatter":{"title":"Sooftware NLP - Tokenizer","userDate":"13 September 2021","date":"2021-09-13T23:46:37.121Z","tags":["nlp"],"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","backgroundColor":"#c8b8b8","images":{"fallback":{"src":"/static/8e92507f0141c508e4d5a30d3ca6fe53/4d1d2/writing.jpg","srcSet":"/static/8e92507f0141c508e4d5a30d3ca6fe53/6cce3/writing.jpg 750w,\n/static/8e92507f0141c508e4d5a30d3ca6fe53/fb319/writing.jpg 1080w,\n/static/8e92507f0141c508e4d5a30d3ca6fe53/6a5de/writing.jpg 1366w,\n/static/8e92507f0141c508e4d5a30d3ca6fe53/4d1d2/writing.jpg 1400w","sizes":"100vw"},"sources":[{"srcSet":"/static/8e92507f0141c508e4d5a30d3ca6fe53/d2a19/writing.webp 750w,\n/static/8e92507f0141c508e4d5a30d3ca6fe53/72f43/writing.webp 1080w,\n/static/8e92507f0141c508e4d5a30d3ca6fe53/8992a/writing.webp 1366w,\n/static/8e92507f0141c508e4d5a30d3ca6fe53/d652b/writing.webp 1400w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6678571428571428}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"children":[{"gatsbyImageData":{"layout":"fullWidth","backgroundColor":"#282838","images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/0d6f4/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/248f9/soohwan.png 40w,\n/static/a9e6b445142b247ee4cfa66155398bb2/fd435/soohwan.png 80w,\n/static/a9e6b445142b247ee4cfa66155398bb2/0d6f4/soohwan.png 120w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/e7f45/soohwan.webp 40w,\n/static/a9e6b445142b247ee4cfa66155398bb2/589ec/soohwan.webp 80w,\n/static/a9e6b445142b247ee4cfa66155398bb2/71a38/soohwan.webp 120w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6833333333333333}}]}}]}},"relatedPosts":{"totalCount":50,"edges":[{"node":{"id":"6529d72e-80e7-5d61-a672-d66276a3f641","excerpt":"MoE(Mixture of Experts) 기초부터 DeepSeek 혁신까지 작년 이맘때쯤 DeepSeek-V3가 저비용으로 엄청난 성능을 보이면서 화제가 되었습니다. 그 핵심 기술인 MoE(Mixture of Experts…","frontmatter":{"title":"MoE(Mixture of Experts) 기초부터 DeepSeek 혁신까지","date":"2026-01-15T01:11:55.000Z"},"fields":{"readingTime":{"text":"8 min read"},"slug":"/developed-moe/"}}},{"node":{"id":"8b49d3ef-4ce3-568c-9d71-240ff17fc3e0","excerpt":"BERT는 사실 Diffusion 모델이였다?! 최근 굉장히 흥미로운 글을 읽게되어 공유합니다. 원문 : link BERT와 Diffusion이 같은 방식이다?! NLP 연구자들에게 BERT는 너무 익숙한 모델입니다. 201…","frontmatter":{"title":"BERT는 사실 Diffusion 모델이였다?!","date":"2025-10-21T12:00:00.000Z"},"fields":{"readingTime":{"text":"8 min read"},"slug":"/bert_diffusion/"}}},{"node":{"id":"7c7e1676-ea84-58de-970c-ddec9aa10660","excerpt":"RLHF는 수다쟁이를 만든다?! (Does RLHF Breed Verbose Chatterboxes?!) RLHF(Reinforcement Learning from Human Feedback)는 OpenAI의 ChatGPT…","frontmatter":{"title":"RLHF는 수다쟁이를 만든다?! (Does RLHF Breed Verbose Chatterboxes?!)","date":"2024-03-13T01:11:55.000Z"},"fields":{"readingTime":{"text":"7 min read"},"slug":"/rlhf-vervosity/"}}},{"node":{"id":"2b9d3e22-1796-5fab-878d-5941d2e76e9d","excerpt":"LLM Paper Abstract - 2023.12 LLM…","frontmatter":{"title":"LLM Paper Abstract - 2023.12","date":"2024-01-05T10:00:00.000Z"},"fields":{"readingTime":{"text":"5 min read"},"slug":"/llm-abs-202312/"}}},{"node":{"id":"f43fa33b-917b-5c00-8462-937d99592ad7","excerpt":"What it MoE? (Mixture of Experts) 현존 최강 LLM인 GPT-4에서 “MoE (Mixture of Experts)” 방식을 채택하여 사용하고 있다고 알려졌는데요, 최근 AI계의 뜨거운 감자 Mistral AI…","frontmatter":{"title":"What is MoE? (Mixture of Experts)","date":"2023-12-22T01:11:55.000Z"},"fields":{"readingTime":{"text":"9 min read"},"slug":"/moe/"}}}]}},"pageContext":{"slug":"/tokenizer/","prev":{"excerpt":"정규 표현식 정규표현식(regular expression)은 일종의 문자를 표현하는 공식으로, 특정 규칙이 있는 문자열 집합을 추출할 때 자주 사용되는 기법입니다. 주로 Prograaming Language나 Text Editor…","frontmatter":{"title":"정규표현식 (regex)","tags":["nlp"],"date":"2021-09-08T10:00:00.000Z","draft":false,"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAYAAAAywQxIAAAACXBIWXMAAAsTAAALEwEAmpwYAAABnElEQVQoz32R3W6bMABG8/6vs91s0rSrrW2StonKVgI0CQnBhhDAJSFNsH0qYG3VqpqlT/6RdWSfb2CNwQiJjgV6E9PurbVYYzHG9uuXwPtYi9EanReYdIt5fGRAO+oarAEM/x2nI/geBB6U+dt5pTBFga3rHjh/SBmNlty7Am8mWMxThldzvn6Z4s0kgZ/ge5LJeM70xy3OzwnjC5+bm5DRcMHlRfDKHliteTockUFEvk7Y7xSHsiIVJYG7IV9JVJKjZEYaZRTViaw8IqRCxjlpnLPdVp2EVsHA7Pfo4RCuLrG/f2EdB3YZ6PObgkT+U2JBCkgERGsoC9BNd6WF0QK7UqqKRsguerfDnM/o0xl9fKIpFafxNefFkkYm6Fa+MehD3RVjtOnKe4F2Du2rAfuhBQvbtC9itexnEffnTYMpFWabodcRJs97YPdUQKma9TIl9AUiLgnDjL/TkI1QRFFBGEiEKMmKGhmm7Is9xvNoZjO062KUeg+UosRxoi4zVzCdrPj+7Q/O3brL7fUC927Fgy/x7jdkqfr0T883/K2xzSR3ygAAAABJRU5ErkJggg=="},"images":{"fallback":{"src":"/static/ebc4f36cae54a4a8d9eea6079daca5bc/acc2d/regex.png","srcSet":"/static/ebc4f36cae54a4a8d9eea6079daca5bc/acc2d/regex.png 699w","sizes":"100vw"},"sources":[{"srcSet":"/static/ebc4f36cae54a4a8d9eea6079daca5bc/8a3ab/regex.webp 699w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.45064377682403434}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAADjElEQVQ4y23Oy48TdQDA8TExEaTttvSBfdHddru7bWc6bWd+8+i0u23n0SdlBRQXZGHDgiiLDwgqXjCIJEaC8cLBmHgwUU9EYzRqQuLJi/4NJB6NF+8evgbOHr7XT76SM5ix7s4QnRHnnm9yZzvBTIswN8JcP5bC67ZoWi7CdtFsn5bloZo+dSNAMYbIIqCmD6mJIVXdR3J6I7r9IQ3T59pZlW+uxzndCbPdC/HZXoq3TikowsfqBOjOmFZ7SrM9p2kfRbXn1O0jKNbsSbJ5BMnu9Gl3+qiiz4OPSvz0QRLHFGwOZH6+neLHT5YRbRerM0J0JrTas/8FZXNKzZgg2es+ZtvDdbs8+iHJl+9lWVP7OG2br26k+PvXJCfmFqoYIpwhTXtMwxpTN0coxhjFmCKbE2rGmKoYPQY9qi2fyzsq//4e5sNXi5TlLqbd5t6lPH/9kuDTG2usqgGi7dO0hzSsIXUzQDaGKGKMLEZUxZCKCJB02+PUpMH9vQwP7ybYnSxyfNrjwumA3dEiD94v8cfnK7xzWsbpujQtH9X0qBs+iuEjC5+a7lPVPSqai9SyelyYFbjkh5hqC3TkKOutLON2nkYpgtuMcT6IszdPslo8SLlcpmUH1A0XRQyQhUtNuFQ1l4o2QFI1h3OTVXbcg+grIfSVBRqlEGopgrYSRi0e4KiIse1mScSeopA/iLADmoZL3Rgg631q+oCq1qeq95FOzhQefpzn7eMpBmqcoBVj04pzwkky0eNMRYIdL8F3twusLe2nslrE6g5pGX1U8/FhD1nvUW1tUJK7SH9+n+DR1zF2vQSvH0lz88wiV45V2J1WuHqsxLXNAnuzNJdfWmK5/BylUgFhb6DZAxR9g7rpIos+Wttja2uE9M9vYb69k+DqfJErWzLOuoVqdKlpDo4jODms4zlFckuHObx4iHwuRi6fYqmYY201x/LqMrnFDNl8gRvvukhNdT9394rcOlNFMxrUhYFmW9imTEsp0mhWKC2nWSokyWSiFA7HSKcXiEb34fayyLVniUQk8rmn2drKIqlqiNcuVrh2ymQS1Nh8sc/GoMEbl7u8eb5D1yqTzkRJJsOkkiGymQVShyLE4yHWKnEq1RjpTIhmI8r2y3mkyloI38uha1n6bpVXrkw5+4LD/S/2uHnrIopaJhZ9hmQqwqHkAXJP8BDhyD46ZoWdMxOGcx/PNfECk/8AbxXdRjRliPoAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/34f77/soohwan.png 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/a94f6/soohwan.png 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png 1148w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/38420/soohwan.webp 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7470d/soohwan.webp 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/b5ef6/soohwan.webp 1148w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6829268292682927}}}}]},"fields":{"readingTime":{"text":"30 min read"},"layout":"","slug":"/regex/"}},"next":{"excerpt":"PyTorch Lightning 대표적인 딥러닝 프레임워크로 , 가 있습니다. 최근에는 보다 를 선호하는 유저가 많아지는 것 같습니다.\nPyTorch Lightning 은 PyTorch에 대한 High-level…","frontmatter":{"title":"Sooftware ML - PyTorch Lightning","tags":["toolkit"],"date":"2021-09-17T10:00:00.000Z","draft":false,"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAHCAYAAAAIy204AAAACXBIWXMAAAsTAAALEwEAmpwYAAABd0lEQVQoz41ROUsDQRjdn6EGryQqHuBVCfoXrKw8CrGUFIrGZNVEo4VpRPACjQeCighCTCEqJAgidhbaiY07s9Fskk08wPvJ920Q7Rx4vPnezLyZ+Z4y7JTwlgoM5Gl/mPR/wZGDU0J1SCiqQ2C0QiIybuL++g1hn8k1mXpKBLNqF/CWCHiKLKaaNVrLGVJNrAwWaDhdfQANcfnKfLKU5c3B5jjGayV8VRKBOh3BljgC9Tp8lRIj5RITDfQqy5g0MmXDq8NnfLx9Ya3bwOfnFy7CT3xwpy+N5fYE9sdMHAQziC1mEJ3PYsuVRNhvYqc/iV13CntqGpEJE/5q3TKMzmX5Zanbd+bjmQy8doHN3iRCnfc4ms4gtpDFbOsdVroMviTUYWC9x8BiWwKRgImzjUeM1ejUQ6tf264Ubs5fmL25HnmKBdw2gclGnUHzoUJLJyaQRt+daorzOYUb6rDSdds0DORrXP+kWCY5CALNGb9TLpM/l1Mo30y9gV9dy4rfAAAAAElFTkSuQmCC"},"images":{"fallback":{"src":"/static/458c786f3433bdba9c9acd81597dd025/ce0ab/pl.png","srcSet":"/static/458c786f3433bdba9c9acd81597dd025/d4b25/pl.png 750w,\n/static/458c786f3433bdba9c9acd81597dd025/ce0ab/pl.png 842w","sizes":"100vw"},"sources":[{"srcSet":"/static/458c786f3433bdba9c9acd81597dd025/f1ea4/pl.webp 750w,\n/static/458c786f3433bdba9c9acd81597dd025/33523/pl.webp 842w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.3586698337292162}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAADjElEQVQ4y23Oy48TdQDA8TExEaTttvSBfdHddru7bWc6bWd+8+i0u23n0SdlBRQXZGHDgiiLDwgqXjCIJEaC8cLBmHgwUU9EYzRqQuLJi/4NJB6NF+8evgbOHr7XT76SM5ix7s4QnRHnnm9yZzvBTIswN8JcP5bC67ZoWi7CdtFsn5bloZo+dSNAMYbIIqCmD6mJIVXdR3J6I7r9IQ3T59pZlW+uxzndCbPdC/HZXoq3TikowsfqBOjOmFZ7SrM9p2kfRbXn1O0jKNbsSbJ5BMnu9Gl3+qiiz4OPSvz0QRLHFGwOZH6+neLHT5YRbRerM0J0JrTas/8FZXNKzZgg2es+ZtvDdbs8+iHJl+9lWVP7OG2br26k+PvXJCfmFqoYIpwhTXtMwxpTN0coxhjFmCKbE2rGmKoYPQY9qi2fyzsq//4e5sNXi5TlLqbd5t6lPH/9kuDTG2usqgGi7dO0hzSsIXUzQDaGKGKMLEZUxZCKCJB02+PUpMH9vQwP7ybYnSxyfNrjwumA3dEiD94v8cfnK7xzWsbpujQtH9X0qBs+iuEjC5+a7lPVPSqai9SyelyYFbjkh5hqC3TkKOutLON2nkYpgtuMcT6IszdPslo8SLlcpmUH1A0XRQyQhUtNuFQ1l4o2QFI1h3OTVXbcg+grIfSVBRqlEGopgrYSRi0e4KiIse1mScSeopA/iLADmoZL3Rgg631q+oCq1qeq95FOzhQefpzn7eMpBmqcoBVj04pzwkky0eNMRYIdL8F3twusLe2nslrE6g5pGX1U8/FhD1nvUW1tUJK7SH9+n+DR1zF2vQSvH0lz88wiV45V2J1WuHqsxLXNAnuzNJdfWmK5/BylUgFhb6DZAxR9g7rpIos+Wttja2uE9M9vYb69k+DqfJErWzLOuoVqdKlpDo4jODms4zlFckuHObx4iHwuRi6fYqmYY201x/LqMrnFDNl8gRvvukhNdT9394rcOlNFMxrUhYFmW9imTEsp0mhWKC2nWSokyWSiFA7HSKcXiEb34fayyLVniUQk8rmn2drKIqlqiNcuVrh2ymQS1Nh8sc/GoMEbl7u8eb5D1yqTzkRJJsOkkiGymQVShyLE4yHWKnEq1RjpTIhmI8r2y3mkyloI38uha1n6bpVXrkw5+4LD/S/2uHnrIopaJhZ9hmQqwqHkAXJP8BDhyD46ZoWdMxOGcx/PNfECk/8AbxXdRjRliPoAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/34f77/soohwan.png 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/a94f6/soohwan.png 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png 1148w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/38420/soohwan.webp 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7470d/soohwan.webp 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/b5ef6/soohwan.webp 1148w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6829268292682927}}}}]},"fields":{"readingTime":{"text":"4 min read"},"layout":"","slug":"/pytorch_lightning/"}},"primaryTag":"nlp"}}, "staticQueryHashes": ["3170763342","3229353822"]}