{ "componentChunkName": "component---src-templates-post-tsx", "path": "/tokenizers/", "result": {"data":{"logo":null,"markdownRemark":{"html":"

최근 NLP 토크나이저를 만드는데 가장 많이 사용되는 tokenizers 라이브러와 실제 사용이 가장 많이 되는 transformers 라이브러리로의 변환에 대한 코드를 담고 있습니다. 해당 내용은 tokenizers==0.10.3 버젼에서 수행되었습니다.

\n

Train

\n

아래 코드는 wordpiece, char-bpe, byte-level bpe 방식을 지원합니다.
\n센텐스피스의 경우 tokenizers에는 현재 sentencepiece가 불안정하여, 오리지널 sentencepiece 학습 방식으로 학습 후 tokenizers.SentencePieceUnigramTokenizer.from_spm() 메서드로 로드하는 방식을 사용하셔야 합니다.

\n\n
import os\nimport argparse\nfrom tokenizers import (\n    BertWordPieceTokenizer, \n    ByteLevelBPETokenizer, \n    CharBPETokenizer, \n    SentencePieceUnigramTokenizer,\n)\n\nparser = argparse.ArgumentParser()\nparser.add_argument('--corpus_file', type=str, required=True)\nparser.add_argument('--vocab_size', type=int, default=0, required=True)\nparser.add_argument('--limit_alphabet', type=int, default=0)\nparser.add_argument('--tokenizer', type=str, default='sentencepiece')\nparser.add_argument('--model_type', type=str, default='gpt')\nparser.add_argument('--save_dir', type=str, default='sp')\nargs = parser.parse_args()\n\nif args.model_type == 'bert':\n    special_tokens = ['<|sos|>', '<|eos|>', '<|pad|>', '<|unk|>', '<|mask|>', '<|sep|>', '<|cls|>']\nelif args.model_type == 'gpt':\n    special_tokens = ['<|endoftext|>', '<|unk|>']\nelse:\n    raise ValueError(f\"Unsupported model type: {args.model_type}\")\n\nif args.tokenizer.lower() == 'wordpiece':\n    tokenizer = BertWordPieceTokenizer(\n        clean_text=True,\n        handle_chinese_chars=True,\n        strip_accents=False,\n        lowercase=False,\n        wordpieces_prefix='##',\n    )\n    tokenizer.train(files=[args.corpus_file], limit_alphabet=args.limit_alphabet, vocab_size=args.vocab_size)\nelif args.tokenizer.lower() == 'bbpe':\n    tokenizer = ByteLevelBPETokenizer(\n        lowercase=False,\n        add_prefix_space=True,\n        unicode_normalizer='nfc',\n    )\n    tokenizer.train(\n        files=[args.corpus_file],\n        vocab_size=args.vocab_size,\n        special_tokens=special_tokens,\n    )\nelif args.tokenizer.lower() == 'char-bpe':\n    tokenizer = CharBPETokenizer(\n        lowercase=False,\n        add_prefix_space=True,\n        unicode_normalizer='nfc',\n    )\n    tokenizer.train(\n        files=[args.corpus_file],\n        vocab_size=args.vocab_size,\n        special_tokens=special_tokens,\n    )\nelse:\n    raise ValueError(\"Error\")\n\nif not os.path.exists(args.save_dir):\n    os.mkdir(args.save_dir)\n\ntokenizer.save(\"tokenizer.json\")
\n

Convert to transformers tokenizer

\n

tokenizers 라이브러리로 학습한 토크나이저는 아래 코드로 간단하게 transformers 토크나이저로 변환이 가능합니다.

\n
from transformers import PreTrainedTokenizerFast\n\ntokenizer = PreTrainedTokenizerFast(tokenizer_file='tokenizer.json')\ntokenizer.save_pretrained('SAVE_DIR')
\n\n
import sentencepiece as spm\n\nSENTENCEPIECE_MODEL_PREFIX = \"SP\"\nSENTENCEPIECE_MODEL_TYPE = \"unigram\"\n\nspm.SentencePieceTrainer.Train(\n        f\"--input=sentencepiece_input.txt \"\n        f\"--model_prefix={SENTENCEPIECE_MODEL_PREFIX} \"\n        f\"--vocab_size={vocab_size} \"\n        f\"--model_type={SENTENCEPIECE_MODEL_TYPE} \"\n        f\"--pad_id=0 \"\n        f\"--bos_id=1 \"\n        f\"--eos_id=2 \"\n        f\"--unk_id=3 \"\n        f\"--user_defined_symbols=[SEP],[CLS],[MASK]\")
\n

Add post process

\n

토크나이저를 만들게 되면 따로 설정해주지 않으면 아래와 같은 형태로 토크나이징이 진행됩니다.

\n\n
from transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained('SAVE_DIR')\ntokenizer.tokenize('튜닙은 자연어처리 테크 스타트업이다.')
\n\n
['▁', '튜', '닙', '은', '▁자연', '어', '처', '리', '▁테', '크', '▁', '스타', '트', '업', '이다.']
\n

보통 버트나 일렉트라 같은 모델들은 파인튜닝의 편리함을 위해 아래와 같은 포맷으로 문장이 토크나이징 됩니다.

\n
['[CLS]', '▁', '튜', '닙', '은', '▁자연', '어', '처', '리', '▁테', '크', '▁', '스타', '트', '업', '이다.', '[SEP]']
\n

이러한 포맷으로 자동으로 토크나이징 해주기 위해서는 post_processor를 추가해주면 됩니다.

\n\n
from tokenizers import SentencePieceUnigramTokenizer\nfrom tokenizers.processors import TemplateProcessing\nfrom transformers import PreTrainedTokenizerFast\n        \ntokenizer = SentencePieceUnigramTokenizer(vocab)      \n\n# '[CLS] SENTENCE [SEP]' format\ntokenizer.post_processor = TemplateProcessing(\n    single=\"[CLS] $A [SEP]\",\n    pair=\"[CLS] $A [SEP] $B:1 [SEP]:1\",\n    special_tokens=[\n        (\"[CLS]\", tokenizer.token_to_id(\"[CLS]\")),\n        (\"[SEP]\", tokenizer.token_to_id(\"[SEP]\")),\n    ],\n)\n\ntokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)\ntokenizer.save_pretrained('SAVE_DIR')
\n

이렇게 저장한 transformers 토크나이저를 아까와 똑같이 로드해서 사용해주면 버트 포맷의 인풋으로 토크나이징 됩니다.

\n\n
from transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained('SAVE_DIR')\ntokenizer.tokenize('튜닙은 자연어처리 테크 스타트업이다.')
\n\n
['[CLS]', '▁', '튜', '닙', '은', '▁자연', '어', '처', '리', '▁테', '크', '▁', '스타', '트', '업', '이다.', '[SEP]']
","htmlAst":{"type":"root","children":[{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"최근 NLP 토크나이저를 만드는데 가장 많이 사용되는 "},{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"tokenizers"}]},{"type":"text","value":" 라이브러와 실제 사용이 가장 많이 되는 "},{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"transformers"}]},{"type":"text","value":" 라이브러리로의 변환에 대한 코드를 담고 있습니다. 해당 내용은 "},{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"tokenizers==0.10.3"}]},{"type":"text","value":" 버젼에서 수행되었습니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Train"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"아래 코드는 wordpiece, char-bpe, byte-level bpe 방식을 지원합니다."},{"type":"element","tagName":"br","properties":{},"children":[]},{"type":"text","value":"\n센텐스피스의 경우 tokenizers에는 현재 sentencepiece가 불안정하여, 오리지널 sentencepiece 학습 방식으로 학습 후 "},{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"tokenizers.SentencePieceUnigramTokenizer.from_spm()"}]},{"type":"text","value":" 메서드로 로드하는 방식을 사용하셔야 합니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"code"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"python"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-python"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-python"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" os\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" argparse\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"from"}]},{"type":"text","value":" tokenizers "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"\n BertWordPieceTokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" \n ByteLevelBPETokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" \n CharBPETokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" \n SentencePieceUnigramTokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\nparser "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" argparse"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"ArgumentParser"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\nparser"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"add_argument"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'--corpus_file'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","builtin"]},"children":[{"type":"text","value":"type"}]},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","builtin"]},"children":[{"type":"text","value":"str"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" required"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"True"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\nparser"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"add_argument"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'--vocab_size'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","builtin"]},"children":[{"type":"text","value":"type"}]},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","builtin"]},"children":[{"type":"text","value":"int"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" default"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","number"]},"children":[{"type":"text","value":"0"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" required"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"True"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\nparser"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"add_argument"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'--limit_alphabet'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","builtin"]},"children":[{"type":"text","value":"type"}]},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","builtin"]},"children":[{"type":"text","value":"int"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" default"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","number"]},"children":[{"type":"text","value":"0"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\nparser"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"add_argument"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'--tokenizer'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","builtin"]},"children":[{"type":"text","value":"type"}]},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","builtin"]},"children":[{"type":"text","value":"str"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" default"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'sentencepiece'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\nparser"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"add_argument"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'--model_type'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","builtin"]},"children":[{"type":"text","value":"type"}]},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","builtin"]},"children":[{"type":"text","value":"str"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" default"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'gpt'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\nparser"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"add_argument"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'--save_dir'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","builtin"]},"children":[{"type":"text","value":"type"}]},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","builtin"]},"children":[{"type":"text","value":"str"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" default"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'sp'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\nargs "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" parser"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"parse_args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"if"}]},{"type":"text","value":" args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"model_type "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"=="}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'bert'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":"\n special_tokens "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"["}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'<|sos|>'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'<|eos|>'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'<|pad|>'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'<|unk|>'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'<|mask|>'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'<|sep|>'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'<|cls|>'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"]"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"elif"}]},{"type":"text","value":" args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"model_type "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"=="}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'gpt'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":"\n special_tokens "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"["}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'<|endoftext|>'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'<|unk|>'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"]"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"else"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"raise"}]},{"type":"text","value":" ValueError"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string-interpolation"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"f\"Unsupported model type: "}]},{"type":"element","tagName":"span","properties":{"className":["token","interpolation"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"{"}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"model_type"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"}"}]}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\""}]}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"if"}]},{"type":"text","value":" args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"tokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"lower"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"=="}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'wordpiece'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":"\n tokenizer "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" BertWordPieceTokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"\n clean_text"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"True"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n handle_chinese_chars"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"True"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n strip_accents"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"False"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n lowercase"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"False"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n wordpieces_prefix"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'##'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n tokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"train"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"files"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"["}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"corpus_file"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"]"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" limit_alphabet"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"limit_alphabet"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" vocab_size"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"vocab_size"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"elif"}]},{"type":"text","value":" args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"tokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"lower"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"=="}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'bbpe'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":"\n tokenizer "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" ByteLevelBPETokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"\n lowercase"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"False"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n add_prefix_space"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"True"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n unicode_normalizer"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'nfc'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n tokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"train"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"\n files"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"["}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"corpus_file"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"]"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n vocab_size"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"vocab_size"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n special_tokens"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"special_tokens"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"elif"}]},{"type":"text","value":" args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"tokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"lower"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"=="}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'char-bpe'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":"\n tokenizer "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" CharBPETokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"\n lowercase"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"False"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n add_prefix_space"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","boolean"]},"children":[{"type":"text","value":"True"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n unicode_normalizer"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'nfc'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n tokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"train"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"\n files"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"["}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"corpus_file"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"]"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n vocab_size"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"vocab_size"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n special_tokens"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"special_tokens"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"else"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"raise"}]},{"type":"text","value":" ValueError"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"Error\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"if"}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"not"}]},{"type":"text","value":" os"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"path"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"exists"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"save_dir"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":":"}]},{"type":"text","value":"\n os"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"mkdir"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"args"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"save_dir"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\ntokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"save"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"tokenizer.json\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Convert to transformers tokenizer"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"tokenizers"}]},{"type":"text","value":" 라이브러리로 학습한 토크나이저는 아래 코드로 간단하게 "},{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"transformers"}]},{"type":"text","value":" 토크나이저로 변환이 가능합니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"python3"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-python3"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-python3"]},"children":[{"type":"text","value":"from transformers import PreTrainedTokenizerFast\n\ntokenizer = PreTrainedTokenizerFast(tokenizer_file='tokenizer.json')\ntokenizer.save_pretrained('SAVE_DIR')"}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"sentencepiece"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"python"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-python"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-python"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" sentencepiece "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"as"}]},{"type":"text","value":" spm\n\nSENTENCEPIECE_MODEL_PREFIX "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"SP\""}]},{"type":"text","value":"\nSENTENCEPIECE_MODEL_TYPE "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" "},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"unigram\""}]},{"type":"text","value":"\n\nspm"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"SentencePieceTrainer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"Train"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","string-interpolation"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"f\"--input=sentencepiece_input.txt \""}]}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","string-interpolation"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"f\"--model_prefix="}]},{"type":"element","tagName":"span","properties":{"className":["token","interpolation"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"{"}]},{"type":"text","value":"SENTENCEPIECE_MODEL_PREFIX"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"}"}]}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":" \""}]}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","string-interpolation"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"f\"--vocab_size="}]},{"type":"element","tagName":"span","properties":{"className":["token","interpolation"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"{"}]},{"type":"text","value":"vocab_size"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"}"}]}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":" \""}]}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","string-interpolation"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"f\"--model_type="}]},{"type":"element","tagName":"span","properties":{"className":["token","interpolation"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"{"}]},{"type":"text","value":"SENTENCEPIECE_MODEL_TYPE"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"}"}]}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":" \""}]}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","string-interpolation"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"f\"--pad_id=0 \""}]}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","string-interpolation"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"f\"--bos_id=1 \""}]}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","string-interpolation"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"f\"--eos_id=2 \""}]}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","string-interpolation"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"f\"--unk_id=3 \""}]}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","string-interpolation"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"f\"--user_defined_symbols=[SEP],[CLS],[MASK]\""}]}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"h3","properties":{},"children":[{"type":"text","value":"Add post process"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"토크나이저를 만들게 되면 따로 설정해주지 않으면 아래와 같은 형태로 토크나이징이 진행됩니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"code"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"python"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-python"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-python"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"from"}]},{"type":"text","value":" transformers "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" AutoTokenizer\n\ntokenizer "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" AutoTokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"from_pretrained"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'SAVE_DIR'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\ntokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"tokenize"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'튜닙은 자연어처리 테크 스타트업이다.'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"output"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"text"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-text"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"['▁', '튜', '닙', '은', '▁자연', '어', '처', '리', '▁테', '크', '▁', '스타', '트', '업', '이다.']"}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"보통 버트나 일렉트라 같은 모델들은 파인튜닝의 편리함을 위해 아래와 같은 포맷으로 문장이 토크나이징 됩니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"text"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-text"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"['[CLS]', '▁', '튜', '닙', '은', '▁자연', '어', '처', '리', '▁테', '크', '▁', '스타', '트', '업', '이다.', '[SEP]']"}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"이러한 포맷으로 자동으로 토크나이징 해주기 위해서는 "},{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"post_processor"}]},{"type":"text","value":"를 추가해주면 됩니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"code"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"python"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-python"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-python"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"from"}]},{"type":"text","value":" tokenizers "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" SentencePieceUnigramTokenizer\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"from"}]},{"type":"text","value":" tokenizers"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"processors "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" TemplateProcessing\n"},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"from"}]},{"type":"text","value":" transformers "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" PreTrainedTokenizerFast\n \ntokenizer "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" SentencePieceUnigramTokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"vocab"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":" \n\n"},{"type":"element","tagName":"span","properties":{"className":["token","comment"]},"children":[{"type":"text","value":"# '[CLS] SENTENCE [SEP]' format"}]},{"type":"text","value":"\ntokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"post_processor "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" TemplateProcessing"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"\n single"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"[CLS] $A [SEP]\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n pair"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"[CLS] $A [SEP] $B:1 [SEP]:1\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n special_tokens"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"["}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"[CLS]\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" tokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"token_to_id"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"[CLS]\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"[SEP]\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":" tokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"token_to_id"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"\"[SEP]\""}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n "},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"]"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":","}]},{"type":"text","value":"\n"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\n\ntokenizer "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" PreTrainedTokenizerFast"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"text","value":"tokenizer_object"},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":"tokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\ntokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"save_pretrained"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'SAVE_DIR'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"p","properties":{},"children":[{"type":"text","value":"이렇게 저장한 "},{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"transformers"}]},{"type":"text","value":" 토크나이저를 아까와 똑같이 로드해서 사용해주면 버트 포맷의 인풋으로 토크나이징 됩니다."}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"code"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"python"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-python"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-python"]},"children":[{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"from"}]},{"type":"text","value":" transformers "},{"type":"element","tagName":"span","properties":{"className":["token","keyword"]},"children":[{"type":"text","value":"import"}]},{"type":"text","value":" AutoTokenizer\n\ntokenizer "},{"type":"element","tagName":"span","properties":{"className":["token","operator"]},"children":[{"type":"text","value":"="}]},{"type":"text","value":" AutoTokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"from_pretrained"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'SAVE_DIR'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]},{"type":"text","value":"\ntokenizer"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"."}]},{"type":"text","value":"tokenize"},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":"("}]},{"type":"element","tagName":"span","properties":{"className":["token","string"]},"children":[{"type":"text","value":"'튜닙은 자연어처리 테크 스타트업이다.'"}]},{"type":"element","tagName":"span","properties":{"className":["token","punctuation"]},"children":[{"type":"text","value":")"}]}]}]}]},{"type":"text","value":"\n"},{"type":"element","tagName":"ul","properties":{},"children":[{"type":"text","value":"\n"},{"type":"element","tagName":"li","properties":{},"children":[{"type":"text","value":"output"}]},{"type":"text","value":"\n"}]},{"type":"text","value":"\n"},{"type":"element","tagName":"div","properties":{"className":["gatsby-highlight"],"dataLanguage":"text"},"children":[{"type":"element","tagName":"pre","properties":{"className":["language-text"]},"children":[{"type":"element","tagName":"code","properties":{"className":["language-text"]},"children":[{"type":"text","value":"['[CLS]', '▁', '튜', '닙', '은', '▁자연', '어', '처', '리', '▁테', '크', '▁', '스타', '트', '업', '이다.', '[SEP]']"}]}]}]}],"data":{"quirksMode":false}},"excerpt":"최근 NLP 토크나이저를 만드는데 가장 많이 사용되는 라이브러와 실제 사용이 가장 많이 되는 라이브러리로의 변환에 대한 코드를 담고 있습니다. 해당 내용은 버젼에서 수행되었습니다. Train 아래 코드는 wordpiece, char-bpe…","fields":{"readingTime":{"text":"4 min read"}},"frontmatter":{"title":"Sooftware NLP - Hugging Face Tokenizers","userDate":"11 August 2021","date":"2021-08-11T15:11:55.000Z","tags":["huggingface","nlp"],"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","backgroundColor":"#484848","images":{"fallback":{"src":"/static/6c6db911d8f477193af9d0fe032dd038/90018/tokenizers.png","srcSet":"/static/6c6db911d8f477193af9d0fe032dd038/f503b/tokenizers.png 750w,\n/static/6c6db911d8f477193af9d0fe032dd038/e4953/tokenizers.png 1080w,\n/static/6c6db911d8f477193af9d0fe032dd038/c2802/tokenizers.png 1366w,\n/static/6c6db911d8f477193af9d0fe032dd038/90018/tokenizers.png 1720w","sizes":"100vw"},"sources":[{"srcSet":"/static/6c6db911d8f477193af9d0fe032dd038/77272/tokenizers.webp 750w,\n/static/6c6db911d8f477193af9d0fe032dd038/317ed/tokenizers.webp 1080w,\n/static/6c6db911d8f477193af9d0fe032dd038/b5c49/tokenizers.webp 1366w,\n/static/6c6db911d8f477193af9d0fe032dd038/e0b55/tokenizers.webp 1720w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.5232558139534883}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"children":[{"gatsbyImageData":{"layout":"fullWidth","backgroundColor":"#282838","images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/0d6f4/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/248f9/soohwan.png 40w,\n/static/a9e6b445142b247ee4cfa66155398bb2/fd435/soohwan.png 80w,\n/static/a9e6b445142b247ee4cfa66155398bb2/0d6f4/soohwan.png 120w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/e7f45/soohwan.webp 40w,\n/static/a9e6b445142b247ee4cfa66155398bb2/589ec/soohwan.webp 80w,\n/static/a9e6b445142b247ee4cfa66155398bb2/71a38/soohwan.webp 120w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6833333333333333}}]}}]}},"relatedPosts":{"totalCount":4,"edges":[{"node":{"id":"01bfefe1-36ae-551f-b213-3eaf935c162d","excerpt":"Huggingface PEFT (Parameter-Efficient Fine-Tuning) 허깅페이스에서 나온 PEFT라는 라이브러리인데 LoRA, Prefix Tuning, P-Tuing, Prompt Tuning…","frontmatter":{"title":"Huggingface PEFT (Parameter-Efficient Fine-Tuning)","date":"2023-03-31T15:11:55.000Z"},"fields":{"readingTime":{"text":"2 min read"},"slug":"/peft/"}}},{"node":{"id":"8a6144d6-6780-530b-b695-d92d023c1219","excerpt":"Sooftware Serving - Huggingface Optimum 허깅페이스에서 나온 Transformers의 Extension 라이브러리이다. 목적은 모델 학습 및 인퍼런스를 더욱 빠르게 해주기 위한 라이브러리이다. Exporting…","frontmatter":{"title":"Sooftware Serving - Huggingface Optimum","date":"2022-10-05T15:11:55.000Z"},"fields":{"readingTime":{"text":"3 min read"},"slug":"/huggingface_optimum/"}}},{"node":{"id":"699a7d22-985a-5874-86f8-489f485f930f","excerpt":"이번에 저희 튜닙에서 공들여 만든 TUNiB Electra 모델을 공개했습니다 !! 🎉 🎉 이번 공개에서는 한-영 bilingual 모델과 한국어 모델을 각각 Small/Base 사이즈로 공개했으며, HuggingFace transformers…","frontmatter":{"title":"TUNiB Electra 공개","date":"2021-09-18T15:11:55.000Z"},"fields":{"readingTime":{"text":"2 min read"},"slug":"/tunib_electra/"}}},{"node":{"id":"996d2b57-1627-59bf-ad52-684205e35ca6","excerpt":"최근 NLP 토크나이저를 만드는데 가장 많이 사용되는 라이브러와 실제 사용이 가장 많이 되는 라이브러리로의 변환에 대한 코드를 담고 있습니다. 해당 내용은 버젼에서 수행되었습니다. Train 아래 코드는 wordpiece, char-bpe…","frontmatter":{"title":"Sooftware NLP - Hugging Face Tokenizers","date":"2021-08-11T15:11:55.000Z"},"fields":{"readingTime":{"text":"4 min read"},"slug":"/tokenizers/"}}}]}},"pageContext":{"slug":"/tokenizers/","prev":{"excerpt":"Efficient Attention: Attention with Linear Complexities Shen Zhuoran et al. Abstract Dot-product attention은 들어오는 인풋 길이에 따라 memory…","frontmatter":{"title":"Sooftware NLP - Efficient Attention Paper Review","tags":["nlp","paper"],"date":"2021-07-17T10:00:00.000Z","draft":false,"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAYAAAAywQxIAAAACXBIWXMAAAsTAAALEwEAmpwYAAABzUlEQVQoz2WS227TQBCG/cg8AHfccIuExAsgxB1C6g0SvSmlAVGklhKpiZ00jont+LA+78GnD8WlIpRfGu3O6h+tZr6xxnHkoK7ryDNBUeTUdc2xDp6HOGgYBvI8oyxLpJT/eKzjwpUvUG0/3XexzXY/Z2TgsUzbY+9SxqMPH2R1rcEoidGKm01EVkpEGnG6fcqH7RNEvcUYQ9e3tFqhlaSWmh+bmNZo9CG0RqoGbSRWubogu35PWhlutxGhqEjTiNnuBZ/953jBgk/2K8JqSXl9QmJ/JS4N87uAWhnGAQYDV3cnXHqvserFKdX3N4S55tutxybM/rSl0G1DnuV8XD7Dr+Y0l28RizPcpGE23xCLhp/Lc+y7G67cd8y8l1i9KunqFKUUF44gEDVj32FMh9ZmaqdQAaaX9I2glQVpKTmzM5TU7CKbtNxh+ppK7/9CGYaeL+uCMFeHMU8ku66d6D8mnuQ1506Oabv/gFlt2yJlQ1EUpFlBUZTT6lRVNb1lWUaSxERRhG0vWa1W+H5AIu49dV3RyIaqul8hSwjBer3Gdhwcx2HjbvA8jyAIcLcuruvi+z7eL2/KgzAg3IfEUTSd+/2eMAwJAp84SfgNdaKop4k5L4oAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/2cdc9b7481cd5e517d106c8618db52d3/a2d8d/efficient_attention.png","srcSet":"/static/2cdc9b7481cd5e517d106c8618db52d3/a74b9/efficient_attention.png 750w,\n/static/2cdc9b7481cd5e517d106c8618db52d3/63493/efficient_attention.png 1080w,\n/static/2cdc9b7481cd5e517d106c8618db52d3/a2d8d/efficient_attention.png 1211w","sizes":"100vw"},"sources":[{"srcSet":"/static/2cdc9b7481cd5e517d106c8618db52d3/ba69e/efficient_attention.webp 750w,\n/static/2cdc9b7481cd5e517d106c8618db52d3/f8adb/efficient_attention.webp 1080w,\n/static/2cdc9b7481cd5e517d106c8618db52d3/a7f95/efficient_attention.webp 1211w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.4706853839801816}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAADjElEQVQ4y23Oy48TdQDA8TExEaTttvSBfdHddru7bWc6bWd+8+i0u23n0SdlBRQXZGHDgiiLDwgqXjCIJEaC8cLBmHgwUU9EYzRqQuLJi/4NJB6NF+8evgbOHr7XT76SM5ix7s4QnRHnnm9yZzvBTIswN8JcP5bC67ZoWi7CdtFsn5bloZo+dSNAMYbIIqCmD6mJIVXdR3J6I7r9IQ3T59pZlW+uxzndCbPdC/HZXoq3TikowsfqBOjOmFZ7SrM9p2kfRbXn1O0jKNbsSbJ5BMnu9Gl3+qiiz4OPSvz0QRLHFGwOZH6+neLHT5YRbRerM0J0JrTas/8FZXNKzZgg2es+ZtvDdbs8+iHJl+9lWVP7OG2br26k+PvXJCfmFqoYIpwhTXtMwxpTN0coxhjFmCKbE2rGmKoYPQY9qi2fyzsq//4e5sNXi5TlLqbd5t6lPH/9kuDTG2usqgGi7dO0hzSsIXUzQDaGKGKMLEZUxZCKCJB02+PUpMH9vQwP7ybYnSxyfNrjwumA3dEiD94v8cfnK7xzWsbpujQtH9X0qBs+iuEjC5+a7lPVPSqai9SyelyYFbjkh5hqC3TkKOutLON2nkYpgtuMcT6IszdPslo8SLlcpmUH1A0XRQyQhUtNuFQ1l4o2QFI1h3OTVXbcg+grIfSVBRqlEGopgrYSRi0e4KiIse1mScSeopA/iLADmoZL3Rgg631q+oCq1qeq95FOzhQefpzn7eMpBmqcoBVj04pzwkky0eNMRYIdL8F3twusLe2nslrE6g5pGX1U8/FhD1nvUW1tUJK7SH9+n+DR1zF2vQSvH0lz88wiV45V2J1WuHqsxLXNAnuzNJdfWmK5/BylUgFhb6DZAxR9g7rpIos+Wttja2uE9M9vYb69k+DqfJErWzLOuoVqdKlpDo4jODms4zlFckuHObx4iHwuRi6fYqmYY201x/LqMrnFDNl8gRvvukhNdT9394rcOlNFMxrUhYFmW9imTEsp0mhWKC2nWSokyWSiFA7HSKcXiEb34fayyLVniUQk8rmn2drKIqlqiNcuVrh2ymQS1Nh8sc/GoMEbl7u8eb5D1yqTzkRJJsOkkiGymQVShyLE4yHWKnEq1RjpTIhmI8r2y3mkyloI38uha1n6bpVXrkw5+4LD/S/2uHnrIopaJhZ9hmQqwqHkAXJP8BDhyD46ZoWdMxOGcx/PNfECk/8AbxXdRjRliPoAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/34f77/soohwan.png 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/a94f6/soohwan.png 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png 1148w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/38420/soohwan.webp 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7470d/soohwan.webp 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/b5ef6/soohwan.webp 1148w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6829268292682927}}}}]},"fields":{"readingTime":{"text":"1 min read"},"layout":"","slug":"/efficient-attention/"}},"next":{"excerpt":"2021 LangCon 발표 - “한국어 음성 인식: KoSpeech 개발기부터 OpenSpeech 개발기까지” 올해로 3회째를 맞는 LangCon에 발표자로 참석해 “한국어 음성 인식: KoSpeech 개발기부터 OpenSpeech…","frontmatter":{"title":"2021 LangCon 발표 - \"한국어 음성 인식: KoSpeech 개발기부터 OpenSpeech 개발기까지\"","tags":["presentation"],"date":"2021-08-28T10:00:00.000Z","draft":false,"excerpt":null,"image":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAcCAYAAABh2p9gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAGXElEQVRIx1WW2VNb1x3H5Zd22syk7/0HmpluT+m0M31ppy+Z9KVt8pBO3M6kfkljB5qa4C0J2DQu2MYEjBE7QmLTDhhqIyE2Y2w2gZar7W5CEthgHAyWAwj06dzLMrZmzhzdc+/96vv9fX/ne2RwKira6BZlekQFh6xijsl0JWR93SEruJMK9tQqPcl1bOl1bKJKy1iABs8CnYtx3Kkl/TnteYNdVqgPJbi1GKcpLOo3tOvGUBxrQsSdStIZzWC728WgtQiX7Soto/PUDYWocU3T9iCsv+NQFH02uBSVxrBIXTBBsyBikxTMMYkOKUVz4jEmv8xg80eITW+Cz8DLXgPT9b+gYXCKaucs1Y5HOCXlWI1B+9IaEelMyDpQUyhBeyJFj2cAb+tJHNfehwkD9/77ax7e+inP7vyI7T4DtqZijBNJnFKClqiol0gjZ9BAjMGEzrI9JmGJJfCubJAeMzFS9FtWO96AUQMb/T/k274fsO8zwLCB7qt/46ZLoGYwQFtMpUc8BNSMaBFEbgfjDKUyCM/W6V16TJ3ZQ9jmg3Q9uamfse8xsO87Qc57gq2B7xM1/ZgV2/cYv/Zzqho9dCaXsIkyBrd64HCfuoT0fJPU5ktSW1k2cpAFfGt5VjZl8qMn2PEYDobXACNvsO36DXvtv6LmSgeWZAZrQjpw2RiK40uvoH2yuRxPv9shn9/jjrTEufsCS4se6Hub3MBP2P3fL1lrfgdXaQkl5QN0XbfSMCVgFmWsooJBc9USl/BmMjpYPp/Xgb/L5UhnX7K+9QLum8F6nnxjIftlZ9n99wXU01dZv1TB88++xv5gnmpBxiSIBzXUDGmLisyuPmUlm2V+7akOurObI5rd5UVgACoL2CspZf9qJRiNUPAFXPoPuxe/ZmBiFstRDTXJGsvaQJw+JYUnnebB4yc6oH9tg3Z/hO3qq1Beyd6lcnL/vMR+RSW5suvkC79i9Uo1nXFZJ2U/6kNTVNIXSh8Kej3F58959nKHC48k7t7ohBtV5E3N7J0rY+eDz9i/XEHe0gpl5cy1OrBklrElJBxHbdMWkfRO/9QXpOShwMZOFu/yBh+bh8ndrtPZ5c5eBlMTe0VXyJVUQEMdT86V070YoVtRcb269bTmrg8muD4X4/xkiHR2k5nVLcauNUDFDXb+Vcru3z8/APu8DC6UsXXmSzptw/QsZ2gXNA+kA8kaasdhDboTMpXzUaxiUneZZjO7Hxaw/aczbL/3KTsfFLJ3qojds1cYs3npyizjFOXjpNEZapK1xrZKB+Zoi7ULMe6mVth4sgaddvKFV+CLa1BlBIsdeSbEwJNVumOibqa2j7Wtp8eXU9ev6lmoBUSTINKzlMSeVulbzfDgxTNCTzLEV9NEvl1jOruJc+0x3aqCI5nUHdaUWQ8TR5esU5YVXEmVjrDIrTt+qtwBynoCVLr8VA0sUNEb4puBOar7ZqlwzFDlnqVzIYZ7Kam/eyz5CMwuKvQ/TlNU3Mdf3jrHJ78voPjdUxS9c4rCP3zM6d+d4eTb5zn/x48ofe8kf37rIn9914zz0N3jxD46AhySBpjipnWaotPNlF2spfzLSm5cvsFXxUZKiusoKmjX58sX6vjkH2ZKKofpzSy9Dnh8oap0zEVpfyRQ7w1SPxyhZUqi9p6A0ROmYSRC3d1Fmsai1HsFWsZDWKYjWOai2EX59cTWQVWV/rCMfSxC11AQ22iYO/MJHOOCPpz3I3QPBemdimEfDeMcj9A/IzIYUnR1utKjGrq0TBREWsaDdMdk2qcELDMRmnwBTJNhmkcCmB9F9AOpfSZCh9argkjrRIjW8RCmhwKuZPIVUxRVN0W7YfQscKt/lmr3DDXuGX2+2fOQb1zTNAzM0+Sep+HeAsZ7fqod0/oPWGPS6y5r7WKejWK86+d23ywdniA9w2HaPAHMniCOiQiNtllcPgG7O0Cze5567wI1rhkavYuv7xQNrHMxRrV9Wj+4W2cjtPoCtE2GaRsPYpkMY5oQaPGFME0K1HsWqB2co2kkQKNvka5AXDf0yFzDkcPd4QT1Q34sEZE7apLlbJb7kST2R3H8okJAVvGLEvLqGukXWXolFdvhv4sjdsdtc1BDmQ5/jJ6ETF9yicX1dYbEFI6wwoSiMC7LjIoS08vLzD9dx60m9fxzvjI0hv8HO3gRxH7m8SQAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/44403c2245dd19b4a40f892e78354c89/e4e7a/langcon.png","srcSet":"/static/44403c2245dd19b4a40f892e78354c89/e4e7a/langcon.png 667w","sizes":"100vw"},"sources":[{"srcSet":"/static/44403c2245dd19b4a40f892e78354c89/0749c/langcon.webp 667w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":1.421289355322339}}},"author":[{"id":"Soohwan Kim","bio":"Co-founder/A.I. engineer at TUNiB.","avatar":{"childImageSharp":{"gatsbyImageData":{"layout":"fullWidth","placeholder":{"fallback":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAABYlAAAWJQFJUiTwAAADjElEQVQ4y23Oy48TdQDA8TExEaTttvSBfdHddru7bWc6bWd+8+i0u23n0SdlBRQXZGHDgiiLDwgqXjCIJEaC8cLBmHgwUU9EYzRqQuLJi/4NJB6NF+8evgbOHr7XT76SM5ix7s4QnRHnnm9yZzvBTIswN8JcP5bC67ZoWi7CdtFsn5bloZo+dSNAMYbIIqCmD6mJIVXdR3J6I7r9IQ3T59pZlW+uxzndCbPdC/HZXoq3TikowsfqBOjOmFZ7SrM9p2kfRbXn1O0jKNbsSbJ5BMnu9Gl3+qiiz4OPSvz0QRLHFGwOZH6+neLHT5YRbRerM0J0JrTas/8FZXNKzZgg2es+ZtvDdbs8+iHJl+9lWVP7OG2br26k+PvXJCfmFqoYIpwhTXtMwxpTN0coxhjFmCKbE2rGmKoYPQY9qi2fyzsq//4e5sNXi5TlLqbd5t6lPH/9kuDTG2usqgGi7dO0hzSsIXUzQDaGKGKMLEZUxZCKCJB02+PUpMH9vQwP7ybYnSxyfNrjwumA3dEiD94v8cfnK7xzWsbpujQtH9X0qBs+iuEjC5+a7lPVPSqai9SyelyYFbjkh5hqC3TkKOutLON2nkYpgtuMcT6IszdPslo8SLlcpmUH1A0XRQyQhUtNuFQ1l4o2QFI1h3OTVXbcg+grIfSVBRqlEGopgrYSRi0e4KiIse1mScSeopA/iLADmoZL3Rgg631q+oCq1qeq95FOzhQefpzn7eMpBmqcoBVj04pzwkky0eNMRYIdL8F3twusLe2nslrE6g5pGX1U8/FhD1nvUW1tUJK7SH9+n+DR1zF2vQSvH0lz88wiV45V2J1WuHqsxLXNAnuzNJdfWmK5/BylUgFhb6DZAxR9g7rpIos+Wttja2uE9M9vYb69k+DqfJErWzLOuoVqdKlpDo4jODms4zlFckuHObx4iHwuRi6fYqmYY201x/LqMrnFDNl8gRvvukhNdT9394rcOlNFMxrUhYFmW9imTEsp0mhWKC2nWSokyWSiFA7HSKcXiEb34fayyLVniUQk8rmn2drKIqlqiNcuVrh2ymQS1Nh8sc/GoMEbl7u8eb5D1yqTzkRJJsOkkiGymQVShyLE4yHWKnEq1RjpTIhmI8r2y3mkyloI38uha1n6bpVXrkw5+4LD/S/2uHnrIopaJhZ9hmQqwqHkAXJP8BDhyD46ZoWdMxOGcx/PNfECk/8AbxXdRjRliPoAAAAASUVORK5CYII="},"images":{"fallback":{"src":"/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png","srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/34f77/soohwan.png 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/a94f6/soohwan.png 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7cf1f/soohwan.png 1148w","sizes":"100vw"},"sources":[{"srcSet":"/static/a9e6b445142b247ee4cfa66155398bb2/38420/soohwan.webp 750w,\n/static/a9e6b445142b247ee4cfa66155398bb2/7470d/soohwan.webp 1080w,\n/static/a9e6b445142b247ee4cfa66155398bb2/b5ef6/soohwan.webp 1148w","type":"image/webp","sizes":"100vw"}]},"width":1,"height":0.6829268292682927}}}}]},"fields":{"readingTime":{"text":"2 min read"},"layout":"","slug":"/langcon_2021/"}},"primaryTag":"huggingface"}}, "staticQueryHashes": ["3170763342","3229353822"]}