github
/
ColossalAI
mirror of https://github.com/hpcaitech/ColossalAI
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
Browse Source

fix bug

pull/6119/head
Tong Li 1 week ago
parent
375e356a16
commit
dacc04ef75
1 changed files with 1 additions and 1 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Unified View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 2
      applications/ColossalChat/coati/dataset/tokenization_utils.py

2
applications/ColossalChat/coati/dataset/tokenization_utils.py
Unescape View File

@ -419,7 +419,7 @@ def tokenize_process_reward(
template = deepcopy(conversation_template) template = deepcopy(conversation_template)
template.append_message("user", input) template.append_message("user", input)
template.append_message("assistant", label) template.append_message("assistant", label)
prompt = template.get_prompt(add_generation_prompt=True) prompt = template.get_prompt(add_generation_prompt=False)
reward_signal_id = tokenizer.convert_tokens_to_ids(template.reward_signal) reward_signal_id = tokenizer.convert_tokens_to_ids(template.reward_signal)
tokenized = tokenizer(prompt, add_special_tokens=False)["input_ids"] tokenized = tokenizer(prompt, add_special_tokens=False)["input_ids"]

Write Preview
Loading…
Cancel
Save