From ee9a22f04a4e049a6f7a269fda5c11f98491a28b Mon Sep 17 00:00:00 2001 From: Wenwei Zhang <40779233+ZwwWayne@users.noreply.github.com> Date: Thu, 18 Jan 2024 11:48:24 +0800 Subject: [PATCH] [Doc] update chat format doc (#619) --- chat/chat_format.md | 137 +++++++++++++++++++++++++++++++++++--- chat/chat_format_zh-CN.md | 136 ++++++++++++++++++++++++++++++++++--- 2 files changed, 251 insertions(+), 22 deletions(-) diff --git a/chat/chat_format.md b/chat/chat_format.md index 2b3519a..eaee8b7 100644 --- a/chat/chat_format.md +++ b/chat/chat_format.md @@ -17,18 +17,65 @@ Hello[UNUSED_TOKEN_145] Hello, I am InternLM2-Chat, how can I assist you?[UNUSED_TOKEN_145] ``` -Here, `[UNUSED_TOKEN_146]` acts as the start token for each turn of dialogue, and `[UNUSED_TOKEN_145]` as the end token. Each turn of dialogue typically starts with `[UNUSED_TOKEN_146]role` and ends with the model's output `[UNUSED_TOKEN_145]`, where role represents `system`, `user`, `assistant`, and `environment`. Currently, the InternLM2-Chat model's vocabulary also maintains the following mappings: +Here, `[UNUSED_TOKEN_146]` acts as the start token for each turn of dialogue, and `[UNUSED_TOKEN_145]` as the end token. Each turn of dialogue typically starts with `[UNUSED_TOKEN_146]role` and ends with the model's output `[UNUSED_TOKEN_145]`, where role represents `system`, `user`, `assistant`, and `environment`. You may refer to the [code in huggingface](https://huggingface.co/internlm/internlm2-chat-7b/blob/main/modeling_internlm2.py#L1138) to see how the chat history is organized. -- `[UNUSED_TOKEN_146]`: Start token for each role's dialogue -- `[UNUSED_TOKEN_145]`: End token for each role's dialogue -- `[UNUSED_TOKEN_144]`: Start token for invoking external plugins -- `[UNUSED_TOKEN_143]`: End token for invoking external plugins -- `[UNUSED_TOKEN_142]`: Code interpreter -- `[UNUSED_TOKEN_141]`: External plugins, regular tools +Currently, the InternLM2-Chat model's vocabulary maintains the following mappings to support full functionalities: + +- `[UNUSED_TOKEN_146]`: Start token for each role's dialogue, the token ID is **92543** +- `[UNUSED_TOKEN_145]`: End token for each role's dialogue, the token ID is **92542** +- `[UNUSED_TOKEN_144]`: Start token for invoking external plugins, the token ID is **92541** +- `[UNUSED_TOKEN_143]`: End token for invoking external plugins, the token ID is **92540** +- `[UNUSED_TOKEN_142]`: Code interpreter, the token ID is **92539** +- `[UNUSED_TOKEN_141]`: External plugins, regular tools, the token ID is **92538** ## Complete Structure -The complete dialogue format of InternLM2-Chat, based on the basic structure, also includes designs for general-purpose AI agents. Its core purpose is to use a streaming format that allows the same format to support various types of plugin extensions and AI environments while being compatible with general dialogue. The state of a general AI agent dialogue is shown below: +The complete chat format of InternLM2-Chat, based on the basic structure, also includes designs for general-purpose AI agents. Its core purpose is to use a streaming format that allows the same format to support various types of plugin extensions and AI environments while being compatible with general chat. + +### Function Call + +InternLM2-Chat support multiple formats (e.g., ReAct) to conduct function call, especially **json format** to ease downstream applications。An example of complete function call is shown below. + +``` +[UNUSED_TOKEN_146]system +You are InternLM2-Chat, a harmless AI assistant[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]system name=[UNUSED_TOKEN_141] +[ + { + "name": "get_current_weather", + "description": "Get the current weather in a given location", + "parameters": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA", + }, + "unit": {"type": "string"}, + }, + "required": ["location"], + }, + } +] +[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]user +I want to know today's weather in Shanghai[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]assistant +Sure, I will search for the weather of Shanghai.[UNUSED_TOKEN_144][UNUSED_TOKEN_141] +{"name": "get_current_weather", "parameters": {"location": "Shanghai"}}[UNUSED_TOKEN_143][UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]environment name=[UNUSED_TOKEN_141] +{"temperature": 22}[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]assistant +The weather in Shanghai is 22 celsius[UNUSED_TOKEN_145] +``` + +1. First，there will be a new system prompt that describe the protocol of tools in json format. The content starts with `[UNUSED_TOKEN_146]system name=[UNUSED_TOKEN_141]\n` and ends with `[UNUSED_TOKEN_145]`. `name=[UNUSED_TOKEN_141]` indicates the system prompt comes from tools. InternLM2-Chat supports and follows multiple system prompts in the chat history, so we can also see the system prompt ask the model to be helpful assistant. +2. Model will call the tools in a streaming format after receving the user prompt, i.e., it will naturally speak something (thoughts, helpful response) then call the tools. The model will output `[UNUSED_TOKEN_144][UNUSED_TOKEN_141]`, where `[UNUSED_TOKEN_144]` indicates the model needs to call extensions and `[UNUSED_TOKEN_141]` indicates it want to use common tools. Then the model will output the calling parameters in json format and end it with `[UNUSED_TOKEN_143]`. +3. The environment returns the function calling outputs, starts with `[UNUSED_TOKEN_146]environment name=[UNUSED_TOKEN_141]`, indicating it comes from the function calling in the environment. The content is in json format, `{"temperature": 22}`, ends with `[UNUSED_TOKEN_145]`. + +### Code Interpreter + +InternLM2-Chat support to use code interpreter to solve complex math problems and data analysis. The whole process is similar to the function calling, shown as below. ``` [UNUSED_TOKEN_146]system @@ -40,7 +87,70 @@ You now have access to a Jupyter notebook environment supporting Python code exe - Programming examples (for understanding programming concepts or language features) - Text processing and analysis (including text analysis and natural language processing) - Machine learning and data science (model training and data visualization) -- File operations and data import (handling CSV, JSON, etc. formats) +- File operations and data import (handling CSV, JSON, etc. formats)[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]user +Please help me process and visualize this dataset. +[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]user name=file +[{"path": "data.csv", size='10K'}][UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]assistant +I have processed the data and visualized it for you. + +[UNUSED_TOKEN_144][UNUSED_TOKEN_142] +```python +import plotly.express as px +import pandas as pd + +# Load the data into a pandas dataframe +df = pd.read_csv('data.csv') + +# Create a scatter plot of rainfall vs wind direction +fig = px.scatter(df, x='WindDir9am', y='Rainfall', color='WindDir3pm', + labels={'WindDir9am': 'Wind Direction 9am', 'Rainfall': ' + +Rainfall', 'WindDir3pm': 'Wind Direction 3pm'}, + title='Rainfall vs Wind Direction', + template='plotly_dark', + width=600, height=500) + +# Add a hover effect to show the date +fig.update_traces(hovertemplate='Date: %{text}
Wind Direction 9am: %{x}
Rainfall: %{y}
Wind Direction 3pm: %{marker.color}') + +# Show the plot +fig.show() +```[UNUSED_TOKEN_143] +[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]environment name=[UNUSED_TOKEN_142] +![image](xxx.png) +[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]assistant +Since the code output is not included here, I cannot provide specific chart content. However, if the code executed correctly, it should display a polar plot with two filled areas representing the relationship between wind direction at 9 am and rainfall, and between wind direction at 3 pm and rainfall, respectively. The values for each direction are based on the average rainfall calculated from the provided dataset. The chart should have a clear title, a legend, and be intuitive for comparing rainfall with different wind directions. Given the use of a dark theme, the overall appearance of the chart should be bright lines and filled areas on a dark background.[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]user +Can you explain the code?[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]assistant +The code uses Python's Plotly library to create a polar plot that shows the relationship between wind direction and rainfall. It starts by creating a Plotly Figure object. Then, I added two `Scatterpolar` traces. The first trace shows the relationship between wind direction at 9 am and average rainfall, and the second trace shows the same for 3 pm. In these traces, the `r` parameter represents the average rainfall, and `theta` represents the wind direction. `fill='toself'` means the shape is filled to its own boundary. Next, I updated the chart layout, hid the radial axis, added a legend, set the title and its font, chose a dark theme, and set the chart's background color, width, and height. Finally, the chart was displayed using `fig.show()`. +[UNUSED_TOKEN_145] +``` + +1. First，there will be a new system prompt that describe the usages of code interpreter. The content starts with `[UNUSED_TOKEN_146]system name=[UNUSED_TOKEN_142]\n` and ends with `[UNUSED_TOKEN_145]`. `name=[UNUSED_TOKEN_142]` indicates the system prompt comes from code interpreter. InternLM2-Chat supports and follows multiple system prompts in the chat history, so we can also see the system prompt ask the model to be helpful assistant. +2. Model will call the tools in a streaming format after receving the user prompt, i.e., it will naturally speak something (thoughts, helpful response) then call the code interpreter. The model will output `[UNUSED_TOKEN_144][UNUSED_TOKEN_142]`, where `[UNUSED_TOKEN_144]` indicates the model needs to call extensions and `[UNUSED_TOKEN_142]` indicates it want to use code interpreter. Then the model will output the code in a markdown-style code block and end it with `[UNUSED_TOKEN_143]`. +3. The environment will execute the code and return outputs, starts with `[UNUSED_TOKEN_146]environment name=[UNUSED_TOKEN_142]`, indicating it comes from the code interpreter in the environment, and the content ends with `[UNUSED_TOKEN_145]`. + +### Function Call + Code Interpreter + +InternLM2-Chat can choose to use tools or code interpreter automatically. We only need to send the system prompt of both to the model respectively. An example is shown as below + +``` +[UNUSED_TOKEN_146]system +You are InternLM2-Chat, a harmless AI assistant[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]system name=[UNUSED_TOKEN_142] +You now have access to a Jupyter notebook environment supporting Python code execution. Just send code to python to run in this stateful environment. This feature is suitable for: +- Data analysis or processing (such as data manipulation and graphic creation) +- Complex calculations (such as math and physics problems) +- Programming examples (for understanding programming concepts or language features) +- Text processing and analysis (including text analysis and natural language processing) +- Machine learning and data science (model training and data visualization) +- File operations and data import (handling CSV, JSON, etc. formats)[UNUSED_TOKEN_145] [UNUSED_TOKEN_146]system name=[UNUSED_TOKEN_141] [ { @@ -104,6 +214,11 @@ The code uses Python's Plotly library to create a polar plot that shows the rela [UNUSED_TOKEN_145] [UNUSED_TOKEN_146]user I want to know today's weather in Shanghai[UNUSED_TOKEN_145] -[UNUSED_TOKEN_144][UNUSED_TOKEN_141] -{"name": "get_current_weather", "parameters": {"location": "Shanghai"}}[UNUSED_TOKEN_143] +[UNUSED_TOKEN_146]assistant +Sure, I will search for the weather of Shanghai.[UNUSED_TOKEN_144][UNUSED_TOKEN_141] +{"name": "get_current_weather", "parameters": {"location": "Shanghai"}}[UNUSED_TOKEN_143][UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]environment name=[UNUSED_TOKEN_141] +{"temperature": 22}[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]assistant +The weather in Shanghai is 22 celsius[UNUSED_TOKEN_145] ``` diff --git a/chat/chat_format_zh-CN.md b/chat/chat_format_zh-CN.md index 657016f..60dc28f 100644 --- a/chat/chat_format_zh-CN.md +++ b/chat/chat_format_zh-CN.md @@ -17,18 +17,128 @@ InternLM2-Chat 采用了全新的对话格式，以灵活地支持工具调用 你好，我是书生浦语，请问有什么可以帮助你的吗[UNUSED_TOKEN_145] ``` -其中 `[UNUSED_TOKEN_146]` 充当了每轮对话开始符，`[UNUSED_TOKEN_145]` 充当了当前轮对话结束符。每轮对话一般以 `[UNUSED_TOKEN_146]role` 开头，以模型输出的 `[UNUSED_TOKEN_145]` 结尾，role 代表 `system`，`user`，`assistant` 和 `environment` 角色。目前，InternLM2-Chat 模型的词表中还维护了如下映射 +其中 `[UNUSED_TOKEN_146]` 充当了每轮对话开始符，`[UNUSED_TOKEN_145]` 充当了当前轮对话结束符。每轮对话一般以 `[UNUSED_TOKEN_146]role` 开头，以模型输出的 `[UNUSED_TOKEN_145]` 结尾，role 代表 `system`，`user`，`assistant` 和 `environment` 角色。你可以参考[huggingface 上的代码](https://huggingface.co/internlm/internlm2-chat-7b/blob/main/modeling_internlm2.py#L1138)来了解对话历史的拼接。 -- `[UNUSED_TOKEN_146]`：每个角色对话的开始符 -- `[UNUSED_TOKEN_145]`：每个角色对话的结束符 -- `[UNUSED_TOKEN_144]`：模型调用外部插件的开始符 -- `[UNUSED_TOKEN_143]`：模型调用外部插件的结束符 -- `[UNUSED_TOKEN_142]`：代码解释器 -- `[UNUSED_TOKEN_141]`：外部插件，常规的 tools +目前，InternLM2-Chat 模型的词表中还维护了如下映射 + +- `[UNUSED_TOKEN_146]`：每个角色对话的开始符，在词表中对应的 token ID 是**92543** +- `[UNUSED_TOKEN_145]`：每个角色对话的结束符，在词表中对应的 token ID 是**92542** +- `[UNUSED_TOKEN_144]`：模型调用外部插件的开始符，在词表中对应的 token ID 是**92541** +- `[UNUSED_TOKEN_143]`：模型调用外部插件的结束符，在词表中对应的 token ID 是**92540** +- `[UNUSED_TOKEN_142]`：代码解释器，在词表中对应的 token ID 是**92539** +- `[UNUSED_TOKEN_141]`：外部插件，常规的 tools，在词表中对应的 token ID 是**92538** ## 完整结构 -InternLM2-Chat 的完整对话格式在上述基本结构的基础上还包含了针对通用智能体的设计，其核心目的是采用流式格式，使得同一套格式在支持各种类插件拓展和智能体环境的同时能够和通用对话兼容。通用的智能体对话状态如下所示 +InternLM2-Chat 的完整对话格式在上述基本结构的基础上还包含了针对通用智能体的设计，其核心目的是采用流式格式，使得同一套格式在支持各种类插件拓展和智能体环境的同时能够和通用对话兼容。 + +### 工具调用 + +InternLM2-Chat 的通用工具调用（function call）支持多种格式，尤其是**json 格式**以便捷下游应用。以此为例，一次完整的工具调用流程如下所示： + +``` +[UNUSED_TOKEN_146]system +你是书生浦语2，一个无害的人工智能助手[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]system name=[UNUSED_TOKEN_141] +[ + { + "name": "get_current_weather", + "description": "Get the current weather in a given location", + "parameters": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA", + }, + "unit": {"type": "string"}, + }, + "required": ["location"], + }, + } +] +[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]user +我想了解今天上海的天气[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]assistant +好的，我将为你查询上海的天气。[UNUSED_TOKEN_144][UNUSED_TOKEN_141] +{"name": "get_current_weather", "parameters": {"location": "Shanghai"}}[UNUSED_TOKEN_143][UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]environment name=[UNUSED_TOKEN_141] +{"temperature": 22}[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]assistant +上海的天气是 22 摄氏度[UNUSED_TOKEN_145] +``` + +1. 首先，在系统提示中以 json 的格式提供工具调用的格式和字段描述。内容以 `[UNUSED_TOKEN_146]system name=[UNUSED_TOKEN_141]\n`开头，`[UNUSED_TOKEN_145]` 结尾，`name=[UNUSED_TOKEN_141]` 体现了这是来自工具的指令。InternLM2-Chat 支持 system 角色对模型的提示和约束多次出现。所以我们会看到前面还有关于对话的要求。 +2. 模型在接受到用户指令后，会以流式的形式调用工具，及自然地生成文字进行思考/回应用户，然后输出`[UNUSED_TOKEN_144][UNUSED_TOKEN_141]`。`[UNUSED_TOKEN_144]`表示要调用外部插件，同时 `[UNUSED_TOKEN_141]` 表示调用的是常规工具。然后模型输出 json 格式的工具调用内容，再以 `[UNUSED_TOKEN_143]` 表示工具调用结束。 +3. 系统返回工具调用结果，以 `[UNUSED_TOKEN_146]environment name=[UNUSED_TOKEN_141]`开头，表示是来自环境关于工具调用的反馈，内容是 json 格式，`{"temperature": 22}`，以`[UNUSED_TOKEN_145]`结尾。 + +### 代码解释器 + +InternLM2-Chat 支持使用代码解释器解决复杂数学问题和进行数据分析。整个调用流程和通用工具调用类似，如下所示 + +``` +[UNUSED_TOKEN_146]system +你是书生浦语2，一个无害的人工智能助手[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]system name=[UNUSED_TOKEN_142] +你现在可以使用一个支持 Python 代码执行的 Jupyter 笔记本环境。只需向 python 发送代码，即可在这个有状态环境中进行运行。这个功能适用于: +- 数据分析或处理（如数据操作和图形制作） +- 复杂计算（如数学和物理问题） +- 编程示例（用于理解编程概念或语言特性） +- 文本处理和分析（包括文本分析和自然语言处理） +- 机器学习和数据科学（模型训练和数据可视化展示） +- 文件操作和数据导入（处理CSV、JSON等格式文件） +[UNUSED_TOKEN_146]user +请帮我对该数据集进行数据处理并可视化。 +[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]user name=file +[{"path": "data.csv", size='10K'}][UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]assistant +我已经帮您处理了数据并进行了可视化。 + +[UNUSED_TOKEN_144][UNUSED_TOKEN_142] +```python +import plotly.express as px +import pandas as pd + +# Load the data into a pandas dataframe +df = pd.read_csv('data.csv') + +# Create a scatter plot of rainfall vs wind direction +fig = px.scatter(df, x='WindDir9am', y='Rainfall', color='WindDir3pm', + labels={'WindDir9am': 'Wind Direction 9am', 'Rainfall': 'Rainfall', 'WindDir3pm': 'Wind Direction 3pm'}, + title='Rainfall vs Wind Direction', + template='plotly_dark', + width=600, height=500) + +# Add a hover effect to show the date +fig.update_traces(hovertemplate='Date: %{text}
Wind Direction 9am: %{x}
Rainfall: %{y}
Wind Direction 3pm: %{marker.color}') + +# Show the plot +fig.show() +```[UNUSED_TOKEN_143] +[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]environment name=[UNUSED_TOKEN_142] +![image](xxx.png) +[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]assistant +由于代码输出没有包含在这里，我无法提供具体的图表内容。但是，如果代码执行无误，它会显示一个极坐标图，其中包含两个填充区域，一个代表上午9点的风向与降雨量的关系，另一个代表下午3点的风向与降雨量的关系。每个方向上的值是根据提供的数据集计算得到的平均降雨量。图表应该有一个清晰的标题、图例，并且能够直观地比较不同风向下的降雨量。由于使用了暗色主题，图表的整体外观应该是深色背景上的亮色线条和填充区域。[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]user +你能解释一下代码吗[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]assistant +我使用的是Python的Plotly库来创建一个极坐标图，以展示风向与降雨量之间的关系。代码中首先创建了一个Plotly的Figure对象。接着，我添加了两个`Scatterpolar`追踪。第一个追踪展示的是上午9点的风向与降雨量的平均值的关系，第二个追踪展示的是下午3点的风向与降雨量的平均值的关系。在这两个追踪中，`r`参数代表降雨量的平均值，`theta`参数代表风向。`fill='toself'`表示图形以自身为边界进行填充。然后我更新了图表的布局，隐藏了径向轴，添加了图例，设置了标题和标题字体，选择了暗色主题，并设定了图表的背景色、宽度和高度。最后，使用`fig.show()`展示了图表。 +[UNUSED_TOKEN_145] +``` + +1. 首先在系统提示中提供代码解释器的格式和字段描述。内容以 `[UNUSED_TOKEN_146]system name=[UNUSED_TOKEN_142]\n`开头，`[UNUSED_TOKEN_145]` 结尾，`name=[UNUSED_TOKEN_142]` 体现了这是来自代码解释器的指令。InternLM2-Chat 支持 system 角色对模型的提示和约束多次出现。所以我们会看到前面还有关于对话的要求。 +2. 用户可以上传一个文件，并对模型提出要求，文件的上传会以单独的形式向模型发出一条指令，以 `[UNUSED_TOKEN_146]user name=file` 开头，以 json 形式给出路径和文件大小` +[{"path": "data.csv", size='10K'}]`，以 `[UNUSED_TOKEN_145]`结尾。 +2. 模型在接受到用户指令后，会以流式的形式调用工具，及自然地生成文字进行思考/回应用户，然后输出`[UNUSED_TOKEN_144][UNUSED_TOKEN_142]`。`[UNUSED_TOKEN_144]`表示要调用外部插件，同时 `[UNUSED_TOKEN_142]` 表示调用的是代码解释器。然后模型输出 markdown 中 python 代码块格式代码内容，再以 `[UNUSED_TOKEN_143]` 表示工具调用结束。 +3. 系统会执行代码块中的代码，然后返回调用结果，以 `[UNUSED_TOKEN_146]environment name=[UNUSED_TOKEN_142]`开头，表示是来自环境关于代码解释器执行的输出，以`[UNUSED_TOKEN_145]`结尾。 + +### 同时使用工具和代码解释器 + +InternLM2-Chat 能够在一个对话过程中自主选择调用工具或代码解释器。在工具和代码解释器同时开启的情况下，只需要将各自的系统提示合并在一起给模型即可。一个调用工具和代码解释器的对话历史样例如下。 ``` [UNUSED_TOKEN_146]system @@ -102,7 +212,11 @@ fig.show() [UNUSED_TOKEN_145] [UNUSED_TOKEN_146]user 我想了解今天上海的天气[UNUSED_TOKEN_145] -[UNUSED_TOKEN_144][UNUSED_TOKEN_141] -{"name": "get_current_weather", "parameters": {"location": "上海"}}[UNUSED_TOKEN_143] -[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]assistant +好的，我将为你查询上海的天气。[UNUSED_TOKEN_144][UNUSED_TOKEN_141] +{"name": "get_current_weather", "parameters": {"location": "Shanghai"}}[UNUSED_TOKEN_143][UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]environment name=[UNUSED_TOKEN_141] +{"temperature": 22}[UNUSED_TOKEN_145] +[UNUSED_TOKEN_146]assistant +上海的天气是 22 摄氏度[UNUSED_TOKEN_145] ```