InternLM/agent/README.md

# InternLM-Chat Agent

English | [简体中文](README_zh-CN.md)

## Introduction

InternLM-Chat-7B v1.1 has been released as the first open-source model with code interpreter capabilities, supporting external tools such as Python code interpreter and search engine.

InternLM2-Chat, open sourced on January 17, 2024, further enhances its capabilities in code interpreter and general tool utilization. With improved and more generalized instruction understanding, tool selection, and reflection abilities, InternLM2-Chat can more reliably support complex agents and multi-step tool calling for more intricate tasks. InternLM2-Chat exhibits decent computational and reasoning abilities even without external tools, surpassing ChatGPT in mathematical performance. When combined with a code interpreter, InternLM2-Chat-20B obtains comparable results to GPT-4 on GSM8K and MATH. Leveraging strong foundational capabilities in mathematics and tools, InternLM2-Chat provides practical data analysis capabilities.

The results of InternLM2-Chat-20B on math code interpreter is as below:

|                                          | GSM8K | MATH  |
| :--------------------------------------: | :---: | :---: |
|            InternLM2-Chat-20B            | 79.6  | 32.5  |
| InternLM2-Chat-20B with Code Interpreter | 84.5  | 51.2  |
|            ChatGPT (GPT-3.5)             | 78.2  | 28.0  |
|                  GPT-4                   | 91.4  | 45.8  |

## Usages

We offer an example using [Lagent](lagent.md) to build agents based on InternLM2-Chat to call the code interpreter. Firstly install the extra dependencies:

```bash
pip install -r requirements.txt
```

Run the following script to perform inference and evaluation on GSM8K and MATH test.

```bash
python streaming_inference.py \
  --backend=lmdeploy \  # For HuggingFace models: hf
  --model_path=internlm/internlm2-chat-20b \
  --tp=2 \
  --temperature=1.0 \
  --top_k=1 \
  --dataset=math \
  --output_path=math_lmdeploy.jsonl \
  --do_eval
```

`output_path` is a jsonl format file to save the inference results. Each line is like

```json
{
    "idx": 41, 
    "query": "The point $(a, b)$ lies on the line with the equation $3x + 2y = 12.$ When $a = 4$, what is the value of $b$?",
    "gt": "0",
    "pred": ["0"],
    "steps": [
        {
            "role": "language",
            "content": ""
        },
        {
            "role": "tool",
            "content": {
                "name": "IPythonInteractive",
                "parameters": {
                    "command": "```python\nfrom sympy import symbols, solve\n\ndef find_b():\n    x, y = symbols('x y')\n    equation = 3*x + 2*y - 12\n    b = solve(equation.subs(x, 4), y)[0]\n\n    return b\n\nresult = find_b()\nprint(result)\n```"
                }
            },
            "name": "interpreter"
        },
        {
            "role": "environment",
            "content": "0",
            "name": "interpreter"
        },
        {
            "role": "language",
            "content": "The value of $b$ when $a = 4$ is $\\boxed{0}$."
        }
    ],
    "error": null
}
```

Once it is prepared, just skip the inference stage as follows.

```bash
python streaming_inference.py \
  --output_path=math_lmdeploy.jsonl \
  --no-do_infer \
  --do_eval
```

Please refer to [`streaming_inference.py`](streaming_inference.py) for more information about the arguments.