diff --git a/doc/aux_ materials/long_text_example.txt b/doc/aux_ materials/long_text_example.txt new file mode 100644 index 0000000..7fdfe2d --- /dev/null +++ b/doc/aux_ materials/long_text_example.txt @@ -0,0 +1 @@ +Please summarize the following passage:\n\nBy Alex Acquisto , BDN Staff • June 14, 2017 2:30 pm\n\nUpdated: June 22, 2017 1:24 pm\n\nHOPE, Maine — While jogging on a familiar, overgrown, wooded trail near her home on a recent warm afternoon, Rachel Borch thought to herself, “what a beautiful day.”\n\nLittle did she know she was about to be attacked by a rabid raccoon she would end up killing with her bare hands.\n\nIn the midst of appreciating the weather and scenery, she looked ahead and noticed a raccoon obstructing the narrow foot path, baring its tiny teeth.\n\nSuddenly, it began “bounding” toward her, Borch recalled Wednesday afternoon during an interview at her home on Hatchet Mountain Road in Hope.\n\n“I knew instantly it had to be rabid,” said Borch, who remembers ripping out her headphones and dropping her phone on the ground.\n\nWhat felt like a split second later, the furry animal was at her feet. Borch said she was “dancing around it,” trying to figure out what to do.\n\n“Imagine the Tasmanian devil,” she said. “It was terrifying.”\n\nThe path was too narrow for Borch to run past the raccoon, which had begun lunging at her. With adrenaline pumping, Borch suspended her disbelief.\n\n“I knew it was going to bite me,” she said.\n\nFiguring she would have the greatest ability to defend herself if she used her hands to hold it down, she decided that probably would be the best place for the aggressive animal to latch on.\n\nThe raccoon sank its teeth into Borch’s thumb and “wouldn’t let go.” Its paws were scratching her arms and legs wildly as Borch screamed and cried.\n\nIn a matter of seconds, Borch, who could not unhinge the raccoon’s jaw to shake it off her hand, noticed that when she had dropped her phone, it had fallen into a puddle in the path and was fully submerged.\n\n“I didn’t think I could strangle [the raccoon] with my bare hands,” she remembers thinking, but holding it under the water might do the trick.\n\nConnecting the dots quickly, Borch, then on her knees, dragged the still biting raccoon, which was scratching frantically at her hand and arms, into the puddle.\n\n“With my thumb in its mouth, I just pushed its head down into the muck,” Borch said.\n\nWith the animal belly-up, she held its head under water. “It was still struggling and clawing at my arms. It wouldn’t let go of my thumb,” she said.\n\nBorch said she held it there for what felt like an eternity until finally it stopped struggling and “its arms sort of of fell to the side, its chest still heaving really slowly.”\n\nHyperventilating and in hysterics, she pulled her thumb out of the raccoon’s mouth, “and then I just bolted as fast as I could through the underbrush,” she said.\n\nBorch remembers looking back once to see if the raccoon had started chasing her again.\n\n“It felt like [Stephen King’s] ‘Pet Sematary,’” she said.\n\nKicking her shoes off because they were soaked, Borch ran the three-quarters of a mile home to her house.\n\nBorch, who was screaming and unsure of how rabies affects humans, remembers thinking, “Oh, God, what if I just start foaming at the mouth and can’t find my way back?”\n\nShe met her mother, Elizabeth, at home, and together they drove immediately to Pen Bay Medical Center.\n\nThe dead raccoon was retrieved by Borch’s dad, who packed it into a Taste of the Wild dog food bag and handed it over to the Maine Warden Service.\n\nHope Animal Control Officer Heidi Blood confirmed Wednesday that the dead raccoon later tested positive for rabies by the Maine Center for Disease Control.\n\n“Not to scare people,” Blood said, but “when there’s one [infected], there’s typically another.”\n\nIt’s important to “let folks know that just because there’s one [infected] and it’s gone now, doesn’t mean the risk still isn’t there,” she said.\n\nInfected animals typically start showing signs within two weeks, Blood said. Humans can start exhibiting symptoms within a few weeks, she said, but often it takes a few months.\n\n“It’s scary stuff,” Blood said. “The No. 1 thing we try to remind people of is that it’s 100 percent fatal [if it goes untreated].”\n\nBorch has received six shots so far, including the rabies vaccine, and immunoglobulin and tetanus injections. She is slated to receive her last injection this weekend.\n\n“If there hadn’t been water on the ground, I don’t know what I would have done,” Borch said of drowning the animal. “It really was just dumb luck. I’ve never killed an animal with my bare hands. I’m a vegetarian. It was self-defense.”\n\nHer advice for others who find themselves facing a rabid animal? Borch said she has none.\n\n“I always thought of raccoons as this cute, cuddly forest animal,” she said. “I just will never look at them the same way.”\n\nBorch is not the only person to have been attacked by a rabid animal so far this season.\n\nEarlier this week, a Wiscasset man was bitten on both hands in Topsham by what was believed to be a rabid fox.\n\nAs of June 7, according to the Maine CDC, there have been 20 animals, including raccoons, red foxes and skunks, that have tested positive for rabies in 2017.\n\nIn 2016, 64 animals in Maine tested positive for rabies, according to CDC data.\n\nLoading… Passage 2: HOPE — A woman who was attacked by a rabid raccoon while running in the woods near her home drowned the animal in a puddle after it latched onto her thumb.\n\nRachel Borch, 21, said she was getting ready to go out for an afternoon run on June 3 when her brother Chris told her to be careful because he had seen a raccoon “skulking” around the yard. He told her he thought it odd that the raccoon, normally a nocturnal animal, was out in the daytime.\n\nBorch said she was running on a fire road next to her house when she encountered the animal.\n\n“Out of nowhere, I see through the underbrush a very ferocious-looking raccoon charging at me with its teeth bared,” Borch said.\n\nShe immediately knew something was wrong with the animal by the way it was acting.\n\n“It was one of those moments like out of the ‘Twilight Zone’ – this isn’t real, this doesn’t happen in real life, but then it was right there and it was right at my feet,” she said.\n\nBorch said she started dancing around the animal, frantically trying to figure out what to do.\n\n“There was nothing I could do, it was going to bite me,” she said. She had dropped her phone and had nothing to protect herself with.\n\nShe figured that if it was going to bite her, it might as well be her hands. She put her hands out in front of her, and the raccoon latched onto her thumb.\n\n“I was screaming and crying and trying to hold it down,” she said. “There was a few inches of really muddy water on the ground – it was a swampy area of the trail – so I just took all my strength and pushed it into the water.”\n\nShe said she pushed the raccoon’s head underwater and held it there for what seemed like a very long time as it clawed her arms and continued to bite down on her thumb.\n\n“It happened so fast, but also in slow motion,” Borch said.\n\nThe state’s Center for Disease Control and Prevention confirmed June 5 that the raccoon tested positive for rabies.\n\nRabies is caused by a virus, affects the brain and spinal cord and can cause death if left untreated, according to the Maine CDC. Rabies in people is very rare in the United States, but rabies in animals – especially wildlife – is common in most parts of the country, including Maine.\n\nThe raccoon continued to move for a while while Borch kept its head underwater. She said she was afraid to let go for fear it would continue attacking her. Eventually, it released its paws and its jaws stopped clenching her thumb.\n\nBleeding and crying, Borch ran home and screamed for her mother to call 911. Borch’s father, Brad, and her brother retrieved the animal so another animal wouldn’t drag it off and become infected.\n\nHope animal control officer Heidi Blood said that if one animal tests positive for rabies, it’s “almost 100 percent” certain there are other animals in the area also infected. Blood said that last summer, two raccoons in Lincolnville and one in Hope tested positive for rabies.\n\nBlood said an animal doesn’t have to be aggressive to be infected; it can also act “delirious or drunklike.”\n\nShe said people and their pets should avoid such animals, even dead ones, because the rabies virus can live outside the body for 24 hours.\n\nUnder Maine law, cats and dogs, even if they stay indoors, must be vaccinated against the disease.\n\nBorch is expected to receive the last of four rabies shots on Saturday, and has been taking antibiotics for the puncture wounds on her hands.\n\n“You just can’t predict something like that,” Borch said. “I’m still processing it, but that does not happen and that is not a normal thing.”\n\nShare\n\nWant the news vital to Maine? Our daily headlines email is delivered each morning. Email *\n\nNewsletter Choices * Daily Headlines and Evening Express Breaking News Business Headlines Maine Cannabis Report High School Sports Real Estate\n\n* I understand the Terms of Service.\n\nName This field is for validation purposes and should be left unchanged.\n\nThis iframe contains the logic required to handle Ajax powered Gravity Forms. diff --git a/doc/imgs/dynamic_ntk_answer.png b/doc/imgs/dynamic_ntk_answer.png new file mode 100644 index 0000000..c582cfe Binary files /dev/null and b/doc/imgs/dynamic_ntk_answer.png differ diff --git a/doc/usage.md b/doc/usage.md index 347ca35..aa7572f 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -371,33 +371,17 @@ $ torchrun --nnodes=1 --nproc_per_node=8 train.py --config ./configs/7B_sft.py - ### 长文本生成 -在推理阶段，您可以在模型配置中通过设置 `use_dynamic_ntk_rope=True` 开启 RoPE 的 Dynamic NTK 选项，从而使得模型适应长文本输入输出，达到 16K 的外推效果: -```python #21 -model_type = "INTERNLM" # 模型类型，默认值为 "INTERNLM"，对应模型结构初始化接口函数 -NUM_ATTENTION_HEAD = 32 -VOCAB_SIZE = 103168 -HIDDEN_SIZE = 4096 -NUM_LAYER = 32 -MLP_RATIO = 8 / 3 -model = dict( - checkpoint=False, # 进行重计算的模型层数比例，可选值为 True/False/[0-1] - num_attention_heads=NUM_ATTENTION_HEAD, - embed_split_hidden=True, - vocab_size=VOCAB_SIZE, - embed_grad_scale=1, - parallel_output=True, - hidden_size=HIDDEN_SIZE, - num_layers=NUM_LAYER, - mlp_ratio=MLP_RATIO, - apply_post_layer_norm=False, - dtype="torch.bfloat16", - norm_type="rmsnorm", - layer_norm_epsilon=1e-5, - use_dynamic_ntk_rope=True -) -``` +在推理阶段，我们可以使用 Dynamic NTK RoPE 来代替原始的 RoPE，从而使得模型能够适应长文本的输入输出，达到 16K 的外推效果。 +目前 InternLM 支持在 huggingface 格式和 InternLM 本身格式的模型中使用 Dynamic NTK RoPE。 + +1. 对于 huggingface 格式的模型，dynamic ntk rope 目前是被默认使用的。如果用户想要关闭该行为，请将 `config.json` 中的 `rotary.type` 修改为 `origin`； +2. 对于 InternLM 本身格式的模型，在推理时，通过在初始化模型的配置字典中添加`use_dynamic_ntk_rope=True`来开启这一行为。 + +用户可以直接通过 web_demo 来直观地对比查看 Dynamic NTK RoPE 是如何生效的。例如文件[长文本示例](./aux_%20materials/long_text_example.txt)中存放着一个token长度超过2200的文本，如果不使用 Dynamic NTK， +模型是完全无法回答该文本对应的问题。而使用 Dynamic NTK RoPE 后 InternLM Chat 7B v1.1 模型的回答如下所示： + +![dynamic_ntk_answer](./imgs/dynamic_ntk_answer.png) 关于 Dyanmic NTK 的原理，详细请参考 - 1. https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases 2. https://kexue.fm/archives/9675