You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Frank Lee 3defa32aee Support TP-compatible Torch AMP and Update trainer API (#27) ... * Add gradient accumulation, fix lr scheduler * fix FP16 optimizer and adapted torch amp with tensor parallel (#18) * fixed bugs in compatibility between torch amp and tensor parallel and performed some minor fixes * fixed trainer * Revert "fixed trainer" This reverts commit 2e0b0b7699. * improved consistency between trainer, engine and schedule (#23) Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: 1SAA <c2h214748@gmail.com> Co-authored-by: ver217 <lhx0217@gmail.com>		3 years ago
..
colossalai.builder.builder.rst	Migrated project	3 years ago
colossalai.builder.pipeline.rst	Migrated project	3 years ago
colossalai.builder.rst	Migrated project	3 years ago
colossalai.checkpointing.rst	Migrated project	3 years ago
colossalai.communication.collective.rst	Migrated project	3 years ago
colossalai.communication.p2p.rst	Migrated project	3 years ago
colossalai.communication.ring.rst	Migrated project	3 years ago
colossalai.communication.rst	Migrated project	3 years ago
colossalai.communication.utils.rst	Migrated project	3 years ago
colossalai.constants.rst	Migrated project	3 years ago
colossalai.context.config.rst	Migrated project	3 years ago
colossalai.context.parallel_context.rst	Migrated project	3 years ago
colossalai.context.parallel_mode.rst	Migrated project	3 years ago
colossalai.context.process_group_initializer.initializer_1d.rst	Migrated project	3 years ago
colossalai.context.process_group_initializer.initializer_2d.rst	Migrated project	3 years ago
colossalai.context.process_group_initializer.initializer_2p5d.rst	Migrated project	3 years ago
colossalai.context.process_group_initializer.initializer_3d.rst	Migrated project	3 years ago
colossalai.context.process_group_initializer.initializer_data.rst	Migrated project	3 years ago
colossalai.context.process_group_initializer.initializer_pipeline.rst	Migrated project	3 years ago
colossalai.context.process_group_initializer.initializer_sequence.rst	Migrated project	3 years ago
colossalai.context.process_group_initializer.initializer_tensor.rst	Migrated project	3 years ago
colossalai.context.process_group_initializer.process_group_initializer.rst	Migrated project	3 years ago
colossalai.context.process_group_initializer.rst	Migrated project	3 years ago
colossalai.context.random.rst	Migrated project	3 years ago
colossalai.context.random.seed_manager.rst	Migrated project	3 years ago
colossalai.context.rst	Migrated project	3 years ago
colossalai.core.rst	Migrated project	3 years ago
colossalai.engine.amp.amp_type.rst	Support TP-compatible Torch AMP and Update trainer API (#27)	3 years ago
colossalai.engine.amp.grad_scaler.rst	Support TP-compatible Torch AMP and Update trainer API (#27)	3 years ago
colossalai.engine.amp.rst	Support TP-compatible Torch AMP and Update trainer API (#27)	3 years ago
colossalai.engine.gradient_handler.rst	Migrated project	3 years ago
colossalai.engine.rst	Support TP-compatible Torch AMP and Update trainer API (#27)	3 years ago
colossalai.engine.schedule.rst	Migrated project	3 years ago
colossalai.initialize.rst	Migrated project	3 years ago
colossalai.logging.logging.rst	Migrated project	3 years ago
colossalai.logging.rst	Migrated project	3 years ago
colossalai.nn.data.base_dataset.rst	Migrated project	3 years ago
colossalai.nn.data.caltech101_dataset.rst	Migrated project	3 years ago
colossalai.nn.data.cifar10_dataset.rst	Migrated project	3 years ago
colossalai.nn.data.rst	update documentation	3 years ago
colossalai.nn.data.sampler.base_sampler.rst	Migrated project	3 years ago
colossalai.nn.data.sampler.data_parallel_sampler.rst	Migrated project	3 years ago
colossalai.nn.data.sampler.rst	Migrated project	3 years ago
colossalai.nn.layer.base_layer.rst	Migrated project	3 years ago
colossalai.nn.layer.parallel_1d.layers.rst	Migrated project	3 years ago
colossalai.nn.layer.parallel_1d.rst	Migrated project	3 years ago
colossalai.nn.layer.parallel_2d.layers.rst	Migrated project	3 years ago
colossalai.nn.layer.parallel_2d.rst	Migrated project	3 years ago
colossalai.nn.layer.parallel_2p5d.layers.rst	Migrated project	3 years ago
colossalai.nn.layer.parallel_2p5d.rst	Migrated project	3 years ago
colossalai.nn.layer.parallel_3d.layers.rst	Migrated project	3 years ago
colossalai.nn.layer.parallel_3d.rst	Migrated project	3 years ago
colossalai.nn.layer.parallel_sequence.layers.rst	Migrated project	3 years ago
colossalai.nn.layer.parallel_sequence.rst	Migrated project	3 years ago
colossalai.nn.layer.parallel_vision_transformer.layers.rst	Migrated project	3 years ago
colossalai.nn.layer.parallel_vision_transformer.rst	Migrated project	3 years ago
colossalai.nn.layer.rst	Migrated project	3 years ago
colossalai.nn.layer.vanilla_resnet.basic_block.rst	Migrated project	3 years ago
colossalai.nn.layer.vanilla_resnet.bottleneck.rst	Migrated project	3 years ago
colossalai.nn.layer.vanilla_resnet.conv.rst	Migrated project	3 years ago
colossalai.nn.layer.vanilla_resnet.reslayer.rst	Migrated project	3 years ago
colossalai.nn.layer.vanilla_resnet.rst	Migrated project	3 years ago
colossalai.nn.layer.vanilla_vision_transformer.layers.rst	Migrated project	3 years ago
colossalai.nn.layer.vanilla_vision_transformer.rst	Migrated project	3 years ago
colossalai.nn.layer.wrapper.lambda_wrapper.rst	Migrated project	3 years ago
colossalai.nn.layer.wrapper.rst	Migrated project	3 years ago
colossalai.nn.loss.base_loss.rst	Migrated project	3 years ago
colossalai.nn.loss.cross_entropy_1d.rst	Migrated project	3 years ago
colossalai.nn.loss.cross_entropy_2d.rst	Migrated project	3 years ago
colossalai.nn.loss.cross_entropy_2p5d.rst	Migrated project	3 years ago
colossalai.nn.loss.cross_entropy_3d.rst	Migrated project	3 years ago
colossalai.nn.loss.rst	Migrated project	3 years ago
colossalai.nn.lr_scheduler.cosine.rst	Migrated project	3 years ago
colossalai.nn.lr_scheduler.delayed.rst	Migrated project	3 years ago
colossalai.nn.lr_scheduler.linear.rst	Migrated project	3 years ago
colossalai.nn.lr_scheduler.multistep.rst	Migrated project	3 years ago
colossalai.nn.lr_scheduler.onecycle.rst	Migrated project	3 years ago
colossalai.nn.lr_scheduler.poly.rst	Migrated project	3 years ago
colossalai.nn.lr_scheduler.rst	Migrated project	3 years ago
colossalai.nn.lr_scheduler.torch.rst	Migrated project	3 years ago
colossalai.nn.model.base_model.rst	Migrated project	3 years ago
colossalai.nn.model.rst	update documentation	3 years ago
colossalai.nn.model.vanilla_resnet.resnet.rst	Migrated project	3 years ago
colossalai.nn.model.vanilla_resnet.rst	Migrated project	3 years ago
colossalai.nn.model.vision_transformer.rst	Migrated project	3 years ago
colossalai.nn.model.vision_transformer.vision_transformer.rst	Migrated project	3 years ago
colossalai.nn.multi_tensor_apply.multi_tensor_apply.rst	Migrated project	3 years ago
colossalai.nn.multi_tensor_apply.rst	Migrated project	3 years ago
colossalai.nn.optimizer.fp16_optimizer.rst	Migrated project	3 years ago
colossalai.nn.optimizer.fused_adam.rst	Migrated project	3 years ago
colossalai.nn.optimizer.fused_lamb.rst	Migrated project	3 years ago
colossalai.nn.optimizer.fused_sgd.rst	Migrated project	3 years ago
colossalai.nn.optimizer.lamb.rst	Migrated project	3 years ago
colossalai.nn.optimizer.lars.rst	update documentation	3 years ago
colossalai.nn.optimizer.loss_scaler.rst	Migrated project	3 years ago
colossalai.nn.optimizer.rst	update documentation	3 years ago
colossalai.nn.optimizer.zero_redundancy_optimizer_level_1.rst	Migrated project	3 years ago
colossalai.nn.optimizer.zero_redundancy_optimizer_level_2.rst	Migrated project	3 years ago
colossalai.nn.optimizer.zero_redundancy_optimizer_level_3.rst	Migrated project	3 years ago
colossalai.nn.rst	Migrated project	3 years ago
colossalai.registry.registry.rst	Migrated project	3 years ago
colossalai.registry.rst	Migrated project	3 years ago
colossalai.rst	Support TP-compatible Torch AMP and Update trainer API (#27)	3 years ago
colossalai.trainer.hooks.rst	Migrated project	3 years ago
colossalai.trainer.metric.rst	Migrated project	3 years ago
colossalai.trainer.rst	Migrated project	3 years ago
colossalai.utils.activation_checkpoint.rst	Migrated project	3 years ago
colossalai.utils.checkpointing.rst	Support TP-compatible Torch AMP and Update trainer API (#27)	3 years ago
colossalai.utils.common.rst	Migrated project	3 years ago
colossalai.utils.cuda.rst	Migrated project	3 years ago
colossalai.utils.memory.rst	Migrated project	3 years ago
colossalai.utils.rst	Support TP-compatible Torch AMP and Update trainer API (#27)	3 years ago
colossalai.utils.timer.rst	Migrated project	3 years ago