Welcome to iMIX’s documentation!

Get Started

• Get started
  • Prerequisites
  • Installation
  • Verification
• Benchmark and Model Zoo
  • Baselines

Quick Run

• 1: Inference and train with existing models and datasets
  • Inference with existing models
  • Test existing models on existing datasets
    • Prepare datasets
    • Prepare model checkpoint
    • Test existing models
  • Train predefined models on existing datasets
    • Prepare dataset
    • Training on single node
    • Training on multiple nodes
• 2: Train with customized datasets
  • Prepare the customized dataset
    • Download dataset files
    • Prepare the dataset config file
    • Add dataset process module
  • Prepare other necessary configs
  • Train model
  • Test and inference
• 3: Train with customized models and existing datasets
  • Prepare dataset
  • Prepare customized model
    • Define a new combine_model (e.g. BranchCombineLayer)
    • Import the module
    • Modify the config file
  • Prepare necessary configs
  • Train model
  • Test and inference

TUTORIALS

• Tutorial: Engine
• Tutorial 1: Learn about Configs
  • The configuration
    • The script configuration
    • The _base_ config file
  • Configuration Reading Process
  • FAQ
    • Use common variables in configs
• Tutorial 2: Customize Datasets
  • Customize dataset in source code
    • Download dataset files
      • Prepare the dataset config file
      • Dataset process pipeline
  • Customize dataset in iMIX
    • Download dataset files
    • Prepare the dataset config file
      • An example of data configuration
    • Dataset process pipeline
      • add dataset_loader.py
      • add dataset_reader.py
      • add dataset_infocpler.py
• Tutorial 3: Customize Models
  • Customize model in source code
  • Customize model in iMIX style
    • Build up model
      • Customize the model.py
    • Develop new components
      • Add a new embedding
      • Add a new encoder
      • Add a new backbone
      • Add new combine_model
      • Add new heads
      • Add new loss
    • An example of customized model config
• Tutorial 4: Customize Schedule and Runtime Settings
  • Customize optimization schedules
    • Customize optimizer supported by Pytorch
    • Customize self-implemented optimizer
    • Additional settings
  • Customize FP16 schedules
  • Customize LR schedules
  • Customize train mode
  • Modify default runtime settings
    • Checkpoint config
    • Log config
    • Additional settings
• Tutorial 5: Customize hooks
  • Customize self-implemented hooks
• Tutorial 6: Finetuning Models
  • Modify dataset
  • Modify head
  • Modify training schedule
  • Set base configs
  • Use pre-trained model

Useful Tools

• Log Visualization
  • Output Runs
  • Visualize Runs

API Reference

• imix.engine package
  • imix.engine.hooks package
    • Subpackages
      • imix.engine.hooks.periods package
        • Submodules
        • imix.engine.hooks.periods.checkpoint module
        • imix.engine.hooks.periods.common_metric module
        • imix.engine.hooks.periods.json_logger module
        • imix.engine.hooks.periods.log_buffer_imix module
        • imix.engine.hooks.periods.tensorboard_logger module
        • Module contents
    • Submodules
    • imix.engine.hooks.base_hook module
    • imix.engine.hooks.builder module
    • imix.engine.hooks.ema module
    • imix.engine.hooks.evaluate module
    • imix.engine.hooks.iteration_time module
    • imix.engine.hooks.lr_scheduler module
    • imix.engine.hooks.momentum_scheduler module
    • imix.engine.hooks.optimizer module
    • imix.engine.hooks.periodic_logger module
    • imix.engine.hooks.text_logger_hook module
    • Module contents
  • imix.engine.base_engine module
  • imix.engine.imix_engine module
  • imix.engine.organizer module
• imix.data package
  • imix.data.infocomp package
    • Submodules
    • imix.data.infocomp.base_infocpler module
    • imix.data.infocomp.clevr_infocpler module
    • imix.data.infocomp.gqa_infocpler module
    • imix.data.infocomp.hatefulememes_infocpler module
    • imix.data.infocomp.lxmertpretrain_infocpler module
    • imix.data.infocomp.ocrvqa_infocpler module
    • imix.data.infocomp.refclef_infocpler module
    • imix.data.infocomp.refcoco2_infocpler module
    • imix.data.infocomp.refcoco_infocpler module
    • imix.data.infocomp.refcocog_infocpler module
    • imix.data.infocomp.refcocop_infocpler module
    • imix.data.infocomp.referit_infocpler module
    • imix.data.infocomp.stvqa_infocpler module
    • imix.data.infocomp.textvqa_infocpler module
    • imix.data.infocomp.vcr_infocpler module
    • imix.data.infocomp.visual_dialog_infocpler module
    • imix.data.infocomp.visualentailment_infocpler module
    • imix.data.infocomp.vizwiz_infocpler module
    • imix.data.infocomp.vqa_infocpler module
    • Module contents
  • imix.data.loaders package
    • Submodules
    • imix.data.loaders.base_loader module
    • imix.data.loaders.clevr_loader module
    • imix.data.loaders.gqa_loader module
    • imix.data.loaders.hatefulmemes_loader module
    • imix.data.loaders.ocrvqa_loader module
    • imix.data.loaders.refclef_loader module
    • imix.data.loaders.refcoco_loader module
    • imix.data.loaders.refcocog_loader module
    • imix.data.loaders.refcocop_loader module
    • imix.data.loaders.referit_loader module
    • imix.data.loaders.stvqa_loader module
    • imix.data.loaders.textvqa_loader module
    • imix.data.loaders.vcr_loader module
    • imix.data.loaders.visual_dialog_dataset module
    • imix.data.loaders.visualentailment_loader module
    • imix.data.loaders.vizwiz_loader module
    • imix.data.loaders.vqa_loader module
    • imix.data.loaders.vqa_loader_uniter module
    • Module contents
  • imix.data.reader package
    • Subpackages
      • imix.data.reader.feature_reader package
        • Submodules
        • imix.data.reader.feature_reader.feature_reader module
        • imix.data.reader.feature_reader.image_features_reader module
        • imix.data.reader.feature_reader.lmdb_feature_reader module
        • imix.data.reader.feature_reader.pth_feature_reader module
        • Module contents
    • Submodules
    • imix.data.reader.annotation_reader module
    • imix.data.reader.base_reader module
    • imix.data.reader.clevr_reader module
    • imix.data.reader.feature_base_data module
    • imix.data.reader.gqa_reader module
    • imix.data.reader.hatefulmemes_reader module
    • imix.data.reader.lxmertpretrain_reader module
    • imix.data.reader.ocrvqa_reader module
    • imix.data.reader.refclef_reader module
    • imix.data.reader.refcoco2_reader module
    • imix.data.reader.refcoco_reader module
    • imix.data.reader.refcocog_reader module
    • imix.data.reader.refcocop_reader module
    • imix.data.reader.referit_reader module
    • imix.data.reader.stvqa_reader module
    • imix.data.reader.textvqa_reader module
    • imix.data.reader.vcr_reader module
    • imix.data.reader.visual_dialog_reader module
    • imix.data.reader.visualentailment_reader module
    • imix.data.reader.vizwiz_reader module
    • imix.data.reader.vqa_reader module
    • imix.data.reader.vqa_reader_uniter module
    • Module contents
  • imix.data.sampler package
    • Submodules
    • imix.data.sampler.builder module
    • imix.data.sampler.sampler_adaptor module
    • imix.data.sampler.uniter_sampler module
    • Module contents
  • imix.data.utils package
    • Submodules
    • imix.data.utils.common module
    • imix.data.utils.data_utils module
    • imix.data.utils.file_utils module
    • imix.data.utils.image_utils module
    • imix.data.utils.io module
    • imix.data.utils.phoc module
    • imix.data.utils.stream module
    • imix.data.utils.tokenization module
    • imix.data.utils.vcr_utils module
    • Module contents
  • imix.data.vqadata package
    • Submodules
    • imix.data.vqadata.answerprocessor module
    • imix.data.vqadata.baseprocessor module
    • imix.data.vqadata.bboxprocessor module
    • imix.data.vqadata.dataset_test module
    • imix.data.vqadata.fasttextprocessor module
    • imix.data.vqadata.processor module
    • imix.data.vqadata.stream module
    • imix.data.vqadata.vocabprocessor module
    • imix.data.vqadata.vqa_dataset module
    • Module contents
  • imix.data.builder module
• imix.evaluation package
  • Submodules
  • imix.evaluation.dataset_evaluator module
  • imix.evaluation.evaluator_imix module
  • imix.evaluation.metric module
  • Module contents
• imix.models package
  • imix.models.backbones package
    • Submodules
    • imix.models.backbones.ban module
    • imix.models.backbones.cagraph_backbone module
    • imix.models.backbones.cmrin_backbone module
    • imix.models.backbones.hgl_backbone module
    • imix.models.backbones.imageembedding module
    • imix.models.backbones.lcgn_backbone module
    • imix.models.backbones.mmt module
    • imix.models.backbones.r2c_backbone module
    • imix.models.backbones.resc_backbone module
    • imix.models.backbones.twobranchembedding module
    • imix.models.backbones.visdial_principles_backbone module
    • Module contents
  • imix.models.combine_layers package
    • Submodules
    • imix.models.combine_layers.branchcombinelayers module
    • imix.models.combine_layers.modalcombinelayer module
    • Module contents
  • imix.models.embedding package
    • Submodules
    • imix.models.embedding.bertimgembedding module
    • imix.models.embedding.textembedding module
    • imix.models.embedding.visual_dialog_embedding module
    • imix.models.embedding.vocab module
    • imix.models.embedding.wordembedding module
    • Module contents
  • imix.models.encoder package
    • Submodules
    • imix.models.encoder.imageencoder module
    • imix.models.encoder.lcgnencoder module
    • imix.models.encoder.oscar module
    • imix.models.encoder.textbert module
    • imix.models.encoder.uniter module
    • imix.models.encoder.visdiag_lstm module
    • imix.models.encoder.visualbert module
    • Module contents
  • imix.models.heads package
    • Submodules
    • imix.models.heads.classifier module
    • imix.models.heads.classifier_mix module
    • imix.models.heads.decoder module
    • imix.models.heads.decoder_mix module
    • Module contents
  • imix.models.losses package
    • Submodules
    • imix.models.losses.base_loss module
    • imix.models.losses.diverse_loss module
    • imix.models.losses.triple_logit_binary_cross_entropy module
    • imix.models.losses.yolo_loss module
    • imix.models.losses.yolo_loss_v2 module
    • Module contents
  • imix.models.visual_dialog_model package
    • Submodules
    • imix.models.visual_dialog_model.vilbert_dialog module
    • Module contents
  • imix.models.vqa_models package
    • Subpackages
      • imix.models.vqa_models.devlbert package
        • Subpackages
          • imix.models.vqa_models.devlbert.datasets package
            • Submodules
            • imix.models.vqa_models.devlbert.datasets.concept_cap_dataset module
            • imix.models.vqa_models.devlbert.datasets.refer_expression_dataset module
            • imix.models.vqa_models.devlbert.datasets.retreival_dataset module
            • imix.models.vqa_models.devlbert.datasets.vcr_dataset module
            • imix.models.vqa_models.devlbert.datasets.vqa_dataset module
            • Module contents
        • Submodules
        • imix.models.vqa_models.devlbert.devlbert module
        • imix.models.vqa_models.devlbert.devlbert_tasks module
        • imix.models.vqa_models.devlbert.postprocess_evaluator module
        • imix.models.vqa_models.devlbert.task_utils module
        • Module contents
      • imix.models.vqa_models.lxmert package
        • Subpackages
          • imix.models.vqa_models.lxmert.datasets package
            • Submodules
            • imix.models.vqa_models.lxmert.datasets.lxmert_gqa module
            • imix.models.vqa_models.lxmert.datasets.lxmert_nlvr2 module
            • imix.models.vqa_models.lxmert.datasets.lxmert_vqa module
            • Module contents
        • Submodules
        • imix.models.vqa_models.lxmert.lxmert module
        • imix.models.vqa_models.lxmert.lxmert_qa_answer_table module
        • imix.models.vqa_models.lxmert.lxmert_task module
        • imix.models.vqa_models.lxmert.postprocess_evaluator module
        • Module contents
      • imix.models.vqa_models.oscar package
        • Subpackages
          • imix.models.vqa_models.oscar.datasets package
            • Submodules
            • imix.models.vqa_models.oscar.datasets.build module
            • imix.models.vqa_models.oscar.datasets.dataset_utils module
            • imix.models.vqa_models.oscar.datasets.gqa_dataset module
            • imix.models.vqa_models.oscar.datasets.nlvr2_dataset module
            • imix.models.vqa_models.oscar.datasets.oscar_tsv module
            • imix.models.vqa_models.oscar.datasets.vqa_dataset module
            • Module contents
          • imix.models.vqa_models.oscar.modeling package
            • Submodules
            • imix.models.vqa_models.oscar.modeling.modeling_bert module
            • imix.models.vqa_models.oscar.modeling.modeling_utils module
            • Module contents
          • imix.models.vqa_models.oscar.utils package
            • Submodules
            • imix.models.vqa_models.oscar.utils.caption_evaluate module
            • imix.models.vqa_models.oscar.utils.cbs module
            • imix.models.vqa_models.oscar.utils.logger module
            • imix.models.vqa_models.oscar.utils.misc module
            • imix.models.vqa_models.oscar.utils.task_utils module
            • imix.models.vqa_models.oscar.utils.tsv_file module
            • imix.models.vqa_models.oscar.utils.tsv_file_ops module
            • Module contents
        • Submodules
        • imix.models.vqa_models.oscar.postprocess_evaluator module
        • imix.models.vqa_models.oscar.run_gqa module
        • imix.models.vqa_models.oscar.run_nlvr module
        • imix.models.vqa_models.oscar.run_vqa module
        • Module contents
      • imix.models.vqa_models.uniter package
        • Subpackages
          • imix.models.vqa_models.uniter.datasets package
            • Submodules
            • imix.models.vqa_models.uniter.datasets.data module
            • imix.models.vqa_models.uniter.datasets.itm module
            • imix.models.vqa_models.uniter.datasets.loader module
            • imix.models.vqa_models.uniter.datasets.mlm module
            • imix.models.vqa_models.uniter.datasets.mrm module
            • imix.models.vqa_models.uniter.datasets.nlvr2 module
            • imix.models.vqa_models.uniter.datasets.pretrain_vcr module
            • imix.models.vqa_models.uniter.datasets.re module
            • imix.models.vqa_models.uniter.datasets.vcr module
            • imix.models.vqa_models.uniter.datasets.ve module
            • imix.models.vqa_models.uniter.datasets.vqa module
            • Module contents
          • imix.models.vqa_models.uniter.model package
            • Submodules
            • imix.models.vqa_models.uniter.model.attention module
            • imix.models.vqa_models.uniter.model.itm module
            • imix.models.vqa_models.uniter.model.layer module
            • imix.models.vqa_models.uniter.model.model module
            • imix.models.vqa_models.uniter.model.nlvr2 module
            • imix.models.vqa_models.uniter.model.ot module
            • imix.models.vqa_models.uniter.model.pretrain module
            • imix.models.vqa_models.uniter.model.pretrain_vcr module
            • imix.models.vqa_models.uniter.model.re module
            • imix.models.vqa_models.uniter.model.vcr module
            • imix.models.vqa_models.uniter.model.ve module
            • imix.models.vqa_models.uniter.model.vqa module
            • Module contents
        • Submodules
        • imix.models.vqa_models.uniter.misc module
        • imix.models.vqa_models.uniter.postprocess_evaluator module
        • imix.models.vqa_models.uniter.train_nlvr2 module
        • imix.models.vqa_models.uniter.train_vcr module
        • imix.models.vqa_models.uniter.train_ve module
        • imix.models.vqa_models.uniter.train_vqa module
        • Module contents
      • imix.models.vqa_models.vilbert package
        • Subpackages
          • imix.models.vqa_models.vilbert.datasets package
            • Submodules
            • imix.models.vqa_models.vilbert.datasets.concept_cap_dataset module
            • imix.models.vqa_models.vilbert.datasets.flickr_grounding_dataset module
            • imix.models.vqa_models.vilbert.datasets.foil_dataset module
            • imix.models.vqa_models.vilbert.datasets.gqa_dataset module
            • imix.models.vqa_models.vilbert.datasets.guesswhat_dataset module
            • imix.models.vqa_models.vilbert.datasets.guesswhat_pointing_dataset module
            • imix.models.vqa_models.vilbert.datasets.nlvr2_dataset module
            • imix.models.vqa_models.vilbert.datasets.refer_dense_caption module
            • imix.models.vqa_models.vilbert.datasets.refer_expression_dataset module
            • imix.models.vqa_models.vilbert.datasets.retreival_dataset module
            • imix.models.vqa_models.vilbert.datasets.vcr_dataset module
            • imix.models.vqa_models.vilbert.datasets.visdial_dataset module
            • imix.models.vqa_models.vilbert.datasets.vismadlibs_dataset module
            • imix.models.vqa_models.vilbert.datasets.visual7w_pointing_dataset module
            • imix.models.vqa_models.vilbert.datasets.visual_entailment_dataset module
            • imix.models.vqa_models.vilbert.datasets.visual_genome_dataset module
            • imix.models.vqa_models.vilbert.datasets.vqa_dataset module
            • imix.models.vqa_models.vilbert.datasets.vqa_mc_dataset module
            • Module contents
        • Submodules
        • imix.models.vqa_models.vilbert.postprocess_evaluator module
        • imix.models.vqa_models.vilbert.task_utils module
        • imix.models.vqa_models.vilbert.utils module
        • imix.models.vqa_models.vilbert.vilbert module
        • imix.models.vqa_models.vilbert.vilbert_tasks module
        • Module contents
    • Submodules
    • imix.models.vqa_models.ban module
    • imix.models.vqa_models.base_model module
    • imix.models.vqa_models.cagraph module
    • imix.models.vqa_models.cmrin module
    • imix.models.vqa_models.hgl module
    • imix.models.vqa_models.lcgn module
    • imix.models.vqa_models.lorra module
    • imix.models.vqa_models.m4c module
    • imix.models.vqa_models.mcan_mix module
    • imix.models.vqa_models.pythia module
    • imix.models.vqa_models.r2c module
    • imix.models.vqa_models.resc module
    • imix.models.vqa_models.visdial_principles module
    • imix.models.vqa_models.visual_dialog_bert module
    • imix.models.vqa_models.visualbert module
    • Module contents
  • imix.models.builder module
• imix.solver package
  • imix.solver.builder module
  • imix.solver.default_constructor module
  • imix.solver.lr_scheduler module
  • imix.solver.optimization module
• imix.utils package
  • imix.utils.Timer module
  • imix.utils.checkpoint module
  • imix.utils.collect_running_env module
  • imix.utils.common_function module
  • imix.utils.config module
  • imix.utils.data_utils module
  • imix.utils.default_argument module
  • imix.utils.disable_ouput module
  • imix.utils.distributed_info module
  • imix.utils.download module
  • imix.utils.enable_ouput module
  • imix.utils.file_io module
  • imix.utils.history_buffer module
  • imix.utils.imix_checkpoint module
  • imix.utils.launch module
  • imix.utils.logger module
  • imix.utils.modeling module
  • imix.utils.registry module
  • imix.utils.text module

Indices and tables

• Index

• Module Index

• Search Page