Utils

SaveLoadMixin

class SaveLoadMixin

save_finetuned(finetuned_delta_path: ~typing.Optional[~typing.Union[str, ~os.PathLike]] = './delta_checkpoints/', save_config: bool = True, state_dict: ~typing.Optional[dict] = None, save_function: ~typing.Callable = <function save>, push_to_dc: bool = False, center_args: ~typing.Optional[~typing.Union[~opendelta.utils.saving_loading_utils.DeltaCenterArguments, dict]] = {}, center_args_pool: ~typing.Optional[dict] = {}, list_tags: ~typing.Optional[~typing.List] = [], dict_tags: ~typing.Optional[~typing.Dict] = {}, delay_push: bool = False, test_result=None, usage: ~typing.Optional[str] = '')

Save a model and its configuration file to a directory, so that it can be re-loaded using the save_finetuned() class method.

Parameters

• finetuned_delta_path – (optional) path to the directory where the model and its configuration file will be saved. If not specified, the model will be saved in the directory ./delta_checkpoints/, which is a subdirectory of the current working directory.

• save_config – (optional) if True, the configuration file will be saved in the same directory as the model file. if False, only the state dict will be saved.

• state_dict – (optional) a dictionary containing the model’s state_dict. If not specified, the state_dict is loaded from the backbone model’s trainable parameters.

• save_function – (optional) the function used to save the model. Defaults to torch.save.

• state_dict_only – (optional) if True, only the state_dict will be saved.

• push_to_dc – (optional) if True, the model will prepare things to pushed to the DeltaCenter. This includes: - creating a configuration file for the model - creating a directory for the model - saving the model’s trainable parameters - pushing the model to the DeltaCenter

• center_args – (optional) the arguments that are used to distinguish between different delta models on the DeltaCenter

• center_args_pool – (optional) a dictionary containing the arguments that are used to distinguish between different delta models on the DeltaCenter

• list_tags – (optional) a list of tags that will be added to the model’s configuration file

• dict_tags – (optional) a dictionary of tags that will be added to the model’s configuration file

• delay_push – (optional) if True, the model will not be pushed to the DeltaCenter. This is useful if you want to push the model later.

load_checkpoint(path, load_func=<function load>, backbone_model=None)

Simple method for loading only the checkpoint

save_checkpoint(path, save_func=<function save>, backbone_model=None)

Simple method for saving only the checkpoint

classmethod from_finetuned(finetuned_delta_path: Optional[Union[str, PathLike]], backbone_model: Module, delta_config=None, cache_dir: Optional[Union[str, PathLike]] = None, state_dict: Optional[dict] = None, *model_args, force_download: Optional[bool] = False, check_hash: Optional[bool] = True, local_files_only: Optional[bool] = False, **kwargs)

Instantiate a finetuned delta model from a path. The backbone_model is set in evaluation mode by default using model.eval() (Dropout modules are deactivated). To further train the model, you can use the freeze_module method.

Parameters

• finetuned_delta_path – (optional) path to the directory where the model and its configuration file will be saved. If not specified, the model will be loaded from the directory cahce directory. (see cache_dir),

• backbone_model – the backbone model that will be used to instantiate the finetuned delta model.

• delta_config – (optional) the configuration file of the finetuned delta model. If not specified, the configuration file is loaded from the directory finetuned_delta_path.

• cache_dir – (optional) path to the directory where the model and its configuration file will be saved. If not specified, we will first look into current working directory, then the cache directory of your system, e.g., ~/.cache/delta_center/,

• state_dict – (optional) a dictionary containing the model’s state_dict. If not specified, the state_dict is loaded from the finetuned_delta_path.

• force_download – (optional) if True, the model will be downloaded from the internet even if it is already present in the cache directory.

• check_hash – (optional) if True, check whether the hash of the model once it’s trained differs from what we load now.

• local_files_only – (optional) if True, the model will be loaded from the local cache directory.

create_delta_center_args(center_args, center_args_pool)

Create the delta center args for the center model. center_args has higher priority than center_args_pool.

create_default_name(**kwargs)

Currently, it’s only a simple concatenation of the arguments.

Visualization

class Visualization(plm: Module)

Better visualization tool for BIG pretrained models.

• Better repeated block representation

• Clearer parameter position

• and Visible parameter state.

Parameters

plm (torch.nn.Module) – The pretrained model, actually can be any pytorch module.

structure_graph(rootname='root', expand_params=False, keep_non_params=False, common_structure=False, mapping=None, only_common=False, printTree=True)

Draw the structure graph in command line.

Parameters

• rootname (str) –

• keep_non_params (bool) –

• expand_params (bool) Display parameter infomation (shape, etc) –

• common_structure (bool) –

• only_common (bool) –

• mapping (dict) –

build_common_tree(module: Module, mapping, tree: Optional[ModuleTree] = None, query='', key_to_root='')

(Unstable) build the common tree structure

Structure Map

class CommonStructureMap(backbone_model, strict=True, warning=False, visualize=True)

A loading structure map.

transform(org_key, strict=True, warning=False)

Transform a key in the original model to the name convention in common structure.

Utility Functions

Hashing

gen_model_hash(model, with_parameters=True)

Get model hash (structure and parameter)

gen_parameter_hash(generator, md5=None)

Get parameter hash. From https://zhuanlan.zhihu.com/p/392942816

Signature

signature(f)

Get the function f ‘s input arguments. A useful gadget when some function slot might be instantiated into multiple functions.

Parameters

f (function) – the function to get the input arguments.

Returns

of args, default, varargs, keywords, respectively.s

Return type

namedtuple

get_arg_names(f)

Get a functions argument name, remove the self argument

get_arg_names_inside_func(func)

Get the functions argument name inside the function itself. Remove self argument.

Named-based addressing

superstring_in(str_a: str, list_b: List[str])

check whether there is any string in list b containing str_a.

Args: Returns:

is_child_key(str_a: str, list_b: List[str])

check whether a string in list_b is the child key in str_a

Args: Returns:

endswith_in_normal(str_a: str, list_b: List[str])

check whether str_a has a substring that is in list_b.

Args: Returns:

endswith_in_regex(str_a: str, list_b: List[str])

check whether str_a has a substring that is in list_b.

Args: Returns: