Data Processor¶
-
class
neural_pipeline.data_processor.data_processor.
DataProcessor
(model: torch.nn.modules.module.Module, device: torch.device = None)[source]¶ DataProcessor manage: model, data processing, device choosing
- Args:
- model (Module): model, that will be used for process data device (torch.device): what device pass data for processing
-
predict
(data: torch.Tensor) → object[source]¶ Make predict by data
Parameters: data – data as torch.Tensor
or dict with keydata
Returns: processed output Return type: the model output type
-
set_pick_model_input
(pick_model_input: callable) → neural_pipeline.data_processor.data_processor.DataProcessor[source]¶ Set callback, that will get output from
DataLoader
and return model input.Default mode:
lambda data: data[‘data’]
- Args:
- pick_model_input (callable): pick model input callable. This callback need to get one parameter: dataset output
- Returns:
- self object
Examples:
data_processor.set_pick_model_input(lambda data: data['data']) data_processor.set_pick_model_input(lambda data: data[0])
-
class
neural_pipeline.data_processor.data_processor.
TrainDataProcessor
(train_config: TrainConfig, device: torch.device = None)[source]¶ TrainDataProcessor is make all of DataProcessor but produce training process.
Parameters: train_config – train config -
get_state
() → {}[source]¶ Get model and optimizer state dicts
Returns: dict with keys [weights, optimizer]
-
predict
(data, is_train=False) → torch.Tensor[source]¶ Make predict by data. If
is_train
isTrue
- this operation will compute gradients. Ifis_train
isFalse
- this will work withmodel.eval()
andtorch.no_grad
Parameters: - data – data in dict
- is_train – is data processor need train on data or just predict
Returns: processed output
Return type: model return type
-
process_batch
(batch: {}, is_train: bool, metrics_processor: AbstractMetricsProcessor = None) → numpy.ndarray[source]¶ Process one batch of data
Parameters: - batch – dict, contains ‘data’ and ‘target’ keys. The values for key must be instance of torch.Tensor or dict
- is_train – is batch process for train
- metrics_processor – metrics processor for collect metrics after batch is processed
Returns: array of losses with shape (N, …) where N is batch size
-
set_data_preprocess
(data_preprocess: callable) → neural_pipeline.data_processor.data_processor.DataProcessor[source]¶ Set callback, that will get output from
DataLoader
and return preprocessed data. For example may be used for pass data to device.Default mode:
_pass_data_to_device()
- Args:
- data_preprocess (callable): preprocess callable. This callback need to get one parameter: dataset output
- Returns:
- self object
Examples:
from neural_pipeline.utils import dict_recursive_bypass data_processor.set_data_preprocess(lambda data: dict_recursive_bypass(data, lambda v: v.cuda()))
-
set_pick_target
(pick_target: callable) → neural_pipeline.data_processor.data_processor.DataProcessor[source]¶ Set callback, that will get output from
DataLoader
and return target.Default mode:
lambda data: data[‘target’]
- Args:
- pick_target (callable): pick target callable. This callback need to get one parameter: dataset output
- Returns:
- self object
Examples:
data_processor.set_pick_target(lambda data: data['target']) data_processor.set_pick_target(lambda data: data[1])
-
Model¶
-
class
neural_pipeline.data_processor.model.
Model
(base_model: torch.nn.modules.module.Module)[source]¶ Wrapper for
torch.nn.Module
. This class provide initialization, call and serialization for itParameters: base_model – torch.nn.Module
object-
model
() → torch.nn.modules.module.Module[source]¶ Get internal
torch.nn.Module
objectReturns: internal torch.nn.Module
object
-