Model#

The cvxlab.Model class includes all the main package APIs necessary to generate, handle and solve the numerical model.

class cvxlab.Model(
model_dir_name='model',
main_dir_path=None,
model_settings_from='xlsx',
detailed_validation=False,
use_existing_data=False,
multiple_input_files=False,
input_data_files_type='xlsx',
log_level='info',
log_format='standard',
)[source]#

Central class for generating and handling a CVXLab models.

The Model class represents a modeling environment that handles SQLite data generation and processing, database interactions, numerical optimization model generation and handling with cvxpy package. This class initializes with a configuration for managing directories, logging, and file management for a specific model. It also sets up various components including a logger, file manager, and core functionalities.

Attributes:

  • logger (Logger): Logger object for logging information, warnings, and errors.

  • files (FileManager): An instance of FileManager to manage file operations.

  • settings (DotDict): A dictionary-like object storing configurations such as model name, file paths, and operational flags.

  • paths (DotDict): A dictionary-like object storing the paths for model directories and associated files.

  • core (Core): An instance of Core that manages the core functionality of the model (it embeds Index, Database and Problem instances).

Parameters:

  • model_dir_name (str)

  • main_dir_path (str | None)

  • model_settings_from (Literal['yml', 'xlsx'])

  • detailed_validation (bool)

  • use_existing_data (bool)

  • multiple_input_files (bool)

  • input_data_files_type (Literal['xlsx', 'csv'])

  • log_level (Literal['info', 'debug', 'warning', 'error'])

  • log_format (Literal['standard', 'detailed'])

Model initialization and setup#

Model.__init__(
model_dir_name='model',
main_dir_path=None,
model_settings_from='xlsx',
detailed_validation=False,
use_existing_data=False,
multiple_input_files=False,
input_data_files_type='xlsx',
log_level='info',
log_format='standard',
)[source]#

Initialize the Model instance with specified configurations.

This constructor sets up the Model instance by initializing logging, file management, and core functionalities. It also checks for the existence of the model directory and required setup files. If the ‘use_existing_data’ flag is set to True, it loads existing sets data and variable coordinates to the Model.Index and initializes numerical problems (the configuration files and model database should have been already generated).

Parameters:

  • model_dir_name (str) – The name of the model directory.

  • main_dir_path (str) – The main directory path where the model directory is located. If None, the current working directory is used.

  • model_settings_from (Defaults.LiteralTypes.SettingsSource, optional) – The format of the model settings file. Can be either ‘yml’ or ‘xlsx’. Defaults to ‘xlsx’.

  • detailed_validation (bool, optional) – if True, performs detailed validation logging of data and model settings during initialization. Defaults to False.

  • use_existing_data (bool, optional) – if True, generation of Model instance is also loading model coordinates and initializing numerical problems. Note that setup files and model database should have been already generated. Defaults to False.

  • multiple_input_files (bool, optional) – if True, input data Excel files are generated as one file per data table. If False, all data tables are generated in a single Excel file with multiple tabs. Defaults to False.

  • input_data_files_type (Defaults.LiteralTypes.DataFileType, optional) – The format of the input data files. Can be either ‘xlsx’ or ‘csv’. Defaults to ‘xlsx’.

  • log_level (Defaults.LiteralTypes.LogLevel, optional) – The logging level for the logger. Defaults to ‘info’.

  • log_format (Defaults.LiteralTypes.LogFormat, optional) – The logging format for the logger. Defaults to ‘standard’.

Model.initialize_model_environment()[source]#

Initialize the model environment for problem generation and solution.

This method prepares the model environment by loading sets data and coordinates, and initializing blank data structures (SQLite database and input data files). This method is typically called after having generated the Model instance, and before generating numerical problems and solving them. If the ‘use_existing_data’ flag is set to True, this method is not meant to be called, since sets data and coordinates are already loaded, and blank data structures are not generated relying on existing ones.

Raises:

exc.SettingsError – If the sets Excel file specified in the settings is missing when loading model coordinates.

Return type:

None

Model.refresh_database_and_initialize_problem(
table_key_list=[],
force_overwrite=False,
)[source]#

Update SQLite database with exogenous data and initialize problems.

This method loads the SQLite database exogenous data, and then initializes numerical problems. This method can be used for both initialization of the model environment, after having generated the Model instance, and for updating the model environment and the problems in case some changes in input data have been made, without the need of re-generating the Model instance.

Parameters:

  • table_key_list (list[str], optional) – A list of data table keys for which to load exogenous data. If empty, all exogenous data tables are loaded. Defaults to [].

  • force_overwrite (bool, optional) – Whether to overwrite/update existing data without asking user permission. Defaults to False.

Return type:

None

Model.run_model(
force_overwrite=False,
integrated_problems=False,
convergence_monitoring=True,
solver=None,
solver_verbose=False,
solver_settings=None,
convergence_norm='l2',
convergence_tables_to_check='all_endogenous',
convergence_tables_to_skip=None,
relative_tolerance=None,
maximum_iterations=None,
keep_previous_iteration_db=False,
**kwargs,
)[source]#

Solve numerical problems defined by the model instance.

This method first performs some coherence checks (if solver is supported, if numerical problems are defined, if integrated problems are possible). Then, it solves the numerical problems, either independently or in an integrated manner, based on the ‘integrated_problems’ flag. Finally, it logs a summary of the problems status.

Parameters:

  • force_overwrite (bool, optional) – If True, overwrites existing results. Defaults to False.

  • integrated_problems (bool, optional) – If True, solve problems iteratively using a block Gauss-Seidel (alternating optimization) scheme, where updated endogenous variables are exchanged until convergence. Defaults to False.

  • convergence_monitoring (bool, optional) – If True, enables convergence monitoring during the solving of integrated problems. Defaults to True.

  • solver (str, optional) – The solver to use for solving numerical problems. Defaults to None, in which case the default solver specified in ‘Defaults.NumericalSettings.CVXPY_DEFAULT_SETTINGS’ is used.

  • solver_verbose (bool, optional) – If True, logs verbose output related to numerical solver operation during the model run. Defaults to False.

  • solver_settings (dict[str, Any], optional) – Additional settings for the solver passed as key-value pairs. Defaults to None.

  • convergence_norm (Defaults.LiteralTypes.NormType, optional) – The norm type to use for convergence monitoring in integrated problems. Defaults to ‘l2’ (Euclidean Norm).

  • | (convergence_tables_to_check (Defaults.LiteralTypes.ConvergenceTables) – List[str], optional): The data tables to consider for convergence monitoring in integrated problems. Can be ‘all_endogenous’, ‘hybrid_only’, or a list of specific data table keys. Defaults to ‘all_endogenous’.

  • convergence_tables_to_skip (Optional[List[str]], optional) – List of data table keys to skip for convergence checking in integrated problems. If None, no tables are skipped. Defaults to None.

  • relative_tolerance (float, optional) – Numerical tolerance for verifying maximum relative change between iterations in integrated problems for each data table. Overrides ‘Defaults.NumericalSettings.MODEL_COUPLING_SETTINGS’.

  • maximum_iterations (int, optional) – The maximum number of iterations for solving integrated problems. Overrides ‘Defaults.NumericalSettings.MODEL_COUPLING_SETTINGS’.

  • keep_previous_iteration_db (bool, optional) – Whether keep or not the database generated during the last-1 iteration. For debugging purpose. Default to False.

  • **kwargs – Additional keyword arguments to be passed to the solver. Useful for setting solver-specific options.

  • convergence_tables_to_check (Literal['all_endogenous', 'hybrid_only'] | ~typing.List[str])

Raises:

  • exc.SettingsError – In case solver is not supported by current cvxpy version.

  • exc.SettingsError – If no numerical problems are found, or if integrated problems are requested but only one problem is found.

Return type:

None

Model data management methods#

Model.generate_input_data_files(table_key_list=[], values_cleanup=True)[source]#

Generate blank Excel files for data input.

This method generates blank Excel files for data input, based on the data tables defined in the model. If the input data directory already exists, it gives the option to erase it and generate a new one, or to work with the existing input data directory. This method is called within the ‘initialize_blank_data_structure’ method. However, the user can call it directly to regenerate input data file/s, for all or for specific data tables (with the ‘table_key_list’ attribute). This is especially useful in adjusting the input data without regenerating the whole blank data structure. This feature works also in case of one single Excel file, since it can overwrite only the tabs related to the specified data tables.

Parameters:

  • table_key_list (List[str], optional) – A list of data table keys for which to generate input data files. If empty, all data tables are generated. Defaults to [].

  • values_cleanup (bool, optional) – Whether to clean up values of database tables before generating input data files. Defaults to True.

Raises:

  • exc.SettingsError – If the input data directory is missing.

  • exc.SettingsError – If any of the specified table keys are invalid (i.e., not exogenous data tables).

Return type:

None

Model.load_results_to_database(
scenarios_idx=None,
force_overwrite=False,
suppress_warnings=False,
)[source]#

Export model results to the SQLite database.

This method exports the results of the numerical problems to the SQLite database. It can export results for all scenarios or for specific scenarios (defined as the linear combinations of inter-problem sets that identify independent numerical problems), based on the ‘scenarios_idx’ attribute

Parameters:

  • scenarios_idx (Optional[List[int] | int], optional) – A list of scenario indices or a single scenario index for which to export results. If None, results for all scenarios are exported. Defaults to None.

  • force_overwrite (bool, optional) – Whether to overwrite/update existing data without asking user permission. Defaults to False.

  • suppress_warnings (bool, optional) – Whether to suppress warnings during the data loading process. Defaults to False.

Return type:

None

Model.reinitialize_sqlite_database(force_overwrite=False)[source]#

Reinitialize SQLite database tables and reimport input data.

This method reinitializes endogenous tables in SQLite database to Null values, and reimports input data to exogenous tables.

Parameters:

force_overwrite (bool, optional) – Whether to force overwrite existing data. Used for testing purposes. Defaults to False.

Return type:

None

Model.update_sets_tables(set_keys_list=[], update_mode='all')[source]#

Update sets tables in the SQLite database.

This method updates sets tables in the SQLite database. The user can choose to update all sets, or only specific sets (with the ‘set_keys_list’ attribute). Additionally, the user can specify the update mode, which can be ‘all’, ‘filters’, or ‘aggregations’.

Parameters:

  • set_keys_list (List[str], optional) – A list of set keys to update. If empty, all sets are updated. Defaults to [].

  • update_mode (Defaults.LiteralTypes.SetUpdateMode, optional) – The update mode. Can be ‘all’ (update all set data), ‘filters’ (update only filters), or ‘aggregations’ (update only aggregations). Defaults to ‘all’.

Return type:

None

Attributes and properties#

Model.sets#

List of sets names available in the model.

Returns:

A list of set names.

Return type:

List[str]

Model.data_tables#

List of data tables names available in the model.

Returns:

A list of data table names.

Return type:

List[str]

Model.variables#

List of variables names available in the model.

Returns:

A list of variable names.

Return type:

List[str]

Model.is_problem_solved#

Status of the problem solution.

Checks if the numerical problem has been solved (even if it has not found a numerical solution).

Returns:

True if the problem has been solved, False otherwise.

Return type:

bool

Helper methods#

Model.set(name)[source]#

Fetch set data.

Useful for inspecting variables data during model generation and debugging.

Parameters:

name (str) – The name of the set.

Returns:

The data for the specified set.

Return type:

Optional[pd.DataFrame]

Model.variable(
name,
scenario_key=None,
intra_problem_key=None,
if_hybrid_var='endogenous',
)[source]#

Fetch variable data.

This method retrieves the data for a specified variable based on optional inter-problem and intra-problem sets cardinality, supporting the data inspection process after a model has run, but before data has exported to the database. This is particularly useful in case multiple runs of the model, to facilitate the control of the numerical data from the user. In case a variable is defined as both endogeous and exogenous, depending on the numerical problem, the user can specify the one to inspect (default as the endogenous one). If the variable is specified for multiple inter- and intra-problem sets, scenario_key defines the cardinality of inter-problem sets, while intra_problem_key defines the cardinality of intra-problem sets.

Parameters:

  • name (str) – The key of the variable in the variables dictionary.

  • scenario_key (Optional[int]) – Defines the cardinality of inter-problem sets. Default to None.

  • intra_problem_key (Optional[int]) – Defines the cardinality of intra-problem sets. Default to None.

  • if_hybrid_var (Defaults.LiteralTypes.HybridVarType) – Defines the type of variable data to inspect in case variable type depends on the problem.

Returns:

The data for the specified variable.

Return type:

Optional[pd.DataFrame]

Model.check_model_results(
other_db_dir_path=None,
other_db_name=None,
numerical_tolerance=None,
)[source]#

Compare model SQLite database with another SQLite reference database.

This method compares the results of the model’s SQLite database with those of another SQLite database, typically for testing purposes. It uses the ‘check_results_as_expected’ method to compare the current model’s computations with the expected results. The expected results can be stored in a specific database (identified by directory path and name), or stored in a test database specified by the SQLite test database default name and located in the model directory. Both ‘other_db_dir_path’ and ‘other_db_name’ must be provided together, or both must be None. If not provided, defaults are used.

Parameters:

  • other_db_dir_path (Optional[Path | str], optional) – The directory path where the other SQLite database is located. If None, it defaults to the model directory. Defaults to None.

  • other_db_name (Optional[str], optional) – The name of the other SQLite database file. If None, it defaults to the SQLite test database default name. Defaults to None.

  • numerical_tolerance (float, optional) – The relative difference (non-percentage) tolerance for comparing numerical values in different databases. If None, it is set to ‘Defaults.NumericalSettings.TOLERANCE_TESTS_RESULTS_CHECK’.

Return type:

None