User Guide#

This section provides a comprehensive guide to using CVXlab for conceptualizing, generating and solving optimization problems. This includes all the necessary steps to generate a model from scratch, or to generate it from existing settings/data. At the end of the section, a set of utilities is provided to generate and handle model directories, inspect model properties and data, save and load model instances, and refresh the model database.

Before diving into individual steps, we suggest reading this page, which provides a concise but comprehensive overview of the CVXlab modeling workflow.

The Tutorials section provides different examples across the same workflow steps used in the current page, from conceptual definition to results export. While the user guide aims to provide general concepts, tutorials are recommended to see how those concepts are instantiated in a concrete model. For CVXlab newbies, it is recommended to start with the resource-constrained production planning problem tutorial.

Guided interface#

Users who prefer a menu-driven workflow can configure and run a model through the Guided Interface. That page explains interactive prompts, preconfiguration settings, the available actions, and how each action maps to the modeling steps and public APIs documented below.

Model generation from scratch#

The CVXlab modeling workflow for generating a model from scratch is summarized in the table below. Each workflow step is described in the description column, and it links to the corresponding detailed page.

CVXlab full modeling workflow#
Workflow step	Description
Conceptual model definition	Mathematical model conceptualization. User defines the mathematical structure of the model (pen on paper!).
Generation of model directory	Model directory and setup template file(s) are generated. API: `cvxlab.create_model_dir()`
Fill model setup file(s)	User fills the setup files.
Generate Model class instance	CVXlab Model instance is created, and conceptual model validated. Blank `sets.xlsx` input file is generated. API: `cvxlab.Model`
Fill sets data	User fills `sets.xlsx` input file with the model coordinates.
Initialization of data structures	Blank SQLite database and Excel input data file(s) are generated. API: `initialize_model_environment()`
Fill exogenous model data	User fills the exogenous Excel input data file(s).
Initialization of numerical problem(s)	Symbolic problem is validated, and numerical problem initialized. API: `refresh_database_and_initialize_problem()`
Run numerical problem(s)	Numerical problem is solved. API: `run_model()`
Export model results	Results are exported from CVXlab Model to the SQLite database. API: `load_results_to_database()`

Step by step description of the CVXlab modeling workflow

Conceptual model definition: the whole CVXlab modeling process must be grounded on a solid conceptualization and mathematical definition of the problem to be solved. This step consists of defining the fundamental model structure pen-on-paper. The following items must be defined:

Conceptual component	Description
Sets	The dimensions of the model, defining its scope.
Data Tables and Variables	Data tables are collections of model data identified by one or more sets. Variables are symbolic items pointing to all or a subset of the data entries in those tables.
Expressions and Problems	Problems are collections of expressions, which are symbolic items constructed from defined variables and symbolic operators. More than one problem can be stated, each defined as a system of linear equalities or as a convex optimization problem.

Once the problem is well defined, the CVXlab modeling process can start.

Generation of model directory: a model directory is generated based on a predefined template, and it contains all the necessary files to transpose the conceptual model into a CVXlab model instance. Setup files can be generated as YAML or Excel formats, depending on the user’s preference. Optionally, Python template modules can be included in the model directory to extend CVXlab functionalities with user-defined symbolic operators and user-defined constants data types (user_defined_operators.py and user_defined_constants.py).

API: cvxlab.create_model_dir()

Fill model setup file(s): the user translates the conceptual model into the setup files available in the model directory. The setup information can be provided either in YAML format (as three separate files) or in Excel format (default model_settings.xlsx workbook with three tabs), as summarized below:

Setup component	YAML file	`model_settings.xlsx` tab
Structure of sets	`structure_sets.yml`	`structure_sets`
Structure of data tables and variables	`structure_variables.yml`	`structure_variables`
Mathematical problem	`problem.yml`	`problem`

Optionally, user-defined symbolic operators and constant data types can be implemented in dedicated Python template modules included in the model directory.

Model class instance generation: the instance of the Model class is generated through the Model class constructor, which translates the information provided by the setup files into a Python object. The model instance includes all the necessary information and the APIs to generate and to solve numerical problems, and to handle exogenous/endogenous model data. The model instance generation automatically triggers the generation of the Excel file sets.xlsx to be filled with the sets coordinates (see Fill sets data (model coordinates)). During the Model class instantiation, a comprehensive validation of model structure is performed to ensure that all model components are correctly and consistently defined.

API (class constructor): cvxlab.Model
Fill sets data (model coordinates): the user fills the sets Excel file with the elements belonging to each set, defined as coordinates. The Excel file includes one tab for each set, where the user can specify other set attributes, such as set filters and aggregation categories. Once this step is completed, all the necessary information defining the model structure is available, and the data structures can be generated.

Initialization of data structures: this step imports the set coordinates into the model instance, checks that variables are defined consistently, and generates the initial model data structures.

Output	Description
Blank SQLite database	Relational database containing set tables and data tables, linked by unique relationships.
Blank Excel/CSV input data files	One or more Excel or CSV files, depending on the user’s preference, containing the exogenous data tables in normalized form to be filled by the user (see Fill exogenous model data).

API: initialize_model_environment()

Fill exogenous model data: the user fills the Excel/CSV input data files with the values in exogenous data tables. In models with a large amount of data, this step can be critical, time-consuming and highly prone to errors. To facilitate this task, CVXlab provides features to limit the number of data entries to be filled (for example, a default value can be defined in the settings for data table entries when data are missing). Once this step is completed, the numerical model can be generated and solved.
Initialization of numerical problem(s): this step includes fetching exogenous data from Excel input data files to the model SQLite database, loading symbolic problems from the model settings, and generating numerical problems (as CVXPY problem object). During the problem generation step, a comprehensive validation of symbolic problem and model data is performed to ensure that the symbolic problem is consistently defined, and that all necessary data entries are correctly provided.

API: refresh_database_and_initialize_problem()

Solution of numerical problem(s): in this step, the numerical problems are solved based on user settings, including selection of the adopted solver, the related verbosity level, and other numerical settings. Solver settings can be specified for each problem, or shared by all problems. If multiple numerical problems, the user can define the solution mode to be adopted:

Solution mode	Description
`parallel`	All numerical problems defined within the Model instance are solved without exchanging information between them (no problem dependencies). This is the default solution mode.
`sequential`	Numerical problems are solved sequentially: results from one problem may be used as inputs for the next in a user-defined problem solution chain. Useful when data need to be processed sequentially.
`integrated`	Numerical problems are solved based on a block Gauss-Seidel (alternating optimization) algorithm: for each iteration, input data for variables shared among problems are updated based on the latest results from previous iteration, until convergence is reached. Useful for decomposing nonlinear problems into smaller convex sub-problems.

API: run_model()

Export endogenous model data: if numerical problems have been successfully solved, numerical results are exported to endogenous data tables of the SQLite database.

API: load_results_to_database()

Model generation from existing settings/data#

The CVXlab modeling workflow for generating a model from existing settings/data is summarized in the table below. Each workflow step is described in the description column, and it links to the corresponding detailed page.

CVXlab modeling workflow from existing settings/data#
Workflow step	Description
Generate Model class instance	CVXlab Model instance is created, and conceptual model validated. `sets.xlsx` file is assumed to be already filled by the user. API: `cvxlab.Model`
Fill exogenous model data	(OPTIONAL) User updates/fills the exogenous Excel input data file(s).
Initialization of numerical problem(s)	(OPTIONAL) SQLite database is refreshed, symbolic problem is validated, and numerical problem initialized. API: `refresh_database_and_initialize_problem()`
Run numerical problem(s)	Numerical problem is solved. API: `run_model()`
Export model results	Results are exported from CVXlab Model to the SQLite database. API: `load_results_to_database()`

Step by step description of the CVXlab modeling workflow

Model class instance generation: the instance of the Model class is generated through the Model class constructor as the first step, specifying that the model relies on existing data (i.e. by passing use_existing_data=True to the constructor). Besides the setup files (see Fill model setup files), other data structures must be present in the model directory, including the sets Excel file filled with the related coordinates (see Fill sets data (model coordinates)), the blank SQLite database (with set tables and data tables, see Initialization of data structures), and the Excel input data directory (see Fill exogenous model data).

The Model constructor checks and validates the model directory structure and the presence of all necessary files. Then, it loads the structures of sets, data tables and variables in the index, and fetches problems from the setup files. Then it loads coordinates from the sets Excel file into the model instance. Finally, it initializes numerical problems and fetches exogenous data from the SQLite database. The model instance includes all the necessary information and the APIs to generate and to solve numerical problems, and to handle model data.

API (class constructor): cvxlab.Model
Fill exogenous model data: (optional) in this step, the user may update exogenous model data in the Excel files. This step is especially useful for multiple consecutive model runs, where only exogenous data are changed. If needed, one or more blank input data files can be regenerated from the blank SQLite database.

API: generate_input_data_files()
Initialization of numerical problem(s): (optional) this step is needed when exogenous data have been updated (see Fill exogenous model data). It fetches exogenous data from the Excel input data files to the SQLite database, updating data in numerical problem variables.

API: refresh_database_and_initialize_problem()
Solution of numerical problem(s): in this step, the numerical problems are solved based on user settings, including the adopted solver, the related verbosity level, and other settings. Solver settings can be specified for each problem, or be the same for all problems. If multiple numerical problems are defined, the user can define the solution mode to be adopted (i.e. parallel, sequential, integrated).

API: run_model()
Export endogenous model data: if numerical problems have been successfully solved, numerical results are exported to the endogenous data tables of the SQLite database.

API: load_results_to_database()

Utilities#

During the modeling process, or once the model is generated and solved, the user may need to inspect model properties or perform some basic operations on the database. CVXlab provides a set of utility functions to facilitate these tasks, reported in the tables below.

CVXlab Model instance inspection utilities#
API	Description
`sets`	Property reporting the list of model set keys.
`data_tables`	Property reporting the list of model data table keys.
`variables`	Property reporting the list of model variable keys.
`is_problem_solved`	Property indicating the status of the numerical problem.
`set()`	Method allowing inspection of a specific set, including related coordinates and other attributes.
`variable()`	Method allowing inspection of a specific variable, including related data table, shape, filters and other attributes.

Other helper methods allow to perform basic operations on the model instance and on the SQLite database, summarized in the table below.

CVXlab Model instance helper utilities#
API	Description
`reinitialize_sqlite_database()`	Method allowing regeneration of a blank SQLite database from the current model structure (sets and data tables). Useful to reset the database to a clean state.
`check_model_results()`	Method allowing comparison of the model SQLite database with a reference SQLite database, to verify that results are coherent with expected values.
`update_sets_tables()`	Method allowing update of sets tables in the SQLite database based on the current model structure and coordinates. Useful after changing set coordinates, for example to add aggregation categories.