User Guide#

This section provides a comprehensive guide to using CVXlab for conceptualizing, generating and solving optimization problems. This includes all the necessary steps to generate a model from scratch, or to generate it from existing settings/data. At the end of the section, a set of utilities is provided to generate/handle model directory, facilitate model properties and data inspections, saving and loading of model instances, refreshing model database.

Before diving into each single step, it is suggested to read all this page, which provides a synthetic but comprehensive overview about the CVXlab modeling workflow.

Tutorial with CVXlab applications are available in Tutorials section. In particular, a complete illustrative example based on a simplified energy system model is available, where the example is distributed across the same workflow presented below.

Guided interface (quick start)#

For users who prefer a plug-and-play experience, CVXlab provides an interactive guided interface that walks through the entire modeling workflow via a menu-driven session. This is the simplest way to get started with CVXlab:

import cvxlab

cvxlab.run()

The run() function launches an interactive session that guides the user through model directory generation, setup, data loading, problem solving, and results export. All parameters (solver, log level, convergence settings, etc.) can be passed directly or configured interactively. The idea is that if arguments are not passed, the user is interactively prompted to provide the necessary information at each step, with sensible defaults provided for all parameters.

The interface is ideal to avoid direct interaction with the underlying APIs; however, model formulation, settings, and data input must still be provided through the expected setup files and data structures, as described in the following sections.

For full control over each modeling step, refer to the programmatic workflows below.

API: cvxlab.run()

Model generation from scratch#

The CVXlab modeling workflow for generating a model from scratch is summarized in the table below. Each workflow step is described in the description column, and it links to the corresponding detailed page.

CVXlab full modeling workflow#

Workflow step

Description

Conceptual model definition

Mathematical model conceptualization. User defines the mathematical structure of the model (pen on paper!).

Generation of model directory

Model directory and setup template file(s) are generated. API: cvxlab.create_model_dir()

Fill model setup file(s)

User fills the setup files.

Generate Model class instance

CVXlab Model instance is created, and conceptual model validated. Blank sets.xlsx input file is generated. API: cvxlab.Model

Fill sets data

User fills sets.xlsx input file with the model coordinates.

Initialization of data structures

Blank SQLite database and Excel input data file(s) are generated. API: initialize_model_environment()

Fill exogenous model data

User fills the exogenous Excel input data file(s).

Initialization of numerical problem(s)

Symbolic problem is validated, and numerical problem initialized. API: refresh_database_and_initialize_problem()

Run numerical problem(s)

Numerical problem is solved. API: run_model()

Export model results

Results are exported from CVXlab Model to the SQLite database. API: load_results_to_database()

Step by step description of the CVXlab modeling workflow

  • Conceptual model definition: the whole CVXlab modeling process must be grounded on a solid conceptualization and mathematical definition of the problem to be solved. This step consists in defining the fundamental model structure pen-on-paper. The following items must be defined:

    Conceptual component

    Description

    Sets

    The dimensions of the model, defining its scope.

    Data Tables and Variables

    Data tables are collections of model data identified by one or more sets. Variables are symbolic items pointing to all or a subset of the data entries in those tables.

    Expressions and Problems

    Problems are collections of expressions, which are symbolic items constructed from defined variables and symbolic operators. More than one problem can be stated, each defined as a system of linear equalities or as a convex optimization problem.

    Once the problem is well defined, the CVXlab modeling process can start.

  • Generation of model directory: a model directory is generated based on a predefined template, and it contains all the necessary files to transpose the conceptual model into a CVXlab model instance. Setup files can be generated as YAML or Excel formats, depending on user’s preference. Eventually, Python template modules can be included in the model directory to extend CVXlab functionalities with user-defined symbolic operators (user_defined_operators.py) and user-defined constants data types (user_defined_constants.py).

    API: cvxlab.create_model_dir()

  • Fill model setup file(s): the user translates the conceptual model into the setup files available in the model directory. The setup information can be provided either in YAML format (as three separate files) or in Excel format (default model_settings.xlsx workbook with three tabs), as summarized below:

    Setup component

    YAML file

    model_settings.xlsx tab

    Structure of sets

    structure_sets.yml

    structure_sets

    Structure of data tables and variables

    structure_variables.yml

    structure_variables

    Mathematical problem

    problem.yml

    problem

    Optionally, user-defined symbolic operators and constant data types can be implemented in dedicated Python template modules included in the model directory.

  • Model class instance generation: the instance of the Model class is generated through the Model class constructor, which translates the information provided by setup file/s into a Python object. The model instance includes all the necessary information and the APIs to generate and to solve numerical problems, and to handle exogenous/endogeous model data. The model instance generation automatically triggers the generation of the Excel file sets.xlsx to be filled with the sets coordinates (see Fill sets data (model coordinates)). During the Model class instantiation, a comprehensive validation of model structure is performed to ensure that all model components are correctly and consistently defined.

    API (class constructor): cvxlab.Model

  • Fill sets data (model coordinates): the user fills the sets Excel file with the elements belonging to each set, defined as coordinates. The Excel file includes one tab per each set, where user can eventually specify other set methibutes, such as set filters and aggregation categories. Once this step is completed, all the necessary information defining the model structure is available, and the data structures can be generated.

  • Initialization of data structures: this step imports the set coordinates into the model instance, checks that variables are defined consistently, and generates the initial model data structures.

    Output

    Description

    Blank SQLite database

    Relational database containing set tables and data tables, linked by unique relationships.

    Blank Excel input data file(s)

    One or more Excel files, depending on user preference, containing the exogenous data tables in normalized form to be filled by the user (see Fill exogenous model data).

    API: initialize_model_environment()

  • Fill exogenous model data: user fills the Excel input data file/s with the exogenous data tables values. In case of models with large amount of data, this step can be critical, time consuming and highly prone to errors. To facilitate this task, CVXlab provides features to limit the amount of data entries to be filled (as example: in settings, a default value can be defined for data table entries in case of missing data). Once this step is completed, the numerical model can be generated and solved.

  • Initialization of numerical problem(s): this step includes fetching exogenous data from excel input data file/s to the model SQLite database, loading symbolic problem/s from model settings, and generating the numerical problem/s (as CVXPY problem object). During the problem generation step, a comprehensive validation of symbolic problem and model data is performed to ensure that the symbolic problem is consistently defined, and that all necessary data entries are correctly provided.

    API: refresh_database_and_initialize_problem()

  • Solution of numerical problem(s): in this step, the numerical problem/s are solved based on user settings, including selection of the adopted solver, the related verbosity level, and other numerical settings.

    API: run_model()

  • Export endogenous model data: in case numerical problem/s have successfully solved, numerical results are exported to endogenous data tables of the SQLite database.

    API: load_results_to_database()

Model generation from existing settings/data#

The CVXlab modeling workflow for generating a model from existing settings/data is summarized in the table below. Each workflow step is described in the description column, and it links to the corresponding detailed page.

CVXlab modeling workflow from existing settings/data#

Workflow step

Description

Generate Model class instance

CVXlab Model instance is created, and conceptual model validated. sets.xlsx file is assumed to be already filled by the user. API: cvxlab.Model

Fill exogenous model data

(OPTIONAL) User updates/fills the exogenous Excel input data file(s).

Initialization of numerical problem(s)

(OPTIONAL) SQLite database is refreshed, symbolic problem is validated, and numerical problem initialized. API: refresh_database_and_initialize_problem()

Run numerical problem(s)

Numerical problem is solved. API: run_model()

Export model results

Results are exported from CVXlab Model to the SQLite database. API: load_results_to_database()

Step by step description of the CVXlab modeling workflow

  • Model class instance generation: the instance of the Model class is generated through the Model class constructor as the first step, specifying that the model relies on existing data (i.e. by passing use_existing_data=True to the constructor). Beside setup file/s (see Fill model setup file/s), other data structures must be present in the model directory, including the sets Excel file filled with the related coordinates (see Fill sets data (model coordinates)), the blank SQLite database (with set tables and data tables, see Initialization of data structures), and the Excel input data directory (see Fill exogenous model data).

    The Model constructor checks and validate the model directory structure and the presence of all necessary files. Then, it loads the structures of sets, data tables and variables in the index, and fetches problems from the setup file/s. Then it loads coordinates from the sets Excel file to the model instance. Finally, it initializes numerical problem/s and fetches exogenous data from the SQLite database. The model instance includes all the necessary information and the APIs to generate and to solve numerical problems, and to handle model data.

    API (class constructor): cvxlab.Model

  • Fill exogenous model data: (optional) in this step, user may update exogenous model data in the Excel file/s. This step is especially useful in case of multiple consecutive model runs, where only exogenous data are changed. If needed, one or more blank input data file/s can be re-generated from the blank SQLite database.

    API: generate_input_data_files()

  • Initialization of numerical problem(s): (optional) this step needs to be made in case of exogenous data have updated (see Fill exogenous model data). It fetches exogenous data from the Excel input data files to the SQLite database, updating data in numerical problem/s variables.

    API: refresh_database_and_initialize_problem()

  • Solution of numerical problem(s): in this step, the numerical problem/s are solved based on user settings, including the adopted solver, the related verbosity level, and other settings.

    API: run_model()

  • Export endogenous model data: in case numerical problem(s) have successfully solved, numerical results are exported to the endogenous data tables of the SQLite database.

    API: load_results_to_database()

Utilities#

During the modeling process, or once the model is generated and solved, the user may need to inspect model properties, or doing some basic operations on the database. CVXlab provides a set of utilities functions to facilitate these tasks, reported in the tables below.

CVXlab Model instance inspection utilities#

API

Description

sets

Property reporting the list of model set keys.

data_tables

Property reporting the list of model data table keys.

variables

Property reporting the list of model variable keys.

is_problem_solved

Property indicating the status of the numerical problem.

set()

Method allowing inspection of a specific set, including related coordinates and other attributes.

variable()

Method allowing inspection of a specific variable, including related data table, shape, filters and other attributes.

Other helper methods allow to perform basic operations on the model instance and on the SQLite database, summarized in the table below.

CVXlab Model instance helper utilities#

API

Description

reinitialize_sqlite_database()

Method allowing regeneration of a blank SQLite database from the current model structure (sets and data tables). Useful to reset the database to a clean state.

check_model_results()

Method allowing comparison of the model SQLite database with a reference SQLite database, to verify that results are coherent with expected values.

update_sets_tables()

Method allowing update of sets tables in the SQLite database based on the current model structure and coordinates. Useful after changing set coordinates, for example to add aggregation categories.