.. _user_guide: User Guide ========== This section provides a comprehensive guide to using CVXlab for conceptualizing, generating and solving optimization problems. This includes all the necessary steps to :ref:`generate a model from scratch `, or to :ref:`generate it from existing settings/data `. At the end of the section, a set of :ref:`utilities ` is provided to generate/handle model directory, facilitate model properties and data inspections, saving and loading of model instances, refreshing model database. Before diving into each single step, it is suggested to read all this page, which provides a synthetic but comprehensive overview about the CVXlab modeling workflow. Tutorial with CVXlab applications are available in :ref:`tutorials` section. In particular, a complete illustrative example based on a :ref:`simplified energy system model ` is available, where the example is distributed across the same workflow presented below. .. _guided_interface: Guided interface (quick start) ------------------------------ For users who prefer a plug-and-play experience, CVXlab provides an interactive guided interface that walks through the entire modeling workflow via a menu-driven session. This is the simplest way to get started with CVXlab: .. code-block:: python import cvxlab cvxlab.run() The ``run()`` function launches an interactive session that guides the user through model directory generation, setup, data loading, problem solving, and results export. All parameters (solver, log level, convergence settings, etc.) can be passed directly or configured interactively. The idea is that if arguments are not passed, the user is **interactively prompted to provide the necessary information at each step**, with sensible defaults provided for all parameters. The interface is ideal to avoid direct interaction with the underlying APIs; however, model formulation, settings, and data input must still be provided through the expected setup files and data structures, as described in the following sections. For full control over each modeling step, **refer to the programmatic workflows below**. API: :py:func:`cvxlab.run` .. _model_generation_from_scratch: Model generation from scratch ----------------------------- The CVXlab modeling workflow for generating a model from scratch is summarized in the table below. Each **workflow step** is described in the **description** column, and it links to the corresponding detailed page. .. list-table:: CVXlab full modeling workflow :header-rows: 1 :widths: 40 58 * - Workflow step - Description * - :ref:`Conceptual model definition ` - Mathematical model conceptualization. *User* defines the mathematical structure of the model (*pen on paper!*). * - :ref:`Generation of model directory ` - Model directory and setup template file(s) are generated. API: :py:func:`cvxlab.create_model_dir` * - :ref:`Fill model setup file(s) ` - *User* fills the setup files. * - :ref:`Generate Model class instance ` - CVXlab Model instance is created, and conceptual model validated. Blank ``sets.xlsx`` input file is generated. API: :py:class:`cvxlab.Model` * - :ref:`Fill sets data ` - *User* fills ``sets.xlsx`` input file with the model coordinates. * - :ref:`Initialization of data structures ` - Blank SQLite database and Excel input data file(s) are generated. API: :py:meth:`~cvxlab.Model.initialize_model_environment` * - :ref:`Fill exogenous model data ` - *User* fills the exogenous Excel input data file(s). * - :ref:`Initialization of numerical problem(s) ` - Symbolic problem is validated, and numerical problem initialized. API: :py:meth:`~cvxlab.Model.refresh_database_and_initialize_problem` * - :ref:`Run numerical problem(s) ` - Numerical problem is solved. API: :py:meth:`~cvxlab.Model.run_model` * - :ref:`Export model results ` - Results are exported from CVXlab Model to the SQLite database. API: :py:meth:`~cvxlab.Model.load_results_to_database` .. rubric:: Step by step description of the CVXlab modeling workflow - :ref:`conceptual-model-definition`: the whole CVXlab modeling process must be grounded on a solid conceptualization and mathematical definition of the problem to be solved. This step consists in defining the fundamental model structure pen-on-paper. The following items must be defined: .. list-table:: :header-rows: 1 :widths: 35 65 * - Conceptual component - Description * - :ref:`defining-sets` - The dimensions of the model, defining its scope. * - :ref:`definig-data-tables-variables` - *Data tables* are collections of model data identified by one or more sets. *Variables* are symbolic items pointing to all or a subset of the data entries in those tables. * - :ref:`definig-expressions-and-problems` - *Problems* are collections of *expressions*, which are symbolic items constructed from defined *variables* and *symbolic operators*. More than one problem can be stated, each defined as a system of linear equalities or as a convex optimization problem. Once the problem is well defined, the CVXlab modeling process can start. - :ref:`generation-of-model-directory`: a model directory is generated based on a predefined template, and it contains all the necessary files to transpose the conceptual model into a CVXlab model instance. Setup files can be generated as *YAML* or *Excel* formats, depending on user's preference. Eventually, *Python template modules* can be included in the model directory to extend CVXlab functionalities with user-defined symbolic operators (``user_defined_operators.py``) and user-defined constants data types (``user_defined_constants.py``). API: :py:func:`cvxlab.create_model_dir` - :ref:`fill-model-setup-files`: the user translates the conceptual model into the setup files available in the model directory. The setup information can be provided either in *YAML* format (as three separate files) or in *Excel* format (default ``model_settings.xlsx`` workbook with three tabs), as summarized below: .. list-table:: :header-rows: 1 :widths: 40 30 30 * - Setup component - YAML file - ``model_settings.xlsx`` tab * - Structure of sets - ``structure_sets.yml`` - ``structure_sets`` * - Structure of data tables and variables - ``structure_variables.yml`` - ``structure_variables`` * - Mathematical problem - ``problem.yml`` - ``problem`` Optionally, user-defined symbolic operators and constant data types can be implemented in dedicated Python template modules included in the model directory. - :ref:`generate-model-class-instance`: the instance of the Model class is generated through the *Model class constructor*, which translates the information provided by setup file/s into a Python object. The model instance includes all the necessary information and the APIs to generate and to solve numerical problems, and to handle exogenous/endogeous model data. The model instance generation automatically triggers the generation of the Excel file ``sets.xlsx`` to be filled with the *sets coordinates* (see :ref:`Fill sets data (model coordinates) `). During the Model class instantiation, a comprehensive validation of model structure is performed to ensure that all model components are correctly and consistently defined. API (class constructor): :py:class:`cvxlab.Model` - :ref:`fill-sets-data`: the user fills the sets Excel file with the elements belonging to each set, defined as *coordinates*. The Excel file includes one tab per each set, where user can eventually specify other set methibutes, such as set *filters* and *aggregation* categories. Once this step is completed, all the necessary information defining the model structure is available, and the data structures can be generated. - :ref:`data-structures-init`: this step imports the set coordinates into the model instance, checks that variables are defined consistently, and generates the initial model data structures. .. list-table:: :header-rows: 1 :widths: 32 68 * - Output - Description * - Blank SQLite database - Relational database containing *set tables* and *data tables*, linked by unique relationships. * - Blank Excel input data file(s) - One or more Excel files, depending on user preference, containing the *exogenous data tables* in normalized form to be filled by the user (see :ref:`Fill exogenous model data `). API: :py:meth:`~cvxlab.Model.initialize_model_environment` - :ref:`fill-exogenous-data`: user fills the Excel input data file/s with the exogenous data tables values. In case of models with large amount of data, this step can be critical, time consuming and highly prone to errors. To facilitate this task, CVXlab provides features to limit the amount of data entries to be filled (as example: in settings, a default value can be defined for data table entries in case of missing data). Once this step is completed, the numerical model can be generated and solved. - :ref:`numerical-problem-init`: this step includes fetching exogenous data from excel input data file/s to the model SQLite database, loading symbolic problem/s from model settings, and generating the numerical problem/s (as *CVXPY* problem object). During the problem generation step, a comprehensive validation of symbolic problem and model data is performed to ensure that the symbolic problem is consistently defined, and that all necessary data entries are correctly provided. API: :py:meth:`~cvxlab.Model.refresh_database_and_initialize_problem` - :ref:`numerical-problem-run`: in this step, the numerical problem/s are solved based on user settings, including selection of the adopted solver, the related verbosity level, and other numerical settings. API: :py:meth:`~cvxlab.Model.run_model` - :ref:`export-model-results`: in case numerical problem/s have successfully solved, numerical results are exported to endogenous data tables of the SQLite database. API: :py:meth:`~cvxlab.Model.load_results_to_database` .. _model_generation_from_existing: Model generation from existing settings/data -------------------------------------------- The CVXlab modeling workflow for generating a model from existing settings/data is summarized in the table below. Each **workflow step** is described in the **description** column, and it links to the corresponding detailed page. .. list-table:: CVXlab modeling workflow from existing settings/data :header-rows: 1 :widths: 40 58 * - Workflow step - Description * - :ref:`Generate Model class instance ` - CVXlab Model instance is created, and conceptual model validated. ``sets.xlsx`` file is assumed to be already filled by the user. API: :py:class:`cvxlab.Model` * - :ref:`Fill exogenous model data ` - (**OPTIONAL**) *User* updates/fills the exogenous Excel input data file(s). * - :ref:`Initialization of numerical problem(s) ` - (**OPTIONAL**) SQLite database is refreshed, symbolic problem is validated, and numerical problem initialized. API: :py:meth:`~cvxlab.Model.refresh_database_and_initialize_problem` * - :ref:`Run numerical problem(s) ` - Numerical problem is solved. API: :py:meth:`~cvxlab.Model.run_model` * - :ref:`Export model results ` - Results are exported from CVXlab Model to the SQLite database. API: :py:meth:`~cvxlab.Model.load_results_to_database` .. rubric:: Step by step description of the CVXlab modeling workflow - :ref:`generate-model-class-instance`: the instance of the Model class is generated through the *Model class constructor* as the first step, specifying that the model relies on existing data (i.e. by passing ``use_existing_data=True`` to the constructor). Beside setup file/s (see :ref:`Fill model setup file/s `), other data structures must be present in the model directory, including the sets Excel file filled with the related coordinates (see :ref:`Fill sets data (model coordinates) `), the blank SQLite database (with set tables and data tables, see :ref:`Initialization of data structures `), and the Excel input data directory (see :ref:`Fill exogenous model data `). The Model constructor checks and validate the model directory structure and the presence of all necessary files. Then, it loads the structures of sets, data tables and variables in the index, and fetches problems from the setup file/s. Then it loads coordinates from the sets Excel file to the model instance. Finally, it initializes numerical problem/s and fetches exogenous data from the SQLite database. The model instance includes all the necessary information and the APIs to generate and to solve numerical problems, and to handle model data. API (class constructor): :py:class:`cvxlab.Model` - :ref:`fill-exogenous-data`: (**optional**) in this step, user may update exogenous model data in the Excel file/s. This step is especially useful in case of multiple consecutive model runs, where only exogenous data are changed. If needed, one or more blank input data file/s can be re-generated from the blank SQLite database. API: :py:meth:`~cvxlab.Model.generate_input_data_files` - :ref:`numerical-problem-init`: (**optional**) this step needs to be made in case of exogenous data have updated (see :ref:`Fill exogenous model data `). It fetches exogenous data from the Excel input data files to the SQLite database, updating data in numerical problem/s variables. API: :py:meth:`~cvxlab.Model.refresh_database_and_initialize_problem` - :ref:`numerical-problem-run`: in this step, the numerical problem/s are solved based on user settings, including the adopted solver, the related verbosity level, and other settings. API: :py:meth:`~cvxlab.Model.run_model` - :ref:`export-model-results`: in case numerical problem(s) have successfully solved, numerical results are exported to the endogenous data tables of the SQLite database. API: :py:meth:`~cvxlab.Model.load_results_to_database` .. _utilities: Utilities --------- During the modeling process, or once the model is generated and solved, the user may need to inspect model **properties**, or doing some basic operations on the database. CVXlab provides a set of **utilities functions** to facilitate these tasks, reported in the tables below. .. list-table:: CVXlab Model instance inspection utilities :header-rows: 1 :widths: 30 68 * - API - Description * - :py:attr:`~cvxlab.Model.sets` - **Property** reporting the list of model set keys. * - :py:attr:`~cvxlab.Model.data_tables` - **Property** reporting the list of model data table keys. * - :py:attr:`~cvxlab.Model.variables` - **Property** reporting the list of model variable keys. * - :py:attr:`~cvxlab.Model.is_problem_solved` - **Property** indicating the status of the numerical problem. * - :py:meth:`~cvxlab.Model.set` - **Method** allowing inspection of a specific set, including related coordinates and other attributes. * - :py:meth:`~cvxlab.Model.variable` - **Method** allowing inspection of a specific variable, including related data table, shape, filters and other attributes. *Other helper methods* allow to perform basic operations on the model instance and on the SQLite database, summarized in the table below. .. list-table:: CVXlab Model instance helper utilities :header-rows: 1 :widths: 30 68 * - API - Description * - :py:meth:`~cvxlab.Model.reinitialize_sqlite_database` - **Method** allowing regeneration of a blank SQLite database from the current model structure (sets and data tables). Useful to reset the database to a clean state. * - :py:meth:`~cvxlab.Model.check_model_results` - **Method** allowing comparison of the model SQLite database with a reference SQLite database, to verify that results are coherent with expected values. * - :py:meth:`~cvxlab.Model.update_sets_tables` - **Method** allowing update of sets tables in the SQLite database based on the current model structure and coordinates. Useful after changing set coordinates, for example to add aggregation categories. .. toctree:: :maxdepth: 1 :hidden: :caption: Modeling workflow steps user_guide_steps/conceptual_model_definition user_guide_steps/model_directory_generation user_guide_steps/fill_model_setup_files user_guide_steps/generate_model_instance user_guide_steps/fill_sets_data user_guide_steps/data_structures_init user_guide_steps/fill_exogenous_data user_guide_steps/numerical_problem_init user_guide_steps/numerical_problem_run user_guide_steps/export_model_results