Note
Click here to download the full example code
Deterministic validation case on a mock model¶
from __future__ import annotations
import logging
from gemseo.datasets.io_dataset import IODataset
from vimseo import EXAMPLE_RUNS_DIR
from vimseo.api import activate_logger
from vimseo.api import create_model
from vimseo.core.model_settings import IntegratedModelSettings
from vimseo.tools.io.reader_file_dataframe import ReaderFileDataFrame
from vimseo.tools.io.reader_file_dataframe import ReaderFileDataFrameSettings
from vimseo.tools.validation_case.validation_case import DeterministicValidationCase
from vimseo.tools.validation_case.validation_case import (
DeterministicValidationCaseInputs,
)
from vimseo.tools.validation_case.validation_case import (
DeterministicValidationCaseSettings,
)
activate_logger(level=logging.INFO)
The model to validate is created. this model contains a 1-D vector input x3,
of variable length.
model_name = "MockModelPersistent"
load_case = "LC1"
model = create_model(
model_name,
load_case,
model_options=IntegratedModelSettings(
directory_archive_root=EXAMPLE_RUNS_DIR / "archive/validation_case",
directory_scratch_root=EXAMPLE_RUNS_DIR / "scratch/validation_case",
cache_file_path=EXAMPLE_RUNS_DIR
/ f"caches/validation_case/{model_name}_{load_case}.hdf",
),
)
Synthetic reference data are read from a csv file.
The input validation space contains the x3 vector.
x3 has different lengths in the csv file (some boxes are left blank to
that this component has no value.
Once this csv data is loaded, the blank components are filled with NaNs.
reference_data = (
ReaderFileDataFrame()
.execute(
settings=ReaderFileDataFrameSettings(
file_name="reference_data_vector_different_lengths.csv",
variable_names=["x3", "x1", "x2", "y4"],
variable_names_to_group_names={
"x1": IODataset.INPUT_GROUP,
"x2": IODataset.INPUT_GROUP,
"x3": IODataset.INPUT_GROUP,
"y4": IODataset.OUTPUT_GROUP,
},
variable_names_to_n_components={
"x3": 3,
},
),
)
.dataset
)
print(reference_data)
Out:
INFO - 16:52:16: Working directory is /home/sebastien.bocquet/PycharmProjects/vimseo/docs/runnable_examples/07_validation_case
GROUP inputs outputs
VARIABLE x3 x1 x2 y4
COMPONENT 0 1 2 0 0 0
0 1 2 NaN 1 6.0 3.3
1 3 4 5.0 2 7.0 14.4
The validation case tool is created and executed.
The only setting is that only output y4 is validated.
validation = DeterministicValidationCase(
working_directory="deterministic_validation_case_on_vectors"
)
validation.execute(
inputs=DeterministicValidationCaseInputs(
model=model, reference_data=reference_data
),
settings=DeterministicValidationCaseSettings(output_names=["y4"]),
)
Out:
INFO - 16:52:16: Working directory is /home/sebastien.bocquet/PycharmProjects/vimseo/docs/runnable_examples/07_validation_case/deterministic_validation_case_on_vectors
INFO - 16:52:16: Working directory is /home/sebastien.bocquet/PycharmProjects/vimseo/docs/runnable_examples/07_validation_case/deterministic_validation_case_on_vectors/CustomDOETool
INFO - 16:52:16: Found 1 entries in the cache file : /home/sebastien.bocquet/PycharmProjects/vimseo/docs/runnable_examples/model_runs/caches/validation_case/MockModelPersistent_LC1_x3_1_2.hdf node : node
INFO - 16:52:16: Found 1 entries in the cache file : /home/sebastien.bocquet/PycharmProjects/vimseo/docs/runnable_examples/model_runs/caches/validation_case/MockModelPersistent_LC1_x3_3_4_5.hdf node : node
ValidationCaseResult(metadata=ToolResultMetadata(generic={'datetime': '11-05-2026_16-52-16', 'version': '0.1.7.dev11+g45528c259'}, misc={}, settings={'input_names': [], 'output_names': ['y4'], 'metric_names': ['SquaredErrorMetric', 'RelativeErrorMetric', 'AbsoluteErrorMetric'], 'description': None}, report={'title': 'Validation of MockModelPersistent LC1'}, model=ModelDescription(name='MockModelPersistent', summary=' A toy model for testing purpose of persistent data - mono-component version', load_case=LC1(name='LC1', domain='', summary='A first mock load case.', plot_parameters=PlotParameters(curves=[]), bc_variable_names=[], load=Load(direction='', sign='', type='')), dataflow={'model_inputs': ['x1', 'x2', 'x3'], 'model_outputs': ['y1', 'y2', 'y3', 'y4', 'y5', 'error_code', 'model', 'load_case', 'description', 'job_name', 'persistent_result_files', 'n_cpus', 'date', 'cpu_time', 'user', 'machine', 'vims_git_version', 'directory_archive_root', 'directory_archive_job', 'directory_scratch_root', 'directory_scratch_job'], 'MockComponentStandalonePersistent_LC1': {'inputs': ['x1', 'x2', 'x3'], 'outputs': ['y1', 'y2', 'y3', 'y4', 'y5', 'error_code']}}, default_inputs={<InputGroupNames.NUMERICAL_VARS: 'numerical variables'>: {}, <InputGroupNames.BC_VARS: 'boundary conditions variables'>: {}, <InputGroupNames.GEOMETRICAL_VARS: 'geometrical variables'>: {'x1': [2.0], 'x2': [5.0], 'x3': [1.0, 2.0, 3.0]}, <InputGroupNames.MATERIAL_VARS: 'material variables'>: {}}, curves=[], verbose=False)), element_wise_metrics=GROUP inputs SquaredErrorMetric RelativeErrorMetric AbsoluteErrorMetric outputs ReferenceOutputs
VARIABLE x1 x2 x3 y4 y4 y4 y4 y4
COMPONENT 0 0 0 1 2 0 0 0 0 0
0 1.0 6.0 1.0 2.0 NaN 0.09 0.090909 0.3 3.0 3.3
1 2.0 7.0 3.0 4.0 5.0 5.76 0.166667 2.4 12.0 14.4, integrated_metrics=defaultdict(<class 'dict'>, {'SquaredErrorMetric': {'y4': 2.9250000000000007}, 'RelativeErrorMetric': {'y4': 0.12878787876831763}, 'AbsoluteErrorMetric': {'y4': 1.35}}), stochastic_point_results=())
The element-wise error metrics can be obtained, together with the input variables.
print(validation.result.element_wise_metrics)
# Standard validation plots can be shown. For vector inputs, each of its
# components are considered as scalar inputs. So for the input space
# ``x1`` and ``x3`` with ``x1`` a scalar, and ``x3`` a vector of length 3,
# the input variables shown in the plots are ``x1``, ``x3[0]``, ``x3[1]``, ``x3[2]``
validation.plot_results(
validation.result, metric_name="RelativeErrorMetric", output_name="y4", show=True
)
validation.save_results()
Out:
GROUP inputs SquaredErrorMetric RelativeErrorMetric AbsoluteErrorMetric outputs ReferenceOutputs
VARIABLE x1 x2 x3 y4 y4 y4 y4 y4
COMPONENT 0 0 0 1 2 0 0 0 0 0
0 1.0 6.0 1.0 2.0 NaN 0.09 0.090909 0.3 3.0 3.3
1 2.0 7.0 3.0 4.0 5.0 5.76 0.166667 2.4 12.0 14.4
INFO - 16:52:16: Working directory is /home/sebastien.bocquet/PycharmProjects/vimseo/docs/runnable_examples/07_validation_case/deterministic_validation_case_on_vectors
INFO - 16:52:17: Working directory is /home/sebastien.bocquet/PycharmProjects/vimseo/docs/runnable_examples/07_validation_case/deterministic_validation_case_on_vectors
/home/sebastien.bocquet/PycharmProjects/vimseo/src/vimseo/utilities/datasets.py:376: FutureWarning:
Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
/home/sebastien.bocquet/PycharmProjects/vimseo/src/vimseo/utilities/datasets.py:376: FutureWarning:
Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
/home/sebastien.bocquet/PycharmProjects/vimseo/src/vimseo/utilities/datasets.py:376: FutureWarning:
Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
/home/sebastien.bocquet/PycharmProjects/vimseo/src/vimseo/utilities/datasets.py:376: FutureWarning:
Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
/home/sebastien.bocquet/PycharmProjects/vimseo/src/vimseo/utilities/datasets.py:376: FutureWarning:
Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
/home/sebastien.bocquet/PycharmProjects/vimseo/src/vimseo/utilities/datasets.py:376: FutureWarning:
Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
/home/sebastien.bocquet/PycharmProjects/vimseo/src/vimseo/utilities/datasets.py:376: FutureWarning:
Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
/home/sebastien.bocquet/PycharmProjects/vimseo/src/vimseo/utilities/datasets.py:376: FutureWarning:
Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
/home/sebastien.bocquet/PycharmProjects/vimseo/src/vimseo/utilities/datasets.py:376: FutureWarning:
Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
/home/sebastien.bocquet/PycharmProjects/vimseo/src/vimseo/utilities/datasets.py:376: FutureWarning:
Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
/home/sebastien.bocquet/PycharmProjects/vimseo/.tox/doc/lib/python3.11/site-packages/plotly/basedatatypes.py:2314: DeprecationWarning:
The append_trace method is deprecated and will be removed in a future version.
Please use the add_trace method with the row and col parameters.
INFO - 16:52:18: Working directory is /home/sebastien.bocquet/PycharmProjects/vimseo/docs/runnable_examples/07_validation_case/deterministic_validation_case_on_vectors
INFO - 16:52:18: Working directory is /home/sebastien.bocquet/PycharmProjects/vimseo/docs/runnable_examples/07_validation_case/deterministic_validation_case_on_vectors
INFO - 16:52:19: Saving result to /home/sebastien.bocquet/PycharmProjects/vimseo/docs/runnable_examples/07_validation_case/deterministic_validation_case_on_vectors/DeterministicValidationCase_result.hdf5
INFO - 16:52:19: Saving result to /home/sebastien.bocquet/PycharmProjects/vimseo/docs/runnable_examples/07_validation_case/deterministic_validation_case_on_vectors/CustomDOETool/CustomDOETool_result.hdf5
Total running time of the script: ( 0 minutes 2.648 seconds)
Download Python source code: plot_deterministic_validation_case_on_vectors.py
Download Jupyter notebook: plot_deterministic_validation_case_on_vectors.ipynb