# Evaluation of Execution Results

The execution result returned by the {py:func}`~amplify.solve` function contains information about the solutions, model conversion, and execution time.
This page explains how to obtain and use this information.

The following is an example of obtaining the optimization result for a model consisting of an objective function and a constraint using the {py:class}`~amplify.AmplifyAEClient`.

```python
from datetime import timedelta
from amplify import VariableGenerator, equal_to, AmplifyAEClient, solve

# Create an array of decision variables
gen = VariableGenerator()
q = gen.array("Binary", 5)

# Create an objective function and a constraint
objective = q[0] * q[1] - q[2]
constraint = equal_to(q[0] + q[1] + q[2], 1)

# Define a model
model = objective + constraint

# Create a solver client
client = AmplifyAEClient()
# client.token = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
client.parameters.time_limit_ms = timedelta(milliseconds=1000)

# Obtaining the result of the run
result = solve(model, client)
```

```{testcode}
:hide:
import os

from datetime import timedelta
from amplify import VariableGenerator, equal_to, AmplifyAEClient, solve

# Create an array of decision variables
gen = VariableGenerator()
q = gen.array("Binary", 5)

# Create an objective function and a constraint
objective = q[0] * q[1] - q[2]
constraint = equal_to(q[0] + q[1] + q[2], 1)

# Define a model
model = objective + constraint

# Create a solver client
client = AmplifyAEClient()
client.url = os.getenv("TEST_AE_URL", "")
client.token = os.getenv("TEST_AE_TOKEN", "")
client.parameters.time_limit_ms = timedelta(milliseconds=1000)

# Obtaining the result of the run
result = solve(model, client)
```

## The Result class

The {py:func}`~amplify.solve` function returns an instance of the {py:class}`~amplify.Result` class. The following explains how to obtain information about solutions, model conversion, and execution time from it.

(result-info-on-solutions)=

### Information on the solutions

The {py:attr}`~amplify.Result.solutions` attribute of the {py:class}`~amplify.Result` class stores the solutions to the input model. The {py:attr}`~amplify.Result.solutions` attribute is an instance of the {py:class}`~amplify.Result.SolutionList` class, which behaves like a list with the {py:class}`~amplify.Result.Solution` class as an element.

```{doctest}
>>> type(result.solutions[0])
<class 'amplify.Result.Solution'>
```

The {py:class}`~amplify.Result.Solution` class represents a solution and has the following attributes.

```{list-table}
:header-rows: 1

- * Attribute name
  * Type
  * Summary
- * {py:attr}`~amplify.Result.Solution.objective`
  * {py:class}`float`
  * Objective function value.
- * {py:attr}`~amplify.Result.Solution.values`
  * {py:class}`amplify.Values`
  * Values of variables in the solution.
- * {py:attr}`~amplify.Result.Solution.feasible`
  * {py:class}`bool`
  * Whether the solution meet the constraints.
- * {py:attr}`~amplify.Result.Solution.time`
  * {py:class}`datetime.timedelta`
  * The timestamp at which the solution is obtained.
```

The {py:class}`~amplify.Values` class is the type of the {py:class}`~amplify.Result.Solution.values` attribute. It represents solution values and acts like a dictionary with variables as keys and solution values as values.

```{doctest}
>>> solution = result.solutions[0]
>>> solution.values
Values({Poly(q_0): 0, Poly(q_1): 0, Poly(q_2): 1})
>>> solution.values[q[0]]
0.0
>>> solution.values[q[2]]
1.0
```

The variable array and polynomial classes have the {py:meth}`~amplify.Poly.evaluate` method, and the constraint class has the {py:meth}`~amplify.Constraint.is_satisfied` method. Use them to assign the solution values represented by the {py:class}`~amplify.Values` class to the variable array, polynomial or constraint conditions. See [Decision variable evaluation](#variable-evaluation), [Penalty evaluation](#polynomial-evaluation), and [Constraint evaluation](#constraint-evaluation) for more details.

The {py:class}`~amplify.Result` class provides several shortcuts to accessing the solution besides the {py:attr}`~amplify.Result.solutions` attribute. First, the {py:attr}`~amplify.Result.best` attribute of the {py:class}`~amplify.Result` class provides the best solution. Accessing the result directly from the {py:class}`~amplify.Result` class by index is also possible.

```{doctest}
>>> result.best.values
Values({Poly(q_0): 0, Poly(q_1): 0, Poly(q_2): 1})
>>> len(result)
1
>>> result[0].values
Values({Poly(q_0): 0, Poly(q_1): 0, Poly(q_2): 1})
```

### Information on the model conversion

The following attributes can obtain information about the model conversion. See [](conversion.md) for details.

```{list-table}
* - {py:attr}`~amplify.Result.intermediate`
  - The information on the intermediate model
* - {py:attr}`~amplify.Result.embedding`
  - The information on the graph embedding
```

### Information on the response of the solver

The {py:attr}`~amplify.solve` function calls the `solve(...)` method of a solver client to run the solver after the model conversion and graph embedding are complete. The object returned by the solver client can be obtained from the {py:attr}`~amplify.Result.client_result` attribute.

```{doctest}
>>> type(result.client_result)
<class 'amplify.AmplifyAEClient.Result'>
```

The type of the {py:attr}`~amplify.Result.client_result` depends on the type of the solver client. See [](solvers.md) and its subpages for more information about the result type of each solver client.

### Information on the execution time

You can obtain information about execution times through the following attributes. See [](timing.md) for more information.

```{list-table}
* - {py:attr}`~amplify.Result.total_time`
  - Time taken to {py:func}`~amplify.solve` 
* - {py:attr}`~amplify.Result.response_time`
  - Time between sending a request to the solver and receiving a response
* - {py:attr}`~amplify.Result.execution_time`
  - Time spent by the solver in optimization
```

(variable-evaluation)=

## Decision variable evaluation

The solution values can be assigned to a variable array used in the formulation. The assigned result is returned as a NumPy array with the same {py:attr}`~numpy.ndarray.shape` as the variable array.

The following performs the assignment to evaluate a variable array. The {py:attr}`~amplify.Result.Solution.values` attribute of the {py:class}`~amplify.Result.Solution` object is passed to the {py:meth}`~amplify.PolyArray.evaluate` method of the {py:class}`~amplify.PolyArray` class.

```{tip}
To obtain the solution in the result, retrieve the best result of the run with the {py:attr}`~amplify.Result.best` attribute. You can also access the elements in the same way as for a list.
```

```{doctest}
>>> print(result.best.values)
{q_0: 0, q_1: 0, q_2: 1}
>>> q_values = q.evaluate(result.best.values)
>>> print(q_values)
[0. 0. 1. 0. 0.]
```

The formulation may not use a variable, as in `q[3]`{l=python} or `q[4]`{l=python} above. The SDK then assigns it one of its possible values by default. In the above example, the Amplify SDK assigns `0`{l=python} as the default value for the binary variable.

If no evaluation is performed, the `default` keyword argument to the {py:meth}`~amplify.PolyArray.evaluate` method can change the value used. If you give the `default` keyword argument as a number, the SDK assigns that value to the variables not passed to the solver.

```{testcode}
:hide:
q_values = q.evaluate(result.best.values, default=3)
```

```{doctest}
>>> q_values = q.evaluate(result.best.values, default=3)    # doctest: +SKIP
Warning: Substituting variable q_3 with 3 is out of bounds.
Warning: Substituting variable q_4 with 3 is out of bounds.
>>> print(q_values)
[0. 0. 1. 3. 3.]
```

```{note}
A warning is printed if a value outside the bounds of the variable is given, such as `default=3`{l=python}, as shown above.
```

If the `default` keyword argument is {py:obj}`None`, variables not passed to the solver remain as they are. Only in this case the {py:meth}`~amplify.PolyArray.evaluate` method return a {py:class}`~amplify.PolyArray`.

```{doctest}
>>> q_values = q.evaluate(result.best.values, default=None)
>>> print(q_values)
[  0,   0,   1, q_3, q_4]
```

```{attention}
Some variables may not be passed to the solver even if they are included in the model. This is because model conversions such as penalty function generation or graph embedding can cause terms to cancel each other out.
```

(polynomial-evaluation)=

## Polynomial evaluation

The {py:func}`~amplify.solve` function returns a solution. The {py:attr}`~amplify.Result.Solution.objective` attribute of the {py:class}`~amplify.Result.Solution` object gives the result of evaluating the objective function of the input model with that solution.

```{doctest}
>>> solution = result.best
>>> solution.objective
-1.0
```

You may want to evaluate a polynomial other than the objective function with the solution returned by the solver. This happens, for example, when the objective function is the sum of multiple polynomials. In this case, we pass the {py:attr}`~amplify.Result.Solution.values` attribute of the {py:class}`~amplify.Result.Solution` object to {py:class}`~amplify.Poly`'s {py:meth}`~amplify.Poly.evaluate` method, as we would evaluate an array of variables.

```{doctest}
>>> objective_1 = q[0] * q[1] # The first term of the objective function
>>> objective_1.evaluate(solution.values)
0.0
```

```{hint}
The behavior when the polynomial contains variables not passed to the solver is similar to the {py:meth}`~amplify.PolyArray.evaluate` method of the {py:class}`~amplify.PolyArray` class. The `default` keyword argument can change this behavior.
```

(constraint-evaluation)=

## Constraint evaluation

If you want to know if the resulting solution satisfies the constraints, you can check with the {py:attr}`~amplify.Result.Solution.feasible` attribute of the {py:class}`~amplify.Result.Solution` class.

```{doctest}
>>> solution.feasible
True
```

By default, this will always be {py:obj}`True` because the {py:func}`~amplify.solve` function retrieves only solutions that satisfy all constraints in the model. To let the solver retrieve solutions that do not satisfy the constraints, change this behavior in one of two ways. Pass a {py:class}`bool` to the `filter_solution` keyword argument of the {py:func}`~amplify.solve` function upfront, or set {py:attr}`~amplify.Result.filter_solution` of the {py:class}`~amplify.Result` class to {py:obj}`False` afterward.

Let's test this by adding a constraint to the model that cannot be satisfied inconsistently.

```{testcode}
# Create an objective function and constraints
objective = q[0] * q[1] - q[2]
constraint1 = equal_to(q[0] + q[1] + q[2], 1, label="sum equals one")
constraint2 = equal_to(q[0] + q[1] + q[2], 2, label="sum equals two")

# Define a model (with conflicting constraints)
model = objective + constraint1 + constraint2

# Get the result of the run
result = solve(model, client)
```

You cannot retrieve the solution from {py:class}`~amplify.Result` if the solution filter is enabled.

```python
>>> result.best.feasible
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: result has no feasible solution
```

Turning off the solution filter allows us to obtain solutions that do not satisfy the constraints.

```{doctest}
>>> result.filter_solution = False
>>> result.best.feasible
False
```

Suppose you obtained a solution that does not satisfy a constraint for some reason, such as model configuration, penalty function weights, or solver settings. In that case, it may be desirable to determine which constraint was not satisfied.

To check whether the constraint conditions are satisfied, pass the solution returned by the {py:func}`~amplify.solve` function to the {py:meth}`~amplify.Constraint.is_satisfied` method of the {py:class}`~amplify.Constraint` class.

```python
>>> constraint1.is_satisfied(result.best.values)
True
>>> constraint2.is_satisfied(result.best.values)
False
```

In the example above, we see that `constraint2` could not be satisfied.

We can also mechanically identify constraints from the list of constraints in the model that are not satisfied, as follows. This method is useful when adjusting and rerunning the penalty function weights.

```python
>>> list(c for c in model.constraints if not c.is_satisfied(result.best.values))
[Constraint({conditional: q_0 + q_1 + q_2 == 2, weight: 1, label: "sum equals two"})]
```