Browse Source

completed documentation for dtmc, ctmc, mdp and ma

refactoring
hannah 5 years ago
committed by Matthias Volk
parent
commit
ef93bdd541
No known key found for this signature in database GPG Key ID: 83A57678F739FCD3
  1. 2
      doc/source/advanced_topics.rst
  2. 15
      doc/source/doc/building_ctmcs.rst
  3. 21
      doc/source/doc/building_dtmcs.rst
  4. 82
      doc/source/doc/building_mas.rst
  5. 111
      doc/source/doc/building_mdps.rst
  6. 5
      examples/building_dtmcs/01-building-dtmcs.py
  7. 49
      examples/building_mas/01-building-mas.py
  8. 45
      examples/building_mdps/01-building-mdps.py

2
doc/source/advanced_topics.rst

@ -12,8 +12,8 @@ This guide is a collection of examples meant to bridge the gap between the getti
doc/building_models
doc/building_dtmcs
doc/building_ctmcs
doc/building_mas
doc/building_mdps
doc/building_mas
doc/engines
doc/exploration
doc/reward_models

15
doc/source/doc/building_ctmcs.rst

@ -30,23 +30,22 @@ In this example, we build the transition matrix using a numpy array::
The following function call returns a sparse matrix with default row groups::
>>> transition_matrix = stormpy.build_sparse_matrix(transitions)
>>> print(transition_matrix)<
>>> print(transition_matrix)
0 1 2 3
---- group 0/3 ----
0 ( 0 1.5 0 0 ) 0
0 ( 0 1.5 0 0 ) 0
---- group 1/3 ----
1 ( 3 0 1.5 0 ) 1
1 ( 3 0 1.5 0 ) 1
---- group 2/3 ----
2 ( 0 3 0 1.5 ) 2
2 ( 0 3 0 1.5 ) 2
---- group 3/3 ----
3 ( 0 0 3 0 ) 3
3 ( 0 0 3 0 ) 3
0 1 2 3
Labeling
================
The state labeling is created analogously to the previous example::
The state labeling is created analogously to the previous example in :doc:`building_dtmcs`::
>>> state_labeling = stormpy.storage.StateLabeling(4)
>>> state_labels = {'empty', 'init', 'deadlock', 'full'}
@ -58,7 +57,7 @@ The state labeling is created analogously to the previous example::
Exit Rates
====================
Lastly, we equip every state with an exit rate::
Lastly, we initialize a list to equip every state with an exit rate::
>>> exit_rates = [1.5, 4.5, 4.5, 3.0]

21
doc/source/doc/building_dtmcs.rst

@ -9,29 +9,30 @@ As described in :doc:`../getting_started`,
Storm can be used to translate a model description e.g. in form of a prism file into a Markov chain.
Here, we use Stormpy to create the single components, to build a DTMC without parsing a model description.
We consider the previous example of the die.
We consider the previous example of the dice.
.. seealso:: `01-building-dtmcs.py <todo /examples/building_dtmcs/01-building-dtmcs.py>`
In the following we create transition matrix, the state labeling and the reward models of a DTMC.
In the following we create the transition matrix, the state labeling and the reward models of a DTMC.
First, we import stormpy::
>>> import stormpy
>>> import stormpy
Transition Matrix
=====================
We begin by creating the matrix representing the transitions in the model in terms of probabilities.
For constructing the transition matrix, we use the SparseMatrixBuilder::
>>> builder = stormpy.SparseMatrixBuilder(rows = 0, columns = 0, entries = 0, force_dimensions = False, has_custom_row_grouping = False)
>>> builder = stormpy.SparseMatrixBuilder(rows = 0, columns = 0, entries = 0, force_dimensions = False, has_custom_row_grouping = False)
Here, we start with an empty matrix to later insert more entries.
If the number of rows, columns and entries is known, the matrix can be constructed using these values.
For DTMCs each state has at most one outgoing probability distribution.
Thus, we create matrix with trivial row grouping where each group contains one row representing the state action.
Thus, we create a matrix with trivial row grouping where each group contains one row representing the state action.
In :doc:`building_mdps` we will revisit the example of the die, but extend the model with nondeterministic choice.
We specify the transitions of the model, by adding values to the matrix where the column represents the target state.
We specify the transitions of the model by adding values to the matrix where the column represents the target state.
All transitions are equipped with a probability defined by the value::
>>> builder.add_next_value(row = 0, column = 1, value = 0.5)
@ -55,12 +56,12 @@ Lastly, we add a self-loop with probability one to the final states::
... builder.add_next_value(s, s, 1)
Finally, we can build the matrix with updated row and columns count that both coincide with the number of states::
Finally, we can build the matrix::
>>> transition_matrix = builder.build(13, 13)
>>> transition_matrix = builder.build()
It should be noted that entries can only be inserted in ascending order, i.e. row by row and column by column.
Stormpy provides the possibility to build a sparse matrix using the numpy library <https://numpy.org/>
Stormpy provides the possibility to build a sparse matrix using the numpy library (https://numpy.org/ )
Instead of using the SparseMatrixBuilder, a sparse matrix can be build from a numpy array via the method stormpy.build_sparse_matrix.
Labeling
@ -77,9 +78,11 @@ In order to specify the state labeling we create an empty labeling for the given
Labels can be asociated with states. As an example, we label the state 0 with 'init'::
>>> state_labeling.add_label_to_state('init', 0)
>>> print(state_labeling.get_states('init'))
bit vector(1/13) [0]
Next, we set the associations between the remaining labels and states.::
>>> state_labeling.add_label_to_state('one', 7)

82
doc/source/doc/building_mas.rst

@ -6,9 +6,11 @@ Markov automata (MAs)
Background
=====================
... works similar as explained in as in :doc:`building_dtmcs`
We already saw the process of building CTMCs with MDPs via Stormpy in :doc:`building_ctmcs` and :doc:`building_mdps`.
In this section, we build a small Markarov automata with five states from which the first four are markovian.
Since we covered labeling and exit rates already in the previous examples we omit the description of these components.
The full example can be found here:
Build simple ma example
.. seealso:: `01-building-mas.py <todo /examples/building_mas/01-building-mas.py>`
First, we import Stormpy::
@ -17,20 +19,72 @@ First, we import Stormpy::
Transition Matrix
==================
Labeling
================
Reward models
In :doc:`building_mdps`, we used the SparseMatrixBuilder to create a matrix with a custom row grouping.
In this example, we use the numpy library:
In the beginning, we create a numpy array that will be used to build the transition matrix of our model.::
>>> import numpy as np
>>> transitions = np.array([
... [0, 1, 0, 0, 0],
... [0.8, 0, 0.2, 0, 0],
... [0.9, 0, 0, 0.1, 0],
... [0, 0, 0, 0, 1],
... [0, 0, 0, 1, 0],
... [0, 0, 0, 0, 1]], dtype='float64')
When building the matrix we define a custom row grouping by passing a list containing the starting row of each row group in ascending order::
>>> transition_matrix = stormpy.build_sparse_matrix(transitions, [0, 2, 3, 4, 5])
>>> print(transition_matrix)
0 1 2 3 4
---- group 0/4 ----
0 ( 0 1 0 0 0 ) 0
1 ( 0.8 0 0.2 0 0 ) 1
---- group 1/4 ----
2 ( 0.9 0 0 0.1 0 ) 2
---- group 2/4 ----
3 ( 0 0 0 0 1 ) 3
---- group 3/4 ----
4 ( 0 0 0 1 0 ) 4
---- group 4/4 ----
5 ( 0 0 0 0 1 ) 5
0 1 2 3 4
Markovian States
==================
Feature of MA: Markovian states.
Those are represented by a BitVector that containing the respective states::
Exit Rates
====================
>>> markovian_states = stormpy.BitVector(5, [1, 2, 3, 4])
Building the Model
====================
Now, we can collect all components::
>>> components = stormpy.SparseModelComponents(transition_matrix=transition_matrix, state_labeling=state_labeling,
markovian_states=markovian_states)
>>> components.choice_labeling = choice_labeling
>>> components.exit_rates = exit_rates
Finally, we can build the model::
>>> ma = stormpy.storage.SparseMA(components)
>>> print(ma)
--------------------------------------------------------------
Model type: Markov Automaton (sparse)
States: 5
Transitions: 8
Choices: 6
Markovian St.: 4
Max. Rate.: 12
Reward Models: none
State Labels: 2 labels
* deadlock -> 0 item(s)
* init -> 1 item(s)
Choice Labels: 2 labels
* alpha -> 1 item(s)
* beta -> 1 item(s)
--------------------------------------------------------------

111
doc/source/doc/building_mdps.rst

@ -2,10 +2,11 @@
Discrete-time Markov decision processes (MDPs)
***********************************************
Background
=====================
Same example of die knuth as in dtmc but mdp (model with nondeterminsm)
In :doc:`building_dtmcs` we modelled Knuth's model of a fair die by the means of a DTMC.
In the following we extend this model with nondeterministic choice by building a Markov Decision process.
.. seealso:: `01-building-mdps.py <todo /examples/mdps/01-building-mdps.py>`
@ -15,25 +16,115 @@ First, we import Stormpy::
Transition Matrix
=====================
more choices than states!
Since we want build a nondeterminstic model, we create a transition matrix with a custom row group for each state::
>>> builder = stormpy.SparseMatrixBuilder(rows=0, columns=0, entries=0, force_dimensions=False, has_custom_row_grouping=True, row_groups=0)
We need more than one row for the transitions starting in state 0 because a nondeterministic choice over the actions is available.
Therefore, we start a new group that will contain the rows representing actions of state 0.
Note that the row group needs to be added before any entries are added to the group::
>>> builder.new_row_group(0)
>>> builder.add_next_value(0, 1, 0.5)
>>> builder.add_next_value(0, 2, 0.5)
>>> builder.add_next_value(1, 1, 0.2)
>>> builder.add_next_value(1, 2, 0.8)
For the remaining states, we need to specify the starting rows of row groups::
>>> builder.new_row_group(2)
>>> builder.add_next_value(2, 3, 0.5)
>>> builder.add_next_value(2, 4, 0.5)
>>> builder.new_row_group(3)
>>> builder.add_next_value(3, 5, 0.5)
>>> builder.add_next_value(3, 6, 0.5)
>>> builder.new_row_group(4)
>>> builder.add_next_value(4, 7, 0.5)
>>> builder.add_next_value(4, 1, 0.5)
>>> builder.new_row_group(5)
>>> builder.add_next_value(5, 8, 0.5)
>>> builder.add_next_value(5, 9, 0.5)
>>> builder.new_row_group(6)
>>> builder.add_next_value(6, 10, 0.5)
>>> builder.add_next_value(6, 11, 0.5)
>>> builder.new_row_group(7)
>>> builder.add_next_value(7, 2, 0.5)
>>> builder.add_next_value(7, 12, 0.5)
>>> for s in range(8, 14):
... builder.new_row_group(s)
... builder.add_next_value(s, s - 1, 1)
Build::
>>> transition_matrix = builder.build()
Labeling
================
We have seen the construction of a state labeling in previous examples. Therefore we omit the description here.
Instead we focus on the choices.
Since in state 0 a nondeterministic choice over two actions is available
The number of choices is 14.
To distinguish those we can define a choice labeling::
>>> choice_labeling = stormpy.storage.ChoiceLabeling(14)
>>> choice_labels = {'a', 'b'}
>>> for label in choice_labels:
... choice_labeling.add_label(label)
We assign the label 'a' to the first action of state 0 and 'b' to the second.
Recall that those actions where defined in row one and two of the transition matrix respectively::
>>> choice_labeling.add_label_to_choice('a', 0)
>>> choice_labeling.add_label_to_choice('b', 1)
>>> print(choice_labeling)
Choice 2 labels
* a -> 1 item(s)
* b -> 1 item(s)
Reward models
==================
Reward models, length of vector coincides with number of choices::
Exit Rates
====================
>>> reward_models = {}
>>> action_reward = [0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
>>> reward_models['coin_flips'] = stormpy.SparseRewardModel(optional_state_action_reward_vector=action_reward)
Building the Model
====================
Collect components::
>>> components = stormpy.SparseModelComponents(transition_matrix=transition_matrix, state_labeling=state_labeling, reward_models=reward_models, rate_transitions=False)
>>> components.choice_labeling = choice_labeling
POMDPS
====================
Give
observations as ..
Build the model::
>>> mdp = stormpy.storage.SparseMdp(components)
>>> print(mdp)
Model type: MDP (sparse)
States: 13
Transitions: 22
Choices: 14
Reward Models: coin_flips
State Labels: 9 labels
* one -> 1 item(s)
* six -> 1 item(s)
* three -> 1 item(s)
* four -> 1 item(s)
* done -> 6 item(s)
* init -> 1 item(s)
* five -> 1 item(s)
* deadlock -> 0 item(s)
* two -> 1 item(s)
Choice Labels: 2 labels
* a -> 1 item(s)
* b -> 1 item(s)
Partially observable Markov decision process (POMDPs)
========================================================
To build a partially observable Markov decision process,
components.observations can be set to a list of numbers that defines the status of the observables in each state.

5
examples/building_dtmcs/01-building-dtmcs.py

@ -27,7 +27,7 @@ def example_building_dtmcs_01():
builder.add_next_value(s, s, 1)
# Build matrix
transition_matrix = builder.build(13, 13)
transition_matrix = builder.build()
print(transition_matrix)
# State labeling
@ -38,10 +38,11 @@ def example_building_dtmcs_01():
for label in labels:
state_labeling.add_label(label)
# Set label to state
# Set label of state 0
state_labeling.add_label_to_state('init', 0)
print(state_labeling.get_states('init'))
# Set remaining labels
state_labeling.add_label_to_state('one', 7)
state_labeling.add_label_to_state('two', 8)
state_labeling.add_label_to_state('three', 9)

49
examples/building_mas/01-building-mas.py

@ -1,9 +1,54 @@
import stormpy
import numpy as np
# hybrid_states example
def example_building_mas_01():
print('todo')
# Building the transition matrix using numpy
transitions = np.array([
[0, 1, 0, 0, 0],
[0.8, 0, 0.2, 0, 0],
[0.9, 0, 0, 0.1, 0],
[0, 0, 0, 0, 1],
[0, 0, 0, 1, 0],
[0, 0, 0, 0, 1]
], dtype='float64')
# Build matrix and define indices of row groups (ascending order)
transition_matrix = stormpy.build_sparse_matrix(transitions, [0, 2, 3, 4, 5])
print(transition_matrix)
# StateLabeling
state_labeling = stormpy.storage.StateLabeling(5)
# Add labels
state_labels = {'init', 'deadlock'}
# Set labeling of states
for label in state_labels:
state_labeling.add_label(label)
state_labeling.add_label_to_state('init', 0)
# Choice labeling
choice_labeling = stormpy.storage.ChoiceLabeling(6)
# Add labels
choice_labels = {'alpha', 'beta'}
# Set labeling of choices
for label in choice_labels:
choice_labeling.add_label(label)
choice_labeling.add_label_to_choice('alpha', 0)
choice_labeling.add_label_to_choice('beta', 1)
exit_rates = [0.0, 10.0, 12.0, 1.0, 1.0]
markovian_states = stormpy.BitVector(5, [1, 2, 3, 4])
# Collect components
components = stormpy.SparseModelComponents(transition_matrix=transition_matrix, state_labeling=state_labeling,
markovian_states=markovian_states)
components.choice_labeling = choice_labeling
components.exit_rates = exit_rates
# Build the model
ma = stormpy.storage.SparseMA(components)
print(ma)
if __name__ == '__main__':

45
examples/building_mdps/01-building-mdps.py

@ -1,13 +1,15 @@
import stormpy
import numpy as np
# Knuth's model of a fair die using only fair coins
def example_building_mdps_01():
nr_states = 13
nr_choices = 14
# Building the transition matrix using
builder = stormpy.SparseMatrixBuilder(rows = 0, columns = 0, entries = 0, force_dimensions = False, has_custom_row_grouping = True, row_groups = 0)
# Transition matrix with custom row grouping: nondeterministic choice over the actions available in states
builder = stormpy.SparseMatrixBuilder(rows=0, columns=0, entries=0, force_dimensions=False,
has_custom_row_grouping=True, row_groups=0)
# New row group, for actions of state 0
builder.new_row_group(0)
@ -15,48 +17,46 @@ def example_building_mdps_01():
builder.add_next_value(0, 2, 0.5)
builder.add_next_value(1, 1, 0.2)
builder.add_next_value(1, 2, 0.8)
# New row group, for actions of state 1
# State 1
builder.new_row_group(2)
builder.add_next_value(2, 3, 0.5)
builder.add_next_value(2, 4, 0.5)
# New row group, for actions of state 2
# State 2
builder.new_row_group(3)
builder.add_next_value(3, 5, 0.5)
builder.add_next_value(3, 6, 0.5)
# New row group, for actions of state 3
# State 3
builder.new_row_group(4)
builder.add_next_value(4, 7, 0.5)
builder.add_next_value(4, 1, 0.5)
# New row group, for actions of state 4
# State 4
builder.new_row_group(5)
builder.add_next_value(5, 8, 0.5)
builder.add_next_value(5, 9, 0.5)
# New row group, for actions of state 5
# State 5
builder.new_row_group(6)
builder.add_next_value(6, 10, 0.5)
builder.add_next_value(6, 11, 0.5)
# New row group, for actions of state 6
# State 6
builder.new_row_group(7)
builder.add_next_value(7, 2, 0.5)
builder.add_next_value(7, 12, 0.5)
# final states
for s in range(8,14):
# Add transitions for the final states
for s in range(8, 14):
builder.new_row_group(s)
builder.add_next_value(s, s-1, 1)
# Build transition matrix, set overridden_row_count = nr_states, no row group count since nondet otherwise set overrridenRowGroupCount
transition_matrix = builder.build(nr_choices, nr_states)
builder.add_next_value(s, s - 1, 1)
transition_matrix = builder.build()
# State labeling
state_labeling = stormpy.storage.StateLabeling(nr_states)
# Add labels
labels = {'init','one', 'two', 'three', 'four', 'five', 'six', 'done', 'deadlock'}
labels = {'init', 'one', 'two', 'three', 'four', 'five', 'six', 'done', 'deadlock'}
for label in labels:
state_labeling.add_label(label)
# Set label for single states
# Set labeling of states
state_labeling.add_label_to_state('init', 0)
state_labeling.add_label_to_state('one', 7)
state_labeling.add_label_to_state('two', 8)
@ -75,18 +75,20 @@ def example_building_mdps_01():
for label in choice_labels:
choice_labeling.add_label(label)
# Set label for single choice
# Set labels
choice_labeling.add_label_to_choice('a', 0)
choice_labeling.add_label_to_choice('b', 1)
print(choice_labeling)
# Reward models
reward_models = {}
# Create a vector representing the state-action rewards.
# Create a vector representing the state-action rewards
action_reward = [0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
reward_models['coin_flips'] = stormpy.SparseRewardModel(optional_state_action_reward_vector = action_reward)
reward_models['coin_flips'] = stormpy.SparseRewardModel(optional_state_action_reward_vector=action_reward)
# Collect components
components = stormpy.SparseModelComponents(transition_matrix=transition_matrix, state_labeling=state_labeling, reward_models=reward_models, rate_transitions = False)
components = stormpy.SparseModelComponents(transition_matrix=transition_matrix, state_labeling=state_labeling,
reward_models=reward_models, rate_transitions=False)
components.choice_labeling = choice_labeling
# Build the model
@ -94,6 +96,5 @@ def example_building_mdps_01():
print(mdp)
if __name__ == '__main__':
example_building_mdps_01()
example_building_mdps_01()
Loading…
Cancel
Save