Skip to content

Fix display of the runnable scenario for protected values#251

Merged
n-osborne merged 2 commits into
ocaml-gospel:mainfrom
n-osborne:fix-runnable-scenario
Oct 2, 2024
Merged

Fix display of the runnable scenario for protected values#251
n-osborne merged 2 commits into
ocaml-gospel:mainfrom
n-osborne:fix-runnable-scenario

Conversation

@n-osborne

Copy link
Copy Markdown
Collaborator

Fixes #248

As suggested in the issue, the PR add a variant with three constructors to Ortac_runtime_qcheck_stm making it possible to pass the information whether the result is a value, a protected value or an exception from the generated tests to the runtime.

We don't really have tests for failing tests, which explains why we didn't spot the bug earlier. I'll open an issue for that, to keep this PR small and focused on the fix.

@n-osborne

Copy link
Copy Markdown
Collaborator Author

We now have this behaviour in case of a test failure (here adding let get a i = get a (i+1) to plugins/qcheck-stm/test/array.ml):

λ dune build @launchtests
File "plugins/qcheck-stm/test/dune.inc", lines 49-52, characters 0-75:
49 | (rule
50 |  (alias launchtests)
51 |  (action
52 |   (run %{dep:array_stm_tests.exe} -v)))
random seed: 276043608
generated error fail pass / total     time test name
[✗]    2    0    1    1 / 1000     0.0s Array STM tests (generating)

--- Failure --------------------------------------------------------------------

Test Array STM tests failed (15 shrink steps):

   protect (fun () -> set <sut> 0 '\186')
   protect (fun () -> get <sut> 0)


+++ Messages ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Messages for test Array STM tests:

Gospel specification violation in function get

  File "array.mli", line 12, characters 12-37:
    a = List.nth t.contents i
  
when executing the following sequence of operations:

  open Array
  let protect f = try Ok (f ()) with e -> Error e
  let sut0 = make 16 'a'
  let _ = protect (fun () -> set sut0 0 '\186')
  (* returned Ok (()) *)
  let r = protect (fun () -> get sut0 0)
  assert (r = Ok '\186')
  (* returned Ok ('a') *)
  

================================================================================
failure (1 tests failed, 0 tests errored, ran 1 tests)

@n-osborne n-osborne force-pushed the fix-runnable-scenario branch from 21821ba to ed755f5 Compare September 25, 2024 14:08
@n-osborne

Copy link
Copy Markdown
Collaborator Author

Rebased. Ready for review.

@nikolaushuber

Copy link
Copy Markdown
Contributor

Not directly related, but since I stumbled upon it again during the review of this PR, there seems to be an issue with showing inbuilt types as return types: #260

Otherwise this PR LGTM :)

@n-osborne

Copy link
Copy Markdown
Collaborator Author

Thanks! Merging.

@n-osborne n-osborne merged commit 95594be into ocaml-gospel:main Oct 2, 2024
@n-osborne n-osborne deleted the fix-runnable-scenario branch October 2, 2024 07:28
n-osborne added a commit to n-osborne/opam-repository that referenced this pull request Oct 8, 2024
This release brings a number of new features and improvements:

- New features:
  + [ocaml#247](ocaml-gospel/ortac#247): Generated tests cover now functions with multiple SUT arguments
  + [ocaml#253](ocaml-gospel/ortac#253): Generated tests cover now functions returning SUT values
  + [ocaml#259](ocaml-gospel/ortac#259): Generated tests can be run in a separated process with a timeout
- Improvements:
  + [ocaml#245](ocaml-gospel/ortac#245): Fix the analysis of function signature to explicitly not support SUTs inside
    another type
  + [ocaml#251](ocaml-gospel/ortac#251): Fix the display of the runnable scenario for value returned by a function
    that could have raised an exception
n-osborne added a commit to n-osborne/opam-repository that referenced this pull request Oct 8, 2024
This release brings a number of new features and improvements:

- New features:
  + [ocaml#247](ocaml-gospel/ortac#247): Generated tests cover now functions with multiple SUT arguments
  + [ocaml#253](ocaml-gospel/ortac#253): Generated tests cover now functions returning SUT values
  + [ocaml#259](ocaml-gospel/ortac#259): Generated tests can be run in a separated process with a timeout
- Improvements:
  + [ocaml#245](ocaml-gospel/ortac#245): Fix the analysis of function signature to explicitly not support SUTs inside
    another type
  + [ocaml#251](ocaml-gospel/ortac#251): Fix the display of the runnable scenario for value returned by a function
    that could have raised an exception
n-osborne added a commit to n-osborne/opam-repository that referenced this pull request Oct 10, 2024
This release brings a number of new features and improvements:

- New features:
  + [ocaml#247](ocaml-gospel/ortac#247): Generated tests cover now functions with multiple SUT arguments
  + [ocaml#253](ocaml-gospel/ortac#253): Generated tests cover now functions returning SUT values
  + [ocaml#259](ocaml-gospel/ortac#259): Generated tests can be run in a separated process with a timeout
- Improvements:
  + [ocaml#245](ocaml-gospel/ortac#245): Fix the analysis of function signature to explicitly not support SUTs inside
    another type
  + [ocaml#251](ocaml-gospel/ortac#251): Fix the display of the runnable scenario for value returned by a function
    that could have raised an exception
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[qcheck-stm] Incorrect runnable scenario on normal behaviour when function could have raised an exception

2 participants