#29962 closed enhancement (fixed)
Introduce randomseed option to allow fuzzing of doctests
Reported by:  ghkliem  Owned by:  

Priority:  major  Milestone:  sage9.2 
Component:  doctest framework  Keywords:  doctests, random 
Cc:  slelievre  Merged in:  
Authors:  Jonathan Kliem  Reviewers:  Markus Wageringel, Matthias Koeppe 
Report Upstream:  N/A  Work issues:  
Branch:  1d99129 (Commits, GitHub, GitLab)  Commit:  
Dependencies:  Stopgaps: 
Description (last modified by )
This is the first step towards #29935.
We introduce an option for doctests: randomseed
.
This allows specifying which seed to use for tests involving randomness.
The seed is displayed in the test log:
sage t long randomseed=9876543210 src/sage/all.py ... Doctesting 1 file. sage t long randomseed=9876543210 src/sage/all.py [16 tests, 0.73 s]  All tests passed!  Total time for all tests: 0.8 seconds cpu time: 0.7 seconds cumulative wall time: 0.7 seconds
which makes it easy to rerun tests with the same seed.
The seed defaults to 0
for now:
sage t long src/sage/all.py ... Doctesting 1 file. sage t long randomseed=0 src/sage/all.py [16 tests, 0.73 s]  All tests passed!  Total time for all tests: 0.8 seconds cpu time: 0.7 seconds cumulative wall time: 0.7 seconds
but the plan in #29935 is to eventually have the random seed itself picked at random by default.
Change History (23)
comment:1 Changed 18 months ago by
 Branch set to public/29962
 Commit set to 998b1b94ce1289ea92451a86e5f6191c37eaeb5a
 Status changed from new to needs_review
comment:2 Changed 18 months ago by
 Commit changed from 998b1b94ce1289ea92451a86e5f6191c37eaeb5a to b6a5dc7c96125b6140e56feafb82dd13599998e7
Branch pushed to git repo; I updated commit sha1. New commits:
23ed583  fix doctest in hyperbolic_space/hyperbolic_point

5283dc4  use abs tol flag

7b244c0  modify doctests to the extend that they hold with fuzz

228f379  Merge branch 'public/29936' of git://trac.sagemath.org/sage into public/29962

5c7e562  fix double description of hypercube

e1bf211  remove set_random_seed

0e7a998  Merge branch 'public/29904' of git://trac.sagemath.org/sage into public/29962

b6a5dc7  fix random test in geometry/linear_expression

comment:3 Changed 18 months ago by
 Commit changed from b6a5dc7c96125b6140e56feafb82dd13599998e7 to 998b1b94ce1289ea92451a86e5f6191c37eaeb5a
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
comment:4 Changed 18 months ago by
 Commit changed from 998b1b94ce1289ea92451a86e5f6191c37eaeb5a to 1d7b00e3fc2ebc1dc9982a2df91d15e3f12e9432
Branch pushed to git repo; I updated commit sha1. New commits:
1d7b00e  dash instead of underscore for command line options

comment:5 Changed 18 months ago by
 Description modified (diff)
comment:6 Changed 17 months ago by
 Reviewers set to Markus Wageringel
Overall, this looks good to me. The following changes seem to be needed:
 sage: subprocess.call(["sage", "t", "warnlong", "0", "randomseed=0", random_seed.rst"], **kwds) # long time + sage: subprocess.call(["sage", "t", "warnlong", "0", "randomseed=0", "random_seed.rst"], **kwds) # long time
  ``random_seed[=seed]``  random seed for fuzzing doctests +  ``randomseed[=seed]``  random seed for fuzzing doctests
comment:7 Changed 17 months ago by
The branch adds this to the documentation, which is not true for this ticket:
 a/src/doc/en/developer/doctesting.rst +++ b/src/doc/en/developer/doctesting.rst ... +Doctests start from a random seed:: + + [kliem@sage sage9.2]$ sage t src/sage/doctest/tests/random_seed.rst ... + sage t warnlong 89.5 randomseed=112986622569797306072457879734474628454 src/sage/doctest/tests/random_seed.rst + **********************************************************************
comment:8 Changed 17 months ago by
 Branch changed from public/29962 to public/29962reb
 Commit changed from 1d7b00e3fc2ebc1dc9982a2df91d15e3f12e9432 to b62f781647851a3f27ecc95ea4b98b53838da112
comment:9 Changed 17 months ago by
 Reviewers changed from Markus Wageringel to Markus Wageringel, Matthias Koeppe
From my side this is a positive review  if patchbot is green
comment:10 Changed 17 months ago by
 Status changed from needs_review to positive_review
Thanks. This successfully passes ptestlong.
comment:11 Changed 17 months ago by
Thank you.
comment:12 Changed 17 months ago by
 Status changed from positive_review to needs_work
Merge conflict
comment:13 Changed 17 months ago by
 Branch changed from public/29962reb to public/29962reb2
 Commit changed from b62f781647851a3f27ecc95ea4b98b53838da112 to 1d99129f26f4a065f9f9e5e13c3d5120a029e89f
 Status changed from needs_work to needs_review
comment:14 Changed 17 months ago by
 Status changed from needs_review to positive_review
comment:15 Changed 17 months ago by
Thank you.
comment:16 Changed 17 months ago by
 Branch changed from public/29962reb2 to 1d99129f26f4a065f9f9e5e13c3d5120a029e89f
 Resolution set to fixed
 Status changed from positive_review to closed
comment:17 Changed 16 months ago by
 Commit 1d99129f26f4a065f9f9e5e13c3d5120a029e89f deleted
What random seeds are allowed? Any integer? Of any sign? Arbitrarily large?
comment:18 Changed 16 months ago by
This is how we will eventually get the random seeds.
sage: import sage.misc.randstate as randstate sage: randstate.set_random_seed() sage: randstate.initial_seed() 11032378495085541661748859066830408537
At least that's the plan for now. Any random seed that you can feed into set_random_seed
should work.
(Edit: But I wouldn't be surprised if there are bugs somewhere with some strange random seed. There is no way, you can avoid all of them. But at least you can prepare for rather probable cases.)
comment:19 Changed 16 months ago by
The random_seed
docstring says seed "must be coercible to a Python long".
Is that from 2**127
to 2**127  1
?
Or are 32bit and 64bit systems different there?
Also isn't "long" a Python 2 concept, with only "int" existing in Python 3?
Could we document the admissible range in a followup ticket, for nonexperts like me?
comment:20 Changed 16 months ago by
And thanks for the work on varying random seeds when testing!
comment:21 Changed 16 months ago by
I added this issue to #29935.
The idea was that once all parts of Sage are ready for it, the default would be to select a "random" random seed.
comment:22 Changed 7 months ago by
 Description modified (diff)
comment:23 Changed 6 months ago by
 Cc slelievre added
 Description modified (diff)
 Summary changed from Allow fuzzing of doctests to Introduce randomseed option to allow fuzzing of doctests
New commits:
start from a "random" random seed for doctesting
make random seed reproducible
document random_seed
default random seed 0 for now