Repository navigation
dev: Add coverage variation to deal with flakiness - #335
Conversation
One known source of flakiness is the interaction between Python multiprocessing and Coverage.py: https://bitbucket.org/ned/coveragepy/src/7b080c304b9219cdc33dd0b303db14d411c419fe/tests/test_concurrency.py?at=default&fileviewer=file-view-default#test_concurrency.py-362 See also issue brainiak#326.
|
Mihai, I'm not familiar with codecov, but looking at the docs it says "threshold: allowed to drop X% and still result in a "success"". |
|
Thanks for having a look at this PR, @gdoubleyew. Codecov is not looking at directly at test results. In fact, it only runs if the tests were successful. So the threshold only allows the test coverage to drop, it does not allow tests to fail. The flakiness does not cause test failures in BrainIAK, as far as I can tell. Therefore, we do not need the |
danielsuo
left a comment
There was a problem hiding this comment.
Looks good to me; the 1% drop seems to work, based on the test reports.
One known source of flakiness is the interaction between Python
multiprocessing and Coverage.py:
https://bitbucket.org/ned/coveragepy/src/7b080c304b9219cdc33dd0b303db14d411c419fe/tests/test_concurrency.py?at=default&fileviewer=file-view-default#test_concurrency.py-362
See also issue #326.