Noissue - convert test_page_outputs to pytest-regtest

Noissue - convert test_page_outputs to pytest-regtest

[Vasily Kuznetsov, 2018-04-18 21:42:40]

Hey Jon,

Here's a rough draft of conversion to regtest.

It's kind of cool that this bunch of expected outputs are just autogenerated
with one command but on the other hand our existing test suite has fewer files
and not that much more code.

It would be possible to have fewer files with regtest too, but then we'd need to
have the list of the tested pages manually hardcoded somewhere, which is more
code.

Another advantage of our current approach is that the files which are not
different between static/dynamic/-r master are only stored once. It doesn't
matter all that much when the files are autogenerated, but still, having fewer
files with less duplication seems better.

I'm a bit less excited about regtest as a result. Still, this is something to
think about and it was interesting to try using regtest.

Cheers,
Vasily

[Jon Sonesen, 2018-04-19 00:15:39]

On 2018/04/18 21:42:40, Vasily Kuznetsov wrote:
> Hey Jon,
> 
> Here's a rough draft of conversion to regtest.
> 
> It's kind of cool that this bunch of expected outputs are just autogenerated
> with one command but on the other hand our existing test suite has fewer files
> and not that much more code.
> 
> It would be possible to have fewer files with regtest too, but then we'd need
to
> have the list of the tested pages manually hardcoded somewhere, which is more
> code.
> 
> Another advantage of our current approach is that the files which are not
> different between static/dynamic/-r master are only stored once. It doesn't
> matter all that much when the files are autogenerated, but still, having fewer
> files with less duplication seems better.
> 
> I'm a bit less excited about regtest as a result. Still, this is something to
> think about and it was interesting to try using regtest.
> 
> Cheers,
> Vasily

Hey,

Thanks for trying this out, I think the code is more simple looking for sure but
once we account for the duplication that might change.

[Jon Sonesen, 2018-04-19 00:23:27]

Sorry, sent the draft before I was finished.

I will think more on this or we can talk more via IRC, but also complexity and
duplication aside I am quite mistrusting of automatically generating all the
expected output since you may create a page to test a feature but that feature
ends up not doing what you think it does, and as long as the behavior does not
change since you generated the last output your test passes.

I also prefer the more concise and verbose expected output filenames (but
regtest does that too so no biggie there).

Cheers,

Jon

[Jon Sonesen, 2018-04-21 00:24:59]

I think for now I want to say this doesnt look good to me

[Vasily Kuznetsov, 2018-04-23 08:57:47]

On 2018/04/21 00:24:59, Jon Sonesen wrote:
> I think for now I want to say this doesnt look good to me

Here's another version, with the page outputs combined, just for completeness.
It doesn't have so many files but still has the redundancy issue.

[Jon Sonesen, 2018-04-23 09:12:25]

On 2018/04/23 08:57:47, Vasily Kuznetsov wrote:
> On 2018/04/21 00:24:59, Jon Sonesen wrote:
> > I think for now I want to say this doesnt look good to me
> 
> Here's another version, with the page outputs combined, just for completeness.
> It doesn't have so many files but still has the redundancy issue.

This still doesn't address my concerns with auto generating outputs. This could
be useful for a CMS dev thing but when testing new features it would be too easy
to make pages which don't do what you think they do but still pass