From publiustemp-perlqa3 at yahoo.com Tue Oct 30 04:50:22 2007 From: publiustemp-perlqa3 at yahoo.com (Ovid) Date: Tue, 30 Oct 2007 02:50:22 -0700 (PDT) Subject: [tap-l] SKIP_ALL tests should not get hidden Message-ID: <166662.31668.qm@web60815.mail.yahoo.com> ----- Original Message ---- > From: Andy Lester > > I was very surprised to find that tests that I was skipping via > > plan skip_all => 'wangdoodle not installed' if $@; > > were not showing up in the results when I ran make test or prove. > In my mind, prove and make test should DEFINITELY be showing that we're > skipping tests, even though they've effectively passed. > > Instead of > > t/html_lint_ok............ok 500 ms > t/has_tag.................ok 1649 ms > > we should have > > t/html_lint_ok............ok 500 ms > # SKIP: HTML::Lint is not installed, test cannot be run. > t/has_tag.................ok 1649 ms This sounds reasonable. It seems to me that TAP::Formatter::Console::Session somewhere should have a line like: if ( $parser->skip_all ) { $formatter->_output( '# SKIP: ' . $parser->skip_all ); } Or something like that. However, TAP::Parser doesn't seem to have a &skip_all method (returning boolean or explanation? What if no skip reason given?). I was also going nuts trying to figure out where I'd put the tests for this. I feel like I've lagged far behind in TAP::Parser development :( Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/ From andy at hexten.net Tue Oct 30 05:45:20 2007 From: andy at hexten.net (Andy Armstrong) Date: Tue, 30 Oct 2007 10:45:20 +0000 Subject: [tap-l] SKIP_ALL tests should not get hidden In-Reply-To: <166662.31668.qm@web60815.mail.yahoo.com> References: <166662.31668.qm@web60815.mail.yahoo.com> Message-ID: <82688CE1-6B43-42CC-9A4D-889200AD0C0A@hexten.net> On 30 Oct 2007, at 09:50, Ovid wrote: > This sounds reasonable. It seems to me that > TAP::Formatter::Console::Session somewhere should have a line like: > > if ( $parser->skip_all ) { > $formatter->_output( '# SKIP: ' . $parser->skip_all ); > } > > Or something like that. > > However, TAP::Parser doesn't seem to have a &skip_all method > (returning boolean or explanation? What if no skip reason given?). That appears to be correct :( > I was also going nuts trying to figure out where I'd put the tests > for this. t/harness.t I think. > I feel like I've lagged far behind in TAP::Parser development :( It's been a long march eh? :) -- Andy Armstrong, Hexten From andy at hexten.net Tue Oct 30 05:46:30 2007 From: andy at hexten.net (Andy Armstrong) Date: Tue, 30 Oct 2007 10:46:30 +0000 Subject: [tap-l] SKIP_ALL tests should not get hidden In-Reply-To: <82688CE1-6B43-42CC-9A4D-889200AD0C0A@hexten.net> References: <166662.31668.qm@web60815.mail.yahoo.com> <82688CE1-6B43-42CC-9A4D-889200AD0C0A@hexten.net> Message-ID: <5A631510-D94B-48FB-8A80-B50949B3E3C0@hexten.net> On 30 Oct 2007, at 10:45, Andy Armstrong wrote: >> However, TAP::Parser doesn't seem to have a &skip_all method >> (returning boolean or explanation? What if no skip reason given?). > > That appears to be correct :( D'you want me to fix that or will you? I don't mind either way. -- Andy Armstrong, Hexten From andy at hexten.net Tue Oct 30 08:56:07 2007 From: andy at hexten.net (Andy Armstrong) Date: Tue, 30 Oct 2007 13:56:07 +0000 Subject: [tap-l] SKIP_ALL tests should not get hidden In-Reply-To: <166662.31668.qm@web60815.mail.yahoo.com> References: <166662.31668.qm@web60815.mail.yahoo.com> Message-ID: <155AA7B6-809C-4339-B56A-FADB09E21412@hexten.net> On 30 Oct 2007, at 09:50, Ovid wrote: > This sounds reasonable. It seems to me that > TAP::Formatter::Console::Session somewhere should have a line like: > > if ( $parser->skip_all ) { > $formatter->_output( '# SKIP: ' . $parser->skip_all ); > } > > Or something like that. I've added logic that produces output like this: 13:54] andy $ prove t/sample-tests/skipall t/sample-tests/ skipall_nomsg t/sample-tests/simple t/sample-tests/skipall..........skipped: rope t/sample-tests/skipall_nomsg....skipped: (no reason given) t/sample-tests/simple...........ok All tests successful. Files=3, Tests=5, 0 wallclock secs ( 0.01 usr 0.01 sys + 0.01 cusr 0.00 csys = 0.03 CPU) Result: PASS Do we like that? > However, TAP::Parser doesn't seem to have a &skip_all method > (returning boolean or explanation? What if no skip reason given?). TAP::Parser->skip_all returns a true value if all tests are skipped. It either returns the skip reason or '(no reason given)' as above. -- Andy Armstrong, Hexten From andy at hexten.net Wed Oct 31 18:51:45 2007 From: andy at hexten.net (Andy Armstrong) Date: Wed, 31 Oct 2007 23:51:45 +0000 Subject: [tap-l] Mad TAP proposal Message-ID: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> We have this ticket in the Test::Harness RT queue: http://rt.cpan.org/Ticket/Display.html?id=29633 Martin Thurn is asking for a SKIP_OUT directive that would skip all remaining test files and return a PASS. I don't like that idea; it depends on the order of execution of test scripts and it's a lot of hassle for what is probably a fairly unusual case. But what about a more general mechanism? A TAP directive that means 'schedule these other tests'. So then you'd have a controller test which was the only one directly visible to Test::Harness and that'd decide which other tests to run. And of course that'd be recursive - so nested controllers could implement selective execution of huge test suites. -- Andy Armstrong, Hexten From scratchcomputing at gmail.com Wed Oct 31 19:11:31 2007 From: scratchcomputing at gmail.com (Eric Wilhelm) Date: Wed, 31 Oct 2007 17:11:31 -0700 Subject: [tap-l] Mad TAP proposal In-Reply-To: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net> Message-ID: <200710311711.31585.ewilhelm@cpan.org> # from Andy Armstrong # on Wednesday 31 October 2007 16:51: >But what about a more general mechanism? A TAP directive that means ? >'schedule these other tests'. So then you'd have a controller test ? >which was the only one directly visible to Test::Harness and that'd ? >decide which other tests to run. It sounds like it would be re-creating a lot of the same functionality needed for declarative extra testing and/or Test::Manifest. http://scratchcomputing.com/tmp/extra_testing.txt I would rather it not be a TAP directive. Yes, it abstracts the "manifesting" into TAP, but it still requires some code to run to determine the manifest, and will therefore be less introspectable. Further, I've never really seen the point of the "controller test" pattern. It has always seemed like a workaround for a lack of harness features or something. Anyone got a better explanation? Yes, there are limitations to a 100% declarative scheme, but it has the advantage that 90% of usage becomes introspectable. Probably another 9% could be covered with a few "canned methods" -- which bottle-up some procedural code into standard phrases (ergo: introspectable.) The remaining 1% can be handled by an "eval this" entry (which could at least be sandboxed.) --Eric -- "It is a mistake to allow any mechanical object to realize that you are in a hurry." --Ralph's Observation --------------------------------------------------- http://scratchcomputing.com ---------------------------------------------------