From andy at hexten.net Sun Mar 18 23:14:58 2007 From: andy at hexten.net (Andy Armstrong) Date: Sun, 18 Mar 2007 23:14:58 +0000 Subject: [tap-l] Testing... Message-ID: <6C69F391-6709-4EC2-B865-0AE4B24E2EA0@hexten.net> Just making sure it's working. Testing anything starting with this list. -- Andy Armstrong, hexten.net From schwern at pobox.com Mon Mar 19 00:31:29 2007 From: schwern at pobox.com (Michael G Schwern) Date: Sun, 18 Mar 2007 17:31:29 -0700 Subject: [tap-l] First post! Message-ID: <45FDD9E1.40805@pobox.com> LOL! :P Oh rats, second post. What's the L stand for anyway? List? From andy at hexten.net Mon Mar 19 00:40:58 2007 From: andy at hexten.net (Andy Armstrong) Date: Mon, 19 Mar 2007 00:40:58 +0000 Subject: [tap-l] First post! In-Reply-To: <45FDD9E1.40805@pobox.com> References: <45FDD9E1.40805@pobox.com> Message-ID: <99EB6F77-135B-4D12-8C73-CBCAF154D1DB@hexten.net> On 19 Mar 2007, at 00:31, Michael G Schwern wrote: > LOL! :P Oh rats, second post. > > What's the L stand for anyway? List? Yeah. I think it's an old skool thing. I copied it from someone else. I assume we'll want a wiki: http://testanything.org/wiki/index.php/Main_Page With BBC testcard inspired logo. Just for shits and giggles. -- Andy Armstrong, hexten.net From andy at hexten.net Mon Mar 19 04:01:48 2007 From: andy at hexten.net (Andy Armstrong) Date: Mon, 19 Mar 2007 04:01:48 +0000 Subject: [tap-l] Wiki Message-ID: I've moved a load of stuff into the wiki[1]. I think we should encourage people to move TAP discussion away from the Perl QA wiki if possible - to put it into a language neutral space. Tomorrow I'll try to track down people who are implementing TAP in languages other than Perl and get them interesting in contributing to the wiki and joining this list. [1] http://testanything.org/wiki/ -- Andy Armstrong, hexten.net From andy at hexten.net Mon Mar 19 11:49:14 2007 From: andy at hexten.net (Andy Armstrong) Date: Mon, 19 Mar 2007 11:49:14 +0000 Subject: [tap-l] Tap, tap, is this thing on? Message-ID: <2E8CBF38-9224-443B-AE6A-65ACEB600772@hexten.net> Am I the only person getting mail from this list? Can someone reply if they see this please. -- Andy Armstrong, hexten.net From adrianh at quietstars.com Mon Mar 19 11:50:43 2007 From: adrianh at quietstars.com (Adrian Howard) Date: Mon, 19 Mar 2007 11:50:43 +0000 Subject: [tap-l] Tap, tap, is this thing on? In-Reply-To: <2E8CBF38-9224-443B-AE6A-65ACEB600772@hexten.net> References: <2E8CBF38-9224-443B-AE6A-65ACEB600772@hexten.net> Message-ID: <4BCAC3FA-024F-491A-8BA1-B2F26BA0A84D@quietstars.com> Nobody here but us chickens... Adrian On 19 Mar 2007, at 11:49, Andy Armstrong wrote: > Am I the only person getting mail from this list? > > Can someone reply if they see this please. > > -- > Andy Armstrong, hexten.net > > _______________________________________________ > tap-l mailing list > tap-l at testanything.org > http://testanything.org/mailman/listinfo/tap-l > From andy at hexten.net Mon Mar 19 11:55:24 2007 From: andy at hexten.net (Andy Armstrong) Date: Mon, 19 Mar 2007 11:55:24 +0000 Subject: [tap-l] Tap, tap, is this thing on? In-Reply-To: <4BCAC3FA-024F-491A-8BA1-B2F26BA0A84D@quietstars.com> References: <2E8CBF38-9224-443B-AE6A-65ACEB600772@hexten.net> <4BCAC3FA-024F-491A-8BA1-B2F26BA0A84D@quietstars.com> Message-ID: <889C5521-F122-455E-8E2E-528BFDC91E79@hexten.net> On 19 Mar 2007, at 11:50, Adrian Howard wrote: > Nobody here but us chickens... Phew. I can talk to myself for quite a while but there /are/ limits :) -- Andy Armstrong, hexten.net From publiustemp-tapx at yahoo.com Mon Mar 19 12:02:52 2007 From: publiustemp-tapx at yahoo.com (Ovid) Date: Mon, 19 Mar 2007 05:02:52 -0700 (PDT) Subject: [tap-l] Tap, tap, is this thing on? In-Reply-To: <2E8CBF38-9224-443B-AE6A-65ACEB600772@hexten.net> Message-ID: <537713.55173.qm@web60815.mail.yahoo.com> --- Andy Armstrong wrote: > Am I the only person getting mail from this list? > > Can someone reply if they see this please. I'm here. Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/ From scratchcomputing at gmail.com Mon Mar 19 17:18:51 2007 From: scratchcomputing at gmail.com (Eric Wilhelm) Date: Mon, 19 Mar 2007 10:18:51 -0700 Subject: [tap-l] Tap, tap, is this thing on? In-Reply-To: <2E8CBF38-9224-443B-AE6A-65ACEB600772@hexten.net> References: <2E8CBF38-9224-443B-AE6A-65ACEB600772@hexten.net> Message-ID: <200703191018.52310.ewilhelm@cpan.org> # from Andy Armstrong # on Monday 19 March 2007 04:49 am: >Can someone reply if they see this please. 1..1 ok 1 -- Turns out the optimal technique is to put it in reverse and gun it. --Steven Squyres (on challenges in interplanetary robot navigation) --------------------------------------------------- http://scratchcomputing.com --------------------------------------------------- From andy at hexten.net Mon Mar 19 17:50:28 2007 From: andy at hexten.net (Andy Armstrong) Date: Mon, 19 Mar 2007 17:50:28 +0000 Subject: [tap-l] Inter-stream punctuation Message-ID: <5AA09C31-DD3F-466F-AADC-ABA99A547C08@hexten.net> Given that TAP is streaming I wonder if it might ever be useful to be able to concatenate a number of TAP streams and have the harness be able to work out that the single stream represented multiple tests? Case: I'm working on a PHP implementation of a YAMLish writer. I'm writing the individual tests in PHP and I'd like to be able to write a single Perl test wrapper that runs the PHP tests one after another. t/wrapper.t <-- Perl, runs these: t/010-load.php <-- PHP, outputs TAP t/020-writer.php <-- Ditto Life would be much easier if the Perl wrapper could output a single stream with some kind of inter-tap marker between the output of the individual test scripts. I can see that it might simplify network testing also to be able to send a single TAP stream that represents the output from multiple tests. It could just be something like this: from t/010-load.php plan 1..1 ok 1 loaded OK from t/020-writer.php plan 1..2 ok 1 simple case: output OK ok 2 nested hash: output OK Of course you could argue that that intersects to some extent with the idea of being able to group tests. If you can have groups of tests each of which has their own plan then that pretty much solves the problem. -- Andy Armstrong, hexten.net From shlomif at iglu.org.il Mon Mar 19 17:51:17 2007 From: shlomif at iglu.org.il (Shlomi Fish) Date: Mon, 19 Mar 2007 19:51:17 +0200 Subject: [tap-l] Tap, tap, is this thing on? In-Reply-To: <2E8CBF38-9224-443B-AE6A-65ACEB600772@hexten.net> References: <2E8CBF38-9224-443B-AE6A-65ACEB600772@hexten.net> Message-ID: <200703191951.18383.shlomif@iglu.org.il> On Monday 19 March 2007, Andy Armstrong wrote: > Am I the only person getting mail from this list? > > Can someone reply if they see this please. 1..1 ok 1 - received message. Regards, Shlomi Fish --------------------------------------------------------------------- Shlomi Fish shlomif at iglu.org.il Homepage: http://www.shlomifish.org/ Chuck Norris wrote a complete Perl 6 implementation in a day but then destroyed all evidence with his bare hands, so no one will know his secrets. From andy at hexten.net Mon Mar 19 20:47:27 2007 From: andy at hexten.net (Andy Armstrong) Date: Mon, 19 Mar 2007 20:47:27 +0000 Subject: [tap-l] YAMLish in PHP Message-ID: <372340D3-51CB-42D0-8D1C-A8E8BAFAD30E@hexten.net> I've released a simple PHP YAMLish writer: http://testanything.org/ftp/yamlishwriter-php-v0.0.1.tar.gz It's not very exciting but handles the same test cases as the Perl version. -- Andy Armstrong, hexten.net From schwern at pobox.com Mon Mar 19 21:46:06 2007 From: schwern at pobox.com (Michael G Schwern) Date: Mon, 19 Mar 2007 14:46:06 -0700 Subject: [tap-l] Wiki In-Reply-To: References: Message-ID: <45FF049E.8080804@pobox.com> Andy Armstrong wrote: > I've moved a load of stuff into the wiki[1]. I think we should > encourage people to move TAP discussion away from the Perl QA wiki if > possible - to put it into a language neutral space. I've deleted from the perl-qa wiki most of what was moved over to the TAP wiki and moved over some other trailing stuff I found. From andy at hexten.net Mon Mar 19 21:47:48 2007 From: andy at hexten.net (Andy Armstrong) Date: Mon, 19 Mar 2007 21:47:48 +0000 Subject: [tap-l] Wiki In-Reply-To: <45FF049E.8080804@pobox.com> References: <45FF049E.8080804@pobox.com> Message-ID: <33D636F2-B750-481D-829F-BEE996972A5D@hexten.net> On 19 Mar 2007, at 21:46, Michael G Schwern wrote: >> I've moved a load of stuff into the wiki[1]. I think we should >> encourage people to move TAP discussion away from the Perl QA wiki if >> possible - to put it into a language neutral space. > > I've deleted from the perl-qa wiki most of what was moved over to > the TAP wiki > and moved over some other trailing stuff I found. Excellent, thanks. -- Andy Armstrong, hexten.net From andy at hexten.net Mon Mar 19 21:55:56 2007 From: andy at hexten.net (Andy Armstrong) Date: Mon, 19 Mar 2007 21:55:56 +0000 Subject: [tap-l] Wiki In-Reply-To: <33D636F2-B750-481D-829F-BEE996972A5D@hexten.net> References: <45FF049E.8080804@pobox.com> <33D636F2-B750-481D-829F-BEE996972A5D@hexten.net> Message-ID: <0A022871-0484-4588-9469-558220E26D6B@hexten.net> On 19 Mar 2007, at 21:47, Andy Armstrong wrote: >> I've deleted from the perl-qa wiki most of what was moved over to >> the TAP wiki >> and moved over some other trailing stuff I found. > > Excellent, thanks. So we seem to have trampled on the nonproliferation treaty with TAP::Parser :) It's also rather Perl-centric. D'you mind if I rework it a bit to make it less language specific? -- Andy Armstrong, hexten.net From schwern at pobox.com Mon Mar 19 22:02:59 2007 From: schwern at pobox.com (Michael G Schwern) Date: Mon, 19 Mar 2007 15:02:59 -0700 Subject: [tap-l] Wiki In-Reply-To: <0A022871-0484-4588-9469-558220E26D6B@hexten.net> References: <45FF049E.8080804@pobox.com> <33D636F2-B750-481D-829F-BEE996972A5D@hexten.net> <0A022871-0484-4588-9469-558220E26D6B@hexten.net> Message-ID: <45FF0893.1030708@pobox.com> Andy Armstrong wrote: > On 19 Mar 2007, at 21:47, Andy Armstrong wrote: >>> I've deleted from the perl-qa wiki most of what was moved over to >>> the TAP wiki >>> and moved over some other trailing stuff I found. >> Excellent, thanks. > > So we seem to have trampled on the nonproliferation treaty with > TAP::Parser :) Not quite, Curtis asked and the rest of us said it was ok. > It's also rather Perl-centric. D'you mind if I rework it a bit to > make it less language specific? I'm not sure how relevant to other languages it is as they tend not to have the same sort of namespace coordination issues we have lacking a Comprehensive Archive Network. From andy at hexten.net Mon Mar 19 22:25:24 2007 From: andy at hexten.net (Andy Armstrong) Date: Mon, 19 Mar 2007 22:25:24 +0000 Subject: [tap-l] Wiki In-Reply-To: <45FF0893.1030708@pobox.com> References: <45FF049E.8080804@pobox.com> <33D636F2-B750-481D-829F-BEE996972A5D@hexten.net> <0A022871-0484-4588-9469-558220E26D6B@hexten.net> <45FF0893.1030708@pobox.com> Message-ID: On 19 Mar 2007, at 22:02, Michael G Schwern wrote: > Not quite, Curtis asked and the rest of us said it was ok. OK. Should it be deleted from the treaty for consistency? >> It's also rather Perl-centric. D'you mind if I rework it a bit to >> make it less language specific? > > I'm not sure how relevant to other languages it is as they tend not > to have > the same sort of namespace coordination issues we have lacking a > Comprehensive > Archive Network. That's pretty much what I was going to say. I just want to prefix the page with a paragraph explaining that it's a Perl-specific issue and then maybe warn that implementers should consider whether there are any similar issues with their own languages. In fact, since I've written that now I've also updated the page. -- Andy Armstrong, hexten.net From schwern at pobox.com Tue Mar 20 00:44:43 2007 From: schwern at pobox.com (Michael G Schwern) Date: Mon, 19 Mar 2007 17:44:43 -0700 Subject: [tap-l] TAP tests Message-ID: <45FF2E7B.8020709@pobox.com> By which I mean acceptance tests to ensure your TAP consumer is consuming properly. We don't have anything official, it sure would be nice if we did. So I'm throwing this out as a project for someone (where someone does not equal "Andy Armstrong" because he's doing too much already) to do. A good place to start is the "sample-tests" directory in the TAP::Parser distribution. The expected results are in t/020-regression.t. I'd suggest the tests be encoded in YAMLish with the TAP and expected results in each document. For example... --- name: simple TAP: | 1..1 ok 1 results: overall: has_plan: 1 planned_tests: 1 passed: 1 version: 12 skipped: [] todo: [] lines: - type: plan planned_tests: 1 - type: test result: ok actual_result: ok directive: '' description: '' ... These should be posted on the Wiki and probably eventually shipped with the TAP module. From andy at hexten.net Tue Mar 20 00:51:21 2007 From: andy at hexten.net (Andy Armstrong) Date: Tue, 20 Mar 2007 00:51:21 +0000 Subject: [tap-l] YAML? In-Reply-To: <45FF26D1.1050201@pobox.com> References: <45FE8755.5070109@modperlcookbook.org> <3CF59515-7734-429E-BAC6-F615AB8B4542@quietstars.com> <844F1585-A79C-4CAA-A86F-81F7710E4D7D@twoshortplanks.com> <45FF26D1.1050201@pobox.com> Message-ID: <1E4B847B-08B0-4712-A10E-2F560A146656@hexten.net> On 20 Mar 2007, at 00:12, Michael G Schwern wrote: >> Why not JSON? http://www.json.org > > Good question. The rationale seems to have been lost in the wiki > move. > Here's the restored rationale. > > http://testanything.org/wiki/index.php/YAMLish#Why_not_JSON.3F Of course the idea of the YAMLish dialect negates this one somewhat: "JSON is, effectively, a subset of YAML. If your producer emits JSON then a YAML parser will read it. The inverse is not true." I'm still in favour of YAMLish on largely aesthetic grounds but I have to say that I don't think any of the objections to JSON are overwhelming. If we /were/ going to allow JSON now would be the time to do it. -- Andy Armstrong, hexten.net From andy at hexten.net Tue Mar 20 00:58:56 2007 From: andy at hexten.net (Andy Armstrong) Date: Tue, 20 Mar 2007 00:58:56 +0000 Subject: [tap-l] TAP tests In-Reply-To: <45FF2E7B.8020709@pobox.com> References: <45FF2E7B.8020709@pobox.com> Message-ID: On 20 Mar 2007, at 00:44, Michael G Schwern wrote: > By which I mean acceptance tests to ensure your TAP consumer is > consuming > properly. We don't have anything official, it sure would be nice > if we did. > So I'm throwing this out as a project for someone (where someone > does not > equal "Andy Armstrong" because he's doing too much already) to do. > > A good place to start is the "sample-tests" directory in the > TAP::Parser > distribution. The expected results are in t/020-regression.t. I'd > suggest > the tests be encoded in YAMLish with the TAP and expected results > in each > document. For example... Now /this/ is a nice reason to be glad we're using YAML instead of JSON. > --- [snip neat use of YAMLish] > ... > > These should be posted on the Wiki and probably eventually shipped > with the > TAP module. I know I've been disqualified from this but I was just pondering today that it'd be nice to distribute parts of our test suite as YAML test specifications - which got me thinking about the whole idea of language independent tests: the test providing part of the specification in a way that's independent of the implementation language. How cool would that be? I'm sure it's already being done but I hadn't thought about it too much before. -- Andy Armstrong, hexten.net From schwern at pobox.com Tue Mar 20 01:45:21 2007 From: schwern at pobox.com (Michael G Schwern) Date: Mon, 19 Mar 2007 18:45:21 -0700 Subject: [tap-l] YAML? In-Reply-To: <1E4B847B-08B0-4712-A10E-2F560A146656@hexten.net> References: <45FE8755.5070109@modperlcookbook.org> <3CF59515-7734-429E-BAC6-F615AB8B4542@quietstars.com> <844F1585-A79C-4CAA-A86F-81F7710E4D7D@twoshortplanks.com> <45FF26D1.1050201@pobox.com> <1E4B847B-08B0-4712-A10E-2F560A146656@hexten.net> Message-ID: <45FF3CB1.1080009@pobox.com> Andy Armstrong wrote: > Of course the idea of the YAMLish dialect negates this one somewhat: > > "JSON is, effectively, a subset of YAML. If your producer emits > JSON then a YAML parser will > read it. The inverse is not true." > > I'm still in favour of YAMLish on largely aesthetic grounds but I > have to say that I don't think any of the objections to JSON are > overwhelming. > > If we /were/ going to allow JSON now would be the time to do it. Producing TAP is so much easier than consuming it (YAML or JSON). Additionally the producer is strongly tied with a particular programming language while a good consumer is language agnostic. For these reasons there will always be far more producers than consumers. Therefore its more important that a producer be easier to write than a consumer. Which its why its more important that a producer can use a JSON library, in place of a possibly harder to find YAML library, then a consumer. Thus it should be a design goal of YAMLish to retain JSON compatibility. As JSON is so simple I don't think it will take much. The inline hash/array syntax, already useful for other reasons, should be all we need to add. The big hurdle to writing a TAP consumer isn't going to be finding a YAML library, its going to be parsing TAP. Folks are going to whinge no matter what we choose. JSON has little going for it over YAML except "more people use it" and even that's probably by a slim margin compared to "those who have neither". From schwern at pobox.com Tue Mar 20 01:47:21 2007 From: schwern at pobox.com (Michael G Schwern) Date: Mon, 19 Mar 2007 18:47:21 -0700 Subject: [tap-l] TAP tests In-Reply-To: References: <45FF2E7B.8020709@pobox.com> Message-ID: <45FF3D29.4010208@pobox.com> Andy Armstrong wrote: > I know I've been disqualified from this but I was just pondering > today that it'd be nice to distribute parts of our test suite as YAML > test specifications - which got me thinking about the whole idea of > language independent tests: the test providing part of the > specification in a way that's independent of the implementation > language. How cool would that be? > > I'm sure it's already being done but I hadn't thought about it too > much before. Go ahead and start doing one or two to get things moving. You probably know best of all what is necessary for a complete test. Then let someone else fill in the rest. I'm just concerned you're doing too much and will be spread too thin. From andy at hexten.net Tue Mar 20 01:58:28 2007 From: andy at hexten.net (Andy Armstrong) Date: Tue, 20 Mar 2007 01:58:28 +0000 Subject: [tap-l] YAML? In-Reply-To: <45FF3CB1.1080009@pobox.com> References: <45FE8755.5070109@modperlcookbook.org> <3CF59515-7734-429E-BAC6-F615AB8B4542@quietstars.com> <844F1585-A79C-4CAA-A86F-81F7710E4D7D@twoshortplanks.com> <45FF26D1.1050201@pobox.com> <1E4B847B-08B0-4712-A10E-2F560A146656@hexten.net> <45FF3CB1.1080009@pobox.com> Message-ID: <77BD4C53-69C3-4824-97E8-10321652F16C@hexten.net> On 20 Mar 2007, at 01:45, Michael G Schwern wrote: > Thus it should be a design goal of YAMLish to retain JSON > compatibility. Yes, agreed. I'll make it happen in TAP::Parser in the next few days. -- Andy Armstrong, hexten.net From andy at hexten.net Tue Mar 20 01:59:43 2007 From: andy at hexten.net (Andy Armstrong) Date: Tue, 20 Mar 2007 01:59:43 +0000 Subject: [tap-l] TAP tests In-Reply-To: <45FF3D29.4010208@pobox.com> References: <45FF2E7B.8020709@pobox.com> <45FF3D29.4010208@pobox.com> Message-ID: On 20 Mar 2007, at 01:47, Michael G Schwern wrote: > Go ahead and start doing one or two to get things moving. You > probably know > best of all what is necessary for a complete test. Yup, will do. > Then let someone else fill > in the rest. I'm just concerned you're doing too much and will be > spread too > thin. I'm going to have to do some paying work in the next few days too - so I may enter a slightly quieter phase :) -- Andy Armstrong, hexten.net From publiustemp-tapx at yahoo.com Tue Mar 20 08:35:24 2007 From: publiustemp-tapx at yahoo.com (Ovid) Date: Tue, 20 Mar 2007 01:35:24 -0700 (PDT) Subject: [tap-l] TAP tests In-Reply-To: Message-ID: <490442.80356.qm@web60821.mail.yahoo.com> --- Andy Armstrong wrote: > > Then let someone else fill > > in the rest. I'm just concerned you're doing too much and will be > > spread too > > thin. > > I'm going to have to do some paying work in the next few days too - > so I may enter a slightly quieter phase :) Perfectly understandable. I, for one, am extremely grateful at how well you've taken up the slack. To be fair, I also feel a bit guilty for it. I've been trying to offload some projects I've been involved with (with limited success, I might add) because I'm doing too much and I think the TAP::Parser is quite possibly the most important out of all of those projects (er, after Grant Committee work. Maybe.) So thank you, Andy! Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/ From adrianh at quietstars.com Tue Mar 20 07:08:02 2007 From: adrianh at quietstars.com (Adrian Howard) Date: Tue, 20 Mar 2007 07:08:02 +0000 Subject: [tap-l] TAP tests In-Reply-To: References: <45FF2E7B.8020709@pobox.com> Message-ID: On 20 Mar 2007, at 00:58, Andy Armstrong wrote: [snip] > which got me thinking about the whole idea of > language independent tests: the test providing part of the > specification in a way that's independent of the implementation > language. How cool would that be? [snip] Shades of fit/fitness? Adrian From adrianh at quietstars.com Tue Mar 20 11:00:08 2007 From: adrianh at quietstars.com (Adrian Howard) Date: Tue, 20 Mar 2007 11:00:08 +0000 Subject: [tap-l] Inter-stream punctuation In-Reply-To: <5AA09C31-DD3F-466F-AADC-ABA99A547C08@hexten.net> References: <5AA09C31-DD3F-466F-AADC-ABA99A547C08@hexten.net> Message-ID: <9B8B9EC1-84E1-402D-8C34-F5B2D1A2EC06@quietstars.com> On 19 Mar 2007, at 17:50, Andy Armstrong wrote: > Given that TAP is streaming I wonder if it might ever be useful to be > able to concatenate a number of TAP streams and have the harness be > able to work out that the single stream represented multiple tests? [snip] I've wanted something like this in the past [snip] > I can see that it might simplify network testing also to be able to > send a single TAP stream that represents the output from multiple > tests. [snip] That was my use case. Cheers, Adrian From andy at hexten.net Tue Mar 20 11:15:16 2007 From: andy at hexten.net (Andy Armstrong) Date: Tue, 20 Mar 2007 11:15:16 +0000 Subject: [tap-l] TAP tests In-Reply-To: <490442.80356.qm@web60821.mail.yahoo.com> References: <490442.80356.qm@web60821.mail.yahoo.com> Message-ID: <9C1F2F27-263B-400C-89E2-C81AE18C7A31@hexten.net> On 20 Mar 2007, at 08:35, Ovid wrote: > Perfectly understandable. I, for one, am extremely grateful at how > well you've taken up the slack. To be fair, I also feel a bit guilty > for it. I've been trying to offload some projects I've been involved > with (with limited success, I might add) because I'm doing too much > and > I think the TAP::Parser is quite possibly the most important out of > all > of those projects (er, after Grant Committee work. Maybe.) > > So thank you, Andy! Thanks :) I've had more fun with it than with anything I've worked on for at least the last couple of years. Hence better than usual productivity :) -- Andy Armstrong, hexten.net From andy at hexten.net Tue Mar 20 12:02:17 2007 From: andy at hexten.net (Andy Armstrong) Date: Tue, 20 Mar 2007 12:02:17 +0000 Subject: [tap-l] TAP tests In-Reply-To: References: <45FF2E7B.8020709@pobox.com> Message-ID: <1FE22739-496D-43D0-B7E3-9BFB15F8CFDE@hexten.net> On 20 Mar 2007, at 07:08, Adrian Howard wrote: > On 20 Mar 2007, at 00:58, Andy Armstrong wrote: > [snip] >> which got me thinking about the whole idea of >> language independent tests: the test providing part of the >> specification in a way that's independent of the implementation >> language. How cool would that be? > [snip] > > Shades of fit/fitness? Yes, I think so. But possible with a richer declarative syntax for the test data. A lot of tests boil down to * construct object / objects * test assertions about the state of objects * verify transformations (got / expected) * Test::LectroTest style assertions * cleanup It's common to write Perl tests with a big data structure and a little driver that chomps through them and spits out results. Big data / small code. Following that trend a bit further you get language neutral test data / no user written code at all. I like the idea that you could write the same thing in, say, Ruby, Perl and PHP - either because you need implementations in multiple languages or because you're migrating from one language to another - and the tests don't change at all. -- Andy Armstrong, hexten.net From andy at hexten.net Tue Mar 20 12:22:49 2007 From: andy at hexten.net (Andy Armstrong) Date: Tue, 20 Mar 2007 12:22:49 +0000 Subject: [tap-l] Inter-stream punctuation In-Reply-To: <9B8B9EC1-84E1-402D-8C34-F5B2D1A2EC06@quietstars.com> References: <5AA09C31-DD3F-466F-AADC-ABA99A547C08@hexten.net> <9B8B9EC1-84E1-402D-8C34-F5B2D1A2EC06@quietstars.com> Message-ID: <3DBE9F44-C21D-4F42-ABA0-7992E6552CBE@hexten.net> On 20 Mar 2007, at 11:00, Adrian Howard wrote: > > On 19 Mar 2007, at 17:50, Andy Armstrong wrote: > > > Given that TAP is streaming I wonder if it might ever be useful to be > > able to concatenate a number of TAP streams and have the harness be > > able to work out that the single stream represented multiple tests? > [snip] > > I've wanted something like this in the past > > Aha. Here's the little driver I wrote yesterday to aggregate a bunch of PHP tests into a single TAP stream: use strict; use warnings; my $php = $ENV{PHP} || 'php'; my $TESTS = 't/*.php'; my $planned = 0; for my $test ( glob( $TESTS ) ) { warn "# $test\n"; my $offset = $planned; my @command = ( $php, $test ); open my $th, '-|', @command or die "Can't run $test ($?)\n"; while ( defined( my $line = <$th> ) ) { chomp $line; if ( $line =~ /^1..(\d+)/ ) { $planned += $1; } else { $line =~ s/^((?:not\s+)?ok\s+)(\d+)/$1 . ($2 + $offset)/e; print "$line\n"; } } close $th or die "Can't run $test ($?)\n"; } # Trailing plan print "1..$planned\n"; Which is almost exactly your: Something that I've wanted in the past is a way to merge TAP streams together. So I could take these: 1..2 ok 1 - foo the first ok 2 - foo the second 1..2 ok 1 - bar the first ok 2 - bar the second and produce something like 1..4 ok 1 - foo the first ok 2 - bar the first ok 3 - bar the second ok 4 - foo the second Except that the output is actually ok 1 - foo the first ok 2 - bar the first ok 3 - bar the second ok 4 - foo the second 1..4 > [snip] > > I can see that it might simplify network testing also to be able to > > send a single TAP stream that represents the output from multiple > > tests. > [snip] > > That was my use case. I think all the TAP grammar would need would be 'begin' and 'end' begin 1..2 ok 1 - foo the first ok 2 - foo the second end begin 1..2 ok 1 - bar the first ok 2 - bar the second end It doesn't then matter how many test scripts that output comes from. -- Andy Armstrong, hexten.net From adrianh at quietstars.com Tue Mar 20 13:22:54 2007 From: adrianh at quietstars.com (Adrian Howard) Date: Tue, 20 Mar 2007 13:22:54 +0000 Subject: [tap-l] Inter-stream punctuation In-Reply-To: <3DBE9F44-C21D-4F42-ABA0-7992E6552CBE@hexten.net> References: <5AA09C31-DD3F-466F-AADC-ABA99A547C08@hexten.net> <9B8B9EC1-84E1-402D-8C34-F5B2D1A2EC06@quietstars.com> <3DBE9F44-C21D-4F42-ABA0-7992E6552CBE@hexten.net> Message-ID: On 20 Mar 2007, at 12:22, Andy Armstrong wrote: [snip] > I think all the TAP grammar would need would be 'begin' and 'end' > > begin > 1..2 > ok 1 - foo the first > ok 2 - foo the second > end > begin > 1..2 > ok 1 - bar the first > ok 2 - bar the second > end > > It doesn't then matter how many test scripts that output comes from. [snip] But doesn't that force us to do all the foo tests, then all the bar tests. I'd like to feed them both to a TAP consumer at the same time. Cheers, Adrian From adrianh at quietstars.com Tue Mar 20 13:24:23 2007 From: adrianh at quietstars.com (Adrian Howard) Date: Tue, 20 Mar 2007 13:24:23 +0000 Subject: [tap-l] TAP tests In-Reply-To: <1FE22739-496D-43D0-B7E3-9BFB15F8CFDE@hexten.net> References: <45FF2E7B.8020709@pobox.com> <1FE22739-496D-43D0-B7E3-9BFB15F8CFDE@hexten.net> Message-ID: <561FC659-4847-4C79-A8FF-74CCD898D30F@quietstars.com> On 20 Mar 2007, at 12:02, Andy Armstrong wrote: > On 20 Mar 2007, at 07:08, Adrian Howard wrote: >> On 20 Mar 2007, at 00:58, Andy Armstrong wrote: >> [snip] >>> which got me thinking about the whole idea of >>> language independent tests: the test providing part of the >>> specification in a way that's independent of the implementation >>> language. How cool would that be? >> [snip] >> >> Shades of fit/fitness? > > Yes, I think so. But possible with a richer declarative syntax for > the test data. Test::Data is something else that is of a similar persuasion (annoyingly Ingy's YAPC presentation seems to have vanished from the interweb...) Adrian From andy at hexten.net Tue Mar 20 13:28:22 2007 From: andy at hexten.net (Andy Armstrong) Date: Tue, 20 Mar 2007 13:28:22 +0000 Subject: [tap-l] Inter-stream punctuation In-Reply-To: References: <5AA09C31-DD3F-466F-AADC-ABA99A547C08@hexten.net> <9B8B9EC1-84E1-402D-8C34-F5B2D1A2EC06@quietstars.com> <3DBE9F44-C21D-4F42-ABA0-7992E6552CBE@hexten.net> Message-ID: On 20 Mar 2007, at 13:22, Adrian Howard wrote: >> It doesn't then matter how many test scripts that output comes from. > [snip] > > But doesn't that force us to do all the foo tests, then all the bar > tests. I'd like to feed them both to a TAP consumer at the same time. When you say at the same time you mean actual concurrency, right? -- Andy Armstrong, hexten.net From andy at hexten.net Tue Mar 20 13:30:33 2007 From: andy at hexten.net (Andy Armstrong) Date: Tue, 20 Mar 2007 13:30:33 +0000 Subject: [tap-l] TAP tests In-Reply-To: <561FC659-4847-4C79-A8FF-74CCD898D30F@quietstars.com> References: <45FF2E7B.8020709@pobox.com> <1FE22739-496D-43D0-B7E3-9BFB15F8CFDE@hexten.net> <561FC659-4847-4C79-A8FF-74CCD898D30F@quietstars.com> Message-ID: On 20 Mar 2007, at 13:24, Adrian Howard wrote: > Test::Data is something else that is of a similar persuasion > (annoyingly Ingy's YAPC presentation seems to have vanished from the > interweb...) Interesting but I don't think that's what I'm talking about. Next time I'm writing some data driven tests I'll have a go at implementing the kind of thing I have in mind. -- Andy Armstrong, hexten.net From adrianh at quietstars.com Tue Mar 20 13:50:13 2007 From: adrianh at quietstars.com (Adrian Howard) Date: Tue, 20 Mar 2007 13:50:13 +0000 Subject: [tap-l] Inter-stream punctuation In-Reply-To: References: <5AA09C31-DD3F-466F-AADC-ABA99A547C08@hexten.net> <9B8B9EC1-84E1-402D-8C34-F5B2D1A2EC06@quietstars.com> <3DBE9F44-C21D-4F42-ABA0-7992E6552CBE@hexten.net> Message-ID: On 20 Mar 2007, at 13:28, Andy Armstrong wrote: > On 20 Mar 2007, at 13:22, Adrian Howard wrote: >>> It doesn't then matter how many test scripts that output comes from. >> [snip] >> >> But doesn't that force us to do all the foo tests, then all the bar >> tests. I'd like to feed them both to a TAP consumer at the same time. > > When you say at the same time you mean actual concurrency, right? Depending what you mean by "actual concurrency" yes :-) What I was doing was getting multiple TAP streams back from different machines by basically forking of a bunch of 'ssh testN.com t/foo.t' and slurping up the results. What I ended up doing was merging in each stream as a unit, so I output something like what you proposed ok 1 test1.com a ok 2 test1.com b ok 3 test1.com c ok 4 test1.com d not ok 5 test2.com a ok 6 test2.com b ok 7 test2.com c ok 8 test2.com d ... etc... which was "good enough". However it wasn't ideal since, with long running test scripts, I didn't get to see early failures when they happened - only on script completion. If I could have been arsed what would have been better would have been something like ok 1 test1.com a not ok 2 test2.com a ok 3 test1.com b ok 4 test2.com b ok 5 test1.com c ok 6 test2.com c ok 7 test1.com d ok 8 test2.com d outputting the test results as I got them so I got the feedback more quickly. Cheers, From andy at hexten.net Tue Mar 20 13:55:45 2007 From: andy at hexten.net (Andy Armstrong) Date: Tue, 20 Mar 2007 13:55:45 +0000 Subject: [tap-l] Inter-stream punctuation In-Reply-To: References: <5AA09C31-DD3F-466F-AADC-ABA99A547C08@hexten.net> <9B8B9EC1-84E1-402D-8C34-F5B2D1A2EC06@quietstars.com> <3DBE9F44-C21D-4F42-ABA0-7992E6552CBE@hexten.net> Message-ID: On 20 Mar 2007, at 13:50, Adrian Howard wrote: > which was "good enough". However it wasn't ideal since, with long > running test scripts, I didn't get to see early failures when they > happened - only on script completion. This is really something the harness should handle and it's something we've talked about for TAP::Parser. I don't think we got as far as specifying how it should actually work though... Logically though the harness should handle merging results from multiple sources. -- Andy Armstrong, hexten.net From scratchcomputing at gmail.com Tue Mar 20 17:42:31 2007 From: scratchcomputing at gmail.com (Eric Wilhelm) Date: Tue, 20 Mar 2007 10:42:31 -0700 Subject: [tap-l] Inter-stream punctuation In-Reply-To: References: <5AA09C31-DD3F-466F-AADC-ABA99A547C08@hexten.net> <3DBE9F44-C21D-4F42-ABA0-7992E6552CBE@hexten.net> Message-ID: <200703201042.32161.ewilhelm@cpan.org> # from Adrian Howard # on Tuesday 20 March 2007 06:22 am: >On 20 Mar 2007, at 12:22, Andy Armstrong wrote: >[snip] >> I think all the TAP grammar would need would be 'begin' and 'end' >> It doesn't then matter how many test scripts that output comes from. >[snip] >But doesn't that force us to do all the foo tests, then all the bar ? >tests. I'd like to feed them both to a TAP consumer at the same time. If you want to have hierarchical tests which are sent in arbitrary order, that has to be completely different than "multiple pieces of tap strung together." The protocol needs to preserve the simplicity of an single feed. This allows the harness and other tools to easily juggle multiple feeds. If we put juggling in the protocol, the tools are then potentially juggling jugglers. Where your use-case is an optimization (ala LWP::Parallel), it either belongs in a parallelized harness or you need a two-step producer of some sort (e.g. an aggregating proxy-consumer does the network pulls and pretends to be a serial, concatenating producer.) In either case, the feed needs to be a single test. Of course, to get the responsiveness that you want via a proxy-consumer, the proxy would have to start all of the feeds and do some math with the plans and test numbers. A parallelized harness would probably be simpler. --Eric -- The opinions expressed in this e-mail were randomly generated by the computer and do not necessarily reflect the views of its owner. --Management --------------------------------------------------- http://scratchcomputing.com --------------------------------------------------- From andy at hexten.net Thu Mar 29 12:28:21 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 29 Mar 2007 12:28:21 +0100 Subject: [tap-l] Non-perl people? Message-ID: <9875E15B-4360-465C-B52F-2762F64A781E@hexten.net> As a matter of interest is anyone here not primarily from the Perl community? -- Andy Armstrong, hexten.net From geoff at modperlcookbook.org Thu Mar 29 16:04:58 2007 From: geoff at modperlcookbook.org (Geoffrey Young) Date: Thu, 29 Mar 2007 11:04:58 -0400 Subject: [tap-l] Non-perl people? In-Reply-To: <9875E15B-4360-465C-B52F-2762F64A781E@hexten.net> References: <9875E15B-4360-465C-B52F-2762F64A781E@hexten.net> Message-ID: <460BD59A.7040008@modperlcookbook.org> Andy Armstrong wrote: > As a matter of interest is anyone here not primarily from the Perl > community? we do now :) --Geoff From chris at shiflett.org Thu Mar 29 16:07:13 2007 From: chris at shiflett.org (Chris Shiflett) Date: Thu, 29 Mar 2007 11:07:13 -0400 Subject: [tap-l] Non-perl people? In-Reply-To: <460BD59A.7040008@modperlcookbook.org> References: <9875E15B-4360-465C-B52F-2762F64A781E@hexten.net> <460BD59A.7040008@modperlcookbook.org> Message-ID: <460BD621.2050409@shiflett.org> Geoffrey Young wrote: > Andy Armstrong wrote: > > As a matter of interest is anyone here not primarily from the > > Perl community? > > we do now :) I guess that would be me. :-) Hi, I'm primarily a PHP guy and the author of test-more.php. Chris From andy at hexten.net Thu Mar 29 16:34:01 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 29 Mar 2007 16:34:01 +0100 Subject: [tap-l] Non-perl people? In-Reply-To: <460BD621.2050409@shiflett.org> References: <9875E15B-4360-465C-B52F-2762F64A781E@hexten.net> <460BD59A.7040008@modperlcookbook.org> <460BD621.2050409@shiflett.org> Message-ID: <893AB9FF-5ED1-41FB-A7F2-C01E0CC5B793@hexten.net> On 29 Mar 2007, at 16:07, Chris Shiflett wrote: >> we do now :) > > I guess that would be me. :-) > > Hi, I'm primarily a PHP guy and the author of test-more.php. Excellent. Welcome Chris :) The purpose of this list is to be a language neutral forum for anyone who's interested in TAP and I guess there's an agenda to actively promote TAP for languages other than Perl too. As you may know work is underway on a new TAP parser for Perl and it supports some TAP syntax extensions as detailed on the Wiki[1]. So that's where we are now. Pull up a chair :) [1] http://testanything.org/wiki -- Andy Armstrong, hexten.net From scratchcomputing at gmail.com Thu Mar 29 18:13:32 2007 From: scratchcomputing at gmail.com (Eric Wilhelm) Date: Thu, 29 Mar 2007 10:13:32 -0700 Subject: [tap-l] Non-perl people? In-Reply-To: <893AB9FF-5ED1-41FB-A7F2-C01E0CC5B793@hexten.net> References: <9875E15B-4360-465C-B52F-2762F64A781E@hexten.net> <460BD621.2050409@shiflett.org> <893AB9FF-5ED1-41FB-A7F2-C01E0CC5B793@hexten.net> Message-ID: <200703291013.32796.ewilhelm@cpan.org> # from Andy Armstrong # on Thursday 29 March 2007 08:34 am: >As you may know work is underway on a new TAP parser for Perl and it ? >supports some TAP syntax extensions as detailed on the Wiki[1]. That should read "a new TAP parser in Perl" ;-) --Eric -- "...the bourgeoisie were hated from both ends: by the proles, because they had all the money, and by the intelligentsia, because of their tendency to spend it on lawn ornaments." --Neal Stephenson --------------------------------------------------- http://scratchcomputing.com --------------------------------------------------- From andy at hexten.net Thu Mar 29 18:24:11 2007 From: andy at hexten.net (Andy Armstrong) Date: Thu, 29 Mar 2007 18:24:11 +0100 Subject: [tap-l] Non-perl people? In-Reply-To: <200703291013.32796.ewilhelm@cpan.org> References: <9875E15B-4360-465C-B52F-2762F64A781E@hexten.net> <460BD621.2050409@shiflett.org> <893AB9FF-5ED1-41FB-A7F2-C01E0CC5B793@hexten.net> <200703291013.32796.ewilhelm@cpan.org> Message-ID: <7841C764-EA28-4DA1-9F4E-83F68757B151@hexten.net> On 29 Mar 2007, at 18:13, Eric Wilhelm wrote: >> As you may know work is underway on a new TAP parser for Perl and it >> supports some TAP syntax extensions as detailed on the Wiki[1]. > > That should read "a new TAP parser in Perl" ;-) Quite so :) -- Andy Armstrong, hexten.net From schwern at pobox.com Thu Mar 29 22:23:11 2007 From: schwern at pobox.com (Michael G Schwern) Date: Thu, 29 Mar 2007 14:23:11 -0700 Subject: [tap-l] Non-perl people? In-Reply-To: <460BD621.2050409@shiflett.org> References: <9875E15B-4360-465C-B52F-2762F64A781E@hexten.net> <460BD59A.7040008@modperlcookbook.org> <460BD621.2050409@shiflett.org> Message-ID: <460C2E3F.7000706@pobox.com> Chris Shiflett wrote: > Geoffrey Young wrote: >> Andy Armstrong wrote: >>> As a matter of interest is anyone here not primarily from the >>> Perl community? >> we do now :) > > I guess that would be me. :-) > > Hi, I'm primarily a PHP guy and the author of test-more.php. Awesome. If I may make one suggestion. I noticed this in the code... TODO: function eq_array() { } function eq_hash() { } function eq_set() { } I might suggest you TODON'T those functions. They've been deprecated from Test::More as they're not really tests, produce no diagnostics and is_deeply() covers all their functionality. In fact, for deep comparisons I'd suggest using Test::Deep as your inspiration. Its done a far more thoughtful analysis of complex structure comparison. From chris at shiflett.org Thu Mar 29 22:34:14 2007 From: chris at shiflett.org (Chris Shiflett) Date: Thu, 29 Mar 2007 17:34:14 -0400 Subject: [tap-l] Non-perl people? In-Reply-To: <460C2E3F.7000706@pobox.com> References: <9875E15B-4360-465C-B52F-2762F64A781E@hexten.net> <460BD59A.7040008@modperlcookbook.org> <460BD621.2050409@shiflett.org> <460C2E3F.7000706@pobox.com> Message-ID: <460C30D6.8060504@shiflett.org> Michael G Schwern wrote: > Awesome. If I may make one suggestion. I noticed this in the > code... > > TODO: > > function eq_array() > { > } > > function eq_hash() > { > } > > function eq_set() > { > } > > I might suggest you TODON'T those functions. They've been deprecated > from Test::More as they're not really tests, produce no diagnostics > and is_deeply() covers all their functionality. Awesome, thanks for the suggestion. :-) Chris From pagaltzis at gmx.de Fri Mar 30 09:13:45 2007 From: pagaltzis at gmx.de (A. Pagaltzis) Date: Fri, 30 Mar 2007 10:13:45 +0200 Subject: [tap-l] Fwd: The next gen TAP, bringing it together. Message-ID: <20070330081345.GU22282@klangraum> ----- Forwarded message from Michael G Schwern ----- > From: Michael G Schwern > To: Andy Armstrong > Subject: The next gen TAP, bringing it together. > Date: Tue, 27 Mar 2007 16:21:40 -0700 > Message-ID: <4609A704.8010804 at pobox.com> > Cc: Gary Hawkins , perl-qa at perl.org > List-Id: > User-Agent: Thunderbird 1.5.0.10 (Macintosh/20070221) > > Andy Armstrong wrote: > > On 27 Mar 2007, at 18:16, Gary Hawkins wrote: > >> What about diag() output. > > > > It goes to STDERR and is therefore not really part of TAP. > > Many uses of diag are expected to be replaced by the YAML > > based mechanism. > > To expand on that, the plan is to transition test module > authors away from using free-format diag() and using the > structured YAML stuff instead. In general when a test module is > using diag() they really want structured diagnostics anyway. > > As for users who use diag(), those messages do tend to really > be free-form. For that diag() will start to use the proposed > TAP logging syntax and no longer print to STDERR. > http://testanything.org/wiki/index.php/TAP_logging_syntax > > diag() would be at the level "notice" and thus normally > displayed. I'll probably add to Test::More an info() function > which outputs at level "info" which is roughly the equivalent > now to printing to STDOUT. Its not normally displayed. > > So, let's bring this together. A simple test like this: > > use File::Temp qw(tempfile); > > use Test::More tests => 3; > > is $dog, "Basset hound", "Basset hounds got long ears"; > > my($fh, $tmpfile) = tempfile(); > print "# Using file $tmpfile\n"; > ok( -w $tmpfile, "tmpfile is writable" ) > or diag "$tmpfile not writable: $!"; > > ok( -x $tmpfile, "tmpfile is executable" ) > or diag "$tmpfile not executable: $!"; > > > Currently outputs this > > 1..3 > not ok 1 - Basset hounds got long ears > # Failed test 'Basset hounds got long ears' > # at /Users/schwern/tmp/foo.t line 8. > # got: 'Mutt' > # expected: 'Basset hound' > # Using file /tmp/HBN5dFYRBm > ok 2 - tmpfile is writable > not ok 3 - tmpfile is executable > # Failed test 'tmpfile is executable' > # at /Users/schwern/tmp/foo.t line 15. > # /tmp/HBN5dFYRBm not executable: > # Looks like you failed 2 tests of 3. > > > And when run through Test::Harness it looks like this: > > $ prove ~/tmp/foo.t > /Users/schwern/tmp/foo.... > # Failed test 'Basset hounds got long ears' > # at /Users/schwern/tmp/foo.t line 8. > # got: 'Mutt' > # expected: 'Basset hound' > /Users/schwern/tmp/foo....NOK 1/3 > /Users/schwern/tmp/foo....ok 2/3# Failed test 'tmpfile is executable' > # at /Users/schwern/tmp/foo.t line 15. > # /tmp/U2x9QTnKoL not executable: > # Looks like you failed 2 tests of 3. > /Users/schwern/tmp/foo....dubious > Test returned status 2 (wstat 512, 0x200) > DIED. FAILED tests 1, 3 > Failed 2/3 tests, 33.33% okay > Failed Test Stat Wstat Total Fail List of Failed > ------------------------------------------------------------------------------- > /Users/schwern/tmp/foo.t 2 512 3 2 1 3 > Failed 1/1 test scripts. 2/3 subtests failed. > Files=1, Tests=3, 0 wallclock secs ( 0.05 cusr + 0.01 csys = 0.06 CPU) > Failed 1/1 test programs. 2/3 subtests failed. > > A bit messy. Since all the failure diagnostics are going to > STDERR Test::Harness can't format their output or control it or > be aware of it at all. So its a mess. They're also unparsable. > Nobody but a human can say why the tests failed. > > The next gen TAP using the same code, except the print to > STDOUT replaced with an info() call, will look something like > this. > > TAP version 16 > --- > datetime: Tue, 27 Mar 2007 15:56:58 -0700 > file: /Users/schwern/tmp/foo.t > hostname: windhund.schwern.org > producer: > name: Test::Builder > version: 0.72 > ... > 1..3 > not ok 1 - Basset hounds got long ears > --- > line: 8 > code: 'is $dog, "Basset hound", "Basset hounds got long ears";' > got: 'Mutt' > expected: 'Basset hound' > display: |2 > got: 'Mutt' > expected: 'Basset hound' > ... > **info** Using file /tmp/HBN5dFYRBm > ok 2 - tmpfile is writable > --- > line: 12 > code: ok( -w $tmpfile, "tmpfile is writable" ) > ... > not ok 3 - tmpfile is executable > --- > line: 15 > code: ok( -x $tmpfile, "tmpfile is executable" ) > ... > **notice** /tmp/HBN5dFYRBm not executable: > **fail** Looks like you failed 2 tests of 3. > > > This incorporates these three proposals as well as the new > version syntax which has been accepted. > http://testanything.org/wiki/index.php/Test_meta_information > http://testanything.org/wiki/index.php/TAP_diagnostic_syntax > http://testanything.org/wiki/index.php/TAP_logging_syntax > > The amount of information to put in the YAML is up to the TAP > producer, all of the keys are optional as is outputting any > YAML at all. This example is probably about the level of > default verbosity I'd expect. The "display" key is just a > suggestion to the TAP producer about how to display the > failure. > > The "code" key is an example of extra information it will be > possible for a producer to send on to the parser. The parser > will then choose what to do with it. > > > So the raw test output is a little more verbose than before and > possibly a little harder for a human to read. But now its all > in one stream (STDOUT) and all parsed by the parser. We trade > off a little verbosity and a little human readability of the > raw stream for a much more flexible TAP display. A hypothetical > display might look like this: > > > $ prove3 ~/tmp/foo.t > Running /Users/schwern/tmp/foo....1/3 > ------------------------------------------------------- > !!!! Failed test #1: 'Basset hounds got long ears' !!!! > !!!! at /Users/schwern/tmp/foo.t line 8. !!!! > !!!! code: 'is $dog, "Basset hound", "Basset hounds got long ears";' > !!!! actual: 'Mutt' !!!! > !!!! expected: 'Basset hound' !!!! > ------------------------------------------------------- > ...........................3/3 > ------------------------------------------------------- > !!!! Failed test #3: 'tmpfile is executable' !!!! > !!!! at /Users/schwern/tmp/foo.t line 15. !!!! > !!!! code: 'ok( -x $tmpfile, "tmpfile is executable" )' > ------------------------------------------------------- > [notice] /tmp/U2x9QTnKoL not executable: > [fail] Looks like you failed 2 tests of 3. > Test returned status 2 (wstat 512, 0x200) > DIED. FAILED tests 1, 3 > Failed 2/3 tests, 33.33% okay > > Failed Test Stat Wstat Total Fail List of Failed > ------------------------------------------------------------------------------- > /Users/schwern/tmp/foo.t 2 512 3 2 1 3 > Failed 1/1 test scripts. 2/3 subtests failed. > Files=1, Tests=3, 0 wallclock secs ( 0.05 cusr + 0.01 csys = 0.06 CPU) > Failed 1/1 test programs. 2/3 subtests failed. > > > Not terribly imaginative, I admit, but still much improved > readability. This is possible because the displayer has full > control over all the diagnostics. It can parse and reformat > them as it likes. Everything in the example above is printed by > the parser. Nothing comes directly from the producer. Its not a > mistake that a changed the "got" key to "actual" as some people > have requested. > > The parser can accept flags to display information about > passing tests, show different levels of logs, be very quiet, be > very verbose, put it all into a file, translate it to XML, send > it out via email... sky's the limit. > > Folks who do more radical things like TAP::HTML::Matrix (which > is radical only in a relative sense) will find this new syntax > extremely useful as they can now archive and display all the > same information you get by running a test on the terminal. > They no longer have to scrape things like file and line number > out of the diagnostics. ----- End forwarded message -----