Fitting Acceptance Tests with DSLs

Reading Ayende's post about Fit prompts me to say that I, too, haven't been overly pleased with the Fit/FitNesse experience in the .NET world; Java maybe a different story...The Fit book was/is excellent at demonstrating what can be done with Fit and why it's valuable to have customer driven/written acceptance tests, but the framework just doesn't seem to have matured enough on the .NET framework.

I've started writing a DSL for acceptance testing based on the ideas presented by Dan North on his blog which I found while looking at Behave#. I believe the Behave# project has good intentions, but the idea is too low-level; customers, business analysts, etc. aren't going to understand what's going on there, especially when asked to use anonymous methods.

Dan North's ideas seem to provide a good context for establishing boundaries around what constitutes a given function of the application, how it is expected to be used/work, and how to verify the application behaved as expected given a number of situations (or scenarios, test cases, what-have-you). And, perhaps most importantly, the language he and his team uses as a template for story writing provides a solid foundation as the basis for an extensible, customer-friendly, acceptance testing DSL. Let's take a look at one of the examples Dan provides:

Story: Account Holder withdraws cash

As an Account Holder

I want to withdraw cash from an ATM

So that I can get money when the bank is closed


Scenario 1: Account has sufficient funds

Given the account balance is $100

And the card is valid

And the machine contains enough money

When the Account Holder requests $20

Then the ATM should dispense $20

And the account balance should be $80

And the card should be returned


Scenario 2: Account has insufficient funds

Given the account balance is $10

And the card is valid

And the machine contains enough money

When the Account Holder requests $20

Then the ATM should not dispense any money

And the ATM should say there are insufficient funds

And the account balance should be $20

And the card should be returned


Scenario 3: Card has been disabled

Given the card is disabled

When the Account Holder requests $20

Then the ATM should retain the card

And the ATM should say the card has been retained


Scenario 4: The ATM has insufficient funds

…

The Story heading presents the title of the story/functionality, followed by three important pieces: the role, desired functionality, and the reason why the functionality is important. These are represented by the As a[n], I want, So that prefixed statements. While it is arguable whether these statements represent any valuable testing information for developers and/or testers, it does provide them with important contextual and business-need information about the desired functionality. From there, each scenario describes the background/"setup" context (the Givens), the action or functionality to execute (the When), and the desired outcome (the Thens).

Given these ideas, here's a DSL that could accomplish this and also yield us (the developers) automated test fixtures for verification during continuous integration cycles:

Story "Account Holder withdraws cash":
    Role "Account Holder"
    Feature "Withdraw cash from ATM"
    Benefit "Access money when the bank is closed"

    Scenario "Account has sufficient funds":
       Given:
          That "The account balance is $100":
            CreateAccount "012324", 100
          And "The card is valid":
            AddATMCardToAccount "012324", "32123456879", Valid
          And "The machine contains enough money":
            SetupAtm 10000
       When "The Account Holder Requests $20":
          UseATM "32123456789", Withdrawal, 20
       Verify:
          That "The ATM should dispense $20":
            VerifyATM LastTransaction, Withdrawal 20
          And "The account balance should be $80":
            VerifyAccount "012324", Balance 80
          And "The card should be returned":
            VerifyATM State NoCardPresent

What if the DSL processor produced the following:

[TestFixture]
public class StoryFixture_Account_Holder_withdraws_cash
{
[Test]
public void Scenario_Account_has_sufficient_funds()
{
  Account a = new Account("012324", 100);
  a.AddAtmCard(new AtmCard("32123456879", AtmCardStatus.Valid));
  AutomatedTeller m = new AutomatedTeller(10000);

  m.Withdrawl(a.AtmCard, 20);

  Assert.AreEqual(Operation.Withdrawl, m.LastTransaction.Operation);
  Assert.AreEqual(20, m.LastTransaction.Amount);
  Assert.AreEqual(80, a.Balance);
  Assert.AreEqual(AtmState.NoCardPresent, m.State);
}
}

Not all of the syntactic sugar has been worked out here, but I think the idea of the foundation DSL extended with the application specific language is clear. While this is better than having to write lower-level C# with anonymous methods, I think users - as in customers, business analysts, etc. - would be better served with a UI that provided them the necessary "guidance", if you will, to create their scenarios as necessary. Something like:

Would then output the DSL file. Now note that none of that UI works, it's all just show right now and very little of the DSL parsing is functional as of yet. With more time, I hope to have this publicly available for contributions, etc. if there's interest.

Finally, Somebody, Got it Right!

My wife and I were heading out of town this weekend but, before we could begin our trip, she had to take a PRAXIS test for her masters so she could student teach this fall and, thus, graduate in Spring of 2008. She's enrolled at Shippensburg University and the test just so happened to be offered there today which, semi-conveinently, wasn't too far out of the way of our destination. So, I tagged along, naturally, as why double back home just to turn around and head in the same general direction. The trick? Find something to occupy myself for 2-3 hours during the test...

Naturally, I took my work-issued laptop and reading material. When I first sat down, I figured I'd crack-open GMail and send a few necessarsy replies before studying. I figured, as most modern-day universities do, Ship would have wireless and I'd be online in no time.

After finding and connecting to the campus WAP, a sinking feeling fell upon me; I suddenly started having flashbacks from college when NIMDA and CODE-RED crippled my alma mater's network. While I was protected by a Linksys router in my dorm then, I did not have such the "luxury" today. This time, it was only me, my laptop, and the Ship campus network with a direct pipe to the Internet. I became very, very afraid.

My first course of action was to double check and see that at least, by some strange occurence of good fortune, that work had, in-fact, enabled Windows Firewall and that I at least had that protection.

Negative.

Four-letter words abound (silently as there was a gentleman sleeping in the chair next to me) and five minutes later, I disconnected, and turned on the firewall. Thank God I run as a non-admin; who knows what could've happened. Right?

Maybe.

Turns out once I felt "secure" about the connection was about to plug into, somebody actually thought out network security at good ol' Shippensburg University. When I fired up my browser to send those e-mails, I was greeted by a solitary web page saying "we don't recognize your computer on our network. Please login using your student ID and password to connect to the network". WOW!!! How 'bout that?! The network admins here had enough sense and forethought to say "with all the laptops, hand-helds and who knows what else roaming around these days, we need to be extra cautious. Anything new/strange that suddenly shows up on our network, isolate it on a separate LAN and require them to authenticate before bringing them in."

Needless to say, I was very, very, very impressed. It only took 4, almost 5, years for academia to "catch-up" on network safety/security, but it looks like it happened anyway. I'm thankful that my wife gave me her login credentials (in case I had to use a lab), or else, it could've turned out to be a worse afternoon -- in many, many regards.

And for those killjoys out there, yeah, I do realize that being summer, it's highly likely that the campus isn't that flooded with too many God-knows how nastily infected student machines; always better safe than sorry.