Beginning.
I'm a coder by trade. A few weeks ago I was setting up Vault and needed to read its source to troubleshoot an error. Turns out it was written in go, perfect opportunity to start learning a new language! This is the story of building conthego, an SBE framework written in golang.
I find it's more fun and effective to learn something by actually using it in a project. I've been using Concordion for a while now and really like the idea of free-form executable specifications (spec). A spec is a collection of example scenarios (ex. given/when/then structure) written in plain language, usually in Markdown.
The spec is backed by fixture code which, when executed, will mark assertions in the spec as either passed or failed (html reporting showing green or red respectively). A quick search tells me nothing like this has been done in go yet so project candidate sorted.
To start, I worked my way through the golang tour to learn the basics. Googling for existing tooling that helps in language conversion from Java to Go gave me nothing but did lead to an excellent article - "Lessons learned porting 50k loc from Java to Go".
My main takeaway was the idea of iterative testing and validation after every little step. I entertained the thought (and did some reading) of AST transforms using antlr but that got me nowhere.
Ideation and initial reading was from Saturday-Monday, May 9-11, around 12 hours effort.
Iterations.
Setup and preparation. Before I can get started on the first test there was the usual required project bootstrapping. I've got Intellij Idea for an IDE, decided on github as repo, and elected to just manage tasks in-project as a list of things to do in markdown. One of the main pre-requisites was to be able to parse markdown and convert it to xhtml.
Go markdown is the best library for that. However I realised during my initial stab that it wasn't able to render link titles with parentheses (which, like Concordion, was to be my mechanism for adding commands in markdown). The library being opensource though, a couple of nights and rounds of PR reviews later, it was ready for prime time (kudos to its maintainers).
This was over Tuesday-Wednesday, May 12-13, around 8 hours effort.
First test, running a fixture method that spits out "Hello World!". Talk about biting off more than you can chew. First up was xml parsing the spec (now in the form of xhtml) into generic nodes, not standard but still pretty straightforward. What I didn't foresee was that I'd need to be diving into Go reflection this early (specs contain only strings, the call to something like "HelloWorld()" needs to be deconstructed and resolved to the relevant fixture function via reflection).
Aside from the official docs, gowalker was my friend. Given a standard library function, it searches Go projects in github and shows you how the function is used. This proved to be an invaluable resource. With the large kubernetes codebase written in Go, there's a lot of examples to learn from. Also I've had to cast my mind back over a couple of decades during my C programming days to get the brain comfortable with pointers again. All up it was 3 days before first commit late on Saturday the 16th, with that first test working, around 16 hours effort.
With the first test under my belt it's time to put the design hat on and think carefully about the commands that conthego will be supporting. These commands are the main interface for consumers using the tool so it pays to put enough upfront investment into them to make them simple and elegant to use. I was under no illusion I'll get everything perfectly right but I was conscious that I needed to start with a 'good set'.
An hour of writing and rewriting got me the command set below. Apart from the one using literal parameters, the rest have stood the test of time (a couple of weeks!). Early on I decided to stick close to what Concordion already offers. Deviations were mostly for ease of parsing.
Command prefixes were a single special character and (if required) appear only at the start of the command instruction string.
set
[World](- "var1")
echo
[](- "$var1")
[](- "$var1.prop2")
exec
[](- "Hello()")
[](- "var1=Hello(TEXT)")
[](- "var2=Hello(var1)")
[](- "var1=Hello('literal')")
assert equals or isTrue
[World](- "?Hello()")
[World](- "?var1")Next lot of functions was to implement variable setting, function parameter passing, echoing of results, inline TEXT processing, and rudimentary dot navigation for struct properties. Of these, dot navigation proved to be a quite tricky as there was no OGNL implementation yet in Go. I rejected the notion of writing one, even the most basic, until my google foo led to the wonderful go-dotnotation.
The trick was to marshal my structs into json then unmarshal them back into a generic tree of json nodes, which go-dotnotation can navigate. Simple and sufficient for my usecase. At this point, passing structs and slices around, it took a fair bit of pointer wrangling to fix some gnarly bugs (I understood the issues then but am sure I'd need to keep re-learning pointer semantics). This lot was part Saturday and most of Sunday, the 17th, at least 8 hours effort.
Next up was execute-rows - executing a set of commands (specified on table headers) over each row of a table. Turning things around my head on how to make it work, I've come to appreciate the beauty of sequential processing of commands: top to bottom, left to right; a perfect match for depth-first traversal of the xhtml node tree.
This meant binning some future (complex) usecases that don't fit the model but I was quite happy in exchange to keep functionality limited and simple. I introduced pre-processing which can transform commands in the xhtml nodes prior to the command collector collecting them for execution. In this case 'transform' means removing the command instructions from the table headers and distributing them to the actual data rows. This was Monday night, the 18th, 4 hours effort.
Verify-rows - there are two variants: first is verifying a simple list of strings, the second verifying a list of structs. This got me back to the drawing board (to be honest, I'm still not fully sold on the solution I came up with thus these commands are currently experimental). In the end I decided to constrain the implementation to support only unstructured lists for the first case, and a table for the second.
I needed to disambiguate the intent for list-processing from the regular commands. The top options were to introduce a new type of token in the command instruction string to signal list processing (ex. square brackets in "?listOfNames[]") or to introduce a new directive command that would impact processing of the succeeding commands.
I opted for the latter for the reasons stated below. Implementation of verification for a list of strings was Tuesday night, 19th, 4 hours effort.
- to keep command instruction strings simple
- because there already was prior art - a directive command '!ExpectedToFail' which impacted succeeding assertions
- I considered list-processing to be 'advanced' commands and merited 'signaling' via use of a directive
Verification for a list of structs is next, with support limited to being applied on a table. A major design decision here is to be able to choose for each column a specific struct property to assert on or echo. You can use echo exclusively (without asserts) for usecases like displaying a table of reference/config items. This was Wednesday night, 20th, 4 hours effort.
I wanted to encourage easy usage, in case anyone would be keen to try, Tidying up of the basic command examples and the README file was Thursday night 21st, 4 hours effort.
I've counted 60 hours above. I'm writing this on Sunday night, 24th, 4 hours effort. I'm a programmer, a serial optimist when it comes to estimates; and 64 is a nice round power of 2.
Ending.
The decision to store the task backlog as a TODO list in Markdown has proven to be handy. It felt natural to keep maintaining it whilst implementing code, all in the IDE. This worked as it was only me working on the codebase. Add another person and I'd probably look for a lightweight board like Trello.
Being able to tick off items (or multiple) on something like a nightly basis builds up confidence and a sense of satisfaction.
I've found that living documentation in the form of executable specifications are extremely useful in understanding how a system works. I always make it a point to refer to these artifacts primarily as specs rather than tests. Specs should be able to stand on their own even without having been executed.
Crafting useful specs takes effort and time from the whole team with the main rewards being shared understanding, increased confidence that the system behaves as expected, and having something like a jumpstart tool for new team members.
The Concordion documentation is the best resource describing how to write good specs.
It's been fun learning and creating something new, and there's still a lot of TODOs to slowly nibble at. But now I feel I've been neglecting my books :) Enjoy!
Commit timeline:
09874b0 Thu May 21 23:14:22 2020 +1200 fix README.md
77b491b Thu May 21 22:57:56 2020 +1200 Merge pull request #1 from fanovilla/feature/tidy-examples
78a09c7 Thu May 21 22:53:24 2020 +1200 better set and echo tests
290460f Thu May 21 22:31:13 2020 +1200 better assert-true test
cadc82e Thu May 21 21:51:21 2020 +1200 better assert-equals test
5af7442 Wed May 20 22:28:45 2020 +1200 reorder backlog
c2a03d6 Wed May 20 22:16:01 2020 +1200 refactor out spec pre-processing routines
1a3622a Wed May 20 22:13:04 2020 +1200 use concordion css
3dc272a Wed May 20 22:00:47 2020 +1200 implement verify table rows
83d58b7 Wed May 20 00:33:50 2020 +1200 implement verify rows for list of strings
ea5cf4d Tue May 19 18:38:59 2020 +1200 add reporting of date time generated and directives
c6fa62e Mon May 18 23:50:44 2020 +1200 implement execute rows
df56ffd Sun May 17 23:17:14 2020 +1200 simplify run spec hook
d3de448 Sun May 17 23:03:18 2020 +1200 update todos
12425cc Sun May 17 22:53:17 2020 +1200 fail on panic
0c83b22 Sun May 17 22:34:09 2020 +1200 implement return slice
620eee8 Sun May 17 22:08:05 2020 +1200 implement assert true
5ea8d27 Sun May 17 21:30:29 2020 +1200 Create LICENSE.txt
2ff8c59 Sun May 17 20:40:03 2020 +1200 implement map return values; failed assert fails test; ExpectedToFail directive
fbf8cca Sun May 17 19:45:15 2020 +1200 implement struct return values; dot.navigation
613b76e Sun May 17 16:51:12 2020 +1200 implement TEXT substitution; assign execute result to var
8961de4 Sun May 17 14:47:20 2020 +1200 implement echo
a953e87 Sun May 17 01:09:23 2020 +1200 rename ReflectUtils to MethodCaller
06d3a5f Sun May 17 01:03:33 2020 +1200 implement variable setting and parameter passing
5d22215 Sat May 16 23:13:56 2020 +1200 Initial commit