Mutation testing for crystal. https://hanneskaeufler.github.io/crytic

Crytic

Crytic logo

InstallationUsageChangelogCredits

Crytic, pronounced /ˈkrɪtɪk/, is a mutation testing framework for the crystal programming language. Mutation testing is a type of software testing where specific statements in the code are changed to determine if test cases find this defect.

Crytic is in a very early state of development. It is not very clever, making it slow as well.

Latest Tag CircleCI codecov Mutation Score

Blog posts

Installation

Add this to your application's shard.yml:

development_dependencies:
  crytic:
    github: hanneskaeufler/crytic
    version: ~> 5

After shards install, this will place the crytic executable into the bin/ folder inside your project.

Usage

Running crytic without any arguments will mutate all of the source files found by src/**/*.cr and use the test-suite containing spec/**/*_spec.cr. Depending on the size of your project and the duration of a full crystal spec, this might take quite a bit of time.

./bin/crytic

Crytic can also be run to only mutate statements in one file, let's call that our subject, or --subject in the command line interface. You can also provide a list of test files to be executed in order to find the defects. This might be helpful to exclude certain long-running integration specs in order to speed up the test suite.

./bin/crytic --subject src/blog/pages/archive.cr spec/blog_spec.cr spec/blog/pages/archive_spec.cr

The above command determines a list of mutations that can be performed on the source code of archive.cr and joins the blog_spec.cr and archive_spec.cr as a test-suite to find suriving mutants.

CLI options

--subject/-s specifies a relative filepath to the sourcecode being mutated.

--min-msi/-m specifies a threshold as to when to exit the program with 0 even when mutants survived. MSI is the Mutation Score Indicator.

--preamble/-p specifies some source code to prepended to the combination of mutated source and specs. By default this will inject a bit of code to enable the "fail fast" mode of crystal spec. This can be used to disable the fail fast behaviour or avoid errors if you don't use crystal spec.

--reporters/-r specifies which reporters to enable via a comma separated list. Reporters Console, Stryker and ConsoleFileSummary exist. Console is enabled by default if the option is omitted.

The rest of the unnamed positional arguments are relative filepaths to the specs to be run.

How to read the output


✅ Original test suite passed.
Running 138 mutations.

    ❌ AndOrSwap
        in source.cr:26:7
        The following change didn't fail the test-suite:
            @@ -26,7 +26,7 @@
                     end
                   end
                   def ==(other : Chunk)
            -        ((type == other.type) && (range_a == other.range_a)) && (range_b == other.range_b)
            +        ((type == other.type) && (range_a == other.range_a)) || (range_b == other.range_b)
                   end
                 end
                 enum Type

    ✅ AndOrSwap
        in source.cr:109:13

Finished in 14:02 minutes:
138 mutations, 85 covered, 36 uncovered, 0 errored, 17 timeout. Mutation Score Indicator (MSI): 73.91%

The first good message here is that the Original test suite passed. Crytic ran crystal spec [with all spec files] and that exited with exit code 0. Any other result on your inital test suite and it would not have made sense to continue. Intentionally breaking source code which is already broken is of no use.

Each occurance of shows that a mutant has been killed, ergo that the change in the source code was detected by the test suite. The line and column numbers are printed to follow the progress through the subject file.

❌ AndOrSwap is signaling that indeed a mutant, an intentional change in the subject, was not detected. The diff below shows the change that was made which was not caught by the test suite.

Mutation Badge

To show a badge about your mutation testing efforts like at the top of this readme you can make use of the dashboard of stryker by letting crytic post the msi score to the stryker API. To do that, make sure to have the following env vars set:

CIRCLE_BRANCH             => "master",
CIRCLE_PROJECT_REPONAME   => "crytic",
CIRCLE_PROJECT_USERNAME   => "hanneskaeufler",
STRYKER_DASHBOARD_API_KEY => "apikey",

It is currently limited to work with Circle CI and assumes your project is hosted on GitHub.

Available mutants

There are many ways a code-base can be modified to introduce arbitrary failures. Crytic only provides mutators which keep the code compiling (at least in theory). Currently, available mutators are:

AndOrSwap

This mutant replaces the && operator by the || operator and vice-versa. A typical mutation is:

- if cool && nice
+ if cool || nice

BoolLiteralFlip

This mutant flips literal occurances of true or false. A typical mutation is:

  def valid
-   return true
+   return false
  end

ConditionFlip

This mutant flips the if and else branch in conditions. It will create an else branch even if there is none. A typical mutation is:

  if true
+ else
    doSomething()
  end

NumberLiteralChange

This mutant changes literal occurances of numbers by replacing them with "0". "0" gets replaces by "1". A typical mutation is:

-  0
+  1

NumberLiteralSignChange

This mutant changes the sign of literal numbers. It ignores literal "0". A typical mutation is:

- 5
+ -5

StringLiteralChange

This mutant changes literal occurances of string by replacing empty strings with __crytic__ and all other strings with the empty string. Typical mutations are:

- "Welcome"
+ ""

AnyAllSwap

This mutant exchanges calls to Enumerable#all? with calls to Enumerable#any? and vice-versa. A typical mutation is:

- [false].all?
+ [false].any?

RegexpLiteralChange

This mutant modifies any regular expression literal to never match anything. A typical mutation is:

- /\d+/
+ /a^/

SelectRejectSwap

This mutant exchanges calls to Enumerable#select with calls to Enumerable#reject and vice-versa. A typical mutation is:

- [1].select(&.nil?)
+ [1].reject(&.nil?)

Credits & inspiration

I have to credit the crystal code-coverage shard which finally helped me create a working mutation testing tool after one or two failed attempts. I took heavy inspirations from its SourceFile class and actually lifted nearly all the code.

One of the more difficult parts of crytic was the resolving of require statements. In order to work for most projects, crytic has to resolve those statements identical to the way crystal itself does. I achieved this (for now) by copying a bunch of methods from crystal-lang itself.

In order to avoid dependencies for tiny amounts of savings I rather copied/adapted a bit of code from timeout.cr and crystal-diff.

Obviously I didn't invent mutation testing. While I cannot remember where I have read about it initially, my first recollection is the mutant gem for ruby. Markus Shirp, author of mutant is also the one who explained the notion of a "neutral" mutant to me in private chat. I took his idea and implemented it in crytic as well. Thanks!

The logo above is free from icons8.com.

Alternatives

Although not having tested it myself yet, the mull libray is supposed to work for any llvm based language, which I believe crystal is.

Execution flow

The following diagram shows a rough sequence of how crytic works. It was generated in mermaidjs.github.io.

Sequence flow

Contributing

  1. Fork it (https://github.com/hanneskaeufler/crytic/fork)
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Run tests locally with crystal spec
  4. Commit your changes (git commit -am 'Add some feature')
  5. Push to the branch (git push origin my-new-feature)
  6. Create a new Pull Request

Contributors

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased] - Unknown

[5.0.1] - 2019-03-09

Fixed

  • When the production code requires e.g. ./html/builder, crytic will mimic crystal langs require and find html/builder/builder.cr

[5.0.0] - 2019-02-17

Added

  • New "File Summary" reporter that list the covered subjects and the number of mutations that were performed on each of those files respectively
  • When the neutral mutation errored, the output is shown as well
  • Ability to enable/disable reporters with --reporters/-r flag. Current reporters are Console, Stryker and ConsoleFileSummary

Changed

  • When no mutations were run, crytic now exits with 1 instead of 0
  • The StringLiteralChange mutant now performs more efficient replacements

Fixed

  • Don't crash in the stryker dashboard reporter when zero mutations were run
  • Fixed reporting of the number of mutations being run

[4.0.0] - 2019-01-31

Added

  • Run a "neutral" mutation before each subjects real mutations. This is @mjb's idea to validate the infrastructure of injecting mutations. Currently a "noop"-mutation is run, which simply doesn't mutate the subject at all.

Changed

  • The AndOrSwap mutant now swaps both || to && and && to ||

[3.2.3] - 2019-01-25

Fixed

  • The AndOrSwap mutant accidentally mutated multiple && at the same time

[3.2.2] - 2019-01-20

Fixed

  • Due to a regression introduced in 5a02821cce6bd27361bc84d5b073b21dc2fa55f0, require statements that don't yield any files could be left in the mutated code, leading to compile errors which looked like killed mutants

[3.2.1] - 2019-01-15

Fixed

  • Fix filename reporting introduced in 3.2.0

[3.2.0] - 2019-01-15

Added

  • Show filename (and line and col numbers) for both killed and surviving mutants in the console output

Fixed

  • Mutants AnyAllSwap and AndOrSwap could skip possible mutations

[3.1.1] - 2019-01-09

Added

  • Add --preamble cli option to pass code that is prepended. Helpful to allow usage together with e.g. (minitest.cr)[https://github.com/ysbaddaden/minitest.cr]

[3.0.1] - 2019-01-08

Fixed

  • Exit after printing usage information, thanks @anicholson
  • Don't mutate unsigned integer literals like 1_u16 with the NumberLiteralSignFlip

[3.0.0] - 2019-01-03

Added

  • Simply running ./bin/crytic without any arguments will now automatically find all src files and specs
  • Introduced a mutant to swap [1].all? for [1].any?
  • Report number of mutations being run in console output
  • Introduced a mutant to swap any RegexLiteral for /a^/ which will never match
  • Enabled the mutant to swap #reject for #select and vice-versa

Fixed

  • When the mutated source code fails to compile this is now being noted correctly
  • Negative numbers are now correctly flipped to positive ones (e.g. -1 => 1 instead of -1 => --1)
  • Timeouts in mutations are printed as "not found" but are actually found and calculated as "killed". Fixed this so that timeouts are not showing a diff in the console output any more.
  • Errors resulting from mutations are printed as "bad" but actually mean that they were detected. Fixed this so that errors are not showing a diff in the console output.

[2.0.0] - 2018-12-06

Added

  • --min-msi cli argument to allow passing the suite (exiting with 0) even when there are mutants that survived. Pass as float like --min-msi=75.0.
  • Post MSI score to stryker dashboard if env vars are set.

Changed

  • NumberLiteralChange mutant now outputs 0 (for everything != 0) and 1 (for 0)
  • Depending on crystal 0.27.0, dropping all previous versions

[1.2.0] - 2018-10-29

Added

  • This changelog
  • Avoid hanging forever by imposing a timeout for mutations
  • Use "fail fast" option of crystal spec runner

Changed

  • Calculate the mutation score as MSI, described in infection

[1.1.0] - 2018-10-23

Added

  • More mutants: AndOrSwap, StringLiteralChange
  • Run CI on crystal 0.26.1 as well
  • Report a summary in the cli output

Changed

  • Don't report the number of times a mutant was run (e.g. (x2)) in the cli output

Fixed

  • Running multiple spec files as the test suite

[1.0.0] - 2018-10-20

Added

  • Everything. First release 🚀 🎉 💃
Github statistic:
  • 34
  • 2
  • 1
  • 3
  • about 1 month ago

License:

MIT License

Links: