created by

Ingo Karkat

script type

utility

description

Every script writer knows how tedious it is to update a published plugin. In addition to the basic functionality, many corner cases (empty line, last line, etc.) and failures (invalid filename, nomodifiable buffer) need to be tested, and people use Vim on different platforms (Windows / Linux), UIs (Console / GUI), with different sets of settings (.vimrc) and other loaded plugins. There already exist multiple unit test plugins and assertion facilities [1], which are good for checking invariants and verifying side effect-free implementation functions, but it is still hard to verify the complete plugin functionality because custom commands and mappings typically change the buffer contents, open additional windows, or produce other side effects. This work aims to show that Vim, together with a shell script driver built around it, allows to write succinct, fully automated regression test suites through a combination of these verification methods: - Comparing buffer contents with a predefined nominal file. - Matching actual Vim message output with a set of expected messages. - Running unit tests or assertions inside Vim and evaluating the test results. With this testing framework, Test-Driven Development can finally be practiced for Vim plugins, too. If you have existing plugins, just add a couple of basic test cases for a start. Soon, further updates and modifications can be done much more rapidly with reduced testing effort, and you can finally tackle that big refactoring that you've been wanting to do all the time, but were too afraid of because of the testing effort. The author has been following both approaches with great success. [1] - vimUnit (by Staale Flock), vimscript #1125 - tAssert (by Tom Link), vimscript #1730 - VimTAP (by Meikel Brandmeyer), vimscript #2213 - spec.vim (by Tom Link), vimscript #2580 - UT (by Luc Hermitte), http://code.google.com/p/lh-vim/wiki/UT - vim-unittest (by h1mesuke), https://github.com/h1mesuke/vim-unittest - Ultimate Test Utility (by Kevin Biskar), vimscript #4724 is a pure Vimscript   implementation providing various test grouping and assertion functions. RELATED WORKS - robot-vim (https://github.com/mrmargolis/robot-vim) by Matt Margolis allows   to TDD Vim scripts using Ruby scripts that launch Vim instances, pass text,   execute commands and then run assertions against the buffer text in Ruby. - vspec (vimscript #3012) by Kana Natsuno allows to write tests BDD-style with   custom matchers, and is driven by a small Bash script. - Vader (vimscript #4832) by Junegunn Choi implements BDD-style testing from   within Vim via a test script in a custom syntax. EXAMPLE Specify an expected output, e.g. in file 'test001.ok'     expected output     and more     expected output and/or specify expected messages in file 'test001.msgok'     /\d\+ lines changed/ use an input file 'test001.in'     ExPeCteD OuTpUt     AND MORE     ExPeCteD OuTpUt write a short test script 'test001.vim'     edit test001.in     normal! gg3gugu     write test001.out     quit! call the test runner     $ runVimTests test001.vim and the framework invokes the test(s) [suites], compares the actual with the expected output, checks that the messages match, and prints any failures and a test summary:     1 file with 2 tests; 0 skipped, 2 run: 2 OK, 0 failures, 0 errors. DESCRIPTION runVimTests implements a testing framework for Vim. Similar to the tests that are part of Vim's source distribution, each test consists of a testXXX.vim file which is executed in a a separate Vim instance. The outcome of a test can be determined by a combination of the following methods: SAVED BUFFER OUTPUT If a testXXX.ok file is provided, the testXXX.vim should save a testXXX.out file at the end of its execution. The contents of the testXXX.out test file must be identical to the provided testXXX.ok file for the test to succeed. The test can either generate the test output itself, or start by editing a testXXX.in (or similar) file and doing modifications to it. Use this method to test commands or mappings that modify buffer contents. CAPTURED MESSAGES If a testXXX.msgok file is provided, the testXXX.vim file should generate Vim messages (from built-in Vim commands, or via :echo[msg]), which are captured during test execution in a testXXX.msgout file. The testXXX.msgok file contains multiple message assertions (separated by empty lines), each of which is compiled into a Vim regexp and then matched against the captured messages. Each assertion can match exactly once, and all assertions must match in the same order in the captured Vim messages. (But there can be additional Vim messages before, after and in between matches, so that you can omit irrelevant or platform-specific messages from the testXXX.msgok file.) For details, see runVimMsgFilter. This method can verify that errors are reported correctly. Also use this method to test commands or mappings that print informational messages. TAP UNIT TESTS If a testXXX.tap file exists at the end of a test execution, it is assumed to represent unit test output in the Test Anything Protocol [2], which is then parsed and incorporated into the test run. This method allows detailed verification of custom commands, mappings as well as internal functions; the entire determination of the test result is done in Vim script. Each TAP unit test counts as one test, even though all those test results are produced by a single testXXX.vim file. If a plan announced more or less tests than what was found in the test output, the test is assumed to be erroneous. Use this method to test the internal implementation functions, or to verify things that can be checked easily with Vim script. [2] web site: http://testanything.org, original implementation: http://search.cpan.org/~petdance/TAP-1.00/TAP.pm, TAP protocol for Vim: http://www.vim.org/scripts/script.php?script_id=2213 A test causes an error if none of these ok-files exist for a test, and no testXXX.tap file was generated (so actually no verification is possible), or if the test execution does not produce the corresponding output files. USAGE A test run is started through the "runVimTests.(sh|cmd)" script:     $ runVimTests [{options}] test001.vim|testsuite.txt|path/to/testdir/ [...] The tests are specified through these three methods, which can be combined: - Directly specify the filespec of testXXX.vim test script file(s). - Specify a directory; all *.vim files inside this directory (except for an   optional special _setup.vim file) will be used as test scripts. - A test suite is a text file containing (relative or absolute) filespecs to   test scripts, directories or other test suites, one filespec per line.   (Commented lines start with #.) The script returns 0 if all tests were successful, 1 if any errors or failures occurred, 2 if it wasn't invoked correctly (i.e. bad or missing command-line arguments) or prerequisites weren't met, 3 in case of an internal error. After test execution, a summary is printed like this:     20 files with 33 tests; 2 skipped, 27 run: 16 OK, 11 failures, 6 errors.     Tests with skips: test006.vim     Skipped tests: test007.vim         2 SKIP (tap): Need 'autochdir' option     Failed tests: test002.vim test012.vim test014.vim test022.vim test032.vim     Tests with errors: test003.vim test013.vim test023.vim test033.vim

install details

INSTALLATION This framework is packaged as a ZIP archive. You can unpack it directly into your-runtime-dir (~/.vim), but you can also install the script executables somewhere else. The script executables are in the bin/ subdirectory:     bin/runVimTests.cmd     bin/runVimTests.sh     bin/runVimMsgFilter.vim They can be put anywhere (preferably somewhere in $PATH for easy invocation). runVimMsgFilter.vim must be in the same directory as the shell script. The doc/ subdirectory contains the documentation. Put the files into ~/.vim/doc and execute :helptags ~/.vim/doc to re-generate the help tags. The autoload/ subdirectory contains optional convenience and helper functions. (E.g. vimtest#Quit(), vimtest#SaveOut(), vimtest#RequestInput(), and more. See vimtest-usage for a full reference.) Their use is not required, but they simplify the writing of tests. The VimTAP plugin (vimscript #2213) needs to be installed separately. The examples here use the API of version 0.3.0; version 0.4.0 of the plugin introduced compatibility-breaking changes, which I don't like; the functions are now cumbersome to use, and the added commands don't offer much convenience, but pollute the test environment in my opinion. If you like, you can use my own fork of version 0.3.0:     https://github.com/inkarkat/VimTAP Note that nothing prevents you from using the latest, original version for your own tests. The default runVimTestsSetup.vim will automatically include VimTAP into 'runtimepath' if it's located in a repository next to runVimTests. The tests/ subdirectory contains example test suites and a self-test of the test framework. For a simple sanity check, execute:     $ runVimTests tests/runVimTests/successful.suite which should print something like:     9 files with 19 tests; 0 skipped, 19 run: 19 OK, 0 failures, 0 errors. If this is the case, you can start exploring the example tests (in the tests/example/ subdirectory) or just start writing your own runVimTests-testscripts! DEPENDENCIES - Requires Vim 7.2 or higher as the default Vim found in $PATH (which is   always used for the matching of Vim message output against the captured   messages). You can use a different Vim version to execute the tests, but at   least Vim 7.0 is required to use captured messages (as this depends on the   'verbosefile' option) and TAP unit tests (vimtap.vim is an autoload script).   The saved buffer output method even works with Vim 6, but the driver will   generate errors in that case. The Windows version requires (* = optional) these ported Unix tools: - grep, sed, diff, sort(*), uniq(*) Windows binaries can be downloaded from the GnuWin32 project:     http://gnuwin32.sourceforge.net/ These binaries must be accessible through %PATH%. Alternatively, you can just make accessible an Windows shell script named "unix.cmd", which is then sourced in order to modify %PATH% to include the Unix tools. This way, you can avoid to permanently place these Unix tools into %PATH%. - Requires the ingo-library.vim plugin (vimscript #4433), version 1.012 or   higher, only for Vim 7.0 and 7.1. CONTRIBUTING The code is hosted in a Git repo at https://github.com/inkarkat/runVimTests Report any bugs, send patches, or suggest features via email (address below), or via the issue tracker at https://github.com/inkarkat/runVimTests/issues

Click on the package to download.

package	script version	date	Vim version	user	release notes
runVimTests-1.30.zip	1.30	2020-02-03	7.2	Ingo Karkat	- Add vimtest#AddDependency() and vimtest#features#SupportsNormalWithCount(). - CHG: Print full absolute path to tests instead of just the test name itself. When running complete suites or tests with subdirectories, it is difficult to locate a failing test with just the name. - ENH: Add -o|--output parameter that redirects all script output into a FILESPEC or &N file descriptor. Piping the entire output of runVimTests is problematic because the started Vim instances expect to write the UI to stdout; without that, the screen does not update / is messed up, and you cannot do debugging in there.
runVimTests-1.25.zip	1.25	2017-05-09	7.0	Ingo Karkat	- vimtest#Quit(): Don't exit Vim when not running inside the runVimTests test framework. This is better behavior when accidentally (or for testing) executing the test script in plain Vim. - FIX: "cd" might print the path when CDPATH is set; discard it. Thanks to Raimondi for sending a pull request. https://github.com/inkarkat/runVimTests/pull/11 - ENH: Make runVimTestsSetup.vim accept any directory starting with VimTAP; thanks to Marcelo Montu for the pull request. https://github.com/inkarkat/runVimTests/pull/9 - With -1 / --default, newer Vim versions still pick up user plugins from the ~/.vim/pack directory. Temporarily modify 'packpath' during Vim startup to avoid that.
runVimTests-1.24.zip	1.24	2014-01-29	7.0	Ingo Karkat	- Don't clobber the default viminfo file with the test results; use a special ~/.vimtestinfo value for the actual test run (to enable tests that use viminfo), and no viminfo for the checking and processing steps. - Show _all_ global, non-test-specific Vim arguments in the initial message. - Convert the filespec passed to --source to an absolute one; relative ones only work when the test driver script doesn't cd into a different directory. - BUG: runVimTestsSetup.vim isn't sourced on Unix when invoked through runVimTests.sh (with the .sh file extension). Reported by Ryan Carney. https://github.com/inkarkat/runVimTests/issues/6 - Replace the distributed escapings.vim autoload script with an optional dependency (for Vim 7.0 and 7.1) to the ingo-library. *** You need to separately install ingo-library (vimscript #4433) version 1.012 (or higher)! ***
runVimTests-1.23.zip	1.23	2013-03-26	7.0	Ingo Karkat	- Add support for Mac OS X; thanks to Israel Chauca Fuentes for sending a pull request. https://github.com/inkarkat/runVimTests/pull/1 - Add support for BSD (tested on PC-BSD 9.1).
runVimTests-1.22.zip	1.22	2013-03-15	7.0	Ingo Karkat	- Include the version 1.21 changes in the Windows runVimTests.cmd, too. - Switch to Git for the plugin's development to prevent such omissions (caused by my manual syncing, which is more complex in this case with files distributed over many different directory trees).
runVimTests-1.21.zip	1.21	2013-03-07	7.0	Ingo Karkat	- FIX: Prevent script errors when the error message containing the full command line from a failing vimtest#System() contains characters like ['"()]. - CHG: Drop comma in the lists of failed / skipped / errored test and add .vim extension, so that the file list can be copy-and-pasted to another runVimTests invocation or :argedit'ed in Vim. - CHG: Change default mode from "user" to "default"; this is what I use all the time, anyway, as the "user" mode is too susceptible to incompatible customizations.
runVimTests.zip	1.20	2012-07-27	7.0	Ingo Karkat	- ENH: Handle file globs in the passed tests and in suite entries on Windows, too. (In contrast to the Unix shell, these must be explicitly expanded on Windows.) - BUG: In the Windows test runner, remove duplicate quoting when vimExecutable isn't found. This actually prevented execution when passing --vimexecutable "C:\Program Files (x86)\vim\vim73\vim.exe"
runVimTests.zip	1.18	2011-10-19	7.0	Ingo Karkat	- BUG: When everything is skipped and no TAP tests have been run, this would be reported as a "No test results at all" error. - CHG: Bail out only aborts from the current recursion level, i.e. it skips further tests in the same directory, suite, or passed arguments, but not testing entirely. Otherwise, a super-suite that includes individual suites would be aborted by a single bail out.
runVimTests.zip	1.17	2011-09-04	7.0	Ingo Karkat	- BUG: When runVimTests.sh is invoked via a relative filespec, $scriptDir is relative and this makes the message output comparison (but not the prerequisite check) fail with "ERROR (msgout): Evaluation of test messages failed." when CWD has changed into $testDirspec. Thanks to Javier Rojas for sending a patch.
runVimTests.zip	1.16	2011-02-28	7.0	Ingo Karkat	- Minor bugfixes and tweaks to the self-test. - Renamed directory that the tests reside in from "test/" to "tests/". This is just a personal preference, you can still put the tests into whatever directory structure. - ENH: Added vimtest#ErrorAndQuit() for convenience. - bugfix and enhancement to the vimtest#System() function.
runVimTests.zip	1.13	2009-05-29	7.0	Ingo Karkat	ENH: Now including SKIP reasons in the summary (identical reasons are condensed and counted) when not running with verbose output. I always wanted to know why certain tests were skipped. (This requires GNU sort and uniq on Windows.)
runVimTests.zip	1.12	2009-03-14	7.0	Ingo Karkat	- Added quoting of regexp in runVimTests.sh, which is needed in bash 3.0 and 3.1. Thanks to Anders Thøgersen for the patch. - Now checking bash version. - Only exiting with exit code 1 in case of test failures; using code 2 for invocation errors (i.e. wrong command-line arguments) and code 3 for internal errors.
runVimTests.zip	1.11	2009-03-12	7.0	Ingo Karkat	- TAP output is now parsed for SKIP and TODO directives, and the "Bail out" message. - TODO TAP tests are included in the test output like failing tests.
runVimTests.zip	1.10	2009-03-11	7.0	Ingo Karkat	- runVimTests drivers now also count test files (*.vim). - ENH: Message output is now parsed for signals to the test driver. Implemented signals: BAILOUT!, ERROR, SKIP, SKIP(out), SKIP(msgout), SKIP(tap).
runVimTests.zip	1.01	2009-03-03	7.0	Ingo Karkat	Added annotated example tests.
runVimTests.zip	1.00	2009-03-02	7.0	Ingo Karkat	Initial upload

ip used for rating: 18.222.10.9