As reported in #16559, bytes of a multibyte character may
be written as separate U+FFFD characters in a ":terminal"
window on a busy machine. The testing facilities currently
offer an optional filtering step to be carried out between
reading and comparing the contents of two screendump files
for each such file. This filtering has been resorted to
(#14767 and #16560) in an attempt to unconditionally replace
known non-Latin-1 characters with an arbitrary substitute
ASCII character and avoid this rendering mishap leading to
syntax tests failures. However, it has been overlooked at
the time that metadata description (in shorthand) to follow
spurious U+FFFD characters may be *distinct* and make the
remainder of such a line, ASCII characters and whatnot, also
unequal between compared screendump files.
While it is straightforward to adapt current filter files to
ignore the line characters after the leftmost U+FFFD,
> It is challenging and error-prone to keep up to date filter
> files because moving around examples in source files will
> likely make redundant some previously required filter files
> and, at the same time, it may require creating new filter
> files for the same source file; substituting one multibyte
> character for another multibyte character will also demand
> a coordinated change for filter files.
Besides, unconditionally dropping arbitrary parts of a line
is rather too blunt an instrument. An alternative approach
is to not use the supported filtering for this purpose; let
a syntax test pass or fail initially; then *if* the same
failure is imminent, drop the leftmost U+FFFD and the rest
of the previously seen line (repeating it for all previously
seen unequal lines) before another round of file contents
comparing. The obvious disadvantage with this filtering,
unconditional and otherwise, is that if there are consistent
failures for _other reasons_ and the unequal parts happen to
be after U+FFFDs, then spurious test passing can happen when
stars align for _a particular test runner_.
Hence syntax test authors should strive to write as little
significant text after multibyte characters as syntactically
permissible, write multibyte characters closer to EOL in
general, and make sure that their checked-in and published
"*.dump" files do not have any U+FFFDs.
It is also practical to refrain from attempting screendump
generation if U+FFFDs can already be discovered, and instead
try re-running from scratch the syntax test in hand, while
accepting other recently generated screendumps without going
through with new rounds of verification.
Reference:
https://github.com/vim/vim/pull/16470#issuecomment-2599848525closes: #17704
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
The process of preparing and submitting syntax tests is
fraught with challenges that can turn away many aspiring
contributors from ever attempting it. (Out of 69 languages
introduced since v9.0.1627, there are only syntax tests for
Tera.)
After v9.1.1176~1, one visual clue for admitting syntax test
failures previously available with e.g. "git status" is gone
after all files under "failed/" have been made ignored for
Git and Mercurial. There isn't a single way to go about it:
some people may move files from "failed/" to "dumps/" after
each iteration; some people may only move "good" iteration
files; when a test file is refactored to a great extent,
some people may prefer deleting all test-related files under
"dumps/" before moving files from "failed/". The usability
of reporting, at any time, that there are some _untracked_
files under "failed/" cannot be overstated. Without it, the
chances are greater for pushing mismatched changesets. And
when tests fail then everyone but the author will be kept in
the dark about the cause: were some updated screendumps not
committed _or_ was a wrong version of the syntax plugin
committed?
Another file, "testdir/Xfilter" (v9.1.0763), that will be
created to establish communication from Make to Vim about
what subset of syntax tests is requested for running, should
also be not ignored but rather deleted once its contents are
read. Unless it is explicitly deleted _after test failure_,
the file may contain new *and* old test names when another
testing attempt is under way. And by virtue of it being
ignored, the reason for also running not requested tests
will be as ever puzzling.
Both Git and Mercurial support per-user configuration; such
wide-reaching settings hardly belong to clonable defaults.
Also, match literal dots in testname filters.
Also, discover and report _some_ disused screendump files
tracked under "dumps/".
References:
- https://git-scm.com/docs/gitignore
- https://www.mercurial-scm.org/help/topics/config#uicloses: #16917
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
As of patch v9.1.1176~1, there are no longer makeshift Make
targets to accommodate language names that can match phoney
targets. For example, "clean_" was previously generated for
Clean because otherwise it clashed with the "clean" target.
Additionally, enable test filtering for makeshift targets.
Reference:
https://wiki.clean.cs.ru.nl/Cleancloses: #16810
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
When a syntax file is changed, timestamps of the corresponding files are
updated.
NOTE: At the moment this script does not strictly track dependency, like
cpp on c.
Also update ignore files
closes#16548
Co-authored-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Eisuke Kawashima <e-kwsm@users.noreply.github.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
The current lightweight synchronisation with ":redraw" needs further
reinforcement in the light of v9.1.1110. And, with v9.1.0820, make
another synchronisation point _before_ the first (or only) screenful is
dumped.
Also add a script to regenerate all screendumps.
closes: #16632
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Currently syntax tests outputs all the Vim commands to the console,
which is annoying for a local developer but also makes the CI output
impossible to parse and just printing all the terminal output comes with
a performance penalty.
So let's just simply redirect all output to /dev/null similar to what
the script tests do.
This can be turned off locally for someone diagnosing issues but it
should not be the default behavior to output all terminal control
sequences to output in CI.
closes: #16612
Signed-off-by: Yee Cheng Chin <ychin.git@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: tests: cannot run single syntax tests
Solution: Support running a subset of syntax tests
(Aliaksei Budavei)
Two methods of assembling a subset of test filenames for
selection are provided:
* Filename and filetype Make targets will be generated, and
multiple such targets can be passed before the mandated
trailing "test" target, e.g. "make html markdown test".
* Filenames and their parts can be specified as a regular
expression that is assigned to a "VIM_SYNTAX_TEST_FILTER"
environment variable, and used with the test Make target,
e.g. "VIM_SYNTAX_TEST_FILTER=html\\\|markdown make test".
(This variable will be ignored and the whole suite will be
run when Make is GNU Make and a parent Makefile is used.)
Methods can be used alone or together, with the Make targets
having the higher precedence. Neither method will influence
the order of test execution.
closes: #15670
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
The whole testdir/runtest.vim is a bit of a mess. So this is an attempt
to improving the current state. First of all, let's move most of the
code into a single RunTest() function so that it's easier to run this
manually.
Then, let's add the possibility to customize filetype specific tests by
applying filetype specific options. We will make use of it in the next
commit, where the provided java file uses tabs, which cause line
wrapping which will then cause the scrolling to miss some lines when
redrawing (see #14105)
related: #14105
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: Syntax test fails when run with non C locale
Solution: Run syntax tests with C locale, clean up Xtestscript file,
strip environment variables from GetVimCommand()
(h-east)
closes: #14007
Co-authored-by: h-east <h.east.727@gmail.com>
Signed-off-by: h-east <h.east.727@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: runtime tests fail with tiny vim
Solution: check for tiny vim, run runtime tests in CI
even for tiny version
closes: #13169closes: #13170
Signed-off-by: Christian Brabandt <cb@256bit.org>
Co-authored-by: Dominique Pellé <dominique.pelle@tomtom.com>
Problem: Insufficient testing for syntax plugins.
Solution: Add shell file examples. (Charles Campbell) Create a messages
file for easier debugging and reporting the test results.
