adam/vim
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

patch 9.1.0681: tests: Analyzing failed screendumps is hard

Browse Source 
Problem:  tests: Analyzing failed screendumps is hard
Solution: Facilitate the viewing of rendered screendumps under src/
          add some documentation on how to use the viewdumps.vim
          script (Aliaksei Budavei)

With the submitted "viewdumps.vim" script, a few manual
steps in typical workflows (see below) can be automated.
The updated "README.txt" contains additional information.

============================================================

Reviewing LOCAL failed screendump tests can be arranged as
follows:

1) Run tests and generate screendumps:
------------------------------------------------------------
cd /path/to/fork/src/testdir
make
------------------------------------------------------------

2) Examine the screendumps from the "failed" directory:
------------------------------------------------------------
../vim -u NONE -S viewdumps.vim
------------------------------------------------------------

============================================================

Reviewing UPLOADED failed screendump tests can be arranged
as follows (it can be further locally scripted):

1) Fetch an artifact with failed screendumps from
"github.com/vim/vim/actions/runs/A_ID/artifacts/B_ID".

2) Extract the archived files:
------------------------------------------------------------
unzip /tmp/failed-tests.zip -d /tmp
------------------------------------------------------------

3) Set up the "dumps" directory.  Create a symlink to
"/path/to/fork/dirs/dumps" in the extracted directories so
that term_dumpdiff() can be used.  (The lookup algorithm
resolves "dumps" for every loaded filename.  So, with
"/tmp/src/testdir/failed/*.dump" files passed as script
arguments, the algorithm will make the files in
"/tmp/src/testdir/dumps" queried.)
------------------------------------------------------------
cd /path/to/fork
ln -s $(pwd)/src/testdir/dumps /tmp/src/testdir/dumps
------------------------------------------------------------

4) Examine the extracted screendumps:
------------------------------------------------------------
./src/vim -u NONE -S src/testdir/viewdumps.vim \
  /tmp/src/testdir/failed/*.dump
------------------------------------------------------------

5) Clean up:
------------------------------------------------------------
unlink /tmp/src/testdir/dumps
rm -rf /tmp/src
------------------------------------------------------------

============================================================

Reviewing SUBMITTED FOR PULL REQUEST screendump tests can be
arranged as follows (it can be further locally scripted):

1) List the fetched changeset and write the changed "dumps"
filenames to "/tmp/filelist":
------------------------------------------------------------
cd /path/to/fork
git switch prs/1234
git diff-index --relative=src/testdir/dumps/ \
  --name-only prs/1234~1 > /tmp/filelist
------------------------------------------------------------

2) Reconcile relative filepaths, and copy next-to-be-updated
"dumps" files in the "failed" directory (note the missing
new screendumps, if any):
------------------------------------------------------------
git switch master
cd src/testdir/dumps
test -d ../failed || mkdir ../failed
cp -t ../failed $(cat /tmp/filelist)
------------------------------------------------------------

3) Remember about the introduced INVERTED relation between
"dumps" and "failed", i.e. the files to be committed are in
"dumps" already and their previous versions are in "failed";
therefore, copy the missing new screendumps from "dumps" to
"failed" (otherwise these won't be shown):
------------------------------------------------------------
git switch prs/1234
cp -t ../failed foo_10.dump foo_11.dump foo_12.dump
------------------------------------------------------------

4) Examine the screendumps from the "failed" directory (new
screendumps will be shown with no difference between their
versions):
------------------------------------------------------------
cd ..
../vim -u NONE -S viewdumps.vim
------------------------------------------------------------

closes: #15515

Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
This commit is contained in:
Aliaksei Budavei
2024-08-19 21:33:26 +02:00
committed by Christian Brabandt
parent 175a87c7f1
commit 6bff6a2fa4
7 changed files with 231 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
runtime/syntax/testdir/README.txt
View File

@ -124,8 +124,8 @@ verify that the tests fail. Then you know your changes are covered by the
test.
Viewing generated screendumps
-----------------------------
Viewing generated screendumps (local)
-------------------------------------
You may also wish to look at the whole batch of failed screendumps after
running "make test". Source the "viewdumps.vim" script for this task:
@ -151,5 +151,60 @@ At any time, you can add, list, and abandon other screendumps:
The listing of argument commands can be found under :help buffer-list.
Viewing generated screendumps (from a CI-uploaded artifact)
-----------------------------------------------------------
After you have downloaded an artifact archive containing failed screendumps
and extracted its files in a temporary directory, you need to set up a "dumps"
directory by creating a symlink:
cd /path/to/fork
ln -s $(pwd)/runtime/syntax/testdir/dumps \
/tmp/runtime/syntax/testdir/dumps
You can now examine the extracted screendumps:
./src/vim --clean -S runtime/syntax/testdir/viewdumps.vim \
/tmp/runtime/syntax/testdir/failed/*.dump
Viewing generated screendumps (submitted for a pull request)
------------------------------------------------------------
First, you need to check out the topic branch with the proposed changes and
write down a difference list between the HEAD commit (index) and its parent
commit with respect to the changed "dumps" filenames:
cd /path/to/fork
git switch prs/1234
git diff-index --relative=runtime/syntax/testdir/dumps/ \
--name-only prs/1234~1 > /tmp/filelist
Then, you need to check out the master branch, change the current working
directory to reconcile relative filepaths written in the filenames list, copy
in the "failed" directory the old "dumps" files, whose names are on the same
list, and follow it by checking out the topic branch:
git switch master
cd runtime/syntax/testdir/dumps
cp -t ../failed $(cat /tmp/filelist)
git switch prs/1234
Make note of any missing new screendumps. Please remember about the
introduced INVERTED relation between "dumps" and "failed", i.e. the files to
be committed are in "dumps" already and their old versions are in "failed".
Therefore, you need to copy the missing new screendumps from "dumps" to
"failed":
cp -t ../failed foo_10.dump foo_11.dump foo_12.dump
After you have changed the current working directory to its parent directory,
you can now examine the screendumps from the "failed" directory (note that new
screendumps will be shown with no difference between their versions):
cd ..
../../../src/vim --clean -S viewdumps.vim
TODO: run test for one specific filetype
TODO: test syncing by jumping around