summaryrefslogtreecommitdiff
path: root/testing/tools/test_runner.py
AgeCommit message (Collapse)Author
2018-06-21Add pass/fail Gold summary to outputchromium/3469Stephan Altmueller
- Changes the URL to retrieve baselines which returns the baselines based on the issue number (if available) - Compare against a single baseline (instead of multiple) - Writes the results of comparing to the baselines to a separate output file that can then be parsed by the recipe code. See complementary CLs: https://skia-review.googlesource.com/c/buildbot/+/131923 https://chromium-review.googlesource.com/c/chromium/tools/build/+/1085853 Change-Id: I90cdd77d0d26e63a5e686f4862fc68dc9c4429d3 Reviewed-on: https://pdfium-review.googlesource.com/33772 Commit-Queue: Stephan Altmueller <stephana@chromium.org> Reviewed-by: Ryan Harrison <rharrison@chromium.org>
2018-05-31Make XFA pixel tests more stablechromium/3447Ryan Harrison
When generating PDFs using the fixup script and a .in, specify them to to use Ahem.ttf as the text font. Ahem is a font designed to reduce flakiness in pixel tests due to font rendering differences between platforms. Specifically the glyphs in the font are standard size black boxes, so though the specific text content cannot be easily compared, things like layout and high level changes to the content can still be tested. Testing things like specific text changes should be done via an embedder test, where the strings in the elements can be extracted and inspected. The font itself is is CID type 2 font, which is relatively complex to correctly embed in a PDF due to subsetting and other features. Instead of embedding it in the generated PDFs, which was originally attempted, it is being supplied to pdfium_test via the --font-dir flag. This flag overrides where the binary looks for system fonts. This works correctly on Mac and Linux, but not on Windows, which is why that platform remains suppressed. BUG=pdfium:1008,pdfium:1020 Change-Id: I00811536de98f736fc599d96b397194ccf8db0cd Reviewed-on: https://pdfium-review.googlesource.com/27790 Commit-Queue: Ryan Harrison <rharrison@chromium.org> Reviewed-by: Lei Zhang <thestig@chromium.org>
2018-05-10Use test_dir instead of 'pdfium' for source typeRyan Harrison
The source type value Gold results is just a logical grouping label. Having a different value for each type will allow the Gold web UI to have a set of labels at the top that allows you to easily switch between specific test tool results. Currently the test scripts are using the test directories 'corpus', 'javascript', and 'pixel', which seem like reasonable labels. BUG=pdfium:1073 Change-Id: I893a1afeb3b87ffb84948c18ae3052b2dd6216ea Reviewed-on: https://pdfium-review.googlesource.com/32350 Commit-Queue: Ryan Harrison <rharrison@chromium.org> Reviewed-by: Henrique Nakashima <hnakashima@chromium.org>
2018-04-26Improve diffing and regeneration of expected pngs.Henrique Nakashima
- If a .pdf had more than one page and any of them except the first had no expectation, that page and the ones after it were not diff'ed. - If a .pdf did not have expectations for any or all pages, --regen did not generate those expectations. Change-Id: Ie6dacf07e44feb4a14a6a92eb20c9aa19858a8f9 Reviewed-on: https://pdfium-review.googlesource.com/25770 Commit-Queue: Henrique Nakashima <hnakashima@chromium.org> Reviewed-by: Lei Zhang <thestig@chromium.org>
2018-04-09Sort test cases in test_runner.py.Lei Zhang
Allow them to run a bit more consistently. Change-Id: I6b0f528a73f2c872813f4644aeae886ec04879f4 Reviewed-on: https://pdfium-review.googlesource.com/29930 Reviewed-by: Henrique Nakashima <hnakashima@chromium.org> Commit-Queue: Lei Zhang <thestig@chromium.org>
2018-02-13Fix nits in Python code.Lei Zhang
Change-Id: I01956bff9e7250c9df84afe6b6de96ab4e9b4851 Reviewed-on: https://pdfium-review.googlesource.com/26650 Commit-Queue: Ryan Harrison <rharrison@chromium.org> Reviewed-by: Ryan Harrison <rharrison@chromium.org>
2017-11-29Read Skia gold json and compare to locally generated pngs.Henrique Nakashima
Change-Id: I73b6a477c1a66beec2b73072f23faab7db7d3342 Reviewed-on: https://pdfium-review.googlesource.com/18513 Commit-Queue: Henrique Nakashima <hnakashima@chromium.org> Reviewed-by: Lei Zhang <thestig@chromium.org>
2017-10-26Add option to regenerate only platform-specific expected pngs.Henrique Nakashima
Change-Id: Id4798fe9a4d297678a76d0511cde7fecbf130e3e Reviewed-on: https://pdfium-review.googlesource.com/16613 Commit-Queue: Henrique Nakashima <hnakashima@chromium.org> Reviewed-by: Nicolás Peña Moreno <npm@chromium.org>
2017-10-25Add --regenerate_expected option to test_runner.py.chromium/3250Henrique Nakashima
After a change in rendering, corpus and pixel test expected pngs need to be regenerated manually. This option reduces the toil by automating renaming and moving the new expected files over the old ones. Change-Id: I9d490369ccf946d4d4567e440b7b5252469eec46 Reviewed-on: https://pdfium-review.googlesource.com/16451 Commit-Queue: Henrique Nakashima <hnakashima@chromium.org> Reviewed-by: Nicolás Peña Moreno <npm@chromium.org>
2017-08-31Implement FORM_OnFocus() API.Lei Zhang
Given a point, if there is an annotation at the point, give if focus if it is not focus already. If there is no annotation at the point, then remove the focus from the focused annotation. BUG=chromium:754594 Change-Id: Iec3070472bbbfbad9d86e517f25da560f82efd4e Reviewed-on: https://pdfium-review.googlesource.com/12530 Commit-Queue: Lei Zhang <thestig@chromium.org> Reviewed-by: dsinclair <dsinclair@chromium.org>
2017-07-18Make progressive renderer default for pdfium_test.Dan Sinclair
When Chrome renders PDF pages it uses the progressive renderer. The one shot renderer is used to render print preview pages. This CL changes pdfium_test to use the progressive renderer by default to better match Chrome. This also adds a --render-oneshot flag which will use the non-progressive renderer. Bug: chromium:736695 Change-Id: If79333ae10a0ed1c9bce1290caad8d531e6709aa Reviewed-on: https://pdfium-review.googlesource.com/8130 Reviewed-by: Tom Sepez <tsepez@chromium.org> Commit-Queue: dsinclair <dsinclair@chromium.org>
2017-06-27Change terminology of image diff to differentiate from pixel tests.Henrique Nakashima
Change-Id: I957199b63f6049a586b8de2688501f5ae3449964 Reviewed-on: https://pdfium-review.googlesource.com/7038 Reviewed-by: Nicolás Peña <npm@chromium.org> Commit-Queue: Henrique Nakashima <hnakashima@chromium.org>
2017-06-27Fixed file matching regex in test_runner.Henrique Nakashima
It was not matching file names with characters like '-' and '&'. Also suppressed tests that are now matched by the regex and were not run. Change-Id: I70c0c8bebf4726748574a9e3ab17db64f1033a13 Reviewed-on: https://pdfium-review.googlesource.com/6835 Commit-Queue: Henrique Nakashima <hnakashima@chromium.org> Reviewed-by: dsinclair <dsinclair@chromium.org>
2017-06-27Fail pixel tests when they do not provide the expected image.Henrique Nakashima
Also showing stats on number of failures, successes, suppressions, etc. Change-Id: Id12faf504ea8d6b1c343d7a8b412d03dc787ca3a Reviewed-on: https://pdfium-review.googlesource.com/6834 Commit-Queue: Henrique Nakashima <hnakashima@chromium.org> Reviewed-by: dsinclair <dsinclair@chromium.org>
2017-02-01Remove DrMemorychromium/3000dan sinclair
The DrMemory bots have been removed, remove the config and infrastructure. BUG=chromium:655521 Change-Id: I065d717b11d615a6dc981b79a8caefd8783b105b Reviewed-on: https://pdfium-review.googlesource.com/2494 Reviewed-by: Tom Sepez <tsepez@chromium.org> Commit-Queue: dsinclair <dsinclair@chromium.org>
2017-01-31Fixup test harnessdan sinclair
When the results of the test runner were converted to return a tuple the following code which checked the results for true/false were not updated to extract the success value from the tuple. This caused the code to think that the results always passed even when they failed. This CL fixes the two tests which were broken (both just minor image result differences) which slipped in during this breakage. Change-Id: I01b56dd7b05013c2c12c83543746cf59b145e561 Reviewed-on: https://pdfium-review.googlesource.com/2456 Reviewed-by: Tom Sepez <tsepez@chromium.org> Commit-Queue: dsinclair <dsinclair@chromium.org>
2017-01-26Add flag to ignore images by their MD5 digestchromium/2994stephana
BUG= Review-Url: https://codereview.chromium.org/2649313005
2017-01-13Ensures that the gold output directory existschromium/2983chromium/2982chromium/2981stephana
This will land after this CL https://chromium-review.googlesource.com/c/424575/ as a way of making sure it works as intended. TBR=borenet BUG=skia:5973 Review-Url: https://codereview.chromium.org/2629943002
2017-01-02Gold support in PDFiumchromium/2970stephana
Extends the PDFium tests to collect images and meta data to be uploaded to Gold. This feature is triggered by adding the --gold_* flags. It extends pdfium_test to output the MD5 hash of the underlying pixel buffer for each page it renders. That output is then processed by test_runner.py to generate the gold meta data. This behavior is modeled after the 'dm' tool in skia. See https://skia.googlesource.com/skia/+/master/dm/DM.cpp#1090 This should not cause any change in the current behavior of the tests, it will be trigger once we update the buildbot recipe. BUG=skia:5973 Review-Url: https://codereview.chromium.org/2578893004
2016-05-17Combine corpus runner into test_runner.pydsinclair
This CL combines the corpus runner into the test_runner.py. This also enables the javascript and pixel runners to do a multi test runner. Review-Url: https://codereview.chromium.org/1952923002
2016-05-04Fix event sending in pdfium_test.tsepez
Passing bad args in pdfium_test -- all void* pointers look alike. Make output less verbose while we're here. Turn on event sending mode for corups tests. No effect until .evt files added to corpus. Review-Url: https://codereview.chromium.org/1946873004
2016-04-25Combine the javascript and pixel test runners.dsinclair
These runners are very similar in their implementation. Combine them into a single test runner. Corpus will, hopefully, be combined into this test runner in the future. Review URL: https://codereview.chromium.org/1906803005
2016-04-20Revert of Re-Reland Combined test runner. (patchset #4 id:80001 of ↵dsinclair
https://codereview.chromium.org/1888673003/ ) Reason for revert: Dr.Memory code is wrong..... Original issue's description: > Re-Reland Combined test runner. > > Original Commit Message: > > This CL revives the old CL to combine the test runners [1] which was reverted > due to failing font_size tests. I've deleted the font_size.pdf as it is not > needed and for testing. Pixel tests are either .in or .pdf files, not both. > > Original description: > This CL takes the three test runners (corpus, javascript, pixel) and combines > the code into a single test_runner file. Each of the individual runners still > exists and calls the test runner with their data directory. > > With this change, the pixel and javascript test will now run in parallel if > multiple processors are available. > > 1-https://codereview.chromium.org/1430623006/ > > BUG= > > Committed: https://pdfium.googlesource.com/pdfium/+/84b26ec6ba8193235ce0df817dec5c0f1da29841 TBR=ochang@chromium.org,zhaoqin@chromium.org,tsepez@chromium.org # Skipping CQ checks because original CL landed less than 1 days ago. NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true BUG= Review URL: https://codereview.chromium.org/1898353004
2016-04-20Re-Reland Combined test runner.dsinclair
Original Commit Message: This CL revives the old CL to combine the test runners [1] which was reverted due to failing font_size tests. I've deleted the font_size.pdf as it is not needed and for testing. Pixel tests are either .in or .pdf files, not both. Original description: This CL takes the three test runners (corpus, javascript, pixel) and combines the code into a single test_runner file. Each of the individual runners still exists and calls the test runner with their data directory. With this change, the pixel and javascript test will now run in parallel if multiple processors are available. 1-https://codereview.chromium.org/1430623006/ BUG= Review URL: https://codereview.chromium.org/1888673003
2016-04-13Revert of Reland Combined test runner. (patchset #4 id:60001 of ↵dsinclair
https://codereview.chromium.org/1886753002/ ) Reason for revert: Dr. Memory still broken: https://build.chromium.org/p/client.pdfium/builders/drm_win_xfa/builds/330/steps/pixel%20tests/logs/stdio Original issue's description: > Reland Combined test runner. > > This reverts commit 7a4374583efc0c41c826122aa26c1198c8d5cc37. > > Original Commit Message: > > This CL revives the old CL to combine the test runners [1] which was reverted > due to failing font_size tests. I've deleted the font_size.pdf as it is not > needed and for testing. Pixel tests are either .in or .pdf files, not both. > > Original description: > This CL takes the three test runners (corpus, javascript, pixel) and combines > the code into a single test_runner file. Each of the individual runners still > exists and calls the test runner with their data directory. > > With this change, the pixel and javascript test will now run in parallel if > multiple processors are available. > > 1-https://codereview.chromium.org/1430623006/ > > Committed: https://pdfium.googlesource.com/pdfium/+/9cec54ab0a5461b3075c585f8f233dbfd06c2cbd TBR=tsepez@chromium.org,ochang@chromium.org # Skipping CQ checks because original CL landed less than 1 days ago. NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true Review URL: https://codereview.chromium.org/1890703002
2016-04-13Reland Combined test runner.dsinclair
This reverts commit 7a4374583efc0c41c826122aa26c1198c8d5cc37. Original Commit Message: This CL revives the old CL to combine the test runners [1] which was reverted due to failing font_size tests. I've deleted the font_size.pdf as it is not needed and for testing. Pixel tests are either .in or .pdf files, not both. Original description: This CL takes the three test runners (corpus, javascript, pixel) and combines the code into a single test_runner file. Each of the individual runners still exists and calls the test runner with their data directory. With this change, the pixel and javascript test will now run in parallel if multiple processors are available. 1-https://codereview.chromium.org/1430623006/ Review URL: https://codereview.chromium.org/1886753002
2016-04-06Revert of Combined test runner. (patchset #2 id:20001 of ↵dsinclair
https://codereview.chromium.org/1850273003/ ) Reason for revert: Broke Dr. Memory Original issue's description: > Combined test runner. > > This CL revives the old CL to combine the test runners [1] which was reverted > due to failing font_size tests. I've deleted the font_size.pdf as it is not > needed and for testing. Pixel tests are either .in or .pdf files, not both. > > Original description: > This CL takes the three test runners (corpus, javascript, pixel) and combines > the code into a single test_runner file. Each of the individual runners still > exists and calls the test runner with their data directory. > > With this change, the pixel and javascript test will now run in parallel if > multiple processors are available. > > 1-https://codereview.chromium.org/1430623006/ > > Committed: https://pdfium.googlesource.com/pdfium/+/03e8f12cd3788038225d0376a164a9993f6cb720 TBR=tsepez@chromium.org,thestig@chromium.org # Skipping CQ checks because original CL landed less than 1 days ago. NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true Review URL: https://codereview.chromium.org/1867643002
2016-04-06Combined test runner.dsinclair
This CL revives the old CL to combine the test runners [1] which was reverted due to failing font_size tests. I've deleted the font_size.pdf as it is not needed and for testing. Pixel tests are either .in or .pdf files, not both. Original description: This CL takes the three test runners (corpus, javascript, pixel) and combines the code into a single test_runner file. Each of the individual runners still exists and calls the test runner with their data directory. With this change, the pixel and javascript test will now run in parallel if multiple processors are available. 1-https://codereview.chromium.org/1430623006/ Review URL: https://codereview.chromium.org/1850273003
2015-11-03Merge to XFA: Revert "One test runner to rule them all."Dan Sinclair
This reverts commit a1215ba51a235fb7abcb995f0e768ea0176d9275. Revert "Return result of the test runner." This reverts commit d1579c9b92b7f9c1d9e0fac1ecb8e3cb23875fca. Reverting the new test_runner code as there appears to be something string happening on the non-Linux boxes on XFA. It looks like it ran the font_size test twice, once with the .in and once with the .pdf. The .in was suppressed, the .pdf faild and wasn't suppressed. Rendering PDF file /Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/gen/pdfium/testing/pixel/font_size.pdf. Rendered 1 pages. Skipped 0 bad pages. Checking /Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/gen/pdfium/testing/pixel/font_size.pdf.0.png FAILURE: font_size.in; Command '['/Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/pdfium_diff', '/Volumes/data/b/build/slave/mac_xfa/build/pdfium/testing/resources/pixel/font_size_expected.pdf.0.png', '/Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/gen/pdfium/testing/pixel/font_size.pdf.0.png']' returned non-zero exit status 1 font_size.in is suppressed, found in SUPPRESSIONS_mac file Rendering PDF file /Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/gen/pdfium/testing/pixel/font_size.pdf. Rendered 1 pages. Skipped 0 bad pages. Checking /Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/gen/pdfium/testing/pixel/font_size.pdf.0.png FAILURE: font_size.pdf; Command '['/Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/pdfium_diff', '/Volumes/data/b/build/slave/mac_xfa/build/pdfium/testing/resources/pixel/font_size_expected.pdf.0.png', '/Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/gen/pdfium/testing/pixel/font_size.pdf.0.png']' returned non-zero exit status 1 TBR=thestig@chromium.org Review URL: https://codereview.chromium.org/1410743008 . (cherry picked from commit e067f7bab4317704de27917c1befa8f57a0e78d7) Review URL: https://codereview.chromium.org/1421133005 .
2015-11-03Merge to XFA: One test runner to rule them all.Dan Sinclair
This CL takes the three test runners (corpus, javascript, pixel) and combines the code into a single test_runner file. Each of the individual runners still exists and calls the test runner with their data directory. With this change, the pixel and javascript test will now run in parallel if multiple processors are available. TBR=thestig@chromium.org Review URL: https://codereview.chromium.org/1430623006 . (cherry picked from commit a1215ba51a235fb7abcb995f0e768ea0176d9275) Review URL: https://codereview.chromium.org/1411553010 .
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-01 04:11:53 +0000