summaryrefslogtreecommitdiff
path: root/testing/tools/run_pixel_tests.py
AgeCommit message (Collapse)Author
2017-06-27Fail pixel tests when they do not provide the expected image.Henrique Nakashima
Also showing stats on number of failures, successes, suppressions, etc. Change-Id: Id12faf504ea8d6b1c343d7a8b412d03dc787ca3a Reviewed-on: https://pdfium-review.googlesource.com/6834 Commit-Queue: Henrique Nakashima <hnakashima@chromium.org> Reviewed-by: dsinclair <dsinclair@chromium.org>
2016-04-25Combine the javascript and pixel test runners.dsinclair
These runners are very similar in their implementation. Combine them into a single test runner. Corpus will, hopefully, be combined into this test runner in the future. Review URL: https://codereview.chromium.org/1906803005
2016-04-20Revert of Re-Reland Combined test runner. (patchset #4 id:80001 of ↵dsinclair
https://codereview.chromium.org/1888673003/ ) Reason for revert: Dr.Memory code is wrong..... Original issue's description: > Re-Reland Combined test runner. > > Original Commit Message: > > This CL revives the old CL to combine the test runners [1] which was reverted > due to failing font_size tests. I've deleted the font_size.pdf as it is not > needed and for testing. Pixel tests are either .in or .pdf files, not both. > > Original description: > This CL takes the three test runners (corpus, javascript, pixel) and combines > the code into a single test_runner file. Each of the individual runners still > exists and calls the test runner with their data directory. > > With this change, the pixel and javascript test will now run in parallel if > multiple processors are available. > > 1-https://codereview.chromium.org/1430623006/ > > BUG= > > Committed: https://pdfium.googlesource.com/pdfium/+/84b26ec6ba8193235ce0df817dec5c0f1da29841 TBR=ochang@chromium.org,zhaoqin@chromium.org,tsepez@chromium.org # Skipping CQ checks because original CL landed less than 1 days ago. NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true BUG= Review URL: https://codereview.chromium.org/1898353004
2016-04-20Re-Reland Combined test runner.dsinclair
Original Commit Message: This CL revives the old CL to combine the test runners [1] which was reverted due to failing font_size tests. I've deleted the font_size.pdf as it is not needed and for testing. Pixel tests are either .in or .pdf files, not both. Original description: This CL takes the three test runners (corpus, javascript, pixel) and combines the code into a single test_runner file. Each of the individual runners still exists and calls the test runner with their data directory. With this change, the pixel and javascript test will now run in parallel if multiple processors are available. 1-https://codereview.chromium.org/1430623006/ BUG= Review URL: https://codereview.chromium.org/1888673003
2016-04-18Exclude XFA-only corpus from non-xfa and roll corpus.tsepez
Test still need modification to process input events. For now, we just suppress anything that diffs. Review URL: https://codereview.chromium.org/1894083003
2016-04-13Revert of Reland Combined test runner. (patchset #4 id:60001 of ↵dsinclair
https://codereview.chromium.org/1886753002/ ) Reason for revert: Dr. Memory still broken: https://build.chromium.org/p/client.pdfium/builders/drm_win_xfa/builds/330/steps/pixel%20tests/logs/stdio Original issue's description: > Reland Combined test runner. > > This reverts commit 7a4374583efc0c41c826122aa26c1198c8d5cc37. > > Original Commit Message: > > This CL revives the old CL to combine the test runners [1] which was reverted > due to failing font_size tests. I've deleted the font_size.pdf as it is not > needed and for testing. Pixel tests are either .in or .pdf files, not both. > > Original description: > This CL takes the three test runners (corpus, javascript, pixel) and combines > the code into a single test_runner file. Each of the individual runners still > exists and calls the test runner with their data directory. > > With this change, the pixel and javascript test will now run in parallel if > multiple processors are available. > > 1-https://codereview.chromium.org/1430623006/ > > Committed: https://pdfium.googlesource.com/pdfium/+/9cec54ab0a5461b3075c585f8f233dbfd06c2cbd TBR=tsepez@chromium.org,ochang@chromium.org # Skipping CQ checks because original CL landed less than 1 days ago. NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true Review URL: https://codereview.chromium.org/1890703002
2016-04-13Reland Combined test runner.dsinclair
This reverts commit 7a4374583efc0c41c826122aa26c1198c8d5cc37. Original Commit Message: This CL revives the old CL to combine the test runners [1] which was reverted due to failing font_size tests. I've deleted the font_size.pdf as it is not needed and for testing. Pixel tests are either .in or .pdf files, not both. Original description: This CL takes the three test runners (corpus, javascript, pixel) and combines the code into a single test_runner file. Each of the individual runners still exists and calls the test runner with their data directory. With this change, the pixel and javascript test will now run in parallel if multiple processors are available. 1-https://codereview.chromium.org/1430623006/ Review URL: https://codereview.chromium.org/1886753002
2016-04-13Add dummy -j flags to pixel and javascript test runnersdsinclair
This is an intermediate step to creating a unified test runner. We need to update Dr Memory to pass the -j 1 flag to the pixel and javascript tests. In order to do so, we need to have the flag available. The flag will be enabled for pixel and javascript tests when [1] lands. 1- https://codereview.chromium.org/1886753002/ Review URL: https://codereview.chromium.org/1882143005
2016-04-06Revert of Combined test runner. (patchset #2 id:20001 of ↵dsinclair
https://codereview.chromium.org/1850273003/ ) Reason for revert: Broke Dr. Memory Original issue's description: > Combined test runner. > > This CL revives the old CL to combine the test runners [1] which was reverted > due to failing font_size tests. I've deleted the font_size.pdf as it is not > needed and for testing. Pixel tests are either .in or .pdf files, not both. > > Original description: > This CL takes the three test runners (corpus, javascript, pixel) and combines > the code into a single test_runner file. Each of the individual runners still > exists and calls the test runner with their data directory. > > With this change, the pixel and javascript test will now run in parallel if > multiple processors are available. > > 1-https://codereview.chromium.org/1430623006/ > > Committed: https://pdfium.googlesource.com/pdfium/+/03e8f12cd3788038225d0376a164a9993f6cb720 TBR=tsepez@chromium.org,thestig@chromium.org # Skipping CQ checks because original CL landed less than 1 days ago. NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true Review URL: https://codereview.chromium.org/1867643002
2016-04-06Combined test runner.dsinclair
This CL revives the old CL to combine the test runners [1] which was reverted due to failing font_size tests. I've deleted the font_size.pdf as it is not needed and for testing. Pixel tests are either .in or .pdf files, not both. Original description: This CL takes the three test runners (corpus, javascript, pixel) and combines the code into a single test_runner file. Each of the individual runners still exists and calls the test runner with their data directory. With this change, the pixel and javascript test will now run in parallel if multiple processors are available. 1-https://codereview.chromium.org/1430623006/ Review URL: https://codereview.chromium.org/1850273003
2016-01-19Merge to XFA: Hook up show-config to testing scripts.Tom Sepez
Review URL: https://codereview.chromium.org/1607923004 . (cherry picked from commit d40e189ba1ab515ef9b3df7928e215be150df336) TBR=ochang@chromium.org Review URL: https://codereview.chromium.org/1607113002 .
2015-11-23Merge to XFA: Enable Dr. Memory to run javascript, pixel, and corpus testsQin Zhao
- add DrMemoryWrapper in common.py to adjust Dr. Memory wrapper - add --wrapper option to run_*_tests.py for Dr. Mempry wrapper - update run_*_tests.py to handle Dr. Memory wrapper - add TestPDFiumTest in pdfium_tests.py to support run_*_tests.py - remove ValgrindTool in valgrind_tests.py R=thestig@chromium.org BUG=pdfium:238 Review URL: https://codereview.chromium.org/1464453003 . (cherry picked from commit 36476923ae5eb8e9283e605f3c85ee8811c86014) Review URL: https://codereview.chromium.org/1465333002 .
2015-11-03Merge to XFA: Revert "One test runner to rule them all."Dan Sinclair
This reverts commit a1215ba51a235fb7abcb995f0e768ea0176d9275. Revert "Return result of the test runner." This reverts commit d1579c9b92b7f9c1d9e0fac1ecb8e3cb23875fca. Reverting the new test_runner code as there appears to be something string happening on the non-Linux boxes on XFA. It looks like it ran the font_size test twice, once with the .in and once with the .pdf. The .in was suppressed, the .pdf faild and wasn't suppressed. Rendering PDF file /Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/gen/pdfium/testing/pixel/font_size.pdf. Rendered 1 pages. Skipped 0 bad pages. Checking /Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/gen/pdfium/testing/pixel/font_size.pdf.0.png FAILURE: font_size.in; Command '['/Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/pdfium_diff', '/Volumes/data/b/build/slave/mac_xfa/build/pdfium/testing/resources/pixel/font_size_expected.pdf.0.png', '/Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/gen/pdfium/testing/pixel/font_size.pdf.0.png']' returned non-zero exit status 1 font_size.in is suppressed, found in SUPPRESSIONS_mac file Rendering PDF file /Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/gen/pdfium/testing/pixel/font_size.pdf. Rendered 1 pages. Skipped 0 bad pages. Checking /Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/gen/pdfium/testing/pixel/font_size.pdf.0.png FAILURE: font_size.pdf; Command '['/Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/pdfium_diff', '/Volumes/data/b/build/slave/mac_xfa/build/pdfium/testing/resources/pixel/font_size_expected.pdf.0.png', '/Volumes/data/b/build/slave/mac_xfa/build/pdfium/out/Debug/gen/pdfium/testing/pixel/font_size.pdf.0.png']' returned non-zero exit status 1 TBR=thestig@chromium.org Review URL: https://codereview.chromium.org/1410743008 . (cherry picked from commit e067f7bab4317704de27917c1befa8f57a0e78d7) Review URL: https://codereview.chromium.org/1421133005 .
2015-11-03Merge to XFA: Return result of the test runner.Dan Sinclair
We need to actually return the result of the test runner.... TBR=thestig@chromium.org Review URL: https://codereview.chromium.org/1413863011 . (cherry picked from commit d1579c9b92b7f9c1d9e0fac1ecb8e3cb23875fca) Review URL: https://codereview.chromium.org/1419953009 .
2015-11-03Merge to XFA: One test runner to rule them all.Dan Sinclair
This CL takes the three test runners (corpus, javascript, pixel) and combines the code into a single test_runner file. Each of the individual runners still exists and calls the test runner with their data directory. With this change, the pixel and javascript test will now run in parallel if multiple processors are available. TBR=thestig@chromium.org Review URL: https://codereview.chromium.org/1430623006 . (cherry picked from commit a1215ba51a235fb7abcb995f0e768ea0176d9275) Review URL: https://codereview.chromium.org/1411553010 .
2015-10-28Merge to XFA: Allow running individual tests.Dan Sinclair
This CL adds the ability to run a given test from the corpus, javascript and pixel test runners. The filename to provide is relative to the testing directory in question. Because the directories for javascript and pixel are flat you just provide the filename (it will rewrite the .pdf to .in if .pdf is provided). For corpus tests you have to provide the path from the corpus directory. Development/pdfium/pdfium % ./testing/tools/run_javascript_tests.py apply.pdf Rendering PDF file /Development/pdfium/pdfium/out/Debug/gen/pdfium/testing/javascript/apply.pdf. Non-linearized path... Rendered 1 pages. Skipped 0 bad pages. Development/pdfium/pdfium % ./testing/tools/run_pixel_tests.py bug_524043_1.pdf Rendering PDF file /Development/pdfium/pdfium/out/Debug/gen/pdfium/testing/pixel/bug_524043_1.pdf. Linearized path... Rendered 1 pages. Skipped 0 bad pages. Checking /Development/pdfium/pdfium/out/Debug/gen/pdfium/testing/pixel/bug_524043_1.pdf.0.png diff: 0.00% passed Development/pdfium/pdfium % ./testing/tools/run_corpus_tests.py third_party/tcpdf/example_065.pdf Rendering PDF file /Development/pdfium/pdfium/out/Debug/gen/pdfium/testing/corpus/example_065.pdf. Non-linearized path... Rendered 1 pages. Skipped 0 bad pages. Checking /Development/pdfium/pdfium/out/Debug/gen/pdfium/testing/corpus/example_065.pdf.0.png diff: 0.14% failed FAILURE: example_065.pdf; Command '['/Development/pdfium/pdfium/out/Debug/pdfium_diff', '/Development/pdfium/pdfium/testing/corpus/third_party/tcpdf/example_065_expected.pdf.0.png', '/Development/pdfium/pdfium/out/Debug/gen/pdfium/testing/corpus/example_065.pdf.0.png']' returned non-zero exit status 1 Summary of Failures: /Development/pdfium/pdfium/testing/corpus/third_party/tcpdf/example_065.pdf TBR=thestig@chromium.org Review URL: https://codereview.chromium.org/1407913005 . (cherry picked from commit da6254ff78553773b7bb637ee1e3f5111c083c74) Review URL: https://codereview.chromium.org/1423523005 .
2015-09-09Merge to XFA: Remove existing generated images before running a test.Lei Zhang
TBR=tsepez@chromium.org Review URL: https://codereview.chromium.org/1314443007 . (cherry picked from commit 8ead9036aab18d45498e310866210dfff20da188) Review URL: https://codereview.chromium.org/1331643004 .
2015-07-16Merge to XFA: Sort unexpected test results.Lei Zhang
No typos here. R=tsepez@chromium.org Review URL: https://codereview.chromium.org/1232203003 . (cherry picked from commit d3611f9db615be393016a8ab5340ccea7f13832b) Review URL: https://codereview.chromium.org/1244503003 .
2015-06-01Get test running scripts to detect and report common error, XFA.Bruce Dawson
testing/tools/run_corpus_tests.py assumes a debug build and will fail cryptically if only a release build is available. Arguably there shouldn't be a default because having one could lead to accidentally running a stale version, but that is probably too much of a change. TBR=tsepez@chromium.org Review URL: https://codereview.chromium.org/1150823003 Review URL: https://codereview.chromium.org/1158883005
2015-04-09Merge to XFA: testing utility combo patchTom Sepez
This pulls in the following CLs from master: Review URL: https://codereview.chromium.org/1072613003 Review URL: https://codereview.chromium.org/1058463004 Review URL: https://codereview.chromium.org/1057983003 Review URL: https://codereview.chromium.org/1036073002 Review URL: https://codereview.chromium.org/1031203003 Review URL: https://codereview.chromium.org/1029193002 Review URL: https://codereview.chromium.org/1016613004 Review URL: https://codereview.chromium.org/1026903002 TBR=thestig@chromium.org Review URL: https://codereview.chromium.org/1062163003
2015-03-17Merge to XFA: Return OS status code from python test driver scripts.Tom Sepez
Original Review URL: https://codereview.chromium.org/1014083003 TBR=thestig@chromium.org Review URL: https://codereview.chromium.org/1011243002
2015-02-24Merge to XFA: Convert pdfium expected test results to PNG format.Tom Sepez
Original Review URL: https://codereview.chromium.org/953723004 TBR=thestig@chromium.org Review URL: https://codereview.chromium.org/953073002
2015-02-13Merge to XFA: Run javascript/pixel tests via automated script.Tom Sepez
This pulls in: 7435e8e Run pixel tests via automated script. 83c87e5 run_javascript_tests.py: Be more flexible about directory layout. 5898509 Test top-level Document JS properties. 9f93baf Create run_javascript_tests.py TBR=thestig@chromium.org Review URL: https://codereview.chromium.org/923293002
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-01 06:50:18 +0000