[Up for Grabs] Switch out PhantomJS for Chromy or Puppeteer

[matthew-dean]

PhantomJS has long created issues for continuous integration tests, where it will timeout or not download, and more recently, fail to execute browser tests under Node 4. If someone had the time to get replace it with a Chromium-based tester, a PR would be welcome.

[TristanNovo]

I think I will be able to make this change, I'm going to take a swing at it.

[matthew-dean]

Cool, good luck!

[TristanNovo]

Ok so after familiarizing myself with the project a bit it looks like the only change that should be required to make this change is to bump the grunt-contrib-jasmine version to ^2.0.3 But doing that causes the tests to hang after the first suite completes.

Here's the truncated output of running grunt browsertest --verbose

Running "jasmine:main" (jasmine) task
Verifying property jasmine.main exists in config...OK
Files: test/less/calc.less, test/less/charsets.less, test/less/colors.less, test/less/comments.less, test/less/comments2.less, test/less/css-3.less, test/less/css-escapes.less, test/less/css-grid.less, test/less/css-guards.less, test/less/detached-rulesets.less, test/less/directives-bubling.less, test/less/extend-chaining.less, test/less/extend-clearfix.less, test/less/extend-exact.less, test/less/extend-media.less, test/less/extend-nest.less, test/less/extend-selector.less, test/less/extend.less, test/less/extract-and-length.less, test/less/functions-each.less, test/less/functions.less, test/less/ie-filters.less, test/less/import-inline.less, test/less/import-interpolation.less, test/less/import-once.less, test/less/import-reference-issues.less, test/less/import-reference.less, test/less/import.less, test/less/lazy-eval.less, test/less/media.less, test/less/merge.less, test/less/mixins-closure.less, test/less/mixins-guards-default-func.less, test/less/mixins-guards.less, test/less/mixins-important.less, test/less/mixins-interpolated.less, test/less/mixins-named-args.less, test/less/mixins-nested.less, test/less/mixins-pattern.less, test/less/mixins.less, test/less/no-output.less, test/less/operations.less, test/less/parse-interpolation.less, test/less/permissive-parse.less, test/less/plugin.less, test/less/property-accessors.less, test/less/property-name-interp.less, test/less/rulesets.less, test/less/scope.less, test/less/selectors.less, test/less/strings.less, test/less/variables-in-at-rules.less, test/less/variables.less, test/less/whitespace.less
Options: version="latest", timeout=10000, styles=[], specs="test/browser/runner-main-spec.js", helpers="test/browser/runner-main-options.js", vendor=["test/browser/vendor/promise.js","test/browser/jasmine-jsreporter.js","test/browser/common.js","test/browser/less.js"], polyfills=[], customBootFile=null, tempDir=".grunt/grunt-contrib-jasmine", outfile="tmp/browser/test-runner-main.html", host="http://localhost:8081/", template="test/browser/test-runner-template.tmpl", templateOptions={}, junit={}, ignoreEmpty=false, display="full", summary=false, keepRunner
Reading C:\github\less.js\node_modules\grunt-contrib-jasmine\tasks\jasmine\reporters\PuppeteerReporter.js...OK
Writing .grunt\grunt-contrib-jasmine\reporter.js...OK
Reading C:\github\less.js\node_modules\grunt-contrib-jasmine\.jasmine\latest\lib\jasmine-core\jasmine.css...OK
Writing .grunt\grunt-contrib-jasmine\jasmine.css...OK
Reading C:\github\less.js\node_modules\grunt-contrib-jasmine\.jasmine\latest\lib\jasmine-core\jasmine.js...OK
Writing .grunt\grunt-contrib-jasmine\jasmine.js...OK
Reading C:\github\less.js\node_modules\grunt-contrib-jasmine\.jasmine\latest\lib\jasmine-core\jasmine-html.js...OK
Writing .grunt\grunt-contrib-jasmine\jasmine-html.js...OK
Reading C:\github\less.js\node_modules\grunt-contrib-jasmine\.jasmine\latest\lib\jasmine-core\json2.js...OK
Writing .grunt\grunt-contrib-jasmine\json2.js...OK
Reading C:\github\less.js\node_modules\grunt-contrib-jasmine\.jasmine\latest\lib\jasmine-core\boot.js...OK
Writing .grunt\grunt-contrib-jasmine\boot.js...OK
Reading C:\github\less.js\node_modules\grunt-contrib-jasmine\.jasmine\latest\images\jasmine_favicon.png...OK
Writing .grunt\grunt-contrib-jasmine\jasmine_favicon.png...OK
Reading test/browser/test-runner-template.tmpl...OK
Processing source...OK
Writing tmp\browser\test-runner-main.html...OK

Testing specs with Jasmine/latest via HeadlessChrome/72.0.3582.0
Jasmine Runner Starting...
jasmine.suiteStarted
 less.js main tests
jasmine.specStarted
   - original-less:test-less-merge should match the expected output...jasmine.specDone
   √ original-less:test-less-merge should match the expected output
...
jasmine.specStarted
   - original-less:test-less-detached-rulesets should match the expected output...jasmine.specDone
   √ original-less:test-less-detached-rulesets should match the expected output
jasmine.suiteDone

This is as far as it ever gets. I've tried setting the timeout smaller to see if it will timeout and it never does, It will only exit if it is forced close with Control-C.

I'm not sure what to do next exactly, am I missing something obvious?

I'm working with 3.9.0 on Windows 10

Thanks in advance

[matthew-dean]

@TristanNovo I ran into the same issue (tests hanging). My guess is that there's some change in reporting. That is, Jasmine is no longer getting a message back from the browser saying tests have completed, probably because the API for reporting has changed. It's really a matter of looking at the template and how Jasmine says that test results should be reported back to the main test process.

[matthew-dean]

@TristanNovo The tests are using a really old form of JS reporting, AFAIK. Really the whole Grunt / Jasmine / PhantomJS stack is rather archaic. But there should be a relatively straightforward process documented somewhere for headless testing of JS within an HTML document. There's also no specific attachment to Jasmine if there's a better-supported test runner.

[ossdev07]

Hi @TristanNovo @matthew-dean

I was trying to replace Phantomjs with puppeteer by updating the grunt-contrib-jasmine version to 2.1.0 and removing the phantomjs dependency from pacakge.json file .

The Jasmine main test cases are not getting run with the latest release, the ChromeHeadless browser gets launch but the test cases are not getting executed.

ossdev@PS-LAB:~/qwert/less.js$ grunt browsertest --verbose
Initializing
Command-line options: --verbose

Reading "Gruntfile.js" Gruntfile...OK

Registering Gruntfile tasks.
Initializing config...OK

Registering "grunt-saucelabs" local Npm module tasks.
Reading /home/qwert/less.js/node_modules/grunt-saucelabs/package.json...OK
Parsing /home/qwert/less.js/node_modules/grunt-saucelabs/package.json...OK
Loading "saucelabs.js" tasks...OK
+ saucelabs-custom, saucelabs-jasmine, saucelabs-mocha, saucelabs-qunit, saucelabs-yui
Loading "Gruntfile.js" tasks...OK
+ benchmark, browsertest, browsertest-lessjs, browsertest-server, default, dist, force, quicktest, sauce, sauce-after-setup, shell-options, shell-plugin, sourcemap-test, test

Running tasks: browsertest

Running "browsertest" task

Running "browsertest-lessjs" task

Loading "grunt-shell" plugin

Registering "/home/qwert/less.js/node_modules/grunt-shell/tasks" tasks.
Loading "shell.js" tasks...OK
+ shell

Running "shell:testbrowser" (shell) task
Verifying property shell.testbrowser exists in config...OK
File: [no files]
Options: stdout, stderr, stdin, failOnError, stdinRawMode=false, preferLocal=false, execOptions={"maxBuffer":null}
Command: node build/rollup.js --browser --out=./test/browser/less.min.js
Writing ./test/browser/less.min.js...

Loading "grunt-contrib-connect" plugin

Registering "/home/qwert/less.js/node_modules/grunt-contrib-connect/tasks" tasks.
Loading "connect.js" tasks...OK
+ connect

Running "connect" task
(node:9403) ExperimentalWarning: The http2 module is an experimental API.

Running "connect:server" (connect) task
Verifying property connect.server exists in config...OK
File: [no files]
Options: protocol="http", port=8081, hostname="0.0.0.0", base=".", directory=null, keepalive=false, debug=false, livereload=false, open=false, useAvailablePort=false, onCreateServer=null, middleware=null
Reading /home/qwert/less.js/node_modules/grunt-contrib-connect/tasks/certs/server.key...OK
Reading /home/qwert/less.js/node_modules/grunt-contrib-connect/tasks/certs/server.crt...OK
Reading /home/qwert/less.js/node_modules/grunt-contrib-connect/tasks/certs/ca.crt...OK
Started connect web server on http://localhost:8081

Loading "grunt-contrib-jasmine" plugin

Registering "/home/qwert/less.js/node_modules/grunt-contrib-jasmine/tasks" tasks.
Loading "jasmine.js" tasks...OK
+ jasmine

Running "jasmine" task

Running "jasmine:main" (jasmine) task
Verifying property jasmine.main exists in config...OK
Files: test/less/calc.less, test/less/charsets.less, test/less/colors.less, test/less/comments.less, test/less/comments2.less, test/less/css-3.less, test/less/css-escapes.less, test/less/css-grid.less, test/less/css-guards.less, test/less/detached-rulesets.less, test/less/directives-bubling.less, test/less/extend-chaining.less, test/less/extend-clearfix.less, test/less/extend-exact.less, test/less/extend-media.less, test/less/extend-nest.less, test/less/extend-selector.less, test/less/extend.less, test/less/extract-and-length.less, test/less/functions-each.less, test/less/functions.less, test/less/ie-filters.less, test/less/import-inline.less, test/less/import-interpolation.less, test/less/import-once.less, test/less/import-reference-issues.less, test/less/import-reference.less, test/less/import.less, test/less/lazy-eval.less, test/less/media.less, test/less/merge.less, test/less/mixins-closure.less, test/less/mixins-guards-default-func.less, test/less/mixins-guards.less, test/less/mixins-important.less, test/less/mixins-interpolated.less, test/less/mixins-named-args.less, test/less/mixins-nested.less, test/less/mixins-pattern.less, test/less/mixins.less, test/less/no-output.less, test/less/operations.less, test/less/parse-interpolation.less, test/less/permissive-parse.less, test/less/plugin.less, test/less/property-accessors.less, test/less/property-name-interp.less, test/less/rulesets.less, test/less/scope.less, test/less/selectors.less, test/less/strings.less, test/less/variables-in-at-rules.less, test/less/variables.less, test/less/whitespace.less
Options: version="latest", timeout=10000, styles=[], specs="test/browser/runner-main-spec.js", helpers="test/browser/runner-main-options.js", vendor=["test/browser/vendor/jasmine-jsreporter.js","test/browser/common.js","test/browser/less.min.js"], polyfills=["./node_modules/phantomjs-polyfill-object-assign/object-assign-polyfill.js","test/browser/vendor/promise.js"], customBootFile=null, tempDir=".grunt/grunt-contrib-jasmine", outfile="tmp/browser/test-runner-main.html", host="http://localhost:8081/", template="test/browser/test-runner-template.tmpl", templateOptions={}, junit={}, ignoreEmpty=false, display="full", summary=false, keepRunner
Reading /home/qwert/less.js/node_modules/grunt-contrib-jasmine/tasks/jasmine/reporters/PuppeteerReporter.js...OK
Writing .grunt/grunt-contrib-jasmine/reporter.js...OK
Reading /home/qwert/less.js/node_modules/grunt-contrib-jasmine/.jasmine/latest/lib/jasmine-core/jasmine.css...OK
Writing .grunt/grunt-contrib-jasmine/jasmine.css...OK
Reading /home/qwert/less.js/node_modules/grunt-contrib-jasmine/.jasmine/latest/lib/jasmine-core/jasmine.js...OK
Writing .grunt/grunt-contrib-jasmine/jasmine.js...OK
Reading /home/qwert/less.js/node_modules/grunt-contrib-jasmine/.jasmine/latest/lib/jasmine-core/jasmine-html.js...OK
Writing .grunt/grunt-contrib-jasmine/jasmine-html.js...OK
Reading /hom/qwert/less.js/node_modules/grunt-contrib-jasmine/.jasmine/latest/lib/jasmine-core/json2.js...OK
Writing .grunt/grunt-contrib-jasmine/json2.js...OK
Reading /home/qwert/less.js/node_modules/grunt-contrib-jasmine/.jasmine/latest/lib/jasmine-core/boot.js...OK
Writing .grunt/grunt-contrib-jasmine/boot.js...OK
Reading /home/qwert/less.js/node_modules/grunt-contrib-jasmine/.jasmine/latest/images/jasmine_favicon.png...OK
Writing .grunt/grunt-contrib-jasmine/jasmine_favicon.png...OK
Reading test/browser/test-runner-template.tmpl...OK
Processing source...OK
Writing tmp/browser/test-runner-main.html...OK

Testing specs with Jasmine/latest via HeadlessChrome/77.0.3844.0

log: Refused to apply style from 'http://localhost:8081/absolute/something.css' because its MIME type ('text/html') is not a supported stylesheet MIME type, and strict MIME checking is enabled.

log: Failed to load resource: the server responded with a status of 404 ()

log: Failed to load resource: the server responded with a status of 404 (Not Found)

log: start spec

log: Refused to apply style from 'http://localhost:8081/absolute/something.css' because its MIME type ('text/html') is not a supported stylesheet MIME type, and strict MIME checking is enabled.

log: Failed to load resource: the server responded with a status of 404 ()
^C

Execution Time (2019-09-20 12:49:15 UTC+5:30)
loading tasks                    131ms  0%
loading grunt-shell                4ms  0%
shell:testbrowser                 5.9s  ▇▇▇ 2%
loading grunt-contrib-connect     29ms  0%
connect:server                     9ms  0%
loading grunt-contrib-jasmine    123ms  0%
jasmine:main                   5m 9.7s  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 98%
Total 5m 15.9s

I have taken a look into the Gruntfile.js file, in it phantomjs-polyfill-object-assign is being used, but by updating grunt-contrib-jasmine it is using puppeteer now, but in its configuration, it is looking for phantomjs-polyfill, it could be one of the issues in starting the test cases.

            options: {
                keepRunner: true,
                host: "http://localhost:8081/",
                polyfills: [
                    "./node_modules/phantomjs-polyfill-object-assign/object-assign-polyfill.js",
                    "test/browser/vendor/promise.js"
                ],
                vendor: [
                    "test/browser/vendor/jasmine-jsreporter.js",
                    "test/browser/common.js",
                    "test/browser/less.min.js"
                ],
                template: "test/browser/test-runner-template.tmpl"
            },
            main: {
                // src is used to build list of less files to compile
                src: [
                    "test/less/*.less",
                    "!test/less/plugin-preeval.less", // uses ES6 syntax
                    // Don't test NPM import, obviously
                    "!test/less/plugin-module.less",
                    "!test/less/import-module.less",
                    "!test/less/javascript.less",
                    "!test/less/urls.less",
                    "!test/less/empty.less"
                ],
                options: {
                    helpers: "test/browser/runner-main-options.js",
                    specs: "test/browser/runner-main-spec.js",
                    outfile: "tmp/browser/test-runner-main.html"
                }
            },

Please share your thoughts on this, if you have any pointer could you please suggest me something.

[ossdev07]

@matthew-dean I am working on replacing PhantomJS with Puppeteer and after analyzing the package with latest and older versions, I have few queries if you could help me with that it will be great:

• Currently, the latest less.js with the latest grunt-contrib-jasmine is giving the following output, i.e the tests are not starting and the console hangs after this log message.

Running "jasmine:main" (jasmine) taskTesting specs with Jasmine/latest via HeadlessChrome/78.0.3882.0log: Refused to apply style from 'http://localhost:8081/absolute/something.css' because its MIME type ('text/html') is not a supported stylesheet MIME type, and strict MIME checking is enabled.log: Failed to load resource: the server responded with a status of 404 ()log: Failed to load resource: the server responded with a status of 404 (Not Found)log: start speclog: Refused to apply style from 'http://localhost:8081/absolute/something.css' because its MIME type ('text/html') is not a supported stylesheet MIME type, and strict MIME checking is enabled.log: Failed to load resource: the server responded with a status of 404 () ^C

• But earlier ( before some commits in less.js ) the package was hanging at a position after successfully running the test cases, below is the output I got.

Running "jasmine:main" (jasmine) taskTesting specs with Jasmine/latest via HeadlessChrome/78.0.3882.0log: Refused to apply style from 'http://localhost:8081/absolute/something.css' because its MIME type ('text/html') is not a supported stylesheet MIME type, and strict MIME checking is enabled.log: Failed to load resource: the server responded with a status of 404 ()log: Failed to load resource: the server responded with a status of 404 (Not Found)log: Refused to apply style from 'http://localhost:8081/absolute/something.css' because its MIME type ('text/html') is not a supported stylesheet MIME type, and strict MIME checking is enabled.log: Failed to load resource: the server responded with a status of 404 ()log: start spec
 less.js main tests
   ✓ original-less:test-less-extend-chaining should match the expected output
   ✓ original-less:test-less-mixins-important should match the expected output
   ✓ original-less:test-less-import-interpolation should match the expected output
   ✓ original-less:test-less-plugin should match the expected output
   ✓ original-less:test-less-extend-clearfix should match the expected output
   ✓ original-less:test-less-comments2 should match the expected output
   ✓ original-less:test-less-property-accessors should match the expected output
   ✓ original-less:test-less-calc should match the expected output
   ✓ original-less:test-less-import-inline should match the expected output
   ✓ original-less:test-less-mixins-interpolated should match the expected output
   ✓ original-less:test-less-extend-exact should match the expected output
   ✓ original-less:test-less-comments should match the expected output
   ✓ original-less:test-less-media should match the expected output
   ✓ original-less:test-less-merge should match the expected output
   ✓ original-less:test-less-mixins-named-args should match the expected output
   ✓ original-less:test-less-no-output should match the expected output
   ✓ original-less:test-less-lazy-eval should match the expected output
   ✓ original-less:test-less-mixins-closure should match the expected output
   ✓ original-less:test-less-mixins-nested should match the expected output
   ✓ original-less:test-less-detached-rulesets should match the expected output
   ✓ original-less:test-less-extend-selector should match the expected output
   ✓ original-less:test-less-operations should match the expected output
   ✓ the global environment
   ✓ original-less:test-less-extend should match the expected output
   ✓ original-less:test-less-mixins-guards-default-func should match the expected output
   ✓ original-less:test-less-mixins-guards should match the expected output
   ✓ original-less:test-less-import-reference-issues should match the expected output
   ✓ original-less:test-less-colors should match the expected output
   ✓ original-less:test-less-css-escapes should match the expected output
   ✓ original-less:test-less-import-once should match the expected output
   ✓ original-less:test-less-directives-bubling should match the expected output
   ✓ original-less:test-less-css-3 should match the expected output
   ✓ original-less:test-less-charsets should match the expected output
   ✓ original-less:test-less-strings should match the expected output
   ✓ original-less:test-less-property-name-interp should match the expected output
   ✓ original-less:test-less-extend-media should match the expected output
   ✓ original-less:test-less-ie-filters should match the expected output
   ✓ original-less:test-less-extend-nest should match the expected output
   ✓ original-less:test-less-rulesets should match the expected output
   ✓ original-less:test-less-variables should match the expected output
   ✓ original-less:test-less-scope should match the expected output
   ✓ original-less:test-less-mixins-pattern should match the expected output
   ✓ original-less:test-less-functions should match the expected output
   ✓ original-less:test-less-variables-in-at-rules should match the expected output
   ✓ original-less:test-less-css-grid should match the expected output
   ✓ original-less:test-less-import should match the expected output
   ✓ original-less:test-less-selectors should match the expected output
   ✓ original-less:test-less-mixins should match the expected output
   ✓ original-less:test-less-extract-and-length should match the expected output
   ✓ original-less:test-less-parse-interpolation should match the expected output
   ✓ original-less:test-less-import-reference should match the expected output
   ✓ original-less:test-less-whitespace should match the expected output
   ✓ original-less:test-less-functions-each should match the expected output
   ✓ original-less:test-less-css-guards should match the expected output
   ✓ original-less:test-less-permissive-parse should match the expected output
^C

• hence to conclude after the commit cae5021 the tests are now even not running with the puppeteer.

It will be great if you could share your thoughts on this.