[CLOSED] Swap out the user prefs when running tests.

[core-ai-bot]

Issue by dangoor
Tuesday Mar 11, 2014 at 19:49 GMT
Originally opened as adobe/brackets#7163

---

Fix for #7054. Note that the exception that appears in the console when running the tests is separate: #7161

This should prevent tests that run in the specrunner window from clobbering
user-level prefs.

cc@redmunds

---

dangoor included the following code: https://github.com/adobe/brackets/pull/7163/commits

[core-ai-bot]

Comment by redmunds
Tuesday Mar 11, 2014 at 23:40 GMT

---

I was able to hit the problem again in this branch on Win7. Here's what I did:

1. Right before opening Spec Runner window:
   • change projects
   • change state of open/closed folders in project tree
   • remove file from working set
   • double-click on a file to open and add it to working set
2. Debug > Run tests, All/All
3. Tests caused Brackets to hang (see note below), so Open Task Manager and End Process for Brackets
4. Restart Brackets

Results
When Brackets re-opened, it remembered the new project and the project tree state, but it did not remember which file was open in the working set (it showed the one I closed and not the one I opened).

So, it seems that you need to force write all of the pending data changes to disk before removing the "user" prefs and state?

Note: I recently got a new lenovo laptop with SSD, and it usually hangs when trying to run All/All tests.

[core-ai-bot]

Comment by dangoor
Wednesday Mar 12, 2014 at 15:29 GMT

---

@redmunds With those steps to reproduce, that sounds like an issue with viewstate generally and not related to the SpecRunner window. Basically, if anything caused Brackets to crash and you had to end the process, you would likely have had the same results. That's probably worth a different bug, I think.

What this change is intended to fix is if any test modifies the preferences or view state it shouldn't change the real prefs or view state of the person running the test.

[core-ai-bot]

Comment by redmunds
Thursday Mar 13, 2014 at 02:59 GMT

---

@dangoor Yes, I guess crash Brackets is a different case, but I'm certain I have also hit that for case of failing test(s). What about this comment?

[core-ai-bot]

Comment by dangoor
Thursday Mar 13, 2014 at 14:53 GMT

---

@redmunds It's possible that I've missed something, but I think with this branch the tests should not be interacting with your own prefs at all.

[core-ai-bot]

Comment by dangoor
Wednesday Mar 19, 2014 at 15:27 GMT

---

Updated to reliably replace the preferences systems used in testing with memory-backed ones that never touch the user's file. It works by reconfiguring RequireJS to change which module is used for creating the PreferenceSystem instances.

[core-ai-bot]

Comment by redmunds
Wednesday Mar 19, 2014 at 21:29 GMT

---

FYI, I opened a separate issue for The default for linting should be off.

[core-ai-bot]

Comment by redmunds
Wednesday Mar 19, 2014 at 22:46 GMT

---

All of the bugs related to tests mentioned so far seem to be fixed!

But I'm having problems running Integration tests on Win7 (Mac 10.8 passes). I switched back to master and they passed except for the known issues (being fixed in this PR). Since those issues are related to prefs, I tried the following:

1. Shutdown Brackets
2. Delete brackets.json and state.json files
3. Make sure git status is clean
4. Start Brackets, Debug > Run Tests
5. Select Integration tab, run "All"

The Results are close to what I am seeing in your branch

Results on master branch with no prefs:

Results in your branch (which seems to have less failures than previous runs):

Note: The Integration tests mostly pass if I run 1 set of tests at a time. I'm not sure how this fits into the formula.

[core-ai-bot]

Comment by RaymondLim
Thursday Mar 20, 2014 at 00:17 GMT

---

@redmunds I'm getting all "Live Development" and "Inspector" tests (10 in total) fail in master on my win7, but only 8 in "Live Development" and two in MultiRangeInlineEditor failing in this pull request branch. If I re-run all Integration tests again, then all of them pass in this branch.

[core-ai-bot]

Comment by redmunds
Thursday Mar 20, 2014 at 00:32 GMT

---

@RaymondLim It's weird that tests (mostly?) pass if I run 1 set at a time. The problem is when I run Integration/All. Does that work for you?

[core-ai-bot]

Comment by RaymondLim
Thursday Mar 20, 2014 at 00:33 GMT

---

Yes, I'm running Integration/All.

[core-ai-bot]

Comment by redmunds
Thursday Mar 20, 2014 at 02:44 GMT

---

OK, I guess I need to run each set one at a time. All test pass on Win7 except for 1 EditorOptionHandlers unit test:

"should also wrap long lines in inline editor by default"

Error: Expected 30 to be less than 30.
    at new jasmine.ExpectationResult (file:///C:/Users/redmunds/dev/github/brackets-shell/Release/dev/test/thirdparty/jasmine-core/jasmine.js:102:32)
    at null.toBeLessThan (file:///C:/Users/redmunds/dev/github/brackets-shell/Release/dev/test/thirdparty/jasmine-core/jasmine.js:1194:29)
    at null.<anonymous> (file:///C:/Users/redmunds/dev/github/brackets-shell/Release/dev/test/spec/EditorOptionHandlers-test.js:101:45)
    at jasmine.Block.execute (file:///C:/Users/redmunds/dev/github/brackets-shell/Release/dev/test/thirdparty/jasmine-core/jasmine.js:1024:15)
    at jasmine.Queue.next_ (file:///C:/Users/redmunds/dev/github/brackets-shell/Release/dev/test/thirdparty/jasmine-core/jasmine.js:1842:31)
    at onComplete (file:///C:/Users/redmunds/dev/github/brackets-shell/Release/dev/test/thirdparty/jasmine-core/jasmine.js:1838:18)
    at jasmine.WaitsForBlock.execute (file:///C:/Users/redmunds/dev/github/brackets-shell/Release/dev/test/thirdparty/jasmine-core/jasmine.js:2322:5)
    at file:///C:/Users/redmunds/dev/github/brackets-shell/Release/dev/test/thirdparty/jasmine-core/jasmine.js:2336:12

[core-ai-bot]

Comment by dangoor
Thursday Mar 20, 2014 at 14:01 GMT

---

I just ran all integration tests on my branch (on Mac) and they all passed. I have sporadically seen failures on master (or other branches) for the Live Development tests. I also just ran EditorOptionHandlers test alone and it passed.

@redmunds just to double check: you do have caching turned off and the devtools open when running the test, right? That EditorOptionHandlers test would fail on master currently if your font size is increased.

With the change introduced in this PR, the only new failure I might expect is one where some piece of code expects userPrefFile to be something other than null, but there's nothing in the code that does that.

[core-ai-bot]

Comment by redmunds
Thursday Mar 20, 2014 at 14:47 GMT

---

@dangoor I haven't specifically turned off caching in devtools, but devtools is not running when I run unit tests. I manually removed all extensions (since I wasn't sure if "Reload Without Extensions" extended to test windows). This test passes for me on Mac 10.8, but fails every time on Win7.

[core-ai-bot]

Comment by dangoor
Thursday Mar 20, 2014 at 18:02 GMT

---

@redmunds from what I've seen, if you don't explicitly set caching off and have dev tools open when you run the tests, the specrunner will run whatever test code was there when you launched Brackets. The only way I reliably see tests running the code that is actually sitting on disk at that moment is to have the dev tools open with caching disabled.

[core-ai-bot]

Comment by dangoor
Thursday Mar 20, 2014 at 18:33 GMT

---

I just ran all of the integration tests on Windows (7) and they all passed for me.