Integration Tests Failing on Develop

[djbrooke]

We're seeing integration tests fail over the past few runs in develop:

• edu.harvard.iq.dataverse.api.HarvestingServerIT.testOaiFunctionality
• edu.harvard.iq.dataverse.api.DatasetsIT.testCreatePublishDestroyDataset
• edu.harvard.iq.dataverse.api.DatasetsIT.testExcludeEmail

Some details about the failures:

https://jenkins.dataverse.org/job/IQSS-dataverse-develop/382/testReport/

[donsizemore]

@djbrooke FWIW I just ran this through in a VM on my new SSD. 8 cores and 8GB of RAM made the tests run cleanly, and the API test suite kept most of the cores pegged for the duration of the tests.

[djbrooke]

Thanks @donsizemore, that's good to hear. Is there an easy way to give the jenkins.dataverse.org instance some more juice so it can get through the tests?

cc @landreev because you mentioned you were going to look into the Harvesting test

[donsizemore]

@djbrooke tweaking the JVM heap didn't help... there are 181 occurrences of the following error, which I haven't seen before:

server.log_2020-03-10T14-18-55: java.lang.IllegalAccessException: Class edu.harvard.iq.dataverse.search.SearchServiceBean can not access a member of class edu.harvard.iq.dataverse.search.SearchFields with modifiers "private static transient" at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102) at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:296) at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:288) at java.lang.reflect.Field.get(Field.java:390) at edu.harvard.iq.dataverse.search.SearchServiceBean.search(SearchServiceBean.java:684)

[landreev]

OK, it appears that the 3 tests that have been failing consistently are the result of #6739.
That in turn was introduced in PR #6670; merged on Feb. 27,and that's when the RestAssured tests started failing.

[landreev]

So let's maybe put this one on hold, until #6739 is fixed.

[landreev]

OK, fixing #6739 appears to have fixed the 3 tests that were failing previously:

edu.harvard.iq.dataverse.api.HarvestingServerIT.testOaiFunctionality
edu.harvard.iq.dataverse.api.DatasetsIT.testCreatePublishDestroyDataset
edu.harvard.iq.dataverse.api.DatasetsIT.testExcludeEmail

However, it also appears that #6490 (also merged yesterday) has introduced 3 new failures:

edu.harvard.iq.dataverse.api.FilesIT.testReplaceFileBadJson
edu.harvard.iq.dataverse.api.FilesIT.test_006_ReplaceFileGood
edu.harvard.iq.dataverse.api.FilesIT.testForceReplaceAndUpdate

[kcondon]

I did a quick replace file test on develop and it failed silently in UI but saw this in log:

[2020-03-24T15:00:11.622-0400] [glassfish 4.1] [WARNING] [] [javax.enterprise.resource.webcontainer.jsf.lifecycle] [tid: _ThreadID=209 _ThreadName=jk-connector(4)] [timeMillis: 1585076411622] [levelValue: 900] [[
  /editFilesFragment.xhtml @144,65 fileUploadListener="#{EditDatafilesPage.handleFileUpload}": java.lang.NullPointerException
javax.el.ELException: /editFilesFragment.xhtml @144,65 fileUploadListener="#{EditDatafilesPage.handleFileUpload}": java.lang.NullPointerException
        at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:111)
        at org.primefaces.component.fileupload.FileUpload.broadcast(FileUpload.java:63)
        at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:755)
        at javax.faces.component.UIViewRoot.processDecodes(UIViewRoot.java:931)
        at com.sun.faces.lifecycle.ApplyRequestValuesPhase.execute(ApplyRequestValuesPhase.java:78)
        at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
        at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:198)
        at javax.faces.webapp.FacesServlet.service(FacesServlet.java:646)
        at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:295)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:226)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673)
        at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282)
        at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167)
        at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201)
        at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175)
        at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
        at edu.harvard.iq.dataverse.datasetutility.AddReplaceFileHelper.runAddReplacePhase1(AddReplaceFileHelper.java:506)
        at edu.harvard.iq.dataverse.datasetutility.AddReplaceFileHelper.runReplaceFromUI_Phase1(AddReplaceFileHelper.java:459)
        at edu.harvard.iq.dataverse.datasetutility.FileReplacePageHelper.handleNativeFileUpload(FileReplacePageHelper.java:117)
        at edu.harvard.iq.dataverse.EditDatafilesPage.handleReplaceFileUpload(EditDatafilesPage.java:1874)
        at edu.harvard.iq.dataverse.EditDatafilesPage.handleFileUpload(EditDatafilesPage.java:1984)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.el.parser.AstValue.invoke(AstValue.java:289)
        at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:304)
        at org.jboss.weld.util.el.ForwardingMethodExpression.invoke(ForwardingMethodExpression.java:40)
        at org.jboss.weld.el.WeldMethodExpression.invoke(WeldMethodExpression.java:50)
        at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105)
        ... 45 more
]]

[djbrooke]

Thanks @kcondon.

@qqmyers based on the log entry is it likely that this was introduced by the work in #6490? If so, could you take a look?

[qqmyers]

I looked briefly this afternoon. It's going to be very hard to debug without seeing the log for the tests to find out where the problem starts. Is there a way to get that? (I didn't see anything obvious in jenkins).
My guesses were either that the test uses a default/dummy value that is no longer valid, or that it doesn't set something that exists for normal datasets but was left out to keep the test simple. As I said, hard to tell without being able to see the details prior to the test failure.

[landreev]

@qqmeyers The server logs from the node that's running the tests are available through jenkins; hold on, I'll find the links.
But, from what @kcondon is saying above, it sounds like it's not just Jenkins - more like file replace is just broken in the develop branch as of now - ?
(and what happens when you just try these 3 RestAssured tests in your own environment?)

[qqmyers]

Ah - didn't look up. His log will help.

[landreev]

@qqmyers The server logs and such from the last Jenkins run:
https://jenkins.dataverse.org/job/IQSS-dataverse-develop/ws/target/

(and of course the full console output log of the jenkins test job is here: https://jenkins.dataverse.org/job/IQSS-dataverse-develop/396/consoleFull)

[landreev]

(also, the EC2 instance that ran the test is still up - they are kept alive for a couple of days - so if you need shell access to it, Don should be able to send the key)