OpenSearch Dashboards map hoverover broken [BUG]

[macrakis]

Describe the bug

Hoverover on the maps in playground's Global Flight Dashboard gives error "Cannot read properties of undefined"

To Reproduce

Open playground: https://playground.opensearch.org/app/home#/
Click on hamburger menu > OpenSearch Dashboards > Dashboard
Click on [Flights] Global Flight Dashboard
Scroll down to "[Flights] Airport Connections (Hover Over Airport)"
Even without performing any actions on that map, you get the error "Cannot read properties of undefined", which starts repeating.
Same thing on the [Flights] Departures Count Map
Also, hoverover is a no-op

Expected behavior

No error message.
Hoverovers should show info.

Plugins
Please list all plugins currently enabled.

Query Workbench
Alerting
Anomaly Detection
Notifications
Observability
Security Analytics
Index Management
Snapshot Management
Search Relevance

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

Mac Chrome 107.0.5304.110 (Official Build) (x86_64)

Additional context
Add any other context about the problem here.

[dblock]

Playground is https://github.com/opensearch-project/dashboards-anywhere/. Moving the issue there once I get access, asked @seraphjiang.

[BionIT]

This issue comes from the official docker image of os and osd, and can be reproduced by using the following docker compose file and run docker-compose up:

services:
  opensearch:
    container_name: open-search-local
    image: opensearchproject/opensearch:2.4.0
    environment:
      - discovery.type=single-node # single node
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    volumes:
      - opensearch-data:/usr/share/opensearch/data
    ports:
      - "9200:9200"
      - "9600:9600"
    networks:
      - opensearch-net
  opensearchdashboard:
    container_name: open-search-dashboard-local
    image: opensearchproject/opensearch-dashboards:2.4.0
    environment:
      OPENSEARCH_HOSTS: "https://opensearch:9200"
    ports:
      - "5601:5601"
    networks:
      - opensearch-net

volumes:
  opensearch-data:

networks:
  opensearch-net:

[seraphjiang]

Thanks @BionIT , could we try reproduce with early version 2.3, 2.2

might be regression introduced by #2352

[BionIT]

interestingly, tried official docker image of 2.3.0, 2.2.0, 2.2.1 and all of they failed to start OpenSearch with the following error - is it expected?? This is due to mounted volume was using 2.4.0 (i.e. - opensearch-data:/usr/share/opensearch/data
)

uncaught exception in thread [main]
java.lang.IllegalArgumentException: Could not load codec 'Lucene94'. Did you forget to add lucene-backward-codecs.jar?
Likely root cause: java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.Codec with name 'Lucene94' does not exist.  You need to add the corresponding JAR file supporting this SPI to your classpath.  The current classpath supports the following names: [Lucene80, Lucene84, Lucene86, Lucene87, Lucene70, Lucene90, Lucene91, Lucene92, KNN80Codec, KNN84Codec, KNN86Codec, KNN87Codec, KNN910Codec, KNN920Codec]
        at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:113)
        at org.apache.lucene.codecs.Codec.forName(Codec.java:118)
        at org.apache.lucene.index.SegmentInfos.readCodec(SegmentInfos.java:511)
        at org.apache.lucene.index.SegmentInfos.parseSegmentInfos(SegmentInfos.java:404)
        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:363)
        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:299)
        at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:88)
        at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:77)
        at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:809)
        at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)
        at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:67)
        at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:60)
        at org.opensearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:309)
        at org.opensearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:453)
        at org.opensearch.env.NodeEnvironment.<init>(NodeEnvironment.java:369)
        at org.opensearch.node.Node.<init>(Node.java:432)
        at org.opensearch.node.Node.<init>(Node.java:341)
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.Command.main(Command.java:101)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)

[seraphjiang]

@BionIT would you share the compose file used to launch 2.3.0, 2.2.0?

We could ask @peterzhuamazon from infra team to help and confirm. if it is an issue of 2.x image, let's report an issue to https://github.com/opensearch-project/opensearch-build

cc: @dblock

[BionIT]

Okay, sorry for the false alarm for 2.3.0. I just re-created 2.3.0 using official docker image using the following docker-compose.yaml and verified that it DOES NOT have same map issue. The issue is for 2.4.0 OS and OSD docker image

version: "3"  # optional
services:
  opensearch:
    container_name: open-search-local2
    image: opensearchproject/opensearch:2.3.0
    environment:
      - discovery.type=single-node # single node
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    volumes:
      - opensearch-data:/usr/share/opensearch/data
    ports:
      - "9200:9200"
      - "9600:9600"
    networks:
      - opensearch-net
  opensearchdashboard:
    container_name: open-search-dashboard-local2
    image: opensearchproject/opensearch-dashboards:2.3.0
    environment:
      OPENSEARCH_HOSTS: "https://opensearch:9200"
    ports:
      - "5601:5601"
    networks:
      - opensearch-net

volumes:
  opensearch-data:

networks:
  opensearch-net:

[peterzhuamazon]

Okay, sorry for the false alarm for 2.3.0. I just re-created 2.3.0 using official docker image using the following docker-compose.yaml and verified that it DOES NOT have same map issue. The issue is for 2.4.0 OS and OSD docker image

version: "3"  # optional
services:
  opensearch:
    container_name: open-search-local2
    image: opensearchproject/opensearch:2.3.0
    environment:
      - discovery.type=single-node # single node
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    volumes:
      - opensearch-data:/usr/share/opensearch/data
    ports:
      - "9200:9200"
      - "9600:9600"
    networks:
      - opensearch-net
  opensearchdashboard:
    container_name: open-search-dashboard-local2
    image: opensearchproject/opensearch-dashboards:2.3.0
    environment:
      OPENSEARCH_HOSTS: "https://opensearch:9200"
    ports:
      - "5601:5601"
    networks:
      - opensearch-net

volumes:
  opensearch-data:

networks:
  opensearch-net:

Have seen this issue on my old machine but disappear on a new EC2 AL2.
Then I removed cache and files from my old machine with at least 30GB storage and it success again.
So I am not sure if that is because of my machine setups.

Right now I cannot reproduce this issue tho.

[BionIT]

@peterzhuamazon

Right now I cannot reproduce this issue tho.

Can you use 2.4.0? This map issue is for 2.4.0 and we just need to change the version in the yaml file. Below is screenshot

[seraphjiang]

Thanks @BionIT for quick verify

could we also verify in 1.3.6 and coming 1.3.7

@peterzhuamazon is there a release candidate for 1.3.7 we could verify.

[peterzhuamazon]

Thanks @BionIT for quick verify

could we also verify in 1.3.6 and coming 1.3.7

@peterzhuamazon is there a release candidate for 1.3.7 we could verify.

Release candidate is this Friday by EOD.

[BionIT]

I just tried the docker-compose file from 1.3.6 by the following steps and did NOT find map issue

1. go to https://opensearch.org/versions/opensearch-1-3-6.html
2. download docker-compose.yaml and move to desired dir
3. run docker-compose up -d
4. go to 0.0.0.0:5601 and add flight sample data
5. map is working as expected
   

[seraphjiang]

we have verified it is regression issue introduced after 2.3.0, and 1.3.6
@dblock @opensearch-project/transfer-request would you help to transfer to https://github.com/opensearch-project/opensearch-dashboards

[peterzhuamazon]

I just tried the docker-compose file from 1.3.6 by the following steps and did NOT find map issue

1. go to https://opensearch.org/versions/opensearch-1-3-6.html

2. download docker-compose.yaml and move to desired dir

3. run docker-compose up -d

4. go to 0.0.0.0:5601 and add flight sample data

5. map is working as expected
   ![Screen Shot 2022-12-01 at 9 56 00 AM](https://user-images.githubusercontent.com/17714150/205125775-b534012b-2023-45ea-a4b0-889ab15b4a77.png)

@BionIT could you compare the diffs between 136 compose 240 compose files?

@seraphjiang I will tranfer

Thanks.