Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 20 additions & 11 deletions doc/sphinx-guides/source/api/search.rst
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ metadata_fields string Includes the requested fields for each dataset in the
geo_point string Latitude and longitude in the form ``geo_point=42.3,-71.1``. You must supply ``geo_radius`` as well. See also :ref:`geospatial-search`.
geo_radius string Radial distance in kilometers from ``geo_point`` (which must be supplied as well) such as ``geo_radius=1.5``.
show_type_counts boolean Whether or not to include total_count_per_object_type for types: Dataverse, Dataset, and Files.
search_service string The name of the search service to use for this query. If omitted, the default search service will be used. For available search services, see :ref:`search-services-api`.
search_service string The name of the search service to use for this query. If omitted, the default search service will be used. For available search services, see :ref:`discovering-available-search-services`.
================ ======= ===========

Basic Search Example
Expand Down Expand Up @@ -779,15 +779,20 @@ Output from iteration example
CORS
</span>

.. _search-services-api:
.. _search-services:

Search Services API
-------------------
Search Services
---------------

.. _discovering-available-search-services:

Discovering Available Search Services
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To discover available search services and their capabilities, you can use the Search Services API endpoint.
Note: Configurable Search Services are an optional, experimental feature than may evolve faster than other parts of Dataverse.

Endpoint: https://demo.dataverse.org/api/search/services
Example API endpoint: https://demo.dataverse.org/api/search/services

This endpoint returns a list of available search services, including their names, display names, and any additional information about their capabilities.

Expand All @@ -804,21 +809,25 @@ Example response:
},
{
"name": "externalSearch",
"displayName": "AI Search for Datasets",
"displayName": "External Search for Datasets",
}
]
}

You can use the `name` values returned by this endpoint in the `search_service` parameter of the main search API to specify which search service to use for a particular query.
You can use the ``name`` values returned by this endpoint in the ``search_service`` parameter of the main search API to specify which search service to use for a particular query.

Using Different Search Services
-------------------------------
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To use a specific search service, include the `search_service` parameter in your search query. For example:
To use a specific search service, include the ``search_service`` parameter in your search query and pass the ``name``. For example:

https://demo.dataverse.org/api/search?q=trees&search_service=externalSearch

This query will use the externalSearch service (assuming it exists) instead of the default search service.
This query will use the ``externalSearch`` service (assuming it exists) instead of the default search service (``solr``).

.. note:: Other search services may not be complete replacements for the included ``solr`` service. For example, they may not support searching for collections or files (just datasets).

.. note:: Other search services may not be complete replacements for the included solr service. For example, they may not support searching for collections or files (just datasets).
Developing Search Services
~~~~~~~~~~~~~~~~~~~~~~~~~~

See :doc:`/developers/search-services` in the Developer Guide.
58 changes: 35 additions & 23 deletions doc/sphinx-guides/source/developers/search-services.rst
Original file line number Diff line number Diff line change
@@ -1,18 +1,23 @@
Search Services
===============
Dataverse now supports configurable search services, allowing developers to integrate new search engines dynamically. This guide outlines the design and provides details on how to use the interfaces and classes involved.

Dataverse supports configurable search services, allowing developers to integrate additional search engines dynamically. This guide outlines the design and provides details on how to use the interfaces and classes involved.

Design Overview
---------------
The configurable search services feature is designed to allow:

1. Dynamic addition of new search engines
2. Configuration of the Dataverse UI to use a specified search engine
3. Use of different search engines via the API
4. Discovery of installed search engines

Key Components
--------------

1. SearchService Interface
^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``SearchService`` interface is the core of the configurable search services. It defines the methods that any search engine implementation must provide.
The ``SearchService`` interface is the core of the configurable search services. It defines the methods that any search engine implementation must provide. (The methods below are accurate as of this writing.)

.. code-block:: java

Expand All @@ -28,9 +33,9 @@ The ``SearchService`` interface is the core of the configurable search services.
default void setSolrSearchService(SearchService solrSearchService);
}

The interface allows you to provide a service name and display name, and to respond to the same search parameters that are normally sent to the solr search engine.
The interface allows you to provide a service name and display name, and to respond to the same search parameters that are normally sent to the Solr search engine.

The setSolrSearchService method is used by Dataverse to give your class a reference to the SolrSearchService, allowing your class to perform solr queries as needed. (See the ExternalSearchServices for an example.)
The ``setSolrSearchService`` method is used by Dataverse to give your class a reference to the ``SolrSearchService``, allowing your class to perform Solr queries as needed. (See the ``ExternalSearchServices`` for an example.)

2. ConfigurableSearchService Interface
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand All @@ -43,15 +48,15 @@ The ``ConfigurableSearchService`` interface extends the ``SearchService`` interf
void setSettingsService(SettingsServiceBean settingsService);
}

The GetExternalSearchServiceBean and PostExternalSearchServiceBean classes provide a use case for this.
The ``GetExternalSearchServiceBean`` and ``PostExternalSearchServiceBean`` classes provide a use case for this.

3. JVM Options for Search Configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dataverse uses two JVM options to configure the search functionality:

- ``dataverse.search.services.directory``: Specifies the local directory where jar files with search engines (classes implementing the SearchService interface) can be found. Dataverse will dynamically load engines from this directory.
- ``dataverse.search.services.directory``: Specifies the local directory where jar files with search engines (classes implementing the ``SearchService`` interface) can be found. Dataverse will dynamically load engines from this directory.

- ``dataverse.search.default-service``: The serviceName of the service that should be used in the Dataverse UI.
- ``dataverse.search.default-service``: The ``serviceName`` of the service that should be used in the Dataverse UI.

Example configuration:

Expand All @@ -65,9 +70,11 @@ Remember to restart your Payara server after modifying these JVM options for the
4. Using Different Search Engines via API
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The loaded search services can be discovered using the /api/search/services endpoint.
The loaded search services can be discovered using the ``/api/search/services`` endpoint.

Queries can be made to different engines by including the optional search_service=<serviceName> query parameter.
Queries can be made to different engines by including the optional ``search_service=<serviceName>`` query parameter.

Use of these endpoints is described for end users in the API Guide under :ref:`search-services`.

Available Search Services
-------------------------
Expand All @@ -91,18 +98,23 @@ or
2. PostExternalSearchServiceBean
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

These classes implement the ConfigurableSearchService interface.
These classes implement the ``ConfigurableSearchService`` interface.
They make a GET or POST call (respectively) to an external search engine that must return a JSON array of objects with "DOI" and "Distance" keys.
The query sent to the external engine use the same query parameters as the Dataverse search API (GET) or have a JSON payload with those keys (POST).
The results they return are then searched for using the solr search engine which enforces access control and provides the standard formatting expected by the Dataverse UI and API.
The Distance values are used to order the results, smallest distances first.
The distance values are used to order the results, smallest distances first.

They can be configured via two settings each:

- GET

- :GetExternalSearchUrl - the URL to send GET search queries to
- :GetExternalSearchName - the display name to use for this configuration

- POST

They can be cofigured via 2 settings each:
:GetExternalSearchUrl - the URL to send GET search queries to
:GetExternalSearchName - the display name to use for this configuration

:PostExternalSearchUrl - the URL to send POST search queries to
:PostExternalSearchName - the display name to use for this configuration
- :PostExternalSearchUrl - the URL to send POST search queries to
- :PostExternalSearchName - the display name to use for this configuration

As these classes use DOIs as identifiers, they cannot reference collections or, unless file DOIs are enabled, files.
Similar classes, or extensions of these classes could search by database ids instead, etc. to support the additional types.
Expand All @@ -113,17 +125,17 @@ Similar classes, or extensions of these classes could search by database ids ins
4. OddlyEnoughSearchServiceBean
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

These classes implement the SearchService interface.
These classes implement the ``SearchService`` interface.
They are intended only as code examples and simple tests of the design and are not intended for production use.
The former simply replaces the user query with a query for entities with a db id < 1000. It demonstrates how a class can leverage the solr engine and achieve results solely by modifying/replacing the user query.
The latter only returns hits from the user's query that also have an odd database id. Since the filtering in the class changes the number of total hits available and pagination, this class demonstrates one way a developer can adjust those aspects of the solr response.
The latter only returns hits from the user's query that also have an odd database id. Since the filtering in the class changes the number of total hits available and pagination, this class demonstrates one way a developer can adjust those aspects of the Solr response.

Notes
-----

1. Unless you use the solr engine to provide access control, you must implement proper access control in your search engine
2. The design currently limits search results to be in the format returned by solr and the hits are expected to be collections, datasets, or files - other classes are not supported.
3. Search services could be designed to completely replace solr or to just support certain use cases (e.g. the external search classes only handling datasets).
1. Unless you use the Solr engine to provide access control, you must implement proper access control in your search engine
2. The design currently limits search results to be in the format returned by Solr and the hits are expected to be collections, datasets, or files - other classes are not supported.
3. Search services could be designed to completely replace Solr or to just support certain use cases (e.g. the external search classes only handling datasets).
4. While search services can be deployed as independent jar files, they currently import multiple Dataverse classes and, unlike exporters, cannot be built using just the Dataverse SPI.
5. As with other experimental features, we expect the SearchService interface may change over time as we learn about how people use it. Please keep in touch if you are developing search services.
5. As with other experimental features, we expect the ``SearchService`` interface may change over time as we learn about how people use it. Please keep in touch if you are developing search services.

10 changes: 10 additions & 0 deletions doc/sphinx-guides/source/installation/config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3566,6 +3566,16 @@ This setting allows admins to highlight a few of the 1000+ CSL citation styles a
These will be listed above the alphabetical list of all styles in the "View Styled Citations" pop-up.
The default value when not set is "chicago-author-date, ieee".

dataverse.search.services.directory
+++++++++++++++++++++++++++++++++++

Experimental. See :doc:`/developers/search-services`.

dataverse.search.default-service
++++++++++++++++++++++++++++++++

Experimental. See :doc:`/developers/search-services`.

.. _dataverse.cors:

CORS Settings
Expand Down