-
Notifications
You must be signed in to change notification settings - Fork 358
[ENG-10028] SHARE is not consistently indexing OSF content #11671
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
adlius
merged 2 commits into
CenterForOpenScience:feature/pbs-26-6
from
mkovalua:fix/ENG-10028-pbs-26-6
Apr 13, 2026
Merged
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Empty file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| from django.urls import re_path | ||
| from . import views | ||
|
|
||
| app_name = 'admin' | ||
|
|
||
| urlpatterns = [ | ||
| re_path(r'^$', views.FailedShareIndexedGuidList.as_view(), name='list'), | ||
| re_path(r'^(?P<resource_type>[^/]+)/$', views.FailedShareIndexedGuidReindex.as_view(), name='reindex-share-resource'), | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| from django.contrib.auth.mixins import PermissionRequiredMixin | ||
| from django.urls import reverse | ||
| from django.shortcuts import redirect | ||
| from django.views.generic import ListView, View | ||
| from osf.models import Guid | ||
| from urllib.parse import urlencode | ||
| from api.share.utils import get_not_indexed_guids_for_resource_with_no_indexed_guid, task__reindex_failed_or_not_indexed_resource_into_share | ||
|
|
||
| class FailedShareIndexedGuidList(PermissionRequiredMixin, ListView): | ||
| paginate_by = 25 | ||
| template_name = 'share_reindex/list.html' | ||
| permission_required = 'osf.update_share_reindex' | ||
| raise_exception = True | ||
| model = Guid | ||
|
|
||
| def get_queryset(self): | ||
| resource_type = self.request.GET.get('type', 'projects') | ||
| return get_not_indexed_guids_for_resource_with_no_indexed_guid(resource_type) | ||
|
|
||
| def get_context_data(self, **kwargs): | ||
| query_set = kwargs.pop('object_list', self.object_list) | ||
| page_size = self.get_paginate_by(query_set) | ||
| paginator, page, query_set, is_paginated = self.paginate_queryset(query_set, page_size) | ||
| kwargs.setdefault('items_to_index', query_set) | ||
| kwargs.setdefault('page', page) | ||
| resource_type = self.request.GET.get('type', 'projects') | ||
| kwargs.setdefault('selected_resource_type', resource_type) | ||
| resource_type_detail_mapping = { | ||
| 'users': 'users:user', 'preprints': 'preprints:preprint', 'registries': 'nodes:node', 'projects': 'nodes:node', 'files': 'files:file' | ||
| } | ||
|
|
||
| kwargs.setdefault('resource_detail', resource_type_detail_mapping.get(resource_type)) | ||
| resource_type_guid_reindex = { | ||
| 'users': 'users:reindex-share-user', 'preprints': 'preprints:reindex-share-preprint', 'registries': 'nodes:reindex-share-node', 'projects': 'nodes:reindex-share-node' | ||
| } | ||
| kwargs.setdefault('resource_guid_reindex', resource_type_guid_reindex.get(resource_type)) | ||
| status_msg = f'Reindex of {resource_type} started, please check later.' if self.request.GET.get('status') == 'indexing' else '' | ||
| kwargs.setdefault('share_reindex_message', status_msg) | ||
| return super().get_context_data(**kwargs) | ||
|
|
||
|
|
||
| class FailedShareIndexedGuidReindex(PermissionRequiredMixin, View): | ||
| permission_required = 'osf.update_share_reindex' | ||
| raise_exception = True | ||
|
|
||
| def post(self, request, *args, **kwargs): | ||
| resource_type = self.kwargs.get('resource_type') | ||
| # reindex 100_000 guids in background task for specific resource_type and resource is public | ||
| task__reindex_failed_or_not_indexed_resource_into_share.delay(resource_type) | ||
| base_url = reverse('share_reindex:list') | ||
| query_string = urlencode({'type': resource_type, 'status': 'indexing'}) | ||
| return redirect(f"{base_url}?{query_string}") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,133 @@ | ||
| {% extends "base.html" %} | ||
| {% load render_bundle from webpack_loader %} | ||
| {% load comment_extras %} | ||
|
|
||
| {% load static %} | ||
| {% block top_includes %} | ||
| <link rel="stylesheet" type="text/css" href="/static/css/institutions.css" /> | ||
| {% endblock %} | ||
| {% block title %} | ||
| <title>Share Reindex</title> | ||
| {% endblock title %} | ||
| {% block content %} | ||
| <h2>Share Reindex</h2> | ||
|
|
||
| {% include "util/pagination.html" with items=page extra_query_params="&type="|add:selected_resource_type %} | ||
|
|
||
|
|
||
| <div class="row" style="margin-bottom: 20px;"> | ||
| <div class="col-md-3"> | ||
| <form method="GET" action=""> | ||
| <select class="form-control" name="type" onchange="this.form.submit()"> | ||
| <option value="projects" {% if selected_resource_type == 'projects' %}selected{% endif %}>Projects</option> | ||
| <option value="preprints" {% if selected_resource_type == 'preprints' %}selected{% endif %}>Preprints</option> | ||
| <option value="registries" {% if selected_resource_type == 'registries' %}selected{% endif %}>Registries</option> | ||
| <option value="users" {% if selected_resource_type == 'users' %}selected{% endif %}>Users</option> | ||
| <option value="files" {% if selected_resource_type == 'files' %}selected{% endif %}>Files</option> | ||
| </select> | ||
|
|
||
| </form> | ||
| </div> | ||
| <div class="col-md-3"> | ||
| <a data-toggle="modal" data-target="#confirmReindexShareNodes" class="btn btn-primary"> | ||
| SHARE Reindex All {{selected_resource_type}} | ||
| </a> | ||
|
|
||
| <div class="modal" id="confirmReindexShareNodes"> | ||
| <div class="modal-dialog"> | ||
| <div class="modal-content"> | ||
|
|
||
| <form method="post" action="{% url 'admin:reindex-share-resource' resource_type=selected_resource_type %}"> | ||
| {% csrf_token %} | ||
| <div class="modal-header"> | ||
| <button type="button" class="close" data-dismiss="modal">×</button> | ||
| <h3>Are you sure you want to reindex {{selected_resource_type}} (SHARE)?</h3> | ||
| </div> | ||
|
|
||
| <div class="modal-footer"> | ||
| <button type="button" class="btn btn-default" data-dismiss="modal">Cancel</button> | ||
| <input class="btn btn-primary" type="submit" value="Confirm Re-index" /> | ||
| </div> | ||
| </form> | ||
|
|
||
| </div> | ||
| </div> | ||
| </div> | ||
| </div> | ||
| </div> | ||
|
|
||
| <div> | ||
| <p>{{share_reindex_message}}</p> | ||
| </div> | ||
|
|
||
|
|
||
| <table class="table table-striped table-hover table-responsive"> | ||
| <thead> | ||
| <tr> | ||
| <th>Guid</th> | ||
| {% if selected_resource_type == 'projects' or selected_resource_type == 'preprints' or selected_resource_type == 'registries' %} | ||
| <th>Title</th> | ||
| {% elif selected_resource_type == 'users' %} | ||
| <th>Fullname</th> | ||
| {% else %} | ||
| <th>Name</th> | ||
| {% endif %} | ||
| <th>Datetime Last Indexed</th> | ||
| <!-- there is no a file indexing option for detail page for now --> | ||
| {% if selected_resource_type != 'files' %} | ||
| <th>Reindex</th> | ||
| {% endif %} | ||
| </tr> | ||
| </thead> | ||
| <tbody> | ||
| {% for item in items_to_index %} | ||
| <tr> | ||
| <td> | ||
| <a href="{% url resource_detail guid=item.first_guid %}"> | ||
| {{item.first_guid}} | ||
| </a> | ||
| </td> | ||
| {% if selected_resource_type == 'projects' or selected_resource_type == 'preprints' or selected_resource_type == 'registries' %} | ||
| <td>{{item.title}}</td> | ||
| {% elif selected_resource_type == 'users' %} | ||
| <td>{{item.fullname}}</td> | ||
| {% else %} | ||
| <th>{{item.name}}</th> | ||
| {% endif %} | ||
|
|
||
| <td>{{item.date_last_indexed}}</td> | ||
|
|
||
| {% if selected_resource_type != 'files' %} | ||
| <td> | ||
| <a data-toggle="modal" data-target="#confirmReindexShareNode-{{ item.first_guid }}" class="btn btn-primary">SHARE Reindex</a> | ||
| </td> | ||
| <div class="modal" id="confirmReindexShareNode-{{ item.first_guid }}"> | ||
| <div class="modal-dialog"> | ||
| <div class="modal-content"> | ||
| <form class="well" method="post" action="{% url resource_guid_reindex guid=item.first_guid %}"> | ||
| <div class="modal-header"> | ||
| <button type="button" class="close" data-dismiss="modal">x</button> | ||
| <h3>Are you sure you want to reindex this node (SHARE)? {{ item.first_guid }}</h3> | ||
| </div> | ||
| {% csrf_token %} | ||
| <div class="modal-footer"> | ||
| <input class="btn btn-danger" type="submit" value="Confirm" /> | ||
| <button type="button" class="btn btn-default" data-dismiss="modal"> | ||
| Cancel | ||
| </button> | ||
| </div> | ||
| </form> | ||
|
|
||
| </div> | ||
| {# Data from above link #} | ||
| </div> | ||
| </div> | ||
| {% endif %} | ||
|
|
||
|
|
||
| </tr> | ||
| {% endfor %} | ||
| </tbody> | ||
| </table> | ||
|
|
||
| {% endblock content %} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
53 changes: 53 additions & 0 deletions
53
osf/migrations/0038_abstractnode_date_last_indexed_and_more.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| # Generated by Django 4.2.26 on 2026-03-31 15:44 | ||
|
|
||
| from django.db import migrations, models | ||
|
|
||
|
|
||
| class Migration(migrations.Migration): | ||
|
|
||
| dependencies = [ | ||
| ('osf', '0037_notification_refactor_post_release'), | ||
| ] | ||
|
|
||
| operations = [ | ||
| migrations.AddField( | ||
| model_name='abstractnode', | ||
| name='date_last_indexed', | ||
| field=models.DateTimeField(blank=True, null=True), | ||
| ), | ||
| migrations.AddField( | ||
| model_name='abstractnode', | ||
| name='has_been_indexed', | ||
| field=models.BooleanField(blank=True, db_index=True, default=None, null=True), | ||
| ), | ||
| migrations.AddField( | ||
| model_name='basefilenode', | ||
| name='date_last_indexed', | ||
| field=models.DateTimeField(blank=True, null=True), | ||
| ), | ||
| migrations.AddField( | ||
| model_name='basefilenode', | ||
| name='has_been_indexed', | ||
| field=models.BooleanField(blank=True, db_index=True, default=None, null=True), | ||
| ), | ||
| migrations.AddField( | ||
| model_name='osfuser', | ||
| name='date_last_indexed', | ||
| field=models.DateTimeField(blank=True, null=True), | ||
| ), | ||
| migrations.AddField( | ||
| model_name='osfuser', | ||
| name='has_been_indexed', | ||
| field=models.BooleanField(blank=True, db_index=True, default=None, null=True), | ||
| ), | ||
| migrations.AddField( | ||
| model_name='preprint', | ||
| name='date_last_indexed', | ||
| field=models.DateTimeField(blank=True, null=True), | ||
| ), | ||
| migrations.AddField( | ||
| model_name='preprint', | ||
| name='has_been_indexed', | ||
| field=models.BooleanField(blank=True, db_index=True, default=None, null=True), | ||
| ), | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The naming of the argument
first_guidcould be a bit more descriptive. How about something likeonly_oldest_guid?