AgentSearch Challenge Section#10
Open
arastun-m wants to merge 2 commits intoAgent-Search:mainfrom
Open
Conversation
arastun-m
commented
Apr 22, 2026
arastun-m
commented
- Added challenge sections details.
- Fixed a typo in README.md website link.
Bingo-W
reviewed
Apr 30, 2026
| <h2 class="section-title">AgentSearch Challenge</h2> | ||
| <div class="section-content"> | ||
| <p>Participate in the first shared task on ranking AI agents given task descriptions. The challenge provides a benchmark for evaluating agent search systems in practical scenarios.</p> | ||
| <p>Participate in the first shared task on retrieving and ranking AI agents given task descriptions. The challenge provides a benchmark for evaluating agent search systems in practical scenarios.</p> |
Collaborator
There was a problem hiding this comment.
There is a mistake in the input. In the challenge, we provide both the task query and task description.
Bingo-W
reviewed
Apr 30, 2026
| <li><strong>Agent Collection:</strong> 500+ documented AI agents with capability descriptions, API specifications, and performance metadata</li> | ||
| <li><strong>Task Queries:</strong> 200+ task descriptions covering information retrieval, data analysis, content generation, and decision support scenarios</li> | ||
| <li><strong>Ground Truth:</strong> Relevance judgments based on agent execution results and expert assessments</li> | ||
| <li><strong>Retrieval</strong>: Given a natural language task description, participants develop methods to retrieve the most relevant agents from a large-scale pool. This track evaluates the ability to surface strong candidate agents efficiently and accurately.</li> |
Collaborator
There was a problem hiding this comment.
same as the above mistake.
Bingo-W
reviewed
Apr 30, 2026
| <ul class="challenge-list"> | ||
| Coming soon. | ||
| <li><strong>Agent Collection:</strong> Collection of real-world agents with capability descriptions and metadata.</li> | ||
| <li><strong>Task Queries:</strong> Task descriptions and associated single-agent and multi-agent task queries.</li> |
Collaborator
There was a problem hiding this comment.
We have defined the input as a task specification. And it contains two different types: (1) task query; and (2) task description.
The current text is not aligned with our benchmark.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.