fix: bypass pagination in download_results to prevent truncated exports (backport #1096)

[mergify]

• Add paginate=True parameter to execute_ibis_query() — skip pagination when False
• Update download_results() to call execute_ibis_query(..., paginate=False, ...)
• Apply a 100,000-row limit on the ibis query before calling execute_ibis_query() in download_results()

Original prompt

Please create a pull request against develop in repository frappe/insights with a minimal, clean fix for query result downloads being truncated by pagination.

Context:

• File already identified: insights/insights/doctype/insights_query_v3/insights_query_v3.py
• download_results() currently calls execute_ibis_query() with a very large page_size, but downloads are still affected by pagination behavior and no longer fetch the full result set.
• We want an explicit pagination control in the lower-level execution helper.

Required changes:

1. Add a new paginate parameter to execute_ibis_query().
2. Keep the change minimal and backwards-compatible for existing callers.
3. Update download_results() in insights_query_v3.py to call execute_ibis_query(..., paginate=False, ...) so exports fetch full results.
4. Ensure existing paginated behavior remains unchanged for normal query execution paths.
5. Update any affected call sites, signatures, and tests as needed, but keep the PR narrowly scoped.

Expected outcome:

• Normal UI/query execution remains paginated as before.
• Download/export paths can explicitly disable pagination and retrieve full results.
• The PR should be clean, minimal, and ready for review.

The following is the prior conversation context from the user's chat exploration (may be truncated):

User: The download function is not affected by page & page_size change, we can't download full results now, suggest an appropriate fix

Assistant: A good fix is to make download_results() explicitly bypass pagination limits imposed by execute_ibis_query, instead of relying on a very large page_size.

Right now it does this:

results, _ = execute_ibis_query(
    ibis_query,
    cache=False,
    page_size=10_00_000,
    reference_doctype=self.doctype,
    reference_name=self.name,
)

If execute_ibis_query() now enforces page/page_size differently, caps page size, or always paginates, downloads will silently return only one page. The safest fix is:

1. add a dedicated “download all” / “no pagination” path to execute_ibis_query, or
2. if that helper already supports it, call it with pagination disabled.

Recommended change

• Extend execute_ibis_query() to accept something like paginate=False or page_size=None.
• In download_results(), call it with pagination disabled so the full result set is exported.

Example intent:

results, _ = execute_ibis_query(
    ibis_query,
    cache=False,
    paginate=False,
    reference_doctype=self.doctype,
    reference_name=self.name,
)

Why this is better than a huge page_size

• avoids hidden server-side caps
• makes the intent explicit: export the full dataset
• keeps UI pagination behavior separate from export behavior
• is more robust if defaults in execute_ibis_query() changed

If you want a minimal local-only fallback

• loop through pages inside download_results() and concatenate all results until fewer than page_size rows are returned

That would look conceptually like:

all_results = []
page = 1
page_size = 5000

while True:
    chunk, _ = execute_ibis_query(
        ibis_query,
        cache=False,
        page=page,
        page_size=page_size,
        reference_doctype=self.doctype,
        reference_name=self.name,
    )
    all_results.append(chunk)
    if len(chunk) < page_size:
        break
    page += 1

results = pd.concat(all_results, ignore_index=True)

Best fix

• Prefer fixing execute_ibis_query() once, because other exports or “full fetch” callers may have the same bug.

Concretely, I’d suggest:

• inspect execute_ibis_query() for any default pagination/cap behavior
• add an explicit full-fetch mode
• update download_results() to use that mode

If you want, I can trace execute_ibis_query() in this repo and point to the exact code change.

User: Okay, add paginate param to execute_ibis_query() and raise a PR with clean minimal changes

Assistant: I’ve started the pull request workflow and we’re waiting for your response to the confirmation dialog to continue.

This pull request was created from Copilot chat.

---

This is an automatic backport of pull request #1096 done by [Mergify](https://mergify.com).

[github-actions]

🎉 This PR is included in version 3.9.5 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀