Add pages for collector components

[jaydeluca]

I am experimenting with automation that scrapes collector metadata and generates these pages. Github action jobs will run after each collector release and identify changes, and then will regenerate the pages and submit PRs accordingly.

This is a starting point, and likely to evolve as we expand the data we include for each component.

cc @tiffany76

---

PREVIEW: https://deploy-preview-8001--opentelemetry.netlify.app/docs/collector/components/

[dmathieu]

This is nice. But looking at it, it could be read that those are the only components anyone can use, which isn't true (if you build your own distribution, you can use any component, including private ones).

Speaking of distributions, there are 5 different official collector distributions. How about adding which of them each component is included into?

[tiffany76]

Thanks for taking a look at this POC, @dmathieu. We're just getting started and can use expert input!

We plan to update the language on the pages and will not only call out that users can use other components, but also cross-reference the registry, which includes many that are not part of core/contrib.

We're starting with a light lift for this POC while we investigate exactly what should be included and how it should be displayed. I'll add the otelcol distribution to our list of what should be included.

Thanks again!

[mx-psi]

This looks great!

Some ideas (not necessarily for this first version, just writing them down somewhere):

1. I wonder if it would make sense to style/order components differently depending on their stability level (since the stability level can vary per signal I guess the following can be done by taking the max stability level).
   For example, we could fade out in development components and strike out those that are unmaintained, and/or order components based on their max stability level (so that first you see stable components, then beta components, then alpha and lastly development). We could even separate alpha components into a collapsed box to signal that they are more experimental.
2. We are missing profiling support from the list! This is an experimental signal so maybe can be excluded
3. I think it would be great to have links to https://opentelemetry.io/docs/collector/configuration/#receivers and similar somewhere in these pages
4. It would also make sense to me to link to https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md somewhere

[chalin]

A few broad questions and comments:

• Have you seen https://opentelemetry.io/status/#collector? The added page content seems to be mostly statuses for components. Might it make sense to put these pages (or at least their status content), under https://opentelemetry.io/status/#collector or a new subsection of https://opentelemetry.io/status/?
• Re. the automation: under which repo will the GH actions run (and where will be the associated scripts, if any)?
• There is https://opentelemetry.io/docs/collector/distributions/. Is there an overlap with what is being proposed here as content? If so, will it lead to confusion for readers?

[tiffany76]

@mx-psi, thank you for the review!

I wonder if it would make sense to style/order components differently depending on their stability level (since the stability level can vary per signal I guess the following can be done by taking the max stability level).

That's an interesting idea. We should consider how to offer filtered/sorted views when we iterate on how the data is presented, especially if we move away from tables. 👍

We are missing profiling support from the list! This is an experimental signal so maybe can be excluded

Noted!

I think it would be great to have links to https://opentelemetry.io/docs/collector/configuration/#receivers and similar somewhere in these pages

I was thinking these pages will merge with that content to actually become the /receivers, /processors, /exporters, etc. pages in the new IA. So the corresponding content on that monolithic configuration page will be broken out into separate pages, and each page will have the OTel-maintained components listed (right now, in a table). WDYT?

It would also make sense to me to link to https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md somewhere

Noted! I think Patrice had the same idea. 👍

[mx-psi]

I was thinking these pages will merge with that content to actually become the /receivers, /processors, /exporters, etc. pages in the new IA. So the corresponding content on that monolithic configuration page will be broken out into separate pages, and each page will have the OTel-maintained components listed (right now, in a table). WDYT?

Ah, right, yeah, that makes sense

[tiffany76]

Thanks for giving this a look, @chalin!

Have you seen https://opentelemetry.io/status/#collector? The added page content seems to be mostly statuses for components. Might it make sense to put these pages (or at least their status content), under https://opentelemetry.io/status/#collector or a new subsection of https://opentelemetry.io/status/?

Although right now it looks like the new content is mostly adding statuses, the ultimate goal is to bridge the gap between component documentation and the official docs. As we iterate, we will hopefully be able to bring in more of the component metadata than just the stability level. So I don't think we should move this content to the /status page.

Re. the automation: under which repo will the GH actions run (and where will be the associated scripts, if any)?

I think @jaydeluca can answer that better than me.

There is https://opentelemetry.io/docs/collector/distributions/. Is there an overlap with what is being proposed here as content? If so, will it lead to confusion for readers?

No, there shouldn't be much overlap. The current Distributions page will be reworked in the new IA to help users focus on choosing a distribution. But I don't think we'll be going into component-level detail there. The only overlap I see (as of now, that could change, of course) is if we indicate which distributions each component is part of, as recommended by Damien.

[svrnm]

Thanks, this is really interesting and going to be helpful. One comment/ask:

While we still have it, can this script populate registry entries first and then fill the pages with that information?

[jaydeluca]

thank you everyone for all the valuable feedback thus far, I'm excited to see what we can do here. Apologies for the delay, I was traveling this week.

Here is my plan for next steps, as of now:

• Work on reworking the page copy, including taking out stability verbiage and link to existing resources for that information
• Include more details about the other distributions and which ones the components are included in (comment)
• Remove the x- components
• Explore integration with the existing registry as well
• Rework things a bit so the automation just manages the list contents on the pages instead of generating the entire page content

As for the tooling itself, I have everything running in https://github.com/jaydeluca/collector-watcher for now, which I think makes sense so I can iterate quickly until I get things in a good place and working - unless there are others who would like to collaborate and be included in code reviews, I'm happy to do whatever makes sense. I can also rewrite it in another language if that makes it easier for others to get involved, although it's not too complex.

My thinking is that I can get everything working end to end in an automated fashion from that repo and let it soak for a bit, monitor that it works as we want, and then we could move it either into this repo or if we want to keep it as a separate repo within the OTel org.

So the goal is that once these pages are finalized and merged, every night the automation will run, detect if there are updates, and will open PR's automatically.

Keep the feedback coming if you have other ideas or concerns. I will hopefully make some progress on this over the next week. Thanks all!

[jaydeluca]

Should we exclude unmaintained components from being included?

[mx-psi]

I think at the very least, they should be faded/clearly marked as something that you should not use

[jaydeluca]

added a marker:

and note at the bottom of each page:

[jaydeluca]

I have implemented most of the feedback. a couple notes:

• I had originally mentioned that I would have automation that runs nightly, but instead it makes more sense to trigger this when a new release is cut, so that's what we'll do instead
• I will work on an integration with the existing registry in a separate PR, since that will be a pretty large changeset

[mx-psi]

This looks great for a first version!

Some feedback looking at the preview:

1. If I look at this with half a screen, I am not able to tell that I need to do horizontal scrolling to see the stability for other signals:
   I am not sure if this is a problem for all tables, but in my particular resolution was confusing. On an iPhone 12 Pro (using Developer Tooling from Chrome) it looks like this which is also not ideal:

2. Connectors have stability defined for pairs of signals, which means currently the stability is empty: . I think for a first version we can just remove these columns for connectors. I suspect the best way of handling this is to have a 'highest stability' column (but that can be for a later PR)

3. I feel like 'components' should be closer to the top of the list on the sidebar, I would expect to want to look at components when I am thinking about how to configure my Collector

[jaydeluca]

If I look at this with half a screen, I am not able to tell that I need to do horizontal scrolling to see the stability for other signals

This seems to be an issue across the site (I tested with the java sdk page on my phone and see the same thing). When @tiffany76 and I first started discussing this, we figured that we'd eventually want to display more than just this basic stability information, so a table is not likely to be the best "final form" for this, as I think the accessibility will only get worse. So at some point we'll come up with a new layout all together, perhaps an actual section for each, with a bit more information included, and ditch the table all together.

In the meantime, maybe the @open-telemetry/docs-approvers might have some suggestions/ideas in case they've come across this before.

think for a first version we can just remove these columns for connectors. I suspect the best way of handling this is to have a 'highest stability' column (but that can be for a later PR)

Done, removed the columns for now

I feel like 'components' should be closer to the top of the list on the sidebar, I would expect to want to look at components when I am thinking about how to configure my Collector

I moved it to below the "Configuration" item, let me know if you want it elsewhere

[mx-psi]

So at some point we'll come up with a new layout all together, perhaps an actual section for each, with a bit more information included, and ditch the table all together.

In the meantime, maybe the @open-telemetry/docs-approvers might have some suggestions/ideas in case they've come across this before.

Alright, I'll leave it to the docs approvers to decide whether this is something that needs to be tackled now or can be left for later :)

I moved it to below the "Configuration" item, let me know if you want it elsewhere

I think that's the closest to the proposed refactor (https://www.mindomo.com/mindmap/c6ececd0512d46edb8f5048d19f18de1) where it would land inside a new "Configure the Collector" section. I guess this will be shuffled around further when the refactor happens, but for now I feel like that is the right place

[mx-psi]

I think this looks good for a first version, my only concern is the table visualization in small screens (which may be something we can leave for later)

[tiffany76]

Wow! I'm super impressed with these pages, especially as a first iteration. And I'm really excited about where we can take this.

@jaydeluca, the only addition I'd make is to link to https://opentelemetry.io/docs/collector/distributions/ in footnote # 1 of the autogenerated text. That link will change sometime soon, but if it's not too much trouble to update, let's add the current link.

The table scrolling issue will have to be discussed separately, so I don't think it should hold up this PR.

With apology, as I'm still catching up, has anyone weighed in on where exactly the watcher tool should live? Can we merge this as-is or do we need to move it out of @jaydeluca's repo? @svrnm @chalin

[jaydeluca]

has anyone weighed in on where exactly the watcher tool should live?

In my opinion, I think we should leave it where it is for now. After this is merged, I'm going to finalize the workflow to trigger it to kick off after each new collector release to automatically create PRs. I imagine I'm going to need to tweak and iterate that a bit. Once I have that in a good place, we can then move it to wherever makes sense. I'm happy to do whatever though

[vitorvasc]

Thanks for bringing this awesome work @jaydeluca, this is looking great! I'm really excited and looking forward to see what we can achieve in the next iterations!

Regarding the watcher and updates, I agree that we could trigger it whenever a release comes out.

For context, the current version update process runs hourly via auto-update-versions.yml. This workflow calls scripts/auto-update/all-versions.sh, iterates over each repository listed there and executes the scripts/auto-update/version-in-file.sh, which is the file where everything happens.

[chalin]

I've merged the latest from main. I would have done a rebase, but apparently there are merge conflicts. Hopefully, it'll all work out.

If all checks are green, then this LGTM as a first iteration. Thanks @jaydeluca 🙌🏻

[chalin]

All checks are green. @tiffany76 - so, are we waiting for a final approval from @open-telemetry/collector-approvers?

[tiffany76]

We've got @mx-psi's approval, which works for me. And @dmathieu's feedback was partially addressed in this first pass. I think it's ready. 🎉

Now we merge...and then we iterate!