compare html -> text methods

[percyliang]

Compare:

1. trafilatura (setup trafilatura as transform operation in pipeline #245 )
2. resiliparse (Add support for resiliparse for content extraction #250 )
3. readability + markdownify (already implemented)

WandB results: https://wandb.ai/marin-community/marin/reports/246-Web-Extraction-Method-Comparison--Vmlldzo5OTg4MTAw?accessToken=8t7elz382va7ftc1uttuo43ccpej2yg220y33y6i9ek7q81q5xfmp9vzcw7av6ih

[percyliang]

Next steps: run these methods on hello_world_fw on Ray cluster and then:

1. Analyze the results to see if there is anything that's off.
2. Train 1.4B models for 144B tokens and eval.

[percyliang]

Add flags to the transform scripts and make it comparable to the DCLM and FineWeb papers to reproduce, and then ablate.

[krypticmouse]

A qualitative review of methods is completed and seems like trafilatura works the best in terms of the text extracted. Up next, we need a quantitative review:

1. Creating a stable subset of fw (use last dump, which we need to compute update CommonCrawl HTML extraction script #311 )
2. Process and store the subset over the 3 extraction methods
3. Train 1B model on 280B tokens for each of the 3 extraction methods
4. Evaluate the 3 models and compare them.

[krypticmouse]

Data Browser: Link to data browser on 5 data config settings

Config:

• Readabilty + Markdownify
• Resiliparse [DEFAULT]
• Resiliparse with preserve_formatting
• Trafilatura with favor_recall [DEFAULT]
• Trafilatura with favor_precision

Qualitative Insights:

• resiliparse with preserve formatting is mostly the same in content terms but I like the formatting and noisy tag removal more, so this setting is better
• traf in our configuration is better but mostly it doesn't matter because output for both favoring recall and precision is same but when it's different I prefer recall one which is our default setting
• overall both traf in our default setting and resiliparse with preserve formatting seems similar, but resiliparse is faster
• One thing to note is that sometimes resiliparse just removes out the content content completely, it's mainly because of table format issues, so maybe we can have trafilatura as fall back in such cases.

[krypticmouse]

WandB Report: https://wandb.ai/stanford-mercury/marin/reports/246-Web-Extraction-Method-Comparison--Vmlldzo5OTg4MTAw
Data Browser Link: https://marlin-subtle-barnacle.ngrok-free.app/experiment?path=gs%3A//marin-us-central2/experiments/246_web_extraction_method_training-efe0cf.json

[krypticmouse]

Evaluation Report

Google Sheet with precise numbers: https://docs.google.com/spreadsheets/d/1jdS8lzF5m0hm__G1AhJ_2LZwJxvKEQKR-fLt62seaiI/edit?usp=sharing