ideas_links_findings.md

Ideas, Links and Findings

Multilingual Training

• Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning
  • LinkedIn
  • arXiv

Multimodal Capabilities

• OpenFlamingo: Twitter, Blog

Data Labeling and Management

• Argilla
• Good German text embedding model: sentence-transformers/paraphrase-multilingual-mpnet-base-v2

Translation

• Helsinki-NLP/opus-mt-en-de
• fairseq

en2de = torch.hub.load('pytorch/fairseq', 'transformer.wmt19.en-de',
                       checkpoint_file='model1.pt:model2.pt:model3.pt:model4.pt',
                       tokenizer='moses', bpe='fastbpe')
en2de.eval()  # disable dropout

Other Tools

• PromptSource: GitHub, arXiv

Other related Organizations and Initiatives

• LAION: Website, Discord, GitHub
• BigScience: Website, Blog
• openGPT-X

Datasets

• https://github.com/databrickslabs/dolly/tree/master/data
• https://huggingface.co/datasets/argilla/databricks-dolly-15k-multilingual