What is The Sum of All Knowledge?

⭐ Is there enough time to count up all the stars?
📚 Is there enough time to read the sum of all knowledge?

🖥️ In servers humming through the endless night,
🌐 Where 14 TB of human thought reside,
📖 C4's multilingual web holds digital light,
📊 While Anna's Archive spans 170 GB wide.

💾 The Pile sits heavy at 825 GB,
🧬 Its gigabytes of wisdom, code, and art,
💻 From PubMed's 194 GB alive
🔢 To GitHub's 102 GB, each plays its part.

📡 We measure meaning now in bytes and bits,
🧠 Wikipedia's 24 GB, a modest sum,
⚖️ While massive datasets store what knowledge permits—
👁️ The fragments of all that we've become.

🔬 From ArXiv's 121 GB academic dreams
📚 To OpenWebText's 135 GB refined,
🤖 Each dataset streams through digital streams,
❓ The quantified collective human mind.

⏰ The paradox grows clear with every byte:
🌌 The more we store, the more we comprehend
🎨 How much remains beyond our finite sight,
❤️ How questions birth new questions without end.

💿 So we collect in terabytes our past,
🚪 Code algorithms, digitize our dreams—
🔍 Though time may not allow us to amass
✨ The sum of all knowledge, or so it seems.

🌟 What drives us is infinite curiosity:
🗝️ Being is becoming, dream is destiny, 
💖 The boundless library of the imagination,
🌈 Where every ending opens up a new start.

Data is the most valuable asset in LLM development. When building a dataset, we target the three following characteristics:

• Accuracy: Samples should be factually correct and relevant to their corresponding instructions. This can involve using solvers for math and unit tests for code.
• Diversity: You want to cover as many use cases as possible to make sure you're never out of distribution. High diversity is essential as it leads to better generalization.
• Complexity: Answers should be both detailed (to maximize helpfulness) and include system 2 techniques like chain of thought (to force step-by-step reasoning).

Measuring accuracy is easy in most cases but near-impossible with open-ended, subjective questions. On the other hand, clustering datasets by topic is a good way of evaluating data mixture diversity. Finally, complexity can be assessed using other LLMs acting like judges.

Text Datasets for Language Model Training

📅 Open SFT datasets

Once a model has been pre-trained on a next-token prediction task, Supervised Fine-Tuning (SFT) is used to turn it into an assistant capable of answering questions and following instructions. These datasets contain pairs of instructions and outputs to train LLMs to understand conversational structure. Unless otherwise noted, all datasets listed here are under permissive licenses (Apache 2.0, MIT, CC-BY-4.0, etc.).

General-purpose mixtures

General-purpose datasets offer balanced mixtures of different types of data, including chat, code, and math. These datasets can be used to create general-purpose models that can handle various types of queries.

Math

LLMs often struggle with mathematical reasoning and formal logic, which has led to the creation of specialized datasets. These datasets can include systematic thinking and step-by-step reasoning.

Code

Code is another challenging domain for LLMs. Code datasets, containing diverse programming language examples, are used to fine-tune LLMs and enhance their ability to understand, generate, and analyze code.

Instruction following

Instruction following corresponds to the ability to properly follow constraints in the user prompt, such as "write only two paragraphs", "write your answer in French", etc. Strong instruction-following capabilities is a must-have for modern LLMs.

Multilingual

Learning new languages "from scratch" is a pre-training task, but providing multilingual instruction samples is useful to boost performance in the languages of interest.

Agent & Function calling

Function calling allows large language models (LLMs) to execute predefined functions with parameters inferred from user prompts, rather than generating standard text responses. This enables LLMs to seamlessly integrate with external systems, perform complex operations, and provide more accurate and contextually relevant responses.

Real conversations

Real-world conversations provide valuable insights into how people naturally interact with LLMs, helping us identify the most important use cases and understand typical usage patterns.

⚖️ Preference alignment

Unlike instruction data, preference datasets consist of chosen and rejected answers. Preference alignment is used to align LLM's answers with human preferences to adopt the desired style and values.

Image Datasets for Visual Model Training

The Pile Component Breakdown (825.18 GB Total)

Largest Components (>100 GB)

Medium Components (50-100 GB)

Smaller Components (10-50 GB)