Resources
Curated collection of papers, repositories, and communities for continued learning.
Foundational Papers
Architecture
- Attention Is All You Need (2017) - arxiv.org/abs/1706.03762
- The original Transformer paper
- BERT (2018) - arxiv.org/abs/1810.04805
- Bidirectional pre-training
- GPT-2 (2019) - openai.com/research/better-language-models
- Scaling demonstration
Scaling & Training
- Scaling Laws for Neural Language Models (2020) - arxiv.org/abs/2001.08361
- Kaplan scaling laws
- Training Compute-Optimal LLMs (2022) - arxiv.org/abs/2203.15556
- Chinchilla paper
- LLaMA (2023) - arxiv.org/abs/2302.13971
- Open foundation models
Alignment
- Training Language Models with Human Feedback (2022) - arxiv.org/abs/2203.02155
- InstructGPT / RLHF
- Constitutional AI (2022) - arxiv.org/abs/2212.08073
- Anthropic's approach
- DPO (2023) - arxiv.org/abs/2305.18290
- Direct Preference Optimization
Efficiency
- LoRA (2021) - arxiv.org/abs/2106.09685
- Low-Rank Adaptation
- Flash Attention (2022) - arxiv.org/abs/2205.14135
- Memory-efficient attention
- QLoRA (2023) - arxiv.org/abs/2305.14314
- Efficient fine-tuning
Code Repositories
Libraries
| Library | Purpose | Link |
|---|---|---|
| Transformers | Model hub & inference | github.com/huggingface/transformers |
| PEFT | Parameter-efficient fine-tuning | github.com/huggingface/peft |
| TRL | Training with RL | github.com/huggingface/trl |
| vLLM | High-throughput serving | github.com/vllm-project/vllm |
| LangChain | LLM applications | github.com/langchain-ai/langchain |
| LiteLLM | Multi-provider API | github.com/BerriAI/litellm |
Training
| Repository | Purpose |
|---|---|
| DeepSpeed | Distributed training |
| Accelerate | Easy distributed training |
| Axolotl | Fine-tuning framework |
| llama.cpp | CPU inference |
Model Hubs
- Hugging Face Hub: huggingface.co/models
- OpenAI API: platform.openai.com
- Anthropic API: console.anthropic.com
- Google AI Studio: ai.google.dev
Learning Resources
Courses
- Stanford CS224N: Natural Language Processing with Deep Learning
- Stanford CS324: Large Language Models
- fast.ai: Practical Deep Learning
Blogs
- Hugging Face Blog: huggingface.co/blog
- OpenAI Blog: openai.com/blog
- Anthropic Research: anthropic.com/research
- The Gradient: thegradient.pub
Newsletters
- The Batch (deeplearning.ai)
- AI Weekly
- Import AI
Communities
- r/MachineLearning: Reddit ML community
- r/LocalLLaMA: Open-source LLMs
- Hugging Face Discord: Model discussion
- EleutherAI Discord: Open research
- LangChain Discord: LLM applications
Benchmarks
| Benchmark | Type | Link |
|---|---|---|
| MMLU | Knowledge | paperswithcode.com/dataset/mmlu |
| HumanEval | Code | github.com/openai/human-eval |
| GSM8K | Math | github.com/openai/grade-school-math |
| HellaSwag | Commonsense | rowanzellers.com/hellaswag |
| TruthfulQA | Truthfulness | github.com/sylinrl/TruthfulQA |
Tools
Development
- Jupyter/Colab: Experimentation
- Weights & Biases: Experiment tracking
- MLflow: ML lifecycle
Deployment
- Modal: Serverless GPU
- Replicate: Model hosting
- RunPod: GPU cloud
- Together AI: Inference API
Keep Learning
The field moves fast! Follow:
- ArXiv CS.CL and CS.LG categories
- Twitter/X ML researchers
- Conference proceedings (NeurIPS, ICML, ACL, EMNLP)