Research
Prompt Trees: Training-time Prefix Caching
We present a method for encoding tree-structured data (like you might get from conversation rollouts while doing RL training for conversational agents) using a standard transformer. This method gives us training speedups of up to 70x over standard training.