Log:

• We met 5:30 - 9:30 PM and attempted to train a sparse autoencoder using Logan’s code.
• We ran into some obstacles loading data.
  • Insight: Pythia 70M chess needs chess data!
  • Insight: load small datasets on to Colab, not a billion tokens!
  • Insight: just ask Logan :)

Todos:

• Naomi might try to make the sparse autoencoders train tonight.
  • Upload to HuggingFace
  • Look at the autoencoder using Neel Nanda’s library
• Try to fine tune the sparse autoencoder on a dataset heavy on one feature. This task will help us practice fine tuning before we move on to RLHF. We’ll start on Friday.