Monday in South Lake

Wonderful time up here in the thin air with family and friends, headed back home tomorrow.

Starship launch was scrubbed, moving the next window back to Wednesday.

LLMs

What I’m reading today re LLM’s:

• RedPajama is a project to create open-source base models (like llama) from the ground up. They’ve released a 1.2T large training dataset that’ll serve as the foundation for an open vicunia-like chat model.
• Minigpt-4 looks to Enhance Vision-language Understanding with Advanced Large Language Models and combines Vicunica with a vision recognition model using a small, easily trained linear layer. So, the “GPT-4” aspect is the vision-language combination, and the SOTA is that this model can be run on commodity GPUs (3090)
• Google web-comic on federated learning. Reads like propaganda, but brought up the concept of secure aggregation, where training data can be collected from end-user devices without exposing personal user data. Particularly relevant when trying to encourage collaboration between entities with silos of private data.