Ryan Yixiang Wang

@RyanYixiang

Phding at @berkeleynlp with @sewon__min | prev @nlp_usc

Berkeley, CA

ryanyxw.github.io

Joined June 2017

587 Following

432 Followers

Replies Reposts Media

Pinned
Ryan Yixiang Wang
@RyanYixiang
May 8
MoEs are everywhere in frontier models, and they are deployed as a monolith system. But many applications only need a narrow slice of capabilities, e.g., math, code, biomedical, etc. So what if "modularity" is actually the missing opportunity for MoEs? Today, we're releasing EMO: an end-to-end pretrained MoE where modularity emerges naturally, enabling selective use of experts!
Ai2
@allen_ai
May 8
Today we’re releasing EMO, a new mixture-of-experts (MoE) model trained so modular structure emerges directly from data without human-defined priors. EMO can use a small subset of its experts for a given task while keeping near full-model performance. 🧵
8
Ryan Yixiang Wang
@RyanYixiang
Nov 5
Almost all model behavior can be traced back to its training data. Unexpected LLM behaviors are often just hard to notice patterns in the data! Human inspection just can’t catch this sort of stuff 🥸
Yaowen Ye
@yaowenye123
Nov 5
To build a system that can predict arbitrary side effects of post-training, we first need a proxy evaluation set of examples. This is a first step toward that. Really enjoyed working on it with the amazing folks at @TransluceAI!
Quote
Transluce
@TransluceAI
Nov 5
We built a pipeline that discovers thousands of datasets with surprising side effects (DSSEs): datasets that read like ordinary post-training data yet induce unexpected model behaviors (e.g. recommends self-harm) after finetuning. 🧵
Ryan Yixiang Wang
@RyanYixiang
Sep 28
When I use coding agents these days, I often get the feeling that some questions are inherently harder than others. I'd be comfortable asking an agent, “Is this README still up to date?”, but much less comfortable asking something broad like, “Are there any remaining bugs in this
Prasann Singhal
@prasann_singhal
Sep 28
Could long-context architectures “find all contradictions” in a science literature? Not yet! 🧵 We study a new class of "high-complexity” tasks whose difficulty scales quadratically with corpus size (as opposed to linearly), reversing common LCLM decisions! (block-sparse
3
Ryan Yixiang Wang
@RyanYixiang
Aug 11, 2024
Presenting our work at ACL on using data watermarks for detecting if an LM used your data during pretraining (with statistical guarantees)! Come find me and @johntzwei Monday 5:45 pm! Also happy to chat about how hot Bangkok is, or anything LM pre-training/memorization, etc
Ryan Yixiang Wang
@RyanYixiang
Feb 19, 2024
We could detect SHA hashes that only occurred in BLOOM-176B pre-training data 90 times! For reference BLOOM has a training corpus of 341B tokens 🫣🥸
Ryan Yixiang Wang
@RyanYixiang
Feb 19, 2024
We could detect SHA hashes that only occurred in BLOOM-176B pre-training data 90 times! For reference BLOOM has a training corpus of 341B tokens 🫣🥸
Johnny Tian-Zheng Wei
@johntzwei
Feb 19, 2024
To detect if your data was used for LLM pretraining, consider using data watermarks: arxiv.org/pdf/2402.10892… Detection can be framed as hypothesis testing (statistical guarantees!), if you contributed multiple training documents and watermarked them before public release. 🧵

Ryan Yixiang Wang

Ryan Yixiang Wang’s posts