Dòng tin
Tất cả
Lighthouse Attention: Sửa đổi attention chi phí thấp cho huấn luyện hiệu quả
Interesting paper. What I like about this is that it is a relatively low-commitment attention modification.
I.e., one can use it during most of training, switch back to vanilla attention near the end, and recover roughly the same modeling performance as if full attention had been used the whole time.
- ›Lighthouse Attention là phương pháp bao bọc attention tiêu chuẩn với một lớp nén subquadratic, giảm chi phí tính toán.