That's interesting results, and a bit surprising. I tried training an NLI model on the T5 architecture, which I think is similar to a lot of Jev-style use cases. But I found the T5 encoder half was better than enc+dec at roughly equal parameters and still worse than ModernBERT/Ettin.
I also found Ettin to be better than ModernBERT, which is not that surprising if you look back at the published benchmarks in their paper.
Qwen3.5 0.8B was looking significantly worse than all of those, but I added LoRA adapters to 27B and that easily beat all of my encoders. So parameter for parameter, encoders were still the best for me, but with a ceiling at 1B for size. I'm planning to train more Qwen sizes and I guess I'll see at what point they can overtake the encoders I have.
That's interesting results, and a bit surprising. I tried training an NLI model on the T5 architecture, which I think is similar to a lot of Jev-style use cases. But I found the T5 encoder half was better than enc+dec at roughly equal parameters and still worse than ModernBERT/Ettin.
I also found Ettin to be better than ModernBERT, which is not that surprising if you look back at the published benchmarks in their paper.
Qwen3.5 0.8B was looking significantly worse than all of those, but I added LoRA adapters to 27B and that easily beat all of my encoders. So parameter for parameter, encoders were still the best for me, but with a ceiling at 1B for size. I'm planning to train more Qwen sizes and I guess I'll see at what point they can overtake the encoders I have.