VLM != LLM. Vision language models basically treat text tokens and image tokens the same. Post-training an LLM on images+text can improve its capabilities. Id recommend searching around the keyword VLM to find more resources on how multi-modal AI works.
transformers were first an image understanding technique, the text processing came later, it's all about the training data and gradient descent, and allegedly attention
VLM != LLM. Vision language models basically treat text tokens and image tokens the same. Post-training an LLM on images+text can improve its capabilities. Id recommend searching around the keyword VLM to find more resources on how multi-modal AI works.
- https://huggingface.co/blog/vlms
- https://en.wikipedia.org/wiki/Multimodal_learning
transformers were first an image understanding technique, the text processing came later, it's all about the training data and gradient descent, and allegedly attention
[dead]