I've been working on a framework to handle the messiness of high-frequency telemetry. The biggest issue I’ve seen is that pipelines usually break the second a sensor tag changes or a schema drifts. I wanted to build something that doesn't need a human to fix it every time a data source fluctuates.
I just hit a milestone where the BERT-based mapping layer is running at sub 20ms latency with 85%+ accuracy. Since I'm the only one building this, the goal was to keep it lean enough for real-time sports telemetry while still being robust enough to resolve semantic mapping issues on the fly.
Technical highlights:
Schema Drift: Instead of rigid mapping, it uses a transformer agent to semantically link incoming data to the target schema.
Speed: Sub 20ms inference means it can actually keep up with live streams like F1 telemetry.
Solo Project: It's been a solo build as part of my PhD work and just passed 2,000 clones on GitHub.
There is a TUI demo in the repo that runs a "chaos stream" to show how it re-maps data when the inputs change. I'd love to get some feedback on the performance or the architecture.
I've been working on a framework to handle the messiness of high-frequency telemetry. The biggest issue I’ve seen is that pipelines usually break the second a sensor tag changes or a schema drifts. I wanted to build something that doesn't need a human to fix it every time a data source fluctuates.
I just hit a milestone where the BERT-based mapping layer is running at sub 20ms latency with 85%+ accuracy. Since I'm the only one building this, the goal was to keep it lean enough for real-time sports telemetry while still being robust enough to resolve semantic mapping issues on the fly.
Technical highlights:
Schema Drift: Instead of rigid mapping, it uses a transformer agent to semantically link incoming data to the target schema.
Speed: Sub 20ms inference means it can actually keep up with live streams like F1 telemetry.
Solo Project: It's been a solo build as part of my PhD work and just passed 2,000 clones on GitHub.
There is a TUI demo in the repo that runs a "chaos stream" to show how it re-maps data when the inputs change. I'd love to get some feedback on the performance or the architecture.
Repo: https://github.com/tarek-clarke/resilient-rap-framework