I have a service that continuously emits data. You start receiving data once you have connected opening a TCP connection and never stops until you terminate the connection.
I'd like to develop a custom plugin to be able to process that data on Dataiku how I can do that as data never ends?
Will "build" log overload the server?
We are loading data from a flight's metasearch service. They expose a data stream we consume polling from a TCP connection (https://github.com/gbrian/Flightmate-Stream). We plan to use Dataiku to parse, sanitize, ... data and the drop into Hadoop apart from applying the corresponding analysis and lab ;)
@alexander Hope this helps