I do not know the exact storage details and configurations, as our instance of DSS is administered by one team, and the file storage systems by another. We have set up a few test cases to monitor and try to reproduce the behavior.
I am certain we do not have any scenarios clearing the data.
I am not sure if this could be related, but we are also running into issues (in the same project) where datasets, after being built, will randomly give a root path error and need to be rebuilt. The timing on this is hard to nail down (as I was not until this week regularly checking folders and datasets to see if they still existed after upload/build), but I do know it has occurred at intervals between 30 minutes, and weeks (super helpful, range, I know!).
I'm mostly trying to figure out if this is a data system issue, a DSS issue, or some combination of the two.
A little background: I had a whole series of notebooks in which I was developing python code. Once all the code worked (reading datasets, doing the stuff, writing output datasets), I turned the notebooks into python recipes. I discovered the folder deletion behavior when my flow failed. Since then, the folders have purged multiple times and I am also running into the root paths issue randomly on other datasets.