0 votes
Hi, I am working with R jupyter notebooks when preparing Flow steps. Unfortunately, due to size of data my notebook keeps crashing. Where can I find jupyter configuration and log files of the notebooks that are used in my workflows?
asked by
Hello, Is your notebook crashing because of a memory error? Or a display error? It could be that you are printing to much data to your browser, which causes your browser to shut down. A way to test it is: reproduce the crash and see if the other parts of DSS are responding well?
We would also like to have an instance diagnostic with the timestamp where the crash happened.
Hi! I only see "Dead kernel" information in notebook UI, therefore I would like to access logs to see what is the actual problem. The crash is probably caused by out of enough memory errors because the data processed is quite big (~1 GB csv file)

1 Answer

0 votes

Hi,

You can find the ipython log file in the DSS administration screen, under Maintenance > Log files > ipython.log. Alternatively, if you have access to the server by command line, the file is located in <DSS_DATA_DIR>/run/ipython.log.

Having said that, it is a bit surprising to get a memory crash on a 1GB csv file. Assuming you use pandas to load and transform it, a good rule of thumb is to have 5-10GB of free memory (see http://wesmckinney.com/blog/apache-arrow-pandas-internals/). Have you checked that you are not printing too much to the notebook output? [EDIT] Sorry, I had read too quickly and did not notice that you were using R. Could you check what is the object.size of the CSV file after it is loaded into an R object? 

Are you at liberty to share your code and underlying data?

Cheers,

Alex

answered by
edited by
Sure, tried for fun some forecasting analysis by rerunning the following code in dataiku R notebook https://www.kaggle.com/merckel/preliminary-investigation-holtwinters-arima/data.

I found that crashing part is the melt of forecast results (probably uniqe() is too much to handle here):

```
meltX <- melt(
  X[, which(names(X) %in% c(unique(keys$Date), "Page")), with = FALSE],
  measure.vars = unique(keys$Date),
  variable.name = "Date",
  value.name = "Visits")
meltX$Date <- as.character(meltX$Date)
```
Which package does the melt function come from?
742 questions
763 answers
612 comments
484 users