Ways to limit job log size

Martin_D
Level 1
Ways to limit job log size
Hi, is there a way to limit the log size when running a job? Sometimes we run a job and that fails. When this happens all temp data and logging is stored which leads to disk space issues. In no time logging of over 100GB is created.
0 Kudos
4 Replies
Alex_Combessie
Dataiker Alumni
Hi Martin,

You can perform such scheduled operations through the use of macros: https://doc.dataiku.com/dss/latest/operations/macros.html. As a matter of fact, we provide a built-in macro to clear old job logs. This macro can be run manually or scheduled through a scenario (if you have production capabilities activated in your license).

Note that this clears existing logs, so you may want to store them elsewhere for audit or compliance reasons.

Hope it helps,

Alex
0 Kudos
cwentz
Level 3

@Alex_Combessie Where is the macro that clears job logs?

0 Kudos
Andrey
Dataiker Alumni

Hi @cwentz , 

 

This macro can be found if you navigate to Project -> "..." -> Macros using the top menu bar.

Regards

Andrey Avtomonov
R&D Engineer @ Dataiku
cwentz
Level 3

TY Andrey!!

0 Kudos

Labels

?
Labels (2)
A banner prompting to get Dataiku