0 votes

Simple question / use case:

  1. I want to to set project level tokens for data connections to per project databases.
  2. I understand, that there exists projects variables  you can declare in json format.
  3. I don't know how the heck access those variables from within a python recipe.
  4. I need help, if this is something that's possible.

Disclaimer: I searched the docs up and down and only found this: https://doc.dataiku.com/dss/latest/advanced/variables_expansion.html#python but it doesn't work as described in the docs. I also searched the source code up and down, but couldn't make sense of it. I'm on 4.0.8

asked by

1 Answer

+1 vote

We would like to understand better what you want to achieve on 1.

Do you want to have multiple projects with similar flows on the same connection, and avoid having users on different projects write to the same tables? If so, you can have a look at https://doc.dataiku.com/dss/latest/connecting/relocation.html. The simplest way would be to add a ${projectKey}_ prefix in the connection menu, for the table name (and Hive name too on HDFS).

Is that what you have in mind?

It is possible to call custom project variables from Python code, but there could be other ways to do it depending on your use case. As for things that do not work, you could please share the code snippets and the error message?


answered by
Ok, I should have elaborated on the context in the first place, sry...

I use Dataiku for ETL projects (ERP migration). I have developed a connector who does all the heavy lifting to the target instance. Now one project in dataiku is one migration project in real life.

Let's say I have tables A, B & C to upload to the target instance with receipte compute_A, compute_B & compute_C, then I only want to set the connection parameters (url, token, etc) once and that every recipe can access those instance parameters.

When everything is OK, I would just switch the staging parameters for production parameters in one single place.
I must have been very tired, when I wrote this post, because now, doing:
`dataiku.get_custom_variables()` results as expected... Wow!
Thanks anyway!
Great news! From the context of your project, it seems indeed that project variables should do the trick.

Note that if you want to create a complete industrialization workflow, you may be interested in the Automation node as well. In this case, we created a feature called Connection Remapping to remap connections from "dev" to "prod": https://doc.dataiku.com/dss/latest/bundles/importing.html#connection-remapping
893 questions
923 answers
1,430 users