How to (if so) access Project Level Variables from (custom) python recipe?

UserBird
Dataiker
How to (if so) access Project Level Variables from (custom) python recipe?

Simple question / use case:




  1. I want to to set project level tokens for data connections to per project databases.

  2. I understand, that there exists projects variables  you can declare in json format.

  3. I don't know how the heck access those variables from within a python recipe.

  4. I need help, if this is something that's possible.



Disclaimer: I searched the docs up and down and only found this: https://doc.dataiku.com/dss/latest/advanced/variables_expansion.html#python but it doesn't work as described in the docs. I also searched the source code up and down, but couldn't make sense of it. I'm on 4.0.8

0 Kudos
9 Replies
Alex_Combessie
Dataiker Alumni
Hello,

We would like to understand better what you want to achieve on 1.

Do you want to have multiple projects with similar flows on the same connection, and avoid having users on different projects write to the same tables? If so, you can have a look at https://doc.dataiku.com/dss/latest/connecting/relocation.html. The simplest way would be to add a ${projectKey}_ prefix in the connection menu, for the table name (and Hive name too on HDFS).

Is that what you have in mind?

It is possible to call custom project variables from Python code, but there could be other ways to do it depending on your use case. As for things that do not work, you could please share the code snippets and the error message?

Cheers,

Alex
UserBird
Dataiker
Author
Ok, I should have elaborated on the context in the first place, sry...

I use Dataiku for ETL projects (ERP migration). I have developed a connector who does all the heavy lifting to the target instance. Now one project in dataiku is one migration project in real life.

Let's say I have tables A, B & C to upload to the target instance with receipte compute_A, compute_B & compute_C, then I only want to set the connection parameters (url, token, etc) once and that every recipe can access those instance parameters.

When everything is OK, I would just switch the staging parameters for production parameters in one single place.
0 Kudos
UserBird
Dataiker
Author
I must have been very tired, when I wrote this post, because now, doing:
`dataiku.get_custom_variables()` results as expected... Wow!
Thanks anyway!
0 Kudos
Alex_Combessie
Dataiker Alumni
Great news! From the context of your project, it seems indeed that project variables should do the trick.

Note that if you want to create a complete industrialization workflow, you may be interested in the Automation node as well. In this case, we created a feature called Connection Remapping to remap connections from "dev" to "prod": https://doc.dataiku.com/dss/latest/bundles/importing.html#connection-remapping
0 Kudos
Gustavo_Brian
Level 2
Sorry not working for me in a "plugin":

self.customVariables = dataiku.get_custom_variables()
[2018/09/05-17:32:47.540] [KNL-python-single-command-kernel-err-42113] [INFO] [dku.utils] - File "/disk2/dataiku-dss-4.2.2/python/dataiku/__init__.py", line 77, in get_custom_variables
[2018/09/05-17:32:47.547] [KNL-python-single-command-kernel-err-42113] [INFO] [dku.utils] - "projectKey" : default_project_key() if project_key is None else project_key,
[2018/09/05-17:32:47.547] [KNL-python-single-command-kernel-err-42113] [INFO] [dku.utils] - File "/disk2/dataiku-dss-4.2.2/python/dataiku/__init__.py", line 66, in default_project_key
[2018/09/05-17:32:47.547] [KNL-python-single-command-kernel-err-42113] [INFO] [dku.utils] - raise Exception("Please specify for which project key you want datasets list")
0 Kudos
Alex_Combessie
Dataiker Alumni
This is working for me for a plugin recipe. What kind of plugin component are you developing? Can you give us more context regarding the goal of this plugin?
0 Kudos
Gustavo_Brian
Level 2
Thanks @Alex Combessie,

This is a custom "Dataset" component developed using the "Plugin developer" tools.
From within the python code I have:
self.customVariables = dataiku.get_custom_variables()

But it causes an error: "Please specify for which project key you want datasets list"
0 Kudos
Alex_Combessie
Dataiker Alumni
Could you retry with dataiku.get_custom_variables(project_key=dataiku.default_project_key()) instead?
0 Kudos
Gustavo_Brian
Level 2
No luck ๐Ÿ˜ž

self.customVariables = dataiku.get_custom_variables(project_key=dataiku.default_project_key())
File "/disk2/dataiku-dss-4.2.2/python/dataiku/__init__.py", line 66, in default_project_key
raise Exception("Please specify for which project key you want datasets list")
Exception: Please specify for which project key you want datasets list
0 Kudos

Labels

?
Labels (2)
A banner prompting to get Dataiku