0 votes

Simple question / use case:

  1. I want to to set project level tokens for data connections to per project databases.
  2. I understand, that there exists projects variables  you can declare in json format.
  3. I don't know how the heck access those variables from within a python recipe.
  4. I need help, if this is something that's possible.

Disclaimer: I searched the docs up and down and only found this: https://doc.dataiku.com/dss/latest/advanced/variables_expansion.html#python but it doesn't work as described in the docs. I also searched the source code up and down, but couldn't make sense of it. I'm on 4.0.8

asked by

1 Answer

+1 vote
Hello,

We would like to understand better what you want to achieve on 1.

Do you want to have multiple projects with similar flows on the same connection, and avoid having users on different projects write to the same tables? If so, you can have a look at https://doc.dataiku.com/dss/latest/connecting/relocation.html. The simplest way would be to add a ${projectKey}_ prefix in the connection menu, for the table name (and Hive name too on HDFS).

Is that what you have in mind?

It is possible to call custom project variables from Python code, but there could be other ways to do it depending on your use case. As for things that do not work, you could please share the code snippets and the error message?

Cheers,

Alex
answered by
Ok, I should have elaborated on the context in the first place, sry...

I use Dataiku for ETL projects (ERP migration). I have developed a connector who does all the heavy lifting to the target instance. Now one project in dataiku is one migration project in real life.

Let's say I have tables A, B & C to upload to the target instance with receipte compute_A, compute_B & compute_C, then I only want to set the connection parameters (url, token, etc) once and that every recipe can access those instance parameters.

When everything is OK, I would just switch the staging parameters for production parameters in one single place.
I must have been very tired, when I wrote this post, because now, doing:
`dataiku.get_custom_variables()` results as expected... Wow!
Thanks anyway!
Great news! From the context of your project, it seems indeed that project variables should do the trick.

Note that if you want to create a complete industrialization workflow, you may be interested in the Automation node as well. In this case, we created a feature called Connection Remapping to remap connections from "dev" to "prod": https://doc.dataiku.com/dss/latest/bundles/importing.html#connection-remapping
Sorry not working for me in a "plugin":

self.customVariables = dataiku.get_custom_variables()
[2018/09/05-17:32:47.540] [KNL-python-single-command-kernel-err-42113] [INFO] [dku.utils]  -   File "/disk2/dataiku-dss-4.2.2/python/dataiku/__init__.py", line 77, in get_custom_variables
[2018/09/05-17:32:47.547] [KNL-python-single-command-kernel-err-42113] [INFO] [dku.utils]  -     "projectKey" : default_project_key() if project_key is None else project_key,
[2018/09/05-17:32:47.547] [KNL-python-single-command-kernel-err-42113] [INFO] [dku.utils]  -   File "/disk2/dataiku-dss-4.2.2/python/dataiku/__init__.py", line 66, in default_project_key
[2018/09/05-17:32:47.547] [KNL-python-single-command-kernel-err-42113] [INFO] [dku.utils]  -     raise Exception("Please specify for which project key you want datasets list")
This is working for me for a plugin recipe. What kind of plugin component are you developing? Can you give us more context regarding the goal of this plugin?
Thanks @Alex Combessie,

This is a custom "Dataset" component developed using the "Plugin developer" tools.
From within the python code I have:
self.customVariables = dataiku.get_custom_variables()

But it causes an error: "Please specify for which project key you want datasets list"
Could you retry with dataiku.get_custom_variables(project_key=dataiku.default_project_key()) instead?
No luck :(

 self.customVariables = dataiku.get_custom_variables(project_key=dataiku.default_project_key())
 File "/disk2/dataiku-dss-4.2.2/python/dataiku/__init__.py", line 66, in default_project_key
     raise Exception("Please specify for which project key you want datasets list")
 Exception: Please specify for which project key you want datasets list
948 questions
978 answers
989 comments
2,141 users

┬ęDataiku 2012-2018 - Privacy Policy