0 votes

I'd like to access SQL output generated by a recipe programatically using Python API client.

Is that possible?

UPDATE:

I see this information is returned by /dip/api/flow/recipes/generic/get-status

by
edited by

1 Answer

0 votes

Hi,

You find below a small code snippet from which you can inspire to apply to your use case:

# HELPER FUNCTION
CODE_RECIPES = ['sql_script', 'sql_query', 'impala', 'hive', 'spark_sql']
def get_recipe_sql(self, project_handle, recipe_name, recipe_type):
	recipe_handle = project_handle.get_recipe(recipe_name)
	if recipe_type in CODE_RECIPES:
		dap = recipe_handle.get_definition_and_payload()
		return dap.get_payload()
	else:
		status = recipe_handle.get_status()
		return status.data.get('sql')


# GO
client = dataiku.api_client()
project_handle = client.get_project('MY_PROJECT')
recipes = project_handle.list_recipes()

#loop through every recipe of the project and print out the SQL if possible
for recipe in recipes:
    recipe_name = recipe.get('name')
    recipe_type = recipe.get('type')
    try:
        sql = get_recipe_sql(project_handle, recipe_name, recipe_type)
    except Exception as e:
        message = "Could not retrieve sql code from recipe {} in project {}: {}"
        print(message.format(recipe_name, project, e))
        continue

 

Cheers,

by
So close!! Most of the recipes are returning the SQL, thanks.
Sadly the one at the screenshot is failing with:

{u'allMessagesForFrontend': {u'anyMessage': True,
  u'error': True,
  u'maxSeverity': u'ERROR',
  u'messages': [{u'code': u'ERR_RECIPE_VALIDATION_FAILED',
    u'details': u'Failed to compute recipe status: NullPointerException',
    u'message': u'Validation failed: Failed to compute recipe status: NullPointerException',
    u'severity': u'ERROR',
    u'title': u'Validation failed'}],
  u'success': False,
  u'warning': False},
 u'engines': [],
 u'output': {u'anyMessage': False,
  u'error': False,
  u'messages': [],
  u'success': False,
  u'warning': False},
 u'topLevelMessages': {u'anyMessage': True,
  u'error': True,
  u'maxSeverity': u'ERROR',
  u'messages': [{u'code': u'ERR_RECIPE_VALIDATION_FAILED',
    u'details': u'Failed to compute recipe status: NullPointerException',
    u'message': u'Validation failed: Failed to compute recipe status: NullPointerException',
    u'severity': u'ERROR',
    u'title': u'Validation failed'}],
  u'success': False,
  u'warning': False}}
1,296 questions
1,325 answers
1,505 comments
11,862 users

┬ęDataiku 2012-2018 - Privacy Policy