Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
I have several Redshift datasets that have already been built in a project, and was trying to deploy some very basic recipes. For instance, on one dataset - let's call it dataset X - I was simply trying to create a new dataset with only the first two columns of X using a Redshift connection.
When I go to deploy the script (there are roughly 1.5 million rows and 2 columns that I want to write out), we see some errors. Upon inspecting the log files, I see the following issue:
It appears that writing out 10,000 lines is taking on average 5 minutes, which seems like a very long amount of time. I have tried implementing this process both using a visual lab and python recipe, but the same issue occurs in both scenarios.
I ended moving the DBs from Redshift and saving them as new DBs using a Postgres connection, and then everything went very fast again. So, I feel as if this must be some issue with SQL and Redshift not playing well together, but I'm not sure exactly why. Any insight into this would be very helpful; thanks!