Debugging a script

Before you start

This guide assumes you have already launched a JupyterLab workspace, as described in the Transformations Overview.

Access prior job logs and data

If you're debugging why a specific job failed, you can learn more about what caused a job to fail by reading the logs.

To start, navigate to the job that failed (this could be under a specific tenant):

Download job dataDownload job data

Download job data

By clicking the download icon next to the job, a zip file will be generated containing the raw data that was was generated during that job. The directory structure is exactly the same as described on the transformation script docs.

By clicking the job name, you can access the logs and understand any errors that may have occurred:

Job logsJob logs

Job logs

Clone a job

While in a JupyterLab workspace, you can clone the data from a prior job. This is especially useful if you're debugging a specific job that failed, or you need some testing data to develop a script.

Inside of Jupyter, select the hotglue tab and press clone job.

Select clone jobSelect clone job

Select clone job

Jupyter will present you with a list of the most recent jobs. Press select on the one you would like to clone.

Choose which job to cloneChoose which job to clone

Choose which job to clone

Once the data is cloned, you'll see a success message as below, and the data will be populated in the sync-output folder.

Job clone successJob clone success

Job clone success

You can preview the data that was cloned in JupyterLab by opening the sync-output folder. As you can see, my subscribe_list data was cloned:

Preview cloned dataPreview cloned data

Preview cloned data

Test the script

Now that your data is cloned, you can run your transform script normally and it should pick up the data automatically.

Run entire transformation scriptRun entire transformation script

Run entire transformation script

If your script fails, you will see the error in the notebook (etl.ipynb). From there, you can change your script and run against the data again.

When your script works correctly, you should see the output files in the etl-output folder:

Transformation script outputTransformation script output

Transformation script output

That's all there is to testing your script in hotglue!


Did this page help you?