Dbt Quick Patterns — Macros Part 2

Yes, Debug Mode Exists… If You Create It.

I spent a lot of effort crafting log messages and worrying about whether they should write to the console or just to file. Looking back, I’m a bit embarrassed. Why not simply use debug mode and control log messages at run time?

While dbt does have debug and log-level arguments, they don’t help with the log messages we write. We have no way to specify a level in the log() macro. In fact, the only configuration for log() is info=true or info=false. So what to do? What to do?

Now to the embarrassing part. We can simply use vars() or envars() to pass in the value for info. It took me years to figure this out. The macro below uses var(), but it would be just as easy to use envar() or both to add flexibility.

{% macro generate_columns() %}
    {% if not execute %}
        return('')
    {% endif %}

    {# Defaults to false #}
    {% set info = var('debug', False) %} 

    {# Contrived code to select columns from a table #}
    {% set metrics = [
        'total_sales',
        'yoy_sales',
        'ytd_sales'
    ]%} 

    {% set columns %}
        {% for metric in metrics %}
            {# Control whether we log to console or not #}
            {{ log('Generating ' + metric, info=info) }}
            {{ metric }} {% if not loop.last %},{% endif %}
        {% endfor%} 
    {% endset %}

    {{ return(columns) }}
{% endmacro %}

Notice the seventh line. “var(‘debug’, false)” By passing — vars ‘{debug: True}’ the log message will appear. This is great for development and finding out where your code is failing. Defaulting to false, keeps our production runs clean, without unnecessary log messages. In practice, we should create our own log() macros that ensure consistency, but this works just fine in a pinch.

Here’s two examples.

Without setting debug to true:

With setting debug to true:

And with a strange and beautiful video from the amazing Heilung.

This article is part of the Understanding Dbt for Data series. The short list of articles is growing, so be sure save the main thread to find what’s new.

Understanding dbt for Data Series