As it is SQLBits 2024 this week, and me being a helper I’ve decided to release ‘Tapestry’ a metadata driven set of notebooks to process your data. It is not a complete process, at the moment it is just a Landing to Raw process, and at this time is a example of how to do these sort of pipelines.
In the Power BI & Data Platform user groups, I get asked questions in the pizza time, as I tend to be a ‘Subject Matter Wanna Be Know It All’, and pathologically helpful, and a few themes come up. One of which is handling multiple data processing pipelines, and I always ask ‘How generic can you make stuff?’. How can you quickly deploy, maintain and debug stuff. So I have created an example, which will be built on, but should give you an idea of some of the steps you can do in a generic way, and also add in those edge cases to keep life simple.

So here it is: Tapestry – https://github.com/mrjonlunn/tapestry a JSON driven set of notebooks: It is here and free to use/fork/update. One thing, let me know of bugs, and things that need adding. At the moment it is very basic and only does landing to raw, but can be updated. All the details are on the wiki page on how to set up and used it, as well as example data and JSON to try it out.