Agile for Data Warehousing
From AgileOpenNorthwest
This session was focused specifically on Data Warehouse projects within an Agile development environment.
A group of us agreed to get together at some as yet undetermined time and place to continue the discussion. If you are interested in joining in on the discussion and/or extending that to form an Agile Data Warehouse user group, please contact Lynne Meddaugh (lmeddaugh@gmail.com).
Some of the challenges discussed include:
- Slicing stories
- How to fit in analysis on source data and schema designs
- The balance between small vertical stories and the need for larger vision to create optimal warehouse design
Suggestions for vertical stories:
- Front-end only (BI report) - Gives feedback first and then build backend design as pieces of that.
- Stage data from source as is; no transformation. Slowly transform data as needed in subsequent sprints.
- Focus on one aggregate at a time
Instead of doing vertical cuts from source to BI, 2 suggestions were made for creating sprints:
- Data source 1 > ETL > Report/Visualization > Backfill. Follow up with Data source 2 (and beyond).
- Analyze source data > Design and create schema > Source to stage > Stage to dimensions > Fact load > Aggregates > Backfill
Questions:
- Can warehouse design really be cut vertically when it could take extensive backend work to create one report?
- How do you create stories around the much-needed source analysis and schema design?
- How do you analyze and design in small pieces when good warehouse design is so critical and so difficult to correct once implemented?
Methods of getting to smaller stories:
- Each story should ask a specific question (ie- how many repeat visitors do we get on our sites within a 30 day window?).
- If someone requests a report, ask how the report will be used.
Here are some links that may be relevant:
- Website: Agile Best Practices for Data Warehousing (DW)/Business Intelligence (BI) Projects
- Book: Agile Data Warehousing: Delivering World-Class Business Intelligence Systems Using Scrum and XP - also referenced in prior link
