Overview
Over the years of being involved with numerous data analysis projects, I have established a structure that I use to design and plan for each new project. Through repetition, I have found that having a structure helps factor in considerations unique to each project. It ensures that . During each phase of my workflow, I try to document relevant information to ensure I don’t miss any details. Throughout each project I often have to update portions based on my work in each phase. A project should generally be documented from start to finish as part of creating an analysis that others can replicate. It also captures best practices over time that can be carried forward to subsequent projects.
As you gain more project experience, you should be getting better and efficient. Each successive iteration should also be a learning experience and improving your understanding of what you are doing.
Any type of analysis justifies documenting the steps you take throughout a project so others may be able to follow what you did and how you arrived at your conclusions. The examples provided throughout will demonstrate the steps taken on a real data set and will demonstrate a group project for AIT-580. Not every project necessarily needs to be this detailed, but if you follow some standard workflow for each project you do, your future projects will flow more naturally and it will help you follow the steps.
Outline
The outline of of my workflow is as follows. Each phase will have a link to the more in-depth explanation via post (if published).
- Planning
- Search
- Storage
- Preparation
- Transformation
- Exploration
- Analysis
- Data Visualizations
- Production of Analysis
- Decisions
Conclusion
No matter the project, you will generally follow the phases of the workflow, though there maybe some back and forth as you learn more about your data. There maybe instances where you may need to even go back to the planning and search phases. Regardless of your actual route from start to end, the goal by the end is to be able to answer your hypothesis or make/ influence a decision.
In your job, you may only be responsible for a phase or two, but its important to know if you are a consumer to others outputs or have consumers to your own outputs.