The Retrospective is designed to provide a formal point for inspection and adaption by the whole Scrum Team with actions from lessons learned incorporated directly into the next Sprint. It should be a safe environment in which the whole Scrum Team can share failures and learn from them.
There’s always talk of whether the Product Owner should be involved, but so long as any hierarchical relationship with the Development Team doesn’t adversely impact the need to be honest, open and transparent, and learn what change is required of the Team to improve their performance in the coming Sprint, it’s important for the whole Scrum Team to use this key event to inspect their way of working and adapt their way of planning for the next Sprint.
Why do traditional, waterfall style projects fail? Some claim its a requirements problem and point to the need for more planning, user research, and design. The truth is we’re not looking at the problem the right way and complex environments require a different way to collecting information and delivering using that knowledge.