Kelly Waters gives 3 scenarios where Agile Software Development does not work. One factor that is often ignored while diving into agile is the team structure.
Companies like to have hierarchical team structures, where few good ones can get work done with many average ones. Economically too, they think, the combination of few higher salaries and many average salaries is beneficial to them. In such cases, usually few do the design, write the pseudo code and the rest work mostly as translators. However, I have seen that flatter team structures, where every individual has the skill and the authority to take most of the technical decisions work better. This helps in faster response to changes and helps in keeping the team small and agile.