Agile software development is really a product development strategy. Scrum has its origins in product development. If you read the Agile Manifesto there are all sorts of hints that we are talking about developing software as a product that is valuable to some known customer. That is product development.
Re-engineering is the activity of changing (hopefully to improve) the structure and design of software. It is distinct from refactoring which is a specific technique. Refactoring can be used to re-engineer, but it can also be used to incrementally improve the design as we add new capabilities. Re-engineering does not have to follow the contract of refactoring (small incremental design changes that preserve observable behavior.) In fact, re-engineering is often done to improve a system by changing its behavior, but not necessarily its user functionality.
If the last sentence in the preceding paragraph doesn’t scare you then I have not yet done the job I intend to do.
Re-engineering is not product development, because it is not primarily concerned with creating useful functionality for customers. Re-engineering can be customer driven, but it tends to be somewhat indirectly customer driven. It tends to be concerned with things that are traditionally considered architectural properties: performance, scalability, security, quality, usability, consistency, etc. The customer might be saying, “This app does what I need, but it is too slow.” Something like that may drive us to re-engineer.
Re-engineering is highly problematic from an Agile point of view. We are not producing functionality. We are not working from user facing stories. There may be business value to what we are doing (such as improving the performance of our application) but it is not directly measurable business value in most cases.
Re-engineering is also too often driven by technical concerns within the team (or its management). We didn’t build it as well as we wish we had. We have learned new things now, and we want to incorporate them.
Learning is the key. Agile is all about learning. The feedback mechanisms in Agile processes are designed to help us learn as we go. The intent is that we will apply this learning immediately rather than waiting until some future (“post mortem”) time to re-evaluate.
For a functional Agile team these learning mechanisms exist at multiple (somewhat redundant) levels: TDD has tiny learning cycles that are as small as a few seconds to a few minutes. Each time a test passes or fails we have learned something. Continuous Integration is a learning cycle (Especially if we have an Informative Build.) When the build passes or fails we are meant to learn something. Stories, in the form of Card-Conversation-Confirmation are learning cycles. Releases, and the consequent customer feedback, are learning cycles. Each time we do these things we learn, and each time we learn we are meant to immediately attempt to improve.
Re-engineering is a poor business proposition. We are spending additional money on a product that is already functional. The return on that investment is hard to anticipate. We can tell the customers that we have improved the product in some ways that they may or may not be able to appreciate (or even notice.) Therefore, the risk is very high, and the best case outcome is the the product does a slightly better job doing what it already does.
Agile is not compatible with re-engineering. Agile is an alternative to re-engineering. Agile expects us to refactor our design and to incorporate our changing understanding (i.e. learning) as we develop our product, not after the fact.