Build a Rough Order Magnitude Estimate (ROM)
If we feel comfortable moving forward under a fixed priced contract, it's time to do the second analysis of how, and how much. We will also do the first analysis of when. We use Excel.
First take a look at your set of features and implementations. It's time to define some milestones. Milestones are waypoints towards your end goal of delivering the product. A good milestone is measurable and defined by the outcome of your work. Write these down as eloquently as possible. Hopefully, you can describe 90 percent of the milestone in one sentence. A detailed description of the milestone is best described as a test plan written by QA. If a developer can test their work before it gets into the hands of QA, the less time you will need to iterate.
Milestone Definition: The first play experience of Level 1.
At each milestone, think about the steps you will need to take to reach that outcome. Look at your implementations and segment them in a way that puts risker items earlier in the development process. Why? Because if an implementation goes awry, you can tell your customers as soon as possible so that you can work together to resolve it. A lack of time is your biggest enemy when things go wrong.
Many project managers forget the following tasks. Set aside time for the following:
- Shims (scaffolding code required to stand-up an implementation or increase testability and efficiency)
- Refactoring before milestones build on new code foundations
- QA
- Reactions to QA (why find a bug if you aren't going to fix it)
- Usability
- Customer Feedback and correction time
Now that you have your milestones, write them down. Add in the implementation tasks. Use a hierarchy to help you organize work and decompose as necessary. A simplified snipped of a ROM might look like the following:
|
Milestone
|
Estimate
|
Confidence
|
| |
|
|
|
Milestone 2 -- User Kills a Simple Enemy
|
|
|
|
Art & Design
|
|
|
|
Build Level
|
32
|
9
|
|
Build Initial Player Character
|
80
|
8
|
|
Build Initial Weapon
|
40
|
8
|
|
Programming
|
|
|
|
Implement Player Damage
|
32
|
7
|
|
Implement Basic Enemy AI
|
16
|
6
|
|
Implement Enemy Damage
|
8
|
8
|
|
Implement Enemy Death
|
32
|
7
|
A snippet from a ROM. We prefer to use estimates in hours, but sometimes days makes more sense.
For each line, develop an estimate. But before you start, we need to think more carefully about what an estimate really is. At GarageGames, we believe an estimate is a time and risk estimate. Let's take a second to really understand how GarageGames determines risk adjusted estimates, it's critically important.
Note: When estimating, we always use work effort which is the total work required to complete the job. Duration measures how much calendar time it will take from start to finish. For example a 16 work effort hours can have a duration of 8 hours if two people are working on the task.
|
Risk Adjusted Estimates
|
|
Let examine two estimates: Task A is a non-risky job you've done before. Task B is a risky job you've done before. Assume for a second that I've asked the engineer to give me an estimate for each and in both cases, they say two weeks.
Beware: In this case, two weeks does not equal two weeks.
More often than not, undiscovered work will cause job B to take longer than expected. It's likely that two weeks REALLY means: "two weeks if everything goes the way I hope it will." It's probably more accurate to assume that the job will take two to four weeks. Since we don't really understand all ramifications of a risky task until we do it, we really can't be very accurate. Some people pad the estimate; we prefer something a bit more pragmatic, descriptive, and standardized.
With every estimate, GarageGames uses confidence to determine a risk adjusted estimate. So if we include confidence numbers, the two estimates might look more like this:
Task A: 2 Weeks, Confidence: 9
Task B: 2 Weeks, Confidence: 4
To calculate the risk adjusted estimate, GarageGames uses the following function:
Risk Adjusted Estimate = Estimate + (Estimate*(10-Confidence)/9)
Task A: Risk Adjusted Estimate: 2.2 Weeks
Task B: Risk Adjusted Estimate: 3.3 Weeks
Having the confidence value represented is useful for more than just determining risk adjusted estimates. As stated earlier, finding ways to reduce cost for a project is something that we openly embrace. Low confidence numbers are expensive and clients are motivated to help you increase confidence numbers to reduce cost. Before writing a single line of code, you can reduce risk the following ways:
Cutting a Feature -- "We didn't need it anyway... It's not that important."
Constraining a Feature -- "What if it did half of what we were hoping for."
Defining a Feature Better -- "Oh, I didn't understand what that meant... That's not so risky now."
Rapid Prototyping - Allowing time for rapid prototyping early in the project (and then updating the estimate ex post facto) --"Eureka!"
This is a great time for leads to share estimates with those who need to actually implement the task. It's hard to have developers who are vested in delivering a project if you didn't consult them initially. They are usually quite motivated to help. It's during this phase that implementers will ask you, "What does this task actually mean?" You will need to work with them and the customer to define what the implementation will or won't do. Save this definition, it will be important later.
|
Now that you have all the estimates, confidence numbers, and risk adjusted estimates, you should be able to provide your first estimate of when the project will be complete. Add up the risk adjusted estimates and divide it by full-time employee team size (two half-time employees equal one).
This isn't a very accurate estimate because it assumes a perfect world where everyone can do every job perfectly parallel -- hardly the case. But at least you will have some data to help you understand the very best case very high-level ball park figures. If someone wants a task done in 6 months and your estimate says you'll be done in seven months, you'll know early that something needs to change.
Same goes for the risk level. Average the confidence numbers. If the confidence numbers are too low (below, say, a 6), then maybe it's time to suggest either 1) moving to a time and material contract or 2) doing a small fixed bid contract up front to rapid prototype in an effort to bring up confidence numbers.
Keep in mind that while you are working on estimates, you will be defining the project or asking questions that will lead to the definition of the product. Write them down in small one or two page descriptions and save them. You will use them to develop the Statement of Work.
The final step is to share it with your customer. Show them where the risk is high and tell them why. Discuss opportunities to save money by reducing risk.
|
Printer-Friendly Version
However, a problem common with comparing agile and waterfall is the labeling. Few, if any, studios are purely "agile" or "waterfall". They are more mindsets that encompass a wide variety of practices and approaches to development. Labels are convenient for helping make an argument, often with cute little straw-man statements to help reinforce preconceived notions.
In reality, its more of a scale from pure iterative, inspect and adapt practices to assembly-line approaches that need little inspection. Studios have *always* been somewhere in the middle of this scale. Individual practices are employed based on uncertainty. To make the argument that "waterfall works" fails here. Sure, *some* waterfall practices work when we know more up front. You might say that you don't need any agile practices, but that's like saying you'll never need to inspect what your making and how your making it...not a state that any game developer has been in for very long, in my experience.
Anyways, I love the confidence scale. I'd love to try doing this on a project with some leads using planning poker! Thanks for writing this Eric.
Well said.
However, I do think that the popularity of Agile methods has somehow colored developers and diminished the discipline and acceptance of prior planning when possible. The pendulum has swung too far in the belief that all development require a reactive only project management process.
http://www.idinews.com/waterfall.html
The misinterpretation of cyclical approaches led to waterfall being executed on many projects since the 70's. I worked on a few of them myself in the late 80's and early 90's on projects that were run using the DOD-STD-2167A model which was waterfall, even if it wasn't called that. This model was used until ~1994.
Some of the projects I did work on are still in development, billions of dollars and almost a decade behind schedule (JSF). On this particular project, I just wrote documentation (40 pounds of it, all told)
His son Walker Royce, a well known IBM agile proponent will show up from time-to-time at agiel and software dev conferences to speak and mentioned that "his dad is innocent" as well.
As for Waterfall, what I think is interesting is how we (as the 'new industry'?) seem to want to invent new words all the time. I've found working in manufacturing the approach is just called "project management." Not as sexy a name perhaps, but when done fully and professionally its about about as waterfally as you can get, covering the scoping and quoting and resourcing and etc. questions quite well.
And when an "agile" approach is needed, a design phase is added. In other words, you don't commit all the resources until you really know what you want to make...whether it is some fun (in our case) or stylish (every bit as squishy, no?!) or ergonomic or what-have-you. Don't know what I mean? Check out: www.pmi.org/, especially: http://www.pmi.org/PMBOK-Guide-and-Standards.aspx
Blame the the person who use a method incorrectly or to religiously (Rigid).....
:)
Speaking from my experience, I've worked with both Agile and Waterfall with games programming teams between 4 and 13 programmers. Although we saw better engineered products with Agile, we ended up with better (more fun) games when we used a more Waterfall-based approach. Was that the methodology, the team or the game design? It's hard to say- these are organic products at the end of the day! Given the choice, I'd try to aim somewhere between the two and see if it is possible to balance them out.
We did risk assessment in a similar 1-10 scale, but found that without a strong PM on the project, they were difficult for clients to understand / deal with. Clients would still ignore them, turn tail later in the project and go ahead regardless!
In my case, I find Boehm Spiral Model to still be the 'best general fit' after all these years and frankly see no advantage of waterfall or agile over this approach 'in the general case'.
Speaking of actuals, I'm curious how accurate your risk modifiers were. Sometimes estimates can be too conservative and adding even more padding can unnecessarily increase your project cost estimate.
In my opinion it is best to be pragmatic about methodology and to not become wedded to a single approach.
@Eric: The Confidence Measurement is fantastic! I'm going to have to try that.
@Christopher: The Spiral Model is an interesting visualization of the development process. I hadn't encountered it before. Thanks for sharing that.
1. Does what we’re doing at this moment support the early and continuous delivery of valuable software?
2. Does our process welcome change and take advantage of change?
3. Does our process lead to and support the delivery of working software?
4. Are the developers and the product owner (stake holders) working together daily? Are customer and business stakeholders working closely with the project team?
5. Does our environment give the development team the support it needs to get the job done?
6. Are we communicating face-to-face more than through phone call and email?
7. Are we measuring progress by the amount of working software produced?
8. Can we maintain this pace indefinitely?
9. Do we support technical excellence and good design that allows for future changes?
10. Are we maximizing the amount of work not done- namely, doing as little as necessary to fulfill the goal of the project?
11. Is this development team self-organizing and self-managing? Does it have the freedom to succeed?
12. Are we reflecting at regular intervals and adjusting our behavior accordingly?
The big issue with Waterfall is change – the team/people, features, tech, etc., true Agile deals with this better, quicker, cheaper and allows you to kill what needs to be killed faster.
Agile, in my humble opinion will never work without a dedicated ScrumMaster and that’s not (always) a traditional Producer/PM – especially when there's Jr. people on the team.
Also ensure to schedule a very healthy polish (around 25-35%) phase at the end of the project to:
1) Reduce risk of over scoping your product by having a shorter production phase
2) Create some buffer if youre budget is reduced, your asked to bring your dates in or if something else goes wrong with staffing/hiring/planning, etc
3) If you avoid the above scenarios, you should be able to create a highly polished product
"If you can make every aspect of your product good, your result will be great".