How can I get things right at the beginning of a software project?

Question

I am programmer with 1 year experience, recently I realized I seldom start a project correctly (most of my side project), normally the project cycle goes like

Start with a few use-cases
Start coding
Realize a few things I did not handle well, and does not fit well in current codebase.
Rewrite most part of code

and this might go a few times

So my questions are

Is such practice common, or it implies I am not competent?
How can I improve myself on this aspect?

Perfect! That's called learning :) And competent != efficient on day 1 — , Jul 06 '16 at 07:33
Your interesting question is off-topic, since it looks like career advice. BTW I would also suggest to contribute to some existing free software project, you'll learn a lot (from a community of developers, some of them being more expert than you are today) — Basile Starynkevitch, Jul 06 '16 at 07:45
I see a gaping chasm there between "use cases" and "start coding". Whatever happened to the analysis and design parts of the SDLC? — Bradley Thomas, Jul 06 '16 at 13:17
If you find a method to start every project perfectly, let us know. Or write a book about it and become millionaire. — Mast, Jul 06 '16 at 14:28
@BradThomas, sorry, I left it out in this post, I believe defining use cases is sort of analysis? I do think about things like what abstraction I am going to have, and how do I wire them up before or at the same time I am coding. So does it mean I should spent more time on them? — Qingwei, Jul 06 '16 at 14:42
@Qingwei, you said start with use cases, not defining them. Defining them would be a kind of analysis, ie. of user requirements. I think it's better to have a more thorough, detailed understanding of most of the use cases early on. I mean finding just one new use case at a later stage, can often mean substantial re-work. Better to do the re-work on the design than on the implementation. — Bradley Thomas, Jul 06 '16 at 15:27
I guess it depends on who you talk to but IMO Use-Cases are purely requirements. They should be written completely devoid of any design decisions. Analysis mainly includes designing/defining the system architecture. Thus, Use-Cases don't belong as part of the Analysis Phase. With that said, what usually happens is you write some use-case details, update your architecture diagrams and iterate back to the Use-Cases to make changes you identified while doing the architecture diagrams and update the diagrams based on changes to the use-cases. Then you keep iterating until you are happy with both. — Dunk, Jul 06 '16 at 15:42
Figuring out which use cases to pursue, drop or adapt is an analytical process, that needs to take into account (among other things) the current state of development. — Bradley Thomas, Jul 06 '16 at 18:27
Therefore you can't sensibly just start with a few use cases, and then start coding, unless you are purely a code monkey and offers management no value add beyond your ability to implement whatever they ask for, no matter how questionable. — Bradley Thomas, Jul 06 '16 at 19:09

anaximander · Accepted Answer · 2016-07-06T11:48:24.357

The cycle you describe is normal. The way to improve things is not to avoid this cycle, but to streamline it. The first step is to accept that:

It's near impossible to know everything on day one of a project.
Even if you do somehow know everything, by the time you've finished the project then something (the client's requirements, the market they're in, the tech you're working with, their customers' wishes) will have changed and made at least part of what you knew invalid or incorrect.

Therefore, it's impossible to plan everything up front, and even if you could, following that plan would lead you to build something imperfect or obsolete. Knowing this, we integrate change into our planning. Let's look at your steps:

Start with a few use-cases

Start coding

Realize a few things I did not handle well, and does not fit well in current codebase.

Rewrite most part of code

That's actually a great starting point. Here's how I'd approach it:

1. Start with a few use-cases

Good. By saying "use cases", you're focusing on what the software is for. By saying "a few", you're not trying to discover everything; you're sticking to a manageable amount of work. All I'd add here is to prioritise them. With your client or end user, work out the answer to this question:

What is the smallest, simplest piece of software I could give you that would improve your situation?

This is your minimum viable product - anything smaller than this isn't helpful to your user, but anything bigger risks planning too much too soon. Get enough information to build this, then move on. Be mindful that you won't know everything at this point.

2. Start coding.

Great. You get working as soon as possible. Until you've written code, your clients have received zero benefit. The more time you spend planning, the longer the client has spent waiting with no payback.

Here, I'd add a reminder to write good code. Remember and follow the SOLID Principles, write decent unit tests around anything fragile or complex, make notes on anything you're likely to forget or that might cause problems later. You want to be structuring your code so that change won't cause problems. To do this, every time you make a decision to build something this way instead of that way, you structure your code so that as little code as possible is affected by that decision. In general, a good way to do this is to separate your code:

use simple, discrete components (depending on your language and situation, this component might be a function, a class, an assembly, a module, a service, etc. You might also have a large component that is built out of smaller ones, like a class with lots of functions, or an assembly with lots of classes.)
each component does one job, or jobs relating to one thing
changes to the way one component does its internal workings should not cause other components to have to change
components should be given things they use or depend on, rather than fetching or creating them
components should give information to other components and ask them to do work, rather than fetching information and doing the work themselves
components should not access, use, or depend upon the inner workings of other components - only use their publicly-accessible functions

By doing this, you're isolating the effects of a change so that in most cases, you can fix a problem in one place, and the rest of your code doesn't notice.

3. Encounter issues or shortcomings in the design.

This will happen. It is unavoidable. Accept this. When you hit one of these problems, decide what sort of problem it is.

Some problems are issues in your code or design that make it hard to do what the software should do. For these problems, you need to go back and alter your design to fix the problem.

Some problems are caused by not having enough information, or by having something that you didn't think of before. For these problems, you need to go back to your user or client, and ask them how they'd like to address the issue. When you have the answer, you then go and update your design to handle it.

In both cases, you should be paying attention to what parts of your code had to change, and as you write more code, you should be thinking about which parts may have to change in the future. This makes it easier to work out what parts might be too interlinked, and what parts might need to be more isolated.

4. Rewrite part of the code

Once you've identified how you need to change the code, you can go and make the change. If you've structured your code well, then this will usually involve changing only one component, but in some cases it might involve adding some components as well. If you find that you're having to change a lot of things in a lot of places, then think about why that is. Could you add a component that keeps all of this code inside itself, and then have all these places just use that component? If you can, do so, and next time you have to change this feature you'll be able to do it in one place.

5. Test

A common cause of issues in software is not knowing the requirements well enough. This is often not the developers' fault - often, the user isn't sure what they need either. The easiest way to solve this is to reverse the question. Instead of asking "what do you need the software to do?", each time you go through these steps, give the user what you've built so far and ask them "I built this - does it do what you need?". If they say yes, then you've built something that solves their problem, and you can stop working! If they say no, then they'll be able to tell you in more specific terms what's wrong with your software, and you can go improve that specific thing and come back for more feedback.

6. Learn

As you go through this cycle, pay attention to the problems you're finding and the changes you're making. Are there patterns? Can you improve?

Some examples:

If you keep finding you've overlooked a certain user's viewpoint, could you get that user to be more involved in the design phase?
If you keep having to change things to be compatible with a technology, could you build something to interface between your code and that technology so you only have to change the interface?
If the user keeps changing their mind about words, colours, pictures or other things in the UI, could you build a component that provides to the rest of the application those so that they're all in one place?
If you find that a lot of your changes are in the same component, are you sure that component is sticking to just one job? Could you divide it into a few smaller pieces? Can you change this component without having to touch any others?

Be Agile

What you're moving towards here is a style of working known as Agile. Agile isn't a methodology, it's a family of methodologies incorporating a whole load of things (Scrum, XP, Kanban, to name a few) but the thing they all have in common is the idea that things change, and as software developers we should plan to adapt to changes rather than avoiding or ignoring them. Some of its core principles - in particular, the ones that are relevant to your situation - are the following:

Don't plan further ahead than you can predict with confidence
Make allowances for things to change as you go
Rather than building something big in one go, build something small and then incrementally improve it
Keep the end user involved in the process, and get prompt, regular feedback
Examine your own work and progress, and learn from your mistakes

Jumping straight from "use cases" to "start coding" neglects sufficient analysis and design and will lead to technical debt or major rewrites. This is the primary cause of his problem. — Bradley Thomas, Jul 06 '16 at 13:20
@BradThomas I'm going to argue that the size and/or complexity of the project makes a difference with that. If its a single form, one table, a few stored procedure type of project I think going from "use cases" to "start coding" is fine. — Timmy, Jul 06 '16 at 13:41
"Great. You get working as soon as possible. Until you've written code, your clients have received zero benefit. The more time you spend planning, the longer the client has spent waiting with no payback." Can't agree with this whatsoever. The less time you spend planning, the longer the client spends waiting for something that actually works properly, with no payback. — Lightness Races in Orbit, Jul 06 '16 at 14:08
I suggest two refinements on this answer: First, include in each use case a narrative (or story) for how the use case would be tested. Second, write decent automated unit tests for every unit, and several rigorous tests for any unit that is fragile or complex. Running a unit test is nearly free! — Codes with Hammer, Jul 06 '16 at 14:08
@LightnessRacesinOrbit I feel this answer is very poor for exactly the reason you give. Lack of adequate planning, diving right into coding, is the #1 reason for excessive total cost of ownership over the lifecycle of a software system. — Bradley Thomas, Jul 06 '16 at 14:12
Lightness, Brad, have you missed the whole Agile movement? Plus, this is a beginner -- he's not primarily responsible for analysis. He should be involved, so he can learn, but most of the work on WHAT to do will be done by more senior developers. — Rob Crawford, Jul 06 '16 at 14:27
@RobCrawford: There's a whole continuum between "no planning" and "excessive planning". Foregoing "planning" or "vision" altogether is likely to get you running... in circles. Agile is not about "not planning", it's about avoiding to rely on uncertain elements and being able to change the goals as you go: you still need some kind of over-arching goal, even if blurry/imprecise, otherwise you cannot measure whether what you develop is progress or egress. — Matthieu M., Jul 06 '16 at 14:30
@RobCrawford: The "Agile movement" does not mean "do insufficient planning and be as inefficient as possible". Quite the opposite, in fact. And, even if it did, you should not do things just because they are a "movement". — Lightness Races in Orbit, Jul 06 '16 at 14:42
Sometimes I feel like "Agile" is little more than an excuse for developers to eat their dessert first. After all, diving into coding new features is the fun part, right? — Bradley Thomas, Jul 06 '16 at 15:05
I think all of you who are objecting to "lack of planning" completely glossed over the fact that the first step is to identify the minimum viable product. This necessarily entails some planning. It seems to me the point of this post is more to discourage trying to get a perfect design up front; instead, so some planning and don't spend forever trying to identify everything up front. — jpmc26, Jul 06 '16 at 15:18
Woah, so this blew up. As commenters have noted, I do NOT advocate doing zero planning. What I'm saying - and what Agile says - is to not do too much planning. I explicitly say "Don't plan further ahead than you can predict with confidence". People saying that I advocate "diving right into coding" should note that coding is step 2, where step 1 is... well, planning. The trick is to do enough planning to determine the smallest product that helps your user, and then give them that product. — anaximander, Jul 06 '16 at 15:57
I would further argue that analysis alone does not eradicate technical debt, because no amount of analysis can enable you to see through time and predict the future perfectly. Technical debt is generally defined as "bad code, caused by doing what is quick and easy instead of what is proper, that we're stuck with because it's hard to change". The solution is not to analyse more, but to build code to good parctices, in such a way that nothing is hard to change. My answer gives guidance on how to do this. — anaximander, Jul 06 '16 at 16:01
In closing, Agile agrees that there is such a thing as too little planning. It merely suggests that there is also such a thing as too much. — anaximander, Jul 06 '16 at 16:02
@LightnessRacesinOrbit Are you advocating for waterfall? That's what's implied by your comment: a direct relationship (in the mathematical sense) between time spent planning and success of the plan (where the design is part of the plan). Perhaps you didn't mean to be as absolute as your comment came out? — jpmc26, Jul 06 '16 at 17:02
@jpmc26: Which comment was absolute? When did I advocate any pre-canned, pre-named methodology? I don't recall actually advocating anything! — Lightness Races in Orbit, Jul 06 '16 at 17:05
@LightnessRacesinOrbit "The less time you spend planning, the longer the client spends waiting for something that actually works properly, with no payback." This reads as an absolute statement. As I said, the implication is that you should maximize planning to maximize success. This would suggest waterfall, which emphasizes planning, is an effective approach. — jpmc26, Jul 06 '16 at 17:06
@jpmc26: Yes, but not one that names any methodology, or promotes any sort of behaviour or approach to either accepting or changing this fact. You are reading too much into it. All I'm saying is that insufficient planning leads to wasted time; that's it. — Lightness Races in Orbit, Jul 06 '16 at 17:07
I suggest to remove the SOLID reference since no programming language is mentioned — sakisk, Jul 06 '16 at 17:10
@ierax Other than the LSP, don't the SOLID principles apply to any programming paradigm? — Andrew, Jul 06 '16 at 17:20
@ierax SOLID is not tied to any language in particular. Some of the principles (like LSP) reference things like polymorphism that only exist in some languages, but I think in general the principles hold for a wide enough range of languages that they can be mentioned without tying the answer to any one language. — anaximander, Jul 06 '16 at 18:03
I agree that "Don't plan further ahead than you can predict with confidence" is a good benchmark. For example, I saw a project where hundreds of features were baked into the contract, but the team decided to not analyze all those features up-front. They had to do costly redesigns partway through, for features known at the start. The proper degree of planning would have been to plan and design for all the contractually required features from the start. — Joeri Sebrechts, Jul 07 '16 at 06:58
@JoeriSebrechts I disagree with your assessment. It is somewhere between unreasonably difficult and impossible to foresee all the impacts on design before doing significant work. A better way to handle this is to try to identify which features would have the biggest impact of the architecture and do them early. Even then, you may fail at correctly identifying them. — jpmc26, Jul 08 '16 at 18:25
@jpmc26 With respect, you weren't there for that project. It would have been possible to cut months of work from the backlog had the team looked in detail at all the features known at the start of the project. Every project is different though, what is true for one is not true for another. In my current project I don't analyze up-front much, because we don't have a clear idea yet of what the product is. My point is that when you know what you need to build, figure out the architectural impact from the start for all of it. You may still get it wrong, but likely less wrong. — Joeri Sebrechts, Jul 11 '16 at 07:18
@JoeriSebrechts We may just be getting hung up on semantics, then. I agree that when possible, it usually does make sense to at least evaluate the feature set and understand broadly what the impact will be so you can get a sense of what broad components your application will need. (Does it need a DB? Is NoSQL more appropriate than relational? Etc.) But "design" implies much more detailed specifications. (I believe this to be a common understanding and not just my own.) I think the kind of evaluation I am describing could be done during the initial prioritization, without getting too detailed. — jpmc26, Jul 11 '16 at 10:30
Amazing how a comment can have things NOT said inserted into it. — Rob Crawford, Jul 19 '16 at 17:03

score 14 · Answer 2 · answered Jul 06 '16 at 07:45

This is normal.

You can take one of two approaches:

Welcome Change

If you assume that you will get it wrong, you must build a code base that is open to change. Mostly this involves taking the code at the end of a book on refactoring, and building your code that way from the start (decomposability, good test coverage, ...).

Avoid Change

In this case you must do a BDUF (big design up front). You have to do a lot of paper prototyping, discussing ideas with potential users or yourself by rubber ducking, trying out various things in prototypes or mockups, writing down a complete specification, and only once you feel you have nailed down the solution then finally you can start coding. Doing all that doesn't actually get rid of unexpected change, it just reduces it a bit, the first year or so. So, you still have to build your code to be easy to change.

So, basically, change is a given. Assume it will happen. Build your code accordingly. In practice you can find a middle ground between the design-up-front and just-start-coding approaches that avoids gratuitous change without getting stuck in analysis paralysis. It just takes a bit of experience.

score 11 · Answer 3 · answered Jul 06 '16 at 07:42

Software development has been described as a series of inherently "wicked" problems.

Horst Rittel and Melvin Webber defined a "wicked" problem as one that could be clearly defined only by solving it, or by solving part of it*. This paradox implies, essentially, that you have to "solve" the problem once in order to clearly define it and then solve it again to create a solution that works. This process has been motherhood and apple pie in software development for decades

This perfectly describes the issue you're facing. Fundamentally, what we do is hard. If there is a part that could be described as "routine", over time we isolate and automate it. So all that's left is either the new or the difficult.

There are other ways to overcome issues like this; some people spend a lot of time considering the issues in advance and not writing code until they're comfortable with the design. Others seek guidance from people who've already dealt with issues like this, either via pair programming or just sites like this.

But certainly your approach doesn't suggest incompetence. The only issue might be if, when you're going back over, you're not reflecting on why you chose to do stuff in this way to begin with, and whether you might have been able to see the "better" way without the rewrite.

In a lot of cases, there was, and you can incorporate it into your design process for next time. In some cases, there wasn't (or the cost would have been as high or higher than the cost of your other approach), and you can just let your concern go.

score 8 · Answer 4 · answered Jul 06 '16 at 07:40

Yes, this is common, except maybe for the "rewrite most of the code" part. You'll never get all requirements right from the beginning, so it's important to deal with change well. That's what the concept of "code maintainability" is all about. Of course it also helps to spend some more time on getting the requirements and design right.
First, think of what changes were required in past projects and how you could have foreseen those or prepared for them better. At the beginning of a project, think more about the details of the use cases. Do some abstract design (what are the main components of the code and how do they communicate, what are their APIs) before you start implementing. Most importantly, try to keep the code as simple and easy to change as possible, read up on concepts such as SOLID, DRY, KISS and YAGNI.

score 6 · Answer 5 · answered Jul 06 '16 at 17:13

Is such practice common, or it implies I am not competent?

Those are not mutually exclusive. The pattern could be common, and you still might be incompetent. Your competence can be determined by measuring your performance against your goals. Are you meeting your goals?

Is this pattern common? Unfortunately, yes. Many people dive into projects with no clear idea of what problem they are solving, how they will engineer a correct solution, and what metrics will constitute success.

How can I improve myself on this aspect?

If you decided to go somewhere and just started walking, and then discovered a day in that what you really needed to do was transport an elephant from Cape Town to New York City, all that time you spent walking was wasted. Figure out what you are doing before you start doing it.

Once you start doing it, consider: what does code that does not have to be rewritten ever look like? That code:

Does one useful thing.
Does it correctly.
Does it with sufficient performance.

So: the more code you write where that code does one useful thing well, correctly, with good performance, the less code you will ever have to rewrite.

score 1 · Answer 6 · answered Jul 06 '16 at 12:08

I think its safe to say you're not so far off from a better way of working, and you're not the only one in this boat.

What I think you're missing is that, although you determine what you want, you don't stop to do the same process for how you'll do it.

This step of stopping and thinking about how to tackle the problem overall is called design, its the step that makes you spend some time in developing the structure or architecture of the system so that everything you do from them on contributes to the final solution, like fitting pieces into a jigsaw puzzle once you've worked out the edges.

Many people don't do this step, yet when I started coding it was mandatory. I think the difference is in development tools - whereas I started with a text editor, you now ave all kinds of features that allow you to jump in and type away.

So, take a little time to figure out the broad areas, components and interoperability between them, and define objects that will comprise your solution before starting to code. It doesn't have to be in immense detail, and understand that you won't get it perfectly right at the start, so it'll evolve over time, but what this does is to help stop you wasting effort revisiting things that shouldn't need to be changed much.

Christian Meißler · Answer 7 · 2016-07-07T06:42:22.550

I'd like to add a few pointers

1) I personally found it incredible useful to start with visualizing stuff. Draw boxes, arrows, lines ... Never mind whatever modelling language you use. In the first place you are doing it FOR YOURSELF. It should help your flow of thoughts.

2) Find a sparring partner - grab a coffee and the flipchart/diagram etc from above and get to town. IMHO it's even better if you dont have matching tech skills. You bounce back and forth the ideas for implementing the usecase. You find a dead-end or two - you find a solution. With an agile mind there is often less time spent in this stage than if you write code, it doesnt work and your have to kill chunks of your work or redo them

3) Find a boss who can understand that you are likely never done improving your written software. If you always plug in new features / requirements that get dumped on your desk and never care for your software, it will strike back at you with bugs, maintenance unfriendliness and lots of hair pulling a few years down the line. Get this software maintenance time/budget allocated! I good round magic number is about 20% of the time needed to develop the software is needed per year to keep it in shape during the course of its lifetime.

4) This process of incremental learning never stops (and shouldn't)! You will look back in 10 years and smile.

Hope this helps a tiny bit and good luck!

Should read "whatever". Edited the post above. – Christian Meißler Jul 07 '16 at 06:42 — Christian Meißler, Jul 07 '16 at 06:42

score 0 · Answer 8 · answered Jul 06 '16 at 13:56

You already have some great answers but your question brings a few things to mind that I thought I would try to touch on.

As you noted running into changes down the road I'd suggest thinking about how things impacted your project and how you could have minimized the impact with design/coding choices while at the same time generating a mental map of where people often make late changes. With experience you can anticipate and code in some flexibility for things that you know are going to be important -- though there is a disagreement about this concept in the industry as some will counter you are investing effort in an area not specifically requested, per se.
One the testing front, throwing a prototype up for your project stakeholder can be a great way to refine requirements. However, during development you can look for ways to "see" what is happening in your code without causing a lot of clutter or complexity. There are certainly tools available to help but you can also do a lot without them if you wanted to. Either way, coming out of the code to look at what is happening with a critical eye may provide various types of insight.
Look for common activities. You are going to find yourself handling the same issues over and over again. After a while you should discover the shortfalls or trade-offs of various choices and zero in on a methodology that helps you avoid them. Of course, if you are working in a framework some of these issues may be wrapped up in the tools you are using already.
If you are working within a framework spend the time, if you can spare it, to consider how to do things from scratch. For example, you can easily assemble a request message, open a socket and issue a GET or POST request manually. If you like you can manually parse XML messages. Whatever you do, the questions you generate and the answers you find will grow your skills. Of course, you get to choose which types of underlying issues are important or of interest. I'd consider this personal development and not expect to spend much on-the-clock time here.
Silver bullets, methodologies and high buzz factor issues are everywhere. Basically, the underlying realities of working with information are not changing that quickly. Take note of whether the framework, toolset, or methodology in place is a help or a hindrance in various situations. In large companies I've seen a lot of methodologies attempted though the company was not able to execute them effectively. Just like frameworks, methodologies are not a failsafe way to function at a high level even if they are based on practices of some highly functional teams.

It's hard to boil down experience and have it accessible. I guess a short way to put all this is to keep your eyes open, think about what you see, and never stop learning.