Stuck due to "knowing too much"

Question

Note more discussion at http://news.ycombinator.com/item?id=4037794

I have a relatively simple development task, but every time I try to attack it, I end up spiraling in deep thoughts - how could it extending the future, what are the 2nd generation clients going to need, how does it affect "non functional" aspects (e.g. Performance, authorization...), how would it best to architect to allow change...

I remember myself a while ago, younger and, perhaps, more eager. The "me" I was then wouldn't have given a thought about all that - he would've gone ahead and wrote something, then rewrote it, then rewrote it again (and again...). The "me" today is more hesitant, more careful.

I find it much easier today to sit and plan and instruct other people on how to do things than to actually go ahead and do them myself - not because I don't like to code - the opposite, I love to! - but because every time I sit at the keyboard, I end up in that same annoying place.

Is this wrong? Is this a natural evolution, or did I drive myself into a rut?

Fair disclosure - in the past I was a developer, today my job title is a "system architect". Good luck figuring what it means - but that's the title.

Wow. I honestly didn't expect this question to generate that many responses. I'll try to sum it up.

Reasons:

Analysis paralysis / Over engineering / gold plating / (any other "too much thinking up-front can hurt you").
Too much experience for the given task.
Not focusing on what's important.
Not enough experience (and realizing that).

Solutions (not matched to reasons):

Testing first.
Start coding (+ for fun)
One to throw away (+ one API to throw away).
Set time constraints.
Strip away the fluff, stay with the stuff.
Make flexible code (kinda opposite to "one to throw away", no?).

Thanks to everyone - I think the major benefit here was to realize that I'm not alone in this experience. I have, actually, already started coding and some of the too-big things have fallen off, naturally.

Since this question is closed, I'll accept the answer with most votes as of today. When/if it changes - I'll try to follow.

Agree with the time pressure. Nothing to make people get stuff done like "this MUST be done at LATEST by Monday, because that is when the customer starts testing ". — , May 28 '12 at 22:42
Your problem is not "knowing too much", but analyzing too much. You don't need to care about performance, future features, etc. too much now, it's not that the world's gonna end if the customer gives you a new feature that's a bit hard to implement — Louis Rhys, May 29 '12 at 02:29
Think "Minimum Viable Product." What's the least you could ship with? Then start there, ship, and add based on market feedback. — Scott C Wilson, May 29 '12 at 11:05
Holy crap - when I read this, it was as though I wrote it myself. After 5 years of churning out bad (but working/profitable) code, I now stare at the screen paralyzed but the thought of doing it wrong. I think it started when I really began trying to use Design Patterns. — MattW, May 29 '12 at 12:44
I don't believe you meant to say it this way but saying "I know too much" comes off a tad arrogant. You might also find the book discussed in this wikipedia article worth your while to read and consider: http://en.wikipedia.org/wiki/The_Paradox_of_Choice:_Why_More_Is_Less — Onorio Catenacci, May 31 '12 at 20:43
This question falls under "software engineering" and (maaaaaybe) "development methodologies", and has a specific answer: How not to over-analyze. Why was it closed? — Izkata, May 31 '12 at 21:40

score 90 · Accepted Answer · edited May 29 '12 at 16:13

Thinking about these things is definitely good, but don't let it stop your progress.

One approach that works really well (especially with iterative development) is to implement a simple solution and then refactor as necessary later. This keeps the code as simple as possible, and avoids over-engineering. Most of the performance or architecture changes you are contemplating probably aren't going to be required anyway, so don't bother writing them until they have officially become necessary. For example, don't worry about performance until a profiler has told you that it's time to improve performance.

One thing you can do to help you adjust is to set a hard time limit on how long you think about something before writing code. Most of the time, the code will turn out better if you think for a little, write it, realize your mistakes, and then fix them by refactoring.

There is a balance to be struck here. You shouldn't just jump in head-first and not think about the consequences, but you also shouldn't try to over-engineer your code either.

15

Official name: YAGNI. – Mark Ransom May 29 '12 at 19:30

score 48 · Answer 2 · edited May 30 '12 at 21:22

Wikipedia names it "Analysis paralysis" in software. The recipe is to stick to agile methodologies. Meaning that any activity or individual action has much more value than attempt to establish practices or policies. Every contributor in the team is valuable, no matter how well or bad the person's abilities fit to architectural ideals. In agile, individuals, egos are first, the policies are last.

My favorite answer is "Architecture is verb". Stop thinking, start acting, no matter how imperfectly and inefficient you and the team will feel. Maybe the first actions can be dismantling inappropriate policies.

score 44 · Answer 3 · answered May 29 '12 at 00:53

44

40 years ago Fred Brooks wrote about this "Write one to throw away, you will anyhow." Nothing has changed........

answered May 29 '12 at 00:53

mattnz

21,362

10

The erstwhile Jeff Atwood says "Version 1 Sucks But Release It Anyway".
http://www.codinghorror.com/blog/2009/12/version-1-sucks-but-ship-it-anyway.html
– Alan B May 29 '12 at 09:27
5

That's true for any version :) – Nemanja Trifunovic May 29 '12 at 13:21
2

@AlanB, here's another codinghorror post which sticks in my memory. – Benjol May 30 '12 at 08:42

score 38 · Answer 4 · answered May 28 '12 at 23:43

38

Is this wrong? Is this a natural evolution, or did I drive myself into a rut?

It depends. This tends to be a common step along the road of a developer.

Start throwing crap together, get bitten in the ass by it
Start over-engineering the living hell out of everything, realize that YAGNI
Settle on some pragmatic middle ground where the easy stuff is slapped together and the hard/change-likely stuff is given enough engineering to make it easy enough to work with/change.

It's only a rut if you stay at number 2.

answered May 28 '12 at 23:43

Telastyn

109,398

4

+1 You'll realize you're in a rut at number 2 when you start over-engineering "Hello World". – Spoike May 29 '12 at 09:00
3

@Spoike - Or Fizzbuzz. Ala, Enterprise Fizzbuzz! – CraigTP May 29 '12 at 10:05
1
Realize that 3 is wrong as well, and only concern yourself with fulfilling the business needs instead of the technical needs. The universal Business Need is that everything will change constant, small or large. The implementation details will fall in line with the bottom line drivers, and will be address when they need to be, no sooner, no later. Just In Time Development.

May 30 '12 at 01:15

Old Pro · Answer 5 · 2012-05-29T01:30:51.710

One of the things I always like to keep in mind is the saying "the future isn't what it used to be."

With a certain amount of experience it becomes tempting to believe you can predict the future, but you cannot. It is easy to envision features that future clients/users/whatever may want, but that doesn't mean they are going to want them right away. It also doesn't mean they're going to want them over some other conflicting feature. So you really do need to limit how much time you spend today planning for the future. You especially need to limit how much time you spend building things today that are only going to be of use in the future.

The question that I ask that keeps me on the straight and narrow is "how much harder will it be to build this feature later than it is to build support for that feature now?" Usually, the answer is that the future effort is about the same or maybe twice what it would be to do it now. In that case, because I can't predict the future, I have no problem not building it now. If the answer gets up to 10x or more, then I'll start asking around about how likely people think it is that we are going to need this in the next year or two. Even then, unless there is widespread agreement, I would simply limit myself to making sure it is not necessary to undo things we are doing today in order to achieve that goal in the future.

For example, I worked on a few projects where we spent a lot of time abstracting out the fact that we were using Hibernate as a data access later. (Never have I ever seen the project built on Hibernate stop using it, so that was a waste of time to begin with, but let's set that aside.) Even if it had been a reasonable possibility that we might want to change later, because we were also using a data access object pattern, it would have been no harder to build in the flexibility to change out Hibernate and change it out at the same time when we needed it than it was to build in the flexibility from the start. Faced with a situation like that now, I would just put off having that flexibility until we really needed it.

Unless you're doing strategic planning for a large corporation, it's hardly even worth it to think about architectural issues more than two or three years away, because technology is changing so quickly. The feature you may be thinking about building today may be freely available in open source in two or three years. Almost certainly the cost-benefit analysis will have changed.

Limit yourself to building a system that does what you need today, that you're proud of, and that you will be happy to work on in a few months whatever the next round of changes bring. Really that's the best you can do.

Premature generalization is responsible for most of the gumph in our current code base. — Benjol, May 30 '12 at 08:35

score 11 · Answer 6 · edited May 30 '12 at 21:27

Here's my personal process of elimination for crazy-awesome designs that (in their first version) can end up being impractical and damage a project from the business perspective.

Identify the epicenter: Think of your project as a hot-dog stand, the epicenter would be the hot dogs. You can take every other spice/dressing/vegetable out from your stand and still be able to sell hot dogs. What is the main purpose of your software? Isolate every other addition and/or added value from it and focus first on the epicenter.
Keep repeating to yourself "doing it later means doing it better": see where it makes sense and put a little "for later" note on it. If you do it well and thinking about its real-world usage you will end up with the same design but prioritized in a roadmap.
Diminish-Decouple-Discard: Whatever module design you have make it as simple/essential/pure/universal as you can (sometimes this can be accomplished without even removing features). When you cannot simplify it further, start decoupling elements that could live by themselves and have a purpose. In the end if you still have some fat in there, you will be able to just cut it off.
Separate "library code" from "production code": There will always be code that cannot be reused, but it always adds noise to design. This code is the one that contains business rules. You will find that sometimes some business rules just are easier and quicker to change (extremely important) than with a robust design. You will find code which you can rely on, and code the customer needs to change or reimplement in the future. You will want those to be kept as separate as possible.

And BTW, step 0 is: "go crazy with the design". This helps me out getting it out of my system and often finding new implications, hidden requirements and even emergent features.

I took 1 & 2 from Rework.

score 9 · Answer 7 · answered May 28 '12 at 20:52

9

Write the tests. You're done when the tests all pass: not before, and certainly not long after during an epic phase of over engineering. Having a suite of tests for the code you're writing will give you an independent, unbiased observer telling you when you can stop.

answered May 28 '12 at 20:52

Why was this downvoted? It gives a concise and objective answer to a pretty vague question. – Andres F. May 28 '12 at 21:53
6

(Not my downvote) When do you stop writing tests? You've just put the problem behind a level of indirection. – MSalters May 28 '12 at 22:11
2

@MSalters I think Graham is referring to TDD, where you write a set of tests before the code. Then you write the simplest code that makes those tests pass. Then you refactor. Following this technique may prevent you from over-thinking your initial development since your goal is to make the test pass, not make perfect code. – Oleksi May 28 '12 at 23:47
2

Testing can never prove the absence of bugs. Just because your toasts pass does not mean you're done. Your tests can at best show that a very very very small subsample of statistical insignificance of the possible inputs to your program produce the outputs you think they should. That's not even close to proving the program is correct. In any case, writing the tests does not help you architect solution that is extensible and maintainable going forward. – Old Pro May 29 '12 at 01:40
1

@OldPro we all know that mantra. The point here is that TDD starts you working instead of thinking! Being test-driven doesn't mean you trust them. – kaoD May 29 '12 at 02:18
1

@kaoD, the architect's job is not just to make code that works, but rather to design a maintainable and extensible framework that makes current development efficient and that also makes future enhancement and maintenace efficient. Writing tests does not help that process. – Old Pro May 29 '12 at 02:52
6

@OldPro the tests are a means, not an end. They encourage good design and focused workflow, and have a side effect of being mildly useful for reducing bugs. Generally speaking. Not always. – Phil May 29 '12 at 02:58
@Phil, my main point is that writing tests is not a good solution to the OP's problem since the OP is talking about a system architect's role. My more controversial opinion is that TDD encourages bad design, not good design, much as teaching to a standardized test in high school encourages memorization over thinking, but that is definitely off-topic. – Old Pro May 29 '12 at 03:07
2

Tests help define the scope and purview of the item. Whether you use TDD or another way, the idea of definging tests and then implementing until those tests are satisfied is what think @Graham is saying here. – Preet Sangha May 29 '12 at 05:24
@Oleksi: I know it's TDD, but the name doesn't matter. How many tests do you write, before you switch to writing code? I've recently been working on a cache implementation, together with a colleague, and it's clear that we can easily write 200 tests. (Caches have the nasty habit of still "working" when there's a bug, just failing to meet the potential efficiency). – MSalters May 29 '12 at 08:12
+1 for focusing on TDD which will help to create maintainable code. i aggree to @MSalters that risk of "over-testing" is high especially for a perfectionist. – k3b May 29 '12 at 08:26
@MSalters It's tricky, which is one of the reasons I didn't recommend this in my answer. However if you wanted to choose this route, I would partition the input space into several disjoint and complete classes: (Cache miss, cache hit), (Cache empty, cache non-empty), etc. This shouldn't generate a lot of tests. Once this is in place, then write your code to pass those basic tests. Next, I would use some code coverage tool to see what parts of the code aren't covered by the current test set, and add tests as required. – Oleksi May 29 '12 at 14:14

score 4 · Answer 8 · edited May 30 '12 at 21:23

There are only two things you really need to keep in mind when writing and designing software: maintainability and correctness.

Correctness is most important short-term and can easily be proven by tests.

Maintainability will help later in the development but is harder to pin down.

My current strategy is to first get a monolithic proof of concept up and then separate the UI from the model (ensuring the model doesn't know anything about the UI) once I'm satisfied that it is viable. If I would wait too long for this step, I'd get something unmaintainable. If I start with the separated layers, I just cannot seem to get started as I get stuck on what the UI needs to know about the model.

score 4 · Answer 9 · answered May 29 '12 at 08:26

When you're young, you don't see risk (possibly the reason junior politicians are scary), but as you get older, your bad experiences paralyze you at every opportunity (possibly the reason senior politicians are stagnant). Adopt an Occam's razor guided approach - go for the solution that has the fewest needs and then evolve from there.

score 3 · Answer 10 · answered May 29 '12 at 08:19

When I've got stuck in situations like these, I've found that it helps to stubbornly imagine that I'm an end-user using the hypothetical program to get something reasonably trivial done. Then I try to focus on what the programmatical entry-points needed to support those actions would be, trying as much as possible to ignore other aspects of the system. From here it's often possible to construct a (small!) 'whish list' of features of the finished system, and write some unrealistic code that starts to implement that. After that exercise, I usually am started, and the rest of the system starts to become more clear. It's all about the entry point - and the entry point of the vast majority of all software is the end-users initial actions with a program.

score 3 · Answer 11 · answered May 29 '12 at 08:21

I think it is a syndrom that the tasks you are doing are too easy for you.

A few years ago the chanllenge for you was to write a code that will accomplish the given task. This was what fully engaged your mind. Now, your mind (your experience etc.) is working more effective and doing the same task requires only a part of the energy that was previously needed. This is why you're ending in that spirale of deep thoughts. Your mind is defending itself from routine and is fighting for challenge.

I think you should consider changing your work. Maybe you should learn a new programming language.

score 3 · Answer 12 · edited May 30 '12 at 21:34

I had the same problem 15 years ago. I wanted to write perfect, reusable, universal, .... code that made the solution much more complicated than necessary. Today I see this as Gold plating. What helped me a lot was an advice of a colleague:

if you have an idea what might improve functionality, make it more univeral, ... write this idea down into a seperate textfile "ideas.txt" but don't implement this now.
keep on implementing the urgent tasks.
after six months, review your "ideas.txt" and analyse which of these changes would really have benefitted the project.

score 2 · Answer 13 · answered May 29 '12 at 08:36

This is simply paralysis by analysis. It happens to many people in many fields. You can break through it.

The answer is - JUST GET ON WITH IT ;-)

I post on a fitness forum and many many times people post about different routines, trying to find the exact perfect one for them. So we tell them just start training and work it as you go along. You want to get stronger, do some strenght exercises and then tweak things as you go.

When you have a large program and your brain works overtime - just code for the simple cases first. Initially the program must run, then must take input etc.
Your challenge is to leave things easy to update and refactor later on but the code MUST be no more complicated than it needs to be to do the task at hand.

Remember in the future it is fine to refactor code to meet new priorities. You cannot predict all of them up front so do not try.

Code for the next task - ONLY. Code is simply and well so it is easy to refactor if you need to. Make sure the program works. Repeat.

score 1 · Answer 14 · answered May 29 '12 at 09:31

Since you're getting "stuck" over-thinking possible end-user use-case scenarios, there is something to consider here if you're going to be publishing an API and expect that people unknown to you will utilize that API. Once an API is published, you either have to continue to support it, even though you later realize how bad your first release is, or you have to break the code of everyone, possibly unknown to you, who has written against it thus risking alienating them for all future time.

The standard solution is to publish with the stipulation that the API might change in any way at any time until you get a sense of what it is your users need and API consumers are doing.

In that solution I think is your own solution. Write just some little thing that does one or two things, maybe does them just OK, keeping the understanding to yourself that anything you do may change in the future.

It's not possible to get it all right, or in some cases even ANY of it right, when you start because design is really a journey of discovery; it's the last to emerge, not the first thing to get done.

You can't design an API right away and never have to break it to your consumers. You get why that is. By the same token you can't write software and not have to possibly throw it all away and start over with a new approach.

I don't think you have a problem in the sense of you've accidentally evolved into something less creative, productive or desirable. I think that you have high standards which you're accidentally misapplying to your situation-- a common mistake in thinking everyone makes.

Experience never counts against you unless you become a cynical know-it-all, done-it-all, and that really sounds like the opposite your situtation.

I have a couple images that I keep in my mind when I go big. One is of playing with Lego. I put it together and take it apart at will. What I start out making may not be what I end up making. I am surfing and advantaging myself of the possibilities that come to my mind as I go along, often recreating my goals entirely on the spot in a flash of inspiration... that's what creativity IS.

The other image is an analogy I heard that describes doing real science. You're groping around in a dark room for a black cat which may not be there. It's unnerving only if you count yourself as a failure for not finding that cat. There is no other way to find black cats. That's the only activity that ever locates them. Anything else is some form of already having what you're allegedly looking for.

score 0 · Answer 15 · answered May 29 '12 at 11:14

You don't know too much; you don't know enough! And you've only recently realized that.

Don't think about your design choices as something that you have to get "right", because there is no "right"--there are many "wrongs", but there are also tradeoffs (in execution speed, time to complete the coding task, extensibility, etc.). The code you write if well-conceived will still have various strengths and weaknesses.

The goal should be to get to the point where understanding these strengths and weaknesses in the context of current and future use and maintenance is not dramatically harder than writing code in the first place.

So don't avoid deep thoughts, but remember that you need experience, not just thought, to become a master at this sort of design. Think until you you reach a point where you are not confident of a particular choice, then implement what you hope is the best one, try it out, and learn from how it went.

score -1 · Answer 16 · answered May 29 '12 at 07:23

Practise coding. Come up with exercises or find them online, and try to finish them correctly as quickly as possible. When you get your speed up, add considerations like maintainability and performance. You'll realize that it's refreshing to code stuff that won't go into production and it might help you get out of your fear of coding.

score -2 · Answer 17 · answered May 29 '12 at 07:54

-2

Do coding in your spare time, the way you did 10 years ago, with less worries and just the fun.

Become a team manager and direct younger developers - who are now in the same mental state as you were 10 years ago.

answered May 29 '12 at 07:54

Sam

171

score -2 · Answer 18 · edited May 30 '12 at 21:35

-2

Yes, this coding horror happens even for me. Getting struck with Stack Overflow. Coding in detail for a small application. But when it's an outsourced project, it doesn't eat much time because of time constraints. And also working with a group of new bunch of people can get over this, I feel.

edited May 30 '12 at 21:35

Peter Mortensen

1,045
2
12
14

answered May 29 '12 at 13:05

Katti

101

score -2 · Answer 19 · answered May 29 '12 at 15:19

No, you don't know enough, yet.

Enrich your knowledge e.g. by these simple rules:

KISS: Keep it small and simple.
YAGNI: You ain't gonna need it.
Worse is Better: Some worse solutions are actually better in maintainability terms.

Do not overengineer. Be ready for change by having refactoring skills, but do not encode every possible change in code. If you see that over time, a class or function grows too big, change it. Divide and conquer. Use interfaces, use more functions. If you feel that you've divided too much (i.e. it became somehow fancier, but less readable), undo your last change.

In summary: Make your code flexible enough, but not more. Instead, make your self flexible, buy a book on refactoring.

score -2 · Answer 20 · answered May 30 '12 at 20:07

Two things:

It's not enough to know a lot. You have to have opinions about what's worth implementing. I personally see TDD as a crutch that enables bad architecture. Given the sheer quantity of popular opinion that works against me on this, I'm probably wrong, but whether to implement TDD in JavaScript is not an issue I put any thought into, because debug has never been a massive headache for me and hey, it wouldn't be the first time popular opinion in the development community was later seen as flawed years later. So learn to be an arrogant prick. At least on the inside. You might be wrong but at least your committing to stuff that works for you rather than overthinking stuff that might not.
It sounds like you're starting with micro. Start with macro. First the tools you need to do the stuff your app needs to do. That part should come easily enough. Then start with architecture/interface concerns. How the plumbing connects and what it connects to should really just be the flimsy bits on top of everything that you can just swipe aside and replace on a whim. Likewise with the details of how things are done. Wrapped properly these can be replaced/edited with ease.

score -2 · Answer 21 · answered May 30 '12 at 21:16

but every time I try to attack it, I end up spiraling in deep thoughts

There is nothing wrong. You have only noticed that it is time to improve your process: in this case, your thinking process.

Many people get lost in analysis or sidetracked in other ways and never even notice it. You have noticed so you can change your thinking process to stay on track.

There are many solutions to this, and the best one I have seen above is setting a time constraint. Before you start, decide how long (1 hour?) you will devote to analyzing and thinking about it. Set a timer.

Then, when the timer goes off, start the coding. If you find yourself thinking about things for too long again, just make an instant decision. Whatever you decide at that moment is the correct decision. You can always make changes and improvements later.

Besides, our environment is always changing. We often need to update our code to handle new requirements, new technology, new hardware, language and system updates, etc.

Salman · Answer 22 · 2012-05-29T08:42:13.423

-3

You Are Not Ruthless Enough

http://playswithfire.com/blog/2012/02/19/you-are-not-ruthless-enough/

Edit: The point of this article is paying attention to small details when developing, but I have found it helps to approach any simple or complex task with a ruthless attitude in order to find the single most efficient solution possible and just get the job done.

edited May 29 '12 at 08:42

answered May 29 '12 at 08:25

Salman

101

Stuck due to "knowing too much"

22 Answers22

No, you don't know enough, yet.

Linked