Data organization strategies for single user desktop QGIS

Question

I have been teaching myself GIS using QGIS for several weeks and I am looking for advice on how to structure my data in a way that is extensible and won't require me to constantly reorganize things. I have been working with the following types of data: topography (DEMS, Hill shades, contours), hydrography, roads, trails, political boundaries and some associated/joined data files.

Currently my data is in a mis-mash of file types and naming conventions based on the source I got the data from.

My questions:

should I use a database, and if so, what which one: SQLite, SpatiaLite or Postgres? Or should I simply use the file system?
are there some rules of thumb for naming/structuring the data I have outlined above?

I realize this may not be the most crisp way of asking the question, but I am looking for a wholistic approach to organizing my data, since it is very painful to reorganize things across multiple projects on a regular basis. Looking for any hints/guidelines you can provide. It seems like many people must use the same general data types and have ways of organizing it in a way that is fairly standard?

score 5 · Answer 1 · answered Apr 14 '16 at 01:40

This is just one fool's opinion.

I separate the file types into specific folders. I will have a main Mapping Folder with sub-folders named specifically. Most of the sub-folders will contain data that will be static, or not edited.

My main folder will contain files that I usually update frequently.

I always have an Archive sub-folder to save files prior to their being changed.

\Mapping
    \Archives
    \Images
    \DEMS
    \Shapefiles
    \Data_From_Whoever
    \Exports
    \Mailouts

and so on.

I typically do not rename anything from outside sources unless I edit it.

As for databases, that choice is up to you, and what you are most comfortable working with. You will find evangelists for SpatiaLite, PostGIS, and so on, and they will all have valid arguments as to why you should choose their preference.

You will probably work with Shape files more than any other vector file since they are the most frequently used type of file (I know, that point is somewhat arguable, but it holds true for me, and many I know.) Get used to them, and their limitations first. Then you can experiment with other databases, and see what works best for you.

If you are supplying data to a client, then you want to work with what they desire, make their life as simple as possible. Good clients will appreciate it, and hopefully provide you with more work.

I have a tendency to explicitly name everything that I generate for the first time, then save it with a date appended to the file name when the original file is going to change, or be updated. This practice has saved me quite a few headaches, and has ended disagreements in short order. The changed files go into my Archive folder.

I also do the same thing when I get data from other sources that has been updated. Being able to track changes is a very important aspect to GIS. I know that is redundant, has been said above, but I cannot stress the importance of tracking what has been done, by you, and others enough!

I frequently work on projects that may have hundreds of files, so using explicitly named folders has been the most efficient method for me.

Then, the biggest problem is deciding how much data to put on a map once the file organization is taken care of.

Welcome to QGIS! I wish you the greatest of fortunes!

Thank you. The more I explore this topic, the more I see that there are many options depending on things like: are you a GIS pro who has many clients (implying a client-centric approach), or if you work with large amounts of data. Still, imaging how crazy Unix would be if the file system didn't have some standard conventions... — craigmcg, Apr 14 '16 at 19:13
You are welcome! There are many ways to organize projects, none of them wrong if they work for you. I would caution against over-organizing. I have fallen into that trap, and ended up burying the data, making it difficult to find.
I used to use a strict vector file naming standard with letter prefixes denoting what the layer was. Such as: A_ for access B_ for buffers H_ for hazards S_ for structures and so on. I decided to simply name my files explicitly after a bunch of experimentation. Have as much fun as you can with this! — jbgramm, Apr 15 '16 at 01:50