How to generate a list of all dates in a range using the tools available in bash?

Question

I want to download a bunch of files named with ISO-8601 dates. Is there a simple way to do this using bash+GNU coreutils? (Or some trick to make wget/curl to generate the list automatically, but I find that unlikely)

Similar to this question, but not restricted to weekdays: How to generate a range of nonweekend dates using tools available in bash?. I guess that there is a simpler way to do it without that restriction.

Also related to How to generate date range for random data on bash, but not restricted to a single year.

Tom Fenech · Accepted Answer · 2020-06-18T11:33:21.043

If you have GNU date, you could do use either a for loop in any POSIX-compliant shell:

# with "for"
for i in {1..5}; do 
    # ISO 8601 (e.g. 2020-02-20) using -I
    date -I -d "2014-06-28 +$i days"

    # custom format using +
    date +%Y/%m/%d -d "2014-06-28 +$i days"
done

or an until loop, this time using Bash's extended test [[:

# with "until"
d="2014-06-29"
until [[ $d > 2014-07-03 ]]; do 
    echo "$d"
    d=$(date -I -d "$d + 1 day")
done

Note that non-ancient versions of sh will also do lexicographical comparison if you change the condition to [ "$d" \> 2014-07-03 ].

Output from either of those loops:

For a more portable way to do the same thing, you could use a Perl script:

use strict;
use warnings;
use Time::Piece;
use Time::Seconds;    
use File::Fetch;

my ($t, $end) = map { Time::Piece->strptime($_, "%Y-%m-%d") } @ARGV; 

while ($t <= $end) {
    my $url = "http://www.example.com/" . $t->strftime("%F") . ".log";
    my $ff = File::Fetch->new( uri => $url );
    my $where = $ff->fetch( to => '.' );  # download to current directory
    $t += ONE_DAY;
}

Time::Piece, Time::Seconds and File::Fetch are all core modules. Use it like perl wget.pl 2014-06-29 2014-07-03.

The first one can be combined with http://stackoverflow.com/a/8903280/939108 to use an explicit date. But I guess the second option is better then. — Hjulle, Sep 06 '14 at 15:16
If you want a format other than ISO-8601, you can do `date -d $d +"%m/%d/"` or whatever formatting string strikes your fancy instead of the `echo $d` line above. — ijoseph, Jun 17 '20 at 18:40

score 22 · Answer 2 · edited Jan 09 '19 at 06:46

22

Using GNU date and bash:

start=2014-12-29
end=2015-01-03
while ! [[ $start > $end ]]; do
    echo $start
    start=$(date -d "$start + 1 day" +%F)
done

edited Jan 09 '19 at 06:46

cweiske

28,704
13
124
186

answered Sep 06 '14 at 14:42

glenn jackman

223,850
36
205
328

Why not put the test in the while clause? `while [[ $start -le $end ]]; do` – Hjulle Sep 06 '14 at 15:01
I mean `while [[ ! $start > $end ]]; do`. The previous one doesn't work. – Hjulle Sep 06 '14 at 15:10

score 6 · Answer 3 · edited Sep 28 '21 at 08:23

6

If you are on macOS, then date works a bit differently from GNU date. Here's a variant to Tom Fenech's date invocation that supports both GNU and Darwin:

if [ $(uname) = 'Darwin' ]; then
    d=$(date -j -v+1d -f %Y-%m-%d $d +%Y-%m-%d)
elif [ $(uname) = 'Linux' ]; then
    d=$(date -I -d "$d + 1 day")
fi

edited Sep 28 '21 at 08:23

tink

12,902
4
44
47

answered Apr 20 '20 at 14:10

Ernst de Haan

378
5
11

3

Or just `brew install coreutils`, then use `gdate` – ijoseph Jun 17 '20 at 18:36
Check the day offset, needs a -v+${i}d for this to work in the @Tom Fenech example. MacOSX 12 zsh: ```start=2020-01-01; for i in {1..5}; do date -j -v+${i}d -f %Y-%m-%d "$start" +%Y-%m-%d; done ``` – Rich Andrews Mar 28 '22 at 18:26

score 2 · Answer 4 · answered Oct 04 '16 at 08:52

2

I use this handy function to work with log files in the format yyyymmdd.log.gz:

function datelist { for dt in $(seq -w $1 $2) ; do date -d $dt +'%Y%m%d' 2>/dev/null ; done ; }

It accepts dates in the format yyyymmdd.

answered Oct 04 '16 at 08:52

runlevel0

2,279
2
22
30

1

This can of course be extremely inefficient if you need big date range. It will generate up to 70 useless days per month and 88 useless months per year, which it will feed to `date` which will give an error. – mivk Oct 08 '21 at 14:11
@mivk: Indeed. Good if you want a couple of years and if you run it on a fat machine. I haven't given it a second thought since I posted it. In my work, back then, I queried the SQL server, which can return a list of dates and does not need checking. But not everybody has an SQL server running and not all servers are as fast as the corporate one I used. – runlevel0 Oct 27 '21 at 15:00

score -1 · Answer 5 · answered Sep 06 '14 at 15:46

-1

This is how I ended up doing it:

d=$(date -I);
while wget "http://www.example.com/$d.log"; do
    d=$(date -I -d "$d - 1 day");
done

This tries to download all files from today's date until we get a 404.

answered Sep 06 '14 at 15:46

Hjulle

2,291
1
21
32

How to generate a list of all dates in a range using the tools available in bash?

5 Answers5

Linked