How to rename glue job output file name?

Question

I am creating a glue job where output is stored in s3 bucket. S3 path is given as follows:

DF.write.format('csv').mode("overwrite").save("s3://test",header='true')

I want a user defined file name for the output which should be appended with system date.

Ex: test20180101.csv

score 0 · Answer 1 · answered Nov 14 '18 at 22:51

0

You can try the following:

DF.write.format('csv').mode("overwrite").save("s3://test/test20180101.csv",header='true')

answered Nov 14 '18 at 22:51

Aida Martinez

This does not work. It creates a new folder `test20180101/` and a new file `part-000-xxx.csv` inside it. – Luke Mar 22 '22 at 08:25

score 0 · Answer 2 · answered Jan 08 '21 at 15:25

Don't use the partitions given in the code.

For system date you can use datetime -

from datetime import datetime
today = datetime.today() #will given the current date.

Append today in the Key of copy_object.

2 Answers2