python-dynamically change log's directory when log content is updated

Question

I'd like to change the log's directory depending on the time (specifically the hour) when new log is created.

For example, let's say the message aaa is saved in a log file in directory d:\log\191105\09\log.log at 09:59 during running a program.

As the time passes, new log bbb is saved in a log file but the directory should be different, d:\log\191105\10.log at 10:00 without terminating the program.

My log manager code is following,

import logging
import datetime
import psutil
import os, os.path

class LogManager:
    today = datetime.datetime.now()
    process_name = psutil.Process().name()
    process_id = str(psutil.Process().pid)
    log_dir = "D:/LOG/" + today.strftime("%Y%m%d") + "/" + today.strftime("%H") + "/"
    log_filename = process_name + '_[' + process_id + ']_' + today.strftime("%Y-%m-%d_%H") + '.log'
    if not os.path.exists(log_dir):
        os.makedirs(log_dir)

    # date/directory config
    log = logging.getLogger('mypython')
    log.setLevel(logging.INFO)
    formatter = logging.Formatter('%(asctime)s||%(levelname)s||%(message)s')
    fileHandler = logging.FileHandler(log_dir+log_filename)
    fileHandler.setFormatter(formatter)

    log.addHandler(fileHandler)

    def update(self):
        self.today = datetime.datetime.now()
        old_log_dir = self.log_dir
        self.log_dir = "D:/LOG/" + self.today.strftime("%Y%m%d") + "/" + self.today.strftime("%H") + "/"

        if (old_log_dir == self.log_dir):
            return

        self.log_filename = self.process_name + '_[' + self.process_id + ']_' + self.today.strftime("%Y-%m-%d_%H") + '.log'
        if not os.path.exists(self.log_dir):
            os.makedirs(self.log_dir)

        self.log = logging.getLogger('mypython')

        # here, i wanna update my filehandler
        for hdlr in self.log.handlers[:]:
            if isinstance(hdlr, self.log.FileHander):
                self.log.removeHandler(hdlr)

        self.fileHandler = logging.FileHandler(self.log_dir+self.log_filename)
        self.fileHandler.setFormatter(self.formatter)

        self.log.addHandler(self.fileHandler)

    def i(self, ex):
        self.update()
        self.log.info(ex)

    def w(self, ex):
        self.update()
        self.log.warning(ex)

and I call these functions like

import LogManager as lm

logManager = lm.LogManager()
logManager.i('message')

It works well, but seems like it does not update its fileHandler after passing the hour.

I tried to find if the log has updateHandler method... but it doesn't .

What should I do??

As a first impression, you are implementing lots of code that might already be available in existing libraries. Do you really need a LogManager? Can't you do the [same](https://stackoverflow.com/questions/6386698/how-to-write-to-a-file-using-the-logging-python-module), with less code, with the `logging` library? [This question](https://stackoverflow.com/questions/13839554/how-to-change-filehandle-with-python-logging-on-the-fly-with-different-classes-a) also addresses your issue, doesn't it fit your case? Apart from that, what is `i(self, ex)` and `w(self, ex)` -> against all python phylosophy. — MikeMajara, Nov 05 '19 at 08:00
Does this answer your question? [How to change filehandle with Python logging on the fly with different classes and imports](https://stackoverflow.com/questions/13839554/how-to-change-filehandle-with-python-logging-on-the-fly-with-different-classes-a) — MikeMajara, Nov 05 '19 at 08:01
@MikeMajara Thank you for your comment, I've already read that answer. That's why I tried removing and then adding the fileHandler, instead of updating. And, i is info and w is warning, I followed someone's code on the internet. Since I'm very new to python so I didn't know that it does not fit into python philosophy. I'm trying to learn. Thanks. :) — W.Cointreau, Nov 05 '19 at 08:04
Yeah, just saying, so you can learn. Advice would be to rename to `log_info(self, ex)` and so on. And so, you read that answer, but it does not work for you? In your code it seems you are not retrieving the root logger, as recommended by answer in that question, but instead 'mypython' logger. — MikeMajara, Nov 05 '19 at 08:30
@MikeMajara Yes. I've editted my code(not in this question yet) and tried using root logger. However I also added ```if isinstance(hdlr,log.FileHander):``` according to the answer by Arun. it made an error message ```'RootLogger' object has no attribute 'FileHander'```. I think I need to try again without that code. But as Arun said it may generate the other errors by removing other handlers. :( — W.Cointreau, Nov 05 '19 at 09:03
With the link from MikeMajara, I finally succeeded what I've wanted. But I still have the concern that it might remove other handlers from other processes + thread safe problem. — W.Cointreau, Nov 06 '19 at 03:59
If you are using a library, thread safety responsibility lies on that library (separation of concerns). If you were doing something wrong it should (and probably will) warn you or throw Exceptions. What do you mean with handlers from other processes? Anyhow you could do a more sofisticated function which would only remove your FileHandler, to add another. I guess. Instead of all like shown in the snippet of linked question. check [here](https://docs.python.org/3/library/logging.html#logger-objects) and [here](https://docs.python.org/3/howto/logging-cookbook.html) for more examples. — MikeMajara, Nov 06 '19 at 06:29
Sorry, just now I located the post by @Arun. So, yes, that is what I mean in my last comment. All that you can check by your self, e.g.: `filename` attribute in `FileHandler` object. About that you should worry. Anyhow, for python starters... this feels a bit too much. Is logging so important right now for you? Can you not redesign your logging policy to keep on? If you were to trust your money on this function, would you bet on it? If your not sure, rethink. Often programming problems have little to do with language specifics. Remember KISS. — MikeMajara, Nov 06 '19 at 06:45

blues · Accepted Answer · 2019-11-06T09:15:27.503

You can do this much easier by using what is already in the logging library. Specifically there is a TimedRotatingFileHandler for which you only need to change how it names the files it creates. I've created a working minimal demo for you which changes the folder that logs get saved into on every second. Edit: change to rollover on every full hour instead of every second as per comment below

import os
import logging
from time import sleep, strftime
from logging.handlers import TimedRotatingFileHandler

def namer(default_name):
    head, tail = os.path.split(default_name)
    folder_name = 'logs'+strftime('%H%M%S') 
    folder = os.path.join(head, folder_name)
    try:
        os.mkdir(folder)
    except:
        # folder already exists
        pass
    return os.path.join(folder, tail)

logger = logging.getLogger()
handler = TimedRotatingFileHandler('base.log', when='H')
handler.namer = namer
# set the rollover time to the next full hour
handler.rolloverAt = datetime.now().replace(microsecond=0,second=0,minute=0).timestamp() + 60*60
logger.addHandler(handler)

logger.warning('test1')
sleep(2)
logger.warning('test2')

Thank you. I looked at ```TimedRotatingFileHandler```. It seems like it has a logging based on time interval or the day. What I exactly need is logging every hour as a new file, not the time interval.. — W.Cointreau, Nov 06 '19 at 01:59
So you want to rollover on every full hour? That is just one more line of code. See the edit above. — blues, Nov 06 '19 at 09:13

python-dynamically change log's directory when log content is updated

1 Answers1