As per recommendations of global researchers, early vaccination for majority of the global population is the only way to restrict the mutations of Covid vaccines. The longers people remain unvaccinated, the higher the chances of new & possibly deadlier variants appearing by mutation. What is even more concerning is the fact that there is not enough knowledge or confidence in the medical community about how useful the current vaccines will be against new variants.

So in summary, the global race to contain Covid & its variants are not very encouraging. One can only that the global leaders & agencies get together to plan & execute the vaccination drive in the most timely & efficient manner in the coming days.

Use

Any script or command which is to be executed in some time interval, can be set with Cron.

Points to remember

• Absolute paths for script or files need to be provided. For eg. I have a python file in a virtual environment which I want to set with Cron. In that case, I have to provide the command as below

/home/username/venv/bin/python  /home/username/venv/project/script.py

• It need to be planned if the output of the command (if any) need to be saved. It is also possible to setup email integration with a Cron job.

Steps

• Below command opens the crontab file for current user. At the bottom, we can add multiple cron entries

  crontab -e
  

• A single cron entry follows m h dom mon dow format, which means minute hour dayofmonth month dayofweek command. Detail instruction & example are also included in the file itself
• To run a python script on 12PM daily, we would add below entry

  0 12 * * * /home/username/venv/bin/python  /home/username/venv/project/script.py
  

• To ignore a field, use *. Then only the non * fields will be considered, as here I have ignored the dom, mon, dow fields, so the Cron will run daily
• As the minimum time field is minute, we can not use seconds for Cron
• After entering the entry, use Ctrl+O & Ctrl+X to save

• Ensure there are no \n character in the entry line, else parsing error will be thrown while saving

• The Cron dameon will check the entries in its own, so not need to restart it after entry
• The generated crontab file can be seen at /var/spool/cron/crontabs/username
• The log of the Cron job can be found in syslog. We can search Cron logs by username

  cat /var/log/syslog | grep "CRON" | grep "username"
  

Conclusion

• Although Cron has its limitation, it provides a very stable way to automate administrative tasks
• It provides many more features & options for various use cases

References:

• https://www.digitalocean.com/community/tutorials/how-to-use-cron-to-automate-tasks-ubuntu-1804
• https://www.cyberciti.biz/faq/how-do-i-add-jobs-to-cron-under-linux-or-unix-oses/

Setup PhpMyAdmin

• We are assuming php, mysql, nginx are already installed in the Ubuntu system
• Install PhpMyAdmin

  sudo apt-get install phpmyadmin
  

• During installation, below screen will appear & ask to choose web server

As we will be using Nginx as web server, don’t select any of the options & enter “Ok”

• In next steps, a new database user for PhpMyAdmin will be created & a database named ‘phpmyadmin’ will be created as well
• After successful installation, phpmyadmin folder can be found at /usr/share/phpmyadmin. We will create a symlink to /var/www/html using following command

  sudo ln -s /usr/share/phpmyadmin /var/www/html/phpmyadmin
  

Create Nginx Server Block

• Create a server block file for phpmyadmin at /nginx/sites-available/phpmyadmin & put below content

  server {
        root /var/www/html;
  
        # Add index.php to the list if you are using PHP
        index index.php index.html index.htm index.nginx-debian.html;
  
        server_name ***.**.***.*;
  
        location ~ \.php$ {
           include snippets/fastcgi-php.conf;
           fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
           fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
           include fastcgi_params;
       }
  }
  

• Test nginx configuration by running sudo nginx -t. If no error found, restart nginx by sudo service restart nginx
• The phpmyadmin web interface will be available at [Your_IP]/phpmyadmin web address

Change name of PhpMyAdmin folder for security

• As automated hacking attack can target the url [IP]/phpmyadmin, it is better practice to use some other name instead of phpmyadmin in the web link.
• To do this, simply change the folder name in /var/www/html by command sudo mv phpmyadmin mydb
• After that the interface will be available at [Your_IP/mydb] web address

Footnote

• There are many more steps regarding setup of Phpmyadmin with mysql & nginx, related to security & best practices. Please follow those as seem fit.

Use with python

• The official sdk for python is Boto3 which provide API for using AWS S3.

Create boto3 configuration using AWS credentials

• Below is an example of basic boto3 configuration. More details, options & best practices can be found in https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

import boto3

s3_resource = boto3.resource(
    "s3",
    aws_access_key_id="**********",
    aws_secret_access_key="**********",
)

S3 File download

Get file objects of a Bucket

• Set bucket variable as name of the S3 bucket
• Set prefix variable as name of S3 folder structure prefix. For eg. for file name like s3://bucketname/[clientid]/file1.csv, clientid can be used as prefix to get only files starting with a particular clientid.
• Using the above, we can use the list_objects_v2 method to get a list of matching file objects

  obj = s3_resource.meta.client.list_objects_v2(Bucket=bucket, Prefix=prefix)
  all_keys_obj = obj['Contents']
  

• The output of the above step is a list of s3 objects, like below

  [
    {'Key': '*****', 'LastModified': datetime.datetime(2020, 3, 3, 12, 16, 16, tzinfo=tzutc()), 'ETag': '"*****"', 'Size': 869061, 'StorageClass': 'STANDARD', 'Owner': {'ID': '*****'}},
    ....
    {'Key': '*****', 'LastModified': datetime.datetime(2020, 3, 3, 12, 18, 10, tzinfo=tzutc()), 'ETag': '"*****"', 'Size': 23333915, 'StorageClass': 'STANDARD', 'Owner': {'ID': '*****'}}
  ]
  

2020-07-20 12:05:30,766 - __main__ - DEBUG - This is a debug message

Alternatively, if we manage to output logs as stream of json objects with pre-defined key-values, it becomes much easier to index & analyse in an organised manner.

Required package

One such python package, which enabled Json logging for python application is https://github.com/thangbn/json-logging-python. Other similar solutions are also available.

Install

• Using pip

  pip install json-logging
  

Example setup

• Create a genetic utility function to return a logger object

def get_logger():
    """
    Generic utility function to get logger object with fixed configurations
    :return:
    logger object
    """
    json_logging.ENABLE_JSON_LOGGING = True
    json_logging.init_non_web()
    logger = logging.getLogger(__name__)
    logging.basicConfig()
    json_logging.config_root_logger()
    logger.setLevel(logging.DEBUG)
    logging.addLevelName(logging.ERROR, 'error')
    logging.addLevelName(logging.WARNING, 'warning')
    logging.addLevelName(logging.INFO, 'info')
    logging.addLevelName(logging.DEBUG, 'debug')
    logger.addHandler(logging.NullHandler())
    return logger

• Create a logger object

  logger = get_logger()
  

• Sample log generation code & output

logger.error(e)

{"written_at": "2020-08-01T19:39:45.359Z", "written_ts": 1596310785359963000, "msg": "Internal Server Error: /api/endpoint1/user_name", "type": "log", "logger": "django.request", "thread": "Thread-3", "level": "error", "module": "log", "line_no": 228, "exc_info": "Traceback (most recent call last):\n...", "filename": "log.py"}

Ideas for advanced use

• In addition to general logs, we can also use log messages with custom key-values for specific insights about the application. For example, we can add some extra keys to keep track of inter-api response time & use it afterwards to understand application performance.

logger.info("Fetched from apiA/search",
                       extra={'props':
                           {
                               'caller_api': 'apiA',
                               'receiver_api': 'apiB',
                               'receiver_endpoint': 'search',
                               'api_response_parsed_duration': 28
                           }
                       }
            )

Conclusion

As json is one of the most flexible, portable & understandable data transfer format, it can provide immense benefit to use json logging for any application (including python).

Generating appropriate log is a primary necessity in a software application. The main benefits of a well configured log output in comparison of print statements are such as -

1. Provides diagnostic details like module name, line number, traceback, loglevel etc

2. Log stream can be directed to various output like file, central logging system, container log etc

3. Loglevel can be modified to control the log output, without modifying code & restarting application.

Python Configuration

Below code create a basic logging configuration using python standard ‘Logging’ package.

import logging

loglevel = os.environ.get('loglevel','DEBUG').upper()
logging.basicConfig(filename='logfile.log',
format='%(asctime)s %(name)-8s %(levelname)-8s %(message)s',
datefmt='%m/%d/%Y %I:%M:%S %p',
level=loglevel)

Here we are setting some common options like log file name, attributes of log output, datetime format, loglevel. We are getting the loglevel from environment variable instead of providing fixed value.

Python logging package have following loglevel — DEBUG, INFO, WARNING, ERROR, CRITICAL, to be used accordingly. By setting the loglevel from environment variable, we can use the below workflow -

1. In a piece of code, we write a debug level log message to track an expected output

    logging.debug('Running for loop with {} value'.format(value))
   

2. In a exception case, we catch the exception and generate a ERROR level log

    logging.error('Exception occurred, unable to process for {} value'.format(value))
   

3. Now, in development stage we would want to view both the logs. However in production, we will only be interested in the ERROR log. So, after deployment, we simply change the loglevel by running export loglevel=ERROR which will limit the log messages to ERROR type only. Again, if we want to debug the production application and want to view the DEBUG level as well, we can simple change the loglevel, without making any code or application state updates.

References

1. Python Logging documentation
2. Loggy — python logging

For example. let’s say we have setup a web API with Nginx. Now we need to monitor various metrics like number of API access, user-agent, response status etc. Below are the steps -

1. Define access & error log file: As per default nginx configuration at /etc/nginx/nginx.conf, below log files are defined

   access_log /var/log/nginx/access.log;
   error_log /var/log/nginx/error.log;
   

   We can define different log files as well, in virtual blocks configuration, at /etc/nginx/sites-available/[virtual_block_file]

2. Create one or more custom log format: We can create log format specifications with a name

   log_format myapilogformat '$remote_addr - $remote_user xxx[$time_local]xxx '
    '"$request" $status $body_bytes_sent '
    '"$http_referer" "$http_user_agent" $response_time ';
   

3. Update log file with log format: Now we have to update the access_log / error_log to work as per provided log format as below

   access_log /var/log/nginx/access.log myapilogformat;
   

4. Save & test if no error present

   nginx -t   #tests all nginx configurations
   

   If error shown, check for syntax/ spelling errors etc. If no error found, restart nginx

   sudo service nginx restart
   

Reference:

• Nginx format options here