Hub 10953 - Anonymize IPs in access logs

anonymize.py

 #!/usr/bin/env python
 """Anonymize data in access log lines.
 
 Read a line from stdin, write it to stdout with the following changes:
 1. IP (v4 or v6) replaced with a salted hash of the IP and the date
 2. Country information (extracted from IP) added after the salted hash.
 
 If the country information is unavailable in the database, '-' is added instead
 of ISO 3166-1 alpha-2 country code (like 'DE').
 
@@ skipping 49 matching lines @@
         to_hash = (ip + date).encode('utf-8')
         token = hmac.HMAC(salt, to_hash, hashlib.sha1).hexdigest()
 
         print(token, country, non_sensitive_info, end='')
 
     reader.close()
 
 
 if __name__ == '__main__':
     parser = argparse.ArgumentParser(
         description='Filter out sensitive data from access logs',

[tlucas, 2018/05/08 16:08:21]

What do you think about
...
        description=__doc__, 
        formatter_class=argparse.RawDescriptionHelpFormatter,
...

That way you'd have all that well-written module documentation as the output for


$ python anonymize.py -h

[l.kryvonos, 2018/05/08 16:26:19]

I like doing `description=__doc__`, but I think the `__doc__` in this case is
too detailed. To my knowledge, this script will be used in between two pipes,
and not invoked manually (in which scenario exploring `-h` comes up often). And
if it fails and someone needs to understand where's the problem, they'll open
the code anyway.
What do you think?

[tlucas, 2018/05/08 16:40:55]

Fair enough - let's see if those encountering the errors are keen enough to dig
through code ;)

     )
 
     parser.add_argument(
         '--salt',
         dest='salt',
         default=os.getenv('ANONYMIZE_SALT'),
         help='Salt for hashing sensitive data, defaults to $ANONYMIZE_SALT'
     )
 
     # https://dev.maxmind.com/geoip/geoip2/geolite2/
     parser.add_argument(
         '--geolite2-db',
         dest='country_db',
         default=os.getenv('ANONYMIZE_GEOLITE2_DB'),
-        help=('Path to MaxMind DB file with GeoLite2 Country data, defaults '

[tlucas, 2018/05/08 16:08:21]

Those parentheses are redundant (and trigger our custom A111 in flake8)  

[l.kryvonos, 2018/05/08 16:26:19]

Acknowledged.

-              'to $ANONYMIZE_GEOLITE2_DB')
+        help='Path to MaxMind DB file with GeoLite2 Country data, defaults '
+             'to $ANONYMIZE_GEOLITE2_DB'
     )
 
     args = parser.parse_args()
 
     if args.salt is None or args.country_db is None:
         parser.print_help()
         sys.exit(1)
 
     main(args.salt, args.country_db)
-

[tlucas, 2018/05/08 16:08:21]

Nit: this blank line is redundant.

[l.kryvonos, 2018/05/08 16:26:19]

Do all of our repositories follow the 'no blank line at the end of file'
convention?

[tlucas, 2018/05/08 16:40:55]

Yes - or at least they should (this is also a built-in flake8 warning; W391)