Search This Blog

Tuesday, September 4, 2012

MaxMind - GeoIP Database Installation Instructions

MaxMind - GeoIP Database Installation Instructions: "/usr/local/share/GeoIP/GeoIP.dat"

GeoIP Database Installation Instructions
Here is a brief outline of the steps needed to install GeoIP Country on Linux/Unix. The installation on Windows is similar, just replace the tar command with WinZip or a similar ZIP program.
Step 1 - Download database
For GeoIP clients, go to the download files page. You may need to enter your username and password. From there, you can download the binary or CSV formats of the GeoIP databases you purchased. Then you will want to upload the databases to your web server.
If you are not a client, you can use the free GeoLite database. You may download it from the GeoLite page.
If you are using the wget program to download the GeoLite file, please use the -N option to only download if the file has been updated:
$ wget -N -q http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
Step 2 - Install database
Once you have uploaded the database, you will want to uncompress it. To uncompress the binary format, you will need to unzip the file. For example, to uncompress the GeoIP Country binary database on Linux or Unix, you could run:
$ tar xvfz GeoIP-106_20051201.tar.gz
Then you will need to install the .dat file into a data directory. For example, on Linux/Unix, you could run:
$ mv GeoIP-106_20051201/GeoIP-106_20051201.dat /usr/local/share/GeoIP/GeoIP.dat
To uncompress the CSV format, you can use any zip program, like WinZip for Windows, or unzip on Linux. Then you can load the CSV database into a SQL database. Learn more 
Step 3 - Query database
For the binary databases, we supply APIs for most major programming languages. You may need to point the API code to the /usr/local/share/GeoIP/GeoIP.dat location. Each API has documentation and example programs inside the downloaded distribution. The fastest way to get started is to use the example program included. You may need to replace the database path in the example program with the location where you installed the database.
For the CSV databases, once you have loaded them into a SQL database, you can issue SQL queries against the database table with the GeoIP data. Learn more 
If you do not have shell access to the server
After you have downloaded the database, unzip it using a program like WinZip. Then rename the *.dat file to GeoIP.dat, GeoIPRegion.dat, or GeoIPCity.dat, for the GeoIP Country, Region or City databases, respectively. Finally upload the *.dat file to the /usr/local/share/GeoIP or other user-specified folder on your server using FTP or another upload mechanism.

Install Geo::IP Perl Module on Debian | MDLog:/sysadmin

Install Geo::IP Perl Module on Debian | MDLog:/sysadmin: "perl Makefile.PL"


Install Geo::IP Perl Module on Debian

This little post will guide you how to quickly install the MaxMind Geo::IP Perl Module that has different usages as shown in other posts. This module should be faster than the otherMaxMind perl module (Geo::IP::PurePerl) but it will require to have the GeoIP C libraryinstalled in order to be compiled.
So first we will need to install the GeoIP C library (that includes also the free GeoLite Country database). Luckily this is available in the Debian repositories so we can simply install it using:
apt-get install libgeoip1 libgeoip-dev
Next, to install the GeoIP perl module, we need to download the perl module locally fromMaxMind or from CPAN.
The latest version available at the time I wrote this post is Geo-IP-1.27, but you should get the latest version available. Uncompress it and compile it as any perl module:
perl Makefile.PL
make
make test
make install
For other Linux distributions you will have to use the same steps (first install the GeoIP C library and then download and compile the perl module).
Note: if you are using Centos4 then you can skip all these steps as they have packaged the perl module in the “extras” repository so you can install it by simply running:
up2date perl-Geo-IP



On Ubuntu 8 you can also install the GeoIP perl library from a package repository:
apt-get install libgeo-ip-perl

Speedup Awstats by using GeoIP instead of DNS Lookups | MDLog:/sysadmin

Speedup Awstats by using GeoIP instead of DNS Lookups | MDLog:/sysadmin:


Speedup Awstats by using GeoIP instead of DNS Lookups

Awstats is probably the best open source web stat program available (with other projects likewebalizer and analog, no longer being maintained). Besides many security problems that are found all the time in Awstats, the main problem that bothered me is its speed. Awstats iswritten in perl, and this means that it will be considerably slower than a similar program written in C for example (like webalizer for example). There is not much we can do to speed awstats, but on high traffic sites (with logs growing over a few Gigs of data per day) this can be a real problem…
What I intended to show here is how DNS lookups can affect awstats performance and why you should not use them.
Why would you want DNS lookups enabled anyway? (by the way, in default configuration DNS lookups are off). You might want to use DNS lookups to resolve the IPs to hostnames and awstats will use this information for its country statistics. I don’t see any other reason why you would want to use DNS lookups while running awstats (but hey, I might be wrong, and you might use it for something else ;) ).
Now why is this not a good idea?
  • first of all the country stats will not be very useful. They will not contain very relevant information if looking only on the hostname to find the country…
  • second, awstats doesn’t have a DNS caching mechanism (like webalizer has for example) to keep the IPs already resolved in a local file during consecutive runs. It has a feature that will cache the results in a file, but only during the same run…
    Now this sucks…
  • DNS lookups will considerably slow down awstats because it will have to resolve each IP from the logs to the hostnames, and this operation is ‘expensive’
So if you are using DNS lookups only to get country stats, then stop doing that… Start usingGeoIP to get much faster results, and much more accurate reports.
To enable the awstats GeoIP plugin you need the perl port of the Maxmind GeoIP library installed first, and you can use the free country database (that in my opinion provides great results). The installation of the perl GeoIP module is peaty simple (just be sure to install theGeoIP C library first), and in case you want to see some short details on the installation you can check this small post: “Install Geo::IP Perl Module on Debian”.
There is also the possibility to use the smaller perl module GeoIP::Free but from my experience the GeoIP module gives much more accurate results.
Once you have the GeoIP perl module installed all you have to do is:
  • have DNS lookups off:
    DNSLookup=0
  • enable the GeoIP plugin:
    #LoadPlugin="geoip GEOIP_STANDARD /pathto/GeoIP.dat"
    LoadPlugin="geoip GEOIP_STANDARD /usr/share/GeoIP/GeoIP.dat"
Now in order to demonstrate this statement, I have done some tests on a large apache log file ~1G (gathered on a live server, with real data). The same identical file was used for each one of the tests.
Hardware: CPU: 2 x Xeon CPU 3.20GHz, RAM: 4G RAM, HDD: SCSI
Log file: 1,8G size, 7169736 records
Here are the results:
  • Test1: no GeoIP and no DNS lookups:
    - time to complete 15m56s 
    - 7499 records / sec
  • Test2: no GeoIP and DNS lookups:
    - time to complete 66m58s
    - 1784 records / sec
  • Test3: GeoIP and no DNS lookups:
    - time to complete 17m32s- 6815 records / sec
Conclusion: as we can see Test1 is the fastest (obviously as no DNS or GeoIP lookups are done). This is the base on how fast awstats can run on this box. As we can see Test2 (with DNS lookups) takes approximately 4 times more to complete (320% decrease in speed), while Test3 (with GeoIP lookups) shows a decrease in performance of 10% (approx. the same as stated in the awstats docs, where it states 8%). Given the advantages GeoIP has over DNS lookups I think that this shows clearly the advantage of using GeoIP.
For official awstats benchmarks, you can check this page:
http://awstats.sourceforge.net/docs/awstats_benchmark.html