# GutenbergPy

> GutenbergPy is a Python library for working with Project Gutenberg books and metadata through a fast SQLite or MongoDB cache.

- Author: Radu Angelescu
- Published: 2017-02-20
- Topic: gutenberg
- Web page: https://raduangelescu.com/post/gutenbergpy/



I have created a library used for interfacing with [Gutenberg](http://www.gutenberg.org/) from python code.
This is the first article about [GutenbergPy](https://github.com/raduangelescu/gutenbergpy)



## Why use my library ?

* Only needs **lxml**  (pymongo only if you use [mongodb](https://www.mongodb.com/))
* [SQLite](https://www.sqlite.org/) cache build time: **about 2 minutes** (instead of more than one day)
* [SQLite](https://www.sqlite.org/) cache size:  **about 120 mb**
* [Mongodb](https://www.mongodb.com/) cache build time: **about 3 minutes** (will probably be less in the future, as it's not optimized)
* [Mongodb](https://www.mongodb.com/) cache size: **about 300 mb** (instead of 2 Gb berkley db previous solution)
* **Fast queries on both solutions** 

<!--more-->


## Just try it !

If you need to use gutenberg meta information and get filtered texts for your project, fast, reliable, without overloading gutenberg servers: try [GutenbergPy](https://github.com/raduangelescu/gutenbergpy).

Check the usage by reading the [Github page](https://github.com/raduangelescu/gutenbergpy)

Install the pypi package: 
```BashSession  
        pip install gutenbergpy
```
Or if you are feeling adventurous you can install from source: 
        
```bash      
        git clone https://github.com/raduangelescu/gutenbergpy
        python setup.py install
```

## Why did you do it?

I decided to continue improving [artisticmachine42](http://artistmachine42.tumblr.com/) quote capabilities. 
As I said in a [previous article]{{< ref "/post/tumblrartisticmachine" >}}, Started doing a simple [reverse index](https://en.wikipedia.org/wiki/Reverse_index) for fast text searching **->** wrote a library to help me scavenge for big-data from the [gutenberg](http://www.gutenberg.org/) site.

Writing something like a reverse index, implies testing it on big data (in my case a lot of books). 
My go-to data source for my quote extraction was [Gutenberg](http://www.gutenberg.org/) (great initiative, I am a big fan). 

List of linked facts that made me do it :)  

* Searched the web for a good Python library for [Gutenberg](http://www.gutenberg.org/) and found: [this](https://pypi.python.org/pypi/Gutenberg)

* Tried using the sqlite interface (great sqlite fan here :) ) after about 1 day + 1 night of running **->** it still didn't populate the cache.

* Thought the sqlite cache feature was added later so it must be still in development **->** used the bsd db cache **->** took around **5 hours** to populate (on a really good pc with a really good internet connection) and **2 GB** of hdd space

* I then proceeded to query all the books that are from a certain category with a certain language **->** it failed. 

* Reported bug to developer

* Read "the maintainer wanted to no longer maintain the project" **->** thought it will never get fixed unless I do something about it

* Downloaded the repository and started debugging **->** saw nasty stuff, the cache base needed to be rewritten (in my opinion)

* I decided to write my own and publish for everyone to use **->** help the open source community

* Wrote it **->** published. 

* Published [GutenbergPy](https://github.com/raduangelescu/gutenbergpy) **->** maintainer of old library said he fixed the query bug :)  (didn't test it yet) 

        
## What about the other promised articles?

I had some crunch time at work so I didn't have the time to write the other (bigger) articles on my site, but rest easy:


* The gabor feature generation code is ready (writing a CUDA port). I didn't forget, the article is coming soon :) . 

* The reverse index code is also ready so an article about that will also be coming soon. :)
