> For the complete documentation index, see [llms.txt](https://adamcysec.gitbook.io/csapp/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://adamcysec.gitbook.io/csapp/overview/features/pypi_data_harvest.py.md).

# pypi\_data\_harvest.py

This tool handles scraping package data from PyPI

{% hint style="info" %}
**pypi\_data\_harvest.py requires an API key from** [**libraries.io**](https://libraries.io/)
{% endhint %}

## Description

pypi\_data\_harvest.py gathers package info from libraries.io and scrapes data from pypi.org for additional metadata. 

By default a new data set CSV file is created, however most of the time you will use the `--update` flag for updating an existing data set CSV file. 

New Python packages are uploaded every day, therefore you will want to update before use.

## Dependencies

pypi\_data\_harvest.py requires the following dependencies:

* [csapptools](https://pypi.org/project/csapptools/)
  * `pip install csapptools`
* [beautifulsoup4](https://pypi.org/project/beautifulsoup4/)
  * `pip install beautifulsoup4`
* [requests](https://pypi.org/project/requests/)
  * `pip install requests`

## Usage

Parameter **--update, -u**

* type : str
* CSV file path to update

Parameter **--apikey, -k**

* type : str
* optional file path to libraries.io api key

Parameter **--verbose, -v**

* type : bool
* True : prints verbose output

#### Example 1

```python
py pypi_data_harvest.py
```

* Starts scraping package data into a new CSV file

#### Example 2

```python
py pypi_data_harvest.py --update "pypi_info_db.csv"
```

* Updates an existing CSV file with new package data

#### Example 3

```python
py pypi_data_harvest.py -u "pypi_info_db.csv" -k "C:\\apikey.txt" -v
```

* Updates an existing CSV file with new package data
* Uses -k to pass in libraries.io api key from a different path
* Prints verbose output

## Configure libraries.io API Key

{% hint style="info" %}
**pypi\_data\_harvest.py requires an API key from** [**libraries.io**](https://libraries.io/)
{% endhint %}

1. Create an account on [libraries.io](https://libraries.io/)
2. Create hidden directory in your user's home folder called `.librariesio`
3. Saved libraries.io api key in a txt file called `api_key.txt`