Pandas download file from s3

It seems that I need to configure pandas to use AWS credentials, but don't know how. You might be able to install boto and have it work correctly. There's some troubles with boto and python 3. If you're on those platforms, and until those are fixed, you can use boto 3 as.

That obj had a. Pandas now uses s3fs to handle s3 coonnections. However, since s3fs is not a required dependency, you will need to install it separately, like boto in prior versions of pandas. If you have already installed s3fs pip install s3fs then you can read the file directly from s3 path, without any imports:. In addition to other awesome answers, if a custom endpoint is required, it is possible to use pd.

Also see: s3fs custom endpoint url. The previous answers are a good basic start but I wanted to achieve advanced objectives stated below. Overall I feel awswrangler is the way to go. For distributed compute with dask, this worked but it uses s3fs afaik and apparently gzip can't be parallized. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams?

Collectives on Stack Overflow. Learn more. Reading a file from a private S3 bucket to a pandas dataframe Ask Question. Asked 5 years, 8 months ago.

Active 2 months ago. Viewed 69k times. I have configured the AWS credentials using aws configure. Improve this question. IgorK IgorK 1 1 gold badge 4 4 silver badges 8 8 bronze badges.

Add a comment. Active Oldest Votes. Improve this answer. TomAugspurger TomAugspurger Works great. Two things: 1. I have boto installed and it imports fine as well as pandas, but still I get the I am trying this method with latest version of pandas 0. Is there a way to convert to stringIO? Sign in Sign up. Instantly share code, notes, and snippets. Last active Nov 8, Code Revisions 2 Stars 55 Forks Embed What would you like to do?

Embed Embed this gist in your website. Share Copy sharable link for this gist. Learn more about clone URLs. Download ZIP. This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.

Learn more about bidirectional Unicode characters Show hidden characters. This comment has been minimized.

Sign in to view. Copy link Quote reply. What is key here?